An efficient human pose estimation method based on decision network and refined features

Abstract

The application discloses a kind of high-efficiency human posture estimation method based on decision network and detailed feature, this method with HRNet-s as backbone network, extract visual feature and send into coarse-grained decision network to remove redundant information, through CG Encoder module, key point token is perceived to the information contained in coarse-grained feature, then send into fine-grained decision network to further segment feature, then through FG Encoder module, fine inference is carried out, so that key point token is perceived to the information contained in fine-grained feature, and send into MLP module, key point token is mapped back to two-dimensional heat map, finally, very competitive result is obtained on COCO and MPII dataset.Therefore, the application can greatly reduce the calculation complexity, and also improve the accuracy, so that the network can estimate human posture with higher efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and human pose estimation, and particularly relates to an efficient human pose estimation method based on a decision network and refined features. BACKGROUND

[0002] Two-dimensional human pose estimation aims to locate the joint positions of human bodies from a given picture. With the development of deep neural networks, human pose estimation models have made great progress in both speed and accuracy, thereby being widely applied in real life, such as intelligent monitoring and motion analysis.

[0003] In the past decade, convolutional neural networks (CNN) have always dominated the field of human pose estimation. However, researchers soon found that CNN is a sparse connection network with local convolution, and therefore cannot well capture the global dependence information of an image. For the complex detection task of human pose estimation, it is very important to fully capture the information contained in the image.

[0004] The Transformer network based on attention mechanism has achieved great success in the field of NLP. Compared with CNN, Transformer has longer dependency modeling ability and stronger global modeling ability. Many studies have shown that Transformer has a higher upper limit and can achieve better results than CNN. Therefore, Transformer has been gradually introduced into the field of human pose estimation. Literature [Yang, Sen, et al. Transpose: Keypoint localization via transformer. in ICCV. 2021, pp. 11802-11812] first introduced Transformer into human pose estimation. The model first extracts features through the backbone network, then flattens the feature map into Token form and sends it into Transformer, and finally predicts the key points through the heat map method. The authors achieved the SOTA effect at that time through such a simple framework. Then, literature [Mao, Weian, et al. Tfpose: Direct human pose estimation with transformers. arXiv:2103.15320 (2021)] removed the heat map method and adopted the direct regression key point method on the basis of TransPose. Literature [Li, Yanjie, et al. Tokenpose: Learning keypoint tokens for human pose estimation. in ICCV. 2021, pp. 11313-11322] rearranged the visual features into one-dimensional Token through CNN and sent them into the Transformer for feature learning. The authors also proposed Keypoints Token to represent each key point entity to explicitly model the constraint relationship between visual clues and human bodies. Literature [Yuan, Yuhui, et al. Hrformer: High-resolution vision transformer for dense predict. in NeurIPS. 2021, 7281-7293] introduced a hierarchical Transformer encoder. Lower transformers process high-resolution low-semantic information, and higher transformers process low-resolution high-semantic information. Unlike the previous methods, HRFormer adopts a pure Transformer structure, which is different from the previous methods that use CNN as the encoder to further improve the information capturing ability of the network.Although the Transformer has made great progress in human pose estimation, the shortcomings are also obvious. The number and length of Token sequences greatly affect the complexity of the model, and this shortcoming will be more pronounced in high-resolution input images or multi-view scenarios. In order to reduce the complexity of the model, the above work will adopt a pruned backbone network to balance the computational overhead of the Transformer. For example, in TokenPose, the authors only use the first three stages of HRNet, and discard the computationally expensive fourth stage. Although this approach allows the entire model to achieve the highest accuracy while achieving an ideal computational complexity, this computational complexity is obtained by pruning the CNN network, while the Transformer has not essentially changed.

[0005] In fact, there are many researchers following up on accelerating Vision Transformer, for example, some researchers focus on distillation and quantization, which are traditional methods, and more researchers focus on Token pruning. Specifically, in the literature [Yuan, Li, et al. Tokens-to-token vit: Training vision transformers from scratch on imagenet. in ICCV. 2021, pp. 558-567], the number of Tokens is reduced by concatenating adjacent Tokens into one Token; in the literature [Rao, Yongming, et al. Dynamic vit: Efficient vision transformers with dynamic token sparsification. in NeurIPS. 13937-13949], the authors prune Tokens through a learnable Token selector; in the literature [Meng, Lingchen, et al. Adavit: Adaptive vision transformers for efficient image recognition. in CVPR. 2022, pp. 12309-12318], the authors propose an end-to-end learning framework that adaptively selects the number of Patches, Heads, and Blocks used in ViT based on the changes in image input to improve the inference efficiency of image recognition while maintaining accuracy. In the literature [Chen, Mengzhao, et al. Cf-vit: A general coarse-to-fine method for vision transformer. in AAAI. 2023, Vol. 37. No. 6], the authors slice the information region through global class attention in a fine-grained manner and connect the two-stage Vision Transformer through a reuse mechanism, achieving a good balance between performance and efficiency. Back to the field of human pose estimation, in the literature [Ma, Haoyu, et al. Ppt: Token-pruned pose transformer for monocular and multi-view human pose estimation. in ECCV, 2022], the authors added a human token identification (HTI) module and pruned the number of Tokens through the attention operation therein, although such a method can greatly improve the computational speed of the network, it is achieved at the expense of a small amount of accuracy. SUMMARY

[0006] In view of the above, the present application provides an efficient human pose estimation method based on decision network and refined features. The two-stage Transformer is used to crop the human contour from coarse to fine, which can greatly reduce the computational complexity while improving the accuracy, so that the network can estimate the human pose with higher efficiency.

[0007] An efficient human pose estimation method based on decision network and refined features, comprising the following steps:

[0008] (1) Obtain a large number of pictures containing multiple people, and label the joint positions of the people in the pictures;

[0009] (2) Divide all the pictures into training set, validation set and test set according to the proportion;

[0010] (3) Construct a CFPose (Efficient PoseNet with Course to Fine Transformer) model framework, which includes:

[0011] a backbone network for extracting visual features from the input picture, and generating a series of visual tokens and key point tokens by cutting, flattening and linear mapping the extracted feature map;

[0012] a coarse-grained decision network for scoring the importance of each visual token, and removing the visual tokens with lower importance scores to preliminarily crop the human mask;

[0013] a coarse-grained feature encoding module (CG Encoder) for concatenating the key point tokens and the visual tokens retained after screening and performing coarse encoding reasoning, so that the key point tokens can fully perceive the information contained in the coarse-grained features;

[0014] a fine-grained decision network for further cutting and scoring the importance of the visual tokens retained after screening, and removing the visual tokens with lower importance scores;

[0015] a fine-grained feature encoding module (FG Encoder) for concatenating the key point tokens in the output encoding information of the coarse-grained feature encoding module and the visual tokens retained after secondary screening, and performing fine encoding reasoning, so that the key point tokens can fully perceive the information contained in the fine-grained features;

[0016] an MLP module for decoding the prediction results of the joint position coordinates in the input picture from the key point tokens in the output encoding information of the fine-grained feature encoding module through the heat map method;

[0017] (3) Using the training set images and their annotation information to train the above CFPose network model;

[0018] (4) Input the test set image into the trained CFPose network model to predict the position coordinates of each joint point of the character in the image.

[0019] Furthermore, in order to balance the cost of the Transformer module, the backbone network only uses the first three stages of the HRNet network pre-trained on the ImageNet dataset, namely HRNet-s, to extract features from the input image and generate feature maps, and then divide the feature maps of each channel into grid, P h and P w are the height and width of the grid respectively, and then each grid is flattened along the channel dimension to a length of P h ×P w ×C one-dimensional sequence, and then the flattened sequence is mapped to a visual token through a linear layer, and a total of N visual tokens are obtained. At the same time, M key point tokens are initialized, and their sequence length is consistent with the visual token, where M is the number of joint points in the input image, and H, W, and C are the height, width, and channel dimensions of the feature map, respectively.

[0020] Furthermore, the coarse-grained decision network uses a self-attention mechanism to calculate and evaluate the importance score of each visual token, prunes the visual tokens with lower importance scores, and only retains the visual tokens with higher importance scores.

[0021] Furthermore, the coarse-grained feature encoding module is composed of a cascade of multiple layers of Transformers, whose input is the concatenation of cropped and retained visual tokens and initialized key point tokens. Each layer of Transformer is composed of a regularization layer L1, a multi-head self-attention mechanism layer, a regularization layer L2, and a feedforward neural network connected in sequence, wherein the input of the regularization layer L2 is the output of the multi-head self-attention mechanism layer superimposed with the input of the regularization layer L1, and the output of the feedforward neural network is superimposed with the input of the regularization layer L2 as the output of the Transformer.

[0022] Further, the specific operation process of the fine-grained decision network is that: for the visual tokens retained by the coarse-grained decision network, the feature map grid corresponding to the visual tokens is further uniformly divided into 2x2 small grids, then each small grid is flattened along the channel dimension and mapped to a visual token by a linear layer, which is equivalent to 1 visual token splitting into 4 visual tokens; finally, a self-attention mechanism is used to calculate the importance score of each visual token, and the visual tokens with low importance score ranking are pruned, and only the visual tokens with high importance score ranking are retained.

[0023] Further, the fine-grained feature encoding module is composed of multiple layers of Transformer cascades, and the input is the key point token in the encoding information output by the coarse-grained feature encoding module and the visual token retained by the fine-grained decision network.

[0024] Further, the multi-head self-attention mechanism layer is composed of multiple Self-Attention, and each Self-Attention multiplies the input of the multi-head self-attention mechanism layer with linear transformation matrices W Q , W K and W V to obtain query vector matrix Q, key vector matrix K and value vector matrix V respectively, then the output result of Self-Attention is calculated by the following formula, and finally the output results of multiple Self-Attention are spliced and passed through a linear layer to obtain the output of the multi-head self-attention mechanism layer.

[0025]

[0026] Wherein: Attention(Q, K, V) is the output result of Self-Attention, s is the column number of query vector matrix Q, T represents transposition.

[0027] Further, the heat map regression module takes the key point token in the encoding information output by the fine-grained feature encoding module as input, and then maps it back to a two-dimensional heat map form through a multi-layer perceptron. In the multi-layer perceptron, the key point token is first mapped back to a two-dimensional heat map P, and the size of the heat map is MxH * xW * , H * and W * are 1 / 4 of the height H and width W of the input picture respectively; finally, the heat map P is reshaped into M heat maps, and the heat map size is the same as the original Figure One On this basis, the coordinates of the joints of the human body are located by finding the maximum response position on the heat map.

[0028] Further, the specific implementation of the step (4) is as follows:

[0029] 4.1 Initialize model parameters, including the bias vector and weight matrix of each layer, learning rate and optimizer;

[0030] 4.2 Input the picture of the training set into the model, and the model forward propagation outputs the corresponding prediction result, i.e. the node position coordinates, and calculates the loss function between the prediction result and the annotation information;

[0031] 4.3 According to the loss function (such as MSE), the model parameters are continuously iteratively updated by using the optimizer (such as Adam) through the gradient descent method until the loss function converges, and the training is completed.

[0032] The application preliminarily extracts the visual features of the image through the backbone network, removes the redundant information by using the coarse-grained decision network, and roughly crops the human contour, then the coarse-grained token and the key point token are sent into the two-stage Transformer together, at the end of the first stage CG Encoder, the key point token has learned the information contained in the coarse-grained feature, at this time we only keep the key point token, and the fine-grained feature is sent into the second stage FGEncoder for training, wherein the fine-grained feature comes from the fine-grained decision network, the function of the network is to further segment the feature to refine the human contour and further remove the redundant information; in the last stage of the CFPose, the key point token output by the Transformer is mapped back to the two-dimensional heat map through the MLP to predict the node, through the above design, the CFPose has achieved impressive results, we have fully verified on the COCO dataset, compared with the baseline network, the computational complexity of the CFPose is reduced by 43%, and the accuracy of the network is further improved to 76.2%; in addition, we also test on the MPII dataset, and all have achieved competitive results. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 It is a comparison diagram of the CFPose model of the application and other Pose Transformer type network frameworks.

[0034] Figure 2 It is a network structure diagram of the CFPose model of the application.

[0035] Figure 3 It is a broken line statistical diagram of the efficiency and accuracy of the mainstream human pose estimation method.

[0036] Figure 4The following is a qualitative comparison of CFPose-B and TokenPose-B on the COCO dataset. DETAILED DESCRIPTION

[0037] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] The present invention provides an efficient human posture estimation method based on a decision network and refined features, comprising the following steps:

[0039] (1) Obtain a large number of pictures containing multiple characters, mark the positions of the characters' joints in the pictures, and then divide all the pictures into training set, validation set and test set according to the proportion.

[0040] (2) Construct Figure 2 The CFPose model framework shown in the figure includes a backbone network, a coarse-grained decision network, a CGEncoder module, a fine-grained decision network, an FG Encoder module, and an MLP module, where:

[0041] The backbone network is used to extract visual features from the input image. In this embodiment, a CNN-based backbone network is used to generate a channel number C and a size of the original Figure 1 / 4 feature map, we use the first three stages of HRNet as our backbone network and named it HRNet-s, which has only 25% of the original parameters to balance the computational overhead of the Transformer part. Assuming the shape of the feature map is Next, the network will transform each size H * ×W * The feature map is divided into grids, where P h ,P w The height and width of each grid are then expanded along the channel dimension. The length of each sequence after expansion is P h ×P w ×C; Since the computational complexity of Transformer is directly related to the length of Token, in order to obtain Visual Token, it is necessary to linearly map the above one-dimensional sequence into a Visual Token with a sequence length of L = 192, which is recorded as At the same time, the network will randomly initialize and generate M learnable Keypoint Tokens as key point entities. KeypointToken can also be understood as a category token, which is used to learn the constraint relationship between key points and the appearance clues from the image.

[0042] Coarse-Grained Decision Network is used to remove redundant information and roughly crop the human contour. For the input V ∈ R N×L , it is sent into a self-attention mechanism to evaluate the importance score of each Visual Token. The network generates a binary decision mask ∈ {0, 1}, and according to the proportion α ∈ (0, 1), it determines which Visual Token is not important and can be discarded; among the N × (1-α) Visual Tokens that need to be discarded, they are filled with 0, which is only used for placeholder after padding and does not participate in subsequent calculations. For the remaining N × α Visual Tokens, they are renamed as Coarse-Grained Token (CG Token), and then position encoding is performed on them. The two-dimensional position embedding pe i can be used to mark the position of each sequence of the input {CG Token} = {V1+pe1, V2+pe2, …, V N×α +pe N×α}, next, the Keypoint Token and the CG Token after position encoding are spliced together and sent into the CG Encoder for training. At this time, the outline of the human is preliminarily cropped through the screening of the Coarse-Grained Decision Network, so that the Transformer network can pay more attention to the human itself, greatly reducing the influence of background noise.

[0043] The CG Encoder module is used for coarse reasoning, so that the Keypoint Token can fully perceive the information contained in the coarse-grained features. The input in the CG Encoder stage is d1 = (N × α) + M, we project T1 into three matrices of the same size through three linear mappings, that is, generate query Q, key K, and value V, and then send the three matrices into the multi-head self-attention module to calculate the attention score:

[0044]

[0045] , where: d k is the dimension of the key.

[0046] Each Attention score determines the attention level of the current query Token, and then it is sent into the FeedForward, which is composed of two fully connected layers with a GELU activation layer in between; through the above operation, the network can quickly identify the features of simple human instances at a lower computational cost and associate them to the Keypoint Token.

[0047] Fine-grained decision network is used to further segment the features to refine the human contour and further remove redundant information. The multi-head self-attention mechanism in the last layer of the CG Encoder module is used as the core component of the fine-grained decision network. First, the N x a grids screened out by the coarse-grained decision network are further uniformly divided to generate 4N x a grids, then they are expanded along the channel dimension and mapped linearly to generate 4N x a Visual Tokens, at this time the length of the Token is L = 96, then the multi-head self-attention mechanism is used to score each Token, at this time the network will still generate a binary decision mask, and according to the proportion β ∈ (0, 1) to determine which Visual Token is not important and can be discarded, and the discarded 4N x a x (1-β) Visual Token is filled with 0, and the reserved 4N x aβ Visual Token is renamed as Fine-Grained Token (FG Token). After the same position coding as above is completed, we linearly map the Keypoint Token output by the CG Encoder to make its size consistent with the FG Token, then concatenate and send it into the FG Encoder for training to further refine the network.

[0048] The FG Encoder module is used for fine inference, so that the Keypoint Token can fully perceive the information contained in the fine-grained features. Let the input be d2 = (4N x aβ) + M, at this time the Keypoint Token is the output part of the CG Encoder, then we repeat the above calculation process to further improve the performance of the network.

[0049] The MLP module maps the Keypoint Token output by the FG Encoder module back to a two-dimensional heat map. In the output stage of the FG Encoder, we only take M Keypoint Tokens as output, then map them back to a two-dimensional heat map form through a multi-layer perceptron. Specifically: assuming that the output of the Transformer is where M is the number of Tokens and L is the length of the Token. In the multi-layer perceptron, X is first mapped back to two dimensions to generate where H * ,W * are 1 / 4 of the original image size, then they are reshaped to At this time, the size of the heat map is the same as the original Figure OneFinally, on this basis, we locate the coordinates of the human joint points by finding the maximum response position on the heat map. In addition, in the training process, we use the MSE loss function to train the network, and the calculation formula of the MSE loss is as follows:

[0050]

[0051] Wherein: M is the number of joint points (COCO: M = 17, MPII: M = 16), P i is the i th real coordinate generated key point heat map, is the predicted i th joint heat map.

[0052] (4) The model framework is trained by using the training set pictures and their labeled information in step (1).

[0053] (5) The test set pictures are input into the trained model, and the position coordinates of each joint point of the pictures can be predicted.

[0054] As can be seen from the above, the human pose estimation network based on the Transformer is redesigned, such as Figure 1 It can be seen that the model of the present application is different from other network models, specifically, the model first extracts features through CNN, then divides each feature map output into a plurality of grids, then expands along the channel dimension, and generates Visual Token through a linear layer, then sends these Visual Token into the coarse-grained decision network for importance scoring, and masks the confidence score lower, and preliminarily cuts out the outline of the human body, then sends the CGToken with a higher confidence score and the Keypoint Token into the first stage CG Encoder of the Transformer for rough reasoning, at the end of the CG Encoder, the Keypoint Token has fully perceived the coarse-grained information contained in the CG Token, we only take the Keypoint Token at this time as the output of the CG Encoder, at the same time we further cut out the fine-grained human slice through the fine-grained decision network, then send the FG Token and the Keypoint Token into the FG Encoder with more layers, at the end of the FG Encoder, the Keypoint Token is mapped back to the two-dimensional heat map through the MLP, and the final joint point coordinates are predicted through the two-dimensional heat map.

[0055] Verification example:

[0056] We divide CFPose into three versions as shown in Table 1. When the Transformer Layer is 12, we use the first 3 layers as CG Encoder and the last 9 layers as FG Encoder. When the Transformer Layer is 6, we use the first 2 layers as CG Encoder and the last 4 layers as FG Encoder.

[0057] Table 1

[0058]

[0059] During the experiment, we cropped the input single-person images to a uniform size (256×192 for the COCO dataset and 256×256 for the MPII dataset). We used the Adam optimizer with an initial learning rate of 1e-3 and a total of 300 epochs. The hardware environment for the experiment consisted of eight NVIDIA RTX 3090 24GB graphics cards and a 48-core Intel(R) Xeon(R) E5-2678 v3 @ 2.50GHz. The software environment used was Pytorch 1.13.1 and Cuda 11.6 with Cudnn 8.3.2.

[0060] As shown in Table 2, we evaluated the performance of CFPose on the mainstream COCO dataset. It can be seen that compared with HRNet-W48 in the CNN-Base model, CFPose-B has an accuracy improvement of 0.8% while reducing GFLOPs by 67%. Compared with TransPose-H-A6 in the CNN+Transformer type, our CFPose-B has a computational complexity reduction of 78% while maintaining almost the same accuracy. T It is only 7% of TransPose-H-A6. Compared with various versions of the baseline model TokenPose, our CFPose has significantly improved in both speed and accuracy; and compared with the lightweight human pose estimation model EfficientPose-C, our CFPose-S has achieved a significant lead in accuracy with a 2.2% improvement while slightly reducing the computational complexity. In addition, through Figure 3 The excellent performance of CFPose can also be seen more intuitively.

[0061] Table 2

[0062]

[0063] Note: The best result in each column of the table is marked in bold, and the second best result is underlined. GFLOPsT For the computational complexity of the Transformer part, Trans represents the Transformer, and SimpleBa represents the SimpleBaseline.

[0064] As shown in Table 3, we also evaluated the performance of CFPose on the MPII dataset. Since the outlines of the figures in the MPII dataset are relatively clear, the prediction accuracy of the model for each joint tends to be saturated. We tested CFPose-B, and it was significantly better than other mainstream human pose estimation models in all indicators except for Hea.

[0065] Table 3

[0066]

[0067] Note: The best result in each column is marked in bold, and the second best result is underlined.

[0068] In addition, we also conducted qualitative experimental result analysis and visualized the test results of CFPose-B and the baseline model TokenPose-B on COCO for comparison, as shown in Fig. 2. Figure 4 As can be seen from the first picture from left to right, in the scene where the features of the black and white pictures are easily confused, TokenPose incorrectly predicted the position of the woman's right leg. As can be seen from the second and third pictures, in the occlusion scene, TokenPose either has unreasonable prediction or missed prediction. As can be seen from the fourth picture, when the figure is small, TokenPose's prediction is slightly difficult. However, CFPose can well cope with the above special scenes, and is superior to the baseline model TokenPose in prediction accuracy and rationality.

[0069] In the present application, we propose a scale factor a to control the retention ratio of CG Tokens in the coarse-grained decision network. In order to verify what value of a is the most appropriate, we use CFPose-B to conduct an ablation experiment on the COCO dataset, and the results are shown in Table 4. As can be seen, when a = 0.5, the proportion is too low to mask a part of useful Tokens, resulting in a decrease in network accuracy. Therefore, a = 0.6 is the optimal value.

[0070] Table 4

[0071]

[0072] Meanwhile, another proportion coefficient β is proposed in the application to control the retention proportion of FG Token in the fine-grained decision network, and the same experimental strategy as described above is adopted, and the results are shown in Table 5, and it can be seen that when β is less than 0.7, useful information will be masked, and therefore the best effect is obtained when β = 0.7.

[0073] Table 5

[0074]

[0075] In addition, we also conducted ablation experiments on CG Token and FG Token to verify that the architecture of Course to Fine proposed in the application is effective, and the same experimental strategy as described above is adopted, and the experimental results are shown in Table 6, and it can be seen that when the idea of Course to Fine is not adopted, that is, the Baseline method achieves 74.7% AP and 5.7 GFLOPs, and from the Base method and the method 1, it can be seen that the coarse-grained decision network contributes 0.4% to the accuracy of CFPose, and the computational complexity is reduced by 0.4. From the method 1 and the method 2, it can be seen that the fine-grained decision network contributes 0.7% AP to the accuracy of CFPose, and the computational complexity is further reduced by 0.5. Therefore, the two-stage gradual refinement strategy is beneficial to the overall performance of the network, and the contribution of FG Token is the largest.

[0076] Table 6

[0077]

[0078] Summary:

[0079] The model of the application takes HRNet-s as the backbone network, extracts visual features, and sends them to the coarse-grained decision network to remove redundant information, and the CG Encoder module enables the key point token to perceive the information contained in the coarse-grained features, and then sends them to the fine-grained decision network for further segmentation, and then the FG Encoder module is used for fine reasoning, so that the key point token perceives the information contained in the fine-grained features, and then the MLP module is used, the key point token is mapped back to the two-dimensional heat map, and finally very competitive results are obtained on the COCO and MPII datasets, and we also conduct sufficient ablation experiments to make each module of the network more convincing; In addition, the application also visualizes the experimental results, which can more intuitively see the performance improvement of CFPose.

[0080] The above description of the embodiments is for the purpose of enabling one of ordinary skill in the art to make and use the application and is not intended to limit the application as construed in the broadest scope possible. Inasmuch as modifications to the above described embodiments can readily be made by persons of ordinary skill in the art, it is intended that the application not be limited to the embodiments described above but should be construed in the broadest scope possible.

Claims

1. An efficient human pose estimation method based on a decision network and refined features, comprising the following steps: (1) Obtain a large number of pictures containing multiple characters and mark the joint positions of the characters in the pictures; (2) Divide all images into training set, validation set and test set in proportion; (3) Construct the CFPose model framework, which includes: The backbone network is used to extract visual features from the input image and generate a series of visual tokens and key point tokens by segmenting, flattening, and linear mapping the extracted feature map; A coarse-grained decision network is used to score the importance of each visual token and remove visual tokens with low importance scores to preliminarily crop the human body mask; The coarse-grained feature encoding module is used to concatenate key point tokens with the filtered and retained visual tokens and then perform coarse coding inference, so that the key point tokens can fully perceive the information contained in the coarse-grained features; A fine-grained decision network is used to further segment and score the importance of the retained visual tokens, and then remove visual tokens with low importance scores; The fine-grained feature encoding module is used to perform fine-grained encoding reasoning by concatenating the key point tokens in the encoded information output by the coarse-grained feature encoding module with the visual tokens retained by the secondary screening, so that the key point tokens can fully perceive the information contained in the fine-grained features; The MLP module is used to decode the key point tokens in the encoded information output by the fine-grained feature encoding module using the heat map method to obtain the predicted results of the position coordinates of each joint point in the input image; (3) Using the training set images and their annotation information to train the above CFPose network model; (4) Input the test set image into the trained CFPose network model to predict the position coordinates of each joint point of the character in the image.

2. The method for efficient human posture estimation according to claim 1, wherein: The backbone network only uses the first three stages of the HRNet network pre-trained with the ImageNet dataset to extract features from the input image to generate feature maps, and then divides the feature maps of each channel into grid, P h and P w are the height and width of the grid respectively, and then each grid is flattened along the channel dimension to a length of P h ×P w ×C one-dimensional sequence, and then the flattened sequence is mapped to a visual token through a linear layer, and a total of N visual tokens are obtained. At the same time, M key point tokens are initialized, and their sequence length is consistent with the visual token, where M is the number of joint points in the input image, and H, W, and C are the height, width, and channel dimensions of the feature map, respectively.

3. The efficient human posture estimation method according to claim 1, wherein: The coarse-grained decision network uses a self-attention mechanism to calculate and evaluate the importance score of each visual token, prunes the visual tokens with lower importance scores, and only retains the visual tokens with higher importance scores.

4. The method for efficient human posture estimation according to claim 1, wherein: The coarse-grained feature encoding module is composed of a cascade of multiple layers of Transformers, whose input is the concatenation of cropped and retained visual tokens and initialized key point tokens. Each layer of Transformer is composed of a regularization layer L1, a multi-head self-attention mechanism layer, a regularization layer L2, and a feedforward neural network connected in sequence. The input of the regularization layer L2 is the output of the multi-head self-attention mechanism layer superimposed with the input of the regularization layer L1, and the output of the feedforward neural network is superimposed with the input of the regularization layer L2 as the output of the Transformer.

5. The efficient human posture estimation method according to claim 1, wherein: The specific operation process of the fine-grained decision network is as follows: for the visual tokens retained by the coarse-grained decision network, the feature map grids corresponding to these visual tokens are further evenly divided into 2×2 small grids, and each small grid is flattened along the channel dimension and mapped to a visual token through a linear layer, which is equivalent to splitting one visual token into four visual tokens; Finally, the self-attention mechanism is used to calculate and evaluate the importance score of each visual token, and the visual tokens with lower importance scores are pruned, leaving only the visual tokens with higher importance scores.

6. The efficient human posture estimation method according to claim 1, wherein: The fine-grained feature encoding module is composed of a multi-layer Transformer cascade, and its input is the concatenation of key point tokens in the output encoding information of the coarse-grained feature encoding module and the visual tokens retained by the fine-grained decision network screening.

7. The efficient human posture estimation method according to claim 4, wherein: The multi-head self-attention mechanism layer is composed of multiple Self-Attention layers, each of which combines the input of the multi-head self-attention mechanism layer with the linear transformation matrix W Q 、W K and W V Multiplying them gives the query vector matrix Q, key vector matrix K, and value vector matrix V. Then, the output of Self-Attention is calculated using the following formula. Finally, the outputs of multiple Self-Attention layers are concatenated and passed through a linear layer to obtain the output of the multi-head self-attention mechanism layer. Where: Attention(Q,K,V) is the output of Self-Attention, s is the number of columns in the query vector matrix Q, and T represents transpose.

8. The efficient human posture estimation method according to claim 1, wherein: The heatmap regression module takes the key point tokens in the encoded information output by the fine-grained feature encoding module as input, and then remaps them back to a two-dimensional heatmap form through a multi-layer perceptron. In the multi-layer perceptron, the key point tokens are first mapped back to a two-dimensional heatmap P, whose size is M×H * ×W * , H * and W * They are 1 / 4 of the height H and width W of the input image respectively; finally, the heat map P is reshaped into M heat maps, and the heat map size is consistent with the original image. On this basis, the coordinates of each joint point of the human body are located by finding the maximum response position on the heat map.

9. The efficient human posture estimation method according to claim 1, wherein: The specific implementation of step (4) is as follows: 4.1 Initialize model parameters, including the bias vector and weight matrix of each layer, learning rate, and optimizer; 4.2 Input the training set images into the model, forward propagate the model output to obtain the corresponding prediction results, i.e., the joint position coordinates, and calculate the loss function between the prediction results and the annotation information; 4.3 Based on the loss function, the optimizer is used to iteratively update the model parameters through the gradient descent method until the loss function converges and the training is completed.

Images