Two-dimensional human pose estimation method based on multi-level fusion and discrimination network

Abstract

The application discloses a two-dimensional human body posture estimation method based on a multi-level fusion and discrimination network, which firstly extracts texture information of an image through a high-resolution CNN network, then improves the feature expression capability of the network by fusing multi-level features, and then positions a rough human body contour through a discrimination network, and then is sent into an efficient Transform module for processing; in different stages of the Transform, confidence scores of each Token are scored, and the score size determines the importance of the Token relative to a skeleton key point, and Tokens with low importance are fused into a new Token to reduce information redundancy and improve the calculation efficiency. Finally, the application is tested on mainstream COCO and MPII data sets, and is superior to mainstream most advanced models in terms of calculation complexity and network accuracy.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and human pose estimation technology, specifically involving a two-dimensional human pose estimation method based on a multi-level fusion and discrimination network. Background Technology

[0002] The Transformer is a sequence-to-sequence network that effectively captures contextual information from input sequences, achieving great success in Natural Language Processing (NLP). Researchers quickly extended its application to the visual domain. Its built-in multi-head self-attention module captures long-range dependencies in human pose, which is crucial for accurately predicting relationships between human joints. Consequently, many classic Transformer-based human pose estimation networks have emerged, as follows:

[0003] The paper [Li, Ke, et al. Pose recognition with cascade transformers. in CVPR. 2021, pp. 1944-1953] proposes a method for pose recognition using cascaded Transformers, and uses Transformers as decoders to directly regress keypoints. The paper [Mao, Weiian, et al. Tfpose: Direct humanpose estimation with transformers. arXiv preprint arXiv: 2103.15320 (2021)] transforms the pose estimation task into a sequence prediction problem, using the Transformer's encoding and decoding structure to regress keypoints. The paper [Yang, Sen, et al. Transpose: Keypoint localization via transformer. in ICCV. 2021, pp. 11802-11812] uses a CNN as the encoder and a Transformer to capture long-range dependencies between keypoints. The paper [Li, Yanjie, et al. Tokenpose: Learning keypoint tokens for human poseestimation. in ICCV.2021, pp.11313-11322] also uses a CNN as the encoder and proposes KeypointToken, which uses a Transformer to explicitly learn the constraints between keypoints and capture appearance cues from images. The paper [Yuan, Yuhui, et al. Hrformer: High-resolution vision transformer for dense prediction. in NeurIPS.2021, 7281-7293] proposes a high-resolution Transformer model for dense prediction tasks such as human pose estimation and semantic segmentation, equivalent to a Transformer version of HRNet.

[0004] While the introduction of Transformers has raised the upper limit of human pose estimation networks, it has also brought higher computational complexity, a common problem faced by the aforementioned methods. In particular, the excessive memory consumption of the method in the paper [Xu, Yufei, et al. Vitpose: Simple vision transformer baselines for human pose estimation. in NeurIPS. 2022, 38571-38584] has rendered it impractical. Therefore, some researchers have begun to attempt to achieve a balance between speed and accuracy, proposing some lightweight human pose estimation schemes, as follows:

[0005] The paper [Zhang, Wenqiang, et al. Efficientpose: Efficient human poseestimation with neural architecture search. Computational Visual Media. 2021, 335-347] found an efficient neural network structure for accurately estimating human pose through automated neural architecture search. The paper [Yu, Changqian, et al. Lite-hrnet: A lightweight high-resolution network. in CVPR. 2021, pp. 10440-10450] achieves network lightweighting by introducing conditional channel weighted units to replace the heavy Shuffle blocks in HRNet. This method significantly reduces the number of network parameters and computational complexity while maintaining high-resolution features. The paper [Ma, Haoyu, et al. Ppt: token-pruned pose transformer for monocular and multi-view human pose estimation. in ECCV. 2022, pp. 424-442] introduces a token-pruned pose transformer that performs self-attention operations only within selected tokens, thereby reducing the computational complexity of the Transformer part. In addition, the authors extend PPT to multi-view human pose estimation and propose a new cross-view fusion strategy called human region fusion, which treats all human foreground pixels as corresponding candidates.

[0006] While the aforementioned methods excel in lightweighting—for example, the paper representing pruned CNNs [Yu, Changqian, et al. Lite-hrnet: A lightweight high-resolution network. in CVPR. 2021, pp. 10440-10450], and the paper representing pruned Transformers [Ma, Haoyu, et al. Ppt: token-pruned pose transformer for monocular and multi-view human poseestimation. in ECCV. 2022, pp. 424-442]—they all essentially come at the cost of accuracy, resulting in varying degrees of accuracy reduction compared to baseline models.

[0007] Using Transformers for human pose estimation is a future trend. The core idea of ​​Pose Transformer is to use image patches as visual tokens and model global dependencies in images through a multi-head self-attention mechanism. However, the global attention mechanism has high computational complexity and is easily affected by severe background noise, which poses a serious challenge to Transformer-based human pose estimation networks. Summary of the Invention

[0008] In view of the above, this invention provides a two-dimensional human pose estimation method based on a multi-level fusion and discriminative network. The algorithm model MADN (Multi-Level Aggregated Pose Transformer with Discriminative Network) used in this invention is superior to the mainstream state-of-the-art models in terms of both computational complexity and network accuracy, and can perform high-quality human pose estimation tasks very well.

[0009] A two-dimensional human pose estimation method based on a multi-level fusion and discriminant network includes the following steps:

[0010] (1) Obtain a large number of images containing people and mark the key points of the people in the images;

[0011] (2) Construct the MADN network model framework, which includes a backbone network, a token generator, a feature fusion module, a discriminator network, an efficient Transformer module, and a heatmap regression module, among which:

[0012] The backbone network is used to extract visual features from the input image;

[0013] The tokenizer is used to divide the feature maps of different levels extracted by the backbone network into several patches, and to splice the patches along the channel dimension and straighten them into a one-dimensional sequence. Finally, the one-dimensional sequence is transformed into the required visual token through a linear layer.

[0014] The Feature Aggregation module is used to splice visual tokens at different levels to achieve the purpose of fusing visual features at different levels.

[0015] The discriminator network is used to score the confidence of each visual token after stitching. The score determines the importance of the features contained in the visual token. Visual tokens with low confidence scores (equivalent to background noise) are removed to initially cut out the human body mask.

[0016] The Efficient Transformer module is used to encode the input tokens (including visual tokens and keypoint tokens), and works with the discriminator network to prune the Transformers at different stages. It also merges visual tokens with low confidence scores into a new visual token, thereby accelerating the Transformer without sacrificing accuracy.

[0017] The heatmap regression module is used to decode the predicted coordinates of each joint in the input image using a heatmap method based on the encoding results.

[0018] (3) Use the images and their annotation information obtained in step (1) to train the above MADN network model;

[0019] (4) Input the image containing the person to be estimated into the trained MADN network model, and the position coordinates of each joint of the person in the image can be predicted.

[0020] Furthermore, to balance the overhead of the Transformer module, the backbone network only uses the first three stages of the HRNet network pre-trained on the ImageNet dataset, namely HRNet-s, with only 25% of the original parameters. Specifically, the input image is first cropped to a uniform size and then fed into HRNet-s for feature extraction, outputting feature maps at three different levels: high, medium, and low. For the i-th level feature map, its height H... i and width W i They are respectively:

[0021] H i =H / 2i+1 W i =W / 2 i+1 i∈{1,2,3}

[0022] Where H and W are the height and width of the input image, respectively.

[0023] Furthermore, the specific operation process of the token generator is as follows:

[0024] First, for the feature map of the i-th level, it is divided into N... i There are 1 grid, each grid having a size of P. h ×P w ,

[0025] Then, each grid is straightened and stitched along the channel dimension to generate N. i There are P one-dimensional sequences, each with a length of P. h ×P w ×C i C i Let be the number of channels in the i-th level feature map;

[0026] Finally, a fully connected approach is used to remap all one-dimensional sequences, generating a visual token for the i-th level feature map, with a size of N. i ×L, N i L represents the number of visual tokens at the i-th level, and L is the length of the visual tokens.

[0027] Furthermore, the specific operation process of the discriminator network is as follows: First, the visual tokens, which are concatenated from multiple layers, are sequentially passed through a multilayer perceptron and a Softmax function to obtain an N-dimensional weight vector, which contains the confidence score of each visual token. Finally, based on the weight vector, the visual tokens ranked in the bottom 30% by confidence score are pruned, and only the top 70% of visual tokens are retained and their positions are encoded.

[0028] Furthermore, the efficient Transformer module is composed of multiple cascaded Transformers. Its input consists of the cropped and preserved visual tokens and the initialized keypoint tokens. Each Transformer layer is composed of a regularization layer L1, a multi-head self-attention mechanism layer, a regularization layer L2, and a feedforward neural network connected in sequence. The input of the regularization layer L2 is the superposition of the output of the multi-head self-attention mechanism layer and the input of the regularization layer L1. The output of the feedforward neural network is superimposed with the input of the regularization layer L2 as the output of the Transformer.

[0029] Furthermore, a discriminator network is set in front of some Transformer layers in the high-efficiency Transformer module. This discriminator network is used to score the confidence of visual tokens in the Tokens input to the Transformer and merge visual tokens with lower confidence scores into a new visual token.

[0030] Furthermore, the multi-head self-attention mechanism layer consists of multiple self-attention mechanisms, each of which modifies the input of the multi-head self-attention mechanism layer with a linear transformation matrix W. Q W K and W V Multiplication yields the query vector matrix Q, the key vector matrix K, and the value vector matrix V. The output of Self-Attention is then calculated using the following formula. Finally, the outputs of multiple Self-Attention layers are concatenated and passed through a linear layer to obtain the output of the multi-head self-attention mechanism layer.

[0031]

[0032] Where: Attention(Q,K,V) is the output of Self-Attention, and s is the number of columns in the query vector matrix Q. T This indicates transpose.

[0033] Furthermore, the heatmap regression module only takes keypoint tokens as output, and then remaps them back to a two-dimensional heatmap form through a multilayer perceptron. Specifically, in the final stage of the efficient Transformer module, only keypoint tokens are taken as output and fed into the multilayer perceptron. In the multilayer perceptron, the keypoint tokens are first mapped back to a two-dimensional generated heatmap P with a size of M×H. * ×W * H * and W * The height H and width W of the input image are respectively 1 / 4, and M is the number of keypoint tokens; finally, the heatmap P is reshaped into M heatmaps, and the dimensions of the heatmaps are the same as the original. Figure 1 Based on this, the coordinates of each joint point of the human body are located by finding the position of maximum response on the heat map.

[0034] Furthermore, the specific implementation of step (3) is as follows:

[0035] 3.1 Initialize model parameters, including the bias vector and weight matrix of each layer, learning rate, and optimizer;

[0036] 3.2 Input the image containing people into the model, and the model outputs the corresponding prediction result, i.e., the coordinates of the key points, through forward propagation. Calculate the loss function between the prediction result and the annotation information.

[0037] 3.3 Based on the loss function (such as MSE), the optimizer (such as Adam) uses gradient descent to iteratively update the model parameters until the loss function converges and the training is complete.

[0038] Applying Transformers to human pose estimation is a future trend. The core idea of ​​Pose Transformers is to use image patches as visual tokens and model global dependencies in the image through a multi-head self-attention mechanism. However, the global attention mechanism has high computational complexity and is easily affected by severe background noise, posing a significant challenge to Transformer-based human pose estimation networks. To address this, this invention proposes a two-dimensional human pose estimation algorithm model, MADN, based on a multi-level fusion and discriminant network. First, a high-resolution CNN network is used to extract texture information from the image. Then, multi-level features are fused to improve the network's feature representation ability. Next, a discriminant network is used to locate the rough human contour, which is then fed into an efficient Transformer module for processing. At different stages of the Transformer, each token is scored with confidence, and the score determines the token's importance relative to the skeletal keypoints. Tokens with lower importance are fused into a new Fused Token to reduce information redundancy and improve computational efficiency. Finally, a heatmap method is used to predict keypoints at the end of the network. Finally, we tested the model on the mainstream COCO and MPII datasets. In terms of both computational complexity and network accuracy, our invention outperforms the mainstream state-of-the-art models. Attached Figure Description

[0039] Figure 1 This is a schematic diagram illustrating the principle framework of the MADN network model of this invention.

[0040] Figure 2 This is a schematic diagram illustrating the principles of the token generator and feature fusion module.

[0041] Figure 3 This is a schematic diagram illustrating the principle of the high-efficiency Transformer module.

[0042] Figure 4 This is a visualization of the prediction results of the MADN network of this invention on the COCO and MPII datasets.

[0043] Figure 5This is a comparison chart showing the visualization prediction results of the MADN network and HRFormer network of this invention on the COCO dataset.

[0044] Figure 6 This diagram illustrates the trade-off between computational speed and accuracy between the present invention and mainstream human pose estimation models. Detailed Implementation

[0045] To describe the present invention in more detail, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] This invention presents a two-dimensional human pose estimation algorithm based on a multi-level fusion and discriminant network, comprising the following steps:

[0047] (1) Obtain a large number of images containing people and mark the key points of the people in the images.

[0048] In this implementation, we selected images from the COCO and MPII datasets for model training. The COCO dataset is a widely used dataset in computer vision for tasks such as human pose estimation, object detection, and semantic segmentation. This dataset contains over 200k images and 250k human instances, each annotated with 17 keypoints. The COCO dataset is divided into training, validation, and test sets. The training set contains approximately 57k images, the validation set contains approximately 5k images, and the test set contains approximately 20k images. In this implementation, we performed thorough validation on both the validation and test sets. The commonly used evaluation metric for the COCO dataset is Average Precision (AP). AP is calculated based on Object Keypoint Similarity (OKS), which considers factors such as the Euclidean distance between predicted and ground truth keypoints, visibility markers, and object scale. The calculation formula is shown below:

[0049]

[0050] Where: k i d represents the weight of the i-th keypoint, used to adjust the contribution of different keypoints to the final OKS; i V represents the detection error of the i-th keypoint, i.e., the Euclidean distance between the predicted keypoint and the actual keypoint; δ represents a constant used to control the degree of error penalty; i V represents the visibility flag of the i-th keypoint. If the keypoint is visible, then V... i =1, otherwise V i =0, s represents the area of ​​the target instance; knowing the formula for OKS, we can derive the formula for AP as follows:

[0051]

[0052] Where: when t = 0.5 and t = 0.75, we denote AP as AP. 50 and AP 75 When 32 2 2 <96 2 and 96 2 2 We will denote AP as AP M and AP L .

[0053] The MPII dataset contains images from real-world activities, each with complete body pose annotations. The dataset contains 40,000 human samples, each with 16 keypoints annotated. The MPII dataset has an image resolution of 256×256 and uses the head-normalized correct keypoint probability (PCKh) as the evaluation metric. PCKh@0.5 is the most commonly used evaluation metric, indicating that a keypoint is considered correct when the Euclidean distance between the predicted keypoint and the ground truth keypoint is less than half the distance between the ground truth keypoint and the head. The calculation formula is as follows:

[0054]

[0055] Where: P and G represent the prediction matrix and the ground truth matrix, respectively; c is the total number of visible markers in V; L head Let be the head length, α be a constant, and n be the batch size. Here, we take PCK@0.5 (α=0.5) as the final experimental result.

[0056] (2) Construct the model architecture of MADN, such as Figure 1 The diagram shows the principle framework of the algorithm model, which consists of a backbone network, a token generator, a feature fusion module, a discriminator network, an efficient Transformer module, and a heatmap regression module.

[0057] Using a 2D image as input, the single-person image is first fed into the backbone network. To balance the high overhead of the Transformer part, we only use the first three stages of HRNet, which has been pre-trained on the ImageNet dataset. We name it HRNet-s, and its parameter count is only 25% of the original. Specifically, for the input single-person image, we first crop it to a uniform size and denote it as... Next, it is fed into HRNet-s for feature extraction, outputting feature maps at three different levels, as shown in the following formula:

[0058] ​​Branch i=Backbone(X),i∈{1,2,3}

[0059] in: And H i W i =H / 2 i+1 W / 2 i+1 The outputs of the three different branches i∈{1,2,3} represent feature maps at different levels. Then we feed the feature maps at different levels into the token generator module.

[0060] Before feature fusion, we first need to convert the feature maps into a one-dimensional sequence and generate visual tokens, such as... Figure 2 As shown, specifically, we first need to divide Branch i, i∈{1,2,3} into... A grid, where P h ×P w In this implementation, we set the size of each grid to 4×3; then we unfold each grid and stitch them together along the channel dimension to generate... There are P one-dimensional sequences, where the length of each one-dimensional sequence is P. h ×P w ×C i ,i∈{1,2,3}. Since the computational complexity of Transformer is related to the number and length of Tokens, we need to shorten the length of each sequence while preserving fine-grained information. Therefore, we use a fully connected form to remap the one-dimensional sequence and generate Tokens. The calculation formula for the above process is as follows:

[0061] Branch Ti=Linear(Stack(Split(Branch i))),i∈{1,2,3}

[0062] in: N represents the token generated from the feature map of the i-th level. i This represents the number of tokens. L represents the length of the token; in this implementation, we set the length of each token to 192.

[0063] After generating tokens at three different levels, we need to perform feature fusion. This not only improves the model's feature representation ability but also enables the network to better capture information about occlusion and end joints, thus improving its ability to detect these situations. The specific calculation process is shown below:

[0064] ML_Token=Concat(Branch Ti),i∈{1,2,3}

[0065] in: This represents a token generated after the fusion of multi-level features. L = 192; In order to preserve the visual information contained in the token to the greatest extent in this invention, we did not use a simple addition method, but retained each token by concatenation operation, and then used a discriminative network to filter out the visual tokens we needed.

[0066] In the above, the number of ML_Tokens generated is These tokens contain many tokens that are irrelevant to the task or serve as noise. The information contained in these tokens is not strongly correlated with the key points of the human body, so we need to prune these tokens.

[0067] Specifically, in the discriminator network, the ML_Token is first fed into a multilayer perceptron for processing. The multilayer perceptron consists of LayerNorm→Linear(N,N)→GELU, where N is the number of ML_Tokens, as shown in the formula below:

[0068] f = MLP(ML_Token) ∈ R N

[0069] Next, we use the Softmax function to assign a confidence score to each token, thereby obtaining the probability of each token being retained. The calculation formula is shown below:

[0070] f′=Softmax(f)∈R N

[0071] At this point, f′ represents a size of We now have the confidence score for each token, and we will prune the tokens from the bottom 30% of the confidence scores to generate the visual tags we need. The calculation formula is as follows:

[0072]

[0073] At this point, N′ = N × 70%. Since the self-attention mechanism in the Transformer lacks position awareness, we need to perform position encoding on the Visual Token. We use a two-dimensional position embedding method. i For each visual tag, i∈(1,N′), positional encoding is performed. Assume the visual tag is represented by v.i If we represent i∈(1,N′), then the formula for position encoding is as follows:

[0074] Visual Token={v1+pe1,v2+pe2,…,v N′ +pe N′}

[0075] At the same time, the network will also generate M learnable [data / methods]. This represents M human body nodes, which, together with the Visual Token, serve as input to the efficient Transformer module.

[0076] Given a one-dimensional order As input, the number of input tokens is C = N′ + M. An efficient Transformer module is composed of multiple layers of Transformers, such as... Figure 3 As shown, each Transformer layer consists of LayerNorm→Multi Head Self Attention→LayerNorm→FeedForward. Specifically, we first need to perform a linear projection on the input T to obtain the key (K), value (V), and query (Q), calculated as follows:

[0077] Q = T × W Q K = T × W K V = T × W V

[0078] Among them: W Q W K W V This is the corresponding weight matrix.

[0079] The formula for calculating the multi-head self-attention score is as follows:

[0080]

[0081] Where: d k This is the key dimension, and each MSA score determines the level of attention given to the current query token. The above calculation strategy is used for each Transformer layer, but for layers 3, 6, 9, or layers 1, 3, 5, we add a discriminative network DN before the Transformer layer to efficiently prune the network. The calculation formula is as follows:

[0082] T l =DN(T) l-1 )+Fused Token

[0083] Wherein: T l-1T represents the input to the previous Transformer. l This represents the output of the current Transformer layer, while the Fused Token is generated by T. l-1 It is formed by merging the tokens with the lowest confidence scores in the bottom 30%.

[0084] The introduction of Fused Tokens can correct early scoring errors of Tokens and retain certain background features to help emphasize foreground character features. By having these less important Tokens participate in subsequent calculations along with other important Tokens, the model can better capture key information about the character. Fused Tokens play a crucial role in the Transformer pruning process at a cost of less than 1% of the total GFLOPs.

[0085] In the final stage of the Transformer, we only take M elements. As output, it is reprojected back through an MLP layer. Dimensions, where H′ and W′ are both 1 / 4 of the original image size; then it is reshaped into The size of the heatmap at this time is the same as the original. Figure 1 Finally, based on this, the coordinates of the human body's joints are located by finding the location of the maximum response on the heat map.

[0086] This implementation uses the MSE loss function. In human pose estimation tasks, due to the diversity and complexity of poses, there may be outliers or noise. The MSE loss function can reduce the impact of these outliers on model training and improve the model's robustness. In addition, the MSE loss function can effectively measure the difference between the predicted keypoint heatmap and the true keypoint heatmap; by minimizing this difference, the model can learn more accurate keypoint locations. The calculation formula is shown below:

[0087]

[0088] Where: M is the number of joints. For the predicted heatmap of the k-th joint, P k A keypoint heatmap generated for the k-th real coordinate.

[0089] (3) Train the MADN model framework using the images and labels from step (1).

[0090] (4) By feeding the image of the person into the trained model, the coordinates of the joints can be predicted.

[0091] Experimental section:

[0092] We followed a top-down paradigm, first detecting single-person images from multi-person images using a human detector with an accuracy of 56% AP. Then, we cropped the single-person images to 256×192 pixels in the COCO dataset and 256×256 pixels in the MPII dataset. Next, we performed data augmentation, including random rotations [-45°, 45°] and horizontal flipping. During training, we used the Adam optimizer with an initial learning rate of 0.001 and a decay factor of 0.1. We set the total number of epochs to 300 and the batch size to 128. For MADN, we extended it with three variants as shown in Table 1 to more comprehensively verify the effectiveness of our method.

[0093] Table 1

[0094]

[0095] The experiment was conducted on a server with eight NVIDIA RTX 3090 24G graphics cards and a 48-core Intel(R) Xeon(R) CPU E5-2678 v3@2.50GHz. The software environment consisted of Linux Ubuntu 20.04.5 LTS, Python 3.7, PyTorch 1.13.1, and CUDA 11.6 + CUDA 8.3.2.

[0096] Table 2 shows the experimental results of the MADN network on the COCO Validation dataset. It can be seen that compared to CNN-type human pose estimation networks, our MADN-L achieves 2.4% higher accuracy than the state-of-the-art HRNetW48 while reducing computational complexity by 42%. Compared to CNN+Transformer type networks, our MADN-L improves accuracy by 2.1% compared to TokenPose-L while reducing computational complexity by 0.7 GFLOPs. MADN-M shows an even more significant reduction in computational complexity while improving accuracy. Compared to other state-of-the-art TransPose and TokenPose networks, MADN shows significant improvements in both speed and accuracy. Compared to the pure Transformer type HRFormer-B, it reduces GFLOPs by 3.8 while improving AP by 1.9. Furthermore, our smaller network, MADN-S, achieves significantly higher speed and comparable accuracy compared to other mainstream networks. In addition, we also compared our network with other state-of-the-art Transformer-based human pose estimation networks on the COCO Validation dataset, such as… Figure 6 As shown, it can be seen more intuitively that the MADN network exhibits excellent performance in both computational complexity and accuracy.

[0097] Table 2

[0098]

[0099] Table 3 shows the comparison between the MADN network and other state-of-the-art human pose estimation networks on the COCO test-dev test set. Compared with CNN-based human pose estimation networks, MADN achieves significant superiority in all metrics. Compared with lightweight human pose estimation networks, our MADN-S has significantly higher AP than EfficientPose and Lite-HRNet with similar computational complexity. Compared with Transformer-based human pose estimation networks, MADN-L improves AP by 0.7 percentage points while significantly reducing GFLOPs by 18.4 percentage points compared to HRFormer-B. And compared with advanced human pose estimation networks such as TokenPose and TransPose, MADN shows significant improvements in all metrics.

[0100] Table 3

[0101]

[0102] As shown in Table 4, we also conducted a simple test on the MPII dataset. On Elb, it achieved 90.6% accuracy, matching PRTR. On other metrics, it significantly outperformed mainstream human pose estimation networks, with the most significant improvement on the Ank metric, reaching 83.7%. When PCKh@0.5, we achieved an accuracy of 90.6%, which is 0.3% higher than HRNet, showing a significant improvement.

[0103] Table 4

[0104]

[0105] We visualized the experimental results on the COCO and MPII datasets as follows: Figure 4 As shown in the COCO dataset, our MADN-L network can accurately predict the joints of the human body in both complex scenes with multiple people and simple scenes with a single person. The same conclusion can be obtained in the MPII dataset. In addition, in some occluded scenes, our model can also reasonably predict the joints of the occluded parts.

[0106] Since the accuracy of the MPII dataset has already reached over 90%, the improvement in the visualized results compared with other models is not significant. Therefore, we compare it with the state-of-the-art model HRFormer on the COCO dataset. Figure 5As shown in the first image from left to right, HRFormer made a significant error in predicting the joints of the left leg, while MADN-L could still accurately predict the joints of the knee and ankle even when the left leg was obscured. The second and third images show that when a part of the figure is completely obscured, MADN-L can reasonably infer the location of the obscured joints. The fourth image shows that in blurry scenes such as overexposure and strong light, HRFormer made an error in predicting the right arm, while MADN-L could accurately predict the joints of the elbow and wrist on the right arm.

[0107] Next, we will conduct extensive ablation experiments on the multi-level feature fusion module, the discriminant network in the Efficient Transformer, and the Fused Token. In the following experiments, we will use the MADN-M network and the COCOValidation dataset.

[0108] Multi-level feature fusion can significantly improve the feature representation ability of a network, greatly impacting its performance. However, mainstream human pose estimation networks only consider high-level features and tend to ignore the role of other levels of features. Therefore, to verify the effectiveness of the multi-level feature fusion module, we conducted ablation experiments as shown in Table 5. Using the Base method and Method 1, we can see that low-level features contribute 0.8% to the accuracy of the MADN network. Using the Base method and Method 2, we can see that medium-level features contribute 1.0% to the accuracy of the MADN network. This shows that deeper and higher-level features have a greater impact on the network, but shallow features also have their value. Using Method 3, we can see that after fusing features from different levels, the contribution to the accuracy of the MADN network is 1.3%, a significant improvement. This demonstrates that the multi-level feature fusion concept proposed in this invention has a significant impact on improving network performance.

[0109] Table 5

[0110]

[0111]

[0112] Note: We name the different network combinations as the Base method and methods 1, 2, and 3, respectively. The Base method uses HRNetW32-s as the backbone network, performs ablation experiments on the output part of the backbone network, and then uses the Efficient Transformer proposed in this invention for the Transformer part. Methods 1, 2, and 3 also have this structure.

[0113] Transformers can significantly improve the modeling ability of human pose estimation networks, but they also bring high computational costs. Some researchers have tried to reduce the computational complexity of Transformers through pruning. While existing techniques such as Transformer pruning have greatly improved computational speed, they also come at a slight performance cost. Therefore, this invention proposes the Efficient Transformer module. For the discriminant network and Fused Token proposed in this invention within the Efficient Transformer module, we conducted extensive ablation experiments, as shown in Table 6. Through the Base method and Method 1, it can be seen that the discriminant network can reduce the computational complexity of the Transformer by 47.9%, but it also loses 1% accuracy. Through Method 1 and Method 2, it can be seen that although the discriminant network causes a 1% loss in accuracy, the Fused Token brings a 2% improvement in accuracy. Therefore, compared to the ordinary Pose Transformer, the Efficient Transformer achieves a 47.9% reduction in computational complexity while improving accuracy by 1%. In addition, we compared the discriminant network with the HTI module in [Yu, Changqian, et al. Lite-hrnet: A lightweight high-resolution network. in CVPR.2021, pp.10440-10450] and the pruned Transformer-based [Ma, Haoyu, et al. Ppt: token-pruned pose transformer for monocular and multi-view human pose estimation. in ECCV.2022, pp.424-442]. While both serve the same purpose, their performance differs. Methods 1 and 3 show that DN and HTI reduce complexity by 47.9% and 38.2%, respectively, and reduce accuracy by 0.1% and 0.3%, respectively, demonstrating the superior performance of the discriminant network proposed in this invention. Furthermore, methods 3 and 4 also show that applying the Fused Token concept to HTI pruning also results in a 0.2% performance improvement. In conclusion, both the discrimination network and the Fused Token have made significant contributions to the MADN network while maintaining their own advanced performance.

[0114] Table 6

[0115]

[0116]

[0117] Note: For different network combinations, we name them the Base method and methods 1, 2, 3, and 4, respectively. DN = Discriminative Network, FT = Fused Token, HTI is a token pruning strategy proposed in the PPT, and GPLOPs. T The computational complexity of the Transformer is given. The Base method uses HRNetW32-s as the backbone network, performs multi-level feature fusion on the output, and then conducts ablation experiments on the discriminative network module inside the Efficient Transformer and the Fused Token. Methods 1, 2, 3, and 4 also have this structure.

[0118] After conducting ablation experiments on each module internally, we return to the overall picture to verify the effectiveness of the three main innovations of this invention: multi-level feature fusion, the discriminant network, and the Fused Token, and their respective contributions to the MADN network, as shown in Table 7. The Base method and Method 1 show that multi-level feature fusion brings a 1.3% improvement in accuracy. Methods 1 and 2 show that the discriminant network sacrifices 0.1% accuracy for a 19.3% reduction in computational complexity of the MADN network. The Base method and Method 3 show that the EfficientTransformer module improves the accuracy of the MADN network by 0.1% while reducing computational complexity by 19.3%. Other comparisons yield the same conclusion.

[0119] Table 7

[0120]

[0121] Note: For different network combination methods, we name them Base method and Method 1, 2, 3, 4 respectively. FA = Feature Aggregation, DN = Discriminative Network, FT = Fused Token, EfficientTrans = Efficient Transformer, GPLOPs is the computational complexity of the MADN-M network; among them, the Base method uses HRNetW32-s as the backbone network, and only uses high-level features for the output part, which are then fed into the unpruned PoseTransformer.

[0122] Summarize:

[0123] In this invention, addressing the issue of mainstream Pose Transformer networks sacrificing performance for speed, we propose the MADN network. Through our meticulous design, this network achieves breakthroughs in both speed and accuracy compared to mainstream human pose estimation models. We propose a multi-level feature fusion strategy and an Efficient Transformer module, which introduces a novel discriminative network and Fused Tokens. In this invention, we validate the superior performance of our proposed method by dividing MADN into three versions and conducting extensive experiments and visualizations. We also performed numerous ablation experiments to fully demonstrate each innovation and its comparison with similar methods. The experimental results further confirm the reliability of our proposed method.

[0124] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. Those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.

Claims

1. A two-dimensional human pose estimation method based on a multi-level fusion and discriminant network, comprising the following steps: (1) Obtain images containing people and mark the key points of the people in the images; (2) Construct the MADN network model framework, which includes a backbone network, a token generator, a feature fusion module, a discriminator network, an efficient Transformer module, and a heatmap regression module, wherein: The backbone network is used to extract visual features from the input image; The token generator is used to divide the feature maps of different levels extracted from the backbone network into several patches, and to splice the patches along the channel dimension and straighten them into a one-dimensional sequence. Finally, the one-dimensional sequence is transformed into the required visual token through a linear layer. The feature fusion module is used to splice visual tokens at different levels to achieve the purpose of fusing visual features at different levels. The discriminator network is used to score the confidence of each visual token after splicing. The score determines the importance of the features contained in the visual token. The visual tokens with the lowest confidence scores are removed to initially cut out the human body mask. The high-efficiency Transformer module is used to encode the input token and, together with the discriminator network, prune the Transformer at different stages. It also merges the visual tokens ranked in the bottom 30% of confidence scores into a new visual token, thereby accelerating the Transformer without sacrificing accuracy. The heatmap regression module is used to decode the predicted coordinates of each joint in the input image using a heatmap method based on the encoding results. (3) Use the images and their annotation information obtained in step (1) to train the above MADN network model; (4) Input the image containing the person to be estimated into the trained MADN network model, and the position coordinates of each joint of the person in the image can be predicted.

2. The two-dimensional human pose estimation method according to claim 1, characterized in that: The backbone network uses only the first three stages of the HRNet network pre-trained on the ImageNet dataset, i.e., HRNet-s, with only 25% of the original parameters. Specifically, the input image is first cropped to a uniform size and then fed into HRNet-s for feature extraction, outputting feature maps at three different levels: high, medium, and low. For the first... i Each level of feature map, its height and width They are respectively: ; in: H and W The height and width of the input image are respectively.

3. The two-dimensional human pose estimation method according to claim 2, characterized in that: The specific operation process of the token generator is as follows: First, regarding the first i The feature maps at each level are segmented into... There are 1 grid, and the size of each grid is 1. , ; Then, each grid is straightened and stitched along the channel dimension to generate... There are n one-dimensional sequences, each with a length of n. , For the first i The number of channels in each hierarchical feature map; Finally, a fully connected approach is used to remap all one-dimensional sequences to generate the first one-dimensional sequence. i A visual token of a hierarchical feature map, the size of which is , Indicates the first i The number of visual tokens at each level L The length of the visual token.

4. The two-dimensional human pose estimation method according to claim 3, characterized in that: The specific operation process of the discriminator network is as follows: First, the visual tokens, which are concatenated from multiple layers, are sequentially passed through a multilayer perceptron and a Softmax function to obtain a... N A weight vector of dimensions, which contains the confidence score for each visual token. Finally, based on the weight vector, the visual tokens ranked in the bottom 30% of the confidence scores are pruned, and only the top 70% of the visual tokens are retained and their positions are encoded.

5. The two-dimensional human pose estimation method according to claim 1, characterized in that: The efficient Transformer module consists of multiple cascaded Transformers. Its input consists of cropped and preserved visual tokens and initialized keypoint tokens. Each Transformer layer is composed of a regularization layer L1, a multi-head self-attention mechanism layer, a regularization layer L2, and a feedforward neural network connected in sequence. The input of the regularization layer L2 is the superposition of the output of the multi-head self-attention mechanism layer and the input of the regularization layer L1. The output of the feedforward neural network is superimposed with the input of the regularization layer L2 as the output of the Transformer.

6. The two-dimensional human pose estimation method according to claim 5, characterized in that: In the high-efficiency Transformer module, a discriminator network is set in front of some Transformer layers. It is used to score the confidence of visual tokens in the tokens input to the Transformer and merge the visual tokens with the lowest confidence scores into a new visual token.

7. The two-dimensional human pose estimation method according to claim 5, characterized in that: The multi-head self-attention mechanism layer consists of multiple Self-Attention mechanisms. Each Self-Attention mechanism modifies the input of the multi-head self-attention mechanism layer with a linear transformation matrix W. Q W K and W V Multiplication yields the query vector matrix Q, the key vector matrix K, and the value vector matrix V. The output of Self-Attention is then calculated using the following formula. Finally, the outputs of multiple Self-Attention layers are concatenated and passed through a linear layer to obtain the output of the multi-head self-attention mechanism layer. ; in: This is the output of Self-Attention, where s is the number of columns in the query vector matrix Q. T This indicates transpose.

8. The two-dimensional human pose estimation method according to claim 1, characterized in that: The heatmap regression module only takes keypoint tokens as output, and then remaps them back to a two-dimensional heatmap form through a multilayer perceptron. Specifically: in the final stage of the efficient Transformer module, only keypoint tokens are taken as output and fed into the multilayer perceptron. In the multilayer perceptron, the keypoint tokens are first mapped back to a two-dimensional generated heatmap P, the size of which is... , and The heights of the input images are respectively H and width W 1 / 4 M The number of keypoint tokens; finally, the heatmap P is reshaped into M A heatmap is generated, with the same size as the original image. Based on this, the coordinates of each joint point of the human body are located by finding the position of maximum response on the heatmap.

9. The two-dimensional human pose estimation method according to claim 1, characterized in that: The specific implementation method of step (3) is as follows: 3.1 Initialize model parameters, including the bias vector and weight matrix of each layer, learning rate, and optimizer; 3.2 Input the image containing people into the model, and the model outputs the corresponding prediction result, i.e., the coordinates of the key points, through forward propagation. Calculate the loss function between the prediction result and the annotation information. 3.3 Based on the loss function, the optimizer iteratively updates the model parameters using gradient descent until the loss function converges, and the training is complete.

Images