A full-scale perceptual balancing high-resolution multi-person pose estimation method

Abstract

This invention discloses a high-resolution multi-person pose estimation method with full-scale perceptual balance. The proposed multi-person pose estimation model, FSA Net, uses HRNet as its backbone. The outputs of the four branches in the last stage are fed into the DCB module. The DCB module introduces dilated convolutions with different dilation rates through a parallel structure to expand the receptive field of each branch. Then, the AOB module performs attention operations at the branch and channel levels to enhance high-frequency features and reduce the influence of noise. Finally, a heatmap representation is used for prediction. The entire model can recognize images with rich scales and more complex semantic information. Experiments show that FSA Net achieves competitive results on the MSCOCO and MPII datasets, which also verifies the effectiveness of this invention.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and human pose estimation technology, specifically involving a high-resolution multi-person pose estimation method with full-scale perceptual balance. Background Technology

[0002] Human pose estimation is a hot research topic in the field of computer vision. It uses key points to simulate human joints and has been widely applied in various fields in recent years, such as autonomous driving, motion tracking, and intelligent monitoring.

[0003] Currently, the mainstream methods for human pose estimation fall into two categories: top-down and bottom-up. The former is a multi-stage approach, typically involving first detecting a single person image using an object detector before performing pose estimation. The latter is mostly a single-stage approach, used for multi-person pose estimation, commonly predicting all keypoints in the image and then performing classification and aggregation. Clearly, the bottom-up approach is more challenging. During network learning, larger-scale human features are often more readily apparent and easier for the network to learn; however, smaller-scale human features are not only less defined but also more susceptible to ambient noise. Therefore, enabling the network to better perceive targets of different scales is a key challenge, and this is the focus of this research.

[0004] Since each person is detected individually in single-person pose estimation, there is no scale variation issue, making it much simpler than multi-person pose estimation. In multi-person pose estimation tasks, because mainstream multi-person pose estimation models are all single-stage models, the design of the backbone network is particularly important. The network depth, downsampling rate, and receptive field size all greatly affect the network's performance.For example, the paper [Newell, Alejandro, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14. Springer International Publishing, 2016] proposes a cascaded cone-shaped hourglass network for pose estimation by directly regressing keypoints; the paper [Pishchulin, Leonid, et al. Deepcut: Joint subset partition and labeling for multi personpose estimation. Proceedings of the IEEE conference on computer vision and pattern recognition. 2016] uses Faster R-CNN for human target detection, followed by pose estimation using an integer linear programming method; while the paper [Insafutdinov, Eldar, et al. Deepercut: A deeper, stronger, and faster multi-person pose estimation model. Computer Vision–ECCV 2016: 14th European Conference] uses Faster R-CNN for human target detection, followed by pose estimation using an integer linear programming method; and the paper [Insafutdinov, Eldar, et al. Deepercut: A deeper, stronger, and faster multi-person pose estimation model. Computer Vision–ECCV 2016: 14th European Conference] uses Faster R-CNN for human target detection, followed by pose estimation using an integer linear programming method; [Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14. Springer International Publishing, 2016] uses ResNet to extract body parts, and then uses the Image-Conditioned PairwiseTerms method to predict key points, which improves both accuracy and speed compared to Deepcut.

[0005] Scale variation is a key research focus in multi-person pose estimation. In scenes with people in both the foreground and background, models perform better for larger-scale people, while their performance on smaller-scale people is often unsatisfactory. Therefore, balancing the detection of targets at different scales is very challenging. Currently, most networks use HRNet as their backbone. However, deeper networks have larger receptive fields, which improves the ability to handle large targets but may overlook smaller ones. In addition, excessive downsampling layers can impair the network's ability to detect small-scale targets. All of these factors affect the performance of HRNet, preventing it from effectively balancing the detection of targets at different scales. Summary of the Invention

[0006] In view of the above, this invention proposes a high-resolution multi-person pose estimation method with full-scale perceptual balance, which can identify images with rich scales and more complex semantic information. Combined with the research on the influence of receptive field on detection, it provides a new approach for pose estimation tasks.

[0007] A high-resolution multi-person pose estimation method with full-scale sensing balance includes the following steps:

[0008] (1) Obtain a large number of images containing multiple people and mark the key points of the people in the images;

[0009] (2) Divide all images into training set, validation set and test set according to the proportions;

[0010] (3) Construct the FSA Net (Full Scale-Aware Balanced High-Resolution Network for Multi-Person Pose Estimation) model framework, which includes:

[0011] The backbone network is used to extract features from the input image and output four sets of feature maps.

[0012] The DCB (dilated convolution based module) module consists of four parallel branches, each containing dilated convolutions with different dilation rates, thereby enhancing the receptive field of the four sets of feature maps.

[0013] The AOB (attention operation based module) module is used to perform attention operations on the feature map output by each branch from the branch and channel dimensions, thereby enhancing important features and suppressing the influence of noise.

[0014] The Decoder module generates a key point heatmap by decoding the feature map output by the AOB module.

[0015] (4) Input the training set images and their label information into the FSA Net model framework for training to obtain a multi-person pose estimation model;

[0016] (5) Input the test set images into the trained multi-person pose estimation model to predict the position coordinates of each joint of the person in the image.

[0017] Furthermore, the backbone network adopts a multi-layer convolutional neural network based on the HRNetW48 structure. After the input image is processed by the multi-layer convolutional neural network for feature extraction, four sets of feature maps are output.

[0018] Furthermore, each branch of the DCB module uses BottleNeck to process the input feature map. BottleNeck is composed of convolutional layer C1, batch normalization layer B1, ReLU activation function R1, dilated convolutional layer, batch normalization layer B2, ReLU activation function R2, convolutional layer C2, and batch normalization layer B3 connected in sequence. The kernel size of C1 and C2 is 1×1. The output of C1 and the output of B3 are superimposed to serve as the output of the branch.

[0019] Furthermore, the dilation rates of the hollow convolutional layers in the four branches of the DCB module are 9, 7, 5, and 3, respectively.

[0020] Furthermore, the AOB module first performs upsampling and cross-swapping operations on the feature maps w1 to w4 output by the four branches of the DCB module to generate feature maps branchw1 to branchw4; then it performs attention operations on the feature maps branchw1 to branchw4 from the branch and channel dimensions to obtain feature maps featurew1 to featurew4; finally, it performs a superposition operation on the feature maps featurew1 to featurew4 to generate feature maps w1' to w4'.

[0021] Furthermore, the upsampling and downsampling cross-stitching operation is that for any branch output feature map w... i First, output the feature maps w from other branches. j The size is made similar to w through upsampling or downsampling operations. i The feature maps w1 to w4 are then concatenated to obtain the feature map branchw. i i and j = 1, 2, 3, 4 and j ≠ i; the superposition operation is to superimpose the feature maps featurew1 to featurew4 along the channel dimension for any branch i, so that the superimposed feature map w i The number of channels and w i Consistent.

[0022] Furthermore, attention operations are performed on feature maps branchw1 to branchw4 from the branch and channel dimensions according to the following formula:

[0023] featurew i =branchw i ×[AAP(branchw i )+AMP(branchw i )]

[0024] Where: AAP() represents adaptive average pooling operation, AMP() represents adaptive max pooling operation, i = 1, 2, 3, 4.

[0025] Furthermore, the specific process of training the model in step (4) is as follows:

[0026] 4.1 Initialize model parameters, including the bias vector and weight matrix of each layer, learning rate, and optimizer;

[0027] 4.2 Input the training set images into the model, and the model outputs the corresponding prediction results, i.e., the position coordinates of each joint of the person in the image, through forward propagation. Calculate the loss function between the prediction results and the labels.

[0028] 4.3 Based on the loss function, the optimizer iteratively updates the model parameters using gradient descent until the loss function converges, and the training is complete;

[0029] 4.4 After training is completed, the model is validated using validation set images. The model that performs best on the validation set is used as the final multi-person pose estimation model.

[0030] In real life, the scenarios we encounter are often complex. For example, in the application of pose estimation in intelligent surveillance, any person in the surveillance footage, regardless of their size, distance, or whether they are occluded or blurred, could be a target. Therefore, the network should treat targets of different scales equally. A pose estimation network with balanced perception across all scales has significant practical application value. This invention uniquely introduces parallel dilated convolutions with different dilation rates into pose estimation and has achieved good results. This is a common practice in object detection and semantic segmentation tasks, but its application in pose estimation still needs further exploration. Therefore, this invention, based on practical applications and combined with research on the impact of receptive field on detection, provides a new approach to pose estimation tasks.

[0031] To improve network performance in complex scenarios, this invention proposes a novel multi-person pose estimation model, FSA Net, which uses HRNet as its backbone. The outputs of the four branches in the final stage are fed into the DCB module. The DCB module expands the receptive field of each branch by introducing dilated convolutions with different dilation rates through a parallel structure. Then, the AOB module performs attention operations at the branch and channel levels to enhance high-frequency features and reduce the impact of noise. Finally, heatmap representation is used for prediction. The entire model can recognize images with rich scales and more complex semantic information. Experiments show that FSA Net achieves competitive results on the MSCOCO and MPII datasets, which also verifies the effectiveness of this invention. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the network structure of the FSA Net of the present invention.

[0033] Figure 2 This is a schematic diagram of the DCB module.

[0034] Figure 3 This is a schematic diagram of the AOB module.

[0035] Figure 4 This is a visualization of the prediction results of the FSA Net model of this invention.

[0036] Figure 5 This is a visualization comparing FSA Net with other algorithm models. Detailed Implementation

[0037] To describe the present invention in more detail, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] The present invention provides a high-resolution multi-person pose estimation method based on full-scale sensing balance, comprising the following steps:

[0039] (1) Obtain a large number of images containing people and mark the key points of the people in the images.

[0040] (2) Constructing such Figure 1 The FSA Net model framework shown includes a backbone network, a DCB module, an AOB module, and a Decoder module, wherein:

[0041] The backbone network is used to extract features from the input image and output feature maps of four branches. In this embodiment, the backbone network adopts a multi-layer convolutional neural network based on the HRNetW48 structure. After the input image is processed by the multi-layer convolutional neural network for feature extraction, the output is a feature map of four branches.

[0042] The DCB module consists of four parallel branches, each containing dilated convolutions with different dilation rates. The four branches are sequentially dilated with rates of 9, 7, 5, and 3 to enhance the receptive field of each branch; for example... Figure 2 As shown, in this embodiment, the DCB module consists of four parallel branches, each containing dilated convolutions with different dilation rates. Specifically, the feature map output by each branch is processed by the BottleNeck module. First, it undergoes a 1×1 convolution, then a 3×3 dilated convolution with different dilation rates, and then another 1×1 convolution. The BottomNeck module does not have an activation function at the end, but uses residual operations instead. Finally, it outputs the feature maps of the four branches, which we denote as w1, w2, w3, and w4 respectively.

[0043] The AOB module then performs attention operations on the feature map of each branch from the branch and channel dimensions to enhance important features while suppressing unnecessary features.

[0044] After processing by the DCB module, the network has initially acquired the ability to enhance perception across all scales. However, for small-scale targets, due to their relatively blurred features, they are very susceptible to noise. Therefore, this invention proposes the AOB module, which performs attention operations on each branch and each channel. The attention operation on the channel is to enhance important features and suppress the influence of noise, which greatly improves the performance of small person detection. The attention operation on each branch is to emphasize the semantic differences between branches, allowing the network to better focus on targets of different scales.

[0045] like Figure 3 As shown, the DCB module outputs feature maps w1, w2, w3, and w4, which are then fed into the AOB module. First, upsampling and cross-swapping are performed to generate branches w1, w2, w3, and w4, as shown in the following equation:

[0046]

[0047] Where: i represents the i-th branch, and j represents the number of branches to be used.

[0048] Next, regarding branch w i Attention operations are performed along the branch and channel dimensions. First, each branch is processed using adaptive average pooling and adaptive max pooling. Then, the results of the two pooling operations are summed to obtain an attention score. Finally, the attention score is applied to branch w. i The weights are readjusted, and the calculation process is shown in the following formula:

[0049] feature w i =branch w i ×[AAP(branch w i )+AMP(branch w i )]

[0050] Where: AAP represents adaptive average pooling operation, and AMP represents adaptive max pooling operation.

[0051] Finally, feature fusion is performed on the feature maps after attention operations to regenerate the four-branch feature maps. The calculation process is shown in the following formula:

[0052]

[0053] Where: i represents the i-th branch, and j represents the number of branches to be used.

[0054] The Decoder module generates the heatmap needed for keypoint prediction. During the decoding phase, the Decoder module upsamples and fuses the outputs of each branch to generate a heatmap that is 1 / 4 the size of the original image. It integrates features extracted from all resolution branches and has a strong ability to predict keypoints of targets at different scales. Finally, after decoding the predicted coordinates, the L2 loss function is used to calculate the loss, comparing the estimated heatmap with the heatmap generated from the ground truth.

[0055] (3) Use the images and their annotation information obtained in step (1) to train the above model framework.

[0056] (4) Input the image containing the person to be estimated into the trained model, and the position coordinates of each joint of the person in the image can be predicted.

[0057] In summary, this invention employs a bottom-up approach, using HRNet as the backbone network and incorporating our carefully designed post-processing module. The newly generated network is named FSA Net, and its structure is as follows: Figure 1As shown. Specifically, in the baseline model of the original HRNet, when the four branches are merged into one branch in the last stage, the present invention only uses a simple upsampling and then addition. This approach ignores the semantic differences between them and assumes that all features are equally important, which is obviously unreasonable. Therefore, we feed the outputs of the four branches of the backbone network into the DCB module. This module consists of four parallel branches, each containing dilated convolutions with different dilation rates. For the four branches, dilated convolutions with dilation rates of 9, 7, 5, and 3 are used sequentially to enhance the receptive fields of different branches. After processing by the DCB module, the feature maps of the four branches are then cross-sampling and concatenated to form a new four-branch feature map. Then, attention operations are performed on the feature map of each branch from the branch and channel dimensions to strengthen important features while suppressing unnecessary features. Finally, the final keypoints are predicted using heatmap representation.

[0058] Verification example:

[0059] In FSA Net, we adopted a bottom-up approach for multi-person pose estimation. During the experiment, we set the initial learning rate to 0.001, used the Adam optimizer to obtain gradients, and set the total number of epochs to 300. We trained and evaluated the network using the mainstream COCO and MPII datasets. Due to the use of heatmaps, a large amount of GPU memory was required. The experimental hardware and software environment is shown in Table 1.

[0060] Table 1

[0061]

[0062] This experiment uses the widely used datasets MSCOCO and MPII in the field of human pose estimation. MSCOCO, a large-scale, multi-purpose dataset from Microsoft, plays a crucial role in mainstream vision tasks. It contains over 200K images, with 250K labeled human keypoint instances. The MSCOCO training set contains 118K images, and the test sets include COCO Validation and COCO test-dev. COCO Validation contains 5K images, primarily used for simple testing and ablation experiments, while COCO test-dev contains 20K images, mainly used for online testing and fair comparison with mainstream models. The main evaluation metrics in the COCO dataset are Mean Precision (AP) and Mean Recall (AR), calculated as follows:

[0063]

[0064] Where: p is the detected human instance number, and t is the threshold used to refine the evaluation index. When t is 0.5 and 0.75, it is denoted as AP. 50 and AP 75 OKS is the similarity between an object and its keypoints, and the calculation formula is as follows:

[0065]

[0066] Where: i represents the i-th key point, d i V represents the Euclidean distance between the actual and predicted values, where s is the area of ​​the human instance, and V is the area of ​​the predicted value. i To determine whether an instance is visible, δ is the regularization parameter for the keypoint; when 32 2 <s 2 <96 2 and 96 2 <s 2 We will call it AP. m and AP L Similarly, AR can also be written as AR 50 AR 7S AR m AR L .

[0067] The MPII dataset contains 25k images and 40k labeled human keypoint instances, with each instance labeled with 16 keypoints. The evaluation metric for the MPII dataset is the percentage of correct keypoints (PCK), calculated as follows:

[0068]

[0069] Where: d0 represents the detector, and δ represents the threshold for whether the key point matches the ground truth value.

[0070] As shown in Table 2, we tested FSA Net on the COCO Validation dataset. The results show significant improvements across various metrics. Specifically, when the backbone network is HRNet-W32, FSA Net improves by 0.9 percentage points compared to HigherHRNet, and the improvement is even more pronounced compared to networks like Personlab. When the backbone network is HRNet-W48, the network's AP reaches 71.2, an improvement of 1.3 percentage points compared to HigherHRNet. Furthermore, we can also see that AP... m With AP L All of these have been improved to varying degrees, which further confirms that the performance of the model in this invention has been enhanced to varying degrees across all scales.

[0071] Table 2

[0072]

[0073]

[0074] As shown in Table 3, we also conducted tests on the COCO test dev dataset. When the backbone network is HRNetW48, the AP reaches 69.8, which is significantly better than other mainstream bottom-up models. It is worth noting that HigherHRNet is similar to the working idea of ​​this invention. It expands the feature map area by transposing the final output of HRNet through convolution, thereby better perceiving small-scale targets. FSA Net, on the other hand, focuses on the performance of each branch, expands the receptive field through dilated convolution in parallel structure, and removes noise through branch attention operation. Experimental results show that the model of this invention is obviously better.

[0075] Table 3

[0076]

[0077] As shown in Table 4, in order to make the model more convincing, we also tested it on the MPII dataset. It can be seen that SimpleBaseline achieved 89.2 on the Hip metric, which is the best. On other metrics, the model of this invention also achieved the best results. Compared with the basic version of HRNet, all metrics have been improved to varying degrees, with Mean improving by 0.4 percentage points.

[0078] Table 4

[0079]

[0080]

[0081] To more intuitively demonstrate the effectiveness of the model of this invention, we respectively... Figure 4 and Figure 5 The experimental results were visualized, specifically the test results on COCO validation when using HRNetW48 as the backbone network. For example... Figure 4 As shown, we can clearly see that our model can easily handle scenarios with rich scale information; such as... Figure 5 As shown, the left side is the original image, the middle side is HigherHRNet, and the right side is FSA Net. It can be seen that in complex scenes, when faced with problems such as self-occlusion of people, foreground occlusion, small size of people, and complex semantic information, the model of this invention makes more accurate and reasonable predictions than other models.

[0082] This invention modifies the baseline model HRNet by adding DCB and AOB modules. To verify the effectiveness of each module, we conducted ablation experiments on the COCO Validation dataset, using the same training strategy as before. Table 5 shows that the DCB module improved AP by 5.3% compared to the baseline model, and the AOB module improved AP by 5.7%. This demonstrates that both modules contribute to the overall performance improvement, with AOB contributing more. When the two modules are combined, the overall AP improves by 7.8%, a significant improvement.

[0083] Table 5

[0084]

[0085] The model of this invention uses HRNet as a baseline and feeds the output of its four branches into the DCB module to expand the receptive field of each branch. Then, the AOB module is used to perform attention operations on each branch to enhance the network's feature representation ability. Finally, we tested it on mainstream datasets and the results are better than the baseline model and similar networks. In addition, this invention visualizes the experimental results, which can more intuitively show the improvement in FSA Net's performance.

[0086] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. Those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.

Claims

1. A high-resolution multi-person pose estimation method based on full-scale sensing balance, comprising the following steps: (1) Obtain a large number of images containing multiple people and mark the key points of the people in the images; (2) Divide all images into training set, validation set and test set according to the proportions; (3) Construct the FSA Net model framework, which includes: The backbone network is used to extract features from the input image and output four sets of feature maps. The DCB module consists of four parallel branches, each containing dilated convolutions with different dilation rates, in order to enhance the receptive field of the four sets of feature maps. The AOB module is used to perform attention operations on the feature map output by each branch from the branch and channel dimensions, thereby enhancing important features and suppressing the influence of noise. The Decoder module generates a key point heatmap by decoding the feature map output by the AOB module. Each branch of the DCB module uses BottleNeck to process the input feature map. BottleNeck consists of a convolutional layer C1, a batch normalization layer B1, a ReLU activation function R1, a dilated convolutional layer, a batch normalization layer B2, a ReLU activation function R2, a convolutional layer C2, and a batch normalization layer B3 connected in sequence. The kernel size of C1 and C2 is 1×1. The output of C1 is superimposed with the output of B3 as the output of the branch. The dilation rates of the dilated convolutional layers in the four branches of the DCB module are 9, 7, 5, and 3, respectively. The AOB module first processes the feature maps output by the four branches of the DCB module. w 1~ w 4. Perform upsampling and downsampling cross-stitching operations to generate feature maps. branchw 1~ branchw 4; Then, the feature map is processed according to the following formula. branchw 1~ branchw 4. Perform attention operations on the branch and channel dimensions to obtain feature maps. featurew 1~ featurew 4; Finally, the feature map featurew 1~ featurew 4. Perform the overlay operation to generate feature maps. w 1'~ w 4'; ; Where: AAP() represents adaptive average pooling operation, and AMP() represents adaptive max pooling operation. i =1,2,3,4; The upsampling and downsampling cross-concatenation operation refers to the feature map output from any branch. w i First, output the feature maps from other branches. w j The size is adjusted by upsampling or downsampling operations. w i Consistent, and then the feature map w 1~ w 4. The feature map is obtained by splicing them together. branchw i , j =1,2,3,4 and j ≠ i The superposition operation refers to the operation performed on any branch. i Feature maps along the channel dimension featurew 1~ featurew 4. Perform superposition to obtain the feature map. w i ' channel number and w i Consistent; (4) Input the training set images and their label information into the FSA Net model framework for training to obtain a multi-person pose estimation model; (5) Input the test set images into the trained multi-person pose estimation model to predict the position coordinates of each joint of the person in the image.

2. The high-resolution multi-person pose estimation method according to claim 1, characterized in that: The backbone network adopts a multi-layer convolutional neural network based on the HRNetW48 structure. After the input image is processed by the multi-layer convolutional neural network for feature extraction, four sets of feature maps are output.

3. The high-resolution multi-person pose estimation method according to claim 1, characterized in that: The specific process of training the model in step (4) is as follows: 4.1 Initialize model parameters, including the bias vector and weight matrix of each layer, learning rate, and optimizer; 4.2 Input the training set images into the model, and the model outputs the corresponding prediction results, i.e., the position coordinates of each joint of the person in the image, through forward propagation. Calculate the loss function between the prediction results and the labels. 4.3 Based on the loss function, the optimizer iteratively updates the model parameters using gradient descent until the loss function converges, and the training is complete; 4.4 After training is completed, the model is validated using validation set images. The model that performs best on the validation set is used as the final multi-person pose estimation model.

Images