A few-shot 2D human pose estimation method

Abstract

This invention discloses an effective method for two-dimensional human pose estimation using deep learning in the case of few samples. The method includes the following steps: (1) For labeled samples, a pose estimation model is trained using a general supervised learning method; (2) For unlabeled samples, pseudo-labels are obtained using the "consistency training" or "multi-teacher" method, and the prior and posterior uncertainty information of the pseudo-labels are estimated using an additional uncertainty branch; (3) The uncertainty estimation results are introduced into the training loss term of the unlabeled samples: the prior uncertainty is introduced as the overall weight of the loss term, and the posterior uncertainty is introduced as the loss balancing factor to complete the training of the unlabeled sample part; (4) For the model trained using the "multi-teacher" method, the uncertainty estimation results of the pseudo-labels are introduced into the fusion process to achieve higher quality pseudo-label fusion. The method proposed in this invention can effectively extract uncertainty information from unlabeled samples with as few model parameters as possible, guide the model to learn better from the pseudo-labels of unlabeled samples, and achieve high-quality two-dimensional human pose estimation performance in the case of a small number of labeled samples.

Description

Technical Field

[0001] This invention relates to a few-sample two-dimensional human pose estimation method, specifically belonging to the field of human pose estimation in computer vision. Background Technology

[0002] Human pose estimation is an important and widely applied task in computer vision. Its goal is to locate the keypoints of a given 2D image containing a human body and then arrange these keypoints in a specific order to form the human skeletal framework, known as human pose. Since the combination of keypoints is relatively fixed, accurately locating each keypoint is crucial to human pose estimation. The results of human pose estimation can serve as the basis for many downstream computer vision tasks, such as AIGC tasks controlled by human pose, and can also be directly applied to industrial fields such as gaming, virtual reality technology, and smart security.

[0003] General human pose estimation research typically uses a fixed dataset as its research object, proposing and optimizing pose estimation neural networks (i.e., models) to achieve excellent performance on that dataset. However, in practical applications, obtaining high-quality human pose datasets often means a large amount of data collection and labeling work, placing significant demands on both time and manpower. While labeling high-quality human pose data is difficult, obtaining unlabeled human images is relatively easy. Based on this, few-shot human pose estimation schemes utilizing a small amount of labeled data and a large amount of unlabeled data have become a compromise in practical applications. Existing few-shot human pose estimation schemes typically use semi-supervised learning in machine learning as a benchmark, generating pseudo-labels for unlabeled samples through self-training or mutual training. These pseudo-labels serve as supervision signals for training this portion of the data, and are used together with manually labeled labels for model training.

[0004] However, unlike manually labeled tags, the quality of pseudo-labels is unknown. High-quality pseudo-labels have a positive impact on model performance, while pseudo-labels containing high noise will hinder performance improvement. Some existing models use the confidence level of the output (in the human pose estimation task, confidence level is defined as the highest response value of the model's output heatmap) as a standard to judge its quality. However, numerous experiments show that the confidence level of such uncalibrated model outputs often differs from the quality of the output results, and sometimes it can even be deceptive. In summary, if the quality of pseudo-labels can be estimated directly or indirectly, the model can learn to discriminate between pseudo-labels of different qualities based on their quality, thereby achieving better performance. Summary of the Invention

[0005] Technical Problem: This invention provides a simple uncertainty estimation method for pseudo-label uncertainty in semi-supervised human pose estimation based on pseudo-label technology. Without altering the original model structure, this method introduces only two uncertainty estimation branches with a small number of parameters, thus achieving effective estimation of pseudo-label uncertainty. Furthermore, it utilizes uncertainty information to guide the model in discriminative learning and the fusion of multiple pseudo-labels in a multi-teacher model, ultimately improving model performance.

[0006] Technical Solution: To achieve the above objectives, the technical solution adopted by this invention is: a few-sample two-dimensional human pose estimation method, comprising the following steps:

[0007] Step 1: Collect 2D images containing human bodies and construct a dataset for 2D human pose estimation in the case of few samples. The dataset contains a small number of labeled samples (a sample includes an image containing a human body and the corresponding key point label) and a large number of unlabeled samples (a sample only includes an image containing a human body).

[0008] Step 2: Construct a model for semi-supervised human pose estimation, whose basic structure includes an encoder (backbone network, usually ResNet or HRNet) and a decoder (containing only some deconvolution layers);

[0009] Step 3: Construct the prior uncertainty estimation branch and the posterior uncertainty branch;

[0010] Step 4: For labeled samples, a fully supervised training method is used: the mean squared error loss is calculated using the sample labels and the model's prediction results, and the prior uncertainty branch is trained using manually labeled labels.

[0011] Step 5: For unlabeled samples, perform two different data augmentations (scaling and rotation) on the images they contain. The two data augmentations have different intensities (i.e., the scaling and rotation parameters are different; the parameter with a larger range of values ​​is defined as "strong augmentation", and the opposite is defined as "weak augmentation"), resulting in "strongly augmented" and "weakly augmented" images.

[0012] Step 6: Estimate the prior uncertainty of the "weakly enhanced" image using the prior uncertainty branch;

[0013] Step 7: Use the model to predict the results of the "weakly enhanced" image and the "strongly enhanced" image respectively. The former is used as the pseudo-label of the latter. At the same time, the posterior uncertainty branch is used to estimate the posterior uncertainty of the prediction process of the "strongly enhanced" image. The heteroscedasticity regression loss is calculated using the above three factors.

[0014] Step 8: If the "multi-teacher" method is used for model training, the multiple pseudo-labels are weighted and fused based on the uncertainty information estimated in Step 6 and Step 7 to obtain higher quality fused pseudo-labels for training the student model.

[0015] Step 9: Use the trained optimal model to perform inference on the test data to obtain the prediction results for the test data.

[0016] Beneficial effects

[0017] This invention addresses the issue of unreliable pseudo-label quality in existing semi-supervised human pose estimation models based on consistency training or multi-teacher networks. It proposes a method for estimating pseudo-label uncertainty and, based on this, a semi-supervised human pose estimation method with uncertainty awareness for two-dimensional human pose estimation tasks with few samples. The proposed method improves the efficiency of pseudo-label utilization and achieves better prediction accuracy on public datasets. Its advantages include:

[0018] 1. By utilizing the uncertainty prediction branch with a small number of parameters, the prior and posterior uncertainties of the human posture estimation model can be effectively estimated, providing a new index for measuring the uncertainty of false labels;

[0019] 2. In the heteroscedastic regression loss function, a method based on the uncertainty of pseudo-labels is proposed for discriminative learning, so that pseudo-labels with different uncertainties contribute differently to the loss function, thereby achieving effective learning of pseudo-labels;

[0020] 3. The proposed method achieves better prediction accuracy on public datasets compared with other existing methods. Moreover, the proposed training method does not significantly increase the number of model parameters and training time. At the same time, it has the same inference complexity as the fully supervised model, realizing an efficient semi-supervised training and inference process. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the SimpleBaseline basic model.

[0022] Figure 2 The flowchart shows a semi-supervised human pose estimation scheme based on consistency training.

[0023] Figure 3 A structural diagram of the prior uncertainty estimation branch and a schematic diagram of its application on labeled and unlabeled data;

[0024] Figure 4 A structural diagram of the branch for posterior uncertainty estimation and a schematic diagram of its application on labeled and unlabeled data;

[0025] Figure 5 The flowchart shows a semi-supervised human pose estimation scheme based on a multi-teacher network.

[0026] Figure 6 A schematic diagram illustrating the pseudo-label fusion process of multiple teacher models;

[0027] Figure 7 This is a schematic diagram of the training phase workflow for a few-sample 2D human pose estimation system. Detailed Implementation

[0028] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0029] First, we will define some basic concepts at the model level involved in this invention.

[0030] This invention uses SimpleBaseline as the basic model, which can regress multiple keypoint heatmaps from a given input image. Its structure is as follows: Figure 1 As shown, the SimpleBaseline model (hereinafter referred to as the basic model) can be roughly divided into two parts: the "encoder" and the "decoder". The encoder, also known as the "backbone network", is mainly responsible for extracting image features from the given input image. The backbone network used in the basic model is usually ResNet or HRNet. The decoder consists of a series of deconvolution layers. By performing several deconvolution operations on the input image features, the key point heatmap of the input can be obtained.

[0031] This invention proposes different uncertainty estimation schemes and model training methods for different semi-supervised human pose estimation schemes based on consistency training (using only a single network) and multi-teacher networks, which are described below.

[0032] 1. A semi-supervised human pose estimation scheme based on consistency training

[0033] The basic scheme for human pose estimation in the few-sample case used in this invention is based on consistency training. This scheme was first proposed in the literature (An Empirical Study of the Collapsing Problem in Semi-Supervised 2D Human Pose Estimation), and its process is as follows: Figure 2 As shown. In Figure 2In this study, labeled and unlabeled samples from the original data are used for model training using "supervised training" and "unsupervised training," respectively. In "supervised training," the input image is processed through a basic model, and the loss is calculated using the keypoint heatmap and corresponding ground truth labels obtained from the model output; this is called "supervised loss." In "unsupervised training," the input image is split into two groups for data augmentation. One group, with stronger augmentation, is called Sample 1, and the other, with weaker augmentation, is called Sample 2 (the strength of data augmentation varies depending on the scaling and rotation parameters; a larger range of parameter values ​​is defined as "strong augmentation," and vice versa; see the next paragraph for details). Subsequently, the basic model is used to predict the results of the two augmented samples, resulting in Prediction 1 and Prediction 2. When inferring from Sample 2, the model parameters are frozen, meaning the gradient of Prediction 2 is not calculated. Prediction 2 is used as the pseudo-label for Sample 1, and the loss between it and Prediction 1 is calculated; this is called "unsupervised loss." The consistency-based training method originates from the smoothness and clustering assumptions in semi-supervised learning, and it can extract as much information as possible from unlabeled samples to improve model performance. Two points need attention during consistency training: First, the labeled samples used for supervised training also need to undergo a weak augmentation before being fed into the model, rather than no data augmentation at all; second, since the basic model used in this invention is based on keypoint heatmap regression, and the heatmap will deform accordingly with the rotation and scaling of the input samples, the heatmap output by the basic model needs to undergo an inverse transformation of the input sample data augmentation before being used for loss calculation. Because the training process of this scheme only involves one model (network), it is also called a "single network structure".

[0034] The strong enhancements employed in this invention include: random rotation within the range of [-60°, 60°], random scaling within the range of [0.5, 1.5], random keypoint region masking (Joint Cutout, which uses gray squares to mask square regions near certain keypoints in the input image), and random keypoint region swapping (Cut-Occlude, which swaps square regions near certain keypoints in the input image); the weak enhancements include: random rotation within the range of [-30°, 30°] and random scaling within the range of [0.75, 1.25].

[0035] The main performance improvement of the proposed solution comes from the estimation of the uncertainty of pseudo-labels obtained during the aforementioned consistency training process. In consistency training, a strong-weak enhancement pair approach is used for unlabeled samples, and the model generates pseudo-labels for the samples themselves. However, not all pseudo-labels generated by the model have quality close to that of human annotations. If the pseudo-labels deviate too much from the ground truth labels, it will affect the model's performance. To address this, the present invention proposes a method for estimating the uncertainty of pseudo-labels and guiding the model training process based on the estimation results: the uncertainty introduced during the prediction process of the baseline model is modeled into two parts, namely, prior uncertainty and posterior uncertainty, whose sources correspond to the encoder and decoder in the model structure, respectively. When estimating the uncertainties of the encoder and decoder, it is assumed that the uncertainty of the other structure is zero, that is, the prior and posterior uncertainties are estimated separately in a decoupled manner. The uncertainty estimation method used in this invention has the following characteristics: First, the smallest unit of uncertainty estimation is the keypoint, meaning each keypoint has a corresponding uncertainty; second, it does not change the structure of the basic model, but only introduces two additional prediction branches, thus it can be used as a plug-and-play module; third, when using the trained model for inference, the two introduced branches do not participate in the inference process, therefore the model used in the inference stage is the same size as the model obtained through fully supervised learning and consistency training, without introducing additional inference overhead; fourth, the uncertainty prediction branches are not only compatible with the SimpleBaseline model, but can also be used with other models with similar structures. It should be noted that only unlabeled samples need uncertainty estimation; therefore, the uncertainty estimation result only affects the unsupervised training process (unsupervised loss), while the unlabeled samples actually input to the uncertainty estimation branches are their weakly enhanced images.

[0036] Prior uncertainty represents the model's "familiarity" with the input unlabeled samples, i.e., whether similar samples appear in the labeled samples, and assigns an uncertainty score to this. The prior uncertainty estimation method used in this invention is as follows: Figure 3As shown. The structure of the prior uncertainty estimation branch contains only a max pooling layer (MaxPool) and a fully connected classification layer (FC). The input of this branch comes from the image features extracted by the encoder and the key point heatmap regressed by the decoder. By upsampling the image features to the same scale as the heatmap and then multiplying them with the heatmap, the feature map is locally activated through the highlighted areas in the heatmap to obtain the local features of the corresponding key points. This invention memorizes the labeled samples by training the prior uncertainty estimation branch on labeled samples to classify local features, and uses the classification score output by the branch as the value of the prior uncertainty on unlabeled samples. Since the classification layer uses Sigmoid as the activation function, the output uncertainty score is between 0 and 1. The loss function used to train the classification layer is shown in Equation (1), where x is the input sample (image), y is the loss function of the image, and y is the loss function of the image. c and P c Both are J-dimensional vectors (J being the number of keypoints). The former is the output vector of the classification layer, and the latter is the ground truth vector of keypoint categories using one-hot encoding (for the i-th keypoint, the i-th position of the vector is 1, and the other positions are 0). For unlabeled samples, the estimated prior uncertainty value is directly applied to the unsupervised loss function as a loss mask, denoted as M. The prior uncertainty estimated by the above method is proportional to the model's familiarity with the input samples; that is, the higher the prior uncertainty, the higher the theoretical accuracy of the model's prediction results for the input samples should be.

[0037] L cls (x)=-y c *logP c -(1-y c )*log(1-P c )#(1)

[0038] Posterior uncertainty measures the uncertainty of the model's decision-making process, specifically the uncertainty introduced by the baseline model in generating pseudo-labels. This invention models a normal distribution with variable variance of the pseudo-label distribution by introducing a "heteroscedasticity assumption" into the pseudo-label data distribution space, and uses the variance of the normal distribution as the posterior uncertainty corresponding to the pseudo-label. Specifically, general regression tasks typically assume that the regressed object follows a normal distribution in the data space, with the mean of this distribution being the true label and the variance being a fixed value. The process of completing a regression task through deep learning can be understood as fitting the variable parameters of this distribution, i.e., the mean. Therefore, the loss function used in common regression tasks is the mean squared error (MSE) loss. Unlike general regression tasks, this invention models the spatial distribution of each pseudo-label as a normal distribution with variable mean and variance, meaning that the variance of the distribution corresponding to each pseudo-label is different. Therefore, the variable parameters of the distribution that the model needs to fit simultaneously include both the mean and variance. This is achieved by adding an extra branch to the original baseline model for variance prediction, such as... Figure 4 As shown. After introducing variable variance into the normal distribution, the corresponding "heteroscedasticity loss" can be obtained through maximum likelihood estimation, as shown in the equation. The posterior uncertainty estimation branch used in this invention first includes a channel attention layer, the implementation of which is derived from the literature (Squeeze-and-Excitation Networks); secondly, it includes an uncertainty estimation module consisting of three 3×3 convolutional layers and two fully connected layers. Through this branch, the model can output an additional variance value, i.e., the posterior uncertainty value, for each input sample based on the heatmap output by the baseline model. On labeled samples, this prediction branch does not participate in training, while on unlabeled samples, heteroscedasticity loss is used for training. The posterior uncertainty estimated by the above method is proportional to the dispersion of the pseudo-labels generated by the model in space, that is, the higher the posterior uncertainty, the lower the theoretical accuracy of the model's prediction results for the input samples should be. In the equation, x u For unlabeled samples (images), T h and T e These are strong enhancement and weak enhancement, respectively. f represents the forward inference process of the baseline model, σ is the variance, and M is the predicted value of the prior uncertainty, which is directly added to the heteroscedasticity loss in the form of a loss mask.

[0039]

[0040] The heteroscedasticity loss proposed in this invention replaces the unsupervised loss used in the original consistency training and is incorporated into the overall model loss, as shown in the equation. Where L sup The supervised loss for the labeled data is λ1 and λ2, which are loss balancing factors. In this invention, they are both set to 1.0.

[0041] L = L sup +λ1L het +λ2L cls #(3)

[0042] 2. A semi-supervised human pose estimation scheme based on multi-teacher networks

[0043] Consistency-based training schemes can largely utilize a single network to extract information from unlabeled samples, and the introduction of uncertainty information can further enhance this process. However, the information-gathering capacity of a single network is limited, thus constraining the quality of pseudo-label generation. To address this, the papers (Semi-Supervised 2D Human Pose Estimation Driven by Position Inconsistency Pseudo Label Correction Module) and (Denoising and Selecting Pseudo-Heatmaps for Semi-Supervised Human Pose Estimation) have proposed schemes utilizing multiple networks for mutual learning, such as... Figure 5 As shown, unlike consistency training which uses one model to generate pseudo-labels for unlabeled samples and then uses the same model for prediction, the multi-teacher network scheme includes two models (referred to as Model 1 and Model 2). For the same input sample, the prediction result of the weakly augmented sample generated by Model 1 is used as the pseudo-label for the prediction result of Model 2, and the prediction result of the weakly augmented sample generated by Model 2 is used as the pseudo-label for the prediction result of Model 1. This is a kind of "mutual training". This training method can overcome the performance limitations of a single network in data cognition and enhance the overall perception ability of the model through interactive learning between the two models.

[0044] The uncertainty estimation method and corresponding model training method proposed in this invention are also applicable to multi-teacher network schemes. The specific implementation steps are as follows: Within one training epoch, a supervised learning process is first completed on labeled samples using a baseline model. Then, pseudo-labels are generated for unlabeled samples through the interactive learning process described above (models 1 and 2 use different strong and weak augmentation methods). Next, the prior and posterior uncertainty estimation branches are used to estimate the prior and posterior uncertainties of the two models respectively (implementation method consistent with single-network structure). Finally, the prediction results of the two models for weakly augmented samples are swapped as pseudo-labels for the prediction results of the other model for strongly augmented samples, and substituted into the unsupervised loss function (now heteroscedasticity regression loss) to calculate the loss. In summary, the difference in the use of the multi-teacher network structure lies only in using the pseudo-label predicted by one network as the regression target of the prediction result of another network. The uncertainty method proposed in this invention is used in this structure in a way that is completely consistent with the scheme based on consistency training. After training the two models, usually one model is selected as the model used in the final inference process, rather than using both models simultaneously, thus ensuring consistency with the inference overhead of the consistency training scheme. Because the training process of this scheme involves two models (networks), it is also called a "dual network structure".

[0045] Based on the above scheme, another model (called Model 3) can be introduced as the student model for Model 1 and Model 2 (Model 1 and 2 are called teacher models), such as... Figure 6 As shown. Specifically, the above dual-network structure only selects one of Model 1 and Model 2 as the final inference model. However, if the prediction results of the two can be fused, better prediction results can be achieved. Therefore, in one training cycle, the following steps can be added: use Model 1 and Model 2 to obtain the prediction results of two weakly augmented samples, fuse the two prediction results to obtain the final pseudo-label; use Model 3 to predict the strongly augmented samples, and use the fused pseudo-label as the regression target of the prediction result, adding its loss to the unsupervised loss term. Since the fused pseudo-label has higher accuracy, the inference performance of Model 3 will be higher than that of Model 1 and Model 2. Since the training process of this scheme involves three models (networks), it is also called a "three-network structure". The general three-network structure usually calculates the maximum response coordinate position of the two pseudo-labels (heatmaps), calculates the midpoint of the two positions, and then renders a new heatmap centered on the midpoint as the fused pseudo-label. For this process, this invention proposes to use the uncertainty estimation result of the two pseudo-labels as an indicator to guide the fusion process. The specific method is: for the pseudo-labels predicted by two or more teacher models, calculate the position P of its maximum response. i, referred to as candidate positions; such that the distance between each candidate position and the final synthesis position P is proportional to the uncertainty of the pseudo-label corresponding to each position, that is, the higher the uncertainty of the pseudo-label, the smaller its contribution to the final synthesis result. The calculation process is shown in the formula. Where, n c This represents the number of candidate positions (i.e., the number of teacher models). Let m be the prior uncertainty. i and posterior uncertainty σ i The synthesis results are shown above. The process can be understood as replacing the general arithmetic mean with a weighted average, and using uncertainty as a weight to guide the fusion of pseudo-labels. Since candidate labels with lower uncertainty contribute more, the pseudo-labels obtained through weighted fusion have higher accuracy than those obtained through arithmetic averaging. It should be noted that for Model 3 in the three-network structure, uncertainty estimation is not used in the inference stage; Model 3 is only used as the model for inference, therefore its inference cost is consistent with that of the single-network structure.

[0046]

[0047] Example

[0048] I. Configuration and Model Description of a Few-Sample Two-Dimensional Human Pose Estimation System

[0049] The proposed few-shot 2D human pose estimation system is a software program based on modern computers, requiring an NVIDIA graphics card and support for the CUDA computing library and the PyTorch deep learning framework. This invention provides a method for efficiently training a human pose estimation network on a small number of labeled samples, specifically by training a benchmark model using images and labeled data organized into a dataset, and then performing inference and testing on additional data. The workflow of the system training phase is as follows: Figure 7 As shown.

[0050] Once trained, the model can be used for inference and testing, just like a regular deep learning model.

[0051] II. Implementation Process of a Few-Sample Two-Dimensional Human Pose Estimation System

[0052] The following describes a complete training and inference process for a few-shot 2D human pose estimation system. The training method used in the example is a semi-supervised human pose estimation scheme based on consistency training.

[0053] The training and testing data used in the examples are both from the Microsoft COCO dataset, which contains 118,000 images and provides 150,000 human body instances, each annotated with 17 keypoints (some of which may not be visible in the image). Sampling is performed on all data in this dataset to obtain the few-sample human pose data required for this invention. The COCO dataset is sampled to obtain subsets containing 1000, 5000, and 10000 labeled samples, respectively, referred to as COCO-1K, COCO-5K, and COCO-10K. In these subsets, all samples except the labeled ones are unlabeled, therefore the total number of samples in each subset is consistent.

[0054] The example uses a server running Ubuntu 20.04 to train and test the model. The server is equipped with an Intel(R) Xeon(R) Gold 6226R processor and an NVIDIA GeForce RTX 3090 graphics card (with 24GB of VRAM). The deep learning framework used in the experiment is PyTorch 2.3.0, with CUDA 12.1 for computational acceleration. The training model consists of 100 epochs. The optimizer used is Adam, with PyTorch's default initialization parameters; the initial learning rate is 0.001, which decays to 0.0001 and 0.00001 in the 70th and 90th epochs, respectively.

[0055] like Figure 7 As shown, within one training epoch, labeled and unlabeled images are subjected to supervised and unsupervised training, respectively. Before training, all images in the samples are cropped to a 4:3 aspect ratio, scaled to 256×192 pixels, and undergo one data augmentation (weak augmentation). Therefore, to obtain strongly augmented samples later, only weak augmentation needs to be performed again on the weakly augmented samples.

[0056] When training the model using samples, batch training is used to speed up the training process. The batch size is set to 128. Subsequent discussions of the samples are based on the batch data, which are presented as tensors in PyTorch.

[0057] For labeled samples (size 128×3×256×192), after passing through the baseline model, the predicted results (keypoint heatmap, size 128×17×64×48) are obtained. The supervised loss is calculated based on the corresponding labels (size consistent with the predicted results). Figure 3As shown, the key point classification result is calculated using the image features of the labeled samples (size 128×512×8×6) and the prediction results. The cross-entropy loss is calculated as the classification loss using an identity matrix of size 128×17×17.

[0058] For unlabeled samples (size 128×3×256×192), the model weights are first frozen. The weakly augmented samples are then predicted, and their predictions are treated as pseudo-labels (size 128×17×64×48). The model is then unfrozen, and the strongly augmented samples are used for prediction to obtain the predicted results. According to... Figure 3 and Figure 4 The network structure shown is used to calculate the prior and posterior uncertainties of the pseudo-labels (both are 128×17). After obtaining the prediction results of the pseudo-labels, the strongly enhanced samples, and the two uncertainties, the unsupervised loss in the case of heteroscedasticity is calculated according to Equation (2).

[0059] After applying supervised loss, unsupervised loss, and classification loss, these losses are weighted and fused together to optimize the overall loss of the system until the training rounds reach the set value and training is completed.

Claims

1. A method for estimating two-dimensional human pose with a small number of samples, characterized in that, Includes the following steps: Step 1: Collect 2D images containing human bodies and construct a dataset for 2D human pose estimation in the case of few samples, which includes a small number of labeled samples and a large number of unlabeled samples. Step 2: Construct a model for semi-supervised human pose estimation, whose structure includes an encoder and a decoder; Step 3: Construct the prior uncertainty estimation branch and the posterior uncertainty branch; Step 4: For labeled samples, train using a fully supervised approach; Step 5: For unlabeled samples, perform two different data augmentations on the images they contain, including scaling and rotation, to obtain "strongly augmented" and "weakly augmented" images; Step 6: Estimate the prior uncertainty of the "weakly enhanced" image using the prior uncertainty branch; Step 7: Use the model to predict the results of "weakly enhanced" and "strongly enhanced" images respectively. The former is used as the pseudo-label of the latter. At the same time, the posterior uncertainty branch is used to estimate the posterior uncertainty of the prediction process of "strongly enhanced" images. The heteroscedasticity regression loss is calculated using the above three factors. Step 8: If the "multi-teacher" method is used for model training, the multiple pseudo-labels are weighted and fused based on the uncertainty information estimated in Step 6 and Step 7 to obtain higher quality fused pseudo-labels for training the student model. Step 9: Use the trained optimal model to perform inference on the test data to obtain the prediction results for the test data; In step 6, the structure of the prior uncertainty branch estimation includes a max pooling layer and a fully connected classification layer. Its input comes from the image features extracted by the encoder and the key point heatmap regressed by the decoder. By upsampling the image features to the same scale as the heatmap and then multiplying them with the heatmap, the feature map is locally activated through the highlighted areas in the heatmap to obtain the local features of the corresponding key points. The prior uncertainty estimation branch is trained on labeled samples to classify the local features to memorize the labeled samples, and the classification score output by the branch is used as the value of the prior uncertainty on unlabeled samples. In step 7, the posterior uncertainty branch estimation first includes a channel attention layer; secondly, it includes an uncertainty estimation module consisting of three 3×3 convolutional layers and two fully connected layers. By using the posterior uncertainty branch, the model can output an additional variance value, i.e., the posterior uncertainty value, for each input sample, based on the heatmap output by the baseline model.

2. The method for estimating two-dimensional human pose with few samples according to claim 1, characterized in that, In step 1, a labeled sample is a sample that includes an image containing a human body and the corresponding key point labels, while an unlabeled sample is a sample that only includes an image containing a human body.

3. The method for estimating two-dimensional human pose with few samples according to claim 1, characterized in that, In step 4, the fully supervised training method is as follows: the mean squared error loss is calculated using the sample labels and the model's prediction results, and the prior uncertainty branch is trained using manually labeled labels.

4. The method for estimating two-dimensional human pose with few samples according to claim 1, characterized in that, In step 5, the two types of data augmentation have different intensities, that is, the scaling and rotation parameters are different. The parameter with a larger range of values ​​is defined as "strong augmentation", and the opposite is defined as "weak augmentation".

5. The method for estimating two-dimensional human pose with few samples according to claim 4, characterized in that, The strong enhancement includes: Random rotation within the range Random scaling within the range, random keypoint region masking, random keypoint region swapping; weak enhancements include: Random rotation within the range Random scaling within the range.

6. The method for estimating two-dimensional human pose with few samples according to claim 1, characterized in that, The classification layer backend uses Sigmoid as the activation function, and the output uncertainty score is between 0 and 1; the loss function used to train the classification layer is shown in the following formula: ； in, For the input sample, The output vector of the classification layer. This is a truth vector of keypoint categories using one-hot encoding.

7. The method for estimating two-dimensional human pose with few samples according to claim 6, characterized in that, On labeled samples, the posterior uncertainty branch is not used in training. On unlabeled samples, heteroscedasticity loss is used for training, as shown in the following equation: ； in, These are unlabeled samples. and These are "strong enhancement" and "weak enhancement," respectively. Represents the forward inference process of the baseline model. For variance, This is the predicted value of the prior uncertainty; The heteroscedasticity loss is replaced by the unsupervised loss used in the original consistency training and added to the overall loss of the model, as shown in the following equation: ； in, Supervision loss for labeled data, and This is the loss balance factor.

8. The method for estimating two-dimensional human pose with few samples according to claim 1, characterized in that, In step 8 For pseudo-labels predicted by two or more teacher models, calculate the location of their maximum response. These are called candidate positions; each candidate position is associated with a final synthesized position. The distance is proportional to the uncertainty of the pseudo-label corresponding to each position; that is, the higher the uncertainty of the pseudo-label, the smaller its contribution to the final synthesis result. The calculation process is shown in the following formula: ； in, This represents the number of candidate positions, i.e., the number of teacher models. For prior uncertainty and posterior uncertainty The synthesis results.

Images