Few-sample two-dimensional human body posture estimation method

Abstract

The invention discloses an effective method for carrying out two-dimensional human body posture estimation by using deep learning under the condition of few samples, and the method comprises the following steps: (1) for labeled samples, training a posture estimation model by using a universal supervised learning mode; (2) for an unlabeled sample, obtaining a pseudo label of the unlabeled sample by adopting a'consistency training 'or'multi-teacher' method, and respectively estimating prior and posterior uncertainty information of the pseudo label by utilizing an additional uncertainty branch; (3) introducing an uncertainty estimation result into the training loss item of the unlabeled sample: introducing the prior uncertainty as the overall weight of the loss item, and introducing the posterior uncertainty as a loss balance factor to complete the training of the unlabeled sample part; and (4) for a model trained by adopting a multi-teacher method, introducing an uncertainty estimation result of the pseudo tag into a fusion process to realize higher-quality pseudo tag fusion. According to the method provided by the invention, the uncertainty information in the label-free sample can be effectively extracted under the condition that the model parameter quantity as few as possible is introduced, the model is guided to better learn from the pseudo label of the label-free sample, and the high-quality two-dimensional human body posture estimation performance is realized under the condition of a small number of labeled samples.

Description

Technical Field

[0001] The present invention relates to a method for estimating two-dimensional human body posture with a small number of samples, and in particular belongs to the field of human body posture estimation in the field of computer vision. Background Art

[0002] Human pose estimation is an important and widely used task in the field of computer vision. Its goal is to locate the positions of the key points of a given two-dimensional image containing a human body, and combine the located key points into a human skeleton framework in a specific order, which is called human pose. Since the combination of key points is relatively fixed, how to accurately locate each key point is the focus of the human pose estimation task. The results of human pose estimation can not only be the basis of many downstream computer vision tasks: such as AIGC tasks with human pose as the control condition, but can also be directly applied to industrial fields such as games, virtual reality technology and intelligent security.

[0003] General human pose estimation research usually takes a fixed data set as the research object, proposes and optimizes the pose estimation neural network (i.e., model) to achieve excellent performance on the data set. However, in practical applications, obtaining a high-quality human pose data set often means a lot of data collection and annotation work, which places great demands on both time and manpower. Although it is difficult to annotate high-quality human pose data, it is relatively easy to obtain unlabeled human images. Based on this, the few-shot human pose estimation scheme using a small amount of labeled data and a large amount of unlabeled data has become a compromise solution in practical applications. Existing few-shot human pose estimation schemes usually use semi-supervised learning in machine learning as a benchmark, and use self-training or mutual training to generate pseudo labels for unlabeled samples as supervisory signals for training this part of the data, and are used together with manually labeled labels for model training.

[0004] However, unlike manually annotated labels, the quality of pseudo labels is unknown. High-quality pseudo labels have a positive effect on model performance, while pseudo labels with high noise will hinder the improvement of model performance. Some existing models use the confidence of the output results (taking the human pose estimation task as an example, the confidence is defined as the highest response value of the model output heat map) as the standard for judging their quality, but a large number of experiments have shown that the confidence of such uncalibrated model outputs is often different from the quality of the output results, and sometimes even deceptive. In summary, if the quality of pseudo labels can be estimated directly or indirectly, the model can discriminate between pseudo labels of different qualities according to their quality to achieve better performance. Summary of the invention

[0005] Technical problem: The present invention provides a simple uncertainty method for estimating the uncertainty of pseudo labels for a semi-supervised human posture estimation method based on pseudo labeling technology. Without changing the original model structure, this method only introduces two uncertainty estimation branches with very few parameters, which effectively estimates the uncertainty of pseudo labels, and uses uncertainty information to guide the model to perform discriminative learning and guide the fusion of multiple pseudo labels in the "multi-teacher" model, ultimately improving the model performance.

[0006] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is: a method for estimating a two-dimensional human body posture using a small number of samples, comprising the following steps:

[0007] Step 1: Collect 2D images containing human bodies and construct a dataset for 2D human pose estimation in the case of few samples, including a small number of labeled samples (one sample includes an image containing human bodies and corresponding key point labels) and a large number of unlabeled samples (one sample only includes images containing human bodies);

[0008] Step 2: Build a model for semi-supervised human pose estimation, whose basic structure includes an encoder (backbone network, usually ResNet or HRNet) and a decoder (containing only some convolution layers);

[0009] Step 3: Construct a priori uncertainty estimation branch and a posteriori uncertainty branch;

[0010] Step 4: For labeled samples, a fully supervised approach is used for training: the mean square error loss is calculated using the sample labels and the model's prediction results, and the prior uncertainty branch is trained using the manually labeled labels;

[0011] Step 5: For the unlabeled samples, two different data enhancements (scaling and rotation) are performed on the images they contain. The two data enhancements have different strengths (i.e., the parameters of scaling and rotation are different in size, and the parameter with a larger value range is defined as "strong enhancement", and vice versa, it is defined as "weak enhancement"), and a "strongly enhanced" image and a "weakly enhanced" image are obtained;

[0012] Step 6: Use the prior uncertainty branch to estimate the prior uncertainty of the “weakly enhanced” image;

[0013] Step 7: Use the model to predict the results of the "weakly enhanced" image and the "strongly enhanced" image respectively. The former is used as the pseudo label of the latter. At the same time, the posterior uncertainty branch is used to estimate the posterior uncertainty of the "strongly enhanced" image prediction process. The above three are used to calculate the heteroscedastic regression loss.

[0014] Step 8: If the "multi-teacher" method is used for model training, multiple pseudo labels are weighted fused according to the uncertainty information estimated in steps 6 and 7 to obtain a higher quality fused pseudo label for training the student model;

[0015] Step 9: Use the trained optimal model to perform inference on the test data to obtain the prediction results of the test data.

[0016] Beneficial Effects

[0017] The present invention aims to solve the problem that the quality of pseudo labels cannot be guaranteed in existing semi-supervised human pose estimation models based on consistency training or multi-teacher networks. A method for estimating the uncertainty of pseudo labels is proposed, and based on this, a semi-supervised human pose estimation method with uncertainty perception is proposed for two-dimensional human pose estimation tasks under few-sample conditions. The method proposed by the invention improves the utilization efficiency of pseudo labels and achieves better prediction accuracy on public data sets. Its advantages include:

[0018] 1. The uncertainty prediction branch with a small number of parameters is used to effectively estimate the prior and a posteriori uncertainties of the human pose estimation model, providing a new indicator for measuring the uncertainty of pseudo labels;

[0019] 2. In the heteroskedastic regression loss function, a method of discriminative learning based on the uncertainty of pseudo labels is proposed, so that pseudo labels with different uncertainties contribute differently to the loss function, thus achieving effective learning of pseudo labels;

[0020] 3. The proposed method achieved better prediction accuracy on public datasets compared with other existing methods. The proposed training method did not significantly increase the number of model parameters and training time. At the same time, it had the same reasoning complexity as the fully supervised model, realizing an efficient semi-supervised training and reasoning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a schematic diagram of the structure of the basic model SimpleBaseline;

[0022] Figure 2 Flowchart of the semi-supervised human pose estimation scheme based on consistency training;

[0023] Figure 3 The structural diagram of the prior uncertainty estimation branch and its schematic diagram of its use on labeled and unlabeled data;

[0024] Figure 4 The structure diagram of the posterior uncertainty estimation branch and its use on labeled and unlabeled data;

[0025] Figure 5 Flowchart of the semi-supervised human pose estimation scheme based on multi-teacher networks;

[0026] Figure 6 Schematic diagram of the pseudo-label fusion process of multiple teacher models;

[0027] Figure 7 Schematic diagram of the workflow of the training phase of the few-sample 2D human pose estimation system. DETAILED DESCRIPTION

[0028] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0029] First, some basic concepts at the model level involved in the present invention are defined.

[0030] The present invention uses SimpleBaseline as the basic model, which can complete the task of regressing multiple key point heat maps from a given input image. Its structure is as follows: Figure 1 As shown in the figure. The structure of the SimpleBaseline model (hereinafter referred to as the basic model) can be roughly divided into two parts: "encoder" and "decoder". The encoder part can also be called the "backbone network", which is mainly used to extract image features from a given input image. The backbone network used by the basic model is usually ResNet or HRNet; the decoder part is a series of deconvolution layers, which can obtain the input key point heat map by performing several deconvolution operations on the input image features.

[0031] The present invention proposes different uncertainty estimation schemes and model training methods for different semi-supervised human posture estimation schemes based on consistency training (using only a single network) and multi-teacher networks, which are described below respectively.

[0032] 1. Semi-supervised human pose estimation scheme based on consistency training

[0033] The basic scheme of human pose estimation in the case of few samples used in the present invention is based on consistency training, which was first proposed by the literature (An Empirical Study of the Collapsing Problem in Semi-Supervised 2D Human Pose Estimation). The process is as follows Figure 2 As shown. Figure 2In the above, the labeled samples and unlabeled samples in the original data are used for model training by "supervised training" and "unsupervised training" respectively. In the "supervised training" method, the input image is passed through the basic model, and the loss is calculated by the key point heat map obtained by the model output and the corresponding true value label, which is called "supervised loss". In the "unsupervised training" method, the input image is divided into two paths for data enhancement, one of which is a stronger enhancement method, called sample 1, and the other is a weaker enhancement method, called sample 2 (the strength of data enhancement varies according to the size of the scaling and rotation parameters. The parameter with a larger range of values ​​is defined as "strong enhancement", and vice versa, it is defined as "weak enhancement", see the next paragraph for details). Subsequently, the basic model is used to predict the results of the two enhanced samples to obtain predictions 1 and 2. Among them, the model parameters need to be frozen when reasoning about sample 2, that is, the gradient of prediction 2 is not calculated. Prediction 2 is used as the pseudo label of sample 1, and the loss between it and prediction 1 is calculated, which is called "unsupervised loss". The consistency-based training method originates from the smoothing hypothesis and clustering hypothesis in the field of semi-supervised learning, and can obtain information from unlabeled samples as much as possible to improve the performance of the model. During the consistency training process, two points need to be noted: first, the labeled samples used for supervised training also need to be weakly enhanced before being sent to the model, rather than without data enhancement; second, since the basic model used in the present invention is based on key point heat map regression, and the heat map will undergo corresponding deformation with the rotation and scaling of the input sample, the heat map output by the basic model needs to be inversely transformed after the input sample data enhancement before being used for loss calculation. Since the training process of this scheme only involves one model (network), it is also called a "single network structure."

[0034] The strong enhancement adopted by the present invention includes: random rotation in the range of [-60°, 60°], random scaling in the range of [0.5, 1.5], random key point area masking (Joint Cutout, using gray blocks to cover the square areas near certain key points in the input image), and random key point area swapping (Cut-Occlude, exchanging the square areas near certain key points in the input image); weak enhancement includes: random rotation in the range of [-30°, 30°], and random scaling in the range of [0.75, 1.25].

[0035] The main performance improvement of the solution proposed in the present invention comes from the estimation of the uncertainty of the pseudo-labels obtained in the above-mentioned consistency training process. In consistency training, a strong-weak enhancement pair method is adopted for unlabeled samples, and a model is used to generate pseudo-labels for the samples themselves. However, not all pseudo-labels generated by the model have a quality close to that of manual labeling. If the deviation between the pseudo-label and the true value label is too large, it will affect the performance of the model. In this regard, the present invention proposes a method for estimating the uncertainty of pseudo-labels and guiding the model training process based on the estimation results: the uncertainty introduced in the prediction process of the benchmark model is modeled into two parts, namely, a priori uncertainty and a posteriori uncertainty, and their sources correspond to the encoder and decoder in the model structure respectively. When estimating the uncertainty of the encoder and decoder, it is respectively assumed that the uncertainty of the other structure is zero, that is, the a priori and a posteriori uncertainties are estimated in a decoupled manner. The uncertainty estimation method used in the present invention has the following characteristics: first, the minimum unit of uncertainty estimation is the key point, that is, each key point has a corresponding uncertainty; second, the structure of the basic model is not changed, but only two additional prediction branches are introduced, so it can be used as a plug-and-play module; third, when the trained model is finally used for reasoning, the two introduced branches do not participate in the reasoning process, so the model used in the reasoning stage is consistent with the model obtained by full supervised learning and the model obtained by consistency training, and no additional reasoning overhead is introduced; fourth, the uncertainty prediction branch is not only compatible with the SimpleBaseline model, but also can be used for other models with similar structures. It should be noted that only unlabeled samples need to estimate uncertainty, so the uncertainty estimation result only acts on the unsupervised training process (unsupervised loss), and the unlabeled samples actually input into the uncertainty estimation branch are their weakly enhanced images.

[0036] The prior uncertainty represents the model's "familiarity" with the input unlabeled samples, that is, it determines whether similar samples appear in the labeled samples and gives an uncertainty score for this. The prior uncertainty estimation method used in the present invention is as follows: Figure 3As shown. The structure of the prior uncertainty estimation branch only includes a maximum pooling layer (MaxPool) and a fully connected classification layer (FC). The input of this branch comes from the image features extracted by the encoder and the key point heat map obtained by the decoder regression. The image features are upsampled to the same scale as the heat map and then multiplied by the heat map to locally activate the feature map through the highlighted area in the heat map to obtain the local features of the corresponding key points. The present invention memorizes the labeled samples by training the prior uncertainty estimation branch on the labeled samples to classify the local features, and uses the classification score output by the branch on the unlabeled samples as the value of the prior uncertainty. Since the back end of the classification layer uses Sigmoid as the activation function, the output uncertainty score is between 0 and 1. The loss function used for training the classification layer is shown in formula (1), where x is the input sample (image), y is the input sample (image), c and P c Both are J-dimensional vectors (J is the number of key points). The former is the output vector of the classification layer, and the latter is the true value vector of the key point category encoded using One-hot (for the i-th key point, the i-th position of the vector is 1, and the rest are 0). For unlabeled samples, the estimated prior uncertainty value will be directly applied to the unsupervised loss function as a loss mask, denoted as M. The prior uncertainty estimated by the above method is proportional to the model's familiarity with the input sample, that is, the higher the prior uncertainty, the higher the theoretical accuracy of the model's prediction results for the input sample.

[0037] L cls (x) = -y c *logP c -(1-y c )*log(1-P c )#(1)

[0038] The posterior uncertainty is used to measure the uncertainty of the model decision process, that is, the uncertainty introduced by the process of generating pseudo-labels by the benchmark model. The present invention introduces the "heteroskedasticity assumption" in the pseudo-label data distribution space, thereby modeling a normal distribution with variable variance of the pseudo-label distribution, and uses the variance of the normal distribution as the posterior uncertainty corresponding to the pseudo-label. Specifically, general regression tasks usually assume that the regression object presents a normal distribution in the data space, the mean of the distribution is its true value label, and the variance is a fixed value. The process of completing the regression task through deep learning can be understood as fitting the variable parameter of the distribution, that is, the mean. Therefore, the loss function used in common regression tasks is the mean square error (MSE) loss. Unlike general regression tasks, the present invention models the spatial distribution of each pseudo-label as a normal distribution with variable mean and variance, that is, the variance of the distribution corresponding to each pseudo-label is different. Therefore, the variable parameters of the distribution that the model needs to fit include both the mean and the variance. The implementation process is to add an additional branch for the prediction of the variance on the basis of the original benchmark model, such as Figure 4 As shown. After introducing the variable variance into the normal distribution, the corresponding "heteroscedasticity loss" can be obtained through maximum likelihood estimation, as shown in the formula. The posterior uncertainty estimation branch adopted by the present invention first includes a channel attention layer, and its implementation method comes from the literature (Squeeze-and-Excitation Networks); secondly, it includes an uncertainty estimation module composed of 3 3×3 convolutional layers and 2 fully connected layers. Through this branch, the model can output an additional variance value, that is, the posterior uncertainty value, for each input sample based on the heat map output by the baseline model. On the labeled samples, the prediction branch does not participate in the training, while on the unlabeled samples, heteroscedasticity loss is used for training. The posterior uncertainty estimated by the above method is proportional to the degree of dispersion of the pseudo-labels generated by the model in space, that is, the higher the posterior uncertainty, the lower the theoretical accuracy of the model's prediction results for the input samples. In the formula, x u is an unlabeled sample (image), T h and T e are strong enhancement and weak enhancement respectively, f represents the forward reasoning process of the baseline model, σ is the variance, and M is the predicted value of the prior uncertainty, which is directly added to the heteroscedastic loss in the form of a loss mask.

[0039]

[0040] The heteroscedasticity loss proposed in the present invention replaces the unsupervised loss used in the original consistency training and is added to the overall loss of the model, as shown in the formula. sup is the supervision loss of the labeled data, λ1 and λ2 are loss balancing factors, and are both set to 1.0 during use in the present invention.

[0041] L=L sup +λ1L het +λ2L cls #(3)

[0042] 2. Semi-supervised human pose estimation scheme based on multi-teacher network

[0043] The scheme based on consistency training can utilize a single network to obtain information from unlabeled samples to a great extent, and the introduction of uncertainty information can also make this process more efficient. However, the ability of a single network to obtain information is limited, which restricts the quality of pseudo-label generation. In this regard, the literature (Semi-Supervised 2D Human PoseEstimation Driven by Position InconsistencyPseudo Label Correction Module) and the literature (Denoising and Selecting Pseudo-Heatmaps forSemi-Supervised Human PoseEstimation) also proposed a scheme to use multiple networks for mutual learning, such as Figure 5 As shown. Unlike consistency training, which uses one model to generate pseudo labels for unlabeled samples and then uses the same model for prediction, the multi-teacher network solution contains two models (called model 1 and model 2). For the same input sample, the weakly enhanced sample prediction results generated by model 1 are used as pseudo labels for the prediction results of model 2, and the weakly enhanced sample prediction results generated by model 2 are used as pseudo labels for the prediction results of model 1, which is a "mutual training" method. This training method can overcome the performance limitations of a single network in data cognition and enhance the overall perception ability of the model through interactive learning of the two models.

[0044] The uncertainty estimation method and the corresponding model training method proposed in the present invention are also applicable to the multi-teacher network scheme. The specific implementation steps are as follows: within one training cycle (epoch), first use the benchmark model to complete the supervised learning process on the labeled samples, then generate pseudo labels for the unlabeled samples through the above-mentioned interactive learning process (the strong and weak enhancement methods used by model 1 and model 2 are different), and then use the prior and a posteriori uncertainty estimation branches to estimate the prior and a posteriori uncertainties of the two models respectively (the implementation method is consistent with the single network structure), and finally exchange the prediction results of the two models for the weakly enhanced samples as the pseudo labels of the prediction results of the other party's strongly enhanced samples, and substitute them into the unsupervised loss function (now the heteroscedastic regression loss) to calculate the loss. In short, the difference in the use of the multi-teacher network structure is only that the pseudo labels predicted by a certain network are used as the regression target of the prediction results of another network, and the use of the uncertainty method proposed in the present invention under this structure is completely consistent with the scheme based on consistency training. After completing the training of the two models, one of the models is usually selected as the model used in the final reasoning process, rather than using both models at the same time, so that the reasoning overhead consistent with the consistency training scheme can be guaranteed. Since the training process of this scheme involves two models (networks), it is also called a "dual network structure".

[0045] Based on the above scheme, another model (called model 3) can be introduced as the student model of model 1 and model 2 (models 1 and 2 are called teacher models), such as Figure 6 As shown. Specifically, the above dual network structure only selects one of model 1 and model 2 as the final inference model, and if the prediction results of the two can be fused, better prediction results can be achieved. Therefore, in a training cycle, the following steps can be added: use model 1 and model 2 to obtain the prediction results of two weakly enhanced samples, fuse the two prediction results, and obtain the final pseudo-label; use model 3 to predict the strongly enhanced samples, and use the fused pseudo-label as the regression target of the prediction result, and add its loss to the unsupervised loss term. Since the fused pseudo-label has a higher accuracy, the inference performance of model 3 will be higher than that of model 1 and model 2. Since the training process of this scheme involves three models (networks), it is also called a "three-network structure". The general three-network structure usually calculates the maximum response coordinate position of two pseudo-labels (heat maps), calculates the midpoint of the two positions, and then renders a new heat map with the midpoint as the center as the fused pseudo-label. In view of this process, the present invention proposes to use the uncertainty estimation results of the two pseudo-labels as an indicator to guide the fusion process. The specific method is: for the pseudo-labels predicted by two or more teacher models, calculate the position P of the maximum response i, called candidate positions; the distance between each candidate position and the final synthesis position P is proportional to the uncertainty of the pseudo-label corresponding to each position, that is, the pseudo-label with higher uncertainty contributes less to the final synthesis result. The calculation process is shown in the formula. c is the number of candidate positions (i.e. the number of teacher models), is the a priori uncertainty m i and the a posteriori uncertainty σ i The above process can be understood as replacing the general arithmetic mean with a weighted average, and using uncertainty as a weight to guide the fusion of pseudo labels. Since candidate labels with lower uncertainty contribute more to it, the pseudo labels obtained by weighted fusion have higher accuracy than the pseudo labels obtained by arithmetic averaging. It should be noted that for Model 3 in the three-network structure, the uncertainty estimation method is not used in the inference stage, and only Model 3 is used as the model for inference, so its inference overhead is also consistent with the single network structure.

[0046]

[0047] Example

[0048] 1. Few-sample 2D human pose estimation system configuration and model description

[0049] The few-sample 2D human pose estimation system proposed in the present invention is a software program based on a modern computer. It requires the computer to be equipped with an NVIDIA graphics card and support the CUDA computing library and the PyTorch deep learning framework. The present invention provides a method for efficiently training a human pose estimation network on a small number of labeled samples, that is, training a benchmark model by organizing images and labeled data in the form of a data set, and completing reasoning and testing on other data. The workflow of the system training phase is as follows: Figure 7 shown.

[0050] After training, the model can be used for reasoning and testing, and the process is consistent with that of a regular deep learning model.

[0051] 2. Implementation process of the few-sample 2D human pose estimation system

[0052] The following describes an example of the complete training and inference process of a few-shot 2D human pose estimation system. The training method used in the example is a semi-supervised human pose estimation scheme based on consistency training.

[0053] The training and test data used in the sample are all from the Microsoft COCO dataset, which contains a total of 118,000 images and provides 150,000 human body instances. Each human body instance is annotated with 17 key points (some may not be visible in the image). Sampling is performed on the basis of all the data in the dataset to obtain the few-sample human posture data required by the present invention. The COCO dataset is sampled to obtain sub-datasets containing 1000, 5000 and 10,000 annotated samples, respectively, called COCO-1K, COCO-5K and COCO-10K. In the above-mentioned sub-datasets, all the samples except the annotated samples are unannotated samples, so the total number of samples in each sub-dataset is consistent.

[0054] The sample uses a server running the Ubuntu 20.04 operating system to complete the training and testing of the model. The server is equipped with an Intel(R) Xeon(R) Gold 6226R processor and an NVIDIA GeForce RTX 3090 graphics card (with 24GB video memory). The deep learning framework used in the experiment is PyTorch 2.3.0, and CUDA12.1 is used for computing acceleration. The training model has a total of 100 epochs. The optimizer used is Adam, and its parameters use the default initialization parameters of PyTorch; the initial learning rate is 0.001, and it decays to 0.0001 and 0.00001 at the 70th and 90th epochs, respectively.

[0055] like Figure 7 As shown in the figure, in one training round, the images of labeled samples and unlabeled samples are respectively subjected to supervised training and unsupervised training. The images in the samples are cropped to 4:3 before training, scaled to 256×192 pixels, and subjected to one data enhancement (weak enhancement). Therefore, in order to obtain strongly enhanced samples later, it is only necessary to perform weak enhancement again on the weakly enhanced samples.

[0056] When training the model in the sample, batch training is used to speed up the training, and the batch size is set to 128. The subsequent discussion of the samples is based on batch data, which is presented in the form of tensors in PyTorch.

[0057] For the labeled samples (size is 128×3×256×192), the prediction results (key point heat map, size is 128×17×64×48) are obtained after passing through the benchmark model, and the supervision loss is calculated with the corresponding labels (size is consistent with the prediction results); Figure 3As shown in the figure, the key point classification results are calculated using the image features of the labeled samples (size is 128×512×8×6) and the prediction results, and the cross entropy loss is calculated with the unit matrix of size 128×17×17 as the classification loss.

[0058] For unlabeled samples (size is 128×3×256×192), the model weights are first frozen, and the weakly enhanced samples are predicted to obtain their prediction results, which are regarded as pseudo labels (size is 128×17×64×48). Then, the model is unfrozen and the strong enhanced samples are used for prediction to obtain the strong enhanced sample prediction results. Figure 3 and Figure 4 The network structure shown calculates the prior and a posteriori uncertainties of the pseudo labels (both of size 128×17). After obtaining the pseudo labels, the prediction results of the strongly enhanced samples and the two uncertainties, the unsupervised loss under heteroscedasticity is calculated according to formula (2).

[0059] After the supervised loss, unsupervised loss and classification loss, they are weighted and fused and optimized as the overall loss of the system until the training round reaches the set value and the training is completed.

Claims

1. A method for estimating a two-dimensional human body posture using a small number of samples, characterized in that: The steps include: Step 1: Collect two-dimensional images containing human bodies and construct a dataset for two-dimensional human posture estimation in the case of few samples, including a small number of labeled samples and a large number of unlabeled samples; Step 2: Build a model for semi-supervised human pose estimation, whose structure includes an encoder and a decoder; Step 3: Construct a priori uncertainty estimation branch and a posteriori uncertainty branch; Step 4: For labeled samples, use a fully supervised approach for training; Step 5: For the unlabeled samples, perform two different data augmentations on the images they contain, including scaling and rotation, and obtain "strongly enhanced" images and "weakly enhanced" images; Step 6: Use the prior uncertainty branch to estimate the prior uncertainty of the "weakly enhanced" image; Step 7: Use the model to predict the results of the "weakly enhanced" image and the "strongly enhanced" image respectively. The former is used as the pseudo label of the latter. At the same time, the posterior uncertainty branch is used to estimate the posterior uncertainty of the "strongly enhanced" image prediction process. The above three are used to calculate the heteroscedastic regression loss. Step 8: If the "multi-teacher" method is used for model training, multiple pseudo labels are weighted fused according to the uncertainty information estimated in steps 6 and 7 to obtain a higher quality fused pseudo label for training the student model; Step 9: Use the trained optimal model to perform inference on the test data to obtain the prediction results of the test data.

2. The method for estimating a two-dimensional human body posture using a small number of samples according to claim 1, characterized in that: In step 1, a labeled sample is a sample including an image containing a human body and corresponding key point labels, and an unlabeled sample is a sample including only an image containing a human body.

3. The method for estimating a two-dimensional human body posture using a small number of samples according to claim 1, characterized in that: In step 4, the fully supervised training is as follows: the mean square error loss is calculated using the sample labels and the prediction results of the model, and the prior uncertainty branch is trained using the manually annotated labels.

4. The method for estimating a two-dimensional human body posture using a small number of samples according to claim 1, characterized in that: In step 5, the two data enhancements have different strengths, that is, the parameters of scaling and rotation have different sizes, and the parameter with a larger value range is defined as "strong enhancement", otherwise it is defined as "weak enhancement".

5. The method for estimating a two-dimensional human body posture using a small number of samples according to claim 4, characterized in that: The strong enhancement includes: random rotation in the range of [-60°, 60°], random scaling in the range of [0.5, 1.5], random key point area masking, and random key point area swapping; the weak enhancement includes: random rotation in the range of [-30°, 30°], and random scaling in the range of [0.75, 1.25].

6. The method for estimating a two-dimensional human body posture using a small number of samples according to claim 1, characterized in that: In step 6, the structure of the prior uncertainty branch estimation includes a maximum pooling layer and a fully connected classification layer, whose input comes from the image features extracted by the encoder and the key point heat map obtained by decoder regression. The image features are upsampled to the same scale as the heat map and then multiplied by the heat map to locally activate the feature map through the highlighted area in the heat map to obtain the local features of the corresponding key points; the local features are classified by training the prior uncertainty estimation branch on the labeled samples to memorize the labeled samples, and the classification scores output by the branch are used as the numerical values ​​of the prior uncertainty on the unlabeled samples.

7. The method for estimating a two-dimensional human body posture using a small number of samples according to claim 6, characterized in that: The classification layer backend uses Sigmoid as the activation function, and the output uncertainty score is between 0 and 1; the loss function used to train the classification layer is as follows: L cls (x)=-y c *logP c -(1-y c )*log(1-P c ) Among them, x is the input sample, y c is the output vector of the classification layer, P c is the key point category truth vector using One-hot encoding.

8. The method for estimating a two-dimensional human body posture using a small number of samples according to claim 7, characterized in that: In step 7, the posterior uncertainty branch estimation first includes a channel attention layer; secondly, includes an uncertainty estimation module consisting of three 3×3 convolutional layers and two fully connected layers; Through the posterior uncertainty branch, the model can output an additional variance value, i.e., the posterior uncertainty value, for each input sample based on the heat map output by the baseline model.

9. The method for estimating a two-dimensional human body posture using a small number of samples according to claim 8, characterized in that: On labeled samples, the posterior uncertainty branch does not participate in training. On unlabeled samples, heteroscedastic loss is used for training, as shown in the following formula: Among them, x u is an unlabeled sample, T h and T e are "strong enhancement" and "weak enhancement" respectively, f represents the forward reasoning process of the benchmark model, σ is the variance, and M is the predicted value of the prior uncertainty; The heteroscedasticity loss replaces the unsupervised loss used in the original consistency training and is added to the overall loss of the model, as shown in the following formula: L=L sup +λ1L het +λ2L cls Among them, L sup is the supervision loss of the labeled data, and λ1 and λ2 are loss balancing factors.

10. The method for estimating a two-dimensional human body posture using a small number of samples according to claim 1, characterized in that: In step 8, for the pseudo labels predicted by two or more teacher models, the position P of the maximum response is calculated. i , called candidate positions; the distance between each candidate position and the final synthesis position P is proportional to the uncertainty of the pseudo-label corresponding to each position, that is, the pseudo-label with higher uncertainty contributes less to the final synthesis result. The calculation process is shown in the following formula: Among them, n c is the number of candidate positions, that is, the number of teacher models, is the a priori uncertainty m i and the a posteriori uncertainty σ i The synthetic result.

Images