Efficient Pose Estimation through Iterative Refinement

Abstract

Some aspects of the present disclosure provide a method that includes processing input data at a feature extraction stage of a machine learning model to generate a feature map, applying an attention map to the feature map to generate an augmented feature map, processing the augmented feature map at a refinement stage of the machine learning model to generate an improved feature map, processing the improved feature map at a first regression stage of the machine learning model to generate multi-dimensional task output data, and processing the improved feature data at an attention stage of the machine learning model to generate an updated attention map.

Description

Claiming priority

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. patent application Ser. No. 17 / 203,607, filed Mar. 16, 2021, the entire contents of which are incorporated herein by reference. [Technical field]

[0002] Aspects of the present disclosure relate to machine learning, and more particularly, to pose estimation using machine learning models. [Background technology]

[0003] Introduction

[0003] Machine learning is generally the process of creating trained models (e.g., artificial neural networks) that represent a generalized fit to a set of training data that is known a priori. Applying the trained model to new data allows for the creation of inferences that can be used to gain insight into the new data.

[0004]

[0004] Pose estimation (e.g., of human body parts) is a key component of most interactive augmented and virtual reality (collectively "extended reality") systems. For hand-based interaction with objects in an extended reality environment, it is important to obtain highly accurate estimates of the positions of the hand joints in 3D world coordinates (as well as image plane coordinates).

[0005]

[0005] Unfortunately, many state-of-the-art pose estimation models achieve high accuracy through large and complex model architectures, which incur significant computational costs. Thus, such models are generally not suitable for relatively lower power computing devices, such as mobile devices, which are ironically the devices where the capabilities can be most useful.

[0006]

[0006] Therefore, there is a need for an improved model architecture for pose estimation. Summary of the Invention

[0007]

[0007] Some aspects provide a method comprising processing input data at a feature extraction stage of a machine learning model to generate a feature map, applying an attention map to the feature map to generate an augmented feature map, processing the augmented feature map at a refinement stage of the machine learning model to generate a refined feature map, processing the refined feature map at a first regression stage of the machine learning model to generate multidimensional task output data, and processing the refined feature data at an attention stage of the machine learning model to generate an updated attention map.

[0008]

[0008] Another aspect provides a method for training a machine learning model comprising: processing training data with a machine learning model, where the machine learning model comprises a feature extraction stage configured to generate a feature map, a refinement stage configured to process an augmented feature map and generate an improved feature map based on the feature map and the attention map, an attention stage configured to process the improved feature map and generate an updated attention map, and a first regression stage configured to process the improved feature map to generate multidimensional task output data; determining a training loss based on a training loss function, where the training loss function comprises an uncertainty loss component and a task loss component; and updating the machine learning model based on the training loss.

[0009]

[0009] Other aspects provide a processing system configured to perform the above-mentioned methods as well as methods described herein; a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the above-mentioned methods as well as methods described herein; a computer program product embodied on a computer-readable storage medium comprising code for performing the above-mentioned methods as well as methods further described herein; and a processing system comprising means for performing the above-mentioned methods as well as methods further described herein.

[0010] The following description and the annexed drawings set forth in detail certain illustrative features of the one or more aspects.

[0011]

[0011] The accompanying drawings illustrate several aspects of the one or more aspects and therefore should not be considered as limiting the scope of the present disclosure. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an example pose estimation model architecture. [Diagram 2]

[0013] FIG. 1 illustrates an example of a repetitive backbone network. [Diagram 3]

[0014] FIG. 1 illustrates an example of a progressive training algorithm. [Figure 4]

[0015] FIG. 3 illustrates an example method for performing pose prediction, such as using the model architecture described with respect to FIGS. 1 and 2. [Diagram 5]

[0016] FIG. 3 illustrates an example method for training a pose prediction model architecture, as described with respect to FIGS. 1 and 2. [Figure 6]

[0017] FIG. 1 illustrates an example processing system that may be configured to perform the methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013]

[0018] For ease of understanding, wherever possible, identical reference numbers have been used to designate identical elements that are common to the figures. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further recitation.

[0014]

[0019] Aspects of the present disclosure provide apparatus, methods, processing systems, and non-transitory computer-readable media for performing pose estimation using machine learning models. In particular, aspects described herein relate to machine learning model architectures for efficiently estimating body element (e.g., hand) poses in three-dimensional space based on two-dimensional input images.

[0015]

[0020] The pose estimation model architectures described herein may generally include a recurrent backbone network, a pose predictor, an uncertainty estimator, and a decision gating function. Beneficially, these model architectures continuously refine predictions over recursive iterations of the refinement module of the recurrent backbone network and the attention map generator until the decision gating function terminates the loop.

[0016]

[0021] The pose estimation model architecture described herein may further use a different batch normalized layer for each iteration through the refinement module, which beneficially stabilizes the gradients during training and mitigates statistical perturbations in the batch normalized layer caused by the attention map generator.

[0017]

[0022] The gating functions of the pose estimation model architectures described herein generally take into account a confidence measure of the network's own estimates (e.g., uncertainty) to determine discrete gating via an objective function. For example, the gating function may estimate the uncertainty of model inferences, such as multi-dimensional (e.g., 2D and 3D) keypoint predictions, to determine when to terminate the recurrent backbone network.

[0018]

[0023] Finally, aspects described herein relate to progressive training techniques for pose estimation model architectures that improve inference efficiency by terminating the recurrent backbone network after fewer iterations on more samples.

[0019]

[0024] Thus, the pose estimation model architecture described herein produces inference that is efficient in terms of computational cost and memory usage, which beneficially enables such model architectures to be deployed across a wider range of devices, such as mobile devices, always-on devices, Internet of Things devices, edge processing devices, etc. Thus, the iterative pose estimation model architecture described herein provides a technical solution to the shortcomings of conventional approaches, as described above. Exemplary Pose Estimation Model Architecture

[0025] FIG. 1 illustrates an example pose estimation model architecture 100.

[0020]

[0026] In general, model architecture 100 illustrates a modular network in which model parameters can be iteratively exploited for more accurate predictions through adaptive spatial scoping. In particular, the adaptive nature of model architecture 100 is provided by a gated looping mechanism that enables the model to use only the amount of complexity necessary to generate reliable pose predictions.

[0021]

[0027] As shown, the model architecture 100 consists of a recurrent backbone network 104, which itself comprises a feature extractor 106, a refinement module 108, and an attention map generator 110. The model architecture 100 further includes a pose predictor 112, which in this example includes a hand model 118, but in other examples may include other body element models, including a full human body skeletal model including the bones and joints of the human body. The model architecture further includes an uncertainty estimator 124 and a decision gating function 128, which are configured to control looping within the recurrent backbone network 106, thus enabling the adaptive scoping aspect of the model architecture 100.

[0022]

[0028] In the illustrated example, a monocular image of a hand is provided as input data 102 to the feature extractor 106, which generates features based on the input data, which are then provided to the refinement module 108. The output of the refinement module 108 is then provided to (1) the pose predictor 112, (2) the uncertainty estimator 124, and (3) the attention map generator 110. Note that in the illustrated example, the output of the refinement module 108 is provided to the attention map generator 110 based on the output of a decision of the gating function 128, which may be compared to a predefined uncertainty threshold. In some cases, the predefined uncertainty threshold is a hyperparameter to be tuned, where different values ​​for this threshold provide different accuracy vs. efficiency tradeoffs. Thus, the uncertainty estimator 124 and the gating function 128 generally control the looping of the refinement module 108 and the attention map generator 110 until a condition is met, and then the output 120 and / or 122 are provided as the output of the model architecture 100. The condition may be, for example, that a gating threshold is met or that a loop limit is reached.

[0023]

[0029] Each of the above-mentioned components of model architecture 100 will now be described in more detail. Iterative Backbone Network

[0030] In the illustrated example, the iterative backbone network 104 is a residual network configured to perform recursive inference, which generally allows the use of higher-level features for the refinement of lower-level features to exploit the capacity of the network parameters. In other words, the iterative backbone network 104 has a low default complexity, which can be increased with each recursive use (e.g., in each loop). This beneficially allows smaller model sizes (e.g., smaller neural networks) to have similar predictive accuracy as more complex models, which reduces memory requirements and allows implementation on a wider variety of devices.

[0024]

[0031] The feature extractor 106 is generally configured to encode low-level image features of the input data 102. In some cases, the feature extractor 106 may be implemented by a neural network model (or a portion thereof), such as a convolutional neural network model. An example of the feature extraction stage of a neural network model is shown and described in more detail with respect to FIG.

[0025]

[0032] Because the feature extractor 106, like the refinement module 108 and the attention map generator 110, is not operated recursively, its feature encoding may be configured to incorporate heavy downsampling of the feature maps, so that their recursive calculation is less costly. For example, the feature extractor 106 may be configured to reduce the spatial dimensions of the image input (e.g., 102) from, e.g., 224×224 pixels to 28×28 pixels, to make the iterations through the refinement module 108 and the attention map generator 110 more efficient. This is just one example, and in general, the input size and downsizing ratio may be configured as needed.

[0026]

[0033] During inference, the output from the feature extractor 106 may be stored in a memory (not shown) so that it is not generated multiple times for recursive loops of other aspects of the iterative backbone network 104. This further improves the computational efficiency of the model architecture 100.

[0027]

[0034] The refinement module 108 is generally configured to be "looped" recursively, so that at every iteration, the attention-extended feature map generated by the attention map generator 110 may be used to refine the prediction of the previous loop. In particular, this attention extension may change the distribution of the input feature map at every loop iteration. Thus, as shown in FIG. 2, a separate batch normalization layer may be used at every iteration to account for statistical shifts in the attention-extended input feature map.

[0028]

[0035] In particular, Figure 2 illustrates an example of a recurrent backbone network 200 that includes a feature extractor 202, a refinement module 204, and an attention map generator 206. l ), where l is a configurable number of loops allowed in the iterative backbone network 200. In other words, for computational efficiency, it may be desirable to limit the loops to a configurable number, such as four loops, to prevent excessive looping and unnecessary latency.

[0029]

[0036] Returning to Figure 1, the attention map generator 110 may be implemented as an upscale decoder that outputs as many attention maps as the number of feature maps for element-wise multiplication, in some examples, 2. In some cases, the attention maps generated by the attention map generator 110 may generally include values ​​in the range of [0, 1], which may then be applied (element-wise) to the feature maps output by the refinement module 108 to enable an "attention" mechanism.

[0030]

[0037] In some aspects, to effectively upscale the output vector of the refinement module 108 without significantly increasing the computational cost, the decoder may be configured for a pixel shuffle layer, which reduces the computational cost by making the channel features into spatial features. In general, upscaling may be done in many ways, including bilinear interpolation, deconvolution, and pixel shuffle operations, to name a few. Pixel shuffling beneficially preserves more information than other alternatives. Pose Predictor

[0038] Generally, the pose predictor 112 is configured to generate 3D and / or 2D pose predictions (or estimates) based on the input feature maps provided by the recurrent backbone network 104.

[0031]

[0039] As in the illustrated example, the pose predictor 112 may include one or more fully-connected (FC) layers (e.g., layer 114) of a neural network model. In such an example, the fully-connected layer 114 of the pose predictor 112 generates a multidimensional output 116 that is used for input to a hand model 118. The fully-connected layer 114 may, in some cases, be referred to as a regression stage because the outputs are regression values ​​instead of categorical values.

[0032]

[0040] In this example, the multi-dimensional output 116 includes multiple parameters including a rotation matrix R, a translation t, a scaling factor s, a pose θ, and a shape β. In general, the pose θ and shape β parameters may be used to generate a 3D pose prediction using the hand model 118, which in this example may then be projected onto a 2D plane using the R, t, and s parameters. In this example, R, t, and s may be referred to as extrinsic parameters of the camera capturing the input data.

[0033]

[0041] In one example, the hand model 118 is θ The MANO hand mesh deformation model is configured to take input low-dimensional vectors as parameters for θ and β to control the 3D hand mesh output according to (J(β)), where J is a function for predicting body element positions, e.g., the locations of the 21 joints in the hand, in this example J(β) is a function for predicting the 3D hand mesh output according to R θ It should be noted that in other examples, the hand model 118 may be replaced with an alternative body element model, or even a complete skeletal model.

[0034]

[0042] In one example, a cropped hand image is provided as a data input x (102), and the output F(x) of the feature extractor 108 is fed to a refinement module R (108) for each recursive iteration (or loop) l ∈ {1, 2, ..., l max}, the attention map M generated by the attention map generator 110 l In some examples, the cropped hand image may be generated by a hand detector model, an image segmentation model, or the like.

[0035]

[0043] The pose predictor is R(F(x)) and R(F(x),Ml ) as input, the rotation matrix R∈SO(3), the transformation

[0036]

number

[0037] and the scaling factor

[0038]

number

[0039] We predict,x ...

[0040]

number

[0041] Here, Ψ pose (·) represents the two fully connected layer operations. The 3D locations of the joints J(θ,β) are defined by the hand model 118, and the detected keypoints are then reprojected to the 2D image plane using a weak-perspective camera model parameterized by the R, t, and s parameters of Eq. 1:

[0042]

number

[0043] According to the estimation of the camera parameters, we estimate the 2D location (J 2D ), where Π represents the orthographic projection. With the reprojected 2D joint location estimation, the network is able to learn the 3D joint locations even when only 2D labels are present in the training set.

[0044]

[0044] In some examples, the criterion for training the pose predictor 112 is the L1 and L2 losses between the predictions and the ground-truths.

[0045]

number

[0046]

[0045] In general, either the L1 norm or the L2 norm can be used for the loss function. Since 2D prediction is a function of more variables (compared to 3D prediction with projected variables), the L1 norm loss may provide a more robust prediction for 2D estimation. Dynamic exiting of iterative backbone networks

[0046] The model architecture 100 may implement multiple exiting policies for the iterative backbone network 104. A first exiting policy option is heuristic based on the uncertainty estimate 130 (e.g., the area of ​​the total variance). For example, the loop may terminate when the variance is less than a threshold. In such an example, the gating function 128 is not required since the uncertainty estimate 130 is used directly for gating. A second exiting policy option, in one example, utilizes a gating function 128 with a single full connection. Beneficially, the gating function 128 is trainable, as further described below with respect to Equation 12.

[0047] 1, terminating the loop with the iterative backbone network 104 is performed adaptively for each input data sample using a gating function 128. Thus, the model architecture 100 may be referred to as an adaptive iterative scope network.

[0048] In general, to decide to proceed to the next loop of the recursive inference of the iterative backbone network 104, the model should be aware of how certain it is about its current prediction. To this end, the uncertainty estimator 124 estimates the variance for the 2D / 3D joint location (e.g., keypoint) predictions (e.g., 120 and 122) by estimating a probability distribution instead of the joint locations alone. In one example, to simplify the problem, the joint coordinates are assumed to be linearized as a single variate Gaussian

[0049]

number

[0050] In order to use, it is assumed that the joint locations are independent, where W refers to the trainable weights. The coordinates for the joint locations are denoted as J,

[0051]

number

[0052] represents the estimated joint location. The closer the standard deviation σ is to 0, the more reliable the model is in its estimation. Having the target joint coordinates as a Gaussian distribution with σ → 0, it can be expressed as J gt denotes the ground truth joint positions, and

[0053]

number

[0054] and The model is P W (J) and P D Based on the Kullback-Leibler (KL) divergence between (J) and the uncertainty loss component (L var ) to a minimum,

[0055]

number

[0056] The objective is to learn the reliability estimate according to Here, α=log(σ 2 ), which is what the model architecture 100 expects as the uncertainty estimate 130. As an output, the uncertainty estimator 124

[0057]

number

[0058] Predict the variance of its own 2D / 3D joint prediction according to

[0059]

number

[0060] represents the fully connected layer before variance estimation.

[0061]

[0049] In 3D joint prediction, the loss of variance estimation for 3D joint localization is

[0062]

number

[0063] It can be set as:

[0064]

[0050] The 2D joint locations are regressed with a smooth L1 loss, so the variance loss for 2D joint localization is

[0065]

number

[0066] It can be defined according to:

[0067]

[0051] L calculated by Equation 3 2D , L 3D With the sum of both losses for the 2D / 3D joint localization variances,

[0068]

number

[0069] is given.

[0070]

[0052] Thus, the uncertainty estimator 124 may learn the variance for its joint location prediction as an l (letter L) dimensional vector.

[0071] The above variances can be used directly to determine termination (e.g., instead of using the gating function 128). For example, the threshold τ var may be set for the average variance for the current joint estimate. If the average variance is greater than a predefined threshold, another loop of the refinement module 108 and attention map generator 110 is performed. Gating Functions In addition to a heuristic uncertainty threshold in each loop that may be used to terminate the loop, the gating function 128 allows the model architecture 100 to learn optimal termination decisions. In other words, the gating function 128 provides a learnable gating mechanism instead of a heuristic method based on the uncertainty estimates 130. In some examples, the gating function 128 may be implemented as a neural network model, which may be part of a larger model, such as the model architecture 100.

[0072] For example, given an input x and the lth (letter L) loop M lFor the attention map generated in , the gating function 128 outputs a stochastic categorial decision of termination after being trained based on the reward signal from Equation 11:

[0073]

number

[0074] As shown in FIG. 1 and in Equation 7, f(126) is an intermediate feature output (e.g., a vector) that is used as an input to a gating function 128.

[0075] For this purpose, the gating function G(A l |f l ) is the uncertainty feature vector f in loop l (the English letter l). l Based on the two possible categories of actions A l , terminate, or not, which is trained with on-policy vanilla policy gradient (a reinforcement learning algorithm used to train agents based on reward signals). With more iterations the loss decreases, but with a penalty for increasing the computational cost for each additional loop. The difference between the loss and the computational cost is the reward signal (r l )

[0076]

number

[0077] Here, λ denotes a scaling constant, and the computational cost C is, in one example, giga-floating point operations per second (GFLOPs). With such a reward, the policy ultimately tries to minimize the error in the pose estimation while also minimizing the required computational cost. The learning gradient of the gating policy function is

[0078]

number

[0079] where w represents a learnable parameter of the gating function 128.

[0080]

[0057] Note that the gating function 128 may be trained after training the rest of the model architecture 100, including the uncertainty estimator 124. Thus, during training of the gating function 128, other parts of the model architecture 100 are decoupled.

[0081] In one example, the gating function 128 is trained using a reinforcement algorithm such that during inference, the action of the gating function 128 (e.g., terminate or not terminate) is sampled from a softmax probability distribution. Thus, the output of the gating function 128 is probabilistic and not deterministic in such cases.

[0082] For the learned gating function 128, the temperature parameter τ gate can be utilized to allow further control of the harshness of the decision even after training. The gating function 128 is, in this example, a categorical softmax function so that the tradeoff between computational cost and predictive accuracy can be further controlled during testing. In general, τ gate Higher values ​​of σ result in a softer distribution of the softmax function. Progressive Training

[0060] The model architecture 100 may be initially trained by looping through the partial layers and computing a loss based on the inference results in each loop. max For a predefined maximum number of

[0083]

number

[0084] It can be trained by minimizing the sum of losses from all loops according to

[0085]

number

[0086] is the hyperparameter γ 2D is the 2D prediction loss weighted by

[0087]

number

[0088] is the hyperparameter γ 3D is the 3D prediction loss weighted by

[0089]

number

[0090] is the hyperparameter γ varwhere is the uncertainty loss weighted by . In particular, model architecture 100 can be trained in either an end-to-end or incremental manner. In an end-to-end training protocol, a single training is performed for a maximum number of loops, whereas in an incremental training protocol, model architecture 100 is trained multiple times during incremental increments of loop tolerance.

[0091] FIG. 3 illustrates an example of a progressive training algorithm 300.

[0092] Generally, during incremental training by algorithm 300, model architecture 100 max It is trained once and loops prog ∈{0,1,2,...,l max}. Thus, the model architecture 100 trains on each of the maximum number of cases for l prog If l = 0, it is first trained without loops. This initial training phase of the incremental training protocol requires one batch normalization layer at the beginning of the refinement module 108 for a single inference path. Also, l prog If >0, model architecture 100 is prog +1 batch normalization layers. The model architecture 100 then removes the extra batch normalization layers and prog If it is -1, it will be reinitialized with the trained parameters.

[0093] The model architecture 100 includes: prog When trained with l = 0, the feature extractor 106 learns a meaningful feature coding layer, prog When it is >0, it is cut off from further learning and the learning rate is

[0094]

number

[0095] The attention map generator 110 is trained with the original base learning rate. For example, training may start with a learning rate of 0.1 for the 0th loop. Then, in the first loop, the feature extractor is decoupled from training and the refinement module 108

[0096]

number

[0097] The training rate is, at each iteration,

[0098]

number

[0099] The attention map generator is trained with the original learning rate of 0.1.

[0100]

[0064] Beneficially, the incremental training method 300 has been empirically shown to ensure maximization of the capacity of the network in each loop case, which results in a higher rate of termination in the early loop case and a lower average computation cost for validation samples. Exemplary Methods for Pose Prediction FIG. 4 shows an example method 400 for performing pose prediction, such as using the model architecture described with respect to FIGS.

[0101]

[0066] The method 400 begins at step 402 with processing input data in a feature extraction stage of a machine learning model to generate a feature map. For example, the feature extraction stage may comprise a feature extractor, such as 106 in Figure 1 and 202 in Figure 2.

[0102]

[0067] Although not shown, the feature maps may be stored in a memory, such as a local memory, for reuse during iterative processing of the machine learning model, which saves processing compared to reprocessing the input data at the feature extraction stage during each iteration.

[0103]

[0068] The method 400 then proceeds to step 404, which involves applying the attention map to the feature map to generate an augmented feature map. In some cases, the augmented feature map may be referred to as an attention augmented feature map. For example, the attention map generator 110 of FIG. 1 or the attention map generator 206 of FIG. 2 may provide the attention map.

[0104] As explained above, step 404 will generally occur after the first pass through the model, where there are no extensions, such as loop number l>0.

[0105]

[0070] The method 400 then proceeds to step 406, which involves processing the augmented feature map in a refinement stage of the machine learning model to generate an refined feature map. For example, the refinement stage may comprise a refinement module, such as 108 in FIG. 1 or 204 in FIG. 2.

[0106]

[0071] In some aspects, during the refinement phase of the machine learning model, processing the augmented feature map comprises applying a batch normalization layer to the augmented feature map, where the batch normalization layer is selected from a plurality of batch normalization layers based on a loop count.

[0107]

[0072] The method 400 then proceeds to step 408, which involves processing the refined feature map in a first regression stage of a machine learning model to generate multi-dimensional task output data (e.g., 116 in FIG. 1). For example, the multi-dimensional task output data may include a rotation matrix R, a translation t, a scaling factor s, a pose θ, and / or a shape β, as described above. In one example, the regression stage comprises one or more fully connected layers of a pose predictor, such as the fully connected layer 114 of the pose predictor 112 in FIG. 1.

[0108]

[0073] The method 400 then proceeds to step 410, which involves processing the refined feature data in the attention stage of the machine learning model to generate an updated attention map.

[0109]

[0074] Note that the refined feature data may generally be the same as that used in step 408. If parallel processing of the various stages is implemented, the refined feature data may be stored in multiple copies to be accessed and processed by the various stages at once.

[0110]

[0075] The method 400 then proceeds to step 412, which involves processing the multi-dimensional task output data with a 3D position generator to generate a 3D position estimate, such as 3D prediction 120 in Figure 1. For example, the 3D position generator can be a body element pose model, such as hand model 118 in Figure 1, and the 3D position estimate comprises one or more body element joint positions in 3D space, such as keypoint estimates.

[0111]

[0076] The method 400 then proceeds to step 414 which involves processing the three-dimensional position estimate together with the multi-dimensional task output data to generate a two-dimensional position estimate.

[0112]

[0077] For example, a rotation matrix R, a translation t, and a scaling factor s may be used to project the pose θ and shape β from 3D to 2D, where R, t, and s are the extrinsic parameters of the camera capturing the input data used in step 402.

[0113] In some aspects, the two-dimensional position estimate comprises one or more body element joint positions in two-dimensional space.

[0114]

[0079] Although not shown in Figure 4, the method 400 may further include processing the refined feature map in a second regression stage of a machine learning model to generate a task uncertainty estimate. For example, the refined feature map may be processed by an uncertainty estimator such as 124 in Figure 1, and in particular by the fully connected layer 125 of the uncertainty estimator 124.

[0115]

[0080] The method 400 may further include determining an uncertainty associated with the three-dimensional position estimate based on the task uncertainty estimate, and determining an uncertainty associated with the two-dimensional position estimate based on the task uncertainty estimate. For example, a fully connected layer (e.g., another regression stage) of the uncertainty estimator 124 of FIG. 1 may generate uncertainty estimates (e.g., variances 130) associated with the two-dimensional and three-dimensional position estimates.

[0116]

[0081] The method 400 may further include processing the task uncertainty estimates in a loop decision stage of the machine learning model to generate a loop decision value, and determining whether to terminate processing of the machine learning model based on the loop decision value. In one example, the loop decision stage comprises the gating function 128 of FIG. 1, and the loop decision value is a softmax value. As described above, in some cases, the loop decision value may be sampled from a softmax probability distribution.

[0117]

[0082] The method 400 may further include displaying the two-dimensional position estimate on a display screen of an extended reality device. In some cases, the extended reality device may be a mobile device such as a smartphone, or a smart wearable device such as glasses or a headset, or other device.

[0118]

[0083] It should be noted that method 400 is one exemplary method and others are possible. In particular, other examples may include fewer, additional, and / or alternative steps as compared to method 400 in accordance with various aspects described herein. Exemplary Method for Training a Pose Model

[0084] FIG. 5 illustrates an example method 500 for training a pose prediction model architecture, such as described with respect to FIGS. 1 and 2.

[0119]

[0085] The method 500 begins at a step in 502 with processing training data with a machine learning model. In some cases, the machine learning model includes a feature extraction stage (e.g., feature extractor 106 in FIG. 1 or feature extractor 202 in FIG. 2) configured to generate a feature map, a refinement stage (e.g., refinement module 108 in FIG. 1 or refinement module 204 in FIG. 2) configured to process an augmented feature map and generate an improved feature map based on the feature map and the attention map, an attention stage (e.g., attention map generator 110 in FIG. 1, attention map generator 206 in FIG. 2) configured to generate an updated attention map based on the improved feature map, and a first regression stage (e.g., fully connected layer 114 of pose predictor 112 in FIG. 1) configured to process the improved feature map to generate multidimensional task output data.

[0120]

[0086] In some cases, the refinement stage of the machine learning model comprises multiple batch normalization layers, where each respective batch normalization layer is associated with a loop count.

[0121]

[0087] The method 500 then proceeds to step 504, which involves determining a training loss based on a training loss function (e.g., L total ) is the uncertainty loss component (e.g., L var ), and the task loss component (e.g.,

[0122]

number

[0123] and

[0124]

number

[0125] ) and.

[0126]

[0088] The method 500 then proceeds to step 506, which involves updating the machine learning model based on the training loss, as illustrated with respect to FIG. 3 .

[0127]

[0089] In some aspects, the machine learning model further comprises a 3D position generator (e.g., hand model 118 in FIG. 1 ) configured to process the multi-dimensional task output (e.g., parameters 116 in FIG. 1 ) and generate a 3D position estimate (e.g., 3D prediction 120 in FIG. 1 ), where the task loss component is a 3D position estimate error component (e.g.,

[0128]

number

[0129] ) is provided.

[0130] In some aspects, the machine learning model further comprises a two-dimensional position generator configured to process the three-dimensional position estimate with the multi-dimensional task output (e.g., 116 in FIG. 1 ) and generate a two-dimensional position estimate (e.g., 122 in FIG. 1 ), where the task loss component is a two-dimensional position estimate error component (e.g.,

[0131]

number

[0132] ) is provided.

[0133]

[0091] In some aspects, the 3D position generator comprises a body element pose model, such as hand model 118 in Figure 1. In some aspects, the 3D position estimate comprises one or more body element joint positions in 3D space, the 2D position estimate comprises one or more body element joint positions in 2D space, and the multi-dimensional task output data comprises a rotation matrix R, a transformation t, and a scaling factor s that can be used to project the pose θ and shape β.

[0134]

[0092] In some aspects, the machine learning model further comprises a second regression stage (e.g., fully connected layer 125 of uncertainty estimator 124 in FIG. 1) configured to generate a task uncertainty estimate, where the task uncertainty estimate comprises a three-dimensional position estimate variance component of the uncertainty loss component, such as uncertainty estimate 130 in FIG. 1, and a two-dimensional position estimate variance component of the uncertainty loss component.

[0135]

[0093] In some aspects, the machine learning model further comprises a loop decision stage (e.g., gating function 128 of FIG. 1) configured to process the task uncertainty estimates and determine a loop decision value.

[0136]

[0094] It should be noted that method 500 is one exemplary method and others are possible. In particular, other examples may include fewer, additional, and / or alternative steps as compared to method 500 in accordance with various aspects described herein. Exemplary Processing System FIG. 6 illustrates an example processing system 600 that may be configured to perform the methods described herein, such as with respect to FIGS.

[0137]

[0096] The processing system 600 includes a central processing unit (CPU) 602, which may be a multi-core CPU in some examples. Instructions executed in the CPU 602 may be loaded from a program memory associated with the CPU 602 or may be loaded from a memory 624, for example.

[0138]

[0097] The processing system 600 also includes additional processing components adapted to specific functions, such as a graphics processing unit (GPU) 604, a digital signal processor (DSP) 606, a neural processing unit (NPU) 608, a multimedia processing unit 610, and wireless connectivity components 612.

[0139]

[0098] In some aspects, one or more of the CPU 602, GPU 604, DSP 606, and NPU 608 may be configured to perform the methods described herein with respect to Figures 3-5.

[0140]

[0099] An NPU, such as 608, is generally a specialized circuit configured to implement all the necessary control and arithmetic logic to run machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), kernel methods, etc. An NPU may alternatively be referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), or a vision processing unit (VPU).

[0141]

[0100] NPUs, such as 608, can be configured to accelerate the performance of common machine learning tasks, such as image classification, machine translation, object detection, and a variety of other tasks. In some examples, multiple NPUs can be instantiated on a single chip, such as a system-on-chip (SoC), while in other examples, they can be part of a dedicated machine learning accelerator device.

[0142]

[0101] NPUs can be optimized for training or inference, or in some cases configured to balance performance between both. For NPUs capable of performing both training and inference, the two tasks can still be performed largely independently.

[0143]

[0102] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly compute-intensive operation that involves inputting an existing dataset (often labeled or tagged), iterating over the dataset, and then adjusting model parameters, such as weights and biases, to improve model performance. Generally, optimizing based on erroneous predictions involves backpropagating through layers of the model and determining gradients to reduce prediction errors.

[0144]

[0103] NPUs designed to accelerate inference are generally configured to operate on complete models. Thus, such NPUs can be configured to input new data and rapidly process it through already-trained models to generate model outputs (e.g., inferences).

[0145]

[0104] In some aspects, the NPU 608 may be implemented as part of one or more of the CPU 602, GPU 604, and / or DSP 606.

[0146] In some aspects, the wireless connectivity component 612 may include subcomponents for, for example, third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity processing component 612 is further connected to one or more antennas 614.

[0147]

[0106] The processing system 600 may also include one or more sensor processing units 616 associated with any type of sensor, one or more image signal processors (ISPs) 618 associated with any type of image sensor, and / or a navigation processor 620, which may include satellite-based positioning system components (e.g., GPS or GLONASS) as well as inertial positioning system components.

[0148]

[0107] The processing system 600 may also include one or more input and / or output devices 622, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, a microphone, etc.

[0149]

[0108] In some examples, one or more of the processors of processing system 600 may be based on the ARM or RISC-V instruction set.

[0150]

[0109] Processing system 600 also includes memory 624, which represents one or more static and / or dynamic memories, such as dynamic random access memories, flash-based static memories, etc. In this example, memory 624 includes computer-executable components that may be executed by one or more of the above-mentioned components of processing system 600.

[0151]

[0110] In particular, in this example, memory 624 includes a feature extraction component 624A, a refinement component 624B, an attention component 624C, a pose prediction component 624D, an uncertainty estimation component 624E, a gating component 626F, a training component 624G, an inference component 624H, and model parameters 624I (e.g., weights, biases, and other machine learning model parameters). One or more of the illustrated components, as well as other components not illustrated, may be configured to perform various aspects of the methods described herein.

[0152]

[0111] In one example, the feature extraction component 624A may be implemented by the feature extractor 106 in Figure 1 and the feature extractor 202 in Figure 2, the refinement component 624B may be implemented by the refinement module 108 in Figure 1 and the refinement module 204 in Figure 2, the attention component 624C may be implemented by the attention map generator 110 in Figure 1 and the attention map generator 206 in Figure 2, the pose prediction component 624D may be implemented by the pose predictor 112 in Figure 1, the uncertainty estimation component 624E may be implemented by the under-training estimator 124 in Figure 1, and the gating component 626F may be implemented by the gating function 128 in Figure 1. Furthermore, the training component 624G may implement the algorithm 300 in some examples.

[0153]

[0112] Generally, the processing system 600 and / or its components may be configured to perform the methods described herein.

[0154]

[0113] Notably, in other aspects, aspects of processing system 600 may be omitted, such as when processing system 600 is a server computer, etc. For example, multimedia components 610, wireless connectivity 612, sensors 616, ISP 618, and / or navigation components 620 may be omitted in other aspects. Additionally, aspects of processing system 600 may be distributed.

[0155]

[0114] It should be noted that FIG. 6 is only one example, and that in other examples, alternative processing systems having fewer, additional, and / or alternative components may be used. Example clauses

[0115] Implementation examples are described in the following numbered clauses.

[0156]

[0116] Clause 1: A method comprising: processing input data at a feature extraction stage of a machine learning model to generate a feature map; applying an attention map to the feature map to generate an augmented feature map; processing the augmented feature map at a refinement stage of the machine learning model to generate an improved feature map; processing the improved feature map at a first regression stage of the machine learning model to generate multidimensional task output data; and processing the improved feature data at an attention stage of the machine learning model to generate an updated attention map.

[0157]

[0117] Clause 2: The method of clause 1, further comprising processing the multi-dimensional task output data with a 3D position generator to generate a 3D position estimate.

[0158]

[0118] Clause 3: The method of clause 2, further comprising processing the three-dimensional position estimate together with multi-dimensional task output data to generate a two-dimensional position estimate.

[0159]

[0119] Clause 4: The method of clause 3, wherein the 3D position generator comprises a body element pose model.

[0160]

[0120] Clause 5: The method described in Clause 4, wherein the three-dimensional position estimate comprises one or more body element joint positions in three-dimensional space, the two-dimensional position estimate comprises one or more body element joint positions in two-dimensional space, and the multidimensional task output data comprises one or more of a rotation estimate, a transformation estimate, or a scaling estimate.

[0161]

[0121] Clause 6: The method of clause 5, wherein the body element pose model comprises a hand model.

[0162]

[0122] Clause 7: A method according to any one of clauses 1 to 6, further comprising processing the improved feature map in a second regression stage of the machine learning model to generate a task uncertainty estimate.

[0163]

[0123] Clause 8: The method described in clause 7, further comprising determining an uncertainty associated with the three-dimensional position estimate based on the task uncertainty estimate, and determining an uncertainty associated with the two-dimensional position estimate based on the task uncertainty estimate.

[0164]

[0124] Clause 9: A method described in any one of clauses 7 to 8, further comprising processing the task uncertainty estimates at a loop decision stage of the machine learning model to generate a loop decision value, and determining whether to terminate processing of the machine learning model based on the loop decision value.

[0165]

[0125] Clause 10: A method described in any one of clauses 1 to 9, wherein during the refinement stage of the machine learning model, processing the augmented feature map comprises applying a batch normalization layer to the augmented feature map, and the batch normalization layer is selected from a plurality of batch normalization layers based on the number of loops.

[0166]

[0126] Clause 11: The method of clause 5, further comprising displaying the two-dimensional position estimate on a display screen of the extended reality device.

[0167]

[0127] Clause 12: The method of clause 9, further comprising determining to terminate processing of the machine learning model based on a loop decision value.

[0168]

[0128] Clause 13: The method described in clause 9, further comprising deciding not to terminate processing of the machine learning model based on a loop judgment value, and deciding to terminate processing of the machine learning model based on the number of loops.

[0169]

[0129] Clause 14: The method of clause 1, further comprising determining a training loss based on a training loss function, wherein the training loss function comprises an uncertainty loss component and a task loss component, and updating the machine learning model based on the training loss.

[0170]

[0130] Clause 15: A method for training a machine learning model comprising: processing training data with a machine learning model, wherein the machine learning model comprises a feature extraction stage configured to generate a feature map, a refinement stage configured to process an augmented feature map and generate an improved feature map based on the feature map and the attention map, an attention stage configured to process the improved feature map and generate an updated attention map, and a first regression stage configured to process the improved feature map to generate multidimensional task output data; determining a training loss based on a training loss function, wherein the training loss function comprises an uncertainty loss component and a task loss component, and updating the machine learning model based on the training loss.

[0171]

[0131] Clause 16: The method of clause 15, further comprising a three-dimensional position generator configured such that the machine learning model processes the multi-dimensional task output and generates a three-dimensional position estimate, and the task loss component comprises a three-dimensional position estimate error component.

[0172]

[0132] Clause 17: The method of clause 15, further comprising a two-dimensional position generator configured such that the machine learning model processes the three-dimensional position estimate together with the multidimensional task output and generates a two-dimensional position estimate, and the task loss component comprises a two-dimensional position estimate error component.

[0173]

[0133] Clause 18: The method of clause 17, wherein the 3D position generator comprises a body element pose model.

[0174]

[0134] Clause 19: The method described in Clause 18, wherein the three-dimensional position estimate comprises one or more body element joint positions in three-dimensional space, the two-dimensional position estimate comprises one or more body element joint positions in two-dimensional space, and the multidimensional task output data comprises one or more of a rotation estimate, a transformation estimate, or a scaling estimate.

[0175]

[0135] Clause 20: A method according to any one of clauses 18 to 19, wherein the body element pose model comprises a hand model.

[0176]

[0136] Clause 21: A method according to any one of clauses 15 to 20, wherein the machine learning model further comprises a second regression stage configured to generate a task uncertainty estimate, wherein the task uncertainty estimate comprises a three-dimensional position estimate variance component of the uncertainty loss component and a two-dimensional position estimate variance component of the uncertainty loss component.

[0177]

[0137] Clause 22: The method described in clause 21, further comprising a loop decision stage in which the machine learning model is configured to process the task uncertainty estimates and determine a loop decision value.

[0178]

[0138] Clause 23: A method described in any one of clauses 15 to 22, wherein the refinement stage of the machine learning model comprises a batch normalization layer parameterized based on the augmented feature map.

[0179]

[0139] Clause 24: A processing system comprising a memory having computer-executable instructions and one or more processors, the one or more processors configured to execute the computer-executable instructions and to cause the processing system to perform a method described in any one of clauses 1 to 23.

[0180]

[0140] Clause 25: A processing system comprising means for carrying out the method according to any one of clauses 1 to 23.

[0181]

[0141] Clause 26: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method according to any one of clauses 1 to 23.

[0182]

[0142] Clause 27: A computer program product embodied on a computer-readable storage medium comprising code for performing the method according to any one of clauses 1 to 23. Additional Considerations

[0143] The above description is provided to enable those skilled in the art to practice the various embodiments described herein. The examples described herein are not intended to limit the scope, applicability, or aspects described in the claims. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, changes may be made in the function and configuration of the elements described without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components, as appropriate. For example, the methods described may be performed in an order different from that described, and various steps may be added, omitted, or combined. Also, features described with respect to some examples may be combined in some other examples. For example, an apparatus may be implemented, or a method may be practiced, using any number of aspects described herein. Furthermore, the scope of the disclosure is intended to cover such apparatus or methods implemented using other structures, functions, or structures and functions in addition to or other than the various aspects of the disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be implemented by one or more elements of a claim.

[0183]

[0144] As used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0184]

[0145] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to encompass a, b, c, ab, ac, bc, and abc, as well as any combination with multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other permutation of a, b, and c).

[0185]

[0146] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database or another data structure), ascertaining, and the like. Also, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Also, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0186]

[0147] The methods disclosed herein comprise one or more steps or actions for achieving the method. The steps and / or actions of the methods may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be changed without departing from the scope of the claims. Furthermore, the various operations of the methods described above may be performed by any suitable means capable of performing the corresponding functions. Those means may include various (one or more) hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs), or processors. Generally, where there are operations illustrated in the figures, those operations may have corresponding counterpart means-plus-function components with similar numbering.

[0187]

[0148] The following claims are not limited to the embodiments set forth herein, but are to be accorded the full scope consistent with the language of the claims. In the claims, reference to an element in the singular does not mean "one and only," unless expressly stated as such, but means "one or more." Unless otherwise expressly stated, the term "several" refers to one or more. No claim element is to be construed under the provisions of 35 U.S.C. 112(f) unless the element is expressly recited using the phrase "means for," or, in the case of a method claim, unless the element is recited using the phrase "step for." All structural and functional equivalents of the elements of the various embodiments described throughout this disclosure that are known or later become known to those skilled in the art are expressly incorporated herein by reference and are encompassed by the claims. Moreover, nothing disclosed herein is made available to the public, whether or not such disclosure is expressly recited in the claims.

Claims

1. A computer-implemented method for a pose estimation task, comprising: processing the received image input data at a feature extraction stage of the machine learning model to generate a feature map; Iteratively, (a) applying an attention map to the feature map to generate an augmented feature map; (b) processing the augmented feature map in a refinement stage of the machine learning model to generate an improved feature map; and (c) processing the refined feature map through an attention stage of the machine learning model to generate an updated attention map for the iteration of steps (a) and (b); and processing the refined feature map through a first regression stage of the machine learning model to generate multi-dimensional task output data; providing the multidimensional task output data as an output of the machine learning model; 1. A computer-implemented method for a pose estimation task comprising:

2. The method of claim 1 , further comprising processing the multi-dimensional task output data with a 3D position generator to generate a 3D position estimate.

3. The method of claim 2 , further comprising processing the three-dimensional position estimate with the multi-dimensional task output data to generate a two-dimensional position estimate.

4. the 3D position generator comprises a body element pose model; the three-dimensional position estimate comprises one or more body element joint positions in three-dimensional space; the two-dimensional position estimate comprises the one or more body element joint positions in two-dimensional space; The multidimensional task output data includes: Rotation estimate (R), the transformed estimate (t), or Scaling estimate (s) The method of claim 3 , comprising one or more of:

5. The method of claim 4 , wherein the body element pose model comprises a hand model.

6. 5. The method of claim 4, further comprising processing the refined feature map in a second regression stage of the machine learning model to generate a task uncertainty estimate.

7. determining an uncertainty associated with the three-dimensional position estimate based on the task uncertainty estimate; and determining an uncertainty associated with the two-dimensional position estimate based on the task uncertainty estimate; The method of claim 6 further comprising:

8. processing the task uncertainty estimates in a loop decision stage of the machine learning model to generate a loop decision value; determining whether to terminate the iterative process of the machine learning model based on the loop decision value; The method of claim 6 further comprising:

9. Processing the augmented feature map during the refinement stage of the machine learning model comprises applying a batch normalization layer to the augmented feature map; The batch normalization layer is selected from a plurality of batch normalization layers based on a loop count. The method of claim 1.

10. The method of claim 4 , further comprising displaying the two-dimensional position estimate on a display screen of an extended reality device.

11. The method of claim 8 , further comprising determining to terminate processing of the machine learning model based on the loop decision value.

12. determining not to terminate the processing of the machine learning model based on the loop determination value; determining to terminate processing of the machine learning model based on the number of loops; The method of claim 8 further comprising:

13. determining a training loss based on a training loss function, wherein the training loss function is The uncertainty loss component, a task loss component; updating the machine learning model based on the training loss; and The method of claim 1 further comprising:

14. 1. A processing system comprising: a memory having computer-executable instructions; one or more processors; 14. A processing system comprising: a processor configured to execute the computer-executable instructions and cause the processing system to perform a method according to any one of claims 1 to 13.

15. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method according to any one of claims 1 to 13.

Images