Probabilistic keypoint regression with uncertainty

Abstract

Keypoints are predicted in an image. A neural network configured to predict each of the keypoints as 2D random variables, which are normally distributed in 2D positions and a 2 * 2 covariance matrix, is performed. The neural network is trained to maximize a log-likelihood value that a sample from each of the predicted keypoints equals a base fact. A trained neural network is used to predict key points of an image without generating a heat map.

Description

Background Art

[0001] Landmark detection is a computer vision task in which key points of images, such as key points (e.g., feature points) of images of human bodies or faces, or more generally of any a priori object, are detected and located in images and videos. For example, key points can be employed to detect the position and rotation of a person's head. Landmark detection can be challenging due to variability and many factors such as pose and occlusion. The disclosures made herein are made with respect to these and other considerations. Summary of the invention

[0002] Landmarks often play a key role in image analysis, but many aspects of identity or expression cannot be represented by a limited number of landmarks. In order to more accurately reconstruct, for example, faces, landmarks are often combined with additional signals (such as depth images) or techniques (such as differentiable rendering). The present disclosure provides a way to use more landmarks in an efficient and cost-effective manner. In addition to faces, the present disclosure can be more generally applied to other types of images.

[0003] In an embodiment, synthetic training data can be used to ensure perfect landmark annotation. By fitting a deformable model to these dense landmarks, state-of-the-art results for monocular 3D face reconstruction can be achieved with real-time responsiveness. Dense landmarks are ideal signals for integrating facial shape information across frames, which can be demonstrated with accurate and expressive facial behavior capture in both monocular and multi-view scenarios.

[0004] The keypoint confidence or certainty is useful when later algorithms consume the keypoints. For example, when fitting a 3D model to 2D keypoints, if the keypoint confidence is low, then the keypoint can be considered unreliable and not considered during model fitting. This can happen, for example, if the keypoint is occluded. Estimated uncertainty can also be used to train better landmark estimators.

[0005] The present disclosure includes an algorithm for directly predicting keypoints (2D points of interest) in an image with uncertainty. In this way, the system (e.g., a neural network) is able to reveal its confidence about each keypoint. Figure 1 , given a single image 110, a large number of landmarks 120 are robustly and accurately predicted. To aid visualization, lines are drawn between the landmarks. The disclosed 3D deformable face model is fitted to these landmarks, thereby reconstructing the face 130 in 3D. Figure 2Compared to a typical sparse set of (e.g., 68) facial landmarks 210, the disclosed dense landmarks 220 cover the entire head in greater detail, including ears, eyes, and teeth. These dense landmarks can provide improved encoding of facial recognition terms and subtle expressions.

[0006] The present disclosure enables direct regression for real-time applications without, for example, a heat map. As used herein, a heat map may be an image in which the value stored at each pixel corresponds to the likelihood that a landmark is located at the location of that pixel in the image. In an embodiment, the task may be changed from a 2D point estimation problem to a 2D random variable estimation problem. Each key point may be predicted as a 2D random variable, typically normally distributed with position (x, y) and standard deviation sigma. The network may be trained to maximize the log-likelihood that a sample from each predicted key point is equal to the ground truth. Key point uncertainty occurs because the network is penalized for making errors and having uncertainty around key point locations. A derivation (assuming a uniform prior based on sigma) is provided below:

[0007] The ground truth consists of a set of keypoint coordinates

[0008]

[0009] Each keypoint is predicted as a 2D random variable, typically normally distributed with position (x', y') and (circular) standard deviation σ. For a predicted keypoint random variable, the relative likelihood of a sample from that random variable will be equal to the ground truth keypoint location is:

[0010]

[0011] For the entire set of N key points, predict a set of coordinates

[0012] S=[σ0,σ1,…,σ N ].

[0013] We work with log-likelihoods because they are more convenient. Since we want to maximize the log-likelihood that the sample from each predicted keypoint is equal to the ground truth, we minimize the loss which is the sum of negative log-likelihoods.

[0014]

[0015]

[0016]

[0017]

[0018] For clarity, this is split into two parts:

[0019] Loss=Loss σ +Loss μ ,in,

[0020] and

[0021] Loss σ The network is penalized for being too uncertain about predictions around key points, and Loss μ The network is penalized for making poorly located keypoint predictions. Additionally, in some examples, the symmetric Gaussian in the above examples can be extended to an asymmetric Gaussian.

[0022] The distribution of uncertainty values ​​can also be influenced at training time by introducing appropriate priors. The above formula assumes a uniform prior on the prediction sigma. The natural choice for the prior of the sigma (or precision) of the (2D) Gaussian distribution is the Wishart distribution (the conjugate prior of the Gaussian distribution), but others can also be used. The prior is the gamma distribution in the univariate case. The effect of doing this is to encourage the network to allocate more neural resources to the current poorly done situation (where sigma is large) and to allocate fewer neural resources to the already quite certain situation (where sigma is small) at training time, so as to balance the usefulness of its key point predictions for downstream model fitting. See below for the derivation assuming a Gamma prior.

[0023] Definition (accuracy, Gaussian distribution, Gamma or Wishart)

[0024] Accuracy

[0025]

[0026]

[0027] a and b are (artificially) adjusted constants "shape and inverse scale"

[0028]

[0029] According to the derivation of the key point log-likelihood value, p is maximized by minimizing -log(p).

[0030]

[0031]

[0032]

[0033] According to the key point loss term for the Gamma prior,

[0034] Item 1 is a constant and thus has no effect on training (ignore it). Add the following term to the loss instead of -log(Gam):

[0035] =>Loss+=bτ-(a-1)log(τ),

[0036] The same is true in terms of σ:

[0037]

[0038] In addition, object detection via keypoint uncertainty can be implemented. A sliding window can be applied on the image, and the average keypoint confidence for each window can be measured. If no window with a high average keypoint certainty is found, it can be determined that the object is not in the image. Otherwise, the window reporting the highest average keypoint confidence can be taken to contain the object.

[0039] Some of the use cases that can be achieved using the described techniques include receiving image input from a conventional color (RGB) camera instead of a depth camera, prediction with many more landmarks, and combining with a model fitter that predicts intrinsic camera parameters (e.g., focal length). This achieves good results for recovering 3D structure from RGB images taken by various cameras. Additional use cases include performing 3D reconstruction from multiple views, where the uncertainty in each view is taken into account, and where the non-intrinsic parameters of each camera are optimized simultaneously. More generally, image input from various types of cameras (such as web cameras, depth cameras, cameras on head-mounted displays (HMDs), IR cameras, event cameras, etc.) can be received, and the images can be RGB images, depth map images, IR images, etc. The placement of the camera can be from the outside-in (e.g., the sensor is fixed, as in a web camera), or introspective position tracking, where the camera or sensor is located on the device being tracked (e.g., HMD). In the case of an HMD, dense landmarks on the observed part of the face can also be used to position the HMD itself relative to the face.

[0040] This summary is not intended to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that overcome any or all disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The detailed description will be described with reference to the accompanying drawings. In the drawings, the left-most digit of a reference number identifies the drawing in which the reference number first appears. The use of the same reference numbers in different drawings indicates similar or identical items.

[0042] Figure 1 is a diagram illustrating an example of the disclosed technology according to one embodiment disclosed herein.

[0043] Figure 2 is a diagram illustrating an example of the disclosed technology according to one embodiment disclosed herein.

[0044] Figure 3 is a diagram illustrating an example of the disclosed technology according to one embodiment disclosed herein.

[0045] Figure 4 is a diagram illustrating an example of the disclosed technology according to one embodiment disclosed herein.

[0046] Figure 5 is a diagram illustrating an example of the disclosed technology according to one embodiment disclosed herein.

[0047] Figure 6 is a diagram illustrating an example of the disclosed technology according to one embodiment disclosed herein.

[0048] Fig. 7A is a flow chart illustrating aspects of an illustrative routine according to one embodiment disclosed herein.

[0049] Figure 7B is a flow chart illustrating aspects of an illustrative routine according to one embodiment disclosed herein.

[0050] Fig. 8A is a flow chart illustrating aspects of an illustrative routine according to one embodiment disclosed herein.

[0051] Figure 8B is a flow chart illustrating aspects of an illustrative routine according to one embodiment disclosed herein.

[0052] Fig. 9 is a computer architecture diagram illustrating aspects of an exemplary computer architecture for a computer capable of executing the software components described herein.

[0053] Fig.10 is a computer architecture diagram illustrating aspects of an exemplary computer architecture for a computer capable of executing the software components described herein.

[0054] Fig.11 is a data architecture diagram showing an illustrative example of a computer environment. DETAILED DESCRIPTION

[0055] Keypoints are typically located in an image by having a neural network generate a heatmap image with high pixel values ​​in portions of the image close to the keypoint and low pixel values ​​in portions of the image far from the keypoint. Another algorithm can run (argmax) on the heatmap image to find the maximum value as a peak. The location of the peak is the 2D location of the keypoint. The value of the peak of the heatmap can sometimes be used as a measure of uncertainty.

[0056] The present disclosure provides a way to directly regress 2D coordinates and uncertainty measures without using heatmap images. This has several technical advantages:

[0057] Generating heatmap images is computationally expensive. In fact, drawing these heatmaps can be the most expensive part of a neural network. With the help of direct keypoint regression, this computation time can be saved.

[0058] In a multi-core system, transferring the heatmap image from one core to another can be a bottleneck. Instead, transferring only the keypoint coordinates and confidence values ​​uses much less bandwidth. For a 128×128 image, this corresponds to an improvement of over 99% in bandwidth.

[0059] In computationally constrained systems, running argmax on the heatmap image can be expensive. By regressing the keypoints directly, running argmax can be avoided. Additionally, argmax is only pixel accurate, while direct regression can be much more accurate at the sub-pixel level.

[0060] Predicting keypoints directly can allow upscaling to more keypoints than is computationally possible when utilizing heatmaps.

[0061] By predicting uncertainty in the manner described, it is clearer how to best use the uncertainty estimate in another system, since the sigma will be in the same units as the prediction. When the heat map peak is used as a measure of uncertainty, it does not have obvious units to integrate into another system.

[0062] By penalizing uncertainty with appropriate priors during training, the resources of the neural network can be automatically allocated to achieve higher equality of results with respect to their predictive accuracy, with more capacity being allocated to harder key points as needed.

[0063] In some embodiments, the disclosed techniques can be used to train a neural network that predicts 2D keypoints in RGB images. For example, the disclosed techniques can also be used to efficiently create depth maps. The rigid pose can then be calculated by using an n-point perspective solution, where these uncertainty values ​​are used to weight the 2D projection error of each keypoint. Keypoints that are occluded by objects or have high motion blur may have higher uncertainty values.

[0064] The disclosed techniques can be applied to multi-camera fitting scenarios, where a model of an object (e.g., a face) is fitted to 2D probabilistic keypoint estimates from multiple cameras. Some keypoints in a particular view may be more uncertain than the same keypoints in a different view. For example, when the face is viewed from the front, both eyes may be clearly visible with high confidence. However, when viewed from the side, only one eye is visible, and when fitting to that view, the other eye may not be considered due to high uncertainty. In this way, although all keypoints are predicted for all views, each view of the face makes the greatest contribution to the most visible part of the face. Additionally, when using probabilistic keypoint energies for multiple cameras, camera correction itself can be used. In some embodiments, the extension of the fitting model to landmarks can allow camera calibration to be performed along with other parameters.

[0065] With respect to object detection via keypoint uncertainty, in existing approaches, training an object detector requires "positive" images that contain the object and "negative" images that do not contain the object. With the disclosed techniques, negative images are not required, enabling the creation of object detectors that are trained using only positive images. This can be useful when negative images are difficult to collect reliably (e.g., images without people in the image).

[0066] Landmarks are points that have correspondence across all faces, such as the tip of the nose or the corners of the eyes. They often play a role in face-related computer vision, for example, being used to extract facial regions of interest, or helping to constrain 3D model fitting. However, many aspects of facial identity or expression cannot be encoded by a typical sparse set of 68 landmarks alone. For example, without landmarks on the cheeks, we cannot tell if someone has high cheekbones. Or, without landmarks around the outer eye region, we cannot tell if someone is gently closing their eyes or scrunching their face.

[0067] Therefore, to reconstruct faces more accurately, previous work has used additional signals besides color images, such as depth images or optical flow. However, these signals may not be available or reliable for computation. Instead, others have approached the problem using analysis-by-synthesis, given color images alone: ​​minimizing the photometric error between a generative 3D face model and the observed image using differentiable rendering.

[0068] However, these schemes are limited by the approximations that must be made in order for differentiable rendering to be computationally feasible. In practice, faces are not purely Lambertian, and many important lighting effects, such as ambient occlusion or the shadow cast by the nose, are not explained when using spherical harmonics alone.

[0069] It would be desirable to use only more landmarks, and the disclosed embodiments describe this. The disclosed examples include methods that are both accurate and robust in predicting over 700 landmarks. Instead of just the frontal "hockey mask" portion of the face, the described landmarks cover the entire head, including the ears, eyeballs, and teeth. As in Figure 1 As shown in , these landmarks provide rich signals for facial recognition terms and expressions. Figure 1 In Figure 2, given a single image (top), we first robustly and accurately predict 703 landmarks (middle). To aid visualization, we draw lines between the landmarks. We then fit the disclosed 3D morphable face model to these landmarks to reconstruct the face in 3D (bottom).

[0070] Even with as few as 68 landmarks, it is difficult to accurately define landmarks that are not aligned with salient image features. Therefore, synthetic training data that ensures consistent annotation can be used. In addition, instead of representing each landmark as just a 2D coordinate, we predict each landmark as a 2D random variable: a 2D circular Gaussian with position and uncertainty. This allows the predictor to express uncertainty about specific landmarks (e.g., occluded landmarks on the back of the head) and also improves landmark accuracy.

[0071] Since the disclosed dense landmarks represent points with correspondence across all faces, we are able to perform 3D face reconstruction by fitting a morphable face model to them. While previous approaches fit a morphable model to the landmarks in a similar manner, this disclosure shows that the landmarks are only the signal required to achieve state-of-the-art results for monocular face reconstruction in the wild. All that is needed are enough landmarks predicted with sufficient accuracy.

[0072] The probabilistic nature of the predictions also makes them ideal for fitting 3D models over a time series or across multiple views. The optimizer can disregard uncertain landmarks and can rely on more certain landmarks. This disclosure demonstrates this with accurate and expressive results for both multi-view and monocular facial action capture. Finally, this disclosure shows that predicting dense landmarks and performing model fitting can be efficient, demonstrating real-time facial action capture at over 60 FPS on a single CPU thread.

[0073] The present disclosure shows that high-quality 3D facial reconstruction does not require parametric appearance models, illumination models, or differentiable rendering, but only requires a certain number of accurate 2D landmarks and a 3D model to be fitted to them. In addition, the present disclosure also shows that combining probabilistic landmarks and model fitting enables intelligent aggregation of facial information across multiple images, which shows robust and expressive results for both multi-view and monocular facial action capture.

[0074] In recent years, methods for 3D face reconstruction have become increasingly sophisticated, involving differentiable rendering and complex neural network training strategies. In contrast, the present disclosure proposes an effective method by keeping it simple. The disclosed scheme consists of two stages: First, we use a conventional convolutional neural network (CNN) to predict probabilistic dense 2D landmarks L. Then, we fit a 3D face model parameterized by Φ to the 2D landmarks in a way that minimizes an energy function E(Φ; L). The image itself is not part of this optimization; the only data used are the 2D landmarks. Reference Figure 3 , given an image 310, we first predict probabilistic dense landmarks L 320, each of which has a location μ and a certainty σ. We then fit our 3D face model 330 to L, minimizing the energy E by optimizing the model parameters Φ 340.

[0075] One difference between the present disclosure and previous approaches is the number and quality of landmarks. Previous approaches have predicted many 2D landmarks with such accuracy that high-quality 3D facial reconstruction can be achieved by fitting a 3D model to these landmarks alone.

[0076] Probabilistic Landmark Regression. We predict each landmark as a random variable using the probability density function of a circular 2D Gaussian.

[0077]

[0078] p(σ)=1

[0079] Thus, Li = {μi,σi}, where μi = [xi,yi] is the expected position of the landmark, and σi (standard deviation) is a measure of uncertainty. Our training data includes labels for the landmark positions μ'i = [x'i,y'i] instead of σ. The network learns to output σ to indicate that it is certain about some landmarks, e.g., visible landmarks on the front of the face, and uncertain about others, e.g., landmarks hidden behind the hair. Figure 4 , when parts of the face are occluded by, for example, hair 400, clothing 410, or hands 420, the corresponding landmarks shown at 405, 415, 425 can be predicted with high uncertainty (red) compared to those landmarks that are visible (green). This can be achieved by training the network with a Gaussian Negative Log-Likelihood (GNLL) loss:

[0080]

[0081] Loss σ The network is penalized for being too uncertain, and Loss μ The network is penalized for making poorly localized landmark predictions while penalizing certainty. i is a per-landmark weight that focuses the loss on specific parts of the face. This is the only loss term used during training.

[0082] The disclosed embodiments enable prediction of occluded keypoints and keypoints outside the image, which can be applied in at least two ways - with respect to ground truth or with respect to network predictions. For ground truth, the use of synthetic training data makes it easier to create representative / accurate ground truth for keypoints not observed by the camera / sensor. For network predictions, keypoints outside the image cannot be predicted when using a heatmap scheme, which is an important advantage of keypoint regression as disclosed herein.

[0083] Landmarks are usually predicted via heatmaps. However, generating heatmaps is computationally expensive. Instead, we keep things simple and directly regress the position and uncertainty using a regular CNN. We take any off-the-shelf architecture, e.g., ResNet, and change the final fully connected layer to output three values ​​per landmark: two for position and one for uncertainty.

[0084] For square images with size S×S, the landmark coordinates are normalized from [0, S] to [-1, 1]. Instead of outputting σ directly, we will predict the logarithm σ and take its exponent to ensure that σ is positive. We use the AdamW optimizer from the timm library to train the PyTorch model. Figure 5 Explains how we extract regions of interest (ROIs) on which we run our dense landmark CNN. To extract facial ROIs from the entire image, we run a sparse landmark CNN over the entire head (a) with sliding windows at multiple scales (b) and select the window with the lowest mean uncertainty (c) as our ROI (d). We perform appearance and geometry data augmentation to assist our synthetic data across the domain gap.

[0085] Given a probabilistic dense 2D landmark L, one goal is to find the best model parameters Φ* that minimize the following energy:

[0086]

[0087] E landmarks is the only term that encourages the 3D model to explain the observed 2D landmarks. The other terms regularize the fit using prior knowledge.

[0088] The advantage of the disclosed scheme is how it scales naturally to multiple images and cameras. The general form of the disclosed method is described herein, which is suitable for F frames on C cameras, i.e., multi-view action capture. In the monocular case, C=1, which simplifies some terms.

[0089] 3D face model. A linear face model can be used, which includes N = 7667 vertices and K = 4 skeletal joints (head, neck, and eyes). The vertex positions can be determined by the mesh generation function M(β,ψ,θ): R|β|+|ψ|+|θ|→R3N, which takes β∈R|β| for identification, ψ∈R|ψ| for expression, and θ∈R3K+3 for skeletal pose (including root joint translation).

[0090]

[0091] in, is the standard linear blend skinning (LBS) function, which makes the vertex positions Around joint position The local joint rotations are measured in θ, where the weight of each vertex is The facial mesh and joint positions in the binding pose are respectively obtained by and to be sure.

[0092] To simplify annotation, it can be assumed that the mesh contains only landmark vertices.

[0093] Camera model. Take a perspective camera. Each pass includes a world-to-camera rigid transformation including rotation and translation And the pinhole camera projection matrix Therefore, the image space projection of the jth landmark in the i-th camera is In the monocular case, X is the discriminant transform and can be ignored.

[0094] The parameters Φ can be optimized to minimize E. The main parameters of interest control the face, but camera parameters can also be optimized if they are unknown or known to be inaccurate.

[0095]

[0096] The facial recognition term β is shared across a sequence of F frames, but the expression Ψ and pose Θ vary per frame. For each of our C cameras, we have six degrees of freedom for rotation R and translation T, and a focal length parameter f (assuming square pixels and a principal point at the center of the image). In the monocular case, we only optimize the focal length.

[0097] E landmarks 3D models are encouraged to explain the predicted 2D landmarks.

[0098]

[0099] Among them, for the kth landmark seen by the jth camera in the i-th frame, [μ ijk , σ ijk ] is the 2D position and uncertainty predicted by our dense landmark CNN, and is the 2D projection of the landmark on our 3D model. Equation 4 is similar to the Loss in Equation 3. μ The similarity of is not accidental: treating landmarks as 2D random variables during both prediction and model fitting allows our scheme to handle uncertainty cleverly, thereby exploiting landmarks that the CNN trusts and disregarding those it is uncertain about.

[0100] E identity Unlikely facial shapes are penalized by maximizing the relative log-likelihood of the shape parameter β under a G-component multivariate Gaussian mixture model (GMM). This GMM was fitted to a library of 3D head scans used to train our 3D face model. Thus, E identity = -loh(p(β)), where v i and Σ i is the mean and covariance matrix of the ith component, and γ i is the weight of this component.

[0101] E expression =||ψ|| 2And E joints =||θ i：i∈[2，K] || 2 The optimizer is encouraged to interpret the data with as little expression and joint rotation as possible. We do this by ignoring the root joint θ1 and not penalizing global translation or rotation.

[0102] The jitter is reduced by encouraging the face mesh vertex x to remain stationary between adjacent frames i-1 and i.

[0103] E intersect The optimizer is encouraged to find solutions with no intersections between the skin and the eyeball or teeth. Figure 6 , E intersect The skin vertices in image 600 are encouraged to remain outside of the convex shape, as shown in image 610. Image 620 shows that there is no E intersect The result is 630 and has E intersect The result is 640.

[0104] Real-time model fitting. We use the Levenberg-Marquardt algorithm to optimize our model fitting energy. We rewrite the energy E about the residual vector r as E(Φ) = ||r(Φ)|| 2 =∑ i r i (Φ) 2 . Then, at each iteration k of our optimization, we can compute r(Φ k ) and Jacobian, And use these to solve the symmetric semidefinite linear system via Cholesky decomposition, (J T J+λdiag(J T J))δ k =-J T r. Then, we apply the update rule Φ k+1 =Φ k +δ k .

[0105] In practice, it is not necessary to form the residual vector r nor the Jacobian matrix J. Instead, for performance reasons, we access each term r of the energy i (Φk) directly calculates the quantity J T J and J T r. Most of the computational cost is incurred in evaluating these products for the landmark data items, as expected. However, the Jacobian of the landmark item residuals is not sufficiently dense. Each individual landmark depends on its own subset of expression parameters and is invariant to the other expression parameters. to perform a static analysis of the sparsity of each landmark term, and

[0106] We use this set of i, j indices to reduce our outer product from O(n 2 )arrive The cost of And m i yes We further enforce sparsity by ignoring any component of the Jacobian with an absolute value below a certain empirically determined threshold.

[0107] By exploiting sparsity in this way, the landmark term residual and its derivatives become very efficient to evaluate. This formulation avoids the correspondence problems often seen when exploiting depth images (which require more expensive optimizations). In addition, adding more landmarks only grows the DNN in the last fully connected layer. Thus, it becomes possible to implement very detailed and well-regularized fitters with relatively little computational burden simply by adding a sufficient number of landmarks. The cost of the Cholesky solution for updating δk is independent of the number of landmarks.

[0108] Now go to Fig. 7A , which illustrates an exemplary operation flow for predicting key points in an image according to the present disclosure. Figures 1 to 6 or Figures 9 to 11 One or more components illustrated in the figure provide such an operation flow. The operation flow can be implemented in a system including one or more computing devices. It should be understood by those skilled in the art that the operations of the methods disclosed herein do not necessarily have to be introduced in any particular order, and it is possible and contemplated to perform some or all of these operations in (one or more) alternative orders. The operations are introduced in the order shown to simplify the description and illustration. Operations can be added, operations can be omitted, several operations can be performed together, and / or several operations can be performed simultaneously without departing from the scope of the appended claims.

[0109] It should also be understood that the illustrated method can be terminated at any time and may not be fully executed. Some or all of the operations of the method and / or substantially equivalent operations can be performed by running computer-readable instructions contained on a computer storage medium. The term "computer-readable instructions" used in the specification and claims and its variations are widely used herein to include routines, applications, application modules, program modules, programs, components, data structures, and algorithms, etc. Computer-readable instructions can be implemented on various system configurations, including single-processor or multi-processor systems, microcomputers, mainframe computers, personal computers, portable computing devices, microprocessor-based programmable consumer electronics, and combinations thereof, etc.

[0110] Thus, it should be noted that the logical operations described herein are implemented as: (1) a sequence of computer-implemented actions or program modules running on a computing system, such as those described herein, and / or (2) interconnected machine logic circuits or circuit modules within the computing system. Such implementation is a choice that depends on the performance and other requirements of the computing system. Accordingly, the logical operations may be implemented by software, firmware, dedicated digital logic, and any combination thereof. Thus, although routine 700 is described as running on a system, it can be appreciated that routine 700 and other operations described herein can be performed on an individual computing device or on several devices.

[0111] refer to Fig. 7A , operation 701 illustrates using a convolutional neural network (CNN) to predict a probabilistic dense 2D landmark L.

[0112] Operation 701 may be followed by operation 703. Operation 703 illustrates fitting the 3D model parameterized by the model parameters Φ to the 2D landmark L by minimizing an energy function E(Φ; L).

[0113] Operation 703 may be followed by operation 705. Operation 703 illustrates fitting the 3D model parameterized by the model parameters Φ to the 2D landmark L by minimizing the energy function E(Φ; L).

[0114] Now turn to Figure 7B , which illustrates another exemplary operational flow for predicting key points by a computing system according to the present disclosure. Referring to FIG7 , operation 711 illustrates instantiating, by the computing system, a neural network configured to predict each of the key points as a 2D random variable, which is generally normally distributed with a 2D position and a 2×2 covariance matrix.

[0115] Operation 711 may be followed by operation 713. Operation 713 illustrates training, by the computing system, the neural network to maximize a log likelihood value that a sample from each of the predicted keypoints is equal to a ground truth.

[0116] Operation 713 may be followed by operation 715. Operation 715 illustrates using a trained neural network to predict key points of an image without generating a heat map.

[0117] Now turn to Fig. 8A , which illustrates another exemplary operation flow for predicting key points in an image according to the present disclosure. Figures 1 to 6 as well as Figures 9 to 11 One or more components illustrated in the figure provide such an operation flow. The operation flow can be implemented in a system including one or more computing devices. It should be understood by those skilled in the art that the operations of the methods disclosed herein do not necessarily have to be introduced in any particular order, and it is possible and contemplated to perform some or all of these operations in (one or more) alternative orders. The operations are introduced in the order shown to simplify the description and illustration. Operations can be added, operations can be omitted, several operations can be performed together, and / or several operations can be performed simultaneously without departing from the scope of the appended claims.

[0118] refer to Fig. 8A , operation 801 illustrates predicting each keypoint as a 2D random variable, which is typically normally distributed with position (x, y) and standard deviation sigma.

[0119] Operation 801 may be followed by operation 803. Operation 803 illustrates training the network to maximize the log-likelihood that the sample from each predicted keypoint is equal to the ground truth.

[0120] Now turn to Figure 8B , which illustrates another exemplary operational flow for predicting key points in an image according to the present disclosure. Referring to FIG8 , operation 811 illustrates generating, by a computing system, a prediction for each of the key points in the image as a 2D random variable, the 2D random variable being normally distributed with a position (x, y) and a standard deviation sigma.

[0121] Operation 811 may be followed by operation 813. Operation 813 illustrates training, by the computing system, the neural network to maximize a log-likelihood value that a sample from each of the predicted keypoints is equal to the ground truth.

[0122] Operation 813 may be followed by operation 815. Operation 815 illustrates using a trained neural network to predict key points of an image without generating a heat map.

[0123] As used herein, a parametric model can be a predefined model that can be modified by a number of parameters. Some of these parameters can control the rotation, translation, and scaling of an object (i.e., some global changes), while other parameters control more local deformations (e.g., a value can control how much a facial model "smiles"). During the fitting process, the goal is to find the best explanation for all of these parameters to predict the value of the keypoint.

[0124] exist Fig. 9 In the exemplary system illustrated in , a system 900 implementing a machine learning (ML) platform 990 is illustrated. The ML platform 990 can be configured to provide output data to various devices 950 and to a computing device 930 via a network 920. A user interface 960 can be rendered on the computing device 930090. The user interface 960 can be provided in conjunction with an application 940, which communicates with the ML platform 990 using an API via the network 920. In some embodiments, the system 900 can be configured to provide product information to a user. In one example, the ML platform 990 can implement a machine learning system to perform one or more tasks. The ML platform 990 uses the machine learning system to perform tasks such as image and writing recognition. The machine learning system can be configured to be optimized using the techniques described herein.

[0125] Fig.10 is a computing system architecture showing an overview of the system disclosed herein for implementing a machine learning model according to one embodiment disclosed herein. Fig.10As shown in , the machine learning system 1000 can be configured to perform analysis and perform identification, prediction, or other functions based on various data collected and processed by the data analysis component 1030 (which can be individually referred to as "data analysis component 1030" or collectively referred to as "data analysis component 1030"). For example, the data analysis component 1030 can include, but is not limited to, a physical computing device, such as a server computer or other type of host associated with hardware components (e.g., memory and mass storage devices), and networking components (e.g., routers, switches, and cables, etc.). The data analysis component 1030 can also include software, such as an operating system, an application, and a container, a network service, a virtual component, such as a virtual disk, a virtual network, and a virtual machine. The database 1050 can include data, such as a database or a database shard (i.e., a portion of a database). Feedback can be used to further update various parameters used by the machine learning model 1020. The data can be provided to the user application 1015, thereby providing results to various users 1010 using the user application 1015. In some configurations, the machine learning model 1020 can be configured to utilize supervised and / or unsupervised machine learning techniques. The model compression framework based on sparsity-induced regularized optimization as disclosed in this article can reduce the amount of data that needs to be processed in such systems or applications. Effective model compression when processing iterations over large amounts of data can provide improved latency for a large number of applications using such techniques, such as image and sound recognition, recommendation systems, and image analysis.

[0126] An exemplary architecture for a real-time system in which the disclosed embodiments can be implemented includes MobileNetV2. An exemplary architecture for an offline system in which the disclosed embodiments can be implemented includes ResNet 101. These architectures are known to those skilled in the art.

[0127] Various aspects of the present disclosure are described herein relative to specific examples and embodiments, which are intended to illustrate rather than limit the present disclosure. It should be appreciated that the subject matter introduced herein can be implemented as a computer process, a computer-controlled device or a computing system or product, such as a computer-readable storage medium. Although the subject matter described herein is introduced in the general context of a program module (executed on one or more computing devices), those skilled in the art will recognize that other implementations can be performed in combination with other types of program modules. In general, a program module includes a routine, a program, a component, a data structure, or other types of structures that perform a specific task or implement a specific abstract data type.

[0128] Those skilled in the art will also appreciate that the subject matter described herein can be practiced on other computer system configurations (including multiprocessor systems) beyond those described herein or in conjunction with these configurations. The embodiments described herein can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices linked through a communication network. In a distributed computing environment, program modules can be located both in a local memory storage device and in a remote memory storage device.

[0129] A network established by or on behalf of a user to provide one or more services (such as various types of cloud-based computing or storage) and accessible to a distributed set of clients via the Internet or other networks may be referred to as a service provider.

[0130] In some embodiments, a server implementing part or all of one or more of the techniques described herein (including techniques implementing methods for predicting key points in an image) may include a general purpose computer system that includes or is configured to access one or more computer-accessible media. Fig.11 Such a general purpose computing device 1100 is illustrated. In the illustrated embodiment, the computing device 1100 includes one or more processors 1110a, 1110b, and / or 1110n (which may be referred to herein individually or collectively as “processors 1110”) coupled to a system memory 1120 via an input / output (I / O) interface 1130. The computing device 1100 also includes a network interface 1140 coupled to the I / O interface 1130.

[0131] In various embodiments, computing device 1100 may be a uniprocessor system including one processor 1110, or a multiprocessor system including a number of processors 1110 (e.g., two, four, eight, or another suitable number). Processor 1110 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 1110 may be a general-purpose or embedded processor that implements any of a variety of instruction set architectures (ISAs), such as x116, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISAs. In a multiprocessor system, each of processors 1110 may typically, but not necessarily, implement the same ISA.

[0132] The system memory 1120 may be configured to store instructions and data that can be accessed by the processor(s) 1110. In various embodiments, the system memory 1120 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash-type memory, or any other type of memory. In the illustrated embodiment, program instructions that implement one or more intended functions (such as those methods, techniques, and data described above) are shown as stored within the system memory 1120 as code 1125 and data 1126.

[0133] In one embodiment, the I / O interface 1130 can be configured to coordinate the I / O traffic between the processor 1110, the system memory 1120 and any peripheral devices (including the network interface 1140 or other peripheral device interfaces) in the device. In some embodiments, the I / O interface 1130 can perform any necessary protocol, timing or other data conversion to convert the data signal from one component (e.g., the system memory 1120) into a format suitable for use by another component (e.g., the processor 1110). In some embodiments, for example, the I / O interface 1130 may include support for devices attached through various types of peripheral device buses (such as variants of the peripheral component interconnect (PCI) bus standard or the universal serial bus (USB) standard). In some embodiments, the functions of the I / O interface 1130 can be divided into two or more separate components. Similarly, in some embodiments, some or all of the functions of the I / O interface 1130 (such as the interface to the system memory 1120) can be directly incorporated into the processor 1110.

[0134] The network interface 1140 may be configured to allow the computing device 1100 to communicate with other one or more devices 1160 attached to one or more networks 1190 (such as Figure 1 7 to other computer systems or devices illustrated in FIG. 7). In various embodiments, for example, the network interface 1140 can support communication via any suitable wired or wireless general-purpose data network (such as various types of Ethernet networks). In addition, the network interface 1140 can support communication via a telecommunications / telephone network (such as an analog voice network or a digital fiber-optic communication network), via a storage area network (such as a Fiber Channel SAN), or via any other suitable type of network and / or protocol.

[0135] In some embodiments, system memory 1120 may be one embodiment of a computer-accessible medium configured to store the data as described above with respect to Figure 1-10The program instructions and data described for implementing the embodiments of the corresponding methods and devices. However, in other embodiments, program instructions and / or data may be received, sent, or stored on different types of computer-accessible media. Computer-accessible media may include non-transitory storage media or memory media, such as magnetic media or optical media, such as a disk or DVD / CD coupled to the computing device 1100 via an I / O interface 1130. Non-transitory computer-accessible storage media may also include any volatile or non-volatile media, such as RAM (e.g., SDRAM, DDRSDRAM, RDRAM, SRAM, etc.), ROM, etc., which may be included in some embodiments of the computing device 1100 as system memory 1120 or other types of memory. In addition, computer-accessible media may include transmission media or signals, such as electrical, electromagnetic, or digital signals conveyed via communication media such as a network and / or wireless link (e.g., which may be implemented via a network interface 1140). Multiple computing devices (e.g., Fig.11 The described functionality in various embodiments may be implemented in part or in whole by a computer (such as those illustrated in the examples); for example, software running on a variety of different devices and servers may cooperate to provide the functionality. In some embodiments, in addition to or as an alternative to implementation using a general-purpose computer system, portions of the described functionality may be implemented using a storage device, a network device, or a dedicated computer system. The term "computing device" as used herein refers to at least all of these types of devices, but is not limited to these types of devices.

[0136] Various storage devices and their associated computer-readable media provide non-volatile storage for the computing devices described herein. The computer-readable media discussed herein may refer to mass storage devices, such as solid-state drives, hard disks, or CD-ROM drives. However, those skilled in the art will recognize that computer-readable media may be any available computer storage media that can be accessed by a computing device.

[0137] By way of example and not limitation, computer storage media may include volatile and nonvolatile, removable and non-removable media implemented by any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology; CD-ROM, digital versatile disk (DVD), HD-DVD, BLU-RAY or other optical storage devices; cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices; or any other medium that can be used to store the desired information and can be accessed by the computing devices discussed herein. For the purposes of the claims, "computer storage medium", "computer-readable storage medium" and their variations do not, by themselves, include waves, signals and / or other transient and / or intangible communication media.

[0138] Encoding the software modules described herein may also transform the physical state of the computer-readable medium described herein. In different embodiments of the present specification, the specific transformation of the physical structure may depend on various factors. Examples of such factors may include, but are not limited to, the technology used to implement the computer-readable medium, whether the computer-readable medium is characterized as primary or secondary, and the like. For example, if the computer-readable medium is implemented as a semiconductor-based memory, the software disclosed herein may be encoded onto the computer-readable medium by transforming the physical state of the semiconductor memory. For example, the software may transform the state of transistors, capacitors, or other discrete circuit elements that constitute the semiconductor memory. The software may also transform the physical state of such components so that data is stored thereon.

[0139] As another example, the computer-readable medium disclosed in this article can be implemented using magnetic or optical technology. In such an embodiment, the software provided herein can transform the physical state of the magnetic or optical medium (when the software is encoded therein). These transformations can include changing the magnetic properties of specific locations in a given magnetic medium. These transformations can also include changing the physical characteristics or properties of specific locations in a given optical medium to change the optical properties of these locations. Other physical medium transformations are also possible without departing from the scope and essence of this specification, wherein the examples provided in the foregoing are only to facilitate this discussion.

[0140] In view of the above, it should be appreciated that many types of physical transformations occur in the disclosed computing device in order to store and execute the software components and / or functions described herein. It is also contemplated that the disclosed computing device may not include Fig.11 All example components shown may include Fig.11 Other components not explicitly shown in the figure or may be used in conjunction with Fig.11Completely different architecture as shown.

[0141] Although various configurations are described in language specific to structural features and / or method operations, it should be understood that the subject matter defined in the attached representations is not necessarily limited to the specific features or operations described. Instead, specific features and operations are disclosed as exemplary forms of implementing the claimed subject matter.

[0142] Furthermore, unless otherwise specifically stated or otherwise understood in the context of use, conditional language used herein, such as "can," "might," "perhaps," "may," "for example," and the like, is generally intended to indicate that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such conditional language is generally not intended to imply that one or more embodiments require certain features, elements, and / or steps in any way, or that one or more embodiments necessarily include logic for determining (with or without author input or prompting) that these features, elements, and / or steps are included in any particular embodiment or will be performed by it. The words "comprise," "include," "have," and the like are synonymous and are used in an open and inclusive manner, which does not exclude additional elements, features, actions, operations, and the like. Moreover, the word "or" is used in its inclusive sense (rather than in its exclusive sense), so when used (for example) to connect a series of elements, the word "or" means one, some, or all of the elements in the list.

[0143] Although certain exemplary embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of the invention disclosed herein. Thus, nothing in the foregoing description is intended to imply that any particular feature, characteristic, step, module, or block is necessary or indispensable. In fact, the novel methods and systems described herein may be embodied in a variety of other forms; in addition, various omissions, substitutions, and changes may be made to the form of the methods and systems described herein without departing from the spirit of the invention disclosed herein. The attached claims and their equivalents are intended to cover such forms or modified forms that fall within the scope and spirit of certain of the inventions disclosed herein.

[0144] It should be recognized that any reference to "first", "second" items and / or abstract concepts, etc. in the description is not intended and should not be interpreted as necessarily corresponding to any reference to "first", "second" elements, etc. in the claims. Specifically, within this invention content and / or the detailed description section below, items and / or abstract concepts (e.g., individual computing devices and / or operating states of a computing cluster) can be distinguished by numerical labels, and such labels do not correspond to such labels in the claims or even other paragraphs of the invention content and / or detailed description. For example, any "first operating state" and "second operating state" markings of a computing cluster within a paragraph of the present disclosure are only used to distinguish between two different operating states within the computing cluster within that specific paragraph (not in any other paragraphs, especially in the claims).

[0145] In the end, although various technologies are described in language specific to structural features and / or method operations, it should be understood that the subject matter defined in the attached representations is not necessarily limited to the specific features or operations described. Instead, specific features and operations are disclosed as exemplary forms of implementing the claimed subject matter.

[0146] The disclosure presented herein covers the subject matter set forth in the following exemplary clauses.

[0147] Clause 1: A method for predicting key points by a computer system, the method comprising:

[0148] instantiating, by the computing system, a neural network configured to predict for each of the keypoints as a 2D random variable, the 2D random variable being normally distributed with a 2D position and a 2×2 covariance matrix;

[0149] training, by the computing system, the neural network to maximize a log-likelihood that a sample from each of the predicted keypoints is equal to a ground truth; and

[0150] Use the trained neural network to predict the keypoints of an image without generating a heatmap.

[0151] Clause 2: The method of clause 1, wherein the distribution of uncertainty values ​​is influenced during said training by introducing appropriate priors.

[0152] Clause 3: The method of any of clauses 1-2, wherein the 2D random variable is a 2D circular Gaussian with position and uncertainty.

[0153] Clause 4: The method of any of clauses 1-3, wherein training the neural network comprises training using a Gaussian negative log-likelihood (GNLL) loss:

[0154]

[0155] Clause 5: The method of any of clauses 1-4, wherein the relative likelihood of a sample from the 2D random variable equal to the position of the ground truth keypoint is:

[0156]

[0157] Wherein, the sample is located at (x', y') and the ground truth is located at (x, y).

[0158] Clause 6: The method of any of clauses 1-5, further comprising minimizing the loss as the sum of negative log-likelihood values.

[0159] Clause 7: The method of any of clauses 1-6, wherein the ground truth comprises a set of keypoint coordinates.

[0160]

[0161] Clause 8: The method of any of clauses 1-7, further comprising predicting occluded keypoints and keypoints external to the image.

[0162] Clause 9: The method of any of clauses 1-8, wherein the keypoints are predicted with sub-pixel accuracy.

[0163] Clause 10: The method of any of clauses 1-9, wherein the images are received from a set of internal view sensors attached to the HMD, the method further comprising:

[0164] 3D facial reconstruction is performed based on the multiple views of the internal perspective sensor.

[0165] Clause 11: The method of any of clauses 1-10, further comprising positioning or calibrating the HMD or an accessory of the HMD.

[0166] Clause 12: The method of any of clauses 1-11, wherein the ground truth images and keypoints comprise image pairs captured by a camera or sensor with annotated ground truth.

[0167] Clause 13: The method of any of clauses 1-12, wherein the ground truth image and key points are computer generated or rendered.

[0168] Clause 14: The method of any of clauses 1-13, wherein the training data for the neural network includes occluded keypoints or keypoints outside the ground truth image.

[0169] Clause 15: A computing system for fitting a model using observational data, the computing system comprising:

[0170] one or more processors; and

[0171] A computer-readable storage medium having computer-executable instructions stored thereon, the computer-executable instructions, when executed by the processor, causing the computing system to perform operations comprising:

[0172] executing a neural network configured to predict for a keypoint as a 2D random variable, typically normally distributed with location (x, y) and standard deviation sigma;

[0173] The network is trained to maximize the log-likelihood that a sample from each predicted keypoint is equal to the ground truth; and keypoints of the image are predicted using the trained network.

[0174] Clause 16: The computing system of clause 15, wherein the probability density function of a circular 2D Gaussian is used To predict each of the key points as a random variable; p(σ)=1.

[0175] Clause 17: The method of any of clauses 15 and 16, wherein the samples from the 2D random variable will be equal to the relative likelihood value of the ground truth keypoint location is:

[0176]

[0177] Wherein, the sample is located at (x', y') and the ground truth is located at (x, y).

[0178] Clause 18: The computing system of any of clauses 15-17, further comprising computer executable instructions stored thereon that, when executed by the processor, cause the computing system to perform operations including minimizing a loss as a sum of negative log-likelihood values.

[0179] Clause 19: The computing system of any of clauses 15-18, wherein the ground truth comprises a set of keypoint coordinates.

[0180]

[0181] Clause 20: A computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a processor of a computing system, cause the computing system to perform operations comprising:

[0182] executing a neural network configured to predictively generate predictions for the keypoints as 2D random variables, typically normally distributed with location (x, y) and standard deviation sigma;

[0183] The network is trained to maximize the log-likelihood that a sample from each predicted keypoint is equal to the ground truth; and the keypoints of the image are predicted using the trained network.

[0184] The disclosure presented herein encompasses the subject matter set forth in the following additional exemplary clauses.

[0185] Clause 1: A method for predicting key points by a computer system, the method comprising:

[0186] generating, by the computing system, a prediction for each of the keypoints of the image as a 2D random variable, the 2D random variable being typically normally distributed with position (x, y) and standard deviation sigma;

[0187] training a neural network by the computing system to maximize a log-likelihood that a sample from each of the predicted keypoints is equal to a ground truth; and

[0188] Use the trained neural network to predict the keypoints of an image without generating a heatmap.

[0189] Clause 2: The method of clause 1, further comprising fitting a parametric model to the predicted key points.

[0190] Clause 3: The method of any of clauses 1-2, wherein the distribution of uncertainty values ​​is influenced during said training by introducing appropriate priors.

[0191] Clause 4: The method of any of clauses 1-3, wherein the image is an image of an object for which an a priori model is available, and the method further comprises preprocessing the image by:

[0192] running a sliding window over the image;

[0193] Measuring an average keypoint confidence for each window in the sliding window;

[0194] If no window with a high average keypoint certainty is found, determining that the object is not in the image; and

[0195] Otherwise the window reporting the highest average keypoint confidence is taken to contain the object.

[0196] Clause 5: The method of any of clauses 1-4, wherein the image is input from one or more of: a conventional color (RGB) camera, a depth camera, an IR sensor, a head mounted camera, an event camera, or a web camera.

[0197] Clause 6: The method of any of clauses 1-5, wherein the method is performed in conjunction with a model fitter that predicts intrinsic camera parameters.

[0198] Clause 7: The method of any of clauses 1-6, wherein the model fitter is configured for a single view, wherein uncertainty of each landmark is taken into account.

[0199] Clause 8: The method of any of clauses 1-7, wherein the model fitter is configured for multiple views.

[0200] Clause 9: The method of any of clauses 1-8, wherein the energy function used to fit the parameter model comprises:

[0201]

[0202] Clause 10: A computing system for fitting a model using observational data, the computing system comprising:

[0203] one or more processors; and

[0204] A computer-readable storage medium having computer-executable instructions stored thereon, the computer-executable instructions, when executed by the processor, causing the computing system to perform operations including:

[0205] Generate predictions for keypoints as 2D random variables, typically normally distributed with location (x,y) and standard deviation sigma;

[0206] The neural network is trained to maximize the log-likelihood that the sample from each predicted keypoint is equal to the ground truth;

[0207] Using a trained neural network to predict key points of an image; and

[0208] Fit a parametric model to the predicted key points.

[0209] Clause 11: The computing system of clause 10, wherein the distribution of uncertainty values ​​is influenced during said training by introducing appropriate priors.

[0210] Clause 12: The computing system of any of clauses 10 and 11, wherein the image is an image of an object, the computing system further comprising computer executable instructions stored thereon, the computer executable instructions, when executed by the processor, causing the computing system to perform operations for pre-processing the image, the operations comprising:

[0211] running a sliding window over the image;

[0212] Measure the average keypoint confidence for each window;

[0213] If no window with a high average keypoint certainty is found, determining that the object is not in the image; and

[0214] Otherwise the window reporting the highest average keypoint confidence is taken to contain the object.

[0215] Clause 13: The computing system of any of clauses 10-12, wherein the predicting and training are performed in conjunction with a model fitter that predicts intrinsic camera parameters.

[0216] Clause 14: The computing system of any of clauses 10-13, wherein the model fitter is configured for a single view, wherein uncertainty of each landmark is taken into account.

[0217] Clause 15: The computing system of any of clauses 10-14, further comprising computer executable instructions stored thereon, which, when executed by the processor, cause the computing system to perform operations including performing 3D reconstruction based on multiple views from multiple cameras, wherein uncertainty in each view is taken into account and wherein non-intrinsic parameters of each camera are optimized simultaneously.

[0218] Clause 16: The computing system of any of clauses 10-15, wherein the camera is an HMD camera.

[0219] Clause 17: The computing system of any of clauses 10-16, further comprising computer executable instructions stored thereon, which, when executed by the processor, cause the computing system to perform operations including using the uncertainty of the neural network to estimate which portions of the object being tracked are visible.

[0220] Clause 18: A method of predicting parameters of a 3D model corresponding to image data using a convolutional neural network (CNN) running on a computing system, the method comprising:

[0221] Predicting probabilistic dense 2D landmarks L using the convolutional neural network (CNN);

[0222] Fitting a 3D model parameterized by model parameters Φ to a 2D landmark L by minimizing an energy function E(Φ; L); and outputting the fitted 3D model.

[0223] Clause 19: The method of clause 18, wherein a model of the object is fit to the 2D probabilistic keypoint estimates from multiple cameras.

[0224] Clause 20: The method of any of clauses 18 or 19, further comprising fitting a 3D model on a time series.

Claims

1. A method for predicting key points by a computing system, the method include: instantiating, by the computing system, a neural network configured to predict each of the keypoints as a 2D random variable, the 2D random variable being normally distributed with a 2D position and a 2×2 covariance matrix; training, by the computing system, the neural network to maximize a log-likelihood that a sample from each of the predicted keypoints is equal to a ground truth; as well as Use the trained neural network to predict the keypoints of an image without generating a heatmap.

2. The method according to claim 1, in, The distribution of uncertainty values ​​is influenced during the training by introducing appropriate priors.

3. The method according to claim 1, in, The 2D random variable is a 2D circular Gaussian with position and uncertainty.

4. The method according to claim 1, in, Training the neural network includes training using the Gaussian Negative Log-Likelihood (GNLL) loss:

5. The method according to claim 1, in, The samples from the 2D random variable will be equal to the relative likelihood of the ground truth keypoint location as: Wherein, the sample is located at (x', y') and the ground truth is located at (x, y).

6. The method of claim 1 further comprising minimizing a loss as a sum of negative log-likelihood values.

7. The method according to claim 1, in, The ground truth includes a set of keypoint coordinates 8. The method of claim 1, further comprising predicting occluded keypoints and keypoints outside the image.

9. The method according to claim 1, in, The keypoints are predicted with sub-pixel accuracy.

10. The method according to claim 1, in, The images are received from a set of internal view sensors attached to the HMD, the method further comprising: 3D facial reconstruction is performed based on the multiple views of the internal perspective sensor. The method of claim 10 , further comprising positioning or calibrating the HMD or an accessory of the HMD.

12. The method according to claim 1, in, The ground truth images and keypoints include image pairs captured by a camera or sensor with annotated ground truth, wherein the training data for the neural network includes occluded keypoints or keypoints outside the ground truth images.

13. The method according to claim 1, in, The ground truth image and the key points are computer generated or rendered.

14. A computing system for fitting a model using observed data, the computing system include: one or more processors; as well as A computer-readable storage medium having computer-executable instructions stored thereon, the computer-executable instructions, when executed by the processor, causing the computing system to perform operations including: executing a neural network configured to predict a keypoint as a 2D random variable, typically normally distributed with location (x, y) and standard deviation sigma; Training the network to maximize the log-likelihood that a sample from each predicted keypoint is equal to the ground truth; and Use the trained network to predict the keypoints of the image.

15. A computer-readable storage medium having computer-executable instructions stored thereon, the computer-executable instructions, when executed by a processor of a computing system, causing the computing system to perform operations comprising: executing a neural network configured to predictively generate predictions for the keypoints as 2D random variables, typically normally distributed with location (x, y) and standard deviation sigma; The network is trained to maximize the log-likelihood that a sample from each predicted keypoint is equal to the ground truth; as well as Use the trained network to predict the keypoints of the image.

Images