Image processing system and method for feature point position estimation with uncertainty

Abstract

A controller for performing a task based on probabilistic image-based feature point localization uses a neural network that is trained to process images of an object of a type having a structured feature point set to generate a parameter probability distribution defined by a value of a parameter for the location of each feature point in each processed image. The controller submits a set of input images to the neural network to generate values ​​of the parameter defining a parameter probability distribution over the location of each feature point in the structured feature point set for each input image. In addition, the controller determines, for each input image, a global feature point uncertainty for the image based on the parameter probability distribution of the feature points in the input image, and performs the task based on the parameter probability distribution of the feature points in each input image and the global feature point uncertainty for each input image.

Description

Technical Field

[0001] The present invention relates generally to image and video processing, and more particularly to estimating the positions of landmarks on objects. Background Art

[0002] Feature point localization refers to estimating the positions of multiple predetermined feature points of an object in an image. Feature point localization is performed on images of different types of objects (such as faces and bodies) that have a structured set of feature points. If feature point localization is performed on a face image, then feature point localization is called face alignment, and includes estimating the positions of a structured set of feature points in the image, including predetermined facial feature points such as the corners of the eyes and the corners of the mouth. If feature point localization is performed on a body image, then feature point localization is called human pose estimation, and includes estimating the positions of a structured set of feature points of the body, including predetermined anatomical feature points such as the elbows and knees, in the image.

[0003] For example, landmark localization on facial images (i.e., face alignment) is a fundamental step in face-related tasks for many computer vision applications, including facial expression recognition, head pose estimation, 3D facial modeling, human-robot interaction, and remote photoplethysmography. The accomplishment of all these tasks depends heavily on the accuracy of facial landmark localization. However, due to the variations in different facial images caused by factors such as facial shape, facial expression and orientation, lighting, and image quality, face alignment is inevitably affected by errors caused by the uncertainty of landmark localization.

[0004] Many approaches to face alignment and human pose estimation are based on deep learning models. Current deep learning models for these tasks typically use a cascade of deep networks, a network architecture developed for human pose estimation.

[0005] Such a deep learning model is heatmap-based because such a deep learning model is trained using the distance (l2 distance, which is the Euclidean distance) between the predicted heatmap and the proxy ground-truth heatmap for each feature point. The proxy ground-truth heatmap can be generated by placing an isotropic Gaussian distribution with a small fixed variance at the true feature point location. In addition, the deep learning model uses the argmax of each predicted heatmap (i.e., the pixel location where the heatmap has its maximum value) to determine the location of the feature point.

[0006] Deep learning models improve the accuracy of feature point localization. However, this approach has several drawbacks. For example, since the argmax of the heat map can only be determined to the nearest pixel, quantization errors are introduced during the determination of feature point locations.

[0007] Most deep learning methods for feature point localization output a single position for each feature point, and the network cannot output the uncertainty level of each position estimate. A deep learning method for face alignment uses a nonparametric probability density function to estimate the position of each feature point. In the nonparametric method, a kernel density network is obtained by convolving a heat map with a fixed isotropic Gaussian kernel. The nonparametric kernel density network model is a method that approximates a large continuous distribution as a union of a very large discrete set of simple parametric models. However, nonparametric methods may require a large amount of training data to obtain accurate results. In addition, when the distribution being modeled nonparametrically is close to the shape of a simple parametric distribution, the nonparametric method is suboptimal. In addition, this nonparametric method is limited by the underlying pixel grid resolution of the heat map and the fixed kernel size; for example, a distribution narrower than the kernel size cannot be accurately modeled using a nonparametric method. Therefore, it is necessary to develop systems and methods to verify and improve the accuracy of the estimation of feature point positions. Summary of the invention

[0008] Some embodiments aim to improve the accuracy of feature point location of a plurality of predetermined feature points of an object in an image. Additionally or alternatively, some embodiments aim to improve the accuracy of task performance based on the accuracy of feature point location. Additionally or alternatively, some embodiments aim to improve the accuracy of performance of these tasks even in the presence of errors in feature point location.

[0009] Some embodiments are based on the recognition that errors in feature point localization are possible, but that these errors are recoverable for the purpose of subsequent execution of tasks that rely on the feature point locations. For example, for a vehicle driver monitoring system that performs facial alignment in a sequence of images of the driver's face, if facial alignment is uncertain for some images, then the image can be reduced for the monitoring task, e.g., the weight of the image in the set execution can be reduced, or the image can be skipped altogether.

[0010] To this end, some embodiments are based on the recognition that, in addition to feature point localization, it is beneficial to estimate the uncertainty of such estimates and use these uncertainties to guide the performance of subsequent tasks. In addition, some embodiments are based on the recognition that the estimation of the uncertainty of feature point localization should not be a separate process, but should be integrated into the feature point localization itself. Specifically, if a neural network is trained to perform feature point localization, such a neural network should output not only the feature point position, but also the uncertainty of the estimated position. In this way, the neural network can be trained end-to-end to achieve two goals.

[0011] However, the configuration and training of such a dual-objective neural network creates additional challenges related to the complexity of non-parametric data and the difference between the uncertainty of feature point positioning for each feature point and the uncertainty of the entire image. Some embodiments address these issues in two steps. First, the neural network used in some embodiments does not output the location of the feature points, but rather outputs a parameter probability distribution over the location of each feature point in the image. Parameter probability estimation involves determining a small number of parameters of a parametric model to represent the probability distribution over the location of each feature point in the image. Therefore, a small parameter model is used instead of a large non-parametric model to represent the probability distribution over the location of each feature point. In some embodiments, different parameters of the parameter probability distribution of each feature point define a point estimate of the location of the feature point and the uncertainty of the location estimate of each feature point. In this way, a single output of the neural network can represent both the location of the feature point and the uncertainty of its estimate. In this way, the neural network of some embodiments is trained for dual objectives without significantly increasing the computational burden and the amount of output data.

[0012] An example of a parameter probability distribution defined by the values ​​of the parameters for the location of each feature point in each processed image is a Gaussian distribution, where the parameters determine the mean and covariance matrix of the Gaussian distribution. In this example, the mean defines the location of the feature point and the covariance matrix defines the uncertainty. Note that the covariance matrix defines the multi-dimensional (e.g., two-dimensional) shape and structure of the Gaussian probability distribution over the feature point locations.

[0013] In addition, some embodiments determine a parameter distribution for each feature point in the input image. In this way, uncertainty estimates can be customized for each individual feature point. However, some embodiments determine the global uncertainty of the input image based on a combination of parameters of the parameter probability distribution determined for each feature point of the input image. For example, in some implementations, the global uncertainty is a single scalar measurement that summarizes the uncertainty of the positions of all individual feature points in the image. Task execution can use this single scalar measurement of the input image as a simple measurement to determine how to process the input image. In this way, the accuracy of task execution is improved.

[0014] Some embodiments are based on the recognition that the parameter probability distribution for each feature point includes an estimate of the location of the feature point, such that the location of the feature point and the uncertainty of the location estimate of each feature point are derived from the parameters of the parameter probability distribution. To this end, some parameters of the parameter probability distribution define point estimates of the location of the feature point, and other parameters of the parameter probability distribution define the uncertainty of the estimate of the location of the feature point, such as the shape and spread of the probability distribution. However, a heat map based representation may be suboptimal for estimating some of these parameters. In fact, different mechanisms may be used to optimally estimate different parameters.

[0015] For example, in some embodiments where a Gaussian probability distribution is estimated for each feature point, the mean of the Gaussian probability distribution (which is a point estimate of the feature point location) is obtained from one or more heat maps, and the parameters for determining the covariance matrix are obtained by direct regression. Some embodiments have experimentally determined that obtaining a covariance matrix (which determines the uncertainty of the distribution, i.e., its shape and extension) of a Gaussian covariance matrix by direct regression (e.g., from low-dimensional eigenvalues ​​at the bottleneck layer of a neural network) provides more accurate results than obtaining a covariance matrix from a heat map. This may be because the heat map does not have a high enough resolution to give a good estimate of the size and shape of the distribution. In addition, some embodiments have also experimentally found that obtaining a point estimate of the feature point location (the mean of the Gaussian) from a heat map works better than obtaining a point estimate by direct regression. To this end, some embodiments calculate a point estimate of the feature point location (e.g., the mean of the Gaussian) from a heat map, while calculating the uncertainty of the feature point location (e.g., the shape and extension of the probability distribution) via direct regression.

[0016] Additionally or alternatively, some embodiments are based on the recognition that the loss function used to train such a neural network with probabilistic outputs should capture this probabilistic nature. To this end, some embodiments train the neural network to provide a mapping from an input image to a probability distribution that maximizes the likelihood of the true feature point locations. For example, the negative log-likelihood can be used as a loss function to train the neural network.

[0017] In this way, outputting a parameter distribution (e.g., a Gaussian distribution) for each landmark position allows the neural network to be trained using a loss function that matches the output distribution. For example, the Gaussian log-likelihood loss function enables the simultaneous estimation of landmark positions and their uncertainties. This joint estimation produces a good prediction of the uncertainty, but this also produces state-of-the-art face alignment results. In this way, a dual-objective neural network with probabilistic outputs allows the simultaneous estimation of landmark positions and their uncertainties, as well as improving the accuracy of landmark localization.

[0018] Additionally or alternatively, some embodiments aim to provide a feature point detection system for safety-critical applications (e.g., driver monitoring). Together with feature point position detection, uncertainty estimation provides additional support for performing tasks related to safety-critical applications (e.g., driver monitoring). In some embodiments, a neural network is trained to process an image of an object having a structured set of feature points to generate a parameter probability distribution over the position of each feature point in the image.

[0019] In some embodiments, the neural network is a densely connected U-net (DU-Net) architecture to which two additional branch components are added from one or more U-nets (also referred to as hourglass networks or hourglasses). In some embodiments, new components are added from all hourglasses; in other embodiments, new components are added only from the final hourglass. The first additional component is a mean estimator that calculates estimated feature point positions in a differentiable manner, wherein the estimated position of each feature point is calculated as the weighted spatial mean of the positive elements of the corresponding heat map. The second component is a Cholesky estimator network (CEN), which estimates the Cholesky coefficient of the covariance matrix of the 2D Gaussian probability distribution of each feature point position. An estimate of the Gaussian log-likelihood loss (GLL) enables estimation of feature point positions and uncertainties associated with the feature point positions, wherein the estimated feature point positions are used as the mean of the Gaussian distribution and the estimated covariance matrix is ​​used as the covariance of the Gaussian distribution. This estimate is referred to as uncertainty with Gaussian log-likelihood (UGLLI), which produces both uncertainty estimates and feature point position estimates.

[0020] For example, the choice of method for calculating the mean and covariance matrix can be a distinguishing factor. Some embodiments are based on the recognition that obtaining point estimates of feature point locations from a heat map produces a more accurate estimate of the feature point locations. However, some embodiments are based on the recognition that other parameters of the probability distribution are more accurate when they are obtained by direct regression rather than from a heat map. Therefore, in one embodiment, the probability distribution of feature point locations (e.g., the mean) is obtained from a heat map, while the shape and spread (e.g., the covariance matrix) of the feature point probability distribution is obtained by direct regression.

[0021] Additionally or alternatively, some embodiments are based on the recognition that, because the dual-objective neural network outputs parameters of a probability distribution, the probabilistic feature point position estimation can be combined with other probabilistic filters to further improve the accuracy of feature point positioning. For example, in some embodiments, the parameter probability distribution of the feature points of the current input image is a Gaussian probability distribution, and a Kalman filter is used to determine the Gaussian probability distribution, and the Kalman filter uses the Gaussian probability distribution determined for the previous input image to update the Gaussian probability distribution determined for the current input image. Kalman filtering is based on the assumption that the probability distribution involved is a Gaussian distribution, so Kalman filtering can be well suited for sequential estimation of feature point positions and uncertainties across subsequent frames of a video.

[0022] In some embodiments, the task is performed based on the estimated global uncertainty. Additionally or alternatively, in some example embodiments, the task is performed based on both the estimated feature point positions and the estimated global uncertainty. In some example embodiments, if the estimated global uncertainty about the positions of the plurality of feature points in the image is above a threshold, the controller rejects the feature points estimated from the image for a safety-critical application and subsequently does not perform the task. In some embodiments, if the estimated global uncertainty of the image is below a threshold, the task is performed.

[0023] Thus, one embodiment discloses a controller for performing a task based on probabilistic image-based feature point localization. The controller includes: an input interface configured to accept a set of input images of an object having a structured set of feature points; a memory configured to store a neural network trained to process images of the object having the structured set of feature points to generate a parameter probability distribution of the position of each feature point in each processed image defined by a value of a parameter; and a processor configured to submit the set of input images to the neural network to generate a value of a parameter, the value of the parameter defining a parameter probability distribution over the position of each feature point in the structured set of feature points for each input image; determine a global feature point uncertainty of the image for each input image based on the parameter probability distribution of the feature points in the input images; and perform the task based on the parameter probability distribution of the feature points in each input image and the global feature point uncertainty of each input image.

[0024] The presently disclosed embodiments will be further explained with reference to the accompanying drawings. The drawings shown are not necessarily drawn to scale, with emphasis instead generally being placed upon illustrating the principles of the presently disclosed embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] [ Figure 1A ]

[0026] Figure 1A A block diagram of a controller for performing tasks based on probabilistic image-based feature point localization in accordance with some embodiments is shown.

[0027] [ Figure 1B ]

[0028] Figure 1B Shown is a collection of input images of human faces in accordance with some embodiments.

[0029] [ Figure 1C ]

[0030] Figure 1C Shown is a collection of input images of a human body in accordance with some embodiments.

[0031] [ Figure 2 ]

[0032] Figure 2 A schematic overview of the principles of a controller for performing tasks based on probabilistic image-based feature point localization according to some embodiments is shown.

[0033] [ Figure 3 ]

[0034] Figure 3 A schematic diagram illustrating the principles of a probabilistic neural network used by some embodiments is shown.

[0035] [ Figure 4A ]

[0036] Figure 4A A schematic diagram illustrating training a neural network to estimate feature point locations and the uncertainty of the location estimates according to some embodiments.

[0037] [ Figure 4B ]

[0038] Figure 4B A scheme of an Uncertainty with Gaussian Log-Likelihood Loss (UGLLI) method for obtaining a parameter probability distribution over the location of each feature point for each input image is shown in accordance with some embodiments.

[0039] [ Figure 5 ]

[0040] Figure 5 Exemplary results of feature point position estimates and the uncertainty of the position estimates on a test image are shown.

[0041] [ Fig. 6A ]

[0042] Fig. 6Ais a table of Normalized Mean Error (NME) comparisons between different methods on the 300-W public, challenge, and full datasets (fork 1) according to some embodiments.

[0043] [ Figure 6B ]

[0044] Figure 6B is a graph comparing the NME and area under the curve (AUC) between different methods on the 300-W test (fork 2) and Menpo datasets.

[0045] [ Figure 7 ]

[0046] Figure 7 A plot of the normalized uncertainty of the position estimate relative to the normalized error of the point estimate of the feature point locations on all 300-W test images is shown (Bifurcation 2).

[0047] [ Figure 8 ]

[0048] Figure 8 is a graph showing ablation studies on the 300-W test and Menpo datasets using the UGLLI method trained on the 300-W fork 2.

[0049] [ Fig.9A ]

[0050] Fig.9A A schematic diagram illustrating the use of uncertainty feedback to improve feature point localization according to some embodiments.

[0051] [ Fig. 9B ]

[0052] Fig. 9B Exemplary results of feature point position estimation and uncertainty of feature point position estimation without and with uncertainty feedback are shown.

[0053] [ Fig.10 ]

[0054] Fig.10 A schematic diagram is shown for performing tasks based on probabilistic image-based feature point localization in a vehicle driver monitoring environment, according to some embodiments.

[0055] [ Fig.11 ]

[0056] Fig.11 A schematic diagram for incorporating probabilistic image-based feature point localization into a Bayes filter (such as a Kalman filter) for video feature point tracking is shown. DETAILED DESCRIPTION

[0057] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other examples, the apparatus and method are shown only in block diagram form to avoid obscuring the present disclosure.

[0058] As used in this specification and claims, when used in conjunction with a list of one or more components or other items, the terms "for example," "for example," and "such as," and the verbs "include," "have," "comprises," and their other verb forms are understood to be open-ended, meaning that the list should not be viewed as excluding other additional components or items. The term "based on" means based at least in part on. In addition, it should be understood that the words and terminology used herein are for descriptive purposes and should not be considered limiting. Any headings used in this specification are for convenience only and have no legal or limiting effect.

[0059] System Overview

[0060] Figure 1A A block diagram of a controller 100 for performing a task based on probabilistic image-based feature point positioning according to some embodiments is shown. For example, in some embodiments, the task is part of a vehicle driver monitoring system and may include, for example, determining the facial alignment of the vehicle driver. In these embodiments, performing the task involves performing or initiating an action process, such as reducing the speed of the vehicle and / or changing the steering of the vehicle. However, different embodiments enable different systems to be performed based on feature point positioning in an image.

[0061] The controller 100 includes an input interface 102 configured to accept a set 122 of input images of an object having a structured set of feature points. In addition, the controller 100 includes a network interface controller (NIC) 118, which is suitable for connecting the controller 100 to a network 120 via a bus 116. Through the network 120, wirelessly or wired, the controller 100 can receive the input images 122. As used herein, the structured set of feature points indicates that the input images are of the same type, such as images of human faces or human bodies.

[0062] Figure 1BA set of input images of human faces according to some embodiments is shown. As shown, the input images can have different faces with different expressions and orientations. However, the landmarks of these faces have the same structure to determine the position, shape and configuration of the mouth 138, nose 140, eyebrows 142, eyes 144, etc. In this case, the structured set of landmarks defines specific facial landmarks that exist on a typical face. The set of one or more input images can include faces of different people, but can be a time series of images of a single person's face (e.g., video frames).

[0063] Figure 1C A collection of input images of a human body according to some embodiments is shown. In this case, a structured collection of feature points defines specific joints 146 and 148 of the human body. In one embodiment, the images are independent of each other. In a different embodiment, the input image collection is a time series of images of the human body. In another embodiment, the collection contains a single image.

[0064] The controller 100 includes a memory 106 that stores instructions that can be executed by the processor 104. The processor 104 can be configured to execute the stored instructions. The processor 104 can be a single-core processor, a multi-core processor, a graphics processing unit (GPU), a computing cluster, or any number of other configurations. The memory 106 can include a random access memory (RAM), a read-only memory (ROM), a flash memory, or any other suitable memory system. The processor 104 is connected to one or more input and output devices via a bus 116. In addition, the controller 100 includes a storage device 110, which is configured to store the neural network 112 and other components such as the probabilistic feature point estimator 114 and the task executor 136. The storage device 110 can be implemented using a hard drive, an optical drive, a flash drive, a drive array, or any combination thereof. In some embodiments, the neural network 112 is trained to process images of objects of a type having a structured set of feature points to generate a parameter probability distribution defined by a parameter value at the location of each feature point in each processed image. In the case where the type of object is a face, the neural network 112 is trained to process images of different faces, such as Figure 1B In the case where the object is a human body, the neural network is trained to process images of different human bodies, such as Figure 1C Human body image shown.

[0065] The processor 104 is configured to submit a set of input images to the neural network 112 to generate values ​​of parameters of a parameter probability distribution on the position of each feature point in a structured set of feature points defining each input image. The parameter probability distribution of each feature point includes an estimate of the position of the feature point, so that the position of the feature point and the uncertainty of the position estimate of the feature point are derived from the parameters of the parameter probability distribution. In some embodiments, the parameter probability distribution of the feature point is a Gaussian probability distribution, and the estimated position of the feature point is the mean of the Gaussian probability distribution. The uncertainty of the position estimate is determined by the covariance matrix of the Gaussian probability distribution. The mean of the Gaussian probability distribution is obtained from one or more heat maps, and the parameters of the covariance matrix are determined by direct regression. To this end, some embodiments are based on the recognition that the neural network 112 is further trained to provide a mapping from the input image to the Gaussian probability distribution, which maximizes the possibility of the true feature point position. In some embodiments, the true feature point position refers to the actual or exact position of the feature point. In some embodiments, the true feature point position refers to the position of the feature point indicated by the human labeler.

[0066] The processor 104 executes the probabilistic feature point estimator 114 to determine the parameter probability distribution at each feature point position using the neural network 112, and determine the global feature point uncertainty of the image for each input image based on the parameter probability distribution of the feature points in the input image. For example, in some implementations, the determined uncertainties of the feature point estimates in the input image are combined to generate the global feature point uncertainty of the input image. In addition, the processor 104 is configured to run the task executor 136 to perform tasks based on the global feature point uncertainty of each input image.

[0067] In some embodiments, the execution of the task by the processor 104 is based on an estimate of the location of the feature points in each image and the global feature point uncertainty of each input image. In some embodiments, the threshold of the task executor may be predefined. In some embodiments, the threshold is task-specific, which means that the threshold may vary from one task to another. If the global feature point uncertainty of the input image is below the threshold, the task execution involves taking a primary course of action, and if the global feature point uncertainty of the input image is above the threshold, a secondary course of action is taken.

[0068] In some implementations, the controller 100 includes a display interface 124 adapted to connect the controller 100 to a display device 126 , where the display device 126 may include, among other things, a computer monitor, a television, a projector, or a mobile display device.

[0069] Additionally or alternatively, the controller 100 may include an imaging interface 128 adapted to connect the controller 100 to an imaging device 130. The imaging device 130 may include a video camera, a computer, a mobile device, a webcam, or any combination thereof.

[0070] Additionally or alternatively, in some embodiments, the controller 100 is connected to an application interface 132 via the bus 116, and the application interface 132 is suitable for connecting the controller 100 to an application device 134 that can operate based on the results of the controller 100. For example, in one embodiment, the application device 134 is a car navigation system that uses the estimated feature point positions to obtain the driver's vital signs, and uses the global feature point uncertainty to verify whether the estimated feature point positions and the vital signs obtained thereby can be trusted, and the car navigation system performs tasks including deciding how to control (e.g., steer) the car. For example, in one embodiment, the application device 134 is a driver monitoring system that uses feature point positioning with uncertainty to determine and verify the driver's vital signs to determine when the driver can drive safely, for example, whether the driver is drowsy, or whether the driver is incapacitated.

[0071] Figure 2 A schematic overview of the principles of a controller for performing tasks based on probabilistic image-based feature point localization according to some embodiments is shown. An input image 200 having a structured set of feature points is input to a neural network 202. The input image 200 may include objects, such as faces and / or bodies. In some embodiments, a set of input images, such as a sequence of video frames, may be input to the neural network 202. The neural network 202 outputs an estimate 204 of the location of each feature point and an uncertainty 206 of the estimate of the location of each feature point, such as the shape and scale of a probability distribution. The output parameters, namely the estimate 204 of the location of each feature point and the uncertainty 206 of the location of each feature point, define a parametric probability distribution 208 over the location of each feature point.

[0072] In some embodiments, the parameter probability distribution 208 over the position of each feature point is a 2D Gaussian distribution. For a 2D input image, the Gaussian distribution is defined by a 2×2 covariance matrix as the mean of the 2D vectors defining the positions in the image space and as a positive definite (or at least positive semidefinite) matrix whose diagonal elements are positive or at least non-negative. In such embodiments, the mean of the Gaussian distribution can be used as a point estimate of the position of each feature point. In some other embodiments, the median position (calculated separately in the x-direction and the y-direction) or the position where the probability distribution reaches its maximum value can be used as a point estimate of the position of each feature point. The covariance matrix can be used to determine the uncertainty of the position of each feature point.

[0073] Additionally or alternatively, in some embodiments, the estimated probability distribution at the location of each feature point can be parameterized in a manner where there is no single point estimate of the location of the feature point. For example, the probability distribution at the location of each feature point can be a mixture of two Gaussians, which will be parameterized by two means and two covariance matrices.

[0074] For the input image 200, a global feature point uncertainty 210 is determined based on a parameter probability distribution over the location of each feature point 208 in the input image 200. The global feature point uncertainty 210 can be a single scalar measure that summarizes the uncertainty of the estimated location of all individual feature points in the input image 200. Based on the global feature point uncertainty 210, task execution 212 is performed. In some embodiments, the task execution is based on both the parameter probability distribution over the location of each feature point 208 and the global feature point certainty 210. Additionally or alternatively, in some embodiments, the task execution is based on the estimated location of each feature point and the global feature point uncertainty.

[0075] In some embodiments, for each input image, the controller 100 may compare the determined global feature point uncertainty with a threshold. The threshold may be predefined. If the global feature point uncertainty of the input image is less than the threshold, the controller 100 performs the main action process. However, if the global feature point uncertainty of the input image is greater than the threshold, the controller 100 performs an auxiliary action process that may be different from the main action process.

[0076] Additionally or alternatively, in some embodiments, global feature point uncertainty may be calculated for a subset of feature points. For example, if the object is a face, global feature point uncertainty may be calculated based on the uncertainty of the positions of feature points that determine the positions of the eyes. This may be advantageous, for example, for applications that rely on knowing the precise position of the eyes (e.g., in order to perform subsequent gaze direction estimation).

[0077] Figure 3 A schematic diagram illustrating the principles of a probabilistic neural network used by some embodiments is shown. A neural network 302 processes an input image 300 to output a parameter probability distribution 304 defined by parameter values ​​for the location of each feature point in each processed image. In the case where the probability distribution 304 is a Gaussian distribution, the parameters of such a distribution include a mean μ 306 and a covariance matrix Σ 308. Because images are multi-dimensional (e.g., two-dimensional), the parameters of the parameter probability distribution 304 are also multi-dimensional.

[0078] The different parameters of the parameter probability distribution 304 for each feature point define 310 point estimates of the location of the feature point and the uncertainty of the location estimate of each feature point. Figure 3 As shown, the center of the ellipse 312 indicates the estimated position of the feature point, while the size and shape of the ellipse 312 indicates the degree of uncertainty in each direction. For example, the larger the ellipse, the greater the uncertainty. In the case of a Gaussian probability distribution, the ellipse can represent an equiprobable contour of the parameter probability distribution.

[0079] The neural network 302 may be trained using back propagation following a gradient-based optimization algorithm utilizing the chain rule. The main feature of back propagation is its iterative, recursive, and efficient method for calculating updates to the weights to improve the network until it is able to perform the task for which it is being trained. However, different implementations vary between the layer structure of the neural network, the loss function, and the optimization techniques to improve the accuracy of feature point localization.

[0080] Figure 4A A schematic diagram of training a neural network to estimate feature point locations and uncertainty in the location estimates according to some embodiments is shown. Since the estimation of feature point locations is performed on a face image, it is referred to as face alignment. An input RGB image of a face is applied to a pre-trained densely connected U-Net (DU-Net) 400. Each U-Net is referred to as an hourglass. Two components 402 are added to each hourglass. The first component is used to calculate the estimated position of each feature point as a weighted spatial average (e.g., mean) 404 of the positive elements of the heat map corresponding to the feature point, where the value of the heat map at each pixel position determines the weight applied to the pixel position in the spatial average. The second component is used to estimate the covariance matrix 406 of the 2D Gaussian probability distribution of each feature point location. The second component emerges from the bottleneck layer of each hourglass. For each hourglass i and each feature point j, the feature point position estimate is given by μ ij Given, the covariance matrix is ​​given by Σ ij In addition, the feature point position estimate μ ij and the covariance matrix Σ ij The loss function is tied together 408, which enables end-to-end optimization of the entire face alignment and uncertainty estimation system.

[0081] In some embodiments, the method for training a neural network is the Uncertainty Under Gaussian Log-Likelihood Loss (UGLLI) method, which is a neural network-based method. In some embodiments, the UGLLI method is also referred to as a UGLLI model. The UGLLI method produces more accurate results. This will refer to FIG. 6A to FIG. 8Explained in detail in the specification. In some embodiments, UGLLI is used to obtain the parameter probability distribution and global feature point uncertainty of each feature point position for each image. In such an embodiment, the two components are the mean estimator and the Cholesky estimator network (CEN). The mean estimator corresponds to the first component, and the Cholesky estimator network corresponds to the second component. In some other embodiments, the feature point position estimation μ is used to obtain the parameter probability distribution and global feature point uncertainty of each feature point position for each image. In such an embodiment, the two components are the mean estimator and the Cholesky estimator network (CEN). The mean estimator corresponds to the first component, and the Cholesky estimator network corresponds to the second component. ij and the covariance matrix Σ ij The loss function tied together is the Gaussian log-likelihood (GLL) loss function Unlike the non-parametric kernel density network model, the uncertainty estimation method disclosed here is parametric, that is, the parameters of a single Gaussian distribution are directly estimated. In addition, some embodiments are based on the recognition that the Gaussian covariance matrix is ​​not constrained to be diagonal. In some embodiments, other parametric models can be used to estimate the probability distribution of each feature point in the image, such as a Laplace distribution, a multivariate exponential distribution, or a probability mixture of a small number of 2D multivariate Gaussians (a mixture of Gaussians). In addition, in some embodiments, the UGLLI method is used for feature point localization problems, such as face alignment, body posture estimation, etc.

[0082] Figure 4B The UGLLI method for obtaining the parameter probability distribution of the location of each feature point for each input image according to some embodiments is shown. The input RGB image 410 of a face is input into a pre-trained neural network whose architecture is a densely connected U-net (DU-Net) 412. Each U-net is called an hourglass. In other embodiments, the neural network architecture is a cascade (i.e., sequence) of hourglass networks, called a stacked hourglass. The UGLLI method uses a heat map H ij 442, 443 and 444 are used to estimate the position of each feature point (which is the mean μ of the Gaussian parameter probability distribution ij 480), but not for estimating the covariance matrix Σ of the Gaussian parameter probability distribution ij 470 represents the uncertainty of the position of each feature point.

[0083] In existing heatmap-based methods for feature point regression (e.g., face alignment and body pose estimation), each estimated feature point location is obtained by taking the argmax of the corresponding heatmap (the pixel location where the heatmap reaches its maximum value). However, argmax is not differentiable (thus, existing networks for feature point regression cannot be trained in an end-to-end manner that generally produces excellent results), and the feature point locations are quantized because the feature point locations are limited to the grid points of the underlying grid of the heatmap, resulting in suboptimal accuracy of the feature point location estimates. Therefore, the UGLLI method uses a mean estimator for the heatmap instead of the argmax of the heatmap because the mean function is differentiable and achieves sub-pixel accuracy. In some embodiments, a function σ is applied to the heatmap before calculating the mean. In some embodiments, the function σ is a rectified linear unit (ReLU) function, σ(X) = max(X, 0), which is restricted to taking the mean on only the positive elements of the heatmap, while ignoring the negative elements of the heatmap. Alternatively or additionally, in some embodiments, a different differentiable function or sub-differentiable function σ can be applied before taking the mean. For example, σ can be a softmax function or a temperature-controlled softmax function, which can be used to concentrate the mean primarily at locations where the heatmap takes its maximum value.

[0084] For example, in one implementation, the neural network includes a stack 412 of hourglass subnetworks 422, 423, and 424. Each hourglass outputs a heat map H of each feature point from the last, largest feature map layer 432, 433, and 434 of the hourglass. ij 442, 443 and 444. The heat map is an intensity image indicating the location of the feature point, and the weighted mean of the locations of the positive values ​​of the heat map is used to obtain the mean 442, 443 and 444 of the parameter probability distribution of the corresponding feature point. In addition, the hourglass sub-networks 422, 423 and 424 have a second output from the bottleneck, minimum feature layer 452, 453 and 454. The eigenvalues ​​of the bottleneck layer are processed by a sub-network called a Cholesky estimator network (CEN) 462, 463 and 464 to produce a lower triangular matrix L ij 468, of which, from L ij The product of its transpose 469 obtains the covariance matrix Σ of the parameter probability distribution of each feature point ij 470.

[0085] For example, consider H ij (x, y), which represents the heat map value at pixel position (x, y) of the heat map of the jth feature point from the i-th hourglass. The position estimate μ of the feature point ij =[μ ijx ,μ ijy ] T 480 is obtained by applying the function σ to the heat map H ijThe pixels are post-processed and then the weighted spatial mean of the post-processing results is taken. Various functions are considered for the function σ, such as the rectified linear unit (ReLU) function, the softmax function, the temperature-controlled softmax function, etc. The ReLU function eliminates the negative values ​​of the heat map. The softmax function modifies the mean estimator to a soft version of the argmax of the heat map. Depending on the temperature setting, the temperature-controlled softmax function provides a continuum from the hard argmax to the uniformly distributed softmax function. In some embodiments, it is observed that selecting the function σ to be the ReLU function produces the simplest and most accurate mean estimator. Estimating the feature point position from the positive heat map by taking the spatial mean can be considered as the maximum likelihood estimate (MLE) of the mean of the 2D Gaussian distribution. The mean of the 2D Gaussian distribution is sampled on a regular grid, where the heat map value represents the sampling frequency at each grid position.

[0086] Some embodiments are based on the realization that computing the covariance matrix directly from the heatmap produces inaccurate results, because many images contain some feature points that can be located very accurately, and the uncertainty of these feature point positions is very small (fraction of a pixel) in at least one direction. The uncertainty of each feature point position is represented as having a covariance matrix Σ ij 470 (2×2 symmetric positive definite matrix) Gaussian distribution. Covariance matrix Σ ij 470 has three degrees of freedom captured by its Cholesky decomposition, i.e., the lower triangular matrix L ij 468, making in, Indication L ij The transpose of .

[0087] To estimate the lower triangular matrix L ij 468, a Cholesky estimator network (CEN) is added from the bottleneck of each hourglass. CEN is a fully connected linear layer whose input is the bottleneck of the hourglass, which can have dimensions of, for example, 128×4×4=2048, and whose output is N p ×3-dimensional vector, where N p is the number of feature points in the structured set of feature points. For example, if the number of feature points is 68, then the output of CEN has 68×3=204 dimensions. If the lower triangular matrix L ij Including the diagonal elements, the lower triangular matrix L ij is the Cholesky decomposition of the covariance matrix. To ensure that the lower triangular matrix L ij 468 includes the diagonal elements, and the exponential linear unit (ELU) activation function is applied to the lower triangular matrix L ijIn some embodiments, the ELU function with a constant is used to ensure that the lower triangular matrix L ij Includes diagonal elements.

[0088] Given the estimated Gaussian distribution of the feature points of the input image at each hourglass i, the feature point j at image position p j The probability of being is given by:

[0089]

[0090] in Matrix Σ ij The determinant is marked as |Σ ij |,μ ij is the estimated position of the feature point.

[0091] Therefore, for each feature point in each input image, the neural network output is given by μ ij and L ij Parameterized Gaussian distribution.

[0092] The neural network is trained to provide a mapping from the input image to a Gaussian distribution such that the likelihood of the true feature point location (over all feature points and all training images) is as large as possible. Maximizing the likelihood (1) is equivalent to minimizing the negative log likelihood. Therefore, the negative log likelihood is used as the loss function. For feature point j, the loss function at each hourglass i is can be written as the sum of two terms T1 and T2:

[0093]

[0094] Where T1 is used as a regularization term to ensure that the Gaussian uncertainty distribution does not become too large, and T2 is the true feature point position p j In some embodiments, the actual feature point position p in the training image is j is the location of feature point j in the training image indicated by the human expert labeler. If the covariance matrix Σ ij is the unit matrix, then equation (2) is reduced to the estimated feature point position μ ij and the actual feature point position p j The standard l2 distance (i.e. Euclidean distance) between them.

[0095] By p ) to obtain a single hourglass objective function, and the total loss for each input image is is the weighted sum of the losses of each hourglass i and is given by:

[0096]

[0097] in,

[0098] In some embodiments, at test time, only the estimated mean and covariance matrix for each feature point are derived from the final hourglass K424.

[0099] Figure 5 5 shows exemplary results of feature point position estimation and the uncertainty of the position estimation on a test image. In images 500, 502, and 504, white dots represent true feature point positions and black dots represent estimated feature point positions. The ellipse represents the estimated uncertainty of the estimated position of the feature point, i.e., the position p with a Mahalanobis distance equal to 1 based on the estimated mean and covariance matrix parameters of the Gaussian probability distribution defining the feature point j. j 502 ). Larger ellipses (e.g., the ellipse over the hand in image 502 ) represent larger uncertainties, and similarly, smaller ellipses (e.g., the ellipse over the nose in image 500 ) represent smaller uncertainties. It is observed that feature points that are occluded (e.g., by the hand in image 502 ) tend to have larger uncertainties, indicating that the estimate of the uncertainty of the feature point position correctly determines that the estimated position of the occluded feature point is less uncertain.

[0100] Fig. 6A is a graph comparing the Normalized Mean Error (NME) between different methods on the 300-W public, challenge, and full datasets (fork 1). The face alignment results or landmark location estimates for the training / test set pairs referred to as 300-W fork 1 are summarized in Fig. 6A , and are reported as the NME averaged over all images in the test set on the 300W common subset, the challenge subset, and the full dataset. Fig. 6A As can be observed from the results in the graph of , the UGLLI method disclosed in this article outperforms some other recent methods in face alignment. In addition to producing the most accurate point estimate of the landmark location (the mean of the Gaussian parameter probability distribution output by the UGLLI method), the UGLLI method provides an estimate of the uncertainty of each estimated landmark location in the form of the covariance matrix of the Gaussian parameter probability distribution. Fig. 6A None of the other methods in provide an estimate of the uncertainty in their feature point position estimates.

[0101] Figure 6B Figure 2 is a graph comparing the normalized mean error (NME) and area under the curve (AUC) between different methods on the 300-W test (fork 2) and Menpo datasets. Figure 6BThe face alignment results or landmark location estimates for the training / test set pair called 300-W split 2 are summarized in the graph of . In addition, Figure 6B The graph shows the results of the model (trained on fork 2) on the Menpo dataset (6679 frontal training images with 68 feature points). Figure 6B From the results in Figure 2, it is clear that the UGLLI method is significantly better than other methods.

[0102] A scalar quantity is determined to measure the uncertainty of the position estimate of each feature point in the image. In addition, the values ​​of such uncertainty for the position estimates of all feature points in the image are combined to obtain a single scalar global feature point uncertainty value for the entire image. In some embodiments, the uncertainty of the position estimate of each feature point is obtained from the covariance matrix of a Gaussian probability distribution over the feature point positions.

[0103] For example, the uncertainty of the position estimate of the feature point can be the average of the eigenvalues ​​of the covariance matrix. In some embodiments, the nth root of the determinant of the covariance matrix (e.g., the square root or fourth root of the determinant) is used as the uncertainty of the position estimate of the feature point. In one embodiment, the fourth root K of the determinant of the covariance matrix estimated from the final hourglass is used as the scalar uncertainty of the position estimate of each feature point, that is, |Σ Kj | 1 / 4 .

[0104] Figure 7 A plot of the normalized uncertainty of the position estimate relative to the normalized error of the point estimate for the feature point locations on all 300-W test images is shown (Bifurcation 2). In plot 700, each point represents a feature point in one image, and the y-coordinate of each point represents a normalized version of the uncertainty of the position estimate for that feature point. In plot 702, each point represents the average of all feature points in one image, and the y-coordinate of each point represents the global feature point uncertainty for the image. Here, the global feature point uncertainty for the image is calculated as the average of the normalized uncertainty of the position estimate over all feature points in the image. The Pearson correlation coefficient for each plot written on top of each plot shows that our predicted uncertainty is highly correlated with the actual error.

[0105] Observe that the correlation coefficient in plot 702 is greater than that of plot 700. In other words, while the uncertainty of the feature point position for each feature point in the image is a good predictor of the error in the estimated position of that feature point, the global feature point uncertainty for the image is a better predictor (of the average feature point location error for the image). Furthermore, the global feature point uncertainty for the image has a significantly higher correlation coefficient. Therefore, in some embodiments, the global feature point uncertainty for the image is applied as an input to the controller for task execution, rather than using the uncertainty of the feature point position for each individual feature point.

[0106] Figure 8 is a graph showing ablation studies on the 300-W test and Menpo datasets using the UGLLI method trained on the 300-W fork 2. Figure 8 The graphs show that computing the loss function only on the last hourglass (HG) during training causes the network to perform less efficiently and less accurately than computing the loss on all hourglasses during training, probably because it improves the vanishing gradient problem. In addition, the Gaussian log-likelihood (GLL) loss of the UGLLI method outperforms the mean squared error (MSE) loss on the feature point locations (equivalent to setting the estimated covariance matrix to the identity matrix, Σ ij =I). For the mean estimator, it is inefficient to use direct regression (directly) to obtain the mean from the bottleneck layer of each hourglass (instead of using a heat map). This may be because neural networks can produce unreliable predictions of continuous real values. In addition to the ReLU function, two other functions of σ are compared, namely soft argmax (s-amax) and temperature-scaled soft argmax (τ-s-amax). The results of temperature-scaled soft argmax and ReLU are basically tied, but temperature-scaled soft argmax is more complicated and requires adjustment of the temperature parameter. Therefore, in some embodiments, the ReLU function is selected for the UGLLI method.

[0107] Fig.9AA schematic diagram of using uncertainty feedback to improve feature point localization according to some embodiments is shown. The UGLLI model is applied to a collection of images 900. Based on the output of the UGLLI model, the global feature point uncertainty of each image in the image collection is estimated 902. Based on the estimated global feature point uncertainty of each image in the image collection, images whose global feature point uncertainty is higher than a threshold are selected 904. In some embodiments, the threshold is predefined. The selected images (i.e., images whose global feature point uncertainty is higher than the threshold) are images whose estimated feature point positions are estimated to be the most uncertain, and therefore these images are images where feature point localization needs the most improvement. In order to improve the feature point estimation of the selected images, the UGLLI model is applied to each selected image multiple times, each time using a different random perturbation 906. For example, in some embodiments, the random perturbation 906 is applied to a face box used to initialize the algorithm. In some embodiments, the face box determined by an automatic face detector such as the Viola-Jones face detection algorithm is used to initially scale, translate, rotate, or otherwise transform the image of the object (e.g., via a 2D affine transformation) for input into a neural network. Applying random perturbations to one or more of the position, scale, rotation angle, and aspect ratio of the face box results in slightly different initializations of the method, which also results in slightly different outputs of the method. In particular, some initializations result in more accurately estimated feature point locations.

[0108] For each selected image, multiple perturbed versions of the image are fed through the UGLLI model, and the global feature point uncertainty is calculated for each perturbed version of the image. The global feature point uncertainties of all perturbed versions of the image are compared 908, and then the result is selected 910 from the perturbation that produces the smallest global feature point uncertainty. It is observed that the feature point position estimates of these final results have a smaller normalized mean error (NME) than the original feature point position estimates of the method. In other words, the average error of the feature point position estimates is therefore reduced due to uncertainty feedback. Additionally or alternatively, different types of random perturbations can be applied. For example, the feature values ​​at certain layers of the neural network can be randomly perturbed.

[0109] Fig. 9B Exemplary results of feature point position estimation and uncertainty of feature point position estimation without and with uncertainty feedback are shown. Fig. 9B In the figure, image 912 is the result of feature point position estimation without uncertainty feedback and the uncertainty of feature point position estimation. Image 914 is the result of feature point position estimation using uncertainty feedback and the uncertainty of feature point position estimation. The average NME (%) and the prediction uncertainty (UCA) values ​​on the feature points in each of images 912 and 914 are shown at the top of the images, respectively. Fig. 9BFrom the results in , it is obvious that the results with uncertainty feedback have smaller errors compared to the initial results (i.e., the results without uncertainty feedback).

[0110] Fig.10 A schematic diagram for performing tasks according to probabilistic image-based feature point positioning in a vehicle driver monitoring environment according to some embodiments is shown. In the vehicle driver monitoring environment 1000, a vehicle driver 1002 is driving a vehicle 1004, and the vital signs (one or more of facial alignment and body posture alignment) of the vehicle driver 1002 need to be monitored. In some example embodiments, a video tracking method is applied to facial tracking or human posture tracking. In this case, video-based feature point tracking can be the final task, or video-based feature point tracking can be an intermediate task whose results are used for subsequent downstream tasks. Here, video facial alignment (i.e., facial tracking) is considered for ease of describing the embodiments. However, in some embodiments, other video tracking applications are also considered, including video human posture estimation (i.e., human posture tracking).

[0111] The camera 1006 is configured to capture video and convert the captured video into a time sequence of frames (i.e., images). The controller 1008 is configured to perform probabilistic image-based feature point localization (e.g., UGLLI method) on the images in the time sequence of images. The image-based feature point localization is initialized by a face frame 1010 given by an automatic face detector such as the Viola-Jones face detection algorithm. The face frame 1010 covers the entire face of the vehicle driver 1002. In some embodiments, the face frame covers a significant portion of the entire face of the vehicle driver 1002, such as a portion of the face containing feature points.

[0112] For the first image in the sequence (first frame of the video), the image-based feature point localization is initialized using the first face box given by the automatic face detector. The image-based feature point localization outputs feature point estimates and global uncertainty estimates for the first image.

[0113] Based on these outputs, the controller 1008 determines the quality of the feature point localization and task performance. In some embodiments, if the estimated global feature point uncertainty of the first image is below a threshold, the controller 1008 determines that the quality of the feature point localization in the first image is good and performs a main action process. In some embodiments, the main action process includes initializing image-based feature point localization for a second image of the sequence using a second face box determined by the estimated feature point locations from the first frame.

[0114] In some embodiments, if the estimated global feature point uncertainty of the first image is below a threshold, the controller 1008 determines that the quality of the feature point localization in the first image is poor and performs an auxiliary action process. In some embodiments, the auxiliary action process includes initializing the image-based feature point localization of the second image using a second face frame given by an automatic face detector.

[0115] In some implementations, an image-based landmark estimator is used for facial landmark tracking using a Bayesian filter (eg, a Kalman filter).

[0116] Fig.11 A schematic diagram of using a Bayesian filter (e.g., a Kalman filter) for probabilistic feature point tracking in combination with image-based parametric probabilistic feature point localization is shown. In some embodiments, image-based feature point localization such as the UGLLI method provides a Gaussian probability distribution 1112 for each feature point in each input image k 1118 in the image sequence. The Gaussian probability distribution 1112 of each feature point in each input image in the image sequence is used as a noise measurement obtained from the input image 1118, and the Gaussian probability distribution 1112 is composed of a mean μ k 1116 and the covariance matrix Σ k 1114 parameterization. In addition, a Kalman filter is used to combine the Gaussian probability distributions 1112 of each feature point over the sequence of time steps to obtain a final Gaussian probability distribution 1110 over the location of the feature point in image k 1118. The Gaussian probability distribution 1110 is obtained by calculating the point estimate of the feature point location (i.e., the mean x k 1108) and the estimated uncertainty of the feature point position (which is the covariance matrix P k 1106). This probability distribution 1110 is referred to as the Gaussian distribution output by the Kalman filter for each image k in the image sequence. For each image in the image sequence, the output estimate of the Kalman filter is used as the parameter probability distribution of the task execution of the controller.

[0117] In some embodiments, the controller is further configured to update the Gaussian probability distribution determined for the second image in the image sequence based on the Kalman filter using the Gaussian probability distribution determined for the first image in the image sequence. Then, the output estimate of the Kalman filter for each input image (each time step) is used for subsequent images (subsequent time steps) as the prior knowledge of the state (i.e., the prior probability distribution), and the output mean 1108 and covariance matrix 1106 from each time step are used in the subsequent time step as the mean 1104 and covariance matrix 1102 of the prior distribution 1100. For several reasons, it is advantageous to use filtering (e.g., Kalman filtering) over time. First, it can improve the accuracy of feature point location and uncertainty estimates. Second, it can appropriately use the uncertainty of the estimate of each image, so that the estimate 1112 with larger uncertainty will have less impact on the output feature point estimate, while the estimate 1112 with smaller uncertainty will have more impact on the output feature point estimate.

[0118] The above description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the above description of the exemplary embodiments will provide a description for enabling one or more exemplary embodiments to those skilled in the art. It is expected that various changes can be made to the function and arrangement of the elements without departing from the spirit and scope of the disclosed subject matter set forth in the appended claims.

[0119] Specific details are given in the above description to provide a comprehensive understanding of the embodiments. However, if it is understood by those of ordinary skill in the art, the embodiments can be practiced without these specific details. For example, the systems, processes and other elements in the disclosed subject matter can be shown as components in block diagram form so as not to obscure the embodiments with unnecessary details. In other examples, well-known processes, structures and technologies can be shown without unnecessary details to avoid obscuring the embodiments. In addition, the same reference numerals and numbers in the various drawings indicate the same elements.

[0120] Moreover, various embodiments may be described as processes depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow diagram may describe an operation as a sequential process, many operations may be performed in parallel or simultaneously. In addition, the order of the operations may be rearranged. A process may terminate when its operations are completed, but may have additional steps not discussed or included in the figure. In addition, not all operations in any particularly described process may appear in all embodiments. A process may correspond to a method, function, process, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the function may correspond to the function returning to the calling function or main function.

[0121] In addition, embodiments of the disclosed subject matter may be implemented at least in part manually or automatically. Manual or automatic implementations may be performed or at least assisted by the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program codes or code segments for performing the necessary tasks may be stored in a machine-readable medium. A processor may perform the necessary tasks.

[0122] The various methods or processes outlined herein may be encoded as software that can be executed on one or more processors using any of a variety of operating systems or platforms. In addition, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may also be compiled into executable machine language code or intermediate code that is executed on a framework or virtual machine. Typically, in various embodiments, the functionality of the program modules may be combined or distributed as desired.

[0123] Embodiments of the present disclosure may be implemented as a method, examples of which have been provided. The actions performed as part of the method may be ordered in any suitable manner. Thus, embodiments may be constructed in which the actions are performed in a different order than shown, which may include performing some actions simultaneously, although shown as sequential actions in illustrative embodiments.

[0124] Although the present disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adaptations and modifications may be made within the spirit and scope of the present disclosure. Therefore, the aspects of the appended claims cover all such changes and modifications that fall within the true spirit and scope of the present disclosure.

Claims

1. A controller for performing a task based on probabilistic image-based feature point localization, the controller comprising: an input interface configured to accept an input image set of an object type having a structured feature point set; a memory configured to store a neural network trained to process images of the type of object having the structured set of feature points to produce a parameter probability distribution defined by a value of a parameter at the location of each feature point in each processed image; as well as A processor configured to: 1) submit the set of input images to the neural network to generate values ​​of the parameters of the parameter probability distribution at the location of each feature point in the structured feature point set defining each input image; 2) for each input image, determine a global feature point uncertainty for the image based on the parameter probability distribution of the feature points in the input image; and 3) perform the task based on the parameter probability distribution of the feature points in each input image and the global feature point uncertainty for each input image.

2. The controller according to claim 1, wherein: Different parameters of the parameter probability distribution of each feature point define a point estimate of the position of the feature point and an uncertainty of the position estimate of the feature point.

3. The controller according to claim 2, wherein: The processor performs the task based on the point estimates of the locations of feature points in each image and the global feature point uncertainties for each input image.

4. The controller according to claim 1, wherein: In order to determine the accuracy of the feature point positioning of each input image, the processor is configured to: determining an uncertainty in a position estimate of each feature point in the input image based on a measure of uncertainty in the parameter probability distribution generated for the feature point; and For each input image, the determined estimated uncertainties of feature points in the image are combined to produce the global feature point uncertainty for the input image.

5. The controller according to claim 1, wherein: The object type is a human face, the neural network is trained to process images of different faces, and the set of structured feature points defines specific facial feature points that appear on a typical face.

6. The controller according to claim 5, wherein: The input image set is a time series of images of the human face.

7. The controller according to claim 1, wherein: The object is a human body, the neural network is trained to process images of different human bodies, and the set of structured feature points defines specific joints of the human body.

8. The controller according to claim 7, wherein: The input image set is a time series of images of the human body.

9. The controller according to claim 2, wherein: The parameter probability distribution of the feature point is a Gaussian probability distribution, wherein the point estimate of the position of the feature point is determined by the mean of the Gaussian probability distribution, and the uncertainty of the position estimate of the feature point is determined by the covariance matrix of the Gaussian probability distribution.

10. The controller according to claim 9, wherein: The mean of the Gaussian probability distribution is obtained from one or more heat maps, and the parameters determining the covariance matrix are obtained by direct regression.

11. The controller according to claim 9, wherein: The uncertainty of the position estimate of the feature point is the nth root of the determinant of the covariance matrix of the Gaussian probability distribution.

12. The controller according to claim 9, wherein: The uncertainty of the position estimate of the feature point is the average of the eigenvalues ​​of the covariance matrix of the Gaussian probability distribution.

13. The controller according to claim 9, wherein: The neural network comprises a stack of hourglass subnetworks, wherein each hourglass outputs a heatmap for each feature point from a last, largest feature map layer of the hourglass, wherein the heatmap is an intensity image indicating the location of the feature point, wherein the mean of the parameter probability distribution is a weighted mean of the locations of the positive valued locations of the heatmap, wherein each hourglass outputs features from a bottleneck layer, wherein the eigenvalues ​​of the bottleneck layer are processed by a Cholesky estimator network to produce values ​​that derive a covariance matrix for each feature point, The Cholesky estimator network is configured to estimate the Cholesky coefficient of the covariance matrix of the 2D Gaussian probability distribution of each feature point position.

14. The controller according to claim 9, wherein: The neural network is trained to provide a mapping from the input image to the Gaussian probability distribution that maximizes the likelihood of true feature point locations.

15. The controller according to claim 14, wherein: The neural network was trained using negative log-likelihood as the loss function.

16. The controller according to claim 1, wherein: The parameter probability distribution of the feature points of the current input image is a Gaussian probability distribution, and wherein the processor is configured to use a Kalman filter to obtain a final Gaussian probability distribution at the position of the feature points in the image, and the Kalman filter uses the Gaussian probability distribution determined for the previous input image to update the Gaussian probability distribution output by the Kalman filter for the current input image.

17. The controller according to claim 1, wherein: Task execution involves taking a primary action course if the uncertainty of the global feature points of the input image is below a threshold, and taking a secondary action course if the uncertainty of the global feature points of the input image is above a threshold.

18. The controller according to claim 1, wherein: The global feature point uncertainty of an input image provides uncertainty feedback, which is used to improve the feature point localization of the image.

19. The controller according to claim 18, wherein: If the uncertainty of the global feature points of the input image is higher than a threshold, the processor is configured to: Performing the feature point positioning on the input image multiple times, each time using a different random perturbation and calculating global feature point uncertainty; comparing the global feature point uncertainties; and The execution result of the feature point localization that produces the smallest global feature point uncertainty is selected.

20. The controller according to claim 1, wherein: The task is part of a vehicle driver monitoring system.

Images