Heat map based facial feature keypoint occlusion detection for occupant monitoring systems and applications

Abstract

In various examples, heat map based facial feature keypoint occlusion detection for occupant monitoring systems and applications is disclosed. Facial feature keypoints can be individually detected and evaluated one by one using a facial feature keypoint estimation model trained to generate a confidence heat map estimate of a set of facial feature keypoint locations from an image of an occupant. Based on the confidence heat map, the model can predict a facial feature keypoint location and an occlusion score for that facial feature keypoint indicating an estimate of the likelihood that the facial feature keypoint is occluded. In some embodiments, the facial feature keypoint estimation model can be further trained to perform an occlusion classification that predicts an occlusion type for the facial feature keypoint that is predicted to be occluded.

Description

Background Technology

[0001] Occupant monitoring can be used within the vehicle compartment to assess the presence, gaze, alertness, or other conditions of the driver and occupants in real time. For example, Occupant Monitoring System (OMS) sensors can be used to track the direction of the driver's gaze, head posture, or blinking (e.g., detecting drowsiness, fatigue, and / or distraction), for hand position and / or posture detection, for the presence of children and / or pets, and / or for actions combined with these features, such as, but not limited to, seatbelt reminders, seat heating, and / or smart airbag deployment. The ability to draw accurate conclusions from OMS sensor data is determined at least in part by the ability to identify and track facial feature keypoints from images of occupants. Summary of the Invention

[0002] Embodiments of this disclosure relate to heatmap-based facial feature keypoint occlusion detection for occupant monitoring systems and applications. Systems and methods are disclosed that involve detecting and tracking features (e.g., facial features) when an occupant region (e.g., an occupant's face) captured by an occupant monitoring system (OMS) sensor is occluded.

[0003] Compared to conventional systems (such as those described above), embodiments of this disclosure provide heatmap-based feature keypoint occlusion detection, wherein feature keypoints are detected and evaluated individually using a feature keypoint estimation model trained to generate a set of confidence heatmap estimates of feature keypoint locations from an image of an occupant. Based on the confidence heatmap, the model can predict the location of a feature keypoint and an occlusion score for that feature keypoint, which indicates an estimate of the probability that the feature keypoint is occluded.

[0004] In some embodiments, the eigenkey point estimation model is a facial eigenkey point estimation model, and facial eigenkey points can be located based on a version of the argmax function. The argmax function is a mathematical function that returns the index of the maximum value in a set of values. In the context of deep neural networks (DNNs), the argmax function can be used to select the most probable class or output from a set of possibilities, i.e., the class with the highest predicted probability. That is, during estimation, each eigenkey point is treated as a single two-dimensional class map (e.g., a pixel with a pixel location that can be mapped to the input image), which can also be interpreted as a heatmap. The function of the soft argmax layer is to determine the peaks in this heatmap to find the location of the eigenkey points. In addition to identifying the peaks in the individual eigenkey point heatmaps, the extent of diffusion in the heatmap carries information about how confident the model is in locating the facial eigenkey point estimation model. For example, the eigenkey point heatmap can include a 2D Gaussian distribution, where the peak probabilities of the Gaussian distribution provide the estimated location of the eigenkey points, and the standard deviation of the Gaussian distribution provides an indication of whether the eigenkey point is occluded, which can be normalized to an occlusion score for that eigenkey point.

[0005] In some embodiments, the feature keypoint estimation model can be further trained to perform occlusion classification, which predicts the type of occlusion that obscures the predicted feature keypoint. That is, the feature keypoint estimation model can infer the category of the object obscuring the feature keypoint from an image. In some embodiments, the classification may include a first category indicating that the feature keypoint is obscured by an object located between the occupant and the OMS camera (e.g., a mobile phone, sunglasses, goggles, face mask, hair, shirt collar, hat, etc.), or due to glare, reflections, and / or poor lighting conditions. The classification may include a second category indicating that the feature keypoint is self-occluded (e.g., obscured by the occupant's face itself due to the occupant's posture). Examples of self-occlusion include, but are not limited to, a nose obscured by the forehead or cheek, eyes obscured by eyebrows or eyelids, and a mouth obscured by the nose or chin. In some embodiments, the classification may include a third category indicating that the feature keypoint is obscured because it is outside the field of view of the OMS camera and is therefore invisible according to the captured image. This classification can be used by downstream systems (e.g., gaze detection, head pose detection, driver drowsiness, etc.) to determine how and / or whether to use the predicted feature keypoint locations generated by the feature keypoint estimation model based on a given captured image, using the output of the feature keypoint estimation model. When no occlusion is detected, occlusion classification can include categories indicating the absence of occlusion. By addressing the challenges of self-occlusion and object occlusion described herein, the facial keypoint estimation algorithm based on the embodiments disclosed herein is more robust and accurate, enabling its wide application in computer vision, machine learning, and human-computer interaction. Attached Figure Description

[0006] The following describes in detail, with reference to the accompanying drawings, the system and method for detecting facial feature key point occlusion based on heatmaps for occupant monitoring systems and applications, wherein: Figure 1 This is a data flow diagram of an example facial feature keypoint prediction system for a vehicle according to some embodiments of the present disclosure; Figure 2A , Figure 2B and Figure 2C These are example facial feature key point heatmaps according to some embodiments of this disclosure; Figure 3 This is a diagram illustrating an example training architecture for training a facial feature keypoint estimation model according to some embodiments of the present disclosure; Figure 4A , Figure 4B and Figure 4C This is a diagram illustrating examples of facial keypoint location data and / or occlusion classification predicted by a facial keypoint estimation model according to some embodiments of the present disclosure. Figure 5A and Figure 5B This is a diagram illustrating an example facial image automatic annotation process using a facial feature keypoint estimation model according to some embodiments of the present disclosure; Figure 6 This is a flowchart illustrating an example method for detecting occlusion of facial feature key points according to some embodiments of the present disclosure; Figure 7A These are illustrations of example autonomous vehicles according to some embodiments of the present disclosure; Figure 7B According to some embodiments of this disclosure Figure 7A Examples of camera positions and fields of view for autonomous vehicles; Figure 7C According to some embodiments of this disclosure Figure 7A A block diagram of an example system architecture for an example autonomous vehicle; Figure 7D Cloud-based servers and according to some embodiments of this disclosure Figure 7A A schematic diagram of a system for communication between autonomous vehicles; Figure 8 This is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and Figure 9 This is a block diagram of an example data center applicable to implementing some embodiments of this disclosure. Detailed Implementation

[0007] Systems and methods related to heatmap-based facial feature keypoint occlusion detection for occupant monitoring systems and applications are disclosed. Although this disclosure may relate to exemplary autonomous or semi-autonomous vehicles or machines 700 (which may be alternatively referred to herein as "vehicle 700" or "this machine 700"), examples are provided for... Figures 7A-7D The description herein is intended to be concise and not limiting. For example, the systems and methods described herein may be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more advanced driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, ships, space shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, trains, underwater vehicles, remotely controlled vehicles (e.g., drones), and / or other vehicle types. Furthermore, while this disclosure may relate to vehicle occupant monitoring systems, this is not intended to be limiting, and the systems and methods described herein may be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technological space that may utilize human feature keypoints.

[0008] This disclosure relates to vehicle occupant monitoring technology. More specifically, the systems and methods proposed in this disclosure relate to detecting and tracking facial features when an occupant's facial region captured by an occupant monitoring system (OMS) sensor is occluded.

[0009] Machine learning-based (e.g., facial) ephemeris estimation models represent a technique for detecting ephemeris points from images and videos. For example, deep neural networks (DNNs) can be trained on input images and predict the locations of ephemeris points (e.g., features of an occupant's eyes, nose, mouth, and / or jawline) with high accuracy and robustness by extracting relevant features such as edges, corners, shapes, and textures. Such models are typically trained using very large datasets of annotated facial images, from which the model learns to generalize facial ephemeris detection to new, unseen faces that vary in pose, expression, and lighting.

[0010] However, feature keypoint estimation models may be challenged to identify feature keypoints from images in which parts of an object's face are occluded, thus inhibiting the model from extracting features from those regions of the input image. As a non-limiting example, parts of an occupant's face may be occluded by: hair, glasses, a mask, facial hair, hands and fingers, shadows, reflections (e.g., reflections from other surfaces may create reflections or glare in the captured image, making it difficult to detect facial keypoints), and / or pose changes (e.g., self-occlusion, where parts of the face are occluded by other facial features, such as when the occupant turns their head).

[0011] Occlusion can lead to detection errors from the facial feature keypoint estimation model (e.g., keypoints may be falsely detected or not detected at all), inaccurate keypoint location estimation (e.g., inaccurately estimated keypoints may not accurately represent the occupant's true facial structure), and robustness issues regarding performance consistency across different driver and / or lighting conditions. Furthermore, inaccurate or unreliable data from the facial feature keypoint estimation model can impact other processes that use facial keypoint estimations from that model. For example, drowsiness detection functions may operate based on facial keypoints marked on the driver's pupils, which serve as reliable indicators of whether the driver's eyes are open or closed and / or the frequency of blinking. Obstruction of these facial keypoints can inhibit the drowsiness detection function from accurately assessing driver drowsiness, resulting in the loss of key safety features.

[0012] To address these challenges, various techniques have been proposed. One approach uses multiple views or cameras to capture an object (e.g., the object's face) from different angles, reducing the impact of occlusion. However, this approach relies on complex processing of multiple image streams, the need for precise external calibration between cameras, and the additional costs associated with the multiple cameras required, and does not address situations where a complete set of keypoints cannot still be obtained due to occlusion. For facial keypoints, such as, but not limited to, occlusion might be caused by the position of hair, masks, glasses, hands, or other elements relative to the camera's position with respect to the keypoints. Other methods attempt to evaluate images to identify when a face is at least partially occluded and assign a face-level occlusion score to the image. For example, a face-level occlusion score can be calculated, indicating the occlusion percentage (e.g., 5%, 50%, 90%, etc.) and a threshold established for using the image. Images with face-level occlusion scores below the threshold can be discarded, while images with face-level occlusion scores above the threshold can be considered acceptable for use. However, even when an image is discarded, it may still contain one or more clearly observable facial keypoints that could be used in some applications (e.g., a face occluded by a hand may still have valid pupil facial keypoints that could be used for drowsiness detection). Conversely, an accepted image with a face-level occlusion score may still contain one or more occluded facial keypoints, which could reduce the accuracy or reliability of inferences made from that image.

[0013] Other methods optimize and / or fit observable facial keypoints to a 3D head model and predict when one or more facial keypoints will become blurred due to head rotation. However, these methods remain susceptible to occlusion, which can obscure otherwise observable facial keypoints on the face exposed to the camera. This can impair the accuracy of the head model optimization and consequently reduce the accuracy of the entire facial keypoint estimation set.

[0014] Compared to these prior art techniques, embodiments of this disclosure provide heatmap-based feature keypoint occlusion detection for occupant monitoring systems and applications. More specifically, facial feature keypoints are detected and evaluated individually using a facial feature keypoint estimation model trained to generate a set of confidence heatmap estimates of facial feature keypoint locations based on an image of an occupant. Based on the confidence heatmap, the model can predict the location of the facial feature keypoint (e.g., a set of image pixel coordinates) and an occlusion score for that facial feature keypoint, indicating an estimate of the probability that the facial feature keypoint is occluded. In some embodiments, the facial feature keypoint estimation model may include a deep neural network (DNN) architecture, such as a convolutional neural network (CNN), a recurrent neural network (RNN), or other DNN-based models. As discussed herein, the facial feature keypoint estimation model can be trained using a loss function designed to optimize the prediction of keypoint occlusion scores.

[0015] In some embodiments, facial keypoint estimation models can compute facial keypoint localization based on a version of the argmax function. The argmax function is a mathematical function that returns the index of the maximum value in a set of values. In the context of a DNN, the argmax function can be used to select the most probable class or output from a set of possibilities, i.e., the class with the highest predicted probability. For example, a soft argmax layer is a layer type used in DNNs to compute a soft version of the argmax function (e.g., a smoothed approximation of the argmax function). A soft argmax layer is a differential version of the argmax function, meaning it can be used as a layer in a DNN and trained using backpropagation. In some embodiments, a soft argmax layer in a facial keypoint estimation model can be used for class activation mapping. That is, during estimation, each facial keypoint is treated as a separate two-dimensional class map (e.g., a pixel with a pixel location that can be mapped to the input image), which can also be interpreted as a heatmap. The role of the soft argmax layer is to determine the peaks in this heatmap, finding the location of the facial keypoint.

[0016] For example, given a selected facial keypoint (e.g., the outer corner of an occupant's left eye), the facial keypoint estimation model can take a facial image (which may be a cropped version of the full image captured by an OMS camera) as input and compute a heatmap with coordinates that can be mapped to the facial image. For pixel locations relatively far from a given facial keypoint, the pixel value may be zero or close to zero, indicating that the model has low confidence in the location of that pixel at the selected facial keypoint. However, for locations on the heatmap immediately adjacent to the selected facial keypoint, the facial keypoint estimation model can compute higher confidence values, with the heatmap location having the highest confidence value corresponding to the location where the facial keypoint estimation model predicts most confidently to be the selected facial keypoint. The facial keypoint estimation model can output this peak confidence location (e.g., in the x, y coordinates of the pixel location) as a prediction of where the selected facial keypoint appears in the facial image, and can output the confidence value it computed for that prediction. In some embodiments, the facial feature keypoint estimation model can output a synthetic keypoint heatmap, which includes multiple data channels for each pixel location, each data channel corresponding to a specific facial feature keypoint (e.g., the outer corner of the left eye, the inner corner of the left eye, the center of the left pupil, the outer corner of the right eye, the inner corner of the right eye, the center of the right pupil, the left corner of the mouth, the right corner of the mouth, etc.).

[0017] In embodiments of this disclosure, in addition to identifying the peaks of individual facial keypoint heatmaps, the diffusion of the heatmaps also carries information about the confidence level of the model in locating the facial keypoint estimation model. For example, the facial keypoint heatmap may include a 2D Gaussian distribution, where the peak probability of the Gaussian distribution provides the estimated location of the facial keypoint, and the standard deviation of the Gaussian distribution provides an indication of whether an occluded facial keypoint can be normalized to its occlusion score. That is, for visible and easily distinguishable facial keypoints, the distribution of the predicted facial keypoint heatmap should be relatively narrow, and the facial keypoint estimation model produces a heatmap with a fairly high peak confidence, such that a dense set of pixels corresponds to the facial keypoint, and the confidence value rapidly decreases to zero as the distance from the peak increases. This distribution will have a relatively small standard deviation. Conversely, when facial keypoints are occluded, the facial keypoint estimation model can still produce a heatmap that indicates the peak used to predict the estimated location of the facial keypoint. However, for occluded facial keypoints, the facial keypoint estimation model will not be able to detect facial keypoints from the image with any significant confidence and will rely more heavily on training to determine where specific facial keypoints should be located. Thus, the distribution of the predicted facial keypoint heatmap expands over a relatively large range of pixels and is relatively flattened by a Gaussian distribution compared to uncrowded facial keypoints. The standard deviation (or similar distribution statistics) can then be used to calculate a normalized occlusion score (e.g., a range of 0 to 1, where 0 indicates a low probability of occlusion and 1 indicates a high probability of occlusion). Therefore, the facial keypoint estimation model can generate an occlusion score for each facial keypoint in the predicted facial keypoint heatmap.

[0018] In some embodiments, the facial keypoint estimation model can be further trained to perform occlusion classification, which predicts the type of occlusion that obscures the predicted facial keypoint. For example, in some embodiments, when the occlusion score of a facial keypoint exceeds an occlusion score threshold, the facial keypoint estimation model can infer the category of the object obscuring the facial keypoint from the image. In some embodiments, the classification may include a first category indicating that the facial keypoint is obscured by an object located between the occupant and the OMS camera (e.g., a mobile phone, glasses, goggles, face mask, hair, shirt collar, hat, etc.), or due to glare, reflections, and / or poor lighting conditions. The classification may include a second category indicating that the facial keypoint is self-occluded (e.g., the facial keypoint is being obscured by the occupant's face itself due to the occupant's posture). Examples of self-occlusion include, but are not limited to, a nose obscured by the forehead or cheek, eyes obscured by eyebrows or eyelids, and a mouth obscured by the nose or chin. In some embodiments, the classification may include a third category indicating that a facial keypoint is occluded because it is located outside the field of view of the OMS camera and is therefore invisible in the captured image. This classification can be used by downstream systems (e.g., gaze detection, head pose detection, driver drowsiness, etc.) to determine how and / or whether to use the predicted facial keypoint location prediction generated by the facial keypoint estimation model from a given captured facial image, using the output of the facial keypoint estimation model. When no occlusion is detected, the occlusion classification may include a category indicating that no occlusion exists.

[0019] By addressing the challenges of self-occlusion and object occlusion described in this paper, facial keypoint estimation algorithms can become more robust and accurate, enabling their widespread application in computer vision, machine learning, and human-computer interaction.

[0020] In some embodiments, training data for a facial keypoint estimation model can be generated based on labeled ground truth images. For example, in some embodiments, facial keypoints can be labeled by human annotators on a set of facial images. The human annotators manually identify and annotate (e.g., label) facial keypoints such as eyes, nose, mouth, and facial contours (e.g., assigning different keypoint identifiers (IDs) to each based on a set of standard keypoints). The human annotators can annotate the location of each facial keypoint, and for each keypoint, further annotate using occlusion labels. For example, occlusion labels can indicate whether a facial keypoint is: 1) visible (unoccluded), 2) occluded by an object, 3) self-occluded, or 4) truncated occlusion (occluded due to being outside the boundaries of the facial image). During training, the model is fed facial images and generates a keypoint heatmap for each facial keypoint. Based on the keypoint heatmap for each facial keypoint, the model determines the location with peaks (e.g., x, y coordinates) to define the estimated location of that facial keypoint (e.g., using soft argmax). The model can compute the standard deviation of the keypoint heatmap distribution to generate an occlusion score (e.g., a score from zero to one) for that facial keypoint. The loss function compares the model-predicted location of each keypoint with the corresponding ground truth locations (e.g., annotated by a human annotator) of those keypoints and uses misalignment to compute an misalignment loss component for iteratively adjusting the loss feedback of the facial keypoint estimation model. A second component of the loss feedback can include an occlusion loss (e.g., keypoint occlusion loss) based on the error between the model's occlusion score and the occlusion classification, and an annotated occlusion label from the ground truth training data used to predict the occlusion type of the facial keypoints being occluded. The model is iteratively trained (e.g., thousands of iterations) on samples of training data, and tuned during iterations to minimize the feedback loss. In the process of reducing the loss, the model is driven to produce heatmaps that are more noticeably narrower or wider based on the predictions. In other words, for observable, unoccluded facial keypoints, the model should produce a narrow heatmap because it should be able to confidently identify the precise location of the keypoint and, similarly, be confident that it is not there. A narrow distribution produces a low standard deviation, resulting in a low occlusion score. Conversely, for occluded facial keypoints where the facial features themselves are not visible in the image, the model relies more on training to infer where it should predict the location of the facial keypoint and is therefore less confident in accurately estimating its location. Thus, the heatmap for this keypoint includes a wider distribution with lower confidence scores, leading to a higher standard deviation and consequently a higher occlusion score.

[0021] In some embodiments, occlusion classifications attached to specific facial keypoints can inform downstream systems using the data how to use the estimated facial keypoint locations. For example, OMS can apply data filtering. OMS can determine that the predicted locations of facial keypoints with high occlusion scores and occlusion classifications indicating object occlusion should be discarded (e.g., not used) or assigned low weights because of low confidence in the accuracy of the estimated locations. OMS can similarly determine that the predicted locations of facial keypoints with high occlusion scores and occlusion classifications indicating truncated occlusion should be discarded (e.g., not used) or assigned low weights because of low confidence in the accuracy of the estimated locations. Conversely, OMS can determine that the predicted locations of facial keypoints with high occlusion scores and occlusion classifications indicating self-occlusion can effectively indicate the locations of keypoints occluded by head pose, and use this data, along with unoccluded keypoints, to predict facial geometry, such as, but not limited to, head pose tracking, gaze detection and tracking, and / or conditions such as driver alertness. OMS or other systems can use combinations of keypoint locations, occlusion scores, and / or occlusion classification data for other purposes. For example, based on this data, OMS can detect when a driver is wearing infrared (IR) blocking sunglasses that obscure the observability of key points in the pupil. This detection can be used to prompt the driver to remove the sunglasses or to make them aware of safety features that may be unavailable while wearing sunglasses (e.g., gaze detection, blink detection, etc.). In some embodiments, the facial key point occlusion detection described herein can be used for applications beyond occlusion, such as for fine-grained occlusion estimation, which can be used as input for specific tasks, such as drowsy user adaptation of OMS functions to drowsiness. Facial key points marked on the pupil can serve as an indication of whether the eyes are open or closed, which is a reliable signal for detecting blinks and thus estimating whether the driver is drowsy. In some embodiments, when OMS generates baseline data (e.g., baseline blink frequency data), OMS can request the driver to remove the IR blocking sunglasses (or mask, or other items obscuring key points) for a limited time, and then allow the driver to put the sunglasses back on once the baseline is established—using the baseline data in conjunction with key point location estimation, which may have a higher occlusion score due to the sunglasses. In some embodiments, pupil occlusion can be used as a signal to determine a quantitative measurement of eye open / closed. For example, a typical perclos (percentage of eyes closed) signal sets a threshold using a measurement such as 80% eye closure, which can be derived by using pupil visibility as an alternative.

[0022] In some embodiments, for diagnostic purposes, the downstream system may use occlusion and / or keypoint location data from a facial keypoint estimation model. For example, warnings and / or other signals may be generated to indicate OMS system degradation when a threshold number of keypoints are consistently identified as occluded. OMS degradation may occur due to camera obstruction, incorrect camera placement or mounting, the presence of foreign objects applied to the camera lens, and / or improper camera positioning (e.g., knocking off its mounting structure), each of which can be detected as occlusion using the heatmap-based facial keypoint occlusion detection described herein. In some embodiments, occlusion patterns may be evaluated to detect physiological differences in humans (e.g., beards, application of facial makeup, etc.), and / or further identify occlusion types such as face masks, goggles, hats, helmets, etc.

[0023] In some embodiments, occlusion and / or keypoint location data from a facial keypoint estimation model can be used for enhanced n-point perspective (PnP)-based head pose estimation. For example, typically, a subset of facial keypoints can be used to estimate head pose by fitting a PnP-based head model. By performing keypoint-level occlusion estimation, embodiments of this disclosure allow OMS to classify and select estimated facial keypoint locations that are found to be unoccluded in order to predict more robust head poses.

[0024] In some embodiments, another application of facial keypoint estimation models involves techniques for automatically annotating facial images for training data. More specifically, facial keypoint estimation models can be applied to methods for classifying and annotating self-occluded facial keypoints from facial images. Facial keypoint datasets can include a combination of annotated ground-value 2D keypoints and 3D keypoints. In 2D keypoint datasets, when the image describes an extreme head pose (e.g., the head turned left or right), for self-occluded keypoints (e.g., on the unobservable side of the face), human annotators can assign keypoint labels to locations along the edges of the facial image; this annotation method is called the "collapsed point" technique. Models trained solely on 2D keypoint datasets do not learn to predict the locations of self-occluded keypoints on the unobservable side of the face. 3D keypoint datasets refer to datasets that use 2D projections of 3D keypoints onto self-occluded keypoints. That is, human annotators can make informed guesses about the locations of self-occluded keypoints and annotate the image by assigning a 2D projection location to that keypoint, indicating the location projected onto the opposite side of the face. When a facial keypoint estimation model is trained on training data including samples from both 2D and 3D datasets, it can generate a first set of facial keypoint locations based on training on the 2D dataset and a second set based on training on the 3D dataset. If a facial image is inferred through such a model, the corresponding unoccluded keypoint locations from the two sets of predicted keypoint locations will be aligned. However, for head poses that cause one or more facial keypoints to become self-occluded, the occluded keypoint locations from the two sets of predicted keypoint locations will be different. The occluded keypoint locations from the first set (2D) will be displayed as collapsed points, and the occluded keypoint locations from the second set (3D) will be based on educated guesses learned from training on projected 2D keypoint locations (e.g., it has learned from predicting keypoint locations on the face). Therefore, the corresponding self-occluded keypoint locations from the two sets of predicted keypoint locations will not be aligned—and may be substantially misaligned. Thus, a model trained to predict keypoint locations using two sets of keypoints can be used to automatically annotate the locations of unoccluded keypoints (those aligned between groups) and self-occluded keypoints (those substantially misaligned between groups) in a large dataset (e.g., to generate a training dataset), without requiring human annotators. Human annotators can still be used to annotate keypoint locations with truncated occlusion and / or object occlusion, which would constitute a substantially simpler annotation task that can be performed faster. Furthermore, in some embodiments, the model can apply multi-class automatic annotation of facial feature keypoints; for example, based on the detection of an object located between the occupant and the OMS camera, keypoints classified and labeled as self-occluded keypoints are further classified and labeled as having object occlusion.

[0025] refer to Figure 1 , Figure 1 This is an example data flow diagram of the process of a facial feature keypoint prediction system 100 according to some embodiments of the present disclosure. It should be understood that the above and other arrangements described herein are merely presented as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used to supplement or replace those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or combined with other components, and can be implemented in any suitable combination and location. The various functions performed by the entities described herein can be implemented by hardware, firmware, and / or software. For example, various functions can be implemented by a processor that executes instructions stored in memory. In some embodiments, a combination of... Figures 7A-7D Example of autonomous vehicles 700 Figure 8 Example computing devices 800 and / or Figure 9 The example data center 900 uses similar components, features, and / or functions to perform the systems, methods, and processes described herein.

[0026] like Figure 1 As shown, the facial landmark prediction system 100 may include a facial landmark detector 110, which can generate facial landmark prediction data 130 based on facial image data 107. The facial image data 107 may include one or more image frames capturing a human face, such as, but not limited to, a vehicle occupant. In some embodiments, the facial image data 107 may represent a facial image cropped from a larger image (e.g., cropped by the functionality of the facial landmark prediction system 100 based on a bounding box recognizing a human facial region). In some embodiments, the facial image data 107 may include one or more image frames captured by one or more optical image sensors 106. The optical image sensors 106 may include, for example, one or more occupant monitoring system (OMS) sensors 701, as described with respect to vehicle 700. In some embodiments, the facial image data 107 may be captured by the optical image sensors 106, which include cameras, such as red, green, blue (RGB), infrared (IR), and / or RGB-IR cameras. In some embodiments, facial image data 107 may include facial images captured by different optical image sensors 106 at different locations to capture facial data from different angles. In some embodiments, facial image data 107 may include image frames captured simultaneously from multiple optical image sensors 106, which are stitched together to form a synthetic image frame for input to facial feature keypoint detector 110.

[0027] like Figure 1 As shown and discussed herein, the facial keypoint detector 110 may include a facial keypoint estimation model 120. The facial keypoint estimation model 120 includes a neural network architecture, which may be implemented, for example, using one or more of a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), and / or other DNN-based model or machine learning model architecture. Based on facial image data 107, the facial keypoint estimation model 120 can generate facial keypoint prediction data 130. More specifically, using the facial keypoint estimation model 120 to detect and evaluate facial keypoints individually, the facial keypoint estimation model 120 is trained to generate a set of confidence heatmap estimates of the locations of facial keypoints based on the facial image data 107. For example, as... Figure 1 As shown, the facial feature keypoint estimation model 120 may include one or more layers performing keypoint heatmap prediction 122 to generate one or more keypoint heatmaps 124. Keypoint heatmap prediction 122 may take facial image data 107 as input and compute one or more keypoint heatmaps 124 for one or more facial feature keypoints. In some embodiments, the generated keypoint heatmaps 124 may be mapped pixel-wise to facial image frames from the facial image data 107. That is, the keypoint heatmaps 124 may include multiple data channels, each corresponding to a specific facial feature keypoint (e.g., left outer corner of eye, left inner corner of eye, left pupil center, right outer corner of eye, right inner corner of eye, right pupil center, left corner of mouth, right corner of mouth, etc.). The facial feature keypoint estimation model 120 may include a keypoint heatmap estimation evaluation 126 algorithm and / or a keypoint occlusion classification prediction 128 algorithm for processing the keypoint heatmaps 124 to generate facial feature keypoint prediction data 130 (e.g., as referenced). Figure 2A , 2B(As described in 2C). In some embodiments, one or more neural layers of the facial keypoint estimation model 120 may be used to implement keypoint heatmap estimation evaluation 126 and / or keypoint occlusion classification prediction 128. For one or more facial keypoints, the facial keypoint prediction data 130 generated by the facial keypoint estimation model 120 may include facial keypoint locations 132, facial keypoint occlusion scores 134, and / or occlusion classifications 136. In some embodiments, the facial keypoint locations 132, facial keypoint occlusion scores 134, and / or occlusion classifications 136 define a set of facial keypoint prediction data 130 corresponding to data channels of the keypoint heatmap 124 associated with a particular facial keypoint. The facial keypoint estimation model 120 is trained to potentially learn to identify which regions of a facial image should be of interest to facilitate accurate keypoint prediction (e.g., static regions). Based on training, the facial feature keypoint estimation model 120 can infer the locations of potential facial feature keypoints and output a keypoint heatmap 124 that can be aligned pixel by pixel with the facial image data 107.

[0028] Figure 2A , Figure 2B and Figure 2C This is a schematic diagram illustrating the keypoint heatmap estimation evaluation 126 used to generate facial feature keypoint locations 132 and / or corresponding occlusion scores 134. For example, Figure 2AThis is an example keypoint heatmap 124A corresponding to the data channel of the keypoint heatmap 124 for unoccluded facial feature keypoints. In this example, pixels in heatmap 124A within region 210 have pixel values ​​representing the highest confidence values, which correspond to the locations where the facial feature keypoint estimation model 120 predicts the facial feature keypoint with the highest confidence. In this text, pixels within region 210 represent clearly discernible peaks in the confidence value distribution, where the confidence value of a pixel decreases rapidly in adjacent region 212 and further in region 214, approaching zero confidence in region 216 of keypoint heatmap 124A. By applying the soft argmax function to keypoint heatmap 124A, keypoint heatmap estimation evaluation 126 can compute the predicted facial feature keypoint location 132 at heatmap pixel coordinates x1, y1 corresponding to the peak confidence location. The predicted heatmap pixel coordinates x1, y1 can be mapped to the pixel coordinates of the input facial image to determine the location 132 of facial feature keypoints within the facial image. Regarding the facial feature keypoint occlusion score 134, this value can be calculated based on the x-axis and y-axis distribution curves of the confidence values ​​of the keypoint heatmap 124A, as shown in 220 and 224. As shown in 222, the x-axis distribution curve 221 includes a well-defined peak at x=x1, and a fairly narrow spread (distribution) 223, which can be characterized as the standard deviation of the x-axis distribution curve 221. As shown in 226, the y-axis distribution curve 225 includes a well-defined peak at y=y1, and a fairly narrow spread (distribution) 227, which can be characterized as the standard deviation of the y-axis distribution curve 225. Then, based on the keypoint heatmap 124A, the predicted facial feature keypoint occlusion score 134 can be calculated as follows: Occlusion score = (For example, occlusion score = (or other statistical measures). In addition, the facial keypoint estimation model 120 can use keypoint occlusion classification prediction 128 to infer occlusion classification 136 (e.g., visible (unoccluded) keypoint classification for this particular example) based at least on facial image data 107 and / or facial keypoint occlusion scores 134.

[0029] As another example, Figure 2BThis diagram illustrates an example keypoint heatmap 124B corresponding to the data channels of keypoint heatmap 124, used for facial feature keypoints that may be at least partially occluded, at least in the vicinity of the facial feature keypoint. Pixels in heatmap 124B within region 230 have pixel values ​​representing the highest confidence value, which corresponds to the location where the facial feature keypoint estimation model 120 predicts the facial feature keypoint with the highest confidence. In this example, the area of ​​region 230 includes a lower confidence value extending over a larger area than region 210 of heatmap 124A, indicating that the facial feature keypoint estimation model 120 is less certain about the precise location of the facial feature keypoint. The confidence value of the pixels decreases in adjacent region 232 and further decreases in region 234, approaching zero confidence in region 236 of keypoint heatmap 124B. By applying the soft argmax function to the keypoint heatmap 124B, the keypoint heatmap estimation evaluation 126 can calculate the predicted facial feature keypoint location 132 for the keypoint at the heatmap pixel coordinates x2, y2 corresponding to the peak confidence position. The predicted heatmap pixel coordinates x2, y2 can be mapped to the pixel coordinates of the facial image input to determine the facial feature keypoint location 132 within the facial image. Regarding the facial feature keypoint occlusion score 134, this value can be calculated based on the x-axis and y-axis distribution curves of the confidence values ​​of the keypoint heatmap 124B, as shown in 240 and 244. As shown in 242, the x-axis distribution curve 241 includes a spherical peak at x=x2, and can be characterized as the standard deviation of the x-axis distribution curve 241. The y-axis distribution curve 245 has a wide extension (distribution) 243. As shown in 246, the y-axis distribution curve 245 includes a well-defined peak at y=y2, and a fairly narrow extension (distribution) 247, which can be characterized as the standard deviation of the y-axis distribution curve 245. Based on the keypoint heatmap 124B, the predicted facial feature keypoint occlusion score 134 can then be calculated as: Occlusion score = (For example, occlusion score = (or other statistical measures). In addition, the facial keypoint estimation model 120 can use keypoint occlusion classification prediction 128 to infer occlusion classification 136 (e.g., object occlusion and / or self-occlusion keypoint classification for this particular example) based at least on facial image data 107 and / or facial keypoint occlusion scores 134.

[0030] As another example, Figure 2CThis is an illustration of example keypoint heatmap 124C, which corresponds to the data channel of keypoint heatmap 124 for facial feature keypoints that may be more occluded than the keypoints in heatmaps 124A and 124B. In this paper, pixels in heatmap 124C within region 250 have pixel values ​​representing the highest confidence values, corresponding to the location predicted by facial feature keypoint estimation model 120 as the location of the facial feature keypoint with the highest confidence. In this example, the area of ​​region 250 includes lower confidence values ​​extending over a larger area than region 230 of heatmap 124B, indicating that facial feature keypoint estimation model 120 is less certain about the precise location of the facial feature keypoint. The confidence values ​​of pixels decrease in adjacent regions 252 and further decrease, approaching zero confidence values ​​in region 254 of keypoint heatmap 124C. By applying the soft argmax function to the keypoint heatmap 124C, the keypoint heatmap estimation evaluation 126 can calculate the predicted facial feature keypoint location 132 for the keypoint at the heatmap pixel coordinates x3, y3 corresponding to the peak confidence position. The predicted heatmap pixel coordinates x3, y3 can be mapped to the pixel coordinates of the face image input to determine the facial feature keypoint location 132 within the face image. In this example, heatmap 124C indicates that the facial feature keypoint estimation model 120 may have a high degree of uncertainty regarding the location of the facial feature keypoint and may rely on its own training on where the facial feature keypoint should be, rather than data that can be extracted from the face image data 107. Regarding the facial feature keypoint occlusion score 134, this value can be calculated based on the x-axis and y-axis distribution curves of the confidence values ​​of the keypoint heatmap 124C, as shown in 260 and 264. As shown in 262, the x-axis distribution curve 261 includes a spherical peak at x=x3, and the standard deviation can be characterized as the x-axis distribution curve 261. The y-axis distribution curve 265 has a wide extension (distribution) 263. As shown in 266, the y-axis distribution curve 265 includes a well-defined peak at y=y3, and a fairly narrow extension (distribution) 267, which can be characterized as the standard deviation of the y-axis distribution curve 265. Then, the facial feature keypoint occlusion score 134 predicted based on the keypoint heatmap 124C can be calculated as: Occlusion score = (For example, occlusion score = (or other statistical measures). In addition, the facial keypoint estimation model 120 can use keypoint occlusion classification prediction 128 to infer occlusion classification 136 (e.g., for this particular example, object occlusion, self-occlusion keypoint classification, and / or truncated occlusion) based at least on facial image data 107 and / or facial keypoint occlusion scores 134.

[0031] Back Figure 1 Based at least in part on facial keypoint prediction data 130 (e.g., facial keypoint locations 132, facial keypoint occlusion scores 134, and / or occlusion classifications 136), the interior monitoring system 150 (which may implement one or more components of the OMS) can generate one or more outputs 154. Outputs 154 can be generated using one or more machine learning models and / or deep neural networks (DNNs) 152. For example, the interior monitoring system 150 can use facial keypoint prediction data 130 (alone or in combination with other data such as optical image data from optical image sensor 106) to predict the presence, location, posture, and / or gaze direction of occupants within the vehicle's interior space. Other systems of the vehicle 700 can determine one or more actions to be taken and / or control other tasks or operations based on the predictions. For example, based on output 154, alarms or warnings can be generated, door locks and / or windows can be operated, various functions can be opened / closed, data for digital assistants, chatbots, digital avatars, etc., can be generated, and / or air conditioning or air recirculation functions can be operated. Facial feature keypoint prediction data 130 and / or output 154 can be used for drowsy detection OMS functions, drowsy user adaptation of drowsy detection OMS functions, diagnostic functions of OMS cameras such as detecting sensor blockage and / or mispositioning, and / or other operations controlling one or more aspects of vehicle 700. For example, in some embodiments, airbag deployment, driver monitoring systems, occupant recognition, human-machine interface (HMI) applications, and / or other vehicle functions can be controlled based at least on data derived from facial feature keypoint prediction data 130.

[0032] Now for reference Figure 3This document describes an example training architecture 300 for training a facial keypoint estimation model 120 according to embodiments of the present disclosure. In this example, the facial keypoint estimation model 120 can be trained to generate facial keypoint prediction data 130 based on input to training data 305, which includes annotated data samples 306, wherein each annotated data sample 306 includes facial image data 307 (e.g., images captured from one or more optical image sensors 106 and / or synthetic facial images) and ground truth (GT) facial keypoint annotations 308. The facial keypoint annotations 308 may include annotations indicating the location of the facial keypoints relative to image frames of the facial image data 307. The facial keypoint annotations 308 may include annotations that label the keypoints with occlusion classifications (e.g., visible (unoccluded), object occlusion, self-occlusion, or truncated occlusion). The training architecture 300 uses these annotated data samples 306 to teach the facial keypoint estimation model 120 features that characterize different facial keypoints (left outer corner of the eye, left inner corner of the eye, left pupil center, right outer corner of the eye, right inner corner of the eye, right pupil center, left corner of the mouth, right corner of the mouth, etc.) as well as the relationships between facial keypoints and other features appearing in the facial image data 307. Thus, the facial keypoint estimation model 120 learns to infer various facial keypoint data, such as facial keypoint location 132, facial keypoint occlusion score 134, and / or occlusion classification 136.

[0033] like Figure 3As shown, the training architecture 300 includes a loss function 340 to generate a loss feedback 348, which is used to iteratively update the facial keypoint estimation model 120 during training, as the facial keypoint estimation model 120 processes training data 305 to produce facial keypoint prediction data 330. The loss feedback 348 can be generated by the loss function 340 to optimize keypoint predictions. During training, the facial keypoint estimation model 120 is fed annotated data samples 306 to generate keypoint heatmaps (e.g., keypoint heatmap 124) for a set of predefined facial keypoints. Based on the keypoint heatmap for each facial keypoint, the facial keypoint estimation model 120 produces facial keypoint prediction data 330, which includes facial keypoint locations 332 (such as facial keypoint locations 132 described herein), facial keypoint occlusion scores 334 (such as facial keypoint occlusion scores 334 described herein), and / or occlusion classifications 336 (such as occlusion classifications 136 described herein). Loss function 340 can compare the model-predicted location data 332 for each of the keypoints with the corresponding ground truth locations of those keypoints indicated by keypoint annotations 308. The misalignment between the model-predicted location data 332 and the location data from the ground truth annotations of keypoint annotations 308 can be used by loss function 340 to compute a misalignment loss 342 component of loss feedback 348, which is applied to the facial feature keypoint estimation model 120 during iterations of the annotated data samples 306 to optimize keypoint location predictions. A second component of the loss feedback 348 computed by loss function 340 may include occlusion loss 344 (e.g., keypoint occlusion loss). In some embodiments, occlusion loss 344 may be computed based on the model's predicted occlusion scores 334 and / or occlusion classifications 336 and their deviations from occlusion labels from one or more annotations of keypoint annotations 308. The facial keypoint estimation model 120 is iteratively trained (e.g., thousands of iterations) on annotated data samples 306 and adjusted during iterations to drive the loss feedback 348 to a minimum. In minimizing misalignment and occlusion losses, the facial keypoint estimation model 120 is driven to produce a facial keypoint heatmap with a confidence value distribution that is either significantly narrower or wider depending on the model's confidence in keypoint location prediction. As discussed in this paper, for observable, unoccluded facial keypoints, the facial keypoint estimation model 120 should produce a heatmap with a narrow distribution because the model should be able to confidently identify the precise location of the keypoint and similarly confident that the keypoint is not located.Conversely, for occluded facial keypoints, the facial keypoint estimation model 120 should produce heatmaps with a flatter and wider distribution because the model has less confidence in identifying the precise location where keypoints are present or absent.

[0034] Figure 4A , Figure 4B and Figure 4C This is an illustration of an example of facial landmark location data and / or occlusion classification predicted by a facial landmark estimation model 120 according to some embodiments. Specifically, Figure 4A , Figure 4B and Figure 4C The illustration shows an example frame of facial image data 107 where multiple facial keypoint locations predicted by the facial keypoint estimation model 120 are covered by occlusion detection data.

[0035] For example, in Figure 4A In the example frames 410 and 420 of the facial image data, each includes multiple keypoint location predictions 412 corresponding to keypoints around the contour of the occupant's right eye socket 414. Although the occupant is wearing glasses, the facial keypoint estimation model 120 still has an unobstructed view of these keypoints around the contour of the occupant's right eye socket 414, allowing the facial keypoint estimation model 120 to still infer keypoint heatmaps 124 to generate these keypoint locations and to infer visible (unobstructed) occlusion classifications of these keypoint location predictions 412 (shown as "o"). In frame 410, the occupant's eyes are open, and the facial keypoint estimation model 120 has already predicted a keypoint location prediction 416 corresponding to the occupant's right pupil. Again, although the occupant is wearing glasses, the facial keypoint estimation model 120 still has an unobstructed view of the occupant's right pupil, allowing the facial keypoint estimation model 120 to infer visible (unobstructed) occlusion classifications of the keypoint location predictions 416. Conversely, in frame 420, the occupants' eyes are closed. (As...) Figure 4A As shown, the facial keypoint estimation model 120 can still infer a keypoint heatmap 124 to generate a keypoint location prediction 422 (shown as "x") corresponding to the occupant's occluded right pupil. However, the facial keypoint estimation model 120 does not have an unoccluded view of the occupant's right pupil, and therefore infers an occlusion classification (e.g., self-occlusion and / or object occlusion) indicating that the keypoint is occluded and not visible.

[0036] In the example Figure 4BIn the example frames 430 and 440 of the facial image data, each includes multiple keypoint location predictions corresponding to keypoints surrounding the contours of the occupant's jawline, eyebrows, eyes, mouth, and nose. In these examples, the occupant's face 431 is partially occluded due to the placement of the occupant's hand 432. The facial feature keypoint estimation model 120 still has an unoccluded view of the keypoints not occluded by the hand 432 and can infer keypoint heatmaps 124 to generate keypoint location predictions 434 (shown as "o") corresponding to those keypoints, and infer visible (unoccluded) occlusion classifications of these keypoint location predictions 434. However, the facial feature keypoint estimation model 120 does not have an unoccluded view of keypoints blocked outside the sensor's field of view by the occupant's hand 432. The facial feature keypoint estimation model 120 can still infer keypoint heatmap 124 to generate keypoint location prediction 436 (shown as "x") corresponding to the occluded keypoint, and infer occlusion classification (e.g., self-occlusion and / or object occlusion) indicating that the keypoint is occluded and not visible.

[0037] In the example Figure 4C In the example frame 450 of the facial image data, multiple keypoint location predictions are included, corresponding to keypoints surrounding the contours of the occupant's jawline, eyebrows, eyes, mouth, and nose. In this example, the occupant's face 451 is partially occluded due to facial hair 452 and self-occluded due to the occupant's head pose, which blocks one or more keypoints from the image sensor's field of view. Frame 450 provides an unoccluded view of the unoccluded keypoints, allowing the facial feature keypoint estimation model 120 to infer a keypoint heatmap 124 to generate keypoint location predictions 454 (shown as "o") corresponding to those unoccluded keypoints, which have a visible (unoccluded) occlusion classification. Frame 450 does not provide an unoccluded view of keypoints blocked outside the sensor's field of view by the occupant's facial hair 452. The facial keypoint estimation model 120 can still infer keypoint heatmap 124 to generate keypoint location predictions 456 (shown as "x") corresponding to keypoints occluded by facial hair 452, and infer an occlusion classification (e.g., object occlusion) indicating that the keypoint is occluded and not visible. Frame 450 also does not provide an unoccluded view of self-occluded keypoints 458 that are blocked from the sensor due to the occupant's head pose. As described above, the facial keypoint estimation model 120 can infer keypoint heatmap 124 to generate keypoint location predictions 436 (shown as "x") corresponding to self-occluded keypoints 458, and infer an occlusion classification (e.g., self-occlusion) indicating that these keypoints 458 are occluded and not visible.

[0038] Figure 5A and Figure 5B This is a diagram illustrating an example facial image automatic annotation process 500 (e.g., for developing training data for a machine learning model) using a facial feature keypoint estimation model 505 according to embodiments described herein. In some embodiments, it can be used with... Figures 7A-7D Example of autonomous vehicles 700 Figure 8 Example computing devices 800 and / or Figure 9 The example data center 900 uses similar components, features, and / or functions to perform the systems, methods, and processes described herein for automatic annotation of facial images.

[0039] In this example, the facial feature keypoint estimation model 505 can be applied to a method for classifying and labeling self-occluded facial feature keypoints from facial image data 507. In some embodiments, facial image data 507 may include one or more image frames capturing the face of a person (such as, but not limited to, a vehicle occupant). Facial image data 507 may represent a facial image cropped from a larger image. In some embodiments, facial image data 107 may include one or more image frames captured by one or more optical image sensors 106 (e.g., one or more occupant monitoring system (OMS) sensors 701, such as those described with respect to vehicle 700).

[0040] The facial keypoint estimation model 505 may include an inference model performing 3D facial keypoint estimation 510 and an inference model performing 2D facial keypoint estimation 512. Regarding 3D facial keypoint estimation 510, model 505 can be trained using a ground truth dataset that includes 2D projections of 3D keypoints of self-occluded keypoints. That is, a human annotator can make an informed guess about the location of the self-occluded keypoint and label the image by assigning a 2D projection location to that keypoint, indicating the location projected onto the opposite side of the face. Regarding 2D facial keypoint estimation 512, model 505 can be trained using a ground truth dataset that uses a "collapsed point" technique for annotation. That is, for 2D facial keypoint estimation 512, model 505 does not learn to predict the location of self-occluded keypoints on the unobservable side of the face, but is instead trained to assign self-occluded keypoint labels to locations along the edges of the facial image. Thus, since the facial keypoint estimation model 505 is trained on training data including samples from 2D and 3D datasets, performing 3D facial keypoint estimation 510 can generate a first keypoint location set including 3D facial keypoint location data 520 (e.g., in...). Figure 5B (as further shown in 550) and a second keypoint location set including 2D keypoint location data 522 of the collapse point (as ...). Figure 5B (See 552 in the text). Figure 5B As shown, in frame 550 of 3D facial keypoint location data 520, 3D facial keypoint estimation 510 annotates the locations of unoccluded keypoints at the predicted locations of those keypoints (as shown in 560), and also annotates the locations of self-occluded keypoints based on a 2D projection of the predicted locations from the self-occluded side of the face to the visible side of the face (as shown in 562). Conversely, in frame 552 of collapsed point 2D keypoint location data 522, 2D facial keypoint estimation 512 annotates the locations of self-occluded keypoints along the edges of the object's facial image (as shown in 564). Therefore, in some embodiments, the automatic facial image annotation process 500 may include automatic facial feature occlusion annotation 530, which generates annotated facial image data 535 based on a comparison of 3D facial keypoint location data 520 and collapsed point 2D keypoint location data 522.

[0041] When a facial image from facial image data 507 is processed by facial keypoint estimation model 505 to generate 3D facial keypoint location data 520 and 2D keypoint location data 522 for collapsed points, the predicted keypoint locations can be compared to identify consistent keypoint locations between two frames (e.g., aligned) and inconsistent keypoint locations between two frames (e.g., misaligned). For example, regarding... Figure 2BIn frames 550 and 552, the corresponding predicted keypoint locations 560 are aligned between frames (e.g., within an alignment threshold) because no collapsed points are used to represent these keypoint locations in either frame. Thus, the predicted keypoint locations 560 can be annotated as unoccluded keypoints by facial feature occlusion auto-annotation 530. Conversely, the predicted keypoint locations 562 in frame 550 are substantially offset and misaligned with their corresponding predicted keypoint locations 564 in frame 552 (e.g., exceeding the alignment threshold). The predicted keypoint locations 562 in frame 550 are annotated at their estimated locations based on 2D projection, while their corresponding predicted keypoint locations 564 in frame 552 are annotated at the edges of the object's face. Based on this offset, facial feature occlusion auto-annotation 530 can output annotated facial image data 535 based on the predicted keypoint locations 560 and 562 in frames 550, where predicted keypoint locations 560 are annotated as unoccluded and predicted keypoint locations 562 are annotated as occluded. In some embodiments, the facial keypoint estimation model 505 may also function as described with respect to the facial keypoint estimation model 120 described herein. The facial keypoint estimation model 505 may apply multi-class automatic annotation of facial keypoints, for example, where keypoints classified and labeled as self-occluding keypoints are further classified and labeled as having object occlusion based on the detection of an object located between the occupant and the OMS camera. For example, for a predicted keypoint location 562 annotated as occluded, the facial keypoint estimation model 505 may generate an automatic annotation applied to the facial image data 507 (e.g., based on heatmap 124 and / or occlusion classification) indicating that the labeled location of the self-occluding keypoint 562 is also occluded due to object occlusion.

[0042] Now for reference Figure 6 , Figure 6 This is a flowchart illustrating a method 600 for detecting occlusion of facial feature key points according to some embodiments of the present disclosure. This document relates to... Figure 6 The features and elements described in method 600 can be used in combination, pairing, or substitution with elements of any other embodiment discussed herein, and vice versa. Furthermore, Figure 6 The functionality, structure, and other descriptions of the elements of the embodiments described herein can be applied to similar or like named or described elements in any of the accompanying drawings and / or embodiments described herein, and vice versa.

[0043] Each block of the method 600 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by one or more processors, including processing circuitry and the execution of instructions stored in memory. These methods can be additionally or alternatively embodied as computer-usable instructions stored on a computer storage medium. These methods can be provided by a standalone application, service, or managed service (standalone or in conjunction with another managed service) or plug-in to another product, to name just a few examples. Furthermore, method 600 is illustrated by way of example and references... Figure 1 The facial landmark prediction system 100 and / or facial landmark estimation model 120 described herein are used to describe these methods. However, these methods may be performed additionally or alternatively by any one of these systems or any combination of systems, including but not limited to those described herein.

[0044] In some embodiments, method 600 may typically be directed to generating keypoint prediction data representing one or more feature keypoints based on: at least one or more facial images of an object, at least based on a graph of keypoint location confidence values ​​inferred from one or more facial images to calculate peaks, and a statistical distribution of the graph of keypoint location confidence values.

[0045] In box B602, method 600 includes generating a map of keypoint location confidence values ​​for one or more feature keypoints based on at least one or more images of an object. The one or more facial images of the object may include facial image data 107, as referenced herein. Figure 1The facial image data 107 may include one or more image frames capturing the face of a person (such as, but not limited to, a vehicle occupant). In some embodiments, the facial image data 107 may represent a facial image cropped from a larger image (e.g., cropped by the facial keypoint prediction system 100 based on bounding boxes recognizing facial regions of a person). In some embodiments, the facial image data 107 may include one or more image frames captured by one or more optical image sensors 106. The optical image sensors 106 may include, for example, one or more occupant monitoring system (OMS) sensors 701, as described with respect to vehicle 700. In some embodiments, the facial image data 107 may be captured by the optical image sensors 106, which include cameras such as RGB, IR, and / or RGB-IR cameras. In some embodiments, the facial image data 107 may include image frames captured simultaneously from multiple optical image sensors 106, which are stitched together to form a synthetic image frame for input to the facial keypoint detector 110. In some embodiments, the graph of keypoint location confidence values ​​may include one or more heatmaps, such as keypoint heatmap 124. Facial keypoint estimation model 120 may include performing keypoint heatmap prediction 122 to generate one or more layers of one or more keypoint heatmaps. In some embodiments, the heatmap includes a heatmap where pixels represent one or more data channels, each of the one or more data channels representing a corresponding keypoint location confidence value for a facial keypoint (e.g., left outer corner of eye, left inner corner of eye, left pupil center, right outer corner of eye, right pupil center, left corner of mouth, right corner of mouth, etc.). In some embodiments, the facial keypoint estimation model may infer occlusion classification based at least on one or more facial images and a graph, wherein the occlusion classification indicates the type of occlusion that occludes one or more facial keypoints. The occlusion classification may include classifications such as 1) visible (unoccluded), 2) object occlusion, 3) self-occlusion, or 4) truncated occlusion (occluded due to being outside the boundaries of the facial image). Facial image training data (such as...) can be used. Figure 3A facial keypoint estimation model can be trained using the following method (as shown): The training data samples of the facial images include a first annotation based on the location of one or more facial keypoints and a second annotation based on occlusion indications of one or more facial keypoints. Specifically, the facial keypoint estimation model can be trained to generate facial keypoint prediction data based on input to training data 305, which includes annotated data samples 306. Each annotated data sample 306 includes facial image data 307 (e.g., images captured from one or more optical image sensors 106 and / or synthetic facial images) and ground truth (GT) facial keypoint annotations 308. Facial keypoint annotations 308 can include annotations indicating the location of facial keypoints relative to image frames of facial image data 307. Facial keypoint annotations 308 can include annotations that label keypoints with occlusion classifications (e.g., visible (unoccluded), object occlusion, self-occlusion, or truncated occlusion). The facial keypoint estimation model can be trained based on optimization of the facial keypoint estimation model using at least a keypoint location misalignment loss and an occlusion loss (e.g., keypoint occlusion loss). In minimizing misalignment and occlusion losses, the facial keypoint estimation model 120 is driven to produce a facial keypoint heatmap with a confidence value distribution that is either significantly narrower or wider depending on the model’s confidence in keypoint location prediction.

[0046] In box B604, method 600 includes determining the position of each feature keypoint relative to a map for one or more feature keypoints, based at least on the peak of the keypoint location confidence value. Based on the keypoint heatmap for each facial feature keypoint, the method can determine the location with the peak (e.g., x, y coordinates) to define the estimated location of the facial feature keypoint (e.g., using soft argmax), as per [the context of the image / image]. Figure 2A , 2B As illustrated and discussed in Figure 2C, by applying the soft argmax function to the keypoint heatmap 124A, the facial feature keypoint estimation model 120 can calculate the predicted facial feature keypoint locations 132 at the heatmap pixel coordinates x1, y1 corresponding to the peak confidence location. The predicted heatmap pixel coordinates x1, y1 can be mapped to the pixel coordinates of the facial image input to determine the facial feature keypoint locations 132 within the facial image.

[0047] In box B606, method 600 includes calculating an occlusion score for at least one individual feature keypoint among one or more feature keypoints, based at least on a statistical distribution of keypoint location confidence values. The facial feature keypoint heatmap may include a distribution (e.g., a 2D Gaussian distribution), where the peak probability of the Gaussian distribution provides the estimated location of the facial feature keypoints, and the standard deviation of the Gaussian distribution provides an indication of whether an occluded facial feature keypoint can be normalized to an occlusion score for that facial feature keypoint (e.g., as referenced). Figure 2A , 2B (As illustrated and discussed in 2C). The facial feature keypoint occlusion score predicted based on the keypoint heatmap can be used as the x-axis standard deviation. and y-axis standard deviation Use a function to calculate (e.g., occlusion score = (or other statistical measures). Furthermore, facial landmark estimation models can infer occlusion classification based at least on facial image data and / or facial landmark occlusion scores.

[0048] In box B608, method 600 includes generating keypoint predictions based at least on the positions of individual feature keypoints and occlusion scores for at least one individual feature keypoint. The method can associate the positions of individual facial feature keypoints relative to a map with the positions of one or more keypoints relative to one or more facial images of an object to generate keypoint prediction data output. That is, the predicted heatmap pixel coordinates x1, y1 can be mapped to the pixel coordinates of the facial image input to determine the positions 132 of facial feature keypoints within the facial image.

[0049] Such as about Figure 5A and 5BIn some embodiments discussed, a machine learning model trained on at least a two-dimensional facial keypoint dataset and a three-dimensional facial keypoint dataset including one or more two-dimensional projections of self-occluded facial keypoints is used to generate a heatmap representing keypoint location confidence values. That is, when facial images from facial image data are processed by a facial keypoint estimation model to produce 3D facial keypoint location data and 2D keypoint location data for collapsed points, the predicted keypoint locations can be compared to identify consistent keypoint locations between two frames (e.g., aligned) and inconsistent keypoint locations between two frames (e.g., misaligned). Occluded keypoint locations can be determined (and labeled) based on the identification that predicted keypoint locations in a first frame are substantially offset and misaligned (e.g., exceeding an alignment threshold) compared to their corresponding predicted keypoint locations in a second frame. In some embodiments, the method may include applying one or more labels to automatically annotate one or more facial images of an object based on one or more inferences from one or more facial images indicating that at least one individual facial keypoint is a self-occluded keypoint with object occlusion.

[0050] In box B610, method 600 includes at least one or more operations for controlling the vehicle based on predicting incoming calls using key points. For example, as referenced... Figure 1Based at least in part on facial keypoint prediction data 130 (which may include facial keypoint locations 132, facial keypoint occlusion scores 134, and / or occlusion classifications 136), the interior monitoring system 150 may generate one or more outputs 154. Outputs 154 may be generated using one or more machine learning models and / or deep neural networks (DNNs) 152. The interior monitoring system 150 may use facial keypoint prediction data (alone or in combination with other data such as optical image data from optical image sensor 106) to predict the presence, location, posture, and / or gaze direction of occupants within the vehicle interior space. Other systems of the vehicle 700 may determine one or more actions to be taken and / or control other tasks or operations based on the predictions. For example, based on output 154, alarms or warnings may be generated, door locks and / or windows may be operated, various functions may be opened / closed, data for digital assistants, chatbots, digital avatars, etc., may be generated, and / or air conditioning or air recirculation functions may be operated. Facial feature key point prediction data 130 and / or output 154 can be used for drowsy detection OMS functions, drowsy user adaptation of drowsy detection OMS functions, diagnostic functions of OMS cameras such as detecting sensor blockage and / or mispositioning, and / or other operations controlling one or more aspects of vehicle 700. For example, in some embodiments, airbag deployment, driver monitoring systems, occupant recognition, HMI applications, and / or other vehicle functions can be controlled based at least on data derived from facial feature key point prediction data 130.

[0051] In some embodiments, the systems and methods described herein can be executed within or in combination with a simulated environment (e.g., NVIDIA's DriveSIM) using simulated data (e.g., simulated sensor data from simulated sensors of virtual or simulated machines). For example, simulated sensor data can be used, including applying realistic facial keypoint prediction data generated from the simulated environment, and the simulation can use the facial keypoint prediction data to perform operations associated with virtual machines in the environment (e.g., navigation, vehicle safety features, etc.). These simulated operations can be used to test the performance of the underlying algorithms, systems, and / or processes before deploying them to the real world. In some cases, simulations can be used to generate synthetic training data, e.g., training data including regions of interest and / or subregions of interest within the simulation. The synthetic training data (supplementing or replacing real-world data) can then be processed for various OMS operations, such as determining the gaze direction and / or drowsiness of the driver and / or other occupants. In any example, such as when a simulated environment is used for testing, validation, training, etc., one or more optical transport algorithms (such as ray tracing and / or path tracing algorithms) can be used to render or generate the simulated environment and / or associated training data. In some embodiments, the simulated environment and / or one or more of its objects, features, or components may be generated or managed within a 3D content collaboration platform (e.g., NVIDIA's Omniverse) for industrial digitization, generative physical artificial intelligence (AI), and / or other use cases, applications, or services. For example, the content collaboration platform or system may include systems for managing objects, features, scenes, etc., using or developing generic scene descriptors (USD) (e.g., OpenUSD) data. In simulated environments, digital environments, etc., the platform may include realistic physical simulations, such as using NVIDIA's PhysX SDK, to simulate real physics and physical interactions with simulations hosted on the platform. The platform may integrate OpenUSD with ray tracing / path tracing / light transport simulations (e.g., NVIDIA's RTX rendering technology) into software tools and simulation workflows for building, training, deploying, or testing AI systems, such as systems for testing, validating, training (e.g., machine learning models, neural networks, etc.) and / or other tasks related to automobiles, robots, machines, or other applications.

[0052] The systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, ships, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, trains, underwater vehicles, remotely controlled vehicles (such as drones), and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, as examples and not limitations, for machine control, machine mobility, machine actuation, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital pairing, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, generative artificial intelligence, and / or any other suitable application.

[0053] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, ship systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems combining one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems implementing one or more language models (e.g., one or more large language models (LLMs) and / or one or more visual language models (VLMs)), systems for performing optical transmission simulations, systems for performing collaborative content creation for 3D assets, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0054] Example autonomous vehicles Figure 7AThis is an illustration of an example autonomous vehicle 700 according to some embodiments of the present disclosure. The autonomous vehicle 700 (or, alternatively, referred to herein as “vehicle 700”) may include, but is not limited to, passenger vehicles such as cars, trucks, buses, ambulances, shuttles, electric or motorized bicycles, motorcycles, fire trucks, police cars, ambulances, boats, engineering vehicles, underwater vessels, robotic vehicles, drones, aircraft, vehicles coupled to trailers (e.g., semi-trailer trucks for transporting goods) and / or other types of vehicles (e.g., driverless and / or capable of accommodating one or more passengers). Autonomous vehicles are typically described according to the levels of automation defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) in its "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 700 is capable of performing one or more functions that meet Level 3 through Level 5 of autonomous driving. Vehicle 700 is capable of performing one or more functions that meet Level 1 through Level 5 of automated driving. For example, depending on the embodiment, vehicle 700 is capable of driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). The term “autonomy” as used herein may include any and / or all types of autonomy for the 700 or other machines, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, providing auxiliary autonomy, semi-autonomy, primary autonomy, or other names.

[0055] Vehicle 700 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 700 may include a propulsion system 750, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. Propulsion system 750 may be connected to the drivetrain of vehicle 700, which may include a transmission, to allow propulsion of vehicle 700. Propulsion system 750 may be controlled in response to receiving a signal from throttle / accelerator 752.

[0056] A steering system 754, which may include a steering wheel, can be used to steer the vehicle 700 (e.g., along a desired path or route) when the propulsion system 750 is operating (e.g., when the vehicle is in motion). The steering system 754 may receive signals from the steering actuator 756. For fully automatic (level 5) functions, the steering wheel may be optional.

[0057] The brake sensor system 746 can be used to operate the vehicle brakes in response to receiving signals from the brake actuator 748 and / or the brake sensor.

[0058] It may include one or more System-on-Chip (SoC) 704 (such as Figure 7C The controllers 704(A) and 704(B) and / or one or more GPUs and one or more controllers 736 can provide signals (e.g., signals indicating commands) to one or more components and / or systems of vehicle 700. For example, one or more controllers can send signals to operate vehicle brakes via one or more brake actuators 748, to operate steering system 754 via one or more steering actuators 756, and to operate propulsion system 750 via one or more throttles / accelerators 752. One or more controllers 736 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals indicating commands) to allow autonomous driving and / or assisting a human driver in driving vehicle 700. One or more controllers 736 may include a first controller 736 for autonomous driving functions, a second controller 736 for functional safety functions, a third controller 736 for artificial intelligence functions (e.g., computer vision), a fourth controller 736 for infotainment functions, a fifth controller 736 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 736 can handle two or more of the functions described above, and two or more controllers 736 can handle a single function, and / or any combination thereof.

[0059] One or more controllers 736 may provide signals for controlling one or more components and / or systems of vehicle 700 in response to sensor data (e.g., sensor input) received from one or more sensors. Sensor data, such as, but not limited to, facial image data 107, may be received from, for example, but not limited to, Global Navigation Satellite System (“GNSS”) sensor 758 (e.g., Global Positioning System sensor), radar (RADAR) sensor 760, ultrasonic sensor 762, lidar (LIDAR) sensor 764, inertial measurement unit (IMU) sensor 766 (e.g., accelerometer, gyroscope, magnetic compass, magnetometer, etc.), microphone 796, stereo camera 768, wide-angle camera 770 (e.g., fisheye camera), infrared camera 772, surround camera 774 (e.g., 360-degree camera), long-range and / or medium-range camera 798, speed sensor 744 (e.g., for measuring the rate of vehicle 700), vibration sensor 742, steering sensor 740, braking sensor (e.g., as part of braking sensor system 546), one or more occupant monitoring system (OMS) sensors 701 (e.g., one or more interior cameras) and / or other sensor types. One or more controllers 736 may provide signals for controlling one or more components and / or systems of vehicle 700 in response to facial keypoint prediction data 130 generated by facial keypoint estimation model 120 as described herein.

[0060] One or more of the controllers 736 may receive inputs (e.g., represented by input data) from the instrument cluster 732 of the vehicle 700 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 734, an auditory signaling device, a speaker, and / or via other components of the vehicle 700. These outputs may include information such as vehicle speed, rate, time, map data (e.g., ...). Figure 7C Information such as high-definition (HD) maps 722, location data (e.g., the location of vehicle 700 on the map), orientation, and the location of other vehicles (e.g., occupying grids), as well as information about objects and their states perceived by the controller 736. For example, the HMI display 734 may display information about the existence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, leaving 34B in two miles, etc.).

[0061] The vehicle 700 further includes a network interface 724, which can communicate via one or more networks using one or more wireless antennas 726 and / or a modem. For example, the network interface 724 may be able to communicate via Long Term Evolution (LTE), Wideband Code Division Multiple Access (WCDMA), Universal Mobile Telecommunications System (UMTS), Global System for Mobile Communications (GSM), IMT-CDMA Multicarrier (CDMA2000), etc. One or more wireless antennas 726 may also use one or more local area networks such as Bluetooth, Bluetooth Low Energy (LE), Z-Wave, ZigBee, etc., and / or one or more low-power wide area networks (LPWANs) such as LoRaWAN, SigFox, etc., to allow communication between objects in the environment (e.g., vehicles, mobile devices, etc.).

[0062] Figure 7B For use in accordance with some embodiments of this disclosure Figure 7A This is an example of the camera position and field of view of an example autonomous vehicle 700. The camera and its respective field of view are an example embodiment and are not intended to be limiting. For example, additional and / or replaceable cameras may be included, and / or these cameras may be located at different positions on the vehicle 700.

[0063] The camera type used for the camera may include, but is not limited to, a digital camera suitable for use with components and / or systems of vehicle 700. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The camera may be able to use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a sharp-pixel camera, such as a camera with RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to improve light sensitivity. In some embodiments, one or more optical image sensors 106 may include reference... Figure 7A and 7B One or more of the described sensors and / or cameras.

[0064] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more of the cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0065] One or more of the cameras can be mounted in mounting components such as custom-designed (3D-printed) components to cut off stray light and reflections from inside the vehicle (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capabilities. Regarding the wing mirror mounting components, the wing mirror components can be custom-3D-printed so that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side-view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0066] Cameras with a field of view that includes the environment in front of the vehicle 700 (e.g., a front-facing camera) can be used for surround view to help identify forward paths and obstacles, and, with the assistance of one or more controllers 736 and / or control SoCs, to provide information crucial for generating an occupancy grid and / or determining the preferred vehicle path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including Lane Departure Warning (LDW), Autonomous Cruise Control (ACC), and / or other functions such as traffic sign recognition.

[0067] A variety of cameras can be used in front-facing configurations, including monocular camera platforms such as complementary metal-oxide-semiconductor (CMOS) color imagers. Another example could be a wide-angle camera 770, which can be used to perceive objects entering the field of view from the periphery (such as pedestrians, traffic at intersections, or bicycles). Although Figure 7B The middle image shows only one wide-angle camera, but any number (including zero) of wide-angle cameras 770 can exist on vehicle 700. Furthermore, any number of remote cameras 798 (e.g., long-view stereo camera pairs) can be used for depth-based object detection, especially for objects for which neural networks have not yet been trained. Remote cameras 798 can also be used for object detection and classification, as well as basic object tracking.

[0068] Any number of stereo cameras 768 can also be included in the front-mounted configuration. In at least one embodiment, one or more stereo cameras 768 may include an integrated control unit that includes a scalable processing unit that can provide a multi-core microprocessor and programmable logic (FPGA) with an integrated controller area network (CAN) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternative stereo cameras 768 may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip that can measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 768 may be used in addition to those described herein or alternatively.

[0069] Cameras with a field of view including the side portion of the vehicle 700 (e.g., side-view cameras) can be used for surround view, providing information for creating and updating occupancy grids and generating side-impact collision warnings. For example, surround camera 774 (e.g., ... Figure 7B The four surround cameras 774 shown can be mounted on the vehicle 700. The surround cameras 774 can include a wide-angle camera 770, a fisheye camera, a 360-degree camera, and / or the like. Four examples are provided; the four fisheye cameras can be positioned at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 774 (e.g., left, right, and rear) and can utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround-view camera.

[0070] A camera with a field of view that includes the environment behind the vehicle 700 (e.g., a rear-view camera) can be used for parking assistance, surround view, rear collision warning, and creating and updating occupancy grids. A wide variety of cameras can be used, including but not limited to those also suitable as front-facing cameras as described herein (e.g., long-range and / or mid-range camera 798, stereo camera 768, infrared camera 772, etc.).

[0071] Cameras (e.g., one or more OMS sensors 701) that cover a portion of the interior environment of the vehicle 700's passenger compartment can be used as part of an Occupant Monitoring System (OMS), such as, but not limited to, a Driver Monitoring System (DMS). For example, the OMS sensors (e.g., OMS sensor 701) can (e.g., by controller 736) track the gaze direction, head posture, and / or blinking of occupants and / or drivers. This gaze information can be used to determine the level of attention of the occupant or driver (e.g., detect drowsiness, fatigue, and / or distraction), and / or to take responsive actions to prevent harm to the occupant or operator. In some embodiments, data from the OMS sensors can be used to allow gaze control operations triggered by the driver and / or non-driver passengers, such as, but not limited to, adjusting the passenger compartment temperature and / or airflow, opening and closing windows, controlling passenger compartment lighting, controlling the entertainment system, adjusting rearview mirrors, adjusting seat positions, and / or other operations. In some embodiments, the OMS can be used for applications such as determining when objects and / or passengers remain in the passenger compartment (e.g., by detecting the presence of passengers after the driver has left the vehicle). In some embodiments, based on facial feature keypoint prediction data 130, OMS can track the occupant's and / or driver's gaze direction, head posture and / or blinking, or perform other functions.

[0072] Figure 7C For use in accordance with some embodiments of this disclosure Figure 7A The example autonomous vehicle 700 is illustrated in the block diagram of an example system architecture. It should be understood that this arrangement, and other arrangements described herein, are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities, which may be implemented as discrete or distributed components or in combination with other components, and in any suitable combination and location. The various functions described herein as being performed by these entities can be implemented via hardware, firmware, and / or software. For example, the various functions can be implemented by a processor executing instructions stored in memory.

[0073] Figure 7CEach component, feature, and system in vehicle 700 is illustrated as being connected via bus 702. Bus 702 may include a Controller Area Network (CAN) data interface (or, alternatively, referred to herein as the "CAN bus"). CAN may be a network within vehicle 700 used to assist in the control of various features and functions of vehicle 700, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, etc. The CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0074] Although bus 702 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet may be used in addition to or alternatively to a CAN bus. Furthermore, although bus 702 is represented by a single line, this is not intended to be limiting. For example, any number of buses 702 may exist, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 702 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 702 may be used for collision avoidance functions, and a second bus 702 may be used for drive control. In any example, each bus 702 may communicate with any component of vehicle 700, and two or more buses 702 may communicate with the same component. In some examples, each SoC 704, each controller 736, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors of vehicle 700) and may be connected to a common bus such as a CAN bus.

[0075] Vehicle 700 may include one or more controllers 736, such as those described herein. Figure 7A The controllers described herein. Controller 736 can be used for a wide variety of functions. Controller 736 can be coupled to any other different components and systems of vehicle 700 and can be used for the control of vehicle 700, artificial intelligence of vehicle 700, infotainment and / or the like for vehicle 700.

[0076] Vehicle 700 may include one or more System-on-Chip (SoC) 704. SoC 704 may include a CPU 706, GPU 708, processor 710, cache 712, accelerator 714, data storage 716, and / or other components and features not shown. SoC 704 can be used to control vehicle 700 across a wide variety of platforms and systems. For example, one or more SoCs 704 may be combined with an HD map 722 in a system (e.g., the system of vehicle 700), the HD map being transmitted via a network interface 724 from one or more servers (e.g., [server name missing]). Figure 7D One or more servers (778) receive map refresh and / or updates.

[0077] The CPU 706 may include CPU clusters or CPU complexes (or, alternatively, referred to herein as "CCPLEX"). The CPU 706 may include multiple cores and / or L2 cache. For example, in some embodiments, the CPU 706 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 706 may include four dual-core clusters, each with a dedicated L2 cache (e.g., 2MB L2 cache). The CPU 706 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, allowing any combination of clusters of the CPU 706 to be active at any given time.

[0078] The CPU 706 can implement power management capabilities including one or more of the following features: automatic clock gating of hardware blocks when idle to conserve dynamic power; clock gating of each core when the core is not actively executing instructions due to the execution of WFI / WFE instructions; independent power gating of each core; independent clock gating of each core cluster when all cores are clock-gated or power-gated; and / or independent power gating of each core cluster when all cores are power-gated. The CPU 706 can further implement enhanced algorithms for managing power states, where allowed power states and desired wake-up times are specified, and the hardware / microcode determines the optimal power state to enter for the core, cluster, and CCPLEX. The processing core can support simplified power state entry sequences in software, with this work offloaded to the microcode.

[0079] The GPU 708 may include an integrated GPU (or, alternatively, referred to herein as an "iGPU"). The GPU 708 may be programmable and efficient for parallel workloads. In some examples, the GPU 708 may use an enhanced tensor instruction set. The GPU 708 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with at least 96KB of storage capacity), and two or more of these streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage capacity). In some embodiments, the GPU 708 may include at least eight streaming microprocessors. The GPU 708 may use a computation application programming interface (API). Furthermore, the GPU 708 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0080] In automotive and embedded applications, the GPU 708 can be power-optimized for optimal performance. For example, the GPU 708 can be fabricated on FinFETs. However, this is not intended to be limiting, and the GPU 708 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can combine several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, dispatch units, and / or a 64KB register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to leverage the mixture of computation and addressing computations for efficient execution of workloads. The streaming microprocessor can include independent thread scheduling capabilities to allow for finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors can include a combination of L1 data cache and shared memory units to improve performance while simplifying programming.

[0081] The GPU 708 may include, in some examples, a High Bandwidth Memory (HBM) and / or a 16GB HBM2 memory subsystem providing a peak memory bandwidth of approximately 900GB / s. In some examples, in addition to HBM memory or alternatively, Synchronous Graphics Random Access Memory (SGRAM), such as Generation 5 Graphics Double Data Rate Synchronous Random Access Memory (GDDR5), may be used.

[0082] The GPU 708 may include unified memory technology, which includes access counters to allow memory pages to be migrated more precisely to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, Address Translation Service (ATS) support can be used to allow the GPU 708 to directly access the CPU 706 page tables. In such examples, when the GPU 708 Memory Management Unit (MMU) experiences a miss, an address translation request can be transferred to the CPU 706. In response, the CPU 706 can look up the virtual-physical mapping for the address in its page tables and transfer the translation back to the GPU 708. Thus, unified memory technology can allow a single unified virtual address space for the memory of both the CPU 706 and GPU 708, simplifying GPU 708 programming and porting applications to the GPU 708.

[0083] In addition, the GPU 708 may include access counters that track how frequently the GPU 708 accesses the memory of other processors. These access counters help ensure that memory pages are moved to the physical memory of the processor that accesses those pages most frequently.

[0084] SoC 704 may include any number of caches 712, including those described herein. For example, cache 712 may include an L3 cache available to both CPU 706 and GPU 708 (e.g., it is connected to both CPU 706 and GPU 708). Cache 712 may include a write-back cache, which can track the state of rows, for example, using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache may include 4MB or more, but a smaller cache size may also be used.

[0085] SoC 704 may include one or more arithmetic logic units (ALUs) that can be used to perform processing of any of a variety of tasks or operations relating to vehicle 700, such as processing a DNN. Furthermore, SoC 704 may include a floating-point unit (FPU) or other mathematical coprocessor or digital coprocessor type for performing mathematical operations within the system. For example, SoC 704 may include one or more FPUs integrated as execution units within CPU 706 and / or GPU 708. In some embodiments, one or more functions of the facial feature keypoint estimation model 120 may be performed at least in part using CPU 706, GPU 708, and / or SoC 704.

[0086] SoC 704 may include one or more accelerators 714 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, SoC 704 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. This large on-chip memory (e.g., 4MB SRAM) can allow the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster can be used to supplement GPU 708 and offload some tasks from GPU 708 (e.g., freeing up more cycles of GPU 708 to perform other tasks). As an example, accelerator 714 can be used for targeted workloads (e.g., perceptrons, convolutional neural networks (CNNs), etc.) that are stable enough to be easily controlled for acceleration. When used herein, the term "CNN" can include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0087] Accelerator 714 (e.g., a hardware acceleration cluster) may include a Deep Learning Accelerator (DLA). The DLA may include one or more Tensor Processing Units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may be an accelerator configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and optimized for performing image processing functions. The DLA may be further optimized for a specific set of neural network types and floating-point operations as well as inference. The DLA is designed to provide higher performance per millimeter than a general-purpose GPU and significantly outperform CPUs. The TPU can perform several functions, including single-instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, and post-processor functions.

[0088] DLA can execute neural networks, especially CNNs, quickly and efficiently on processed or unprocessed data for any function across a wide variety of applications, such as, but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and recognition using data from microphones; CNNs for face recognition and vehicle owner recognition using data from camera sensors; and / or CNNs for safety and / or safety-related events.

[0089] The DLA can perform any function of the GPU 708, and by using inference accelerators, for example, a designer can make either the DLA or the GPU 708 target any function. For example, a designer can focus the CNN processing and floating-point operations on the DLA and leave other functions to the GPU 708 and / or other accelerators 714.

[0090] Accelerator 714 (e.g., a hardware acceleration cluster) may include a programmable vision accelerator (PVA), which may alternatively be referred to herein as a computer vision accelerator. The PVA can be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA may include, for example, but not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0091] RISC cores can interact with image sensors (such as the image sensor of any camera described herein), image signal processors, and / or the like. Each of these RISC cores may include any amount of memory. Depending on the embodiment, the RISC core may use any of several protocols. In some examples, the RISC core may execute a real-time operating system (RTOS). RISC cores may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0092] DMA allows PVA components to access system memory independently of the CPU 706. DMA can support any number of features used to provide optimizations to the PVA, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which can include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0093] A vector processor can be a programmable processor designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as, for example, a Single Instruction Multiple Data (SIMD) or Very Long Instruction Word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0094] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. Consequently, in some examples, each of the vector processors may be configured to execute independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even different algorithms on a sequence of images or portions of an image. Among other things, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each of these PVAs. Furthermore, the PVA may include additional error correction code (ECC) memory to enhance overall system security.

[0095] Accelerator 714 (e.g., a hardware acceleration cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for accelerator 714. In some examples, on-chip memory may include at least 4MB of SRAM consisting of, for example, but not limited to, eight field-configurable memory blocks, accessible by both PVA and DLA. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. PVA and DLA may access memory via a backbone that provides high-speed memory access to PVA and DLA. The backbone may include (e.g., using an APB) an on-chip computer vision network that interconnects PVA and DLA to memory.

[0096] On-chip computer vision networks can include interfaces that ensure both the PVA and DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such interfaces can provide separate phases and channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0097] In some examples, the SoC 704 may include, for example, a real-time ray tracing hardware accelerator as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. This real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) to generate real-time visualization simulations for RADAR signal interpretation, sound propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison with LiDAR data for localization and / or other functional purposes, and / or other uses. In some embodiments, one or more Tree Traversal Units (TTUs) may be used to perform one or more ray tracing-related operations.

[0098] Accelerators 714 (e.g., hardware accelerator clusters) have broad applications in autonomous driving. PVAs can be programmable vision accelerators used in critical processing stages of ADAS and autonomous vehicles. PVAs are well-suited to algorithmic domains requiring predictable processing, low power, and low latency. Therefore, PVAs perform well in semi-dense or dense rule computation, even on small datasets requiring predictable runtimes with low latency and low power. Thus, in the context of platforms for autonomous vehicles, PVAs are designed to run classical computer vision algorithms because they are efficient in object detection and integer arithmetic.

[0099] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. In some examples, algorithms based on semi-global matching may be used, but this is not intended to be limiting. Many applications for Level 3–5 autonomous driving require instantaneous motion estimation / stereo matching (e.g., from moving structures, pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions on input from two monocular cameras.

[0100] In some examples, PVA can be used to perform intensive optical flow, providing processed RADAR data from raw RADAR data (e.g., using 4D Fast Fourier Transform). In other examples, PVA is used for time-of-flight depth processing, which, for example, involves processing raw time-of-flight data to provide processed time-of-flight data.

[0101] DLA can be used to run any type of network to enhance control and driving safety, including, for example, neural networks that output a confidence metric for each object detection. Such a confidence value can be interpreted as a probability or as providing a relative “weight” for each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a threshold for the confidence and only consider detections exceeding the threshold as true positives. In an Automatic Emergency Braking (AEB) system, false positives can cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run a neural network to regress the confidence value. This neural network can take at least a subset of parameters as input, such as bounding box dimensions, ground plane estimates (e.g., from another subsystem), inertial measurement unit (IMU) sensor 766 outputs related to vehicle orientation and distance, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LiDAR sensor 764 or RADAR sensor 760), etc.

[0102] SoC 704 may include one or more data storage units 716 (e.g., memory). The data storage unit 716 may be on-chip memory of SoC 704, which may store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and security, the data storage unit 716 may be large enough to store multiple instances of the neural network. The data storage unit 716 may include L2 or L3 cache 712. References to the data storage unit 716 may include references to memory associated with PVA, DLA, and / or other accelerators 714 as described herein.

[0103] SoC 704 may include one or more processors 710 (e.g., embedded processors). Processor 710 may include a startup and power management processor, which may be a dedicated processor and subsystem for handling startup power and management functions, as well as safety implementation. The startup and power management processor may be part of the SoC 704 startup sequence and may provide runtime power management services. The startup power and management processor may provide clock and voltage programming, auxiliary system low-power state transitions, SoC 704 thermal and temperature sensor management, and / or SoC 704 power state management. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and SoC 704 may use the ring oscillator to detect the temperature of CPU 706, GPU 708, and / or accelerator 714. If it is determined that the temperature exceeds a threshold, the startup and power management processor may enter a temperature fault routine and place SoC 704 into a lower power state and / or place vehicle 700 into a driver-safe parking mode (e.g., safely stop vehicle 700).

[0104] The processor 710 may further include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio via multiple interfaces, as well as a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor and dedicated RAM.

[0105] The processor 710 may further include an always-on-processor engine that can provide the necessary hardware features to support low-power sensor management and wake-up use cases. This always-on-processor engine may include a processor core, tightly coupled RAM, support for peripherals such as timers and interrupt controllers, various I / O controller peripherals, and routing logic.

[0106] The processor 710 may further include a security cluster engine, which includes a dedicated processor subsystem for handling security management for automotive applications. The security cluster engine may include two or more processor cores, tightly coupled RAM, support for peripheral devices (e.g., timers, interrupt controllers, etc.), and / or routing logic. In secure mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic that detects any differences between their operations.

[0107] The processor 710 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0108] The processor 710 may further include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0109] Processor 710 may include a video image compositer, which may be (e.g., implemented on a microprocessor) a processing block, implementing video post-processing functions required by the video playback application to generate the final image for the player window. The video image compositer may perform lens distortion correction on the wide-angle camera 770, the surround camera 774, and / or the in-cabin monitoring camera sensor. The in-cabin monitoring camera sensor is preferably monitored by a neural network running on another instance of an advanced SoC, configured to recognize in-cabin events and respond accordingly. The in-cabin system may perform lip reading to activate mobile phone services and make calls, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. Some functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled in other situations.

[0110] Video image compositers can include enhanced temporal denoising for both spatial and temporal noise reduction. For example, in the case of motion in the video, denoising appropriately weights spatial information, reducing the weight of information provided by neighboring frames. In cases where the image or part of the image does not contain motion, the temporal denoising performed by the video image compositer can use information from previous images to reduce noise in the current image.

[0111] The video image compositer can also be configured to perform stereo correction on input stereo lens frames. When the operating system desktop is in use and the GPU 708 does not need to continuously render new surfaces, the video image compositer can be further used for user interface components. Even when the GPU 708 is powered on and active, performing 3D rendering, the video image compositer can be used to offload the GPU 708 to improve performance and responsiveness.

[0112] The SoC 704 may further include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functions for receiving video and input from a camera. The SoC 704 may further include an input / output controller that can be software-controlled and can be used to receive I / O signals not assigned to a specific role.

[0113] SoC 704 may further include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management and / or other devices. SoC 704 can be used to process data from cameras and sensors (e.g., LIDAR sensor 764, RADAR sensor 760, etc., which can be connected via Gigabit Multimedia Serial Link and Ethernet), data from bus 702 (e.g., vehicle 700 speed, steering wheel position, etc.), and data from GNSS sensor 758 (connected via Ethernet or CAN bus). SoC 704 may further include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine, and which can be used to free up CPU 706 from routine data management tasks.

[0114] The SoC 704 can be an end-to-end platform with a flexible architecture spanning Automation Levels 3-5, providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS technologies for diversity and redundancy, along with deep learning tools to deliver a flexible and reliable driving software stack. The SoC 704 can be faster, more reliable, and even more energy- and space-efficient than conventional systems. For example, when combined with the CPU 706, GPU 708, and data storage 716, the accelerator 714 can provide a fast and efficient platform for Level 3-5 autonomous vehicles.

[0115] Therefore, this technology offers capabilities and functionalities that cannot be achieved through conventional systems. For example, computer vision algorithms can be executed on CPUs, which can be configured using high-level programming languages ​​such as C to execute a wide variety of processing algorithms across a diverse range of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for automotive ADAS applications and practical Level 3-5 autonomous vehicles.

[0116] In contrast to conventional systems, the techniques described in this paper, by providing CPU complexes, GPU complexes, and hardware acceleration clusters, allow multiple neural networks to be executed simultaneously and / or sequentially, and the results to be combined to enable Level 3–5 autonomous driving capabilities. For example, a CNN executed on a DLA or dGPU (e.g., GPU 720) could include text and word recognition, allowing a supercomputer to read and understand traffic signs, including those for which neural networks have not yet been specifically trained. The DLA could further include a neural network capable of recognizing, interpreting, and providing semantic understanding of the signs, and passing that semantic understanding to a path planning module running on the CPU complex.

[0117] As another example, multiple neural networks can operate simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of "Caution: Flashing lights indicate icy conditions" along with a light can be interpreted independently or jointly by several neural networks. The sign itself can be recognized as a traffic sign by a deployed first neural network (e.g., a trained neural network), and the text "Flashing lights indicate icy conditions" can be interpreted by a deployed second neural network that informs the vehicle's path planning software (preferably executing on a CPU complex) that icy conditions exist when the flashing lights are detected. The flashing lights can be identified by a deployed third neural network operating across multiple frames, informing the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks can operate simultaneously, for example, within a DLA and / or on a GPU 708.

[0118] In some examples, the CNN used for facial recognition and owner identification can use data from camera sensors to identify the presence of an authorized driver and / or owner of vehicle 700. A processing engine always on the sensors can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in safe mode, to disable the vehicle when the owner leaves. In this way, SoC 704 provides security against theft and / or carjacking.

[0119] In another example, the CNN used for emergency vehicle detection and identification can use data from microphone 796 to detect and identify emergency vehicle siren. In contrast to conventional systems that use a general classifier to detect siren and manually extract features, SoC 704 uses a CNN to classify environmental and urban sounds as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative shut-off rate of emergency vehicles (e.g., by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the localized area in which the vehicle operates, as identified by GNSS sensor 758. Thus, for example, when operating in Europe, the CNN will seek to detect European siren, and when operating in the United States, the CNN will seek to identify siren only in North America. Once an emergency vehicle is detected, with the assistance of ultrasonic sensor 762, the control program can be used to execute emergency vehicle safety routines, causing the vehicle to slow down, pull over to the side of the road, stop, and / or idle until the emergency vehicle passes.

[0120] The vehicle may include a CPU 718 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 704 via a high-speed interconnect (e.g., PCIe). The CPU 718 may include, for example, an x86 processor. The CPU 718 can be used to perform any of a wide variety of functions, including, for example, arbitrating the results of potential inconsistencies between ADAS sensors and the SoC 704, and / or monitoring the status and health of the controller 736 and / or the infotainment SoC 730.

[0121] Vehicle 700 may include a GPU 720 (e.g., a discrete GPU or dGPU) that can be coupled to SoC 704 via a high-speed interconnect (e.g., NVIDIA's NVLINK). GPU 720 may provide additional artificial intelligence capabilities, for example by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based on inputs from sensors of vehicle 700 (e.g., sensor data).

[0122] Vehicle 700 may further include a network interface 724, which may include one or more wireless antennas 726 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). Network interface 724 can be used to allow wireless connectivity via the Internet to the cloud (e.g., with server 778 and / or other network devices), with other vehicles, and / or with computing devices (e.g., a passenger's client device). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across networks and via the Internet). A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide vehicle 700 with information about vehicles approaching vehicle 700 (e.g., vehicles in front, to the side, and / or behind vehicle 700). This functionality can be part of vehicle 700's cooperative adaptive cruise control function.

[0123] Network interface 724 may include a SoC that provides modulation and demodulation functions and enables controller 736 to communicate via a wireless network. Network interface 724 may include an RF front-end for up-conversion from baseband to RF and down-conversion from RF to baseband. Frequency conversion can be performed using known processes and / or using a superheterodyne process. In some examples, the RF front-end functionality may be provided by a separate chip. The network interface may include wireless functions for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0124] Vehicle 700 may further include data storage 728, which may include off-chip (e.g., outside of SoC 704) storage devices. Data storage 728 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0125] Vehicle 700 may further include GNSS sensor 758. GNSS sensor 758 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used for assisted mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 758 can be used, including, for example, but not limited to, GPS using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0126] Vehicle 700 may further include a RADAR sensor 760. The RADAR sensor 760 can be used by vehicle 700 for remote vehicle detection even in dark and / or inclement weather conditions. The RADAR functional safety level may be ASIL B. The RADAR sensor 760 can use CAN and / or bus 702 (e.g., to transmit data generated using the RADAR sensor 760) for control and access to object tracking data, and in some examples, Ethernet access for accessing raw data. A wide variety of RADAR sensor types can be used. For example, and without limitation, the RADAR sensor 760 can be adapted for front, rear, and side RADAR use. In some examples, a pulse Doppler RADAR sensor is used.

[0127] The RADAR sensor 760 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In some examples, the long-range RADAR can be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view (e.g., within 250m) achieved through two or more independent scans. The RADAR sensor 760 can help distinguish between stationary and moving objects and can be used by ADAS systems for emergency braking assist and forward collision warning. The long-range RADAR sensor can include a single-site multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record the vehicle 700's surroundings at higher rates with minimal traffic interference from adjacent lanes. The other two antennas can extend the field of view, enabling rapid detection of vehicles entering or leaving the vehicle 700's lane.

[0128] As an example, a mid-range RADAR system can include a range of up to 760m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 750 degrees (rear). Short-range RADAR systems can include, but are not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor blind spots behind and beside the vehicle.

[0129] Short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0130] Vehicle 700 may further include ultrasonic sensors 762. Ultrasonic sensors 762, which may be positioned at the front, rear, and / or sides of vehicle 700, can be used for parking assistance and / or creating and updating occupancy grids. A wide variety of ultrasonic sensors 762 can be used, and different ultrasonic sensors 762 can be used for different detection ranges (e.g., 2.5m, 4m). Ultrasonic sensors 762 can operate at functional safety level ASIL B.

[0131] Vehicle 700 may include a LIDAR sensor 764. The LIDAR sensor 764 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensor 764 may be of functional safety level ASIL B. In some examples, vehicle 700 may include multiple LIDAR sensors 764 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0132] In some examples, the LiDAR sensor 764 may be able to provide a list of objects and their distances within a 360-degree field of view. A commercially available LiDAR sensor 764 may have an advertising range of, for example, approximately 700m, with an accuracy of 2cm-3cm, and support for 700Mbps Ethernet connectivity. In some examples, one or more non-protruding LiDAR sensors 764 may be used. In such examples, the LiDAR sensor 764 may be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of a vehicle 700. In such examples, the LiDAR sensor 764 may provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, even for low-reflectivity objects, with a range of 200m. A front-mounted LiDAR sensor 764 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0133] In some examples, LiDAR technologies such as 3D flash LiDAR can also be used. 3D flash LiDAR uses a flash of laser light as the emission source to illuminate the vehicle's surroundings up to approximately 200 meters. A flash LiDAR unit includes a receiver that records the laser pulse propagation time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LiDAR allows for the generation of highly accurate and distortion-free images of the surrounding environment using each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of the vehicle. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras (e.g., non-browsing LiDAR devices) without moving parts other than a fan. Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-registered intensity data. By using a flash LiDAR, and because a flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 764 is less susceptible to motion blur, vibration, and / or shock.

[0134] The vehicle may further include an IMU sensor 766. In some examples, the IMU sensor 766 may be located at the center of the rear axle of the vehicle 700. The IMU sensor 766 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 766 may include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 766 may include an accelerometer, a gyroscope, and a magnetometer.

[0135] In some embodiments, the IMU sensor 766 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 766 can allow the vehicle 700 to estimate heading by directly observing and correlating velocity changes from GPS to the IMU sensor 766 without input from a magnetic sensor. In some examples, the IMU sensor 766 and the GNSS sensor 758 can be combined into a single integrated unit.

[0136] The vehicle may include a microphone 796 placed in and / or around the vehicle 700. Among other things, the microphone 796 may be used for emergency vehicle detection and identification.

[0137] The vehicle may further include any number of camera types, including stereo camera 768, wide-angle camera 770, infrared camera 772, surround camera 774, long-range and / or mid-range camera 798, and / or other camera types. These cameras can be used to capture image data around the entire perimeter of the vehicle 700. The types of cameras used depend on the embodiment and the requirements of the vehicle 700, and any combination of camera types can be used to provide the necessary coverage around the vehicle 700. Furthermore, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As an example and without limitation, these cameras may support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described herein with respect to... Figure 7A and Figure 7B It was described in more detail.

[0138] Vehicle 700 may further include vibration sensor 742. Vibration sensor 742 can measure vibrations of vehicle components such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 742 are used, differences between vibrations can be used to determine friction or slippage on the road surface (e.g., when there is a vibration difference between a power drive shaft and a free-rotating shaft).

[0139] Vehicle 700 may include ADAS system 738. In some examples, ADAS system 738 may include SoC. ADAS system 738 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic warning (RCTW), collision warning system (CWS), lane centering (LC) and / or other features and functions.

[0140] The ACC system can use a RADAR sensor 760, a LIDAR sensor 764, and / or a camera. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to vehicles immediately in front of vehicle 700 and automatically adjusts the vehicle speed to maintain a safe distance. Lateral ACC performs distance holding and, if necessary, advises vehicle 700 to change lanes. Lateral ACC is associated with other ADAS applications such as LCA and CWS.

[0141] CACC uses information from other vehicles, which can be received indirectly from other vehicles via a wireless link or network connection (e.g., via the Internet) through network interface 724 and / or wireless antenna 726. Direct links can be provided by vehicle-to-vehicle (V2V) communication links, while indirect links can be infrastructure-to-vehicle (I2V) communication links. Typically, the V2V communication concept provides information about vehicles immediately ahead (e.g., vehicles immediately in front of vehicle 700 and in the same lane), while the I2V communication concept provides information about traffic further ahead. A CACC system can include either or both I2V and V2V information sources. Given information about vehicles ahead of vehicle 700, CACC can be more reliable, and it has the potential to improve traffic flow and reduce road congestion.

[0142] The Forward-Looking Warning (FCW) system is designed to alert the driver to hazards, enabling the driver to take corrective action. The FCW system uses a front-facing camera and / or RADAR sensor 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components. The FCW system can provide warnings in the form of, for example, audible, visual, haptic, and / or rapid braking pulses.

[0143] The AEB system detects an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system can use a front-facing camera and / or RADAR sensor 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes to attempt to prevent or at least mitigate the effects of the predicted collision. The AEB system may include technologies such as dynamic brake support and / or collision proximity braking.

[0144] The Lane Departure Warning (LDW) system provides visual, auditory, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle crosses lane markings. When the driver indicates intentional lane departure, the LDW system is deactivated by activating a turn signal. The LDW system can utilize a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0145] The LKA system is a variation of the LDW system. If vehicle 700 begins to leave the lane, the LKA system provides corrective steering input or braking to vehicle 700.

[0146] The BSW system detects and warns the driver of vehicles in the blind spot. The BSW system can provide visual, auditory, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses turn signals. The BSW system can utilize a rear-facing camera and / or RADAR sensor 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0147] The RCTW system can provide visual, auditory, and / or tactile notifications when an object is detected outside the range of the rear camera while the vehicle is reversing. Some RCTW systems include AEB to ensure the application of the vehicle's brakes to avoid a collision. The RCTW system may use one or more rear-mounted RADAR sensors 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0148] Conventional ADAS systems can be prone to false positives, which can be annoying and distracting for the driver, but typically not catastrophic, as ADAS systems alert the driver and allow them to determine whether a safe condition truly exists and take appropriate action. However, in an autonomous vehicle 700, in the event of conflicting results, the vehicle 700 itself must decide whether to heed the results from the main computer or auxiliary computer (e.g., the first controller 736 or the second controller 736). For example, in some embodiments, the ADAS system 738 may be a backup and / or auxiliary computer for providing perception information to a backup computer rationality module. The backup computer rationality monitor may run redundant and diverse software on hardware components to detect faults in perception and dynamic driving tasks. Outputs from the ADAS system 738 may be provided to a supervisory MCU. If the outputs from the main computer and the auxiliary computer conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0149] In some examples, the master computer can be configured to provide a confidence score to the supervisory MCU, indicating the master computer's confidence level in the selected result. If the confidence score exceeds a threshold, the supervisory MCU can follow the master computer's direction regardless of whether the auxiliary computer provides conflicting or inconsistent results. If the confidence score does not meet the threshold and the master and auxiliary computers indicate different results (e.g., conflict), the supervisory MCU can arbitrate between these computers to determine the appropriate result.

[0150] The supervisory MCU can be configured to run a neural network trained and configured to determine the conditions under which the auxiliary computer provides a false alarm based on outputs from both the host and auxiliary computers. Thus, the neural network in the supervisory MCU can learn when the output of the auxiliary computer can be trusted and when it cannot. For example, when the auxiliary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying a metallic object that is not actually dangerous, such as a drain grid or manhole cover that triggers an alarm. Similarly, when the auxiliary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU may include at least one of a DLA or GPU suitable for running the neural network using associated memory. In a preferred embodiment, the supervisory MCU may include components of and / or be included as components of the SoC 704.

[0151] In other examples, ADAS system 738 may include an auxiliary computer that performs ADAS functions using conventional computer vision rules. This allows the auxiliary computer to use classic computer vision rules (if-then), and the presence of neural networks in the supervising MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identity make the entire system more fault-tolerant, especially for failures caused by software (or software-hardware interface) functionality. For instance, if a software vulnerability or bug exists in the software running on the host computer and non-identical software code running on the auxiliary computer provides the same overall result, the supervising MCU can be more confident that the overall result is correct and that the vulnerability in the software or hardware on the host computer does not cause a substantial error.

[0152] In some examples, the output of ADAS system 738 can be fed to the perception block and / or the dynamic driving task block of the main computer. For example, if ADAS system 738 issues a forward collision warning because an object is immediately in front, the perception block can use this information when identifying the object. In other examples, the assistance computer can have its own neural network, which is trained and thus reduces the risk of false positives as described herein.

[0153] Vehicle 700 may further include an infotainment SoC 730 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 730 may include a combination of hardware and software that can be used to provide vehicle 700 with audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.) and / or information services (e.g., navigation system, rear parking assistance, radio data system, vehicle-related information such as fuel level, total coverage distance, brake fuel level, fuel level, door opening / closing, air filter information, etc.). For example, the infotainment SoC 730 may include a radio, disc player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, Wi-Fi, steering wheel audio controls, hands-free voice controls, head-up display (HUD), HMI display 734, telematics device, control panel (e.g., for controlling and / or interacting with various components, features, and / or systems) and / or other components. The infotainment SoC 730 may further be used to provide information (e.g., visual and / or auditory) to the vehicle's users, such as information from the ADAS system 738, autonomous driving information like planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0154] The infotainment SoC 730 may include GPU functionality. The infotainment SoC 730 can communicate with other devices, systems, and / or components of the vehicle 700 via bus 702 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 730 may be coupled to a supervisory MCU, allowing the GPU of the infotainment system to perform some autonomous driving functions in the event of a failure of the main controller 736 (e.g., the primary and / or backup computer of the vehicle 700). In such an example, the infotainment SoC 730 may place the vehicle 700 into a driver-safe parking mode as described herein.

[0155] Vehicle 700 may further include an instrument cluster 732 (e.g., a digital instrument panel, electronic instrument cluster, digital instrument panel, etc.). The instrument cluster 732 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 732 may include a set of instruments such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 730 and the instrument cluster 732. Therefore, the instrument cluster 732 may be included as part of the infotainment SoC 730, or vice versa.

[0156] Figure 7D For cloud-based servers and according to some embodiments of this disclosure Figure 7A This is a system diagram illustrating communication between example autonomous vehicles 700. System 776 may include server 778, network 790, and vehicles including vehicle 700. Server 778 may include multiple GPUs 784(A)-784(H) (collectively referred to herein as GPU 784), PCIe switches 782(A)-782(D) (collectively referred to herein as PCIe switch 782), and / or CPUs 780(A)-780(B) (collectively referred to herein as CPU 780). GPUs 784, CPUs 780, and PCIe switches may be interconnected with high-speed interconnects and / or PCIe connections 786, such as, but not limited to, NVLink interface 788 developed by NVIDIA. In some examples, GPUs 784 are connected via NVLink and / or NVSwitch SoCs, and GPUs 784 and PCIe switches 782 are connected via PCIe interconnects. Although eight GPUs 784, two CPUs 780, and two PCIe switches are shown in the diagram, this is not intended to be limiting. Depending on the embodiment, each of the servers 778 may include any number of GPUs 784, CPUs 780, and / or PCIe switches. For example, each of the servers 778 may include eight, sixteen, thirty-two, and / or more GPUs 784.

[0157] Server 778 can receive image data from vehicles via network 790, representing images of unexpected or changed road conditions such as recently commenced roadworks. Server 778 can also transmit neural network 792, updated neural network 792, and / or map information 794, including information about traffic and road conditions, to vehicles via network 790. Updates to map information 794 may include updates to HD map 722, such as information about construction sites, potholes, bends, floods, or other obstacles. In some examples, neural network 792, updated neural network 792, and / or map information 794 may have been generated from new training and / or data received from any number of vehicles in the environment, and / or based on experience gained from training performed at a data center (e.g., using server 778 and / or other servers).

[0158] Server 778 can be used to train machine learning models (e.g., neural networks) based on training data. In some embodiments, facial feature keypoint estimation model 120 can be trained at least partially using server 778 as described herein. Training data can be generated using a vehicle, and / or can be generated in a simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., where the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not labeled and / or preprocessed (e.g., where the neural network does not require supervised learning). Training can be performed according to any one or more classes of machine learning techniques, including but not limited to: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component analysis and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, the machine learning model can be used by the vehicle (e.g., transmitted to the vehicle via network 790), and / or the machine learning model can be used by server 778 to remotely monitor the vehicle.

[0159] In some examples, server 778 can receive data from vehicles and apply that data to state-of-the-art real-time neural networks for real-time intelligent inference. Server 778 may include a deep learning supercomputer powered by GPU 784 and / or a dedicated AI computer, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, server 778 may include a deep learning infrastructure in a data center that uses only CPU power.

[0160] The deep learning infrastructure of server 778 may be capable of rapid, real-time inference and can be used to assess and verify the health status of the processor, software, and / or associated hardware in vehicle 700. For example, the deep learning infrastructure may receive periodic updates from vehicle 700, such as image sequences and / or objects located in those image sequences by vehicle 700 (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure may run its own neural network to identify objects and compare them with objects identified by vehicle 700. If the results do not match and the infrastructure concludes that the AI ​​in vehicle 700 has malfunctioned, then server 778 may transmit a signal to vehicle 700 instructing its fail-safe computer to take control, notify passengers, and complete a safe stopping operation.

[0161] For inference, server 778 may include GPU 784 and one or more programmable inference accelerators (such as NVIDIA's TensorRT 3). The combination of GPU-powered servers and inference acceleration enables real-time response. In other examples, such as where performance is less critical, CPU, FPGA, and other processor-powered servers can be used for inference.

[0162] Example computing device Figure 8 The block diagram is provided for an example computing device 800 suitable for implementing some embodiments of the present disclosure. The computing device 800 may include an interconnect system 802 directly or indirectly coupled to the following devices: memory 804, one or more central processing units (CPUs) 806, one or more graphics processing units (GPUs) 808, a communication interface 810, input / output (I / O) ports 812, input / output components 814, a power supply 816, one or more presentation components 818 (e.g., a display), and one or more logic units 820. In at least one embodiment, the computing device 800 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 808 may include one or more vGPUs, one or more CPUs 806 may include one or more vCPUs, and / or one or more logic units 820 may include one or more virtual logic units. Therefore, computing device 800 may include discrete components (e.g., a complete GPU dedicated to computing device 800), virtual components (e.g., a portion of the GPU dedicated to computing device 800), or a combination thereof.

[0163] although Figure 8The various boxes are shown connected via an interconnect system 802 with wiring, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, a presentation component 818, such as a display device, can be considered an I / O component 814 (e.g., if the display is a touchscreen). As another example, CPU 806 and / or GPU 808 may include memory (e.g., memory 804 may represent a storage device other than the memory of GPU 808, CPU 806, and / or other components). Therefore, Figure 8 The computing devices mentioned are merely illustrative. No distinction is made between categories such as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as all of these are considered within the same category. Figure 8 Within the scope of computing devices.

[0164] Interconnect system 802 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 802 may include one or more link or bus types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Fast (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 806 may be directly connected to memory 804. Furthermore, CPU 806 may be directly connected to GPU 808. In cases where there is a direct or point-to-point connection between components, interconnect system 802 may include a PCIe link to perform that connection. In these examples, a PCI bus is not required in computing device 800.

[0165] Memory 804 may include any medium of a wide variety of computer-readable media. Computer-readable media can be any available medium that can be accessed by computing device 800. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. For example and without limitation, computer-readable media may include computer storage media and communication media.

[0166] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media, implemented in any way or by any method or technique for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 804 may store computer-readable instructions (e.g., representing programs and / or program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by computing device 800. As used herein, computer storage media does not include the signal itself.

[0167] Computer storage media may contain computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and include any information transport medium. The term "modulated data signal" can refer to a signal whose characteristics are set or altered in a manner that encodes information into that signal. For example and without limitation, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as sound, RF, infrared, and other wireless media. Any combination of the above should also be included within the scope of computer-readable media.

[0168] CPU 806 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 800 to perform one or more of the methods and / or processes described herein. Each of CPU 806 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing a large number of software threads simultaneously. CPU 806 may include any type of processor and may include different types of processors depending on the type of computing device 800 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 800, the processor may be an advanced RISC mechanism (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors such as math coprocessors, computing device 800 may also include one or more CPUs 806.

[0169] In addition to or replacing CPU 806, GPU 808 may also be configured to execute at least some computer-readable instructions to control one or more components of computing device 800 to perform one or more of the methods and / or processes described herein. One or more GPUs 808 may be integrated GPUs (e.g., having one or more CPUs 806) and / or one or more GPUs 808 may be discrete GPUs. In embodiments, one or more GPUs 808 may be coprocessors of one or more CPUs 806. Computing device 800 may use GPU 808 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPU 808 may be used for general-purpose computing on a GPU (GPGPU). GPU 808 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. GPU 808 may generate pixel data for outputting an image in response to rendering commands (e.g., rendering commands received via a host interface from CPU 806). GPU 808 may include graphics memory, such as display memory, for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 804. GPU 808 may include two or more GPUs operating in parallel (e.g., via a link). The link may connect the GPUs directly (e.g., using NVLINK) or via a switch (e.g., using NVSwitch). When combined, each GPU 808 may generate different portions of pixel data or GPGPU data for different outputs (e.g., the first GPU for the first image, the second GPU for the second image). Each GPU may include its own memory or may share memory with other GPUs.

[0170] In addition to or replacing CPU 806 and / or GPU 808, logic unit 820 may be configured to execute at least some computer-readable instructions to control one or more components of computing device 800 to perform one or more methods and / or processes described herein. In embodiments, CPU 806, GPU 808, and / or logic unit 820 may perform any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 820 may be part of and / or integrated into one or more CPUs 806 and / or one or more GPUs 808, and / or one or more logic units 820 may be discrete components of CPU 806 and / or GPU 808 or otherwise external thereto. In embodiments, one or more logic units 820 may be processors of one or more CPUs 806 and / or one or more GPUs 808. In some embodiments, one or more functions of the facial landmark detector 110 and / or facial landmark estimation model 120 described herein may be executed at least in part using the CPU 806, GPU 808, and / or logic unit 820. Examples of logic unit 820 include one or more processing cores and / or components thereof, such as data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree traversal unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU), application-specific integrated circuit (ASIC), floating-point unit (FPU), input / output (I / O) element, peripheral component interconnect (PCI) or peripheral component interconnect fast (PCIe) element, etc.

[0171] The communication interface 810 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 800 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communications. The communication interface 810 may include components and functions that enable communication via any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication via Ethernet or InfiniBand), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 820 and / or the communication interface 810 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via interconnect system 802 to one or more GPUs 808 (e.g., memory within GPU 808).

[0172] I / O port 812 allows computing device 800 to be logically coupled to other devices, including I / O component 814, presentation component 818, and / or other components, some of which may be built into (e.g., integrated into) computing device 800. Illustrative I / O component 814 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dish antennas, browsers, printers, wireless devices, and so on. I / O component 814 can provide a Natural User Interface (NUI) for processing user-generated air gestures, voice, or other physiological input. In some instances, the input may be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, on-screen and adjacent-screen gesture recognition, air gestures, head and eye tracking, and touch recognition associated with the display of computing device 800 (described in more detail below). Computing device 800 may include depth cameras such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof for gesture detection and recognition. In addition, computing device 800 may include an accelerometer or gyroscope that enables motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by computing device 800 to render immersive augmented reality or virtual reality.

[0173] Power supply 816 may include hardwired power supply, battery power supply, or a combination thereof. Power supply 816 may supply power to computing device 800 to allow the components of computing device 800 to operate.

[0174] The presentation component 818 may include a display (such as a monitor, touch screen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 818 may receive data from other components (such as GPU 808, CPU 806, DPU, etc.) and output that data (such as as images, videos, sounds, etc.).

[0175] Example Data Center Figure 9 An example data center 900 is illustrated, which can be used in at least one embodiment of this disclosure. Data center 900 may include a data center infrastructure layer 910, a framework layer 920, a software layer 930, and an application layer 940. In some embodiments, one or more functions of the facial landmark detector 110 and / or the facial landmark estimation model 120 described herein may be performed at least in part using data center 900.

[0176] like Figure 9 As shown, the data center infrastructure layer 910 may include a resource coordinator 912, grouped computing resources 914, and node computing resources (“nodes CR”) 916(1)-916(N), where “N” represents any complete positive integer. In at least one embodiment, nodes CR 916(1)-916(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state drives or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules and cooling modules, etc. In some embodiments, one or more nodes CR 916(1)-916(N) may correspond to servers having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CR 916(1)-916(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more of nodes CR 916(1)-916(N) may correspond to virtual machines (VMs).

[0177] In at least one embodiment, the grouped computing resources 914 may include individual groups (not shown) of nodes CR916 housed in one or more racks, or a plurality of racks (also not shown) housed in data centers in various geographic locations. Individual groups of nodes CR916 within the grouped computing resources 914 may include computing, networking, memory, or storage resources that can be configured or allocated to support groups of one or more workloads. In at least one embodiment, several nodes CR916, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0178] Resource coordinator 912 may be configured or otherwise controlled by one or more nodes CR 916(1)-916(N) and / or grouped computing resources 914. In at least one embodiment, resource coordinator 912 may include a Software Design Infrastructure (SDI) management entity for data center 900. Resource coordinator 912 may include hardware, software, or some combination thereof. In some embodiments, one or more functions of the facial landmark detector 110 and / or facial landmark estimation model 120 described herein may be performed at least in part using one or more nodes CRs 916(1)-916(N).

[0179] In at least one embodiment, such as Figure 9 As shown, framework layer 920 may include a job scheduler 933, a configuration manager 934, a resource manager 936, and a distributed file system 938. Framework layer 920 may include a framework of software 932 supporting software layer 930 and / or one or more applications 942 of application layer 940. Software 932 or application 942 may respectively include web-based service software or applications, such as service software or applications provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 920 may be, but is not limited to, a free and open-source software web application framework, such as Apache Spark, which can utilize distributed file system 938 for large-scale data processing (e.g., "big data"). TM(Hereinafter referred to as "Spark"). In at least one embodiment, the job scheduler 933 may include a Spark driver for facilitating the scheduling of workloads supported by various layers of the data center 900. In at least one embodiment, the configuration manager 934 may be able to configure different layers, such as the software layer 930 and the framework layer 920, which includes Spark and a distributed file system 938 for supporting large-scale data processing. The resource manager 936 is able to manage cluster or grouped computing resources mapped to or allocated to support the distributed file system 938 and the job scheduler 933. In at least one embodiment, the cluster or grouped computing resources may include grouped computing resources 914 at the data center infrastructure layer 910. The resource manager 936 may coordinate with the resource coordinator 912 to manage these mapped or allocated computing resources.

[0180] In at least one embodiment, the software 932 included in the software layer 930 may include software used by at least a portion of the nodes CR 916(1)-916(N), the grouped computing resources 914, and / or the distributed file system 938 of the framework layer 920. One or more types of software may include, but are not limited to, Internet web page search software, email virus browsing software, database software, and streaming video content software.

[0181] In at least one embodiment, one or more applications 942 included in application layer 940 may include one or more types of applications used by at least a portion of nodes CR 916(1)-916(N), grouped computing resources 914, and / or the distributed file system 938 of framework layer 920. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments. In some embodiments, one or more functions of the facial landmark detector 110 and / or facial landmark estimation model 120 described herein may be performed at least in part using software 932 and / or application 942.

[0182] In at least one embodiment, any of the configuration manager 934, resource manager 936, and resource coordinator 912 can perform any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible manner. Self-modification actions can alleviate the risk of data center operators of data center 900 making potentially poor configuration decisions and can prevent underutilization and / or skewed portions of the data center.

[0183] Data center 900 may include tools, services, software, or other resources for training one or more machine learning models or using one or more machine learning models to predict or infer information according to one or more embodiments described herein. For example, a machine learning model can be trained by calculating weight parameters based on a neural network architecture using the software and computing resources described above with respect to data center 900. In at least one embodiment, by using weight parameters calculated through one or more training techniques, information can be inferred or predicted using trained machine learning models corresponding to one or more neural networks, such as, but not limited to, those described herein, using the resources described above with respect to data center 900.

[0184] In at least one embodiment, the data center 900 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as a service to allow a user to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services.

[0185] Example network environment A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may... Figure 8 This is implemented on one or more instances of computing device 800—for example, each device may include similar components, features, and / or functions of computing device 800. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of data center 900, examples of which are described here regarding... Figure 9 To describe in more detail.

[0186] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks, or networks within multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. In cases where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0187] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment), and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the server functionality described herein can be implemented on any number of client devices.

[0188] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. A cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework for supporting software at the software layer and / or one or more applications at the application layer. The software or applications may respectively include network-based service software or applications. In embodiments, one or more client devices may use network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open-source software network application framework, such as one that can use a distributed file system for large-scale data processing (e.g., "big data").

[0189] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these various functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers at the state, region, country, global, etc.). If the connection to a user (e.g., a client device) is relatively close to an edge server, the core server can assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0190] Client devices may include those described in this article. Figure 8The example computing device 800 described includes at least some components, features, and functions. By way of example and not limitation, the client device may be a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, aircraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming equipment or system, entertainment system, in-vehicle computer system, embedded system controller, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these described devices, or any other suitable device.

[0191] This disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, which are executed by a computer or other machine such as a personal digital assistant or other handheld device. Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a wide variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. This disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices linked via a communication network.

[0192] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" could include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. Furthermore, "at least one of element A or element B" could include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" could include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0193] This document describes in detail the subject matter of this disclosure to meet legal requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have envisioned that the claimed subject matter may also be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms “step” and / or “box” may be used herein to imply different elements of the method employed, these terms should not be construed as suggesting any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. One or more processors, including processing circuitry, said processing circuitry being used to: Generate a map of keypoint location confidence values ​​for one or more feature keypoints based on at least one or more images of the object; The position of each of the one or more feature key points relative to the graph is determined, based at least on the peak value of the confidence value of the key point position. Based at least on the statistical distribution of the confidence values ​​of the key point locations, calculate the occlusion score of at least one individual key point among the one or more key points. Keypoint prediction is generated based at least on the location of each feature keypoint and the occlusion score of the at least one individual feature keypoint. as well as Control one or more operations of the vehicle based at least on the key point predictions.

2. The processor of claim 1, wherein the processor is further configured to associate the position of each feature keypoint relative to the map with the position of one or more keypoints relative to the one or more facial images of the object to generate the keypoint prediction.

3. The processor of claim 1, wherein the processor is further configured to generate a heatmap from the graph based on keypoints, wherein a pixel represents one or more data channels, and wherein a pixel value in a single data channel of the one or more data channels represents a corresponding keypoint location confidence value of a feature keypoint.

4. The processor of claim 1, wherein the processor is further configured to generate the image based on applying the one or more images to a feature keypoint estimation model.

5. The processor of claim 4 or more, wherein the feature keypoint estimation model is further configured to infer occlusion classification based on at least one of the one or more images and the graph, wherein the occlusion classification indicates the occlusion type of the feature keypoints occluding the one or more feature keypoints.

6. The processor of claim 5 or more, wherein the occlusion classification includes one of the following: no occlusion, self-occlusion, object occlusion, or truncated occlusion.

7. One or more processors according to claim 4, wherein the feature keypoint estimation model is trained using image training data, wherein samples of the image training data include a first annotation based on the location of the one or more feature keypoints and a second annotation based on occlusion indications of the one or more feature keypoints.

8. One or more processors according to claim 4, wherein the feature keypoint estimation model is trained by optimizing the feature keypoint estimation model based at least on keypoint location misalignment loss and keypoint occlusion loss.

9. The processor of claim 1, wherein the processor is further configured to calculate the statistical distribution based on the standard deviation to determine the confidence value of the key point location.

10. The processor of claim 1, wherein the one or more images of the object are captured by one or more image sensors, the image sensors including red, green, and blue RGB sensors, infrared IR sensors, or RGB-IR sensors.

11. The processor of claim 1 or more, wherein the processing circuitry is included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; Systems for performing analog operations; systems for performing digital twin operations; A system for performing optical transmission simulation; A system for creating collaborative content for 3D assets; A system used to perform deep learning operations; A system used to perform remote operations; Systems used for performing real-time streaming; A system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; Systems implemented using edge devices; Systems implemented using robots; Systems used to perform conversational AI operations; A system that implements one or more language models; A system that implements one or more large language model LLMs; A system that implements one or more Visual Language Models (VLMs); A system for generating synthetic data; A system that uses AI to generate synthetic data; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that uses cloud computing resources at least in part.

12. A system comprising one or more processors, said one or more processors being used to: The location of each feature keypoint is determined based at least on the peak values ​​of the heatmap representing the confidence values ​​of the keypoint locations; Based at least on the statistical distribution of the confidence values ​​of the keypoint locations on the heatmap, calculate the occlusion score of at least one individual keypoint among the one or more keypoints; and The keypoint prediction data is output based at least on the location of each feature keypoint and the occlusion score of the at least one individual feature keypoint.

13. The system of claim 12, wherein the one or more processors are further configured to infer occlusion classification for the at least one individual feature keypoint based at least on one or more facial images and the graph, wherein the occlusion classification includes one of the following: no occlusion, self-occlusion, object occlusion, or truncated occlusion.

14. The system of claim 12, wherein the one or more processors are further configured to associate the positions of the respective feature keypoints relative to the map with the positions of one or more keypoints relative to one or more facial images of the object to generate the keypoint prediction data.

15. The system of claim 12, wherein a machine learning model is used to generate the heatmap representing the confidence values ​​of keypoint locations, the machine learning model being trained on at least a two-dimensional feature keypoint dataset and a three-dimensional feature keypoint dataset including one or more two-dimensional projections of self-occluded feature keypoints.

16. The system of claim 12, further comprising a machine learning model trained to generate the heatmap based at least on keypoint location misalignment loss and occlusion loss.

17. The system of claim 12, wherein the one or more processors are further configured to apply one or more labels to automatically annotate one or more facial images of an object based on one or more inferences from the one or more facial images indicating that the at least one individual facial feature keypoint is a self-occluding keypoint with object occlusion.

18. The system of claim 12, wherein the one or more processors are further configured to generate the image as a heatmap, wherein the pixels of the heatmap represent one or more data channels, wherein individual data channels of the one or more data channels represent corresponding keypoint location confidence values ​​of facial feature keypoints.

19. The system of claim 12, wherein the system is included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system for creating collaborative content for 3D assets; a system for performing deep learning operations; A system used to perform remote operations; Systems used for performing real-time streaming; A system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; Systems implemented using edge devices; Systems implemented using robots; systems for performing conversational AI operations; systems that implement one or more language models; A system that implements one or more large language model LLMs; A system that implements one or more Visual Language Models (VLMs); A system for generating synthetic data; A system that uses AI to generate synthetic data; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that uses cloud computing resources at least in part.

20. A method comprising: Keypoint prediction data representing one or more feature keypoints is generated based on at least one or more facial images of an object, at least based on a graph of keypoint location confidence values ​​inferred from the one or more facial images to calculate peaks, and based on the statistical distribution of the graph of keypoint location confidence values.

Images