Depth assisted 3D detection for vehicle occupant detection and monitoring

Abstract

The present disclosure relates to depth-augmented 3D detection for vehicle occupant detection and monitoring. In various embodiments, systems and methods related to depth-augmented 3D detection are provided. The technology described herein can be used to utilize depth-augmented 3D detection to monitor the state of a vehicle occupant with high precision. Sensors can be used to capture image data and depth data, which includes features of a vehicle occupant (e.g., eyes, head, hands, etc.). 2D image coordinates from the image data can be converted to 3D coordinates in a global coordinate system using the depth data and a 3D occupant model, which can be personalized to the vehicle occupant. Based on the 3D coordinates in the global coordinate system, precise values can be determined, and the state of features of the vehicle occupant can be estimated based on these values. The values and / or state of features of the vehicle occupant can be used to detect drowsiness, distraction, and / or other conditions, and provide for safe vehicle operation.

Description

Background Technology

[0001] Autonomous and semi-autonomous vehicles rely on machine learning methods, such as those using deep neural networks (DNNs), to analyze images of the interior space of a vehicle or other machine (e.g., passenger compartment, cockpit). Occupant monitoring systems (OMS) are examples of systems that can be used within a vehicle compartment to perform real-time assessments of the presence, gaze, alertness, or other conditions of an occupant or operator. For example, OMS sensors (e.g., but not limited to red, green, and blue (RGB) sensors, infrared (IR) sensors, depth sensors, cameras, and / or other optical sensors) can be used to track the occupant's or operator's gaze direction, head posture, and / or blinking. This information can be used to determine the occupant's or operator's level of attention (e.g., to detect drowsiness, fatigue, and / or distraction), and / or to take responsive actions to prevent harm to the occupant or operator (e.g., by redirecting their attention to a potential hazard, pulling over, etc.). For example, a DNN can be used to detect whether an operator is asleep at the wheel based on their gaze looking down at the vehicle floor or blink rate, and the detection result can lead to adjustments in the vehicle's speed and direction (e.g., pulling over) or an audible alarm to the operator. Summary of the Invention

[0002] Embodiments of this disclosure relate to depth-assisted 3D detection. The disclosed systems and methods can be used to detect the state of one or more features of a vehicle occupant and / or generate labels for images, which can be used to train one or more models for various tasks, such as blink detection, drowsiness detection, etc., and others.

[0003] Unlike conventional systems, the system and method proposed in this disclosure can generate three-dimensional (3D) coordinates of one or more keypoints in a global coordinate system based on 2D image coordinates of one or more keypoints of at least one feature of a vehicle occupant, a 3D occupant model including the at least one feature, and depth data in a global coordinate system. The 3D occupant model can include a general 3D model of the vehicle occupant or a personalized 3D model of the vehicle occupant. Depth data can be provided by a depth sensor synchronized with an image sensor, generated based on stereo information determined using sensor data from two or more sensors, or estimated based on light rays from the image sensor to a plane fitted to the 3D image coordinates of the one or more keypoints. One or more states of at least one feature of the vehicle occupant can be determined based on the 3D coordinates of the one or more keypoints in the global coordinate system, and one or more indications of the one or more states can be output to other components of the system (e.g., an OMS or a driver monitoring system (DMS)) for further actions (e.g., issuing an alarm, pulling over, etc.). Image data can be automatically labeled based on the 3D coordinates of one or more keypoints in the global coordinate system and can be used to train one or more machine learning models. Attached Figure Description

[0004] The system and method for depth-assisted 3D detection are described in detail below with reference to the accompanying drawings, wherein: Figure 1A This is a flowchart illustration of an example depth-assisted 3D detection system suitable for implementing some embodiments of this disclosure; Figure 1B This is a flowchart illustration of an example depth-assisted 3D detection system suitable for implementing some embodiments of this disclosure; Figure 2 This is a flowchart illustration of an example blink detection system suitable for implementing some embodiments of the present disclosure; Figure 3 This is a flowchart illustration of an example drowsiness detection system suitable for implementing some embodiments of the present disclosure; Figure 4 This is a flowchart illustration of an example gaze detection system suitable for implementing some embodiments of this disclosure; Figure 5 This is an illustration of an example flowchart for depth-assisted 3D detection according to some embodiments of the present disclosure; Figure 6A Examples of sensor locations having a corresponding field of view or sensing field, such as that of an autonomous or semi-autonomous machine, according to at least some embodiments of this disclosure; Figure 6BThis is an illustration of examples of the locations of components and sensors on an autonomous or semi-autonomous vehicle according to at least some embodiments of this disclosure; Figure 6C This is a block diagram of an example system architecture for autonomous or semi-autonomous vehicles, robots, and / or other machine types according to at least some embodiments of this disclosure; Figure 6D This is a block diagram of an example architecture of a computing system (e.g., a system-on-a-chip (SoC)) according to at least some embodiments of the present disclosure; Figure 6E This is a system diagram of communication between a cloud-based server and example autonomous or semi-autonomous vehicles, robots and / or other machine types according to at least some embodiments of this disclosure; Figure 7 This is a system diagram illustrating three computer ecosystems according to at least some embodiments of the present disclosure, including a computing system for generating or creating artificial intelligence (AI) (e.g., AI training and validation data), a computing system for training artificial intelligence, and a computing system for deploying AI at the edge; Figure 8 This is a block diagram of an example computing system for generative artificial intelligence (AI) according to at least some embodiments of the present disclosure; and Figure 9 This is a block diagram of an example computing device according to at least some embodiments of the present disclosure. Detailed Implementation

[0005] Systems and methods related to depth-assisted 3D detection are disclosed. Although this disclosure may relate to examples of autonomous or semi-autonomous vehicles, robots, and / or other machine types 600 (which may alternatively be referred to herein as "vehicle 600," "self-vehicle 600," "machine 600," "self-machine 600," "robot 600," and / or "self-robot 600"), examples are related to... Figures 6A-6E(A description has been provided, but this is not intended to be limiting.) For example, the systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms (e.g., autonomous mobile robots (AMRs), humanoid robots, robotic arms and / or end effectors, warehouse vehicles, off-road vehicles, vehicles connected to one or more trailers, aircraft, sea vehicles, shuttles (e.g., driverless taxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones and / or other vehicle, robot or machine types. Furthermore, although this disclosure may describe 3D detection for eye measurements, it is not intended to be limiting; the systems and methods described herein can be used in augmented reality (AR), virtual reality (VR), mixed reality (MR), robotics, security (e.g., smart cities), autonomous or semi-autonomous machine applications, industrial manufacturing, simulation and / or any other technology space where 3D detection technology can be used. In some embodiments, the systems, methods and / or processes described herein can be used with… Figures 6A-6E Example machine 600 Figure 7 Example calculation of ecosystem 700 Figure 8 Example generative language model system 800 and / or Figure 9 The example computing device 900 uses similar components, features, and / or functions to perform this task.

[0006] Common techniques for blink estimation using facial keypoints involve detecting and analyzing the location of keypoints on an occupant's face to determine whether their eyes are open or closed. This process typically involves locating a face in an image or video frame using face detection algorithms and identifying two-dimensional (2D) keypoints on the face, particularly around the eyes. Keypoints used for blink detection and / or drowsiness detection may include, but are not limited to, points at the corners of the eyes and on the eyelids. Using the identified 2D keypoints, the eye aspect ratio (EAR) is determined, which is a measurement of the distance between eye keypoints in the vertical direction divided by the distance between eye keypoints in the horizontal direction. The EAR changes significantly when the eyes are closed compared to when they are open. For example, a larger EAR indicates the eyes are open, while a smaller EAR indicates the eyes are closed. A threshold is typically set for the EAR; if the EAR is below this threshold, it indicates the eyes are closed. Similarly, if the EAR is above this threshold, it indicates the eyes are open. The EAR can be continuously monitored over time to detect blinks, and blinks are usually detected when the EAR drops below the threshold and then rises above it again.

[0007] While EAR-based blink estimation offers a non-invasive and efficient way to monitor eye closure and detect blinks, it has significant limitations due to changes in head posture (e.g., yaw or pitch rotation), which can lead to occlusion (e.g., self-occlusion), preventing the subject's eyes from being captured in the image frame and / or resulting in inaccurate scaling factors. EAR-based techniques can provide sufficient accuracy for some use cases when the occupant's face is in a frontal pose (e.g., with little or no self-occlusion). However, 2D keypoints can be unreliable when the head rotates. For example, if the occupant's head pitches (e.g., looking up or down), the accuracy of the EAR metric will be limited due to perspective blur caused by the occupant's head posture. Furthermore, EAR-based techniques typically assume the same threshold for both eyes, which may not adequately account for eye asymmetry and lead to incorrect blink estimation.

[0008] For training blink detection or drowsiness detection models, manual annotation is typically used to label blinking segments of vehicle occupants in the training data. This requires manually reviewing each frame of the video and labeling the start and end of each blink. This process is time-consuming and costly. Furthermore, inconsistencies exist in this process due to the insufficient accuracy of the EAR metric, the lack of a standard threshold for the EAR metric as a blink transition point (e.g., due to differences in blinking patterns among different users), and inherent differences in how multiple human annotators define blink transition points (e.g., when multiple people annotate training data).

[0009] Unlike conventional systems such as those described above, the systems and methods proposed in this disclosure can generate three-dimensional (3D) coordinates of one or more facial keypoints (e.g., around the eyes and chin) in a global coordinate system based on 2D image coordinates of these facial keypoints, a 3D occupant model (e.g., a 3D model of the occupant's head), and depth data in a global coordinate system. The 3D occupant model can include a generic 3D model of the vehicle occupant (e.g., a generic 3D model of a human head) or a personalized 3D model of the vehicle occupant (e.g., a personalized 3D model of the occupant's head). Depth data can be provided by a depth sensor synchronized with an image sensor, generated based on stereo information determined using sensor data from two or more sensors, or estimated based on light rays from the image sensor to a plane fitted to the 3D image coordinates of the facial keypoints. One or more states of a vehicle occupant's eyes can be determined based on the 3D coordinates of these facial key points in a global coordinate system, and indications of these states can be output to other components of the system (e.g., an OMS or a driver monitoring system (DMS)) for further action (e.g., issuing an alarm, pulling over, etc.). This state may include measurements of eye openness, eye open / closed state, or gaze direction, which can be determined based on the 3D coordinates of these facial key points in the global coordinate system. Image data can be automatically labeled based on the 3D coordinates of these facial key points in the global coordinate system and used to train blink detection and / or drowsiness detection models.

[0010] One or more image sensors (e.g., RGB sensors, IR sensors, cameras, and / or other image sensors) can be used to capture image data including at least a portion of the face and head of a vehicle occupant. For example, one or more image sensors may be placed near the dashboard, steering wheel, or center console and aimed at the vehicle occupant (e.g., the driver) to monitor occupant drowsiness or alertness. One or more face detection algorithms can be used to locate the occupant's face in the image data, and one or more facial keypoint detection algorithms can be used to identify the occupant's facial keypoints. For example, the occupant's facial keypoints may include, but are not limited to, keypoints of the chin, eyebrows, eyes, mouth, lips, pupils, or ears. The 2D coordinates of the occupant's facial keypoints in the image sensor coordinate system can be determined based on the image data.

[0011] A 3D occupant model can be used to determine the 3D coordinates of facial key points in an image sensor coordinate system. This 3D occupant model can include a generic 3D model of a human head. In some embodiments, the 3D occupant model can include a personalized 3D model of the occupant's head, which can provide greater accuracy in determining the 3D coordinates of facial key points. To generate a personalized 3D model of a vehicle occupant's head, the occupant can sit in a seat within the vehicle and can be guided to perform predefined head movements to cover multiple viewpoints of their head, while depth data is captured using a depth sensor. In some embodiments, feedback can be provided to the occupant using instructions on how to rotate their head to capture a specified range of viewpoints (e.g., visual guidance and / or auditory / voice commands). Predefined head movements can include head rotation in three directions, including roll, pitch, and / or yaw rotation. In some embodiments, the depth data captured during this process can be used to adjust the generic 3D model of the human head to personalize the model and reflect the characteristics of the occupant, which can result in the generation of a personalized 3D model of the occupant's head.

[0012] The 3D occupant model can include the known 3D coordinates of facial key points. Using the Perspective n-Point (PnP) algorithm, the 2D image coordinates of the occupant's facial key points in the image sensor coordinate system can be fitted to the known 3D coordinates of the facial key points in the 3D occupant model, thereby determining the pose and orientation of the occupant's head in the image data. Fitting the known 3D coordinates of the 3D occupant model's facial key points to the 2D image coordinates of the facial key points can include applying estimated rotation and translation vectors to transform the known 3D coordinates of the 3D occupant model to the camera coordinate system.

[0013] Depth data in a global coordinate system (e.g., based on a reference point) can be used to generate 3D coordinates of facial keypoints in a global coordinate system from their 3D coordinates in the image sensor coordinate system. In some embodiments, depth data (e.g., depth point clouds or depth images) can be captured synchronously with image data using one or more depth sensors (e.g., stereo imaging camera depth sensors, time-of-flight camera depth sensors, millimeter-wave radar sensors, Kinect sensors, or other depth sensors), which can be synchronized with the image sensor. Depth data can also be generated based on stereo information determined using sensor data from two or more sensors (e.g., DMS / OMS sensors), which can be synchronized with the image sensor. Depth data can also be estimated by fitting a plane parallel to a set of keypoints on the face (e.g., eyelid keypoints, corners of the eyes, chin, etc.) and determining the light rays from the image sensor to the fitted plane, which moves with the head at various angles. The 3D coordinates of facial key points in the global coordinate system can be generated using the intersection of a 3D model of the head corresponding to the 3D coordinates of facial key points and depth data (e.g., depth point cloud, depth image, fitted plane).

[0014] By utilizing the 3D coordinates of facial keypoints in a global coordinate system, the state of a vehicle occupant's eyes can be determined. For example, one example state that can be determined is the amount of eye opening of one or more of the vehicle occupant's eyes, referred to as a measurement or value of eye openness. For instance, an eye openness measurement can be determined based on the distance between the 3D coordinates corresponding to the vehicle occupant's eyelid keypoints (e.g., upper and lower eyelid keypoints) in the global coordinate system. By converting to 3D coordinates in the global coordinate system, the eye openness measurement can be determined in absolute units (e.g., metric units) and can provide sub-millimeter accuracy.

[0015] Another example of a vehicle occupant's eye state that can be determined is whether the eye is open or closed. In some embodiments, an eye is determined to be closed when a measurement of the occupant's eye opening degree is below a threshold. This determination can be made using a vehicle occupant-specific threshold, or even an individualized threshold for each eye (e.g., since the measured values ​​of eye opening degree differ from each other when the occupant's left and right eyes are fully open or closed). The threshold for each of the vehicle occupant's eyes can be determined based on the occupant's baseline (e.g., the average of previous samples or a calibration procedure) or other factors over time.

[0016] Another example of a definite eye state for a vehicle occupant is gaze direction. In some embodiments, the position of the vehicle occupant's pupil can be determined based on the 3D coordinates of one or more pupil keypoints in a global coordinate system. The gaze direction of the vehicle occupant can be estimated based on the position of the vehicle occupant's pupil relative to other eye keypoints (e.g., eyelid or corner of the eye keypoints).

[0017] 3D coordinates of facial keypoints in a global coordinate system can be continuously generated, and the state of an occupant's eyes can be continuously monitored. Blinking can be detected based on these 3D coordinates of facial keypoints in the global coordinate system. For example, a blink can be detected when a measurement of the openness of an eye drops below a threshold (e.g., closed), and then subsequently the measurement of the openness of that eye exceeds the threshold (e.g., open). The blink rate can be estimated as the number of blinks within a given time period. The blink duration can be estimated as the amount of time taken to complete the blink. In some embodiments, the start of a blink can be determined when the measurement of the openness begins to decrease from an approximate maximum value, and the end of a blink can be determined when the measurement of the openness returns to an approximate maximum value after decreasing below an approximate minimum threshold. Blink duration can alternatively be estimated as the amount of time the eye is closed during the blink. Other metrics (e.g., percentage eyelid closure (PERCLOS)) can also be determined based on the 3D coordinates of facial keypoints in the global coordinate system. Based on measurements of eye opening, eye open / closed status, blink count, blink rate, blink duration, PERCLOS, and / or other metrics determined using the 3D coordinates of facial key points in the global coordinate system, vehicle operation can be adjusted and / or notifications can be provided to occupants.

[0018] Furthermore, while the techniques described herein provide more accurate measurements of eye opening compared to EAR-based techniques, converting to 3D coordinates in a global coordinate system may incur higher computational costs. In some embodiments, it is possible to determine whether the vehicle occupant's face is in a frontal pose, and in response to such a determination, the EAR value can be determined based on the 2D image coordinates of the vehicle occupant's facial key points and used to perform blink detection and / or drowsiness detection, rather than based on 3D coordinates in a global coordinate system. Since EAR-based techniques typically provide sufficient accuracy when the vehicle occupant's face is in a frontal pose, this hybrid approach can provide adequate accuracy while reducing computational costs.

[0019] The embodiments described in this disclosure primarily refer to determining the state of one or more eyes of a vehicle occupant, for example, for blink detection and / or drowsiness detection. However, it should be understood that techniques similar to those described herein can be used for other applications. For example, similar techniques can be used to determine the state of different characteristics of a vehicle occupant, such as, but not limited to, the mouth, hands, or fingers. The embodiments described in this disclosure can be implemented in the context of training or operating a vehicle's DMS or OMS, such as, but not limited to, autonomous vehicles, semi-autonomous vehicles, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, spacecraft, ships, shuttles, emergency response vehicles, engineering vehicles, underwater vehicles, drones, and / or other vehicle types, and can operate in a variety of locations.

[0020] In some embodiments, the systems and methods described herein can be performed in a simulated environment (e.g., simulated environment data and simulated sensor data of simulated sensors of virtual or simulated vehicles, robots, or machines within the simulated environment) using simulated data. For example, simulated input data (e.g., map data, perception data, self-motion data, tactile data, and / or any other data described herein) can be used to determine eye-opening values ​​for blink detection, drowsiness detection, etc., and this information can be used to perform operations associated with virtual machines in the simulated environment. These simulated operations can be used to test the performance of underlying algorithms, systems, and / or processes before deploying them to the real world. In some cases, simulation can be used to generate synthetic training data, such as synthetic image data and / or synthetic depth data from within the simulation. The synthetic training data (in addition to real-world data, or as a substitute for real-world data) can then be used, for example, or processed to generate corresponding 2D image coordinates, 3D image coordinates, and 3D global coordinates.

[0021] In any example, such as when using a simulated environment for testing, validation, training, etc., the simulated environment and / or associated training data may be rendered or otherwise generated using one or more optical transport simulation algorithms, such as one or more ray tracing and / or path tracing algorithms. When using optical transport simulation, the simulation system may employ one or more dedicated ray tracing hardware accelerators and / or processors (e.g., NVIDIA's RTX or another real-time ray tracing GPU, such as a GPU including one or more ray tracing (RT) cores) optimized to perform real-time or near-real-time optical transport simulation operations in collaboration with one or more other processors in the system (e.g., GPUs, CPUs, accelerators, etc.). In some embodiments, the simulated environment and / or one or more of its objects, features, or components may be generated or managed in a three-dimensional (3D) content collaboration platform (e.g., NVIDIA's OMNIVERSE), which may be optimized for or adapted to industrial digitization, generative physics, artificial intelligence, and / or other use cases, applications, and / or services. For example, a content collaboration platform or system may include a system for using or developing common scene descriptors (USD) (such as OpenUSD) data to manage objects, features, scenes, etc., in simulated environments, digital environments, etc. The platform may include simulations of real-world physical phenomena (e.g., using NVIDIA's PhysX software development kit (SDK)) to simulate real-world physical phenomena and physical interactions with simulations hosted on the platform. The platform may integrate OpenUSD with ray tracing / path tracing / light transport simulations (such as NVIDIA's RTX rendering technology) into software tools and simulation workflows for building, training, deploying, and / or testing AI systems—such as systems for testing, validating, and training (e.g., machine learning models, neural networks, etc.)—and / or other tasks related to automobiles, robots, other machine types, and / or other systems and applications. In some examples, the simulated environment may include a digital twin of a real-world environment, such as a specific road segment, warehouse, data center, airport, geographic area, ocean area, and / or any other real-world environment in which autonomous or semi-autonomous vehicles or machines can operate.

[0022] In some embodiments, teleoperation or remote control of vehicles, robots, and / or other machines can be performed using a remote control or teleoperation system. For example, the systems and methods described herein can be used to generate precise values ​​that can be included in a visualization or map of the environment to assist a remote operator in controlling an autonomous or semi-autonomous machine through the environment, or to provide waypoints or other instructions for control or navigation. Therefore, a remote operator can use visual, auditory, textual, and / or other cues or indicators generated using the systems and methods described herein to assist a vehicle, robot, machine, etc., in navigating a real-world environment using a teleoperation system.

[0023] In some embodiments, the systems and methods described herein can be deployed in robotic applications. For example, a robot or robotic system may include one or more onboard processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), hardware-based programmable vision accelerator (PVA) (which may include one or more vector processing units (VPU), direct memory access (DMA) system and / or pixel processing engine (PPE)), hardware-based optical flow accelerator (OFA), SoC, etc.) and memory and / or storage devices (e.g., for storing control algorithms, sensor data, and one or more machine learning models). The robotic system can utilize these processors to execute one or more machine learning models (e.g., language models, visual language models (VLM), large language models (LLM), visual-language-action (VLA) models, multimodal language models (MMLM), etc.) that enable it to perform complex tasks autonomously or semi-autonomously, such as interacting with and / or manipulating static and / or dynamic objects, or navigating its environment using sensors such as cameras, LiDAR, RADAR, and ultrasonic sensors. The system can employ sensor fusion techniques to combine data from multiple sensors (e.g., cameras, infrared sensors, LiDAR, RADAR, accelerometers) to create a comprehensive model of the robot's surrounding environment. This data can be processed locally on the robot or sent to a remote server for computationally intensive tasks such as 3D mapping or SLAM (Simultaneous Localization and Mapping). In one or more embodiments, data from individual robots (e.g., sensor data, task status, or environmental conditions) can be uploaded to the cloud, where a centralized AI model can analyze optimized commands and distribute them across the entire fleet. In some embodiments, the machine learning models described herein (e.g., language models, VLM, VLA, LLM, MMLM, diffusion models, NeRF models, DNN, etc.) can be used to allow a robot to perceive its environment and make inferences about it, and / or communicate with one or more other robots and / or people in the environment. In some embodiments, the robot may communicate with one or more locally hosted servers / computing devices (e.g., using one or more network interface cards (NICs) and / or data processing units (DPUs)), and / or with one or more remotely located servers / computing devices (e.g., in one or more data centers).

[0024] In some embodiments, the systems and methods described herein can be deployed in in-vehicle infotainment (IVI) systems or in-cabin experience (IX) applications. For example, an infotainment system within a vehicle (e.g., a car, truck, drone, construction equipment, robot, semi-autonomous vehicle, or autonomous vehicle) may include one or more in-vehicle processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), hardware-based programmable vision accelerator (PVA) (which may include one or more vector processing units (VPU), direct memory access (DMA) systems, and / or pixel processing engines (PPE)), hardware-based optical flow accelerator (OFA), SoC, etc.) and memory and / or storage devices (e.g., for storing entertainment content, navigation data, and user preferences). The system may utilize these processors to execute one or more machine learning models (e.g., language models) to achieve features such as voice control, personalized media recommendations, dynamic navigation, and real-time communication with other services via network connectivity. The in-vehicle infotainment system may also use natural language processing (NLP) models to achieve voice-based interaction. One or more machine learning models can be stored locally or accessed via one or more APIs connected to a cloud service, enabling the system to process requests in real time or near real time.

[0025] In some examples, the machine learning models described herein (e.g., deep neural networks, language models, LLMs, VLMs, multimodal language models, visual-language-action (VLA) models, perceptual models, tracking models, fusion models, transformer models, diffusion models, encoder-only models, decoder-only models, encoder-decoder models, neural rendering field (NERF) models, etc.) can be packaged into microservices—such as inference microservices (e.g., NVIDIA NIM)—which may include containers (e.g., operating system (OS) level virtualization packages) that may include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model “engine.” For example, an inference microservice may include the container itself as well as the model (e.g., weights and biases). In some cases, such as when the machine learning model is small enough (e.g., with a small number of parameters), the model may be included within the container itself. In other examples (e.g., when the model is large), the model may be hosted / stored in the cloud (e.g., in a data center), and / or hosted locally and / or at the edge (e.g., on a local server or computing device, but outside the container). In these embodiments, the model may be accessed via one or more APIs (e.g., REST APIs). Therefore, in some embodiments, the machine learning models described herein can be deployed as inference microservices to accelerate model deployment on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, an optimized inference engine (e.g., execution software built using standardized AI model deployments, such as NVIDIA's Triton Inference Server), and / or one or more APIs for high-performance deep learning inference, which may include inference runtimes and model optimizations providing low latency and high throughput for production applications, such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). The machine learning models described herein may be included as part of a microservice along with an acceleration infrastructure capable of deployment with a single command and / or orchestrated and automatically scaled on the acceleration infrastructure using a container orchestration system (e.g., reaching data center scale on a single device). Therefore, the inference microservice may include a machine learning model (e.g., optimized for high-performance inference), inference runtime software for executing the machine learning model and providing output / response to inputs (e.g., user queries, prompts, etc.), and enterprise management software for providing health checks, authentication, and / or other monitoring. In some embodiments, the inference microservice may include software for in-situ replacement and / or updating of the machine learning model. During replacement or updating, the software performing the replacement / update may maintain the user configurations of the inference runtime software and the enterprise management software.

[0026] While this article may describe examples using machine learning models (such as neural networks), it is not intended to be limiting. For example, but not limited to, any of the various machine learning models and / or neural networks described herein may include any type of machine learning model, such as those using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoder neural networks, artificial neural networks (ANN), convolutional neural networks (CNN), recurrent neural networks (RNN), perceptrons, long short-term memory (LSTM) networks, multilayer perceptron (MLP) networks, deep stacked networks (DSN), generative pre-trained (GPT) models or networks, feedforward networks, radial basis function ANNs, self-organizing maps (SOM), Kohonen maps, Hopfield networks, Boltzmann machines, deep belief neural networks, deconvolutional neural networks, generative adversarial networks (GAN)), liquid machines, modular neural networks, sequence-to-sequence models, networks using transformer architectures, state-space models (SSM) (e.g., networks using Mamba architectures (e.g., Mamba-1, Mamba...)). 2, etc.), networks using selective state-space models, networks using structured state-space sequence models, etc.), diffusion models (e.g., diffusion probability models, fraction-based generative models, etc.), neural radiation field (NeRF) models, Gaussian sputtering models, Kolmogorov-Arnold networks (KAN), models with encoder-only architectures, models with decoder-only architectures, models with encoder-decoder architectures, generative machine learning models, language models, large language models (LLM), visual language models (VLM), multimodal language models (MMLM), large action models (LAM), visual-language-action (VLA) models, etc., and / or other types of machine learning models.

[0027] The systems and methods described herein may be used, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, spacecraft, ships, shuttles (e.g., robotaxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones, and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, by way of example but not limited to: machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and supervision, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets (e.g., NVIDIA's Omniverse), cloud computing and / or any other suitable application.

[0028] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines, etc.), systems implemented using robots, aviation systems, medical systems, marine systems, smart zone monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing language models (such as large language models (LLM), visual language models (VLM), visual-language-action (VLA) models, and / or multimodal language models), systems using or deploying one or more inference microservices, systems containing one or more machine learning models deployed in services or microservices and OS-level virtualization packages (e.g., containers), systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0029] refer to Figure 1A , Figure 1AThis is an example data flow diagram illustrating the component interconnections and data flow of a depth-assisted 3D inspection system 100 according to some embodiments of this disclosure. It should be understood that the arrangements and other arrangements described herein are merely examples. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to the arrangements, components, features, and elements shown (e.g., machines, interfaces, functions, sequences, functional groups, etc.), and some elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and at any suitable location (e.g., on a local device, vehicle, or machine at the edge; locally deployed—(e.g., locally hosted on a remote server)—e.g., on one or more computing or server devices in one or more data centers in the cloud; and / or other locations). The various functions performed by the entities described herein can be implemented through hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., one or more central processing units (CPUs), one or more graphics processing units (GPUs), microprocessors, microcontrollers, embedded processors, digital signal processors (DSPs), image signal processors (ISPs), physical processing units (PPUs), field-programmable gate arrays (FPGAs), one or more accelerators (e.g., deep learning accelerators (DLAs), deep learning accelerator clusters (XNNs), neural network accelerators (NNAs) and / or neural processing units (NPUs), programmable vision accelerators (PVAs), optical flow accelerators (OFAs), etc.), application-specific integrated circuits (ASICs), data processing units (DPUs), quantum processors, etc.) that execute instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 6A-6E Example machine 600 Figure 7 Example calculation ecosystem 700, Figure 8 Example generative language modeling systems 800 and / or Figure 9 The example computing device 900 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the task.

[0030] exist Figure 1AIn the example shown, the depth-assisted 3D detection system 100 may receive image data 104 from one or more image sensors 102. The image sensors 102 may be located within the vehicle and may be used to capture image data 104 that includes at least a portion of features of the vehicle occupants (e.g., face, head, hands, etc.). For example, the image sensor 102 may be located near the dashboard, steering wheel, or center console and aimed at the vehicle occupants (e.g., the driver) to at least partially monitor occupant drowsiness or alertness. The image sensor 102 may include, but is not limited to, any type of optical sensor (e.g., RGB optical sensor, IR optical sensor, RGB-IR optical sensor, camera, and / or other optical sensors, such as, but not limited to, the sensors described herein—e.g., in some examples, DMS sensors and / or OMS sensors for machine 600, etc.). The image data 104 may include, but is not limited to, sensor data from any type of optical sensor used for the image sensor 102. In some embodiments, the image data 104 may correspond to a 2D image or 2D image frame generated from the image data 104.

[0031] exist Figure 1A In the example shown, the depth-assisted 3D detection system 100 can also receive depth data 108 from one or more sensors 106. The sensors 106 may be located inside the vehicle and can be used to capture depth data 108 that includes at least a portion of features of the vehicle occupants (e.g., face, head, hands, etc.). For example, the sensors 106 may be located near the dashboard, steering wheel, or center console and aligned with the vehicle occupants (e.g., the driver). The depth data 108 can be generated in a global coordinate system based on a common reference point.

[0032] Sensor 106 can operate synchronously with image sensor 102, enabling the simultaneous acquisition of image data 104 and depth data 108. The depth-assisted 3D inspection system 100 can use an onboard synchronization signal provided by machine 600 to adjust the synchronization of image sensor 102 and sensor 106. For example, image sensor 102 and sensor 106 can be synchronized via a public transport synchronization signal generated by hardware or software, which can be used to trigger image sensor 102 and sensor 106 to acquire image data 104 and depth data 108 synchronously.

[0033] Sensor 106 may include, but is not limited to, a stereo imaging camera depth sensor, a time-of-flight camera depth sensor, a millimeter-wave radar sensor, a Kinect sensor, or other depth sensors, while depth data 108 may include, but is not limited to, depth point cloud data or depth image data. In some embodiments, sensor 106 may include two or more sensors that are not depth sensors, and depth data 108 may be generated based on stereo information determined using sensor data captured by these two or more sensors. For example, sensor 106 may include a first sensor (e.g., an OMS sensor) and a second sensor (e.g., a DMS sensor), which may be synchronized with each other, and sensor data from the first and second sensors may be processed (e.g., using feature matching, disparity determination, etc.) to generate depth data 108. In some embodiments, at least one sensor in sensor 106 may also be an image sensor in image sensor 102.

[0034] The depth-assisted 3D detection system 100 may include one or more keypoint detection models 110 that can identify one or more keypoints of one or more features of a vehicle occupant based on image data 104. These features may include, for example, the head, face, hands, and / or other features of the vehicle occupant. The keypoints of these features may include multiple points of features that can be used for other detection and / or measurement tasks. For example, keypoints of the head or face may include one or more points defining facial landmarks (e.g., eyebrows, eyelids, pupils, mouth, chin, etc.). The keypoint detection model 110 may include one or more models (e.g., machine learning models) that can detect one or more features and their one or more keypoints based on image data 104. The keypoint detection model 110 may be trained to identify keypoints of a specific human feature (e.g., head or face), or the keypoint detection model 110 may be trained to identify keypoints of multiple human features. The keypoint detection model 110 may generate 2D image coordinates 112 for each keypoint of one or more features detected in image data 104. The key point detection model 110 can provide the 2D image coordinates 112 to the key point fitting function 116.

[0035] Keypoint fitting function 116 can receive or retrieve 3D occupant model 117 (e.g., from a data store). 3D occupant model 117 can include point cloud data of one or more features of a vehicle occupant. 3D occupant model 117 can include a generic 3D occupant model. For example, 3D occupant model 117 can include a 3D model of at least one human body feature. In some embodiments, 3D occupant model can include a personalized 3D model of one or more features of an occupant. To generate a personalized 3D model of one or more features of a vehicle occupant, the occupant can sit in a vehicle seat and be instructed to perform predefined actions to cover multiple viewpoints of one or more features while capturing depth data using a depth sensor (e.g., sensor 106 or other sensors). In some embodiments, feedback (e.g., visual guidance and / or auditory / voice commands) can be provided to the occupant using instructions on how to rotate one or more features of the occupant to capture a specified range of viewpoints. Predefined actions can include rotation in three directions, including roll, pitch, and / or yaw rotation. In some embodiments, the depth data captured during this process can be used to adjust a generic 3D model of the features to personalize the model to reflect the characteristics of the occupant, resulting in the generation of personalized 3D models of one or more features of the occupant.

[0036] The 3D occupant model 117 may include known 3D coordinates of keypoints of one or more features of the vehicle occupant. The keypoint fitting function 116 may use a perspective n-point (PnP) algorithm to fit the 2D image coordinates 112 of one or more keypoints of one or more features of the occupant in the image sensor coordinate system to the known 3D coordinates of the keypoints of the 3D occupant model 117, thereby solving for the pose and orientation of the occupant features in the image data 104. Fitting the known 3D coordinates of the keypoints of the 3D occupant model 117 to the 2D image coordinates 112 of one or more keypoints of one or more features may include applying estimated rotation and translation vectors to transform the known 3D coordinates of the 3D occupant model 117 into 3D coordinates in the camera coordinate system. The keypoint fitting function 116 may output 3D image coordinates 118 of one or more keypoints in the image coordinate system and provide these 3D image coordinates 118 to the intersection determination function 120 of the depth-assisted 3D detection system 100.

[0037] The intersection determination function 120 of the depth-assisted 3D detection system 100 can receive depth data 108 and 3D image coordinates 118. The intersection determination function 120 can generate 3D global coordinates 122 in the global coordinate system (e.g., based on a reference point) by determining the intersection of the 3D image coordinates 118 and the depth data 108. For example, the intersection determination function 120 can use the intersection of the 3D occupant model 117 represented by the 3D image coordinates 118 and the depth data 108 to generate 3D global coordinates 122 of one or more keypoints of one or more features in the global coordinate system.

[0038] The depth-assisted 3D detection system 100 may include one or more feature state determination functions 124 that receive 3D global coordinates 122 and determine the state of a vehicle occupant's features based on those coordinates. If the vehicle occupant's features include one or more eyes, the state may include the amount or degree of eye opening, referred to as a measurement of eye opening or an eye opening value. For example, the measurement of eye opening may be determined based on the distance between the 3D global coordinates 122 of the corresponding eyelid keypoints (e.g., upper and lower eyelid keypoints) of the vehicle occupant. The feature state determination functions 124 may output one or more indications 126 of the feature state for use by one or more other systems (e.g., vehicle) or automatically labeled image data 104.

[0039] refer to Figure 1B , Figure 1BThis is an example data flow diagram illustrating the component interconnections and data flow of a depth-assisted 3D inspection system 150 according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are merely illustrative. Other arrangements, components, features, and elements may be used in addition to or as alternatives to the arrangements, components, features, and elements shown (e.g., machines, interfaces, functions, sequences, functional groupings, etc.), and certain elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and location (e.g., on a local device, vehicle, or edge machine; locally—e.g., on a locally hosted server; remotely—e.g., on one or more computing or server devices in one or more data centers in the cloud; and / or other locations). The various functions performed by the entities described herein can be implemented by hardware, firmware, and / or software. For example, various functions can be performed using one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, microcontroller, embedded processor, digital signal processor (DSP), image signal processor (ISP), physical processing unit (PPU), field-programmable gate array (FPGA), accelerators (e.g., deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural network accelerator (NNA) and / or neural processing unit (NPU), programmable vision accelerator (PVA), optical flow accelerator (OFA), etc.), application-specific integrated circuit (ASIC), data processing unit (DPU), quantum processor, etc.). In some embodiments, the systems, methods, and processes described herein can be used with... Figures 6A-6E Example machine 600 Figure 7 Example calculation of ecosystem 700, Figure 8 Example generative language modeling systems 800 and / or Figure 9 The example computing device 900 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the task.

[0040] The depth-assisted 3D inspection system 150 can receive and process image data 104 from one or more image sensors 102, in a manner similar to that described above. Figure 1A The depth-assisted 3D detection system 100 shown is similar in manner, and will only be discussed in this discussion. Figure 1A The depth-assisted 3D inspection system 100 shown is... Figure 1B The differences between the 150 depth-assisted 3D inspection systems shown. Figure 1BIn the example shown, the depth-assisted 3D detection system 150 may not receive depth data 108 from sensor 106. The depth-assisted 3D detection system 150 includes a plane fitting function 152 that generates a plane 154 (e.g., a 2D plane) fitted to the 3D image coordinates 118 of a set of key points representing features of a vehicle occupant. For example, plane 154 may be fitted to the 3D image coordinates 118 of a set of key points parallel to the face (e.g., eyelid key points, corners of the eyes, chin, etc.), which moves with the head at various angles. Plane 154 may be provided to a depth estimation function 156, and the depth estimation function 156 can generate estimated depth data 158 by determining the light rays from image sensor 102 to plane 154.

[0041] The intersection determination function 120 of the depth-assisted 3D detection system 150 can receive estimated depth data 158 and 3D image coordinates 118. The intersection determination function 120 can generate 3D global coordinates 122 in the global coordinate system (e.g., based on a reference point) by determining the intersection of the 3D image coordinates 118 and the estimated depth data 158. For example, the intersection determination function 120 can use the intersection of the 3D occupant model 117 represented by the 3D image coordinates 118 and the estimated depth data 158 to generate 3D global coordinates 122 of key points of features in the global coordinate system.

[0042] In some embodiments, such as when the features of a vehicle occupant include the eyes, depth-assisted 3D detection systems 100 and 150 may optionally include a pose estimation function 113 and an eye aspect ratio (EAR) estimation function 114. The pose estimation function can estimate the pose of the feature based on the 2D image coordinates 112 of the keypoints of the feature before the keypoint fitting function 116 generates 3D image coordinates 118. The pose estimation function 113 can estimate the pose of the feature, for example, based on the keypoints of the identified features (e.g., using a keypoint detection model 110). If the estimated pose of the feature is such that self-occlusion is unlikely for a particular application (e.g., a frontal head pose), then the EAR technique can provide sufficient accuracy for certain use cases. In response to determining that a feature of a vehicle occupant (e.g., a face) is in a specific pose (e.g., a frontal pose), depth-assisted 3D detection system 100 or depth-assisted 3D detection system 150 can use the EAR estimation function 114 to generate one or more EAR values ​​115. The EAR estimation function 114 can generate EAR values ​​115 based at least on the 2D image coordinates 112 of key points of the vehicle occupant's eyes detected in image data 104. EAR values ​​115 can be generated as a supplement to feature state indication 126 (e.g., for redundancy or integration purposes), or as a replacement for feature state indication 126 (e.g., to conserve computational resources when EAR provides sufficient accuracy). In some examples, EAR values ​​115 can be used at least partially for blink detection, drowsiness detection, and / or gaze detection.

[0043] refer to Figure 2 , Figure 2 This is a block diagram of an example blink detection system 200 according to some embodiments of the present disclosure. Figure 2 As shown, the blink detection system 200 may include one or more blink threshold determination functions 204, one or more eye state detection models 206, one or more blink count determination functions 208, one or more blink rate determination functions 212, and one or more blink duration determination functions 216. It should be understood that this and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to or as alternatives to the arrangements and elements shown, and certain elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and location. The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory.

[0044] The blink detection system 200 can receive an eye-opening value 202 (e.g., from depth-assisted 3D detection systems 100, 150). The eye-opening value 202 can be determined based on 3D global coordinates 122. For example, for a given image, the eye-opening value 202 can be determined based on the distance between the 3D global coordinates 122 of one or more first keypoints (e.g., upper eyelid keypoints) of the vehicle occupant's eye and the 3D global coordinates 122 of one or more second keypoints (e.g., lower eyelid keypoints) of the eye. Since the amount of eye-opening may differ for each eye (e.g., due to differences in eye shape, deformities, etc.), the eye-opening value 202 can be determined individually for each eye.

[0045] The blink detection system 200's blink threshold determination function 204 can determine one or more thresholds related to the blinking of a vehicle occupant based on the eye opening value 202. The blink threshold determination function 204 can determine a corresponding threshold for each eye of the vehicle occupant, defining when that eye is open or closed, and the corresponding threshold for each eye can be determined based on the history of the eye opening value 202 over time, which can establish a pattern of the degree of eye opening for each eye of the vehicle occupant. In some embodiments, the blink threshold determination function 204 can determine the average maximum degree of eye opening (e.g., indicating that the eye is fully open), the average minimum degree of eye opening (e.g., indicating that the eye is closed), or other metrics for determining the thresholds related to the blinking of a vehicle occupant. These metrics can vary over time, depending on conditions inside and / or outside the vehicle. For example, if sunlight shines directly into the eyes of a vehicle occupant, the occupant may squint, which reduces the average maximum degree of eye opening, and the blink threshold can be adjusted accordingly.

[0046] The eye opening value 202 and the threshold from the blink threshold determination function 204 can be provided to the eye state detection model 206 and used to determine various eye states. A first threshold (e.g., associated with the average minimum degree of eye opening) can be used to determine whether the eye opening value 202 indicates that the eye is open or closed. For example, when the eye opening value 202 is greater than the first threshold, the eye can be considered open; while if the eye opening value 202 is less than the first threshold, the eye can be considered closed. A corresponding first threshold can be determined for each eye, and the first threshold can be determined specifically for vehicle occupants. The eye state detection model 206 can output one or more eye state indicators 207 for further use by other system components. The eye state indicator 207 can indicate whether a particular eye is open or closed at a given time.

[0047] For certain measurements of blink duration, a second threshold (e.g., associated with the average maximum degree of eye opening) can be used to determine when a blink begins. For example, a blink can be considered to have begun when the eye opening value 202 drops below the second threshold, and to have ended when the eye opening value 202 rises above the second threshold. For a complete blink, the eye opening value 202 will sequentially drop below the second threshold, drop below the first threshold, rise above the first threshold, and rise above the second threshold. A corresponding second threshold can be determined for each eye, and a second threshold can be specifically determined for vehicle occupants.

[0048] exist Figure 2 In the example shown, the blink detection system 200 also includes a blink count determination function 208, a blink rate determination function 212, and a blink duration determination function 216. The eye state detection model 206 can provide eye state indications 207 to the blink count determination functions 208, 212, and 216 for various blink-related determinations. The blink count determination function 208 can determine one or more blink count values ​​210 based on the eye state indications 207. For example, the blink count value 210 for each eye can be determined based on the number of times the eye state indications 207 indicate that the eye transitions from open to closed and then back to open, corresponding to the eye opening value 202 dropping below a first threshold and then rising above the first threshold again. The blink count determination function 208 can output the blink count values ​​210 for further use by other system components or for automatic annotation of image data 104.

[0049] The blink rate determination function 212 can determine one or more blink rate values ​​214 based on the eye state indication 207. For example, the blink rate value 214 for each eye can be determined based on the number of blinks within a defined time period (e.g., per minute), which corresponds to the number of times the eye opening value 202 drops below a first threshold and rises above a first threshold within the defined time period. The blink rate determination function 212 can output the blink rate value 214 for further use by other system components or for automatic annotation of image data 104.

[0050] The blink duration determination function 216 may determine one or more blink duration values ​​218 based on an eye state indication 207 determined using a first threshold. For example, the blink duration value 218 for each eye may be determined based on the amount of time the eye closes with each blink, which may correspond to the duration for which a given blink opening value 202 is below the first threshold. In some embodiments, the blink duration determination function 216 may determine the blink duration value 218 based on an eye state indication 207 determined using a first threshold and a second threshold. For example, the blink duration value 218 for each eye may be determined based on the amount of time it takes to complete one blink after the blink begins, which may correspond to the amount of time it takes for the opening value 202 to sequentially decrease below the second threshold, decrease below the first threshold, increase above the first threshold, and increase above the second threshold. The blink duration determination function 216 may generate a blink duration value 218 for each eye by averaging (e.g., a moving average) the blink duration determined for each eye over a given number of samples. The blink duration determination function 216 can output a blink duration value 218 for further use by other system components or for automatic annotation of image data 104.

[0051] refer to Figure 3 , Figure 3 This is a block diagram of an example drowsiness detection system 300 according to some embodiments of the present disclosure. Figure 3 As shown, the drowsiness detection system 300 may include one or more eyelid closure percentage (PERCLOS) threshold determination functions 302, one or more PERCLOS determination functions 304, and one or more drowsiness detection models 308. It should be understood that this and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or as alternatives to the arrangements and elements shown, and certain elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and location. The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory.

[0052] exist Figure 3In the example shown, the drowsiness detection system 300 includes a PERCLOS threshold determination function 302. The PERCLOS threshold determination function 302 of the drowsiness detection system 300 can generate one or more PERCLOS thresholds for the eyes of a vehicle occupant based on eye opening values ​​202. A corresponding PERCLOS threshold can be determined for each corresponding eye, and a corresponding PERCLOS threshold can be specifically determined for the vehicle occupant. The PERCLOS threshold determination function 302 can determine a corresponding PERCLOS threshold for each eye of the vehicle occupant, which defines when the corresponding eye is closed for more than a specific percentage (e.g., 60%, 70%, 80%, etc.), and the corresponding PERCLOS threshold for each eye can be determined based on the history of eye opening values ​​202 over time, which can establish a pattern of eye opening degree for each eye of the vehicle occupant. In some embodiments, the PERCLOS threshold determination function 302 can determine the PERCLOS threshold based on the average maximum degree of eye opening (e.g., indicating eyes fully open) and the average minimum degree of eye opening (e.g., indicating eyes closed). These metrics can vary over time, depending on conditions inside and / or outside the vehicle. For example, if sunlight shines directly into the eyes of vehicle occupants, they may squint, which reduces the average maximum degree of eye opening, and the PERCLOS threshold can be adjusted accordingly.

[0053] The eye-opening value 202 and the PERCLOS threshold from the PERCLOS threshold determination function 302 can be provided to the PERCLOS determination function 304 and used to determine one or more PERCLOS values ​​306. The PERCLOS threshold (e.g., associated with the percentage of eye openness) can be used to determine whether the eye-opening value 202 indicates that the eye is at a specific percentage of closure. For example, if the eye-opening value 202 is less than the PERCLOS threshold corresponding to 80% eye closure for a particular eye, then the eye can be considered to be 80% closed. The PERCLOS determination function 304 can output the PERCLOS value 306 for further use by other system components (e.g., by the drowsiness detection model 308). In some embodiments, the PERCLOS value 306 generated by the PERCLOS determination function 304 can indicate whether the eye-opening value 202 is below a corresponding PERCLOS threshold. The PERCLOS value 306 can indicate the percentage of time intervals (e.g., minutes) during which the eye-opening value 202 is below the corresponding PERCLOS threshold.

[0054] The drowsiness detection model 308 may receive eye-opening value 202, PERCLOS value 306, blink count value 210, blink rate value 214, and / or blink duration value 218. In some embodiments, the drowsiness detection model 308 may determine the level of drowsiness based on the eye-opening value 202, PERCLOS value 306, blink count value 210, blink rate value 214, and / or blink duration value 218, and generate one or more drowsiness notifications 310 (e.g., if the drowsiness detection model 308 determines that the level of drowsiness exceeds a threshold). The drowsiness detection model 308 may include machine learning models trained to determine the level of drowsiness based at least on metrics input to the drowsiness detection model 308. The drowsiness detection model 308 may output drowsiness notifications 310 to one or more other systems. For example, drowsiness notifications 310 may be provided to a vehicle monitoring system that alerts occupants to potential signs of fatigue. In some embodiments, drowsiness notification 310 may be provided to the vehicle's control system (e.g., controller 636 of vehicle 600), and the control system may adjust the vehicle's operation based on drowsiness notification 310. For example, the control system may take over control of the vehicle and pull it over to the side of the road.

[0055] refer to Figure 4 , Figure 4 This is a block diagram of an example gaze detection system 400 according to some embodiments of the present disclosure. Figure 4 As shown, the gaze detection system 400 may include one or more pupil detection models 402 and one or more gaze estimation models 404. It should be understood that this and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to or as alternatives to those shown, and certain elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and location. The various functions performed by these entities can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory.

[0056] The pupil detection model 402 of the gaze detection system 400 can receive 3D global coordinates 122 (e.g., from the depth-assisted 3D detection system 100). In some embodiments, the pupil detection model 402 can receive the 3D global coordinates 122 of all detected keypoints (e.g., using a keypoint detection model 110), and the pupil detection model 402 can filter the detected keypoints to retain only keypoints around the eyes. The pupil detection model 402 can determine the relative position of the occupant's pupils based on the 3D global coordinates 122 of the pupil keypoints and the surrounding eye keypoints. The pupil detection model 402 may include a machine learning model trained to detect and track eye pupils.

[0057] The gaze estimation model 404 of the gaze detection system 400 can receive the position of the eye pupil generated by the pupil detection model 402 and can estimate the gaze direction of the vehicle occupant based on the position of the eye pupil relative to other ocular key points (e.g., eyelid or corner of the eye key points). The gaze estimation model 404 may include machine learning models trained to estimate the occupant's gaze direction based at least on the position of the eye pupil relative to other ocular key points. The gaze estimation model 404 may output one or more gaze direction indicators 406 to one or more other systems. The gaze direction indicator 406 may include predictions about the target the occupant is looking at (e.g., the infotainment console, the center console, etc.). The gaze direction indicator 406 may be provided to a vehicle monitoring system that alerts the occupant to potential signs of inattention (e.g., if the occupant is looking beyond the road). In some embodiments, the gaze direction indicator 406 may be provided to a vehicle control system (e.g., the controller 636 of vehicle 600), and the control system may adjust the vehicle's operation based on the gaze direction indicator 406. For example, if the gaze direction indicator 406 indicates that the occupants of the vehicle are distracted, the control system can take over control of the vehicle and pull it over to the side of the road.

[0058] Now for reference Figure 5 , Figure 5 This is a flowchart illustrating a method 500 for depth-assisted 3D detection according to some embodiments of the present disclosure. Each block of the method 500 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. The method can also be embodied as computer-usable instructions stored on a computer storage medium. The method can be provided by a standalone application, service, or managed service (alone or in combination with another managed service), or a plug-in to another product, etc. Furthermore, method 500 is illustrated by examples regarding... Figures 1A-4The method is described in this document. However, it may be additionally or alternatively implemented by any system or any combination of systems, including but not limited to the system described herein.

[0059] Method 500 at block B502 includes: acquiring 2D image coordinates of one or more key points of one or more features of a vehicle occupant, based at least on image data captured by an image sensor. The image data (e.g., image data 104) may be captured using one or more image sensors (e.g., image sensor 102), including but not limited to RGB sensors, cameras, or other image sensors. The image sensors may be located near the dashboard, steering wheel, or center console and aimed at the vehicle occupant (e.g., the driver) (e.g., to monitor occupant drowsiness or alertness), and the image data may include at least a portion of the vehicle occupant's face, head, hands, or other features. One or more feature detection models may be used to locate occupant features (e.g., face) in the image data, and one or more key point detection models may be used to identify key points of the occupant's features. For example, facial key points of the occupant may include, but are not limited to, key points of the chin, eyebrows, eyes, mouth, lips, pupils, or ears. Feature detection models and key point detection models can be used to determine the 2D image coordinates of key points of occupant features in the image sensor coordinate system based on image data, and these 2D image coordinates can be continuously generated for monitoring purposes.

[0060] Method 500, at block B504, includes: at least based on fitting 2D image coordinates to a 3D occupant model comprising one or more features, generating first 3D coordinates of one or more keypoints of the one or more features in the coordinate system of an image sensor. The 3D occupant model may include known 3D coordinates of the keypoints of the features, and the 2D image coordinates of the keypoints of the occupant features in the image sensor coordinate system may be fitted to the known 3D coordinates of the keypoints of the 3D occupant model using a perspective n-point (PnP) algorithm to solve for the pose and orientation of the occupant features in the image data. Fitting the known 3D coordinates of the keypoints of the 3D occupant model to the 2D image coordinates of the keypoints of the features may include: applying estimated rotation and translation vectors to transform the known 3D coordinates of the 3D occupant model to the camera coordinate system. The first 3D coordinates in the image sensor coordinate system may be continuously generated for monitoring purposes.

[0061] A 3D occupant model may include a generic 3D model of human features (e.g., the head). In some embodiments, the 3D occupant model may include a personalized 3D model of occupant features, which may be generated while the occupant is seated in the vehicle and guided to perform predefined actions covering multiple viewpoints of their features, while depth sensors are used to capture depth data. In some embodiments, feedback (e.g., visual guidance and / or auditory / voice commands) may be provided to the occupant based on instructions on how to rotate the occupant's features to capture a specified range of viewpoints. Predefined actions may include rotation in three directions, including roll, pitch, and / or yaw. In some embodiments, the depth data captured in this process may be used to adjust the generic 3D model of human features to personalize the model and reflect the occupant's characteristics.

[0062] Method 500 at block B506 includes: generating second 3D coordinates in the global coordinate system for keypoints of the feature, based at least on first 3D coordinates and depth data in the global coordinate system. The depth data in the global coordinate system (e.g., depth point cloud or depth image) (e.g., based on reference points) can be captured synchronously with image frames using one or more depth sensors (e.g., sensor 106). The depth sensors can include, but are not limited to, stereo imaging camera depth sensors, time-of-flight camera depth sensors, millimeter-wave radar sensors, Kinect sensors, or other depth sensors that can be synchronized with image sensors used to capture image data. In addition to or instead of using depth sensors, depth data can also be generated based on stereo information determined using sensor data from two or more sensors (e.g., DMS / OMS sensors) synchronized with the image sensors. In some embodiments, such as when depth sensors or multiple image sensors are unavailable, depth data can be estimated by fitting a plane through a set of keypoints parallel to the feature, the plane moving with the feature at various angles. For example, depth data can be estimated by fitting a plane parallel to the face that passes through a set of key points (e.g., eyelid key points, corners of the eyes, chin, etc.), and the fitted plane moves with the face or head at various angles as the occupant moves their head. Depth data can be estimated by determining the light rays from the image sensor to the fitted plane.

[0063] The second 3D coordinates of key points of a feature in the global coordinate system can be generated based on the intersection of the first 3D image coordinates and depth data or estimated depth data. For example, the second 3D coordinates of facial key points in the global coordinate system (e.g., 3D global coordinates 122) can be generated based on the intersection of a 3D model of the head represented by the first 3D image coordinates and depth data or estimated depth data. Similar to the first 3D coordinates in the image sensor coordinate system, the second 3D coordinates of key points of a feature in the global coordinate system can be continuously generated for monitoring purposes.

[0064] Method 500, at box B508, includes: determining the state of a vehicle occupant's features based at least on second 3D coordinates. The state of the occupant's features can be continuously determined using the second 3D coordinates in a global coordinate system. If the vehicle occupant's features include eyes, the state can include eye opening value, eye open / closed state, and / or other states. The eye opening value can be determined based on the distance between the second 3D coordinates of corresponding eyelid keypoints (e.g., upper eyelid keypoints and lower eyelid keypoints) of the vehicle occupant. The eye open / closed state can be determined based on whether the eye opening value is below a threshold, which can be specific to the vehicle occupant and can be specific for each eye (e.g., because the eye opening values ​​of the occupant's left and right eyes can differ from each other when fully open or closed). The threshold for each eye of the vehicle occupant can be determined over time based on the occupant's baseline (e.g., the average of previous samples or a calibration procedure) and other factors.

[0065] Method 500, at box B510, includes outputting one or more indications of the state of a vehicle occupant's characteristics. The indications of the state of the vehicle occupant's characteristics may be output to one or more systems within the vehicle for further processing. For example, the indications of the state of the characteristics may be provided to a blink detection system (e.g., blink detection system 200), which may determine blink count, blink rate, and / or blink duration based on eye opening value or eye opening / closing state. Other indicators (e.g., percentage of eyelid closure (PERCLOS)) may also be determined based on eye opening value or eye opening / closing state. A drowsiness detection system (e.g., drowsiness detection system 300) may receive blink indicators or other indicators and generate one or more notifications indicating that a vehicle occupant (e.g., a driver) may be showing signs of drowsiness, and may adjust vehicle operation and / or provide notifications to the occupant.

[0066] The systems and methods described herein can be used by (but are not limited to) non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, water vehicles, shuttles (e.g., autonomous taxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, by way of example but not limited to machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and supervision, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets (e.g., NVIDIA's Omniverse), cloud computing and / or any other suitable application.

[0067] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines, etc.), systems implemented using robots, aviation systems, medical systems, marine systems, smart zone monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing language models (such as large language models (LLM), visual language models (VLM), and / or multimodal language models), systems using or deploying one or more inference microservices, systems containing one or more machine learning models deployed in services or microservices and OS-level virtualization packages (e.g., containers), systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation for 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0068] Example autonomous or semi-autonomous machines Figure 6AExamples of sensor locations with corresponding fields of view or sensing fields of view of an autonomous or semi-autonomous vehicle 600A, an autonomous mobile robot (AMR) 600B, and a humanoid robot 600C, according to some embodiments of this disclosure. While three types of machines 600 are shown, this is not intended to be limiting, and the machines 600 described herein may include vehicles, cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police or emergency vehicles, ambulances, boats, construction vehicles, underwater vehicles, robots (e.g., AMRs, humanoid robots, robotic arms, end effectors, forklifts, etc.), drones, aircraft, vehicles coupled to trailers (e.g., semi-trailer tractors for hauling goods), and / or another type of vehicle or machine (e.g., driverless and / or vehicles or machines accommodating one or more passengers). In some cases, vehicle 600A, AMR 600B, humanoid robot 600C, and / or other machine types may be collectively referred to herein as machine 600.

[0069] Regarding Vehicle 600A, autonomous and semi-autonomous vehicles are typically described by automation levels, defined by the National Highway Traffic Safety Administration (NHTSA) (a division of the U.S. Department of Transportation) and the Society of Automotive Engineers (SAE) in their "Classification and Definition of Terms Related to Driving Automation Systems for Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 600 may have one or more functions that meet Level 3 through Level 5 of autonomous driving. For example, Vehicle 600 may be able to provide driver assistance (Level 1), partial automation (Level 2, Level 2+, Level 2++), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the specific implementation. The term “autonomy” as used herein can include any and / or all types of autonomy of Machine 600 or other machines, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, provision of auxiliary autonomy, semi-autonomy, primary autonomy or other specified autonomy.

[0070] about Figure 6AThe sensors and their respective fields of view (not shown for clarity) or sensing fields (not shown for clarity) are an example embodiment and are not intended to be limiting. Although not shown, each sensor may have a corresponding field of view (e.g., a 360-degree field of view around camera 668D, a 180-degree field of view of wide-angle camera 668B, a 360-degree sensing field of LiDAR sensor 664, etc.). For example, only a subset of the sensors shown may be included, additional sensors may be included, alternative sensors may be included, the number of each sensor mode may be different, the sensor modes may be different (e.g., LiDAR or RADAR may not be included, SONAR, thermal sensors, etc. may be included), and the sensor locations may differ from those shown on vehicles 600A, AMR 600B, and / or humanoid robots 600C, etc. For example, for vehicle 600A, the location, number, modes, and / or other sensor information may differ depending on the type (e.g., SUV, truck, car, robot, motorcycle, etc.), size (e.g., 18-wheeler, transport vehicle, small car, etc.), and associated functions (e.g., L2 vs. L5). Similarly, for the AMR 600B and / or the humanoid robot 600C, the shape, size, purpose, implementation method, model, etc., can determine the number and type of sensors used.

[0071] like Figure 6A As shown, autonomous or semi-autonomous vehicles 600A, AMR 600B, and humanoid robots 600C may include different sensor types, numbers, and locations. As a non-limiting example, vehicle 600A may include twelve cameras 668, such as a front wide-angle camera (e.g., 120-degree field of view (FOV)), a front telephoto camera (e.g., 30-degree FOV), a side-rear left camera (e.g., 70-degree FOV), a side-rear right camera (e.g., 70-degree FOV), a front fisheye camera (e.g., 200-degree FOV), a rear fisheye camera (e.g., 200-degree FOV), a left fisheye camera (e.g., 200-degree FOV), a right fisheye camera (e.g., 200-degree FOV), a front telephoto satellite camera (e.g., 30-degree FOV), a rear telephoto camera (e.g., 30-degree FOV), a cross-left camera (e.g., 120-degree FOV), and a cross-right camera (e.g., 120-degree FOV). In one embodiment, camera 668 may use a gigabit multimedia serial link (GMSL) interface (e.g., GMSL2) as input / output (I / O).

[0072] In some embodiments, although Figure 6AAs not shown, vehicle 600A may include an in-cabin occupant and / or driver monitoring system, which may include various sensors. For example, in-cabin sensors may include various cameras 668, such as a driver monitoring camera (e.g., located in front of the driver's seat and facing the driver's seat at a 55-degree FOV), a front occupant monitoring camera (e.g., located in front of the front occupant seat and facing the front occupant seat at a 190-degree FOV), and a rear occupant monitoring camera (e.g., located in front of the rear occupant seat and facing the rear occupant seat at a 190-degree FOV). Similar to external cameras 668, in embodiments, internal cameras 668 may use a GMSL (e.g., GMSL2) interface for I / O.

[0073] As another non-limiting example, vehicle 600A may also include nine RADAR sensors 660. For example, vehicle 600A may include a front center imaging RADAR sensor (e.g., 120-degree FOV or sensing field), a left front corner RADAR sensor (e.g., 160-degree FOV or sensing field), a right front corner RADAR sensor (e.g., 160-degree FOV or sensing field), a right rear corner RADAR sensor (e.g., 160-degree FOV or sensing field), a left RADAR sensor (e.g., 160-degree FOV or sensing field), a right RADAR sensor (e.g., 160-degree FOV or sensing field), a left rear RADAR sensor (e.g., 50-degree FOV or sensing field), and a right rear RADAR sensor (e.g., 50-degree FOV or sensing field). In embodiments, the RADAR sensors 660 may use an Ethernet interface as I / O.

[0074] As a non-limiting example, the vehicle 600A may also include twelve ultrasonic sensors 662. Figure 6A As shown, the ultrasonic sensor can be placed along the front and rear bumpers of vehicle 600A and along the sides of vehicle 600A, and can be used to detect objects (static and dynamic) close to vehicle 600A. In some embodiments, the ultrasonic sensor 662 can use a DS13 interface as I / O.

[0075] As a non-limiting example, vehicle 600A may also include a LiDAR sensor 664, such as a front-center LiDAR sensor (e.g., a 120-degree horizontal FOV or sensing field and a 30-degree vertical FOV or sensing field). In some embodiments, such as when using additional or alternative LiDAR sensors, the LiDAR sensor may have different horizontal and vertical fields of view or sensing fields. For example, LiDAR sensor 664 may include a 360-degree horizontal FOV or sensing field (e.g., located in a rotating LiDAR sensor) and a 90-degree vertical FOV or sensing field. In some embodiments, LiDAR sensor 664 may use an Ethernet interface as I / O.

[0076] As a non-limiting example, the autonomous mobile robot (AMR) 600B may include three LiDAR sensors 664. For example, the topmost LiDAR sensor 664 may include a beam or 3D LiDAR sensor (e.g., a 360-degree horizontal and 90-degree vertical FOV or sensing field), and the front and rear LiDAR sensors may include planar or 2D LiDAR sensors (e.g., a 180-degree horizontal FOV or sensing field).

[0077] As a non-limiting embodiment, the AMR600B may further include eight cameras 668, such as a front stereo camera (e.g., 120-degree FOV), a rear stereo camera (e.g., 120-degree FOV), a left stereo camera (e.g., 120-degree FOV), a right stereo camera (e.g., 120-degree FOV), a front fisheye camera (e.g., 202-degree ± 3-degree FOV), a rear fisheye camera (e.g., 202-degree ± 3-degree FOV), a left fisheye camera (e.g., 202-degree ± 3-degree FOV), and a right fisheye camera (e.g., 202-degree ± 3-degree FOV).

[0078] The AMR600B may also include a charging port, charging port contacts, status indicators, one or more (e.g., four) RGB LEDs, one or more IMU sensors 666, a magnetometer, and a barometer. The AMR 600B is capable of high-precision time synchronization between sensors using hardware timestamps and PTPs over Ethernet with sensor acquisition times of less than 10 microseconds. In embodiments, the AMR 600B provides simultaneous camera capture within 100 microseconds of a single hardware trigger on all cameras 668, and can write sensor captures to disk at a rate of 4 GB / s for writing packets (e.g., writing to ROSbags of the Robot Operating System (ROS)). Therefore, the AMR 600B is capable of running ROS (e.g., NVIDIA's ISAACROS), can be remotely operated (as described herein), can map the environment, and can navigate the environment using vision cameras 668, LiDAR 664, and / or other sensor types or modalities.

[0079] The humanoid robot 600C may include (as a non-limiting example) a LiDAR sensor 664. For example, the LiDAR sensor 664 may include a beam or a 3D LiDAR sensor (e.g., a 360-degree horizontal and 90-degree vertical FOV or sensing field), or it may include a planar or 2D LiDAR sensor (e.g., a 180-degree horizontal FOV or sensing field).

[0080] As a non-limiting embodiment, the humanoid robot 600C may also include four cameras 668, such as a front stereo camera (e.g., 120-degree FOV), a rear stereo camera (e.g., 120-degree FOV), a front fisheye camera (e.g., 202-degree ± 3-degree FOV), and a rear fisheye camera (e.g., 202-degree ± 3-degree FOV).

[0081] As a non-limiting embodiment, the humanoid robot 600C may also include four ultrasonic sensors 662, such as a left arm ultrasonic sensor, a right arm ultrasonic sensor, a left leg ultrasonic sensor, and a right leg ultrasonic sensor.

[0082] The humanoid robot 600C may also include any number of actuators, such as those allowing control and manipulation of joints. For example, the humanoid robot 600C may include actuators that allow for various degrees of freedom (DoF) depending on the design. In a non-limiting embodiment, the humanoid robot 600C may have a total of 40 degrees of freedom (DoF) (e.g., 6DoF x2 for arms, 6DoF x2 for hands, 6DoF x2 for legs, 2DoF for the torso, and 2DoF for the neck). Actuators can convert energy into physical motion, thereby allowing actions such as joint movement, locomotion, and grasping / manipulation. For example, motors and servos can be used to perform joint movements to control the rotation of joints in an arm or manipulator and allow for reaching, grasping, and manipulating objects. Locomotion can be achieved by moving through the environment using wheels, tracks, or other mobility devices (robot legs). Grasping and manipulation can be performed using end effectors or hands / fingers, which can be equipped with actuators to grasp objects, apply forces, and perform specific tasks. In some examples, the humanoid robot 600C may include position and orientation sensors, such as encoders, gyroscopes, etc., to determine the robot 600C's position in space, thereby enabling position determination and motion tracking. In embodiments, the humanoid robot 600C may include force and pressure sensors to detect environmental interactions, enabling the robot 600C to grasp objects with appropriate force and avoid obstacles along its path. Perception sensors (e.g., cameras, LiDAR, RADAR, ultrasound, sonar (SONAR), etc.) may be used in conjunction with tactile sensors to enable the robot 600C to perceive objects, shapes, and textures, and to know when to begin and stop touching (in conjunction with force sensors that regulate the force used during touching). As a non-limiting example, the humanoid robot 600C may have a height of approximately 1-2 meters (e.g., 1.7 meters or 5 feet 6 inches), a weight of 50-70 kilograms, be able to move at speeds of 8 km / h or higher, and be able to carry a payload of 20-100 kilograms, depending on the system design and requirements.

[0083] In embodiments, the humanoid robot 600C may include a dialogue system—such as a dialogue system driven by a language model (e.g., LLM, VLM, MMLM, VLA, etc.)—to help understand the environment, reason, and communicate with humans, animals, devices, and / or other robots, and / or make planning, control, and navigation decisions. Therefore, in addition to performing various tasks, the humanoid robot 600C can also use onboard sensors, microphones, and speakers to understand speech, audio, and visual cues, while also being able to communicate with the environment.

[0084] Referring to camera 668 of machine 600, the camera type of camera 668 may include, but is not limited to, digital cameras applicable to components and / or systems of machine 600. For vehicle 600A implementation, camera 668 may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera may be capable of any image capture rate, such as 30 frames per second (fps), 60 fps, 120 fps, 240 fps, etc., depending on the embodiment. The camera may use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red colorless colorless colorless (RCCC) color filter array, a red colorless colorless blue (RCCB) color filter array, a red blue green colorless (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a colorless pixel camera (e.g., a camera with an array of RCCC, RCCB, and / or RBGC color filters) can be used to improve light sensitivity.

[0085] The field of view includes cameras (e.g., front-facing cameras) in front of the machine 600, which can be used for surround view to help identify the path and obstacles ahead, and, with the help of one or more controllers 636 and / or control SoCs, provide information crucial for generating an occupancy grid and / or determining preferred machine movement, trajectory, and / or path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including lane departure warning (“LDW”), adaptive cruise control (“ACC”), and / or other functions such as traffic sign recognition.

[0086] Various cameras can be used in front-mounted configurations, including, for example, monocular camera platforms that include complementary metal-oxide-semiconductor (“CMOS”) color imagers. Another example could be a wide-angle camera 668B, which can be used to perceive objects entering the field of view from the periphery (e.g., pedestrians, warehouse vehicles, other robots, pedestrian traffic, or bicycles). Furthermore, any number of long-range cameras 668E (e.g., a pair of long-range stereo cameras) can be used for depth-based object detection, especially for objects for which neural networks have not yet been trained. The long-range camera 668E can also be used for object detection and classification, as well as basic object tracking.

[0087] Any number of stereo cameras 668A can also be included in front-mounted and / or other (e.g., rear-mounted) configurations. In at least one embodiment, one or more stereo cameras 668A may include an integrated control unit comprising a scalable processing unit that can provide programmable logic (“FPGA”) and a multi-core microprocessor with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the machine 600 environment, including distance estimates of midpoints in the image (e.g., parallax or depth images). Alternative stereo cameras 668A may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip that can measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 668A may be used in addition to those described herein, or as an alternative to those described herein. For example, in some embodiments, cameras other than stereo cameras (e.g., two monocular cameras with at least partially overlapping fields of view) may be used to perform stereo depth estimation.

[0088] Cameras with a field of view including portions of the side environment of machine 600 (e.g., side-view cameras) can be used, for example, for surround view, providing information for creating and updating occupancy grids, and generating side collision warnings and / or indicating to AMR 600B or humanoid robot 600C, for example, the presence of objects, features, and / or people on the side. For example, surround cameras 668D can be mounted on machine 600. Surround cameras 668D can include wide-angle cameras 668B, fisheye cameras, 360-degree cameras, etc. For example, four fisheye cameras can be mounted on the front, rear, and sides of machine 600. In an alternative arrangement, machine 600 can use three surround cameras 668D (e.g., left, right, and rear), and can utilize one or more other cameras (e.g., front-facing cameras) as a fourth surround-view camera.

[0089] A camera 668 (e.g., a rear-view camera) having a field of view including a portion of the environment behind the machine 600 can be used to understand objects, features, people, and / or other information behind the machine 600, such as for parking assistance, surround view, rear collision warning, planning, control, and navigation determination, and / or to create and update occupancy grids, BEV images representing the environment, height maps, etc. A wide variety of cameras 668 can be used, including but not limited to those also suitable for use as front cameras (e.g., long-range and / or mid-range cameras 668E, stereo cameras 668A, infrared cameras 668C, etc.), rear cameras, side cameras, downward cameras, upward cameras, and / or similar cameras 668, as described herein.

[0090] Similarly, for LiDAR sensor 664, RADAR sensor 660, ultrasonic sensor 662 and / or other sensor modes or types, the location and placement of the sensors and their corresponding fields of view or sensing fields can be determined based on the use case, implementation or design of the particular machine 600.

[0091] For example, machine 600 includes a RADAR sensor 660, which can be used by machine 600 for long-range object detection, even in dark and / or inclement weather conditions. In embodiments, the RADAR functional safety level may be ASIL B. RADAR sensor 660 can be controlled and accessed for object tracking data using CAN and / or bus 602 (e.g., for transmitting data generated by RADAR sensor 660), and in some examples, raw data can be accessed using Ethernet. Various types of RADAR sensors can be used. For example, but not limited to, RADAR sensor 660 can be suitable for front, rear, and side radar applications. In some examples, a pulse Doppler RADAR sensor is used.

[0092] The RADAR sensor 660 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range lateral coverage, etc. In some examples, long-range radar can be used for adaptive cruise control (ACC) functions. A long-range RADAR system can provide a wide field of view, for example, within a 250m range, achieved through two or more independent scans. The RADAR sensor 660 can help distinguish between static and moving objects; ADAS systems can use it for emergency braking assistance and forward collision warning, and robots can use it to detect dynamic objects in various environments (e.g., low-light or no-light environments). A long-range RADAR sensor can include a single static multimodal RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record the machine 600's surroundings at higher speeds while minimizing peripheral interference (e.g., traffic from adjacent lanes). The other two antennas expand the field of view, enabling rapid detection of objects entering or leaving the machine's direct path (e.g., lanes).

[0093] Mid-range RADAR systems can include, for example, a range of up to 660m (front) or 80m (rear), and a field of view of up to 42 degrees (front) or 150 degrees (rear). Short-range RADAR systems can include, but are not limited to, RADAR sensors designed to be mounted at both ends of a side surface (e.g., rear bumper), allowing continuous monitoring of blind spots behind and beside a machine (e.g., vehicle, robot, etc.) using two beams. Therefore, short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0094] Machine 600 may also include an ultrasonic sensor 662. The ultrasonic sensor 662 may be located at the front, rear, and / or sides of machine 600 and may be used to assist near-field sensing, such as for parking assistance, collision avoidance (e.g., for robot parts), and / or to create and update occupancy grids, evidence mesh maps (EGMs), height maps, BEV images, and / or other representations of objects and features in the machine 600 environment. Multiple ultrasonic sensors 662 may be used, and different ultrasonic sensors 662 may be used for different detection ranges (e.g., 2.5 m, 4 m). For example, the ultrasonic sensor 662 may operate at an ASIL B functional safety level.

[0095] Machine 600 may include a LiDAR sensor 664. The LiDAR sensor 664 can be used for object and feature detection, pedestrian and other robot detection, emergency braking, collision avoidance, simultaneous localization and mapping (SLAM), free space detection, and / or other functions. In embodiments, the LiDAR sensor 664 may be of functional safety level ASIL B. In some examples, machine 600 may include multiple LiDAR sensors 664 (e.g., two, four, six, etc.), which can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0096] In some examples, the LiDAR sensor 664 may be able to provide a list of objects and their distances within a 360-degree field of view. For example, a commercially available LiDAR sensor 664 may have an advertised range of approximately 600m, an accuracy of 2cm-3cm, and support for a 600Mbps Ethernet connection. In some examples, one or more non-protruding LiDAR sensors 664 may be used. In such examples, the LiDAR sensor 664 can be implemented as a small device that can be embedded in the front, rear, side, top, and / or corner of a machine 600. In such examples, the LiDAR sensor 664 can provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, with a range of up to 200m even for low-reflectivity objects. A front-mounted LiDAR sensor 664 can be configured with a horizontal field of view between 45 and 135 degrees.

[0097] In some examples, LiDAR technology, such as 3D flash LiDAR, can also be used. 3D flash LiDAR uses a laser flash as a transmission source, illuminating the environment around a vehicle up to approximately 200m away. The flash LiDAR unit includes a receiver that records the laser pulse transmission time and the reflected light on each pixel, which in turn corresponds to the distance from the vehicle to the object. Flash LiDAR allows for the generation of highly accurate and distortion-free images of the surrounding environment with each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of machine 600. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras with no moving parts other than a fan (e.g., non-scanning LiDAR devices). Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of 3D distance point clouds and co-registration intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, LiDAR sensor 664 may be less susceptible to motion blur, vibration, and / or shock.

[0098] Figure 6BThis is an illustration of the location of sensors and components of an example autonomous or semi-autonomous vehicle 600A (also referred to herein as "vehicle 600", "self-vehicle 600", "self-machine 600", or "machine 600") according to some embodiments of this disclosure. While vehicle 600A is shown in the figure, this is not intended to be limiting, and similar components and / or sensors may be included on any other machine type without departing from the scope of this disclosure. For example, similar sensors and / or components may be used in vehicles, cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police cars, ambulances, ships, construction vehicles, underwater vehicles, robots (e.g., AMRs, humanoid robots, robotic arms, end effectors, forklifts, etc.), drones, aircraft, vehicles coupled to trailers (e.g., semi-trailers for hauling goods), and / or another type of vehicle or machine (e.g., driverless and / or capable of accommodating one or more passengers).

[0099] Figure 6CThis is a block diagram of an example system architecture for a machine 600 (e.g., an autonomous or semi-autonomous vehicle 600A, an autonomous mobile robot (AMR) 600B, a humanoid robot 600C, and / or other types of machines) according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are presented as examples only. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of the arrangements shown, and some elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components or together with other components, and can be implemented in any suitable combination and location (e.g., on a local device, vehicle, or edge machine, in the field (e.g., a locally hosted server), in a remote location (e.g., in one or more computing or server devices in one or more data centers in the cloud) and / or other locations). The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, microcontroller, embedded processor, digital signal processor (DSP), image signal processor (ISP), physical processing unit (PPU), field-programmable gate array (FPGA), accelerators (e.g., deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural network accelerator (NNA) and / or neural processing unit (NPU), programmable vision accelerator (PVA), optical flow accelerator (OFA), etc.), application-specific integrated circuit (ASIC), data processing unit (DPU), quantum processor, etc.) executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 6A to 6E Example machine 600 Figure 7 Example calculation of ecosystem 700, Figure 8 Example Generative Language Model System 800 and / or Figure 9 The example computing device 900 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the task.

[0100] Figure 6CEach component, feature, and system of machine 600 is shown connected via bus 602 (or referred to as "machine communication network 602" or simply "communication network 602"). Bus 602 may include a Controller Area Network (CAN) data interface (or referred to herein as "CAN bus"). CAN can be a network within machine 600 used to help control various features and functions of machine 600, such as brake actuation, acceleration, braking, steering, windshield wipers, etc. CAN bus can be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CANID). CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button position, and / or other vehicle status indicators. CAN bus may conform to ASIL B standard. In some embodiments, in addition to or as an alternative to the CAN bus, bus 602 may include FlexRay, embedded buses (e.g., SPI, I2C), local interconnect links (LIN), NVIDIA's NVLink, USB (2.0, 3.0 and above), radio frequency (RF), Ethernet (e.g., 10BASE / 100BASE, 1000BASE, 10G, etc.), and / or other communication protocols or functions. Furthermore, while a single line is used to represent bus 602, this is not limiting. For example, there can be any number of buses 602, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 602 may be used to perform different functions and / or for redundancy. For example, a first bus 602 may be used for a collision avoidance function, while a second bus 602 may be used for actuation control. In any example, each bus 602 may communicate with any component of machine 600, and two or more buses 602 may communicate with the same component. In some examples, each computer or computing engine within each SoC 604, each controller 636, and / or machine 600 can access the same input data (e.g., input from sensors in the machine 600) and can be connected to a common bus, such as a CAN bus.

[0101] Machine 600 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, batteries, side mirrors, and / or other parts of the vehicle or machine. Machine 600 may include a propulsion system 650, such as an internal combustion engine, a hybrid power plant, an all-electric motor, a hydrogen fuel cell engine, and / or another type of propulsion system. Propulsion system 650 may be connected to the drivetrain of machine 600, which may include a transmission to enable propulsion of machine 600. Propulsion system 650 may be controlled in response to a signal received from throttle / accelerator 652.

[0102] Steering system 654 may include a steering wheel and / or other steering mechanisms (e.g., remote steering and / or local steering) for steering machine 600 (e.g., along a desired path or route) while propulsion system 650 is in operation (e.g., when the vehicle is moving). Steering system 654 may receive signals from steering actuator 656. In some embodiments, a steering wheel or other steering mechanism may be omitted, for example, for machine 600 capable of fully automated (e.g., level 5) functionality.

[0103] The brake sensor system 646 can be used to operate the vehicle brakes in response to signals received from the brake actuator 648 and / or the brake sensor.

[0104] Machine 600 may include one or more controllers 636, such as those described herein. Figure 6AThe controller 636 is described. It can be used for a variety of functions and can be coupled to any of the various other components and systems of the machine 600. For example, the controller 636 can be used to control the machine 600, artificial intelligence performed on the machine 600, infotainment on the machine 600, etc. For example, one controller 636 can be used for some or all of the functions, or different controllers 636 can be used for different functions, for example, to ensure availability and safety separation between various controllers for different tasks. For example, the controller 636 can use a system-calculated plan (e.g., the path or trajectory of vehicle 600A or AMR 600B, or the movement, component trajectory, motion position or displacement, etc. of joints or components (e.g., manipulators, end effectors, limbs, hands, fingers, legs, feet, etc.) of humanoid robot 600C to control the machine 600 in the environment. In some cases, the controller 636 may include a proportional-integral-derivative (PID) controller, a fuzzy logic controller, a neural controller (e.g., a controller embodied as one or more neural networks), a force control controller, a programmable logic controller (PLC), and / or other types of controllers. For example, in the humanoid robot 600C, the controller 636 can act as the brain, responsible for analyzing sensor data, making decisions, and sending commands to the actuators. The controller 636 may include a low-level controller that handles basic motor control, ensuring accurate and precise movement of the individual joints and actuators. The controller 636 may also include a high-level controller to coordinate multiple actuators and sensors, plan complex movements, and adapt to changing environments.

[0105] In some embodiments, controller 636 may include an artificial intelligence controller that can use AI algorithms (e.g., DNN, MLM, etc.) to learn, make decisions, and autonomously perform tasks of machine 600. In some embodiments, controller 636 may use a fixed open-loop control algorithm and not adjust its actions according to the environment. In other embodiments, closed-loop control may be used, which incorporates a feedback mechanism to monitor the robot's performance and make necessary adjustments. In examples, controller 636 may implement reactive control to directly respond to sensor inputs, enabling rapid reflexive actions and real-time changes. Furthermore, in some examples, deliberate control may be implemented, using internal models and planning algorithms to generate advanced actions, which may be suitable for complex tasks requiring reasoning, decision-making, and long-term planning.

[0106] The controller 636 may include one or more system-on-chip (SoC) 604 ( Figure 6C and Figure 6DThe controller 636 may provide signals (e.g., signals representing commands or messages) to one or more components and / or systems of the machine 600, including components such as CPUs, GPUs, and accelerators. Although the controller 636 is listed separately from the SoC 604, this is not intended to be limiting, and in some embodiments, one or more components of the SoC 604 may perform the operations of the controller 636. For example, the controller may send signals to operate machine brakes via one or more brake actuators 648, to operate steering system 654 via one or more steering actuators 656, to operate propulsion system 650 via one or more throttle / accelerators 652, and so on. The controller 636 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous or semi-autonomous navigation and movement and / or assist human operators using the machine 600. Controller 636 may include a first controller 636 for autonomous control and navigation functions, a second controller 636 for functional safety functions, a third controller 636 for artificial intelligence functions (e.g., computer vision), a fourth controller 636 for infotainment functions, a fifth controller 636 for redundancy in emergency situations, and / or other controllers. For example, hardware for safety monitoring and other safety functions (e.g., functional safety islands) may be discrete or partitioned (physically or by processing separation) relative to hardware for processing sensor data to make perception and vehicle control decisions. Similarly, hardware for controlling in-vehicle infotainment and / or in-cabin monitoring (e.g., controllers, SOCs, etc.) may be separate or independent from hardware for vehicle perception and control. In some examples, a single controller 636 may handle two or more of the above functions, two or more controllers 636 may handle a single function, and / or any combination thereof. In some embodiments, the controller 636 may provide signals for controlling one or more components and / or systems of the machine 600 in response to a drowsiness notification 310 provided by the drowsiness detection system 300 and / or a gaze direction indication 406 provided by the gaze detection system 400.

[0107] The controller 636 may provide signals for controlling one or more components and / or systems of the machine 600 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data can be received from, for example but not limited to, Global Navigation Satellite System (“GNSS”) sensor 658 (e.g., Global Positioning System sensor), RADAR sensor 660, ultrasonic sensor 662, LiDAR sensor 664, Inertial Measurement Unit (IMU) sensor 666 (e.g., accelerometer, gyroscope, magnetic compass, magnetometer, etc.), microphone 696, camera 668 (e.g., stereo camera 668A, wide-angle camera 668B (e.g., fisheye camera), infrared camera 668C, surround camera 668D (e.g., 360-degree camera), long-range and / or medium-range camera 668E and / or other types of cameras), speed sensor 644 (e.g., for measuring the speed of machine 600), vibration sensor 642, steering sensor 640, braking sensor (e.g., as part of braking sensor system 646), actuators and / or other types of sensors.

[0108] One or more controllers 636 may receive input (e.g., represented by input data) from the instrument panel 632 of the machine 600 and provide output (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 634 (e.g., screen, head-up display, mirror display, facial display, robot display, etc.), sound alarms, loudspeakers, speakers, and / or via other components of the machine 600. Outputs may include parameters such as machine speed, rate, time, and... Figure 6C The system can display map data corresponding to map 622 (e.g., from a navigation map, a standard definition (SD) map, a high definition (“HD”) map, etc.), location data (e.g., the location of machine 600, such as its location on map 622), direction, the location of other vehicles (e.g., occupancy map, elevation map, bird's-eye view (BEV) image, grid, etc.), information about objects perceived by the system and their states, system status information, and so on. For example, HMI display 634 can display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.), and / or information about driving maneuvers that the vehicle has performed, is performing, or will perform (e.g., changing lanes now, exiting from exit 34B in two miles, etc.).

[0109] Machine 600 may include one or more System-on-a-Chip (SoC) 604 (in Figure 6D(described in more detail below). SoC 604 may include CPU 606, GPU 608, processor 610, cache 612, accelerator 614, data storage 616, and / or other components and features. SoC 604 can be used to process and provide data for various operations, such as navigation, planning, reasoning, inference, perception, control, and / or actuation operations of machine 600 on various platforms and systems. For example, SoC 604 can process real-time perception data (e.g., from cameras, LiDAR, RADAR, ultrasound, etc.) and map data corresponding to one or more maps 622 (e.g., HD maps, SD maps, navigation maps, occupancy maps, etc.) to perform or assist in performing various operations of machine 600. When using maps and / or AI, maps and / or AI (e.g., model parameter updates, fine-tuning, etc.) are transmitted via network interface 624 from one or more servers (e.g., [unclear]). Figure 6E Refresh and / or update the server 678 (e.g., one or more servers in a cloud-based data center).

[0110] Despite Figures 6A to 6E The SoC 604 is shown, however, additional or alternative components and / or architectures, such as multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiPs), field-programmable gate arrays (FPGAs), heterogeneous integration (HI), and single-board computers (SBCs), may be used without departing from the scope of this disclosure. For example, depending on the type of machine 600, the purpose of machine 600, the model of machine 600, and the capabilities required by machine 600, one or more SoCs 604s and / or alternative architectures and / or components may be used to meet a particular implementation.

[0111] Machine 600 may include CPU 618 (e.g., a discrete CPU or dCPU) which may be coupled to SoC 604 via a high-speed interconnect (e.g., PCIe). CPU 618 may include, for example, an x86 processor. CPU 618 may be used to perform any of a variety of functions, including arbitrating potentially inconsistent results between ADAS sensors and SoC 604, and / or monitoring the status and health of controller 636 and / or infotainment SoC 630.

[0112] Machine 600 may include GPU 620 (e.g., a discrete GPU or dGPU) which may be coupled to SoC 604 via a high-speed interconnect (e.g., NVIDIA's NVLink). GPU 620 may provide additional artificial intelligence capabilities, such as by executing redundant and / or different neural networks, and may be used to train and / or update neural networks based on inputs from sensors (e.g., sensor data) from machine 600.

[0113] The machine 600 may also include a network interface 624, which may include one or more wireless antennas 626 and / or modems (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 624 can be used to establish wireless connectivity with the cloud (e.g., with server 678 and / or other network devices) via the Internet, and with other vehicles and / or computing devices (e.g., passenger client devices). For communication with other vehicles, direct links and / or indirect links (e.g., across networks and via the Internet) can be established between the two vehicles. A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide the machine 600 with information about vehicles near the machine 600 (e.g., vehicles in front, to the side, and / or behind the machine 600). This functionality can be part of the machine 600's cooperative adaptive cruise control function.

[0114] Network interface 624 may include a SoC that provides modulation and demodulation functions and enables controller 636 to communicate over a wireless network. Network interface 624 may include a radio frequency (RF) front-end for up-conversion from baseband to radio frequency (RF) and down-conversion from RF to baseband. Frequency conversion can be performed through well-known processes and / or using superheterodyne processes. In some examples, the RF front-end functionality may be provided by a separate chip. For example, network interface 624 may be able to communicate via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multicarrier (“CDMA2000”), 5G, 6G, and / or other cellular and / or wireless communication standards. The wireless antenna 626 can also enable communication between objects in the environment (such as vehicles, mobile devices, etc.) using local area networks (such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc.) and / or low power wide area networks (“LPWAN”) (such as LoRaWAN, SigFox, etc.).

[0115] Machine 600 may also include data memory 628, which may include off-chip (e.g., outside of SoC 604) storage. Data memory 628 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, Flash, hard disk and / or other components and / or devices capable of storing at least one bit of data.

[0116] The machine 600 may also include a GNSS sensor 658. The GNSS sensor 658 (e.g., a GPS, an auxiliary GPS sensor, a differential GPS (DGPS) sensor, etc.) is used to assist in mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 658 can be used, such as, but not limited to, GPS sensors with a USB connector having an Ethernet-to-serial (RS-232) bridge.

[0117] Machine 600 may also include an IMU sensor 666. In some examples, the IMU sensor 666 may be located at the center of the rear axis of machine 600. The IMU sensor 666 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 666 may include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 666 may include an accelerometer, a gyroscope, and a magnetometer.

[0118] In some embodiments, the IMU sensor 666 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 666 can enable the machine 600 to estimate its heading by directly observing and correlating velocity changes from GPS to the IMU sensor 666, without requiring input from a magnetic sensor. In some examples, the IMU sensor 666 and the GNSS sensor 658 can be combined in a single integrated unit.

[0119] The vehicle may include one or more microphones 696 placed inside and / or around the machine 600. The microphones 696 can be used for emergency vehicle detection and identification, etc.

[0120] Machine 600 may also include vibration sensors 642. Vibration sensors 642 can measure vibrations of machine parts, such as the arm or leg of humanoid robot 600C, or the axle of vehicle 600A or AMR 600B. For example, changes in vibration may indicate changes in roads, walking, or traversable surfaces. In another example, when two or more vibration sensors 642 are used, differences between vibrations can be used to determine friction or slippage on surfaces (e.g., when the vibration difference is between an electrically driven shaft and a freely rotating axle).

[0121] Machine 600 may include ADAS system 638, for example, when machine 600 is vehicle 600A. In some examples, ADAS system 638 may include a dedicated SoC. ADAS system 638 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision or collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), blind spot monitoring (BSM), rear cross traffic warning (RCTW), pedestrian detection, driver monitoring, collision warning system (CWS), traffic sign recognition, speed limit detection, automatic parking, lane centering (LC), high beam safety system and / or other features and functions.

[0122] Machine 600 may also include an infotainment SoC 630 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include one or more discrete components, such as multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiP), heterogeneous integration (HI), single-board computers (SBCs), etc. The infotainment SoC 630 may include a combination of hardware and software that can be used to provide audio (e.g., music, personal digital assistant, navigation commands, news, radio, etc.), video (e.g., television, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., wireless, Wi-Fi, etc.), and / or information services (e.g., navigation system, rear parking assist, radio data system, vehicle-related information (e.g., fuel level, total driving distance, brake fluid level, fuel level, door opening / closing, air filter information, etc.) to the machine 600. For example, the infotainment SoC 630 may be a radio, disk player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, Wi-Fi, steering wheel audio controls, hands-free voice control, head-up display (HUD), HMI display 634, telematics device, control panel (e.g., for controlling various components, features, and / or systems and / or interacting with various components, features, and / or systems), and / or other components. 630 can also be used to provide information to vehicle users (e.g., visual and / or auditory), such as information from ADAS system 638, autonomous driving information (e.g., planned vehicle maneuvers, trajectories), surrounding environment information (e.g., intersection information, vehicle information, road information, etc.) and / or other information.

[0123] The infotainment SoC 630 may include GPU functionality. The infotainment SoC 630 can communicate with other devices, systems, and / or components of the machine 600 via bus 602 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 630 may be coupled to a monitoring MCU, enabling the GPU of the infotainment system to perform some autonomous driving functions in the event of a failure of the main controller 636 (e.g., the main computer and / or backup computer of the machine 600). In such an example, the infotainment SoC 630 may place the machine 600 into a driver-safe stop mode, as described herein.

[0124] In some embodiments, the infotainment system may provide a digital or virtual assistant, which may be voice-only or may have visual components (e.g., in the form of a digital human or digital avatar). The assistant may provide basic functions such as sending text messages, adjusting vehicle settings, controlling music or video, navigation, etc., and / or more advanced functions, such as those supported by one or more language models (e.g., large language model (LLM), visual language model (VLM), multimodal language model (MMLM), etc.). For example, the driver and / or occupants may interact with the assistant in a manner similar to how a user interacts with a language model, such as asking general questions, specific questions, requesting restaurants, gas stations, and / or other recommendations and / or locations, inquiring about vehicle functions or troubleshooting (e.g., asking for tire pressure information, oil change information, battery swap information, etc.). Therefore, machine 600 (whether it is vehicle 600A, AMR 600B, humanoid robot 600C, or other types of machine) may include locally stored language models and / or communicate with remotely hosted language models (e.g., via one or more APIs) to provide users of machine 600 with more detailed and in-depth communication capabilities.

[0125] In some examples, the infotainment SoC 630, SoC 604, and / or another SoC or computing / processing system can perform in-cabin driver and / or occupant monitoring. For example, the computing system can perform facial recognition, and vehicle owner recognition can use data from cameras and / or other sensors to identify the presence of an authorized driver and / or owner of the machine 600. An always-on sensor processing engine can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in a security mode, to disable the vehicle when the owner leaves. In this way, SoC 604 can provide security against theft and / or carjacking.

[0126] In some embodiments, one or more neural networks running on another or a dedicated SoC (e.g., an in-vehicle infotainment or in-vehicle monitoring SoC) can be used to monitor in-cabin monitoring camera sensors. This other or dedicated SoC is configured to recognize in-cabin events and respond accordingly. The in-cabin system can activate cellular services and make phone calls by lip reading, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. The in-cabin system may also include one or more in-cabin AI agents or assistants that can interact with one or more LLMs, VLMs, MMLMs, etc., in the cloud using one or more APIs or plugins. For example, the in-cabin AI agents or assistants can provide directions, vehicle or machine feedback information, answer general questions, handle music / video and / or other requests, activate windows, doors, and / or other vehicle components, etc. Therefore, one or more dedicated SoCs and / or processor groups can be used to perform in-cabin infotainment and / or in-cabin monitoring (e.g., as an occupant monitoring system (OMS)) of the machine 600.

[0127] The device 600 may also include an instrument cluster 632 (e.g., a digital instrument cluster, electronic instrument cluster, digital dashboard, etc.). The instrument cluster 632 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 632 may include a set of instruments such as a speedometer, fuel level, fuel pressure, tachometer, odometer, turn signal indicator, gearshift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 630 and the instrument cluster 632. In other words, the instrument cluster 632 may be included as part of the infotainment SoC 630, and vice versa.

[0128] Figure 6D A computing system according to at least some embodiments of this disclosure (about Figure 6C A block diagram of an example architecture (a subset of the systems described). Although illustrated as SoC 604, this is not intended to be limiting, and the computing system may additionally or alternatively include multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiP), heterogeneous integration (HI), single-board computers (SBCs), and / or other components and / or architectures without departing from the scope of this disclosure.

[0129] The SoC 604 can be an end-to-end platform with a flexible architecture spanning automation levels 2-5, or it can be specifically designed for a particular automation level (e.g., a first SoC 604 for levels 2 to 2++, a second SoC 604 for level 3, a third SoC 604 for level 4, and so on), providing a comprehensive functional safety architecture that leverages and effectively utilizes computer vision, neural network inference, robot planning, control and navigation, ADAS technologies, and more, with diversity and redundancy, to provide a flexible and reliable platform for driving or robot control software stacks, as well as deep learning tools. The SoC 604 can be faster, more reliable, and even more energy-efficient and space-saving than traditional systems. For example, when the accelerator 614 is used in conjunction with the CPU 606, GPU 608, and data repository 616, it can provide a fast and efficient platform for Level 2-5 autonomous vehicles, as well as for the safety planning, navigation, and control of AMR 600B, humanoid robot 600C, and / or other robot or machine types.

[0130] In some embodiments, for example, SoC 604 includes: a GPU 608 having 2000 or more cores (e.g., 2048 cores), 60 or more tensor cores (e.g., 64 tensor cores), and a maximum GPU frequency exceeding 1 GHz (e.g., 1.3 GHz); a CPU 606 including 10 or more cores (e.g., 12 cores), having 64-bit, 3 MB L2 and 6 MB L3 caches, and a maximum frequency of 2 GHz or more (e.g., 2.2 GHz); one or more deep learning accelerators (DLAs); deep learning accelerator clusters (XNNs); neural network accelerators (NNAs) or neural processing units (NPUs) 609 (e.g., two DLAs / XNNs / NNAs / NPUs 609); and a vision accelerator (e.g., a programmable vision accelerator (PVA)) 607. A single SoC 604 may be capable of achieving AI performance of 275 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGX Orin 64GB. The SoC meets these standards and achieves this performance.

[0131] Similarly, in an embodiment, SoC 604 includes: a GPU 608 having 1700 or more cores (e.g., 1792 cores), 50 or more tensor cores (e.g., 56 tensor cores), and a maximum GPU frequency exceeding 900MHz (e.g., 930MHz); a CPU 606 including 8 or more cores (e.g., 8 cores), having 64-bit, 2MB L2 and 4MB L3 cache memory, and a maximum frequency of 2GHz or more (e.g., 2.2GHz); one or more deep learning accelerators (DLAs); a deep learning accelerator cluster (XNN); a neural network accelerator (NNA) or neural processing unit (NPU) 609 (e.g., 2 DLAs / XNNs / NNAs / NPUs 609); and a vision accelerator (e.g., a programmable vision accelerator (PVA) 607), and a single SoC 604 may be capable of achieving AI performance of 200 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGXOrin 32GB SoC meets these standards and achieves such performance.

[0132] In some embodiments, for example, SoC 604 includes: a GPU 608 having 1,000 or more cores (e.g., 1,024 cores), 28 or more tensor cores (e.g., 32 tensor cores) and a maximum GPU frequency exceeding 900 MHz (e.g., 1,173 MHz); a CPU 606 including 8 or more cores (e.g., 8 cores), having 64-bit, 2 MB L2 and 4 MB L3 cache memory, and a maximum frequency of 2 GHz or more (e.g., 2 GHz); one or more deep learning accelerators (DLAs); a deep learning accelerator cluster (XNN); a neural network accelerator (NNA) or neural processing unit (NPU) 609 (e.g., one DLA / XNN / NNA / NPU 609); and a vision accelerator (e.g., a programmable vision accelerator (PVA) 607), and a single SoC 604 may be capable of achieving AI performance of 157 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGX Orin NX 16GB SoC meets these standards and achieves such performance.

[0133] In various embodiments, for example, SoC 604 includes: a GPU 608 having 1000 or more cores (e.g., 1024 cores), 28 or more tensor cores (e.g., 32 tensor cores), and a maximum GPU frequency exceeding 900MHz (e.g., 1020MHz); and a CPU 606 including 6 or more cores (e.g., 6 cores), having 64-bit, 1.5MB L2 and 4MB L3 cache memory, and a maximum frequency of 1.5GHz or higher (e.g., 1.7GHz). A single SoC 604 can achieve AI performance of 67 trillion operations per second (TOPS). For example, NVIDIA's Jetson Orin Nano 8GB SoC meets these standards and achieves such performance.

[0134] SoC 604 may include one or more CPUs 606. In embodiments, CPU 606 may include CPU clusters or CPU complexes (also referred to herein as “CCPLEX”). CPU 606 may include multiple cores and / or (e.g., L2, L3) caches. For example, in some embodiments, CPU 606 may include twelve cores in a coherent multiprocessor configuration. In some embodiments, CPU 606 may include four dual-core clusters, each with a dedicated L2 cache (e.g., 3MB L2 cache). CPU 606 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, thereby allowing any combination of clusters of CPU 606 to be active at any given time.

[0135] SoC 604 may include any type and number of GPUs 608. For example, in some embodiments, an integrated GPU (also referred to herein as an "iGPU") may be used. GPU 608 may be programmable and can be used efficiently for parallel workloads. In some examples, GPU 608 may use an enhanced tensor instruction set. GPU 608 may include one or more streaming microprocessors, wherein each streaming microprocessor may include a cache (e.g., an L1 cache with at least 96KB of storage), and two or more streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage). In some embodiments, GPU 608 may include at least eight streaming microprocessors. GPU 608 may use a computing application programming interface (API). Furthermore, GPU 608 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0136] The GPU 608 can be power-optimized for optimal performance in automotive, robotics, and / or other embedded use cases. For example, the GPU 608 can be fabricated on FinFETs. However, this is not intended to limit it, and the GPU 608 can be fabricated using other semiconductor fabrication methods or processes. Each streaming microprocessor can contain multiple mixed-precision processing cores divided into multiple blocks. For example, but not limited to, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSOR COREs for deep learning matrix algorithms, one (e.g., L0) instruction cache, one warp scheduler, one dispatch unit, and / or one (e.g., 64KB) register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to efficiently perform workloads involving mixed-computation and addressing computations. Streaming microprocessors can include independent thread scheduling capabilities to enable finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors can also include combined L1 data caches and shared memory units to improve performance while simplifying programming.

[0137] The GPU 608 may include high-bandwidth memory (HBM) and / or (e.g., 16GB) HBM2 memory subsystems to provide peak memory bandwidth of approximately 900GB / s in some examples. In some examples, synchronous graphics random access memory (SGRAM), such as graphics double data rate type 5 synchronous random access memory (GDDR5), may be used in addition to or as an alternative to HBM memory.

[0138] The GPU 608 may include unified memory technology, including access counters, to allow more accurate migration of memory pages to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, Address Translation Service (ATS) support may be used to allow the GPU 608 to directly access the CPU 606 page tables. In such examples, when a miss occurs in the GPU 608 Memory Management Unit (MMU), an address translation request may be transferred to the CPU 606. In response, the CPU 606 can look up the virtual-to-physical mapping of the address in its page tables and transfer the translation back to the GPU 608. Therefore, unified memory technology allows the memory of the CPU 606 and GPU 608 to use a single unified virtual address space, thereby simplifying GPU 608 programming and porting applications to the GPU 608.

[0139] SoC 604 may include any number of caches 612, including the caches described herein. For example, cache 612 may include L0 cache, L1 cache, L2 cache, L3 cache (e.g., available for CPU 606 and GPU 608 (e.g., connected to CPU 606 and GPU 608)), etc. Cache 612 may include a write-back cache that can track the state of the line, for example, by using one or more cache coherence protocols (e.g., MEI, MESI, MSI, etc.). According to embodiments, the cache (e.g., L3) may include 4MB or more, although smaller or larger cache sizes may be used.

[0140] SoC 604 may include one or more arithmetic logic units (ALUs) 665, which can be used to perform processing related to various tasks or operations of machine 600, such as computer vision, machine learning or deep learning processing, world model management, etc. Furthermore, SoC 604 may include a floating-point unit (FPU) 667 or other mathematical coprocessors or digital coprocessors for performing mathematical operations within the system. For example, SoC 604 may include one or more FPUs 667, which are integrated as execution units within CPU 606 and / or GPU 608.

[0141] SoC 604 may include one or more accelerators 614 (e.g., hardware accelerators, software accelerators, or combinations thereof). For example, SoC 604 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. Large on-chip memory 615 (e.g., 4MB SRAM, 32GB and / or 64GB 256-bit LPDDR5 (204.8GB / s), 8GB and / or 16GB 128-bit LPDDR5 (102.4GB / s), and / or other memory types and sizes) enables the hardware acceleration cluster to accelerate neural network processing, converter processing, optical flow processing, vision processing, and / or other computations or processing. The hardware acceleration cluster can be used to supplement GPU 608 and offload some tasks from GPU 608 (e.g., freeing up more cycles of GPU 608 to perform other tasks). As an example, Accelerator 614 can be used for target workloads (e.g., perceptual, convolutional neural network (CNN), deep neural network (DNN), language model (LLM, VLM, MMLM, VLA, etc.), transformer model, diffusion model, encoder-only model, encoder-decoder model, etc., which are stable enough to be suitable for acceleration.

[0142] Accelerator 614 (e.g., a hardware acceleration cluster) may include a deep learning accelerator (DLA) 609 (also referred to herein as a “deep learning accelerator cluster (XNN) 609”, a “neural network accelerator (NNA) 609”, or a “neural processing unit (NPU) 609”). DLA 609 may include one or more tensor processing units (TPUs) 641, which may be configured to provide additional, for example, trillions of operations per second for deep learning applications and inference. TPU 641 may be an accelerator configured to perform image processing functions (e.g., for CNNs, RCNNs, DNNs, etc.) and optimized for them. DLA 609 may be further optimized for a specific set of neural network types and floating-point operations and inference. The DLA is designed to provide higher performance per millimeter than general-purpose GPUs and significantly outperforms CPUs. TPU 641 can perform several functions, including single-instance convolution functions, support for INT8, INT16, and FP16 data types for features and weights, and post-processor functions. Although TPU 641 is described as being included as part of DLA 609, this is not intended to be limiting, and TPU 641 may be included in additional or alternative accelerator 614 and / or other components, and / or may be included as a discrete processing component.

[0143] The DLA 609 can quickly and efficiently execute neural networks on processed or unprocessed data to achieve a variety of functions, including but not limited to: object and feature recognition and detection using data from one or more sensor modalities (e.g., vehicles, pedestrians, other robots, lane lines, road boundary lines, debris, potholes, boxes, warehouse items, etc.); distance estimation using data from one or more sensor modalities; emergency vehicle detection and identification using data from microphones and / or vision-based sensors; facial recognition; pick-up and place operations; maneuvering operations; occupant monitoring; vehicle owner identification; and / or other in-cabin operations using data from in-cabin cameras and / or other sensor types; and / or safety and / or safety-related events, to name a few.

[0144] The DLA 609 can perform any function of the GPU 608, and by using inference accelerators, for example, designers can anchor either the DLA 609 or the GPU 608 for any function. For instance, designers can centralize the processing of DNNs and floating-point operations on the DLA 609, leaving other functions to the GPU 608 and / or other accelerators 614. The DLA 609 can be used to run any type of network to enhance control and security, including, for example, neural networks that output a confidence metric for each object detection.

[0145] Accelerator 614 (e.g., a hardware acceleration cluster) may include a programmable vision accelerator (PVA) 607, which may alternatively be referred to herein as a computer vision accelerator or generally as a vision accelerator. The PVA 607 may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), semi-autonomous driving, autonomous driving, robotics applications, safety and supervision applications, augmented reality (AR), virtual reality (VR), and / or mixed reality (MR) applications. The PVA 607 provides a balance between performance and flexibility. For example, each PVA 607 may include (e.g., but not limited to) any number of Reduced Instruction Set Computer (RISC) cores, Direct Memory Access (DMA) systems, Pixel Processing Engines (PPEs), Vector Processors or Vector Processing Units (VPUs), and / or other components. The PVA engine may include an Advanced Very Long Instruction Word (VLIW) or Single Instruction Multiple Data (SIMD) digital signal processor. The PVA 607 may be optimized for image processing and computer vision algorithm acceleration tasks. For example, the PVA607 offers excellent performance with extremely low power consumption and can be used asynchronously and concurrently as part of a heterogeneous computing pipeline with CPU 606, GPU 608 and / or other accelerators in systems such as vehicles, robots and so on.

[0146] The PVA 607 may include one or more (e.g., two) Vector Processing Subsystems (VPSs), each of which may include one or more Vector Processing Unit (VPU) cores, one or more Decoupled Lookup Units (DLUTs), one or more shared or vector memory (VMEMs), and one or more instruction caches (I-caches). The VPU core may be the main processing unit and may include a vector SIMD VLIW DSP 643 optimized for computer vision. The VPU core can fetch instructions via the I-cache and access data via the VMEM. The DLUT may include dedicated hardware components that enhance the efficiency of parallel lookup operations. For example, the DLUT allows parallel lookups using a single copy of the lookup table by performing these lookups in a decoupled pipeline independent of the main processor pipeline. By doing so, the DLUT can minimize or reduce memory usage and increase throughput while avoiding data-related memory library conflicts, ultimately leading to improved overall system performance. The VPU VMEM can provide local data storage for the VPU, allowing for the efficient implementation of various image processing and computer vision algorithms. The VPU VMEM can support access from external VPS hosts, such as Direct Memory Access (DMA) and the CPU 606 (e.g., an ARM Cortex-R5 processor), thereby facilitating data exchange with the CPU 606 and other system-level components. The VPU I-cache can provide instruction data to the VPU upon request, request missing instruction data from system memory, and / or maintain temporary instruction storage for the VPU. For each VPU task, the CPU 606 can configure the DMA system, optionally prefetching the VPU program into the VPU I-cache, and / or initiating each VPU-DMA pair to process the task. The PVA607 may also include L2 SRAM memory shared between one or more (e.g., two) sets of VPS and DMA. In some embodiments, one or more (e.g., two) DMA devices are used to move data between external memory, PVA L2 memory, VMEM (e.g., one per VPS), CPU tightly coupled memory (TCM), DMA descriptor memory, and / or PVA-level configuration registers. In lightly loaded systems, two parallel DMA accesses to DRAM can achieve read / write bandwidth of up to 15 GB / s, while in heavily loaded systems, this bandwidth can reach up to 10 GB / s. Regarding compute capacity, INT8 gigabyte multiply-accumulate operations per second (GMAC) can be 2048 or greater, excluding DLUT. FP32 GMACs can be 32 per PVA instance.

[0147] The RISC core can interact with image sensors (e.g., the image sensor of any camera described herein), image signal processors, etc. Each RISC core may include any amount of memory. The RISC core can use any of a variety of protocols, depending on the implementation. In some examples, the RISC core can execute a real-time operating system (RTOS). The RISC core can be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, the RISC core may include an instruction cache and / or tightly coupled RAM.

[0148] The DMA system enables PVA 607 components to access system memory independently of the CPU 606. DMA can support any number of functions optimized for the PVA 607, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which may include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0149] A vector processor, or VPU, can be a programmable processor designed to efficiently and flexibly execute computer vision algorithms and provide signal processing capabilities. In some examples, the PVA 607 may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA 607 and may include one or more vector processing units (VPUs), one or more pixel processing engines (PPEs) (which may include a 2D layout of interconnected (e.g., for north, south, east, and west communication) processing elements), one or more instruction caches, and / or one or more shared or vector memories (e.g., VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can improve throughput and speed.

[0150] In some embodiments, each vector processor may include an instruction cache and may be coupled to dedicated memory. Therefore, in some examples, each vector processor may be configured to execute independently of other vector processors. In other examples, the vector processors included in a particular PVA 607 may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA 607 may execute the same computer vision algorithm, but for different regions of an image. In other examples, the vector processors included in a particular PVA 607 may execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on consecutive images or portions of an image. Among other things, the hardware acceleration cluster may include any number of PVA 607s, and each PVA may include any number of vector processors. Furthermore, the PVA 607 may include additional error correction code (ECC) memory to enhance overall system security.

[0151] Accelerator 614 (e.g., a hardware accelerator cluster) has broad applications for autonomous and semi-autonomous machine control. PVA 607 can be a programmable vision accelerator used in critical processing stages in perception, robot understanding and reasoning, ADAS, semi-autonomous and autonomous vehicles, etc. PVA 607's capabilities are well-suited for algorithmic domains requiring predictable processing with low power consumption and low latency. In other words, PVA 607 performs well in semi-intensive or intensive rule computation, even on small datasets requiring predictable runtime, low latency, and low power consumption. Therefore, in the context of autonomous vehicles and robotic platforms, PVA 607 is designed to run classic computer vision algorithms, as they are highly efficient in object detection and integer mathematical operations.

[0152] For example, according to one embodiment of this technology, the PVA 607 is used to perform computer stereo vision. In some examples, a semi-global matching-based algorithm can be used, although this is not intended to limit it. Many Level 3-5 autonomous driving applications require on-the-fly motion estimation / stereo matching (e.g., moving structures, pedestrian recognition, lane detection, etc.). The PVA 607 can perform computer stereo vision functions on input from two monocular cameras.

[0153] In some examples, the PVA 607 can be used to perform dense optical flow, providing processed RADAR data based on the raw RADAR data (e.g., using 4D Fast Fourier Transform). In other examples, the PVA 607 is used for time-of-flight depth processing, for example, providing processed time-of-flight data by processing the raw time-of-flight data.

[0154] While the VPU, DMA, RISC Core, VMEM, and decoupled coprocessors (e.g., DLUT) are described as being included within the PVA607, this does not imply limitation. In some embodiments, these components may be included in alternative or additional processing components and / or accelerator 614, and / or may be included as discrete components of the SoC 604 and / or other computing system architectures.

[0155] In some examples, SoC 604 may include a real-time ray tracing hardware accelerator (RTA) 651, which can be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) to generate real-time or near-real-time visualization simulations, for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for simulation of SONAR, RADAR, LiDAR, camera and / or other sensor modalities in a simulation, for general wave propagation simulation, for comparison with LiDAR data for localization, for generating realistic training data for training neural networks, and / or other functions and uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations. For example, machine 600 (or another machine or device) may perform simulations in a simulation environment, and one or more light transport simulation algorithms (e.g., ray tracing, path tracing, etc.) may be used to generate the simulation environment. Therefore, the ray tracing accelerator 651 and / or a ray tracing-optimized GPU 608 (e.g., NVIDIA's RTX GPU) may be used to accelerate these ray tracing algorithms.

[0156] Accelerator 614 (e.g., in a hardware acceleration cluster) may include one or more optical flow accelerators (OFAs) 611. For example, OFA 611 can be used to calculate optical flow and stereo disparity between sensor data frames (e.g., images). Optical flow can be accelerated on OFA 611 for purposes such as object detection and tracking, and / or for stereo depth estimation, where stereo disparity is calculated between stereo image frames (e.g., two or more frames captured using two or more image sensors with at least partially overlapping fields of view).

[0157] SoC 604 may include one or more Camera Serial Interfaces (CSIs) 623. For example, CSI 623 may include a Mobile Industry Processor Interface (MIPI) Camera Serial Interface (CSI) for receiving video and input from cameras, a high-speed interface, and / or a video input block available for camera and associated pixel input functions. SoC 604 may also include a software-controllable input / output controller and may be used to receive I / O signals not assigned to a specific role. For example, CSI 623 may include MIPI CSI-2 connectors, such as a 16-channel MIPI CSI-2 connector, D-PHY 2.1 (up to 40Gbps), and C-PHY 2.0 (up to 164Gbps) to support 16 virtual channels and 6 or more cameras; an 8-channel MIPI CSI-2 connector, D-PHY 2.1 (up to 20Gbps) to support 8 virtual channels and 4 or more cameras; and / or 2x MIPI CSI-2, 22-pin camera connectors, depending on the embodiment and implementation.

[0158] Accelerator 614 (e.g., a hardware acceleration cluster) may include an on-chip computer vision network (CVNOC) 663 and SRAM for providing high-bandwidth, low-latency SRAM to accelerator 614. In some examples, on-chip memory may include at least 4 MB of SRAM, such as, but not limited to, eight field-configurable memory blocks accessible by PVA 607, OFA 611, DLA 609, and / or other accelerators 614. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory 615 may be used. PVA 607, OFA 611, DLA 609, and / or other accelerators 614 may access memory via a backbone providing high-speed memory access to accelerator 614. The backbone may include an on-chip computer vision network that interconnects accelerator 614 to memory (e.g., using APB).

[0159] CVNOC 663 may include an interface that determines whether the accelerator 614 provides ready and valid signals before transmitting any control signals / addresses / data. Such an interface can provide separate stages and separate channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface may conform to ISO 26262 or IEC 61508 standards, although other standards and protocols may be used.

[0160] SoC 604 may include a data repository 616 and / or memory 615. The data repository 616 may be on-chip memory 615 of the SoC 604, which may store neural networks and / or other algorithms to be executed on the CPU 606, GPU 608, and / or one or more accelerators 614. In some examples, the capacity of the data repository 616 may be large enough to store multiple neural network instances for redundancy and security. The data repository 616 may include, for example, L2 and / or L3 cache 612. The memory 615 may include SRAM, LPDDR5, and / or other memory types. For example, the memory 615 may include 4MB SRAM, 32GB and / or 64GB 256-bit LPDDR5 (204.8GB / s), 8GB and / or 16GB 128-bit LPDDR5 (102.4GB / s), and / or other memory types and sizes. References to the data repository 616 may include references to the memory associated with the PVA 607, OFA 611, DLA 609 and / or other accelerators 614, as described herein.

[0161] Data repository 616 may include various storage types, such as eMMC, NVMe, etc. For example, SoC 604 may include storage in the form of an embedded multimedia card (eMMC) (e.g., 64GB eMMC 5.1) and / or an SD card slot, with external NVMExpress (NVMe) capability, for example, via M.2 Key M. For example, data repository 616 and / or other storage may be accessed via, for example, NVMe using PCI Express (PCIe), RDMA, TCP, and / or other protocols.

[0162] SoC 604 may include one or more processors 610 (e.g., embedded processors). Processor 610 may include a boot and power management processor (BPMP) 653, which may be a dedicated processor and subsystem for handling boot power and management functions, as well as associated safety implementations. BPMP 653 may be part of the SoC 604 boot sequence and may provide runtime power management services. BPMP 653 may provide clock and voltage programming, assistance with system low-power state transitions, management of SoC 604 thermal and temperature sensors, and / or management of SoC 604 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and SoC 604 may use the ring oscillator to detect the temperature of CPU 606, GPU 608, accelerator 614, and / or other components. If the temperature is determined to exceed a threshold, BPMP 653 may enter a temperature fault routine and place SoC 604 into a lower power state and / or place machine 600 into a driver-safe-stop mode (e.g., safely stop machine 600).

[0163] The processor 610 may also include a set of embedded processors that can be used as the Audio Processing Engine (APE) 655. The APE 655 can be an audio subsystem capable of providing full hardware support for multi-channel audio through multiple interfaces, as well as a wide and flexible audio I / O interface. In some examples, the APE 655 is a dedicated processor core with a digital signal processor and dedicated RAM.

[0164] The processor 610 may also include an Always-On Processor Engine (AOPE) 657, which provides the necessary hardware features to support low-power sensor management and wake-up use cases. AOPE 657 may include a processor core, tightly coupled RAM, peripheral support (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0165] Processor 610 may also include a security processor 613 (or “security island 613”), which may include a security cluster engine comprising a dedicated processor or processor subsystem for handling security management for automotive, robotic, and / or other applications. Security processor 613 and / or the security cluster engine may include two or more processor cores, tightly coupled RAM, support for peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In secure mode, the two or more cores may operate in lockstep mode and as a single core, with comparison logic to detect any differences in their operation. In some embodiments, security processor 613 may include discrete processors such that failure of other system components may not affect the performance and availability of security processor 613.

[0166] The processor 610 may also include a real-time or near-real-time sensor engine (SE) 659, which may include a dedicated processor subsystem for handling real-time or near-real-time camera, LiDAR, RADAR and / or other sensor modality management.

[0167] The processor 610 may also include one or more image signal processors (ISPs) 627, which may include high dynamic range signal processors and / or hardware engines as part of one or more sensor processing pipelines.

[0168] Processor 610 may include a video image synthesizer (VIC) 661, which may be a processing block (e.g., implemented on a microprocessor) that implements the video post-processing functions required by the video playback application to produce the final image of the player window. VIC 661 can perform lens distortion correction on wide-angle camera 668B, surround camera 668D, in-cabin monitoring camera sensor and / or other camera sensors with distorted field of view.

[0169] VIC 661 may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, when motion occurs in the video, noise reduction appropriately weights spatial information, thereby reducing the weight of information provided by adjacent frames. When an image or part of an image does not contain motion, temporal noise reduction performed by the video image synthesizer can use information from previous images to reduce noise in the current image.

[0170] The VIC 661 can also be configured to perform stereoscopic correction on input stereoscopic camera frames. The video image compositor can also be used for user interface compositing when the operating system desktop is in use and the GPU 608 does not need to continuously render new surfaces. Even when the GPU 608 is powered on and actively performing 3D rendering, the video image compositor can be used to offload the GPU 608 to improve performance and responsiveness.

[0171] SoC 604 may also include various peripheral interfaces for input / output (I / O) 625, for example, to enable communication with peripheral devices, audio codecs, power management and / or other devices. SoC 604 can be used to process data from cameras (e.g., via a gigabit multimedia serial link and / or Ethernet connection), from sensors (e.g., LiDAR sensor 664, RADAR sensor 660, etc., connected via Ethernet), from bus 602 (e.g., speed of machine 600, steering wheel position, etc.), and from GNSS sensor 658 (e.g., connected via Ethernet or CAN bus). SoC 604 may also include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine and can be used to free the CPU 606 from routine data management tasks. In some embodiments, the SoC 604 I / O 625 may include headers (e.g., 40-pin headers or 40-pin extension headers) supporting Universal Asynchronous Receiver / Transmitter (UART), Serial Peripheral Interface (SPI), Inter-Integrated Sound (I2S), Inter-Integrated Sound (I2C), Controller Area Network (CAN), Pulse Width Modulation (PWM), Digital Microphone Interface (DMIC), Digital Speaker Station (DSPK), General Purpose I / O (GPIO), etc.; automation headers (e.g., 12-pin automation headers); audio panel headers (e.g., 10-pin audio panel headers); Joint Test Action Group (JTAG) headers (e.g., 10-pin JTAG headers); fan headers (e.g., 4-pin fan headers); RTC battery spare connectors (e.g., 2-pin battery spare connectors); a microSD slot; a DC power jack; power, force, resume, and reset buttons; and one or more display connectors (e.g., DisplayPort (DP), such as DP 1.4A (+MST), eDP 1.41, HDMI). 2.1 and / or 4K30 multi-model DP1.2 (+MST) connectors) and / or other I / O 625 elements, components or features.

[0172] SoC 604 may include machine intranetting capabilities using, for example, Ethernet (e.g., automotive Ethernet), SERDES, Controller Area Network (CAN), FlexRay, Local Interconnect Network (LIN), Low Voltage Differential Signaling (LVDS), Media-Oriented System Transport (MOST), another network type, and / or combinations thereof. For example, SoC 604 may include RJ45 connectors with up to 10GbE, 1GbE connectors, and / or other network connector types.

[0173] So604 may include one or more digital signal processors (DSPs) 643. For example, DSP 643 may include a dedicated or specialized microprocessor chip optimized for digital signal processing, such as audio signal processing, telecommunications, digital image processing, RADAR, SONAR, LiDAR and / or other sensor processing, speech recognition and / or other applications.

[0174] SoC 604 may include one or more video encoders 619 and / or one or more video decoders 621. For example, video encoder 619 may include a hardware-based (e.g., as part of GPU 608) video encoder (e.g., supporting H.264, H.265, etc., and conforming to the HEVC standard, such as NVIDIA's NVENC) that can process image input (e.g., as YUV, RGB, etc.) to generate a video bitstream. Video decoder 621 may include a video decoder engine that can provide fully accelerated hardware video decoding capabilities (e.g., supporting decoding of various bitstream formats such as AV1, H.264, H.265, VP8, VP9, ​​MPEG-1, MPEG-2, MPEG-4, VC-1, etc., and conforming to the HEVC standard, such as NVIDIA's NVDEC). In some examples, video decoder 621 may be hardware-based (e.g., as part of GPU 608).

[0175] SoC 604 may include one or more General Purpose Computing Acceleration Clusters (GCACs) 629. For example, GCACs 629 may include various processor types that can be used to accelerate computing, such as one or more Vector Microcode Processors (VMPs) 633, one or more Multi-Threaded Processing Clusters (MPCs) 631, one or more Programmable Macroarrays (PMAs) 635, and / or one or more other processor types. For example, GCAC 629 may include a PMA 635, two VMPs 633, and two MPCs 631.

[0176] SoC 604 may include one or more vector microcode processors (VMPs) 633. In an embodiment, VMP 633 may include a wide vector (Very Long Instruction Word (VLIW) and Single Instruction Multiple Data (SIMD)) machine that performs a variety of operations, such as short integral-type operations common in computer vision and deep learning algorithms.

[0177] SoC 604 may include one or more multi-threaded processing clusters (MPCs) 631. MPCs 631 may include processing clusters that are more general-purpose than GPUs and more efficient than CPUs in some embodiments. For example, MPCs 631 may include multi-threaded processors that allow multiple threads to share resources and execute instructions concurrently.

[0178] The SoC 604 may include one or more programmable macro arrays (PMAs) 635. The PMA 635 may include a coarse-grained reconfigurable architecture (CGRA) dataflow machine, which has a unique architecture that delivers powerful performance on intensive computer vision and deep learning algorithms that may not be achievable in classic digital signal processing (DSP) architectures.

[0179] The SoC 604 may include one or more Display Processing Units (DPUs) 645 for performing hardware-accelerated image processing. For example, the DPU 645 may retrieve pixel data from memory 615 and send it to a display peripheral device via a standard interface. Therefore, the DPU 645 can handle display processing and rendering for displays within and / or on the machine.

[0180] The SoC 604 may include one or more Application Processing Units (APUs) 639. For example, the APU 639 may include a quad-core or dual-core processor with 48KB / 32KB L1 cache (with parity and ECC) and a 1MB L2 cache with ECC. The APU 639 may support NEON instructions and single-precision and double-precision floating-point operations.

[0181] The SoC 604 may include one or more Real-Time Processing Units (RTPUs) 669. The RTPU 669 may include a dual-core processor with 32KB / 32KB L1 cache and a 256KB TCM with ECC. The RTPU 669 can support single-precision and double-precision floating-point operations.

[0182] SoC 604 may include one or more built-in self-test (BIST) components 637. For example, BIST component 637 may include a memory BIST (MBIST) for testing the system's memory and / or a logic BIST (LBIST) for testing the system's logic. BIST component 637 may include embedded logic for directly testing the system's logic and / or memory.

[0183] SoC 604 may include one or more dynamically reconfigurable processors (DRPs) 671. For example, DRP 671 can be used to accelerate various computational operations. For instance, in one embodiment, DRP 671 may be combined with a MAC unit to function as an AI accelerator. In another embodiment, DRP 671 can execute an application while dynamically switching the circuit connection configuration of the arithmetic unit (e.g., ALU) on the chip each operating clock according to what needs to be processed. Because it uses only the necessary arithmetic circuitry, DRP 671 can consume less power than a CPU and achieve higher speeds. Furthermore, compared to a CPU, which suffers from performance degradation due to frequent accesses to external memory caused by cache misses and other reasons, DRP 671 can pre-build the necessary data paths in the hardware, thereby reducing performance degradation and operating speed variations (jitter) caused by memory accesses. DRP 671 may include a dynamic loading function that switches circuit connection information each time the algorithm changes, enabling processing with limited hardware resources, even in robotics / automotive applications that require processing multiple algorithms.

[0184] In some embodiments, accelerator 614 may include an OpenCV accelerator for accelerating OpenCV processing, OpenCV being an open-source industry-standard library for computer vision processing. In some embodiments, the combination of one or more DRP 671s deployed as AI accelerators with an OpenCV accelerator can enhance AI computation and image processing algorithms to enable complex and computationally intensive operations such as visual simultaneous localization and mapping (SLAM).

[0185] Compared to traditional systems, the techniques described herein, by providing CPU complexes, GPU complexes, and hardware acceleration clusters, allow multiple neural networks to be executed simultaneously (e.g., at least partially in parallel) and / or sequentially, and the results combined to achieve Level 2-5 autonomous driving capabilities and / or autonomous robot motion, control, planning, and / or navigation operations. Furthermore, since the SoC 604 can include various computing engines (e.g., processor 610, CPU 606, GPU 608, accelerator 614, etc.), tasks can be distributed among the computing engines, and in some cases, common-cause failures are avoided due to the discrete footprint of the computing engines. Additionally, since the SoC 604 can include a dedicated safety processor 613 (or safety island 613), critical safety or redundant operations can be performed without common-cause failures of the SoC 604's main processing components or computing engines. Due to these features, the underlying system of the SoC 604 and / or machine 600 may be able to meet higher safety levels—such as the Automotive Safety Integrity Level (ASIL) D of the ISO 26262 standard.

[0186] Figure 6E According to some embodiments of this disclosure, cloud-based servers (e.g., servers such as those described herein in a data center) and Figure 6A The following is a system diagram illustrating communication between an example autonomous or semi-autonomous vehicle or machine 600. System 676 may include server 678, network 690, and machine 600. Server 678 may include multiple GPUs 684(A)-684(H) (collectively referred to herein as GPU 684), switches 682(A)-682(D) (e.g., PCIe 4.0 / 5.0 switches, M.2 slots, Thunderbolt, USB4, NVIDIA's NVLink, NVIDIA's NVSwitch, GPUDirect RDMA, GPUDirect Storage, etc.), CPUs 680(A)-680(B) (collectively referred to herein as CPU 680), accelerators, and / or other processor types. GPU 684, CPU 680, and PCIe switch 682 may interconnect with high-speed interconnects, such as, but not limited to, NVIDIA-developed NVLink interface 688 and / or PCIe connection 686. In some examples, the GPU 684 is connected via NVLink and / or NVSwitchSoC, and the GPU 684 and PCIe switch 682 are connected via PCIe interconnect. While the figure shows eight GPUs 684, two CPUs 680, and two PCIe switches, this is not limiting. According to embodiments, each server 678 may include any number of GPUs 684, CPUs 680, and / or PCIe switches. For example, each server 678 may include eight, sixteen, thirty-two, and / or more GPUs 684.

[0187] Server 678 may receive sensor data from network 690 and machines 600 indicating information about new or previously unexplored locations, and / or sensor data indicating changes to previously seen / stored locations (e.g., unexpected or changed road conditions, such as recently started road construction). Server 678 may transmit neural network 692, updated neural network 692, map information 694, etc., including information about traffic and road conditions, to machines 600 via network 690. Updates to map information 694 may include updates to HD maps 622, SD maps, navigation maps, etc., such as information about construction sites, potholes, detours, floods, and / or other obstacles. In some examples, neural network 692, updated neural network 692, map information 694, and / or other information may come from new training and / or experience, represented in data received from any number of machines 600 in the environment, and / or based on training performed in a data center (e.g., using server 678 and / or other servers).

[0188] Server 678 can be used to train a machine learning model (e.g., a neural network) based on training data. Training data can be generated by machine 600, and / or can be generated in a simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is unlabeled and / or unprocessed (e.g., the neural network does not require supervised learning). Training can be performed according to any one or more machine learning techniques, including but not limited to the following categories: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component analysis and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, it can be used by machine 600 (e.g., transmitted to machine 600 via network 690), and / or the machine learning model can be used by server 678 to remotely monitor and / or control machine 600.

[0189] In some examples, server 678 can receive data from machine 600 and apply the data to a state-of-the-art real-time neural network for real-time intelligent inference. Server 678 may include a deep learning supercomputer powered by GPU 684 and / or a dedicated AI computer, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, server 678 may include a deep learning infrastructure in a data center using only CPU power.

[0190] The deep learning infrastructure of server 678 may be capable of rapid, real-time inference and can use this capability to assess and verify the health of the processor, software, and / or related hardware in machine 600. For example, the deep learning infrastructure may receive periodic updates from machine 600, such as image sequences and / or objects located by machine 600 in the image sequence (e.g., through computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them with the objects identified by machine 600. If the results do not match and the infrastructure concludes that the AI ​​in machine 600 has malfunctioned, server 678 may send a signal to machine 600 instructing the fail-safe computer of machine 600 to take over control, notify the occupants, and perform safety maneuvers or operations, such as slowing down, returning control to the driver, stopping, and / or pulling over / closing the vehicle.

[0191] For inference, server 678 may include GPU 684 and one or more programmable inference accelerators (such as NVIDIA's TensorRT). The combination of GPU-driven servers and inference acceleration enables real-time response. In other examples, such as in scenarios where performance is less critical, servers driven by CPUs, FPGAs, and other processors can be used for inference.

[0192] Computational ecosystem for generating, training, and deploying AI Figure 7 This is a system diagram illustrating a three-computer ecosystem 700 according to at least some embodiments of the present disclosure, including a first computing system 702 for generating or creating artificial intelligence (AI) (e.g., AI training and validation data), a second computing system 704 for training the AI, and a third computing system 706 for deploying AI at the edge (which may include or correspond to...). Figures 6A to 6E (SoC 604). For example, to develop and deploy materialized or physical AI, a three-computer ecosystem 700 can be used, comprising three accelerated computer systems to handle physical AI training, simulation, and runtime (e.g., edge deployment). These systems can use scalable, physical-based simulations of the machine 600 and its world to generate training data and train multimodal base models (and / or other model types). By doing so, simulations of the machine 600 can be performed at scale, allowing skills (e.g., robotic skills) to be improved, tested, and optimized in virtual worlds that simulate physical laws (e.g., using NVIDIA's OMNIVERSE), thereby helping to reduce the cost of real-world data acquisition and ensuring that the machine 600 can operate safely in a controlled environment.

[0193] Computing system 704 (e.g., NVIDIA's DGX platform) can be used to train and fine-tune powerful foundational and generative AI models. Models such as general foundational models (e.g., NVIDIA's Project GR00T) can be used to enable robots and other machines 600 to understand natural language and mimic actions by observing human movements. Computing system 704 may include a platform that integrates software, infrastructure, and expertise into a modern, unified AI development and training solution. Computing system 704 may include individual computing devices 710 (e.g., NVIDIA's DGX B200, H200, etc.) and / or any number of computing devices 710 (e.g., NVIDIA's DGX SuperPOD) within data center infrastructure 712.

[0194] For example, a standalone computing device 710 may include GPUs (e.g., 8 GPUs with a total GPU memory of 1,440 GB) and CPUs (e.g., 2 CPUs with a total of 112 cores, 2.1 GHz or 4 GHz (with enhancements)), providing up to 72 petaFLOPS of training capability and 144 petaFLOPS of inference capability. The computing device 710 may include memory (e.g., 4 TB of memory) and storage (e.g., 2 x 1.9 TB NVMe M.2 OS storage and 8 x 3.84 TB NVMe U.2 internal storage). The computing device 710 may include various networking and network management components, such as OSFP ports (e.g., 4 OSFP ports) for servicing single-port intelligent host channel adapters (e.g., 8 single-port ConnextX-7 Virtual Protocol Interconnect (VPI)), providing up to 400 GB / s of Infiniband / Ethernet. The computing device 710 may also include, for example, a dual-port quad small pluggable (QSFFP) data processing unit (DPU) (e.g., two dual-port QSFP112 DPUs – such as NVIDIA’s BlueField-3DPU) providing up to 400Gb / s of InfiniBand / Ethernet. The computing device 710 may include an onboard network interface card (NIC) (e.g., a 10Gb / s onboard NIC with RJ45), a dual-port Ethernet NIC (e.g., a 100GB / s dual-port Ethernet NIC), and / or a host board management controller (MBC) (e.g., with RJ45). In some embodiments, the NIC for the computing device 710 may include a SuperNIC (e.g., NVIDIA’s ConnectX-8 SuperNIC) to provide up to 800Gb / s of data throughput for in-network computing acceleration engines, thereby providing the performance and robust feature set required to support trillion-parameter-scale AI factories and scientific computing workloads. In other embodiments, computing device 710 may include a smart host channel adapter (HCA) (e.g., NVIDIA's ConnectX-7) to provide ultra-low latency and 400Gb / s throughput for in-network computing acceleration engines.

[0195] Data center infrastructure 712 may include: any number of computing devices 710, and an operating system (OS) (e.g., a DGX OS extension of a Linux distribution) for maximizing system uptime, security, and reliability; network / storage acceleration libraries and management for accelerating end-to-end infrastructure performance; cluster management for scaling and managing a single node (e.g., a computing device 710) to thousands of nodes; job scheduling and orchestration to ensure the smooth execution of each developer's jobs; AI workflow management and machine learning operations (MLOps) for moving more models from prototypes to production; and enterprise software to accelerate developer success.

[0196] Computational System 702 (e.g., NVIDIA's OVX server) can provide a development and simulation platform for testing and optimizing physics-based AI using APIs and frameworks for simulation (e.g., NVIDIA's DriveSIM, ISAAC Sim, ISAAC Gym, ISAAC Lab, etc.). Computational System 702 allows developers to use simulation frameworks to simulate and validate robot models, and / or generate large amounts of physically-based synthetic data to guide model training. Computational System 702 can support learning frameworks that power robot reinforcement learning and imitation learning to accelerate robot policy training and refinement. For example, Computational System 702 can be used to generate any number of Simulations 708, such as in NVIDIA's OMNIVERSE. Computational System 702 can be used to optimize and accelerate the entire software stack, from training, fine-tuning, and deploying generative AI to supporting industrial digitization within content collaboration platforms (including APIs, software development kits (SDKs), and services). These platforms allow the integration of OpenUSD, ray tracing rendering technologies (e.g., NVIDIA's RTX), and generative physics-based AI into existing software tools and simulation workflows, such as for industrial and robotics use cases (e.g., NVIDIA's OMNIVERSE). Therefore, the computing system 702 can host or support the native OpenUSD software platform, enabling enterprises to connect 3D pipelines and develop advanced real-time 3D applications for industrial digitalization. With powerful ray tracing-accelerated AI and graphics capabilities, the computing system 702 delivers robust performance for workloads such as extended reality (XR), multi-user design collaboration, and digital twins. This allows for the creation of physically accurate models with high-fidelity ray tracing and path tracing material rendering, large-scale, AI-enabled simulation operations, and the generation of realistic 3D synthetic data for training. The computing system 702 can include a single computing device 714 (e.g., an NVIDIA OVX L40S server) and / or any number of computing devices 714 (e.g., NVIDIA OVX systems) within a data center infrastructure 716.

[0197] Computing device 714 (which may include a server) may include CPUs (e.g., two CPUs, each with 32 cores) and GPUs (e.g., four or eight GPUs, each including 48GB GDDR6 with ECC memory, 864GB / s memory bandwidth, a PCIe Gen4x16:64GB / s bidirectional interconnect interface, 18,176 CUDA cores, 142 ray tracing (RT) cores, and 568 Tensor cores). Computing device 714 may include various networking and network management components, such as intelligent host channel adapters (HCAs) (e.g., two or four single-port ConnextX-7s, each with 200Gb / s, providing up to 800Gb / s InfiniBand / Ethernet), one or more DPUs (e.g., dual-port QSFP112 DPUs, such as the NVIDIA BlueField-3 DPU), providing up to 400Gb / s InfiniBand / Ethernet. In some embodiments, the NIC for computing device 714 may include a SuperNIC (e.g., NVIDIA's ConnectX-8 SuperNIC) to provide up to 800Gb / s of data throughput for in-network compute acceleration engines, thereby providing the performance and robust feature set required to support trillion-parameter-scale AI factories and scientific computing workloads. In other embodiments, computing device 714 may include a Smart Host Channel Adapter (HCA) (e.g., NVIDIA's ConnectX-7) to provide ultra-low latency and 400Gb / s throughput for in-network compute acceleration engines. Computing device 714 may include host memory (e.g., 384Gb DDR5 ECC for four GPUs, or 768Gb DDR5 ECC for eight GPUs) and may include dual in-line memory module (DIMM) slots, host boot drives (e.g., 1TB NVMe), and / or host storage (e.g., 24TB NVMe).

[0198] Similar to data center infrastructure 712, data center infrastructure 716 allows any number of computing devices 714 to be combined into a cluster configuration based on a reference architecture.

[0199] Computing system 706 can be used to deploy trained AI models on a runtime computer, such as the SoC 604 described herein. For example, these computing systems 706 can be designed for compact onboard computing needs, including ensembles of models such as control policies, vision, and language models deployed on an energy-efficient onboard edge computing system 706. (See also: [link to document]). Figures 6A to 6E A more detailed description of the components, features, and capabilities of the computing system 706.

[0200] Example Generative Model In at least some embodiments, language models such as Large Language Models (LLM), Visual Language Models (VLM), Multimodal Language Models (MMLM), Visual Language Action (VLA) models, and / or other types of generative artificial intelligence (AI) can be implemented. These models may be able to understand, summarize, translate, and / or otherwise generate text (e.g., natural language text, code, etc.), images, videos, computer-aided design (CAD) assets, OMNIVERSE and / or METAVERSE file information (e.g., USD formats such as OpenUSD), and / or the like, based on context provided in input prompts or queries. In embodiments, these language models may be considered "large" because they are trained on massive datasets and have architectures with a large number of learnable network parameters (weights and biases)—e.g., millions or billions of parameters. This disclosure allows for the implementation of LLM / VLM / MMLM, etc., for summarizing textual data, analyzing data (e.g., text, images, videos, etc.), extracting insights from data (e.g., text, images, videos, etc.), and generating new text / images / videos / , etc., in a user-specified style, tone, and / or format. In some embodiments, the LLM / VLM / MMLM, etc., of this disclosure may be specifically designed for text processing, while in other embodiments, multimodal LLMs may be implemented to accept, understand, and / or generate text and / or other types of content, such as images, audio (sound, synthesized speech, etc.), 2D and / or 3D data (e.g., USD format), and / or video. For example, a visual language model (VLM) or more specifically a multimodal language model (MMLM) may be implemented to accept images, videos, sensor data, audio, text, 3D designs (e.g., CAD), and / or other input data types and / or generate or output images, videos, audio, text, 3D designs, and / or other output data types.

[0201] Various types of LLM / VLM / MMLM / etc. architectures can be implemented in various embodiments. For example, different architectures can be implemented using different techniques to understand and generate outputs (e.g., text, audio, video, images, 2D and / or 3D design or asset data, etc.). In some embodiments, LLM / VLM / MMLM / etc. architectures (e.g., recurrent neural networks (RNNs) or long short-term memory networks (LSTMs)) can be used, while in other embodiments, converter architectures (e.g., architectures relying on self-attention and / or cross-attention (e.g., between contextual data and textual data) mechanisms) can be used to understand and recognize relationships between words or tokens and / or contextual data (e.g., other text, video, images, design data, USD, etc.). One or more generative processing pipelines including LLM / VLM / MMLM / etc. may also include one or more diffusion blocks (e.g., noise reduction blocks). The LLM / VLM / MMLM / etc. of this disclosure may include encoder and / or decoder blocks. For example, discriminative or encoder-only models (e.g., BERT (Bidirectional Encoder Representations from Transformers)) can be implemented for tasks involving language understanding (e.g., classification, sentiment analysis, question answering, and named entity recognition). As another example, generative or decoder-only models (e.g., GPT (Generative Pretrained Transformer)) can be implemented for tasks involving language and content generation (e.g., text completion, story generation, and dialogue generation). LLM / VLM / MMLM / etc., including encoder and decoder components (e.g., T5 (Text-to-Text Transformer)), can be implemented to understand and generate content, such as for translation and summarization. These examples are not intended to be limiting and any architecture type (including, but not limited to, those described herein) can be implemented depending on the specific implementation and the task performed using LLM / VLM / MMLM / etc.

[0202] In various embodiments, LLM / VLM / MMLM / etc. can be trained using unsupervised learning, where LLM / VLM / MMLM / etc. learns patterns from a large amount of unlabeled text / audio / video / image / design / USD / etc. data. Due to the extensive training, in these embodiments, the model may not require task-specific or domain-specific training. An LLM / VLM / MMLM / etc. extensively pre-trained on a large amount of unlabeled data can be referred to as a base model and can excel at various tasks, such as question answering, summarizing, filling in missing information, translation, and image / video / design / USD / data generation. Some LLM / VLM / MMLM / etc. can be customized for specific use cases using techniques such as cue tuning, fine-tuning, retrieval augmentation generation (RAG), adding adapters (e.g., custom neural networks and / or neural network layers to tune or adjust cues or labels to bias the language model towards a specific task or domain), and / or using optimization models for specific tasks and / or other fine-tuning or customization techniques for specific domains.

[0203] In some embodiments, the LLM / VLM / MMLM / etc. disclosed herein can be implemented using various model alignment techniques. For example, in some embodiments, guardrails can be implemented to identify incorrect or unwanted inputs (e.g., prompts) and / or outputs of the model. In this process, the system can use guardrails and / or other model alignment techniques to prevent the processing of specific unwanted inputs using LLM / VLM / MMLM / etc., and / or to prevent the output or presentation of information generated by LLM / VLM / MMLM / etc. (e.g., displays, audio outputs, etc.). In some embodiments, one or more additional models (or layers thereof) can be implemented to identify problems with the model's inputs and / or outputs. For example, these "protective" models can be trained to identify "safe" or otherwise okay or desired inputs and / or outputs and / or "unsafe" or otherwise unwanted inputs and / or outputs for a particular application / implementation. Therefore, the LLM / VLM / MMLM / etc. disclosed herein are unlikely to output language / text / audio / video / design data / USD data / etc. that may be offensive, vulgar, inappropriate, insecure, out of scope, and / or unwanted for a particular application / implementation.

[0204] In some embodiments, an LLM / VLM / etc. can be configured or able to access or use one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc. For example, for certain tasks or operations where the model is not ideally suited, the model may have instructions for accessing one or more plugins (e.g., third-party plugins) to help process the current input (e.g., as a result of training, and / or based on instructions in a given prompt). In such an example, when at least part of the prompt relates to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs) to retrieve relevant information. Another example is that if at least part of the response requires mathematical computation, the model can access one or more mathematical plugins or APIs to help solve the problem, and then the response from the plugins and / or APIs can be used in the model's output. This process can be repeated (e.g., recursively) an arbitrary number of iterations, using any number of plugins and / or APIs, until a response to each query / question / request / process / operation / etc. can be generated in response to the input prompt. Therefore, models can rely not only on their own knowledge gained from training on large datasets, but also on the expertise or optimized properties of one or more external resources (such as APIs, plugins, etc.).

[0205] In some embodiments, multiple language models (e.g., LLM / VLM / MMLM / etc., multiple instances of the same language model, and / or multiple hints provided to the same language model or instances of the same language model) can be implemented, executed, or accessed (e.g., using one or more plugins, user interfaces, APIs, databases, data stores, repositories, etc.) to provide output in response to the same query or in response to separate parts of a query. In at least one embodiment, the same input query and hints (e.g., a set of constraints, condition generators, etc.) can be provided to multiple language models (e.g., language models with different architectures, language models trained on different (e.g., updated) data corpora). In one or more embodiments, the language models can be different versions of the same base model. In one or more embodiments, at least one language model can be instantiated as multiple agents, for example, providing more than one hint to constrain, guide, or otherwise influence the style, content, or character of the provided output. In one or more exemplary non-limiting embodiments, the same language model can be required to provide output corresponding to different roles, perspectives, characters, or different knowledge bases, as defined by the provided hints.

[0206] In any such embodiment, the outputs of two or more (e.g., each) language models, two or more versions of at least one language model, two or more instantiated proxies of at least one language model, and / or provided to two or more prompts for at least one language model can be further processed, such as aggregated, compared, or filtered, or used to determine (and provide) a consensus response. In one or more embodiments, the output from one language model (or version, instance, or proxy) can be provided as input to another language model for further processing and / or validation. In one or more embodiments, the language model can be required to generate or otherwise obtain output about the input source material, wherein the output is associated with the input source material. Such association can include, for example, generating captions or text portions embedded (e.g., as metadata) within the input source text or image. In one or more embodiments, the output of the language model can be used to determine the validity of the input source material for further processing or inclusion in a dataset. For example, the language model can be used to evaluate the presence (or absence) of a target word in a text portion or the presence (or absence) of an object in an image, wherein the text or image is annotated to indicate such presence (or absence). Alternatively, the determination from the language model can be used to determine whether the source material should be included in a curatorial dataset, for example, but not limited to this.

[0207] Figure 8 This is a block diagram of an example generative language model system 800 suitable for implementing at least some embodiments of the present disclosure. Figure 8 In the example shown, the generative language model system 800 includes a retrieval augmentation generation (RAG) component 892, an input processor 805, a tokenizer 810, an embedding component 820, a plugin / API 895, and a generative language model (LM) 830 (which may include LLM, VLM, MMLM, VLA model, etc.).

[0208] At a high level, the input processor 805 can receive input 801, which includes text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasound, etc.), 3D design data, CAD data, generic scene descriptor (USD) data (e.g., OpenUSD, etc.), depending on the architecture of the generative LM 830 (e.g., LLM / VLM / MMLM / etc.). In some embodiments, input 801 includes plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, input 801 may include numerical sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., tabular format, JSON, or XML). In generative LM In some implementations of 830 capable of handling multimodal input, input 801 can combine text (or text that may be omitted) with image data, audio data, video data, design data, USD data, and / or other types of input data (e.g., but not limited to the data described herein). Taking raw input text as an example, input processor 805 can prepare the raw input text in various ways. For example, input processor 805 can perform various types of text filtering to remove noise (e.g., special characters, punctuation marks, HTML tags, stop words, portions of images, portions of audio, etc.) from the relevant text content. In examples involving stop words (common words that often have little semantic meaning), input processor 805 can remove stop words to reduce noise and enable generative LM. 830 focuses on more meaningful content. Input processor 805 can apply text normalization, for example, by converting all characters to lowercase, removing accents, and / or handling special cases (such as abbreviations or abbreviations) to ensure consistency (e.g., converting ¼ to ¼). Similarly, input processor 805 and / or post-processor can perform inverse text normalization (ITN) to convert plain language back to canonical or other forms (e.g., converting ¼ to ¼). These are just a few examples; other types of input and / or output processing can be applied.

[0209] In some embodiments, RAG component 892 (which may include one or more RAG models, and / or may be performed using generative LM 830 itself) may be used to retrieve additional information to be used as part of input 801 or a prompt. RAGs can be used to enhance input to LLM / VLM / MMLM / etc. with external knowledge to make the answer to a specific question or query or request more relevant, for example, where specific knowledge is required. RAG component 892 may obtain this additional information from one or more external sources (e.g., basic information such as basic text / images / videos / audio / USD / CAD / etc.), which can then be fed along with the prompt to LLM / VLM / MMLM / etc. to improve the accuracy of the model's response or output.

[0210] For example, in some embodiments, in addition to the data retrieved using RAG component 892, input 801 may also be generated using query or model inputs (e.g., questions, requests, etc.). In some embodiments, input processor 805 may analyze input 801 and communicate with RAG component 892 (or in embodiments, RAG component 892 may be part of input processor 805) to identify relevant text and / or other data to provide to generative LM 830 as additional context or information sources, typically from which responses, answers, or outputs 890 are identified. For example, when the input indicates that a user is interested in the required tire pressure for a particular brand and model of vehicle, RAG component 892 may use a RAG model, for example, to perform a vector search in the embedding space to retrieve tire pressure information or its corresponding text from a digital (embedded) version of the owner's manual for that particular brand and model of vehicle. Similarly, when a user revisits a chatbot related to a particular product for sale or service, RAG component 892 may retrieve previously stored conversation history (or at least its summary) and provide the previous conversation history along with the current inquiry / request as part of input 801 to generative LM 830.

[0211] RAG component 892 can use various RAG techniques. For example, naïve RAG can be used, where documents are indexed, chunked, and applied to an embedding model to generate embeddings corresponding to chunks. User queries can also be applied to this embedding model and / or another embedding model of RAG component 892, and the embeddings of chunks can be compared with the embeddings of the query to identify the most similar / relevant embeddings to the query. These most similar / relevant embeddings can be provided to generative LM 830 to generate output.

[0212] In some embodiments, more advanced RAG techniques can be used. For example, chunks can undergo pre-retrieval processes (e.g., routing, rewriting, metadata analysis, expansion, etc.) before being passed to the embedding model. Furthermore, post-retrieval processes (e.g., re-ranking, hint compression, etc.) can be performed on the output of the embedding model before generating the final embedding, which is then used for comparison with the input query.

[0213] As a further example, modular RAG techniques can be used, such as those similar to Naive RAG and / or Advanced RAG, but also including features such as hybrid search, recursive retrieval and query engines, StepBack methods, subqueries and hypothetical document embeddings.

[0214] As another example, Graph RAG can use a knowledge graph as a source of context or factual information. Graph RAG can be implemented using a graph database as a source of contextual information sent to LLM / VLM / MMLM / etc. Instead of providing the model with data chunks extracted from larger documents (which can lead to a lack of context, factual accuracy, linguistic accuracy, etc.) (or anything other than providing the model with data chunks extracted from larger documents), Graph RAG can also provide the model with structured entity information by combining structured entity text descriptions with their many attributes and relationships, thus giving the model deeper insights. In implementing Graph RAG, the systems and methods described herein use graphs as content stores and extract relevant document chunks, requiring LLM / VLM / MMLM / etc. to use them to answer questions. In such embodiments, the knowledge graph may contain relevant textual content and metadata about the knowledge graph, or it may be integrated with a vector database. In some embodiments, Graph RAG can use the graph as a subject matter expert, where descriptions of concepts and entities relevant to the query / hint can be extracted and passed to the model as semantic context. These descriptions may include relationships between concepts. In other examples, the graph can be used as a database where a portion of a query / hint can be mapped to a graph query, the graph query can be executed, and LLM / VLM / MMLM / etc. can summarize the results. In such examples, the graph can store relevant factual information and can be used for queries (natural language queries) and entity links to graph query tools (NL to graph query tools). In some embodiments, the graph RAG (e.g., using a graph database) can be combined with standard (e.g., vector database) RAGs and / or other RAG types to benefit from a variety of approaches.

[0215] In any embodiment, the RAG component 892 can implement plugins, APIs, user interfaces, and / or other functions to perform RAG. For example, LLM / VLM / MMLM / etc. can use graph RAG plugins to run queries on knowledge graphs to extract relevant information to feed into the model, and can use standard or vector RAG plugins to run queries on vector databases. For example, the graph database can interact with the plugin's REST interface, thus decoupling the graph database from the vector database and / or the embedded model.

[0216] The tokenizer 810 can segment (e.g., processed) text data into smaller units (tags) for subsequent analysis and processing. Depending on the implementation, the tags can represent individual words, sub-words, characters, audio / video / images, etc. Word-based tokenization divides the text into individual words, treating each word as a separate tag. Sub-word tokenization breaks words down into smaller meaningful units (e.g., prefixes, suffixes, stems), enabling the generative LM 830 to understand morphological changes and process words outside the vocabulary more effectively. Character-based tokenization represents each character as a separate tag, enabling the generative LM 830 to process text at a fine-grained level. The choice of tokenization strategy can depend on factors such as the language being processed, the task at hand, and / or the characteristics of the training dataset. Therefore, the tokenizer 810 can transform (e.g., processed) text into a structured format according to the tokenization scheme implemented in a particular embodiment.

[0217] Embedding component 820 can use any known embedding technique to transform discrete tokens into semantically meaningful (e.g., dense, continuous vector) representations. For example, embedding component 820 can use pre-trained word embeddings (e.g., Word2Vec, GloVe, or FastText), one-hot encoding, Term Frequency-Inverse Document Frequency (TF-IDF) encoding, one or more embedding layers of a neural network, and / or others.

[0218] In some implementations where input 801 includes image data / video data, etc., input processor 805 may resize the data to a standard size compatible with the format of the corresponding input channel and / or normalize pixel values ​​to a common range (e.g., 0 to 1) to ensure consistent representation, and embedding component 820 may encode the image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations where input 801 includes audio data, input processor 805 may resample the audio file to a consistent sampling rate for uniform processing, and embedding component 820 may use any known technique to extract and encode audio features, such as in the form of a spectrogram (e.g., a Mel spectrogram). In some implementations where input 801 includes video data, input processor 805 may extract frames or apply resizing to extracted frames, and embedding component 820 may extract features such as optical flow embedding or video embedding and / or encode temporal information or frame sequences. In some implementations where input 801 includes multimodal data, the embedded component 820 can use techniques such as early fusion (concatenation), late fusion (sequential processing), and attention-based fusion (e.g., self-attention, cross-attention) to fuse representations of different types of data (e.g., text, images, audio, USD, video, design, etc.).

[0219] Other components of the generative LM 830 and / or generative LM system 800 may use different types of neural network architectures depending on the implementation scheme. For example, a transducer-based architecture (such as the one used in models like GPT) may be implemented, and it may include a self-attention mechanism that weights the importance of different words or tokens in the input sequence and / or a feedforward network that processes the output of the self-attention layer, applying a nonlinear transformation to the input representation and extracting higher-level features. Some non-limiting example architectures include transducers (e.g., encoder-decoder, decoder-only, multimodal), RNNs, LSTMs, fusion models, diffusion models, cross-modal embedding models that learn a joint embedding space, graph neural networks (GNNs), hybrid architectures that combine different types of adversarial networks (such as generative adversarial networks or GANs or adversarial autoencoders (AAEs) for joint distribution learning, linear time series modeling using a selective state-space modeling (SSM) architecture (e.g., the Mamba LLM architecture), etc. Therefore, depending on the implementation scheme and architecture, the embedded component 820 can apply the encoded representation of the input 801 to the generative LM 830, and the generative LM 830 can process the encoded representation of the input 801 to generate an output 890, which may include response text and / or other types of data.

[0220] As described herein, in some embodiments, the generative LM 830 may be configured to access or use (or be able to access or use) plugins / APIs 895 (which may include one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc.). For example, for certain tasks or operations where the generative LM 830 is not ideally suited, the model may have instructions (e.g., as a result of training, and / or based on instructions in a given prompt, such as instructions retrieved using RAG component 892) to access one or more plugins / APIs 895 (e.g., third-party plugins) to help process the current input. In such an example, when at least part of the prompt is related to a restaurant or weather, the model may access one or more restaurant or weather plugins (e.g., via one or more APIs), sending at least part of the prompt related to a particular plugin / API 895 to the plugin / API 895, which may process the information and return an answer to the generative LM 830, which may use the response to generate output 890. This process can be repeated (e.g., recursively) an arbitrary number of iterations and repeated with any number of plugins / APIs 895 until an output 890 that resolves each query / question / request / process / action / etc. from input 801 is generated. Therefore, the model can rely not only on its own knowledge acquired from training on a large dataset and / or from data retrieved using the RAG component 892, but also on the expertise or optimized properties of one or more external resources (e.g., plugins / APIs 895).

[0221] In some embodiments, one or more converter engines (TEs) can be implemented. Converter engines can use micro-tensor scaling to optimize performance and accuracy, such as enabling 16-bit floating-point (FP16), 8-bit floating-point (FP8), and / or 4-bit floating-point (FP4) AI processing. For example, a converter engine can use 16-bit or 8-bit floating-point precision and 8-bit or 4-bit floating-point data formats, combined with software algorithms, to improve AI performance and capabilities. By reducing mathematical operations to 8 or 4 bits, TEs can train larger networks faster without compromising accuracy. For example, TEs can include libraries for accelerating converter models on processing devices such as GPUs to provide better performance in training and inference with lower memory utilization. When TEs are combined with other technologies, such as high-speed interconnects between nodes (e.g., using NVLink switches) and tensor cores (which enable mixed-precision computation, such as micro-scaling precision support), server clusters can be more capable of training massive networks (e.g., billions of parameters) at high speeds. Therefore, it can support tensor core precision for FP64, TF32, BF16, FP16, FP8, INT8, FP6 and FP4, as well as CUDA core precision for FP64, FP32, FP16 and BF16.

[0222] The LLM / VLM / MMLM / VLA and other architectures described herein are intended to be examples only, and other suitable architectures may be implemented within the scope of this disclosure.

[0223] Example computing device Figure 9This is a block diagram of an example computing device 900 suitable for implementing some embodiments of the present disclosure. In some embodiments, one or more functions of a depth-assisted 3D detection system 100, a depth-assisted 3D detection system 150, a blink detection system 200, a drowsiness detection system 300, and / or a gaze detection system 400 may be implemented using the computing device 900. The computing device 900 may include an interconnect system 902 that directly or indirectly couples to the following devices: a memory 904, one or more central processing units (CPUs) 906, one or more graphics processing units (GPUs) 908, a communication interface 910, input / output (I / O) ports 912, input / output components 914, a power supply 916, one or more presentation components 918 (e.g., one or more displays, one or more speakers, etc.), and one or more logic units 920. In at least one embodiment, one or more computing devices 900 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 908 may include one or more vGPUs, one or more CPUs 906 may include one or more vCPUs, and / or one or more logic units 920 may include one or more virtual logic units. Thus, one or more computing devices 900 may include discrete components (e.g., a full GPU dedicated to computing device 900), virtual components (e.g., a portion of a GPU dedicated to computing device 900), or combinations thereof.

[0224] although Figure 9 The various blocks are shown as connected via interconnect system 902 using lines, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 918 (such as a display device) may be considered I / O component 914 (e.g., if the display is a touchscreen). As another example, CPU 906 and / or GPU 908 may include memory (e.g., memory 904 may represent a storage device other than the memory of GPU 908, CPU 906, and / or other components). Therefore, Figure 9 The computing devices described are for illustrative purposes only. No distinction is made between such categories as “workstation,” “server,” “laptop computer,” “desktop computer,” “tablet computer,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of this category. Figure 9 Within the scope of computing devices.

[0225] Interconnect system 902 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 902 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 906 may be directly connected to memory 904. Further, CPU 906 may be directly connected to GPU 908. Where there is a direct or point-to-point connection between components, interconnect system 902 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required to be included in computing device 900.

[0226] The memory 904 may include any computer-readable medium from a variety of computer-readable media. A computer-readable medium may be any available medium accessible by the computing device 900. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0227] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 904 may store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, Digital Universal Disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 900. As used herein, computer storage media does not include the signal itself.

[0228] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and includes any information transmission medium. The term "modulated data signal" can refer to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the above should also be included within the scope of computer-readable media.

[0229] CPU 906 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. Each CPU 906 may contain one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling numerous software threads simultaneously. CPU 906 may contain any type of processor and may contain different types of processors depending on the type of computing device 900 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 900, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (such as math coprocessors), computing device 900 may also include one or more CPUs 906.

[0230] In addition to or in lieu of one or more CPUs 906, one or more GPUs 908 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. One or more GPUs 908 may be integrated GPUs (e.g., having one or more CPUs 906) and / or one or more GPUs 908 may be discrete GPUs. In embodiments, one or more GPUs 908 may be coprocessors of one or more CPUs 906. GPUs 908 may be used by computing device 900 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 908 may be used for general-purpose computing on a GPU (GPGPU). GPUs 908 may include hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. GPUs 908 may generate pixel data for an output image in response to rendering commands (e.g., rendering commands received from CPUs 906 via a host interface). GPU 908 may include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 904. GPU 908 may include two or more GPUs operating in parallel (e.g., via a link). The link may be directly connected to the GPUs (e.g., using NVLINK) or may be connected to the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 908 may produce pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for an analog image). Each GPU may include its own memory or may share memory with other GPUs. In some embodiments, one or more functions of depth-assisted 3D detection system 100, depth-assisted 3D detection system 150, blink detection system 200, drowsiness detection system 300, and / or gaze detection system 400 may be implemented using CPU 906 and / or GPU 908.

[0231] In addition to or in lieu of CPU 906 and / or GPU 908, logic unit 920 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 906, one or more GPUs 908, and / or one or more logic units 920 may execute any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 920 may be a portion of one or more CPUs 906 and / or GPUs 908 and / or integrated into one or more CPUs 906 and / or GPUs 908, and / or one or more logic units 920 may be discrete components or otherwise external to CPUs 906 and / or GPUs 908. In embodiments, one or more logic units 920 may be coprocessors of one or more CPUs 906 and / or GPUs 908. In some embodiments, one or more functions of the depth-assisted 3D detection system 100, the depth-assisted 3D detection system 150, the blink detection system 200, the drowsiness detection system 300, and / or the gaze detection system 400 may be implemented using one or more logic units of the logic unit 920.

[0232] Examples of logic unit 920 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree lateral unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), programmable vision accelerator (PVA) (which may include one or more direct memory access (DMA) systems), and one or more vision or vector processing units (VPUs). One or more pixel processing engines (PPEs) (e.g., comprising a 2D array of processing elements, each of which communicates north, south, east, and west with one or more other processing elements in the array), one or more decoupled accelerators or units (e.g., decoupled lookup table (DLUT) accelerators or units), visual processing units (VPUs), optical flow accelerators (OFAs), field-programmable gate arrays (FPGAs), neuromorphic chips, quantum processing units (QPUs), associative processing units (APUs), arithmetic logic units (ALUs), application-specific integrated circuits (ASICs), floating-point units (FPUs), input / output (I / O) elements, peripheral component interconnects (PCIs) or fast peripheral component interconnects (PCIe) elements, etc.

[0233] The communication interface 910 may include one or more receivers, transmitters, and / or transceivers enabling the computing device 900 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). The communication interface 910 may include components and functions for enabling communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wirelessband), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 920 and / or the communication interface 910 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via interconnect system 902 to one or more GPUs 908 (e.g., memory of one or more GPUs 908).

[0234] I / O port 912 enables computing device 900 to be logically coupled to other devices including I / O component 914, one or more presentation components 918, and / or other components, some of which may be built into (e.g., integrated into) computing device 900. Illustrative I / O component 914 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dish antennas, scanners, printers, wireless devices, etc. I / O component 914 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by the user. In some cases, input may be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, pen recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of computing device 900. Computing device 900 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. Additionally, the computing device 900 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables motion detection. In some examples, the computing device 900 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0235] Power supply 916 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 916 may provide power to computing device 900 so that the components of computing device 900 can operate.

[0236] The presentation component 918 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 918 may receive data from other components (e.g., GPU 908, CPU 906, etc.) and output the data (e.g., as images, videos, sounds, etc.).

[0237] Example network environment A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 9It is implemented on one or more instances of one or more computing devices 900—for example, each device may include similar components, features, and / or functions of one or more computing devices 900. Furthermore, in the case of implementing back-end devices (e.g., servers, NAS, etc.), the back-end devices may be included as part of a data center, such as, but not limited to, those described herein.

[0238] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. In cases where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0239] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein for the server can be implemented on any number of client devices.

[0240] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or application may respectively include network-based service software or applications. In embodiments, one or more client devices may use network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs). The framework layer may be, but is not limited to, a free and open-source software network application framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0241] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers at the state, region, country, global, etc.). If the connection to a user (e.g., a client device) is relatively close to an edge server, the core server can assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0242] One or more client devices may include the information described in this article. Figure 9 At least some of the components, features, and functions of one or more example computing devices 900 described. By way of example and not limitation, the client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming device or system, entertainment system, vehicle computer system, embedded system controller, chat booth, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0243] Example Terms Example Clause 1. One or more processors, including a processing circuitry system, the processing circuitry system being configured to: acquire, using image data, two-dimensional 2D image coordinates of one or more key points corresponding to at least one feature of a vehicle occupant; generate, at least based on the 2D image coordinates being fitted to a three-dimensional 3D occupant model, the 3D occupant model including the at least one feature, first 3D coordinates of the one or more key points in a coordinate system; generate, at least based on the first 3D coordinates and depth data of the one or more key points in a global coordinate system, second 3D coordinates of the one or more key points in a global coordinate system to determine a state of the at least one feature; and output an indication of the state of the at least one feature.

[0244] Example Clause 2. One or more processors according to Example Clause 1, wherein the at least one feature includes at least a portion of the head of the vehicle occupant, and the 3D occupant model including the at least one feature is a personalized 3D occupant model including the head of the vehicle occupant.

[0245] Example Clause 3. One or more processors according to Example Clause 1 or Example Clause 2, wherein the processing circuitry is further configured to: synchronously operate the depth sensor and the image sensor to capture the depth data and the image data.

[0246] Example Clause 4. One or more processors according to any one of Example Clauses 1-3, wherein the processing circuitry is further configured to: generate the depth data based at least on stereo information determined using two or more sensors.

[0247] Example Clause 5. One or more processors according to any one of Example Clauses 1-4, wherein the one or more keypoints include at least a first keypoint and a second keypoint of the first eye of the vehicle occupant, wherein the processing circuitry is further configured to: determine the distance between the 3D coordinates of the first keypoint and the 3D coordinates of the second keypoint to generate an eye-opening value; determine whether the eye-opening value exceeds a threshold for the first eye of the vehicle occupant; and output an indication of whether the first eye of the vehicle occupant is open or closed, at least based on the determination that the eye-opening value exceeds the threshold for the first eye of the vehicle occupant.

[0248] Example Clause 6. One or more processors according to any one of Example Clauses 1-5, wherein the processing circuitry is configured to: determine an eye-opening value based at least on the second 3D coordinates of the one or more keypoints in the global coordinate system.

[0249] Example Clause 7. One or more processors according to any one of Example Clauses 1-6, wherein the depth data is determined at least based on light from the image sensor to a plane fitted to one or more eyelid key points and chin key points of the vehicle occupant.

[0250] Example Clause 8. One or more processors according to any one of Example Clauses 1-7, wherein the processing circuitry is further configured to: determine whether the face of the vehicle occupant is in a frontal pose; in response to determining that the face of the vehicle occupant is in the frontal pose, determine an eye aspect ratio (EAR) value based at least on the 2D image coordinates; and perform at least one of blink detection or drowsiness detection based at least on the EAR value.

[0251] Example Clause 9. One or more processors according to any one of Example Clauses 1-8, wherein the processing circuitry is further configured to: automatically annotate the image data based at least on the second 3D coordinates of the one or more keypoints in the global coordinate system.

[0252] Example Clause 10. One or more processors according to any one of Example Clauses 1-9, wherein the at least one feature includes at least a portion of the hand of the vehicle occupant, and the 3D occupant model including the at least one feature includes a personalized 3D occupant model of the hand of the vehicle occupant.

[0253] Example Clause 11. One or more processors according to any one of Example Clauses 1-10, wherein said one or more processors are included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more analog operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models LL A system for performing operations using M; a system for performing operations using one or more visual language models (VLMs); a system for performing operations using one or more multimodal language models (MMLMs); a system for performing operations using one or more visual-language-action (VLA) models; a system for using or deploying one or more inference microservices; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0254] Example Clause 12. A system comprising one or more processors, the one or more processors being configured to: acquire two-dimensional 2D image coordinates of one or more key points of at least one feature of a vehicle occupant, based at least on image data; generate 3D coordinates of the one or more key points in the global coordinate system, based at least on the 2D image coordinates being fitted to a three-dimensional 3D occupant model and depth data in the global coordinate system, the 3D occupant model including the at least one feature of the vehicle occupant; determine a state of the at least one feature of the vehicle occupant based at least on the 3D coordinates of the one or more key points in the global coordinate system; and output an indication of the state of the at least one feature of the vehicle occupant.

[0255] Example Clause 13. The system according to Example Clause 12, wherein the at least one feature of the vehicle occupant includes at least a portion of the head of the vehicle occupant, and the 3D occupant model including the at least one feature of the vehicle occupant includes a personalized 3D model of the head of the vehicle occupant.

[0256] Example Clause 14. The system according to Example Clause 12 or Example Clause 13, wherein the one or more processors are further configured to: synchronously operate the depth sensor and the image sensor to capture the depth data and the image data; or generate the depth data based at least on stereo information determined using two or more sensors.

[0257] Example Clause 15. A system according to any one of Example Clauses 12-14, wherein the one or more processors are further configured to: determine an eye-opening value based at least on the 3D coordinates of the one or more keypoints in the global coordinate system.

[0258] Example Clause 16. A system according to any one of Example Clauses 12-15, wherein the one or more keypoints include at least a first keypoint and a second keypoint of the first eye of the vehicle occupant, wherein the one or more processors are further configured to: determine the distance between the 3D coordinates of the first keypoint in the global coordinate system and the 3D coordinates of the second keypoint in the global coordinate system to generate an eye-opening value; determine whether the eye-opening value exceeds a threshold for the first eye of the vehicle occupant; and output an indication of whether the first eye of the vehicle occupant is open or closed, at least based on the determination that the eye-opening value exceeds the threshold for the first eye of the vehicle occupant.

[0259] Example Clause 17. A system according to any one of Example Clauses 12-16, wherein the one or more processors are further configured to: automatically annotate the image data based at least on the 3D coordinates of the one or more key points of the at least one feature of the vehicle occupant in the global coordinate system.

[0260] Example Clause 18. A system according to any one of Example Clauses 12-17, wherein the one or more processors are further configured to: determine the position of the pupil of the vehicle occupant's eye based at least on the 3D coordinates of the one or more keypoints in the global coordinate system; and determine the vehicle occupant's gaze direction based at least on the position of the pupil of the vehicle occupant's eye.

[0261] Example Clause 19. A system according to any one of Example Clauses 12-18, wherein said one or more processors are included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more analog operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language model LLMs. Systems for performing operations; systems for performing operations using one or more visual language models (VLMs); systems for performing operations using one or more multimodal language models (MMLMs); systems for performing operations using one or more visual-language-action (VLA) models; systems for using or deploying one or more inference microservices; systems for performing one or more conversational AI operations; systems for generating synthetic data; systems for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; systems containing one or more virtual machines (VMs); systems implemented at least partially in a data center; or systems implemented at least partially using cloud computing resources.

[0262] Example Clause 20. A method comprising: generating an indication of the state of at least one feature determined based on three-dimensional 3D coordinates of one or more keypoints of at least one feature of a vehicle occupant in a global coordinate system, wherein the 3D coordinates are determined based at least on two-dimensional coordinates of the one or more keypoints fitted to a 3D occupant model and depth data in the global coordinate system, the 3D occupant model including the at least one feature.

[0263] This disclosure can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions, such as program modules, that are executed by a computer or other machine, such as a personal data assistant or other handheld device. Typically, a program module, which includes routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure can also be practiced in distributed computing environments, where tasks are performed by remote processing devices linked via a communication network.

[0264] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0265] This document describes in detail the subject matter of this disclosure to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the discloser has envisioned that the claimed subject matter may be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of the method employed, these terms should not be construed as suggesting any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. One or more processors, including a processing circuitry system, said processing circuitry system being used for: Using image data, obtain the 2D image coordinates of one or more key points corresponding to at least one feature of a vehicle occupant; At least based on the 2D image coordinates being fitted to a 3D occupant model, a first 3D coordinate of the one or more key points in the coordinate system of the image sensor is generated, the 3D occupant model including the at least one feature; Based at least on the first 3D coordinates and the depth data of the one or more key points in the global coordinate system, generate the second 3D coordinates of the one or more key points in the global coordinate system to determine the state of the at least one feature; as well as Output an indication of the state of the at least one feature.

2. The processor of claim 1 or more, wherein the at least one feature includes at least a portion of the head of the vehicle occupant, and the 3D occupant model including the at least one feature is a personalized 3D occupant model including the head of the vehicle occupant.

3. The processor of claim 1 or more, wherein the processing circuitry is further configured to: synchronously operate the depth sensor and the image sensor to capture the depth data and the image data.

4. The processor of claim 1 or more, wherein the processing circuitry is further configured to: generate the depth data based at least on stereo information determined using two or more sensors.

5. The processor of claim 1 or more, wherein the one or more key points include at least a first key point and a second key point of the first eye of the vehicle occupant, wherein the processing circuitry is further configured to: Determine the distance between the 3D coordinates of the first key point and the 3D coordinates of the second key point to generate an eye opening value; Determine whether the eye opening value exceeds a threshold for the first eye of the vehicle occupant; and Based at least on determining whether the eye opening value exceeds the threshold for the first eye of the vehicle occupant, an indication of whether the first eye of the vehicle occupant is open or closed is output.

6. The processor of claim 1 or more, wherein the processing circuitry is configured to: determine an eye-opening value based at least on the second 3D coordinates of the one or more key points in the global coordinate system.

7. The processor of claim 1 or more, wherein the depth data is determined at least based on light rays from the image sensor to a plane fitted to one or more eyelid key points and chin key points of the vehicle occupant.

8. The processor of claim 1 or more, wherein the processing circuitry is further configured to: Determine whether the faces of the vehicle occupants are in a frontal position; In response to determining that the face of the vehicle occupant is in the frontal pose, the eye aspect ratio (EAR) value is determined at least based on the 2D image coordinates; and At least one of blink detection or drowsiness detection is performed based on the EAR value.

9. The processor of claim 1 or more, wherein the processing circuitry is further configured to: automatically annotate the image data based at least on the second 3D coordinates of the one or more key points in the global coordinate system.

10. One or more processors according to claim 1, wherein the at least one feature includes at least a portion of the hand of the vehicle occupant, and the 3D occupant model including the at least one feature includes a personalized 3D occupant model of the hand of the vehicle occupant.

11. The processor of claim 1 or more, wherein the processor is included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models (MMLMs); A system for performing operations using one or more visual-language-action (VLA) models; A system for using or deploying one or more inference microservices; A system for performing one or more conversational AI operations; A system for generating synthetic data; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

12. A system comprising one or more processors, said one or more processors being used to: Based at least on image data, obtain the two-dimensional 2D image coordinates of one or more key points of at least one feature of the vehicle occupant; At least based on the 2D image coordinates being fitted to a 3D occupant model and depth data in the global coordinate system, the 3D coordinates of one or more key points in the global coordinate system are generated, wherein the 3D occupant model includes at least one feature of the vehicle occupant. The state of at least one feature of the vehicle occupant is determined based at least on the 3D coordinates of the one or more key points in the global coordinate system. as well as Output an indication of the state of at least one characteristic of the vehicle occupant.

13. The system of claim 12, wherein the at least one feature of the vehicle occupant includes at least a portion of the head of the vehicle occupant, and the 3D occupant model including the at least one feature of the vehicle occupant includes a personalized 3D model of the head of the vehicle occupant.

14. The system of claim 12, wherein the one or more processors are further configured to: Simultaneously operate the depth sensor and the image sensor to capture the depth data and the image data; or The depth data is generated based on stereo information determined using at least two or more sensors.

15. The system of claim 12, wherein the one or more processors are further configured to: determine an eye opening value based at least on the 3D coordinates of the one or more key points in the global coordinate system.

16. The system of claim 12, wherein the one or more key points include at least a first key point and a second key point of the first eye of the vehicle occupant, wherein the one or more processors are further configured to: Determine the distance between the 3D coordinates of the first key point in the global coordinate system and the 3D coordinates of the second key point in the global coordinate system to generate an eye-opening value; Determine whether the eye opening value exceeds a threshold for the first eye of the vehicle occupant; as well as Based at least on determining whether the eye opening value exceeds the threshold for the first eye of the vehicle occupant, an indication of whether the first eye of the vehicle occupant is open or closed is output.

17. The system of claim 12, wherein the one or more processors are further configured to: automatically annotate the image data based at least on the 3D coordinates of the one or more key points of the at least one feature of the vehicle occupant in the global coordinate system.

18. The system of claim 12, wherein the one or more processors are further configured to: Based at least on the 3D coordinates of one or more key points in the global coordinate system, determine the position of the pupil of the vehicle occupant's eye; and The gaze direction of the vehicle occupant is determined at least based on the position of the pupil of the vehicle occupant's eye.

19. The system of claim 12, wherein the one or more processors are included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models (MMLMs); A system for performing operations using one or more visual-language-action (VLA) models; A system for using or deploying one or more inference microservices; A system for performing one or more conversational AI operations; A system for generating synthetic data; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

20. A method comprising: Generate an indication of the state of at least one feature determined based on the three-dimensional 3D coordinates of one or more key points of at least one feature of a vehicle occupant in a global coordinate system, wherein the 3D coordinates are determined based at least on the two-dimensional coordinates of the one or more key points fitted to a 3D occupant model and depth data in the global coordinate system, the 3D occupant model including the at least one feature.

Images