Uncertainty estimation for object detection in autonomous and semi-autonomous systems and applications

Abstract

The invention relates to uncertainty estimation of object detection in autonomous and semi-autonomous systems and applications. In various examples, systems and methods for uncertainty estimation of object detection in autonomous and semi-autonomous systems and applications are provided. The systems and methods may use data from one or more sensors (e.g., one or more cameras and / or one or more LiDAR sensors) to generate a representation of features around the machine. The model may be used to generate a probability of presence of an object in the feature representation and an uncertainty estimate corresponding to the probability of presence of the object. The uncertainty estimation may be used to identify scenes that are significantly different from the training data, detect errors in the bounding shape of the object, and / or highlight areas where object detection may have been missed. The systems and methods may also be used to automatically annotate scenes associated with the feature representations, and the automatically annotated scenes may be used for training purposes.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 708,606, filed October 17, 2024, the entire contents of which are incorporated herein by reference. Background Technology

[0003] In recent years, 3D object detection has attracted much attention and is an important task for computer vision and perception applications in autonomous and semi-autonomous vehicles, robots, and other machines. 3D object detection techniques can be broadly classified into three main approaches: camera-based methods, LiDAR-based methods, and multimodal methods. Camera-based methods predict 3D objects based on multi-view camera images and aggregate features from multiple camera views to construct a comprehensive understanding of the geometry. LiDAR-based methods estimate 3D objects in a given point cloud, projecting the point cloud onto a regular grid (such as cylinders, voxels, or distance images), and then a deep learning model is used to obtain features for object detection. Multimodal methods integrate or fuse various sensor data (such as camera and LiDAR data) to further enhance 3D detection capabilities. These multimodal methods allow models to fully utilize the complementary advantages of cameras and LiDAR, resulting in improved detection accuracy compared to single-modal methods.

[0004] One approach to fusing different types of data, such as LiDAR and camera data, includes bird's-eye view (BEV) fusion or top-down view fusion of different types of data. At a higher level, this approach generates a fused or combined feature set in the form of BEV representations. BEV representations effectively capture the relative position and size of objects, making them well-suited for perception and planning. After generating the fused BEV feature set, these features can be used to perform object detection, which involves using the fused BEV features to generate or identify the bounding shape corresponding to the object using an object detection model (e.g., a heatmap model). However, these models may struggle to adequately evaluate or quantify the confidence or uncertainty of predictions made for object detection, leading to poor performance (e.g., in unfamiliar scenes).

[0005] Uncertainty estimation models have been considered for use in conjunction with object detection models to generate estimated uncertainties related to object detection. Sampling-based uncertainty estimation methods (e.g., MC-Dropout, Deep Ensemble, etc.) are the most common approaches for evaluating the reliability of deep neural networks. While intuitive, these methods typically increase the nominal computational, memory, or training costs of the neural network by a factor of two. MC-Dropout involves randomly deleting network weights and observing the effects, while Deep Ensemble involves training multiple networks with different initializations. Compared to some sampling-based uncertainty estimation methods, MC-Dropout has relatively low computational cost but is generally less reliable, while Deep Ensemble provides more reliable uncertainty estimation but has high computational and memory requirements. Therefore, these methods are not suitable for large-scale applications, including 3D detection systems (e.g., for autonomous or semi-autonomous vehicles, robots, or other machines). Summary of the Invention

[0006] Various embodiments of this disclosure relate to uncertainty estimation for object detection in autonomous and semi-autonomous systems and applications. Systems and methods are disclosed that can be used for detecting out-of-distribution scenes, bounding shape errors, and / or missed objects based on uncertainty estimation, as well as adapting operations or labels of ego machines applied to driving scenarios, where these labels can be used to train one or more models for multiple tasks (e.g., ego machine navigation models).

[0007] In contrast to conventional systems, the systems and methods proposed in this disclosure can generate object presence probabilities and corresponding uncertainty estimates of those probabilities using a first model (e.g., an evidence-based deep learning (EDL) model) based at least on a representation of features (e.g., a BEV representation) associated with sensor data from one or more sensors in the environment. The first model can generate object presence probabilities and uncertainty estimates for each unit and class of the feature representation based at least on parameters (e.g., α and β) of the probability distribution of the class (e.g., a Beta distribution) for each unit of the feature representation (e.g., each BEV unit). Using the generated presence probabilities and corresponding uncertainty estimates, the techniques described herein enable the detection of scenes significantly different from the training distribution (out-of-distribution (OOD) scene detection), the detection of bounding shapes with errors, and the highlighting of regions where objects may be missed (missed object detection). Additionally, a unified pipeline for automatically labeling scenes corresponding to feature representations is described, and the labels can be identified as requiring human verification at the scene, bounding shape, and / or missed object levels. This centralized validation can improve the performance of secondary models trained using validated, automatically labeled scenarios, and result in a significant improvement in the final detection metric through uncertainty-driven improvements. Attached Figure Description

[0008] The following section describes in detail the uncertainty estimation for object detection in autonomous and semi-autonomous systems and applications, with reference to the accompanying figures:

[0009] Figure 1 This is a flowchart illustration of an example 3D object detection system applicable to implementing some embodiments of the present disclosure;

[0010] Figure 2 This is a flowchart illustration of an example distributed off-site detection system applicable to implementing some embodiments of the present disclosure;

[0011] Figure 3 This is a flowchart illustration of an example enclosing shape error detection system applicable to implementing some embodiments of the present disclosure;

[0012] Figure 4 This is a flowchart illustration of an example missing object detection system applicable to implementing some embodiments of the present disclosure;

[0013] Figure 5 These are illustrations of example automatic annotation systems applicable to implementing some embodiments of this disclosure;

[0014] Figure 6 This is an illustration of an example flowchart for 3D object detection according to some embodiments of the present disclosure;

[0015] Figure 7A Examples of sensor locations having a corresponding field of view or sensing field, such as an autonomous or semi-autonomous machine, according to at least some embodiments of this disclosure;

[0016] Figure 7B This is an illustration of examples of the locations of components and sensors on an autonomous or semi-autonomous vehicle according to at least some embodiments of this disclosure;

[0017] Figure 7C This is a block diagram of an example system architecture for autonomous or semi-autonomous vehicles, robots, and / or other machine types according to at least some embodiments of this disclosure;

[0018] Figure 7D This is a block diagram of an example architecture of a computing system (e.g., a system-on-a-chip (SoC)) according to at least some embodiments of the present disclosure;

[0019] Figure 7E This is a system diagram of communication between a cloud-based server and example autonomous or semi-autonomous vehicles, robots and / or other machine types according to at least some embodiments of this disclosure;

[0020] Figure 8 This is a system diagram illustrating three computer ecosystems according to at least some embodiments of the present disclosure, including a computing system for generating or creating artificial intelligence (AI) (e.g., AI training and validation data), a computing system for training artificial intelligence, and a computing system for deploying AI at the edge;

[0021] Figure 9 This is a block diagram of an example computing system for generative artificial intelligence (AI) according to at least some embodiments of the present disclosure; and

[0022] Figure 10 This is a block diagram of an example computing device according to at least some embodiments of the present disclosure. Detailed Implementation

[0023] Systems and methods relating to uncertainty estimation in autonomous and semi-autonomous systems and their applications are disclosed. Although this disclosure may relate to example autonomous or semi-autonomous vehicles, robots, and / or other machine types 700 (which may alternatively be referred to herein as...) Figures 7A-7E Examples of “Vehicle 700”, “Self-Vehicle 700”, “Machine 700”, “Self-Machine 700”, “Robot 700”, and / or “Self-Robot 700” are as follows: Figures 7A-7EThe description herein is provided for purposes of description, but is not intended to be limiting. For example, the systems and methods described herein may be, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms (e.g., autonomous mobile robots (AMRs), humanoid robots, robotic arms and / or end effectors), warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, boats, shuttles (e.g., robotaxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones, and / or other types of vehicles, robots or machines. Furthermore, while this disclosure may describe uncertainty estimation for object detection in autonomous and semi-autonomous vehicles, this is not intended to be limiting. The systems and methods described herein can be used in augmented reality (AR), virtual reality (VR), mixed reality (MR), robotics, safety and supervision (e.g., smart cities), autonomous or semi-autonomous machine applications, industrial manufacturing, simulation, and / or any other technological space where object detection can be used. In some embodiments, the systems, methods, and / or processes described herein can be used with... Figures 7A-7E Example machine 700, Figure 8 Example computing ecosystem 800, Figure 9 Example generative language model system 900 and / or Figure 10 Example computing device 1000 uses components, features, and / or functions similar to those of other computing devices to perform the operation.

[0024] In contrast to conventional systems (such as those described above), the systems and methods proposed in this disclosure can generate object presence probabilities and corresponding uncertainty estimates based at least on feature representations associated with sensor data from one or more sensors in the environment using an Evidence-Based Deep Learning (EDL) model. The EDL model generates object presence probabilities and uncertainty estimates based at least on feature representations. Using the generated presence probabilities and corresponding uncertainty estimates, the techniques described herein enable the detection of scenes significantly different from the training distribution (out-of-distribution (OOD) scene detection), the detection of bounding shapes with errors, and the highlighting of regions where objects may be missed (missed object detection). Additionally, a unified pipeline for automatically labeling scenes corresponding to feature representations is described, and the labels can be identified as requiring human verification at the scene, bounding shape, or missed object level. This centralized verification improves the performance of secondary models trained using verified, automatically labeled scenes and results in a significant improvement in the final detection metric through uncertainty-driven improvements.

[0025] During operation, sensor data can be acquired from various types of sensors. In some cases, sensors may be located on the self-machine and capture sensor data relating to the environment surrounding the self-machine. As an example only, LiDAR sensors and image sensors may be positioned (e.g., close to each other) and / or oriented to capture similar or different parts of the environment. Sensor data may include, but is not limited to, point cloud data from LiDAR sensors and / or images (e.g., RGB images) from image sensors.

[0026] Based on sensor data obtained (e.g., from image sensors and / or LiDAR sensors), a representation of the set of features associated with detected objects in the environment can be generated. As used herein, a feature can refer to any feature that captures or indicates spatial patterns or boundaries associated with objects in the environment. Feature representations can (e.g., using tensors) represent any number of features in a viewpoint or space. These features can be transformed to a single viewpoint or space (e.g., BEV viewpoint or BEV space).

[0027] In some embodiments, a unified representation of features can be generated. A unified feature representation typically refers to the identified feature representation associated with multiple sensors (such as different types of sensors). Thus, various features from different types of sensors, such as image sensors and LiDAR sensors, can be combined or fused into a single unified feature representation. For example, in cases where LiDAR sensor features and image sensor features are to be represented by a unified feature representation, the unified feature representation can be in the form of a BEV (Body Image Representation). In this way, features associated with the LiDAR sensor and features associated with the image sensor can be fused or aggregated in a unified BEV space or viewpoint to generate a unified feature representation. Generating a unified feature representation in BEV form makes it easier to identify shapes and orientations. Advantageously, using BEV to generate a unified feature representation maintains both the geometric structure from LiDAR features and the semantic density from image sensor features.

[0028] An EDL model can be provided with a feature representation of the environment, which may also be referred to as an "EDL head" or "EDL heatmap head". The EDL model can generate and output object presence probabilities based on the feature representation. The object presence probability for a given unit and class can be generated based on parameters (e.g., α and β) of the probability distribution of each unit (e.g., each BEV unit) and class (e.g., a Beta distribution). The object presence probability output by the EDL model can include heatmap data indicating the probability that an object is located at a specific 3D coordinate. In addition to predicting the object's location (e.g., where the object is located), the object presence probability can also encompass class prediction (e.g., what type of object is detected). For example, the object presence probability can include the probability that the center of an object of a particular class is located within a unit of the feature representation. In some embodiments, the object presence probability indicates the proportion of positive evidence (e.g., a measure of support from the data that favors or disfavors classifying a sample into a class) in the total evidence (e.g., a measure of support from the data that favors or disfavors classifying a sample into a class) of the unit and class of the feature representation. As positive evidence represents a larger portion of the total evidence, the object presence probability increases.

[0029] EDL models can also generate and output uncertainty estimates corresponding to the object presence probability based on feature representations. Each uncertainty estimate generated corresponds to a corresponding generated object presence probability. Uncertainty estimates corresponding to a particular object presence probability can be generated based on the same parameters (e.g., α and β) used to generate the probability distributions (e.g., Beta distributions) of the units and categories used to generate the corresponding object presence probabilities. In addition to location uncertainty (e.g., where the object is located), uncertainty estimates corresponding to object presence probabilities can also capture category uncertainty (e.g., what type of object is detected). For example, in the case of generating object presence probabilities for each unit (e.g., BEV unit) and category of feature representation, corresponding uncertainty estimates for each unit and category of feature representation can be generated. Uncertainty estimates indicate the level of uncertainty when predicting the object presence probability, and this level of uncertainty is inversely proportional to the total evidence. That is, the level of uncertainty indicated by the uncertainty estimate decreases as the total evidence increases.

[0030] For object detection, there may be an inherent imbalance toward the negative class, which could be because most detections correspond to the background and lead to overconfidence in positive detections (e.g., low uncertainty estimates), thus biasing the uncertainty estimates. In some embodiments, the combined loss function of the EDL model can be used to mitigate the negative class imbalance by using two main terms. The first term can correspond to the cell in the feature representation where the actual object center is located, and the Bayesian risk loss for each of these cells can be computed and scaled (e.g., using a factor based on Gaussian focus loss (GFL)). This helps reduce the influence of well-classified examples and focuses on examples that are harder to misclassify during training. The second term can correspond to the cell in the feature representation where no object is located. The Bayesian risk can be similarly computed and weighted (e.g., using a factor based on GFL) to focus on more difficult negative examples. In some embodiments, a discount term can also be applied, which reduces the penalty for predictions made near the object center. A regularization term can also be included in the loss function to manage uncertainty by penalizing the model when it generates incorrect or overconfident predictions. The goal is to reduce misleading evidence, especially when the model makes incorrect predictions. When predictions are incorrect, regularization can be applied by encouraging the model to revert to a uniform prior representing high uncertainty (e.g., a Dirichlet prior), thereby penalizing misleading evidence and avoiding errors of overconfidence.

[0031] Uncertainty estimates generated from feature representations can be aggregated or combined in various ways to perform different types of detection on scenes corresponding to the feature representations. For example, uncertainty estimates can be used to detect scene-level OOD samples, bounding shapes with errors, and / or missed objects. Uncertainty estimates can also be used to adapt control over self-machines, data storage, or other operations.

[0032] Uncertainty estimates for each class and feature representation unit corresponding to a scene can be generated. For scene-level OOD detection, the uncertainty estimates of the entire feature representation are combined (e.g., averaged) to generate an aggregated uncertainty estimate for the scene. The aggregated uncertainty estimate of the scene can be compared with a threshold to determine whether the scene is OOD, where OOD means that the scene differs sufficiently from the training distribution of the EDL model. For example, if the aggregated uncertainty estimate of the scene exceeds the threshold, an indication that the scene is considered an OOD scene can be output. For bounding shape detection with errors, the uncertainty estimates of units associated with the predicted bounding shape (e.g., box or cube) are combined (e.g., averaged) to generate an aggregated uncertainty estimate of the bounding shape. The aggregated uncertainty estimate of the predicted bounding shape can be compared with a threshold to determine whether a localization error exists in the predicted bounding shape. For example, if the aggregated uncertainty estimate of the predicted bounding shape exceeds the threshold, an indication that a localization error exists in the predicted bounding shape can be output.

[0033] In some cases, EDL models may assign low object presence probabilities to locations where actual objects exist, potentially leading to false negatives or missed object detections. Missed object detections are often accompanied by high uncertainty in prediction, indicating the uncertainty the EDL model faces in identifying objects within certain regions (e.g., feature-represented units). Uncertainty estimates generated by the EDL model can be used to identify potentially missed objects and improve detection performance in such challenging scenarios.

[0034] The second model can be used to process the feature representation, the predicted object presence probability, and the uncertainty estimate for each unit and category from the feature representation of the EDL model. A concatenated vector including these components can be fed into the second model to estimate the confidence value of a potentially missed object in a given unit. In some embodiments, only units where the EDL model produces a low object presence probability (e.g., an object presence probability less than a threshold) are used as candidates for locations where objects may have been missed. In these cases, a subset of the object presence probability and corresponding uncertainty estimate is used. A threshold (e.g., 5%) can be chosen such that no bounding boxes are generated for units identified as having a low object presence probability. The second model can be trained using the same objective, loss, and training procedure as the EDL model, the only difference being that the second model is trained on units with low probabilities and uses the object presence probability and corresponding uncertainty estimate as input in addition to the feature representation.

[0035] The techniques described herein may also include automatically labeling one or more scenarios associated with feature representations to generate one or more automatically labeled scenarios. At least a portion of the automatically labeled scenarios is identified as requiring (e.g., human) verification based on uncertainty estimation. For example, the entire scene, enclosing shapes, or spaces without enclosing shapes (e.g., cells) may be identified as requiring verification based on uncertainty estimation. The scenarios may then be verified (e.g., human) and the verified automatically labeled scenarios can then be used to train an object detection model.

[0036] The techniques described herein can be used for real-time deployment (e.g., real-time edge deployment). The bounding shape (e.g., a box or cube) or a representation of the bounding shape of an object can be generated based on the object existence probability and uncertainty estimate generated by the EDL model. One or more operations corresponding to the environment can be performed based at least on the bounding shape or the representation of the bounding shape. For example, a self-machine can use the bounding shape or the representation of the bounding shape to manipulate itself. Furthermore, if the uncertainty estimate of the bounding shape or the representation of the bounding shape exceeds a threshold, the self-machine can adjust its operations (e.g., return control to the driver).

[0037] The techniques described herein can also be used to determine how a perception system (e.g., a self-driving machine) collects training data. Typically, sensor data is collected using a self-driving machine driving over an extended period, and all sensor data is stored, which requires significant cost and memory. In some embodiments, the aggregate uncertainty of a scene can be determined; and if the aggregate uncertainty of a scene exceeds a threshold, the sensor data for that scene can be stored. This way, only the sensor data corresponding to poorly covered scenarios from the data used to train the EDL model will be stored, saving memory and cost.

[0038] The embodiments presented in this disclosure primarily pertain to 3D object detection in relation to autonomous vehicles. However, it should be understood that techniques similar to those described herein can also be used for other applications of 3D object detection. The embodiments presented in this disclosure can be implemented within the context of developing automated annotation scenarios for training and / or deploying 3D object detection models. Object presence probability and / or uncertainty estimation can be used to train navigation systems, such as, but not limited to, autonomous vehicles, semi-autonomous vehicles, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, spacecraft, ships, shuttles, emergency response vehicles, engineering vehicles, underwater vehicles, drones, and / or other vehicle types, and operate in a variety of locations, such as, but not limited to, warehouses, factories, retail stores, and / or other locations.

[0039] In some embodiments, the systems and methods described herein can be performed in a simulated environment (e.g., NVIDIA DriveSIM, ISAAC Sim, ISAAC Gym, ISAAC Lab, etc.) using simulated data (e.g., simulated environment data and simulated sensor data from simulated sensors of virtual or simulated vehicles, robots, or machines in the simulated environment). For example, simulated input data (e.g., map data, perception data, self-motion data, tactile data, and / or any other data described herein) can be used to determine one or more feature representations for object detection, and this information can be used to perform operations associated with virtual machines in the simulated environment. These simulated operations can be used to test their performance before deploying the underlying algorithms, systems, and / or processes to the real world. In some instances, simulations can be used to generate synthetic training data, such as synthetic sensor data from the simulation to be input into the 3D object detection system 100. The synthetic training data can then be used or processed (e.g., by one or more models 106) to generate corresponding feature representations (e.g., feature representation 108), for example (in addition to or instead of real-world data). In any example (such as where the simulated environment is used for testing, validation, training, etc.), one or more optical transport simulation algorithms (such as one or more ray tracing and / or path tracing algorithms) can be used to render or otherwise generate the simulated environment and / or associated training data. When using optical transport simulation, the simulation system may employ one or more dedicated ray tracing hardware accelerators and / or processors (e.g., NVIDIA's RTX or another real-time ray tracing GPU, such as a GPU including one or more ray tracing (RT) cores) optimized for performing real-time or near-real-time optical transport simulation operations in conjunction with one or more other processors of the system (e.g., GPUs, CPUs, accelerators, etc.). In some embodiments, simulated environments and / or one or more of their objects, features, or components can be generated or managed in a 3D content collaboration platform (e.g., NVIDIA's OMNIVERSE) that can be optimized for or suitable for industrial digitization, generative physics, artificial intelligence, and / or other use cases, applications, and / or services. For example, the content collaboration platform or system may include a system for using or developing generic scene descriptors (USD) (e.g., OpenUSD) data to manage objects, features, scenes, etc., in simulated environments, digital environments, etc. The platform can include realistic physics simulations (e.g., using NVIDIA's PhysX software development kit (SDK)) to simulate real physics and physical interactions with simulations hosted on the platform.This platform can integrate OpenUSD with ray tracing / path tracing / optical transport simulations (e.g., NVIDIA's RTX rendering technology) into software tools and simulation workflows for building, training, deploying, and / or testing AI systems, such as those used for testing, validating, training (e.g., machine learning models, neural networks, etc.) and / or other tasks related to motor vehicles, robots, other machine types, and / or other systems and applications. In some examples, the simulated environment can include digital twins of real-world environments, such as specific road segments, warehouses, data centers, airports, geographic areas, ocean areas, and / or any other real-world environment operable by autonomous or semi-autonomous vehicles or machines.

[0040] In some embodiments, a remote control or teleoperation system can be used to perform teleoperation or remote control of vehicles, robots, and / or other machines. For example, the systems and methods described herein can be used to generate bounding shapes of detected objects that can be included in a visualization or mapping of the environment to assist a remote operator in controlling an autonomous or semi-autonomous machine through the environment (or to provide waypoints or other indications for control or navigation). Thus, a remote operator can use visual, auditory, textual, and / or other cues or indicators generated by the systems and methods described herein to assist in navigating vehicles, robots, machines, etc., through real-world environments using a teleoperation system.

[0041] In some embodiments, the systems and methods described herein can be deployed in robotic applications. For example, a robot or robotic system may include one or more onboard processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), hardware-based programmable vision accelerator (PVA) (which may include one or more vector processing units (VPU)), direct memory access (DMA) system and / or pixel processing engine (PPE), hardware-based optical flow accelerator (OFA), SoC, etc.), and memory and / or storage devices (e.g., for storing control algorithms, sensor data, and one or more machine learning models). Robotic systems can use these processors to execute one or more machine learning models (e.g., language models, visual language models (VLM), large language models (LLM), visual-language-action (VLA) models, multimodal language models (MMLM), etc.) that allow the robotic system to autonomously or semi-autonomously perform complex tasks, such as interacting with and / or manipulating static and / or dynamic objects, or navigating the environment using sensors (e.g., cameras, LiDAR, RADAR, ultrasonic sensors, etc.). The system can use sensor fusion techniques to combine data from multiple sensors (e.g., cameras, infrared, LiDAR, RADAR, accelerometers) to create a comprehensive model of the robot's surrounding environment. This data can be processed locally on the robot or sent to a remote server for computationally intensive tasks such as 3D mapping or SLAM (simultaneous localization and mapping). In one or more embodiments, data from individual robots (e.g., sensor data, task status, or environmental conditions) can be uploaded to the cloud, where a centralized AI model can analyze and optimize commands and distribute them across the entire fleet. In some embodiments, one or more machine learning models described herein (e.g., language models, VLM, VLA, LLM, MMLM, diffusion models, NeRF models, DNN, etc.) may be used to allow the robot to perceive and reason about its environment and / or communicate with one or more other robots and / or people in the environment. In some embodiments, the robot may (e.g., using one or more network interface cards (NICs) and / or data processing units (DPUs)) communicate with one or more locally hosted servers / computing devices and / or with one or more remotely located servers / computing devices (e.g., in one or more data centers).

[0042] In some embodiments, the systems and methods described herein can be deployed in in-vehicle infotainment (IVI) systems or in-cabin experience (IX) applications. For example, an infotainment system within a vehicle (e.g., a car, truck, drone, engineering equipment, robot, semi-autonomous vehicle, or autonomous vehicle) may include one or more onboard processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), hardware-based programmable vision accelerator (PVA) (which may include one or more vector processing units (VPU), direct memory access (DMA) systems and / or pixel processing engines (PPE), hardware-based optical flow accelerators (OFA), SoCs, etc.). This includes memory and / or storage devices (e.g., for storing entertainment content, navigation data, and user preferences). The system can use these processors to execute one or more machine learning models (e.g., language models) to enable features such as voice control, personalized media recommendations, dynamic navigation, and real-time communication with other services via network connectivity. In-vehicle infotainment systems can also use natural language processing (NLP) models to enable voice-based interaction. One or more machine learning models can be stored locally or accessed via one or more APIs connected to cloud services, enabling the system to process requests in real-time or near real-time.

[0043] In some examples, one or more machine learning models described herein (e.g., deep neural networks, language models, LLMs, VLMs, multimodal language models, visual-language-action (VLA) models, perceptual models, tracking models, fusion models, transformer models, diffusion models, encoder-only models, decoder-only models, encoder-decoder models, neural rendering field (NERF) models, etc.) can be packaged as microservices, such as inference microservices (e.g., NVIDIA NIM), which may include containers (e.g., operating system (OS) level virtualization packages) that may include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model "engine". For example, an inference microservice may include the container itself and one or more models (e.g., weights and biases). In some instances (e.g., where one or more machine learning models are small enough (e.g., have a sufficiently small number of parameters)), one or more models may be included within the container itself. In other examples (such as when one or more models are large), one or more models may be hosted / stored in the cloud (e.g., in a data center) and / or hosted on-premises and / or at the edge (e.g., on a local server or computing device, but outside a container). In these embodiments, one or more models may be accessed via one or more APIs (such as REST APIs). Thus, and in some embodiments, one or more machine learning models described herein may be deployed as inference microservices to accelerate the deployment of one or more models on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, optimized inference engines (e.g., execution software built using standardized AI model deployments, such as NVIDIA's Triton Inference Server) and / or one or more APIs for high-performance deep learning inference, which may include inference runtime and model optimizations for delivering low-latency and high-throughput inference for production applications (such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). One or more machine learning models described herein can be included as part of a microservice along with an acceleration infrastructure capable of being deployed with a single command and / or orchestrated and automatically scaled (e.g., to data center scale on a single device) using a container orchestration system. Thus, an inference microservice can include machine learning models (e.g., optimized for high-performance inference), inference runtime software that executes one or more machine learning models and provides outputs / responses to inputs (e.g., user queries, prompts, etc.), and enterprise management software that provides health checks, identity, and / or other monitoring.In some embodiments, the inference microservice may include software that performs in-situ replacement and / or updates to one or more machine learning models. When a replacement or update is performed, the software performing the replacement / update may maintain user configurations for the inference runtime software and enterprise management software.

[0044] While examples of machine learning models such as neural networks are described in this document, this is not intended to be limiting. For example, but not limited to, any of the various machine learning models and / or neural networks described herein can include any type of machine learning model, such as those using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoder neural networks, artificial neural networks (ANN), convolutional neural networks (CNN), recurrent neural networks (RNN), perceptrons, long short-term memory (LSTM) networks, multilayer perceptron (MLP) networks, etc. Deep Stacked Networks (DSN), Generative Pre-trained (GPT) models or networks, Feedforward networks, Radial Basis Function ANNs, Self-Organizing Maps (SOM), Kohonen Maps, Hopfield Networks, Boltzmann Machines, Deep Belief Neural Networks, Deconvolutional Neural Networks, Generative Adversarial Networks (GANs), Liquid Machines, Modular Neural Networks, Sequence-to-Sequence Models, Networks Using Transformer Architectures, State-Space Models (SSM) (e.g., Networks Using Mamba Architectures (e.g., Mamba-1, Mamba...) 2) Networks using selective state-space models, networks using structured state-space sequence models, etc., diffusion models (e.g., diffusion probability models, fraction-based generative models, etc.), Neural radiation field (NeRF) models, Gaussian sputtering models, Kolmogorov-Arnold networks (KAN), models with encoder-only architectures, models with decoder-only architectures, models with encoder-decoder architectures, generative machine learning models, language models, large language models (LLM), visual language models (VLM), multimodal language models (MMLM), large action models (LAM), visual-language-action (VLA) models, etc.) and one or more machine learning models and / or other types of machine learning models.

[0045] The systems and methods described herein may be used by, but are not limited to, the following: non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, spacecraft, ships, shuttles (e.g., robotaxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones, and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, as examples but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and supervision, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets (e.g., NVIDIA's Omniverse), cloud computing and / or any other suitable application.

[0046] The disclosed embodiments can be included in a variety of different systems, such as motor vehicle systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines, etc.), systems implemented using robots, aviation systems, medical systems, rowing systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing language models (such as large language models (LLM), visual language models (VLM), visual-language-action (VLA) models, and / or multimodal language models), systems using or deploying one or more inference microservices, systems containing one or more machine learning models and OS-level virtualization packages (e.g., containers) deployed in services or microservices, systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0047] refer to Figure 1 , Figure 1This is an example data flow diagram illustrating the component interconnections and data information flow of a 3D object detection system 100 according to some embodiments of the present disclosure. It should be understood that the arrangements and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or as alternatives to the arrangements and elements shown, and some elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and location (e.g., at the edge, in the field (e.g., locally hosted servers, remotely located in one or more computing or server devices in one or more data centers such as the cloud), and / or on local devices, vehicles, or machines). The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., central processing units (one or more CPUs), graphics processing units (one or more GPUs), microprocessors, microcontrollers, embedded processors, digital signal processors (DSPs), image signal processors (ISPs), physical processing units (PPUs), field-programmable gate arrays (FPGAs), one or more accelerators (e.g., deep learning accelerators (DLAs), deep learning accelerator clusters (XNNs), neural network accelerators (NNAs) and / or neural processing units (NPUs), programmable vision accelerators (PVAs), optical flow accelerators (OFAs), etc.), application-specific integrated circuits (ASICs), data processing units (DPUs), quantum processors, etc.) that execute instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 7A-7E Example machine 700, Figure 8 Example computing ecosystem 800, Figure 9 Example generative language model system 900 and / or Figure 10 Example computing device 1000 uses components, features, and / or functions similar to those of other computing devices to perform the operation.

[0048] The 3D object detection system 100 can receive sensor data 105 from one or more sensors 102. One or more sensors 102 may be located on the self-machine and capture sensor data 105 relating to the self-machine or its surrounding environment. One or more sensors 102 may be positioned and / or oriented to capture similar or different parts of the environment. Figure 1In the example shown, one or more sensors 102 may include one or more LiDAR sensors 103 and one or more image sensors 104. One or more LiDAR sensors 103 and one or more image sensors 104 may be positioned and / or oriented to capture similar portions of the environment or different portions of the environment. For example, at least some of the LiDAR sensors 103 and one or more image sensors 104 may be positioned (e.g., close to each other) and oriented to capture similar portions of the environment, and / or at least some of the LiDAR sensors 103 and one or more image sensors 104 may be positioned (e.g., far from each other) and oriented to capture different portions of the environment.

[0049] One or more LiDAR sensors 103 may include, but are not limited to, any type of LiDAR-based sensor, such as, but not limited to, those described herein with respect to machine 700 (e.g., one or more LiDAR sensors 764) and / or those described with respect to other vehicles or objects (in some examples, such as robotic devices, virtual reality (VR) systems, augmented reality (AR) systems, mixed reality systems, etc.). Sensor data 105 from one or more LiDAR sensors 103 may include, but is not limited to, point cloud data or other types of sensor data from any type of LiDAR-based sensor used for one or more LiDAR sensors 103.

[0050] One or more image sensors 104 may include, but are not limited to, any type of image sensor, such as, but not limited to, those described herein with respect to machine 700 (e.g., one or more cameras) and / or those described with respect to other vehicles or objects (in some examples, such as robotic devices, virtual reality (VR) systems, augmented reality (AR) systems, mixed reality systems, etc.). Sensor data 105 may include, but is not limited to, red, green, and blue (RGB) image data, infrared (IR) image data, depth image data, or other types of sensor data from any type of image sensor for one or more image sensors 104.

[0051] The 3D object detection system 100 may include one or more models 106 that can generate feature representations 108 based on sensor data 105 input to the 3D object detection system 100. The one or more models 106 of the 3D object detection system 100 may include one or more machine learning models. For example, one or more models 106 may include one or more encoders and may include a single-stage or multi-stage process for generating the feature representations 108. The feature representations 108 may include a representation of a set of features associated with detected objects in the environment. As used herein, a feature can refer to any feature that captures or indicates spatial patterns or boundaries associated with objects in the environment. The feature representations 108 may (e.g., using tensors) represent any number of features in a viewpoint or space, and these features can be transformed to a single viewpoint or space. The feature representations 108 may include, for example, a bird's-eye view (BEV) viewpoint or BEV space that captures features of the scene represented by the sensor data 105.

[0052] In some embodiments, one or more sensors 102 may include a single type of sensor (e.g., one or more LiDAR sensors 103 or one or more image sensors 104), and one or more models 106 may be unimodal. One or more models 106 may receive sensor data 105 from the single type of sensor 102 as input and generate one or more vectors based on the sensor data 105. For example, one or more models 106 may encode the sensor data 105 from the single type of sensor 102 to generate one or more vectors. The one or more vectors may include vector embeddings, encodings, or other complex representations adapted to that type of sensor data 105 in the manner described herein. One or more vectors are output from one or more models 106 as feature representations 108.

[0053] In some embodiments, feature representation 108 may be a unified feature representation. A unified feature representation, or unified feature representation, generally refers to the identified feature representation associated with multiple sensors, such as one or more LiDAR sensors 103 and one or more image sensors 104. Thus, various features from different types of sensors, such as one or more LiDAR sensors 103 and one or more image sensors 104, can be combined or fused into a single unified feature representation. For example, in cases where features derived from sensor data 105 from one or more LiDAR sensors 103 and features derived from sensor data 105 from one or more image sensors 104 are to be represented in a unified feature representation, the unified feature representation may be in the form of a BEV representation. In this way, features associated with one or more LiDAR sensors 103 and one or more image sensors 104 can be fused or aggregated in a unified BEV space or viewpoint to generate a unified feature representation.

[0054] In some embodiments, one or more sensors 102 may include multiple types of sensors, and one or more models 106 may include multiple models that are unimodal. One or more unimodal models 106 may receive sensor data 105 of a corresponding type as input and generate one or more vectors based on the sensor data 105 of that corresponding type. For example, each model in one or more models 106 may encode the sensor data 105 of the corresponding type to generate one or more vectors. One or more vectors may include vector embeddings, encodings, or other complex representations adapted to the sensor data 105 of the corresponding type in the manner described herein. One or more vectors may then be concatenated or otherwise combined (e.g., using another model) to generate feature representation 108.

[0055] In some embodiments, one or more sensors 102 may include multiple types of sensors, and one or more models 106 may include at least one model that is multimodal. One or more multimodal models 106 may receive sensor data 105 of a corresponding type as input and generate one or more vectors based on different types of sensor data 105. For example, one or more multimodal models 106 may encode the sensor data 105 to generate one or more vectors. One or more vectors may include vector embeddings, encodings, or other complex representations that are applied to different types of sensor data 105 in a manner described herein. One or more vectors are output from one or more models 106 as feature representations 108.

[0056] The 3D object detection system 100 may include an evidential deep learning model 110, which may generate one or more object presence probabilities 112 and one or more uncertainty estimates 114 based on feature representation 108. The evidential deep learning model 110 may generate and output the object presence probabilities 112 based on feature representation 108. The object presence probability 112 for each cell and class of feature representation 108 may be generated based on parameters (e.g., α and β) of the probability distribution of each cell (e.g., each BEV cell) and class of feature representation 108. The object presence probability 112 output by the evidential deep learning model 110 may include heatmap data indicating the probability that an object is located at a specific 3D coordinate. In addition to predicting the location of the object (e.g., where the object is located), the object presence probability 112 may also include class prediction (e.g., what type of object is detected). For example, the object presence probability 112 may include the probability that the center of an object of a particular class is located within a cell of feature representation 108. In some embodiments, the object presence probability 112 indicates the proportion of positive evidence (e.g., a measure of the amount of support collected from the data that favors or disfavors classifying a sample into a certain category) in the total evidence (e.g., a measure of the amount of support collected from the data that favors or disfavors classifying a sample into a certain category) of the units and categories of feature representation 108. The object presence probability 112 increases as positive evidence represents a larger portion of the total evidence.

[0057] The evidence-based deep learning model 110 can also generate and output uncertainty estimates 114 corresponding to the object presence probability 112 based on the feature representation 108. Each uncertainty estimate in the uncertainty estimates 114 corresponds to a corresponding object presence probability 112. For example, the uncertainty estimate 114 corresponding to a particular object presence probability 112 can be generated based on the same parameters (e.g., α and β) of the probability distributions (e.g., Beta distributions) of the units and categories of the feature representation 108 used to generate that particular object presence probability 112. In addition to location uncertainty (e.g., where the object is located), the uncertainty estimates 114 corresponding to the object presence probability 112 can also capture category uncertainty (e.g., what type of object is detected). For example, in generating the object presence probability 112 for each unit and category of the feature representation 108, a corresponding uncertainty estimate 114 for each unit and category of the feature representation 108 can be generated. The uncertainty estimate 114 can indicate the level of uncertainty when predicting the object presence probability 112, which can be inversely proportional to the total evidence. That is, the level of uncertainty indicated by the uncertainty estimate 114 decreases as the total evidence increases.

[0058] For object detection, there may be an inherent imbalance towards the negative class, which could be due to most detections corresponding to the background and leading to overconfidence in positive detections (e.g., low uncertainty estimates), thus biasing the uncertainty estimates. To train the evidence deep learning model 110 for multi-label classification, the loss function can be determined by calculating the Bayesian risk with respect to the class predictor. In some embodiments, the combined loss function of the evidence deep learning model 110 can be used as follows:

[0059]

[0060] Where S is the number of training scenarios, and λ≥0 is the regularization parameter.

[0061] Given the i-th data point, the object existence probability 112 and uncertainty estimate 114 can be obtained using a Beta distribution (β(α)). ij ,β ij Modeling can be performed, and the EDL loss term of the combined loss function can be defined as follows:

[0062]

[0063] Where ψ(·) is the digamma function (the logarithmic derivative of the γ function, for example,

[0064] The first term of the EDL loss term in the combined loss function can correspond to the cell in feature representation 108 where the actual object center is located (e.g., Furthermore, a digamma-based Bayesian risk loss can be calculated for each of these units, and this digamma-based Bayesian risk loss can be scaled using a factor based on Gaussian focus loss (GFL). This helps reduce the influence of well-classified examples and focuses on examples that are harder to misclassify during training. The GFL-based factor of the first term of the EDL loss term of the combined loss function can be expressed as:

[0065] (1-α ij / (α ij +β ij )) γ

[0066] The second term of the EDL loss term in the combined loss function can correspond to a unit of feature representation 108 in which no object is located (e.g., The Bayesian risk can be similarly calculated and weighted using GFL-based factors to focus on more difficult negative examples. The GFL-based factor for the second term of the EDL loss term in the combined loss function can be expressed as:

[0067] (α ij / (α ij +β ij )) γ

[0068] The reduction term can also be applied to the second term of the EDL loss term in the combined loss function, which reduces the penalty applied to predictions near the object center. The reduction term of the second term of the EDL loss term in the combined loss function can be expressed as:

[0069]

[0070] The combined loss function of the evidence-based deep learning model 110 may also include a regularization term to manage uncertainty by penalizing the model when it generates incorrect or overconfident predictions. The goal is to reduce misleading evidence, especially when the model makes incorrect predictions. When predictions are incorrect, regularization can be applied by encouraging the model to revert to a uniform prior representing high uncertainty (e.g., a Dirichlet prior), thereby penalizing misleading evidence and avoiding overconfidence errors. The regularization term of the combined loss function can be defined as follows:

[0071]

[0072] in The sum of ⊙ is the Hadamard product. It is a β function that normalizes the distribution and

[0073] In some embodiments, the 3D object detection system 100 includes one or more bounding shape predictors 116 that can generate one or more predicted bounding shapes 118 based on object presence probability 112. The one or more bounding shape predictors 116 may include one or more models (e.g., machine learning models) that can generate one or more predicted bounding shapes 118 based on object presence probability 112 (e.g., heatmap data) and can provide one or more predicted bounding shapes 118 with respect to feature representation 108. The one or more predicted bounding shapes 118 may include one or more bounding boxes or other shapes suitable for establishing the boundaries of objects of one or more categories based on feature representation 108. In addition to or instead of the object presence probability 112 and / or uncertainty estimate 114, the 3D object detection system 100 may also output one or more predicted bounding shapes 118 for further use by other systems.

[0074] refer to Figure 2 , Figure 2This is an example data flow diagram illustrating the component interconnections and data information flow of an out-of-distribution (OOD) detection system 200 according to some embodiments of the present disclosure. Figure 2 As shown, the OOD detection system 200 may include one or more uncertainty estimation aggregators 202 and one or more OOD detection functions 206. It should be understood that the arrangements and other arrangements described herein are illustrative only. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or as alternatives to the arrangements and elements shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and location. The various functions described herein as being performed by entities can be implemented by hardware, firmware, and / or software. For example, various functions can be implemented by a processor that executes instructions stored in memory.

[0075] The OOD detection system 200 can receive (e.g., from the 3D object detection system 100) an uncertainty estimate 114 associated with the scene. The scene can be represented by a feature representation 108, and corresponds to sensor data 105 used to generate the feature representation 108, as described above. Figure 1 The uncertainty estimation 114 discussed may include uncertainty estimates 114 corresponding to each unit and each category of the feature representation 108 of the scene.

[0076] One or more uncertainty estimation aggregators 202 of the OOD detection system 200 may include one or more components that can aggregate uncertainty estimates 114 for a specific scenario to generate a scenario-level uncertainty 204, which represents the aggregated uncertainty estimates of the scenario across all units and categories of the feature representation 108. One or more uncertainty estimation aggregators 202 may include a single aggregator that can receive all uncertainty estimates 114 for a specific scenario and combine these uncertainty estimates 114 to generate the scenario-level uncertainty 204. Alternatively, one or more uncertainty estimation aggregators 202 may include multiple aggregators that can receive and combine subsets of uncertainty estimates 114 to generate intermediate aggregated uncertainty estimates, which can be provided to another aggregator that can combine the intermediate aggregated uncertainty estimates to generate the scenario-level uncertainty 204. Uncertainty estimation aggregator 202 can combine uncertainty estimate 114, for example, by averaging uncertainty estimate 114 or intermediate aggregated uncertainty estimates, and scene-level uncertainty 204 can represent the average uncertainty level of the scene across all units and categories of feature representation 108.

[0077] Scene-level uncertainty 204 can be provided to OOD detection function 206, which can determine whether a scene is an OOD scene based on scene-level uncertainty 204. In some embodiments, OOD detection function 206 can compare scene-level uncertainty 204 with an OOD threshold. The OOD threshold can include a threshold uncertainty level indicating an OOD scene. The OOD threshold can be determined based on a variety of factors, including but not limited to empirical data, system specifications, and / or other system performance requirements. If OOD detection function 206 determines that scene-level uncertainty 204 exceeds the OOD threshold, OOD detection function 206 can output an OOD indication 208, which indicates that the scene associated with scene-level uncertainty 204 is an OOD scene. Similarly, if OOD detection function 206 determines that scene-level uncertainty 204 does not exceed the OOD threshold, OOD detection function 206 can output an OOD indication 208, which indicates that the scene associated with scene-level uncertainty 204 is an in-distribution scene. If the scene-level uncertainty 204 does not exceed the OOD threshold, the OOD detection function 206 will not output an OOD indication 208.

[0078] In some embodiments, the OOD detection function 206 may include one or more models that can generate an OOD indication 208 based on scene-level uncertainty 204. The OOD detection function 206 may include one or more machine learning models trained to identify or predict OOD scenes based on scene-level uncertainty 204. For example, the OOD detection function 206 may be trained on a training set in a dataset (e.g., the training set in the nuScenes dataset) and consider scenes from a corresponding test set in the dataset (e.g., the test set in the nuScene dataset) as in-distribution samples, while considering scenes from another dataset (e.g., the Waymo test set) as OOD samples. The OOD detection function 206 may output the OOD indication 208 based on the predictions of one or more models regarding whether a scene is an OOD scene or an in-distribution scene.

[0079] OOD indicator 208 can be output to another component of the system, which can use OOD indicator 108 to determine how to collect training data. For example, the following section discusses... Figures 7A-7E One or more controllers 736 of the machine 700 under discussion can determine, based on OOD indication 208, when (e.g., in one or more data stores 716 and / or one or more data stores 728) to store sensor data from at least some of the sensors of the machine 700. If OOD indication 208 indicates that the scene is OOD, one or more controllers 736 of the machine 700 can determine to store the sensor data (e.g., in one or more data stores 716 and / or one or more data stores 728). However, if OOD indication 208 indicates that the scene is an in-distribution scene, one or more controllers 736 of the machine 700 can determine that the sensor data should be discarded or not stored. In this way, only sensor data corresponding to situations not well covered in the training data used to train the evidence deep learning model 110 can be stored, which can provide memory and cost savings associated with acquiring real-world training data.

[0080] OOD indicator 208 can also be output to another component of the system, which can use OOD indicator 108 to adjust the operation of the system. For example, one or more controllers of the vehicle (e.g., hereinafter referred to as...) Figures 7A-7EOne or more controllers 736 of the machine 700 under discussion can determine, based on OOD indication 208, when to stop outputting certain operating commands (e.g., signals indicating commands) for controlling the vehicle and hand over control of the vehicle to the driver. If OOD indication 208 indicates that the scenario is OOD, one or more controllers of the vehicle can stop outputting certain operating commands and hand over control of the vehicle to the driver.

[0081] refer to Figure 3 , Figure 3 The illustrations are example data flow diagrams showing the component interconnections and data information flow of the surrounding shape error detection system 300 according to some embodiments of the present disclosure. Figure 3 As shown, the bounding shape error detection system 300 may include one or more uncertainty estimation aggregators 302 and bounding shape error detection functions 306. It should be understood that this arrangement and other arrangements described herein are illustrative only. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or as alternatives to the arrangements and elements shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and location. The various functions described herein as being performed by entities can be implemented by hardware, firmware, and / or software. For example, various functions can be implemented by a processor that executes instructions stored in memory.

[0082] The bounding shape error detection system 300 can receive at least one predicted bounding shape 118 and (e.g., from the 3D object detection system 100) an uncertainty estimate 114 associated with the predicted bounding shape 118. (As mentioned above...) Figure 1 The proposed bounding shape 118 can be generated based on the object existence probability 112. The uncertainty estimate 114 associated with the proposed bounding shape 118 may include the uncertainty estimate 114 corresponding to each unit and each category of the feature representation 108 within the proposed bounding shape 118.

[0083] The bounding shape error detection system 300 may include one or more uncertainty estimation aggregators 302 that can aggregate uncertainty estimates 114 of the predicted bounding shape 118 to generate a bounding shape uncertainty 304, which represents the aggregated uncertainty estimate of the predicted bounding shape 118 on all cells of the category and feature representation 108 in the predicted bounding shape 118. The one or more uncertainty estimation aggregators 302 may include a single aggregator that can receive all uncertainty estimates 114 of the predicted bounding shape 118 and combine these uncertainty estimates 114 to generate the bounding shape uncertainty 304. Alternatively, the one or more uncertainty estimation aggregators 302 may include multiple aggregators that can receive and combine subsets of uncertainty estimates 114 to generate intermediate aggregated uncertainty estimates, which can be provided to another aggregator that can combine the intermediate aggregated uncertainty estimates to generate the bounding shape uncertainty 304. One or more uncertainty estimate aggregators 302 can combine uncertainty estimate 114, for example, by averaging uncertainty estimate 114 or intermediate aggregated uncertainty estimates, and enclosing shape uncertainty 304 can represent the average level of uncertainty in predicting enclosing shape 118.

[0084] Enclosing shape uncertainty 304 can be provided to enclosing shape error detection function 306, which can determine whether the predicted enclosing shape 118 may have an error (e.g., positioning error) based on the enclosing shape uncertainty 304. In some embodiments, the enclosing shape error detection function 306 can compare the enclosing shape uncertainty 304 with an enclosing shape error threshold. The enclosing shape error threshold can include a threshold uncertainty level indicating the enclosing shape error. The enclosing shape error threshold can be determined based on a variety of factors, including but not limited to empirical data, system specifications, and / or other system performance requirements. If the enclosing shape error detection function 306 determines that the enclosing shape uncertainty 304 exceeds the enclosing shape error threshold, the enclosing shape error detection function 306 can output an enclosing shape error indication 308, which indicates that the predicted enclosing shape 118 may have at least one error (e.g., positioning error). Similarly, if the bounding shape error detection function 306 determines that the bounding shape uncertainty 304 does not exceed the bounding shape error threshold, the bounding shape error detection function 306 can output a bounding shape error indication 308, which indicates that no error in the predicted bounding shape 118 associated with the bounding shape uncertainty 304 has been detected. If the bounding shape uncertainty 304 does not exceed the bounding shape error threshold, the bounding shape error detection function 306 does not output a bounding shape error indication 308.

[0085] In some embodiments, the bounding shape error detection function 306 may include one or more models that can generate a bounding shape error indication 308 based on the bounding shape uncertainty 304. The bounding shape error detection function 306 may include one or more machine learning models trained to predict bounding shape errors based on the bounding shape uncertainty 304. The bounding shape error detection function 306 may be trained to perform a binary classification (e.g., having error or being accurate) on the predicted bounding shape 118 based on the bounding shape uncertainty 304. For example, one or more bounding shapes 118 with an intersection-over-union (IoU) ratio below a threshold (e.g., 0.3) with one or more ground truth bounding shapes may be considered to have errors, and the bounding shape error detection function 306 may be trained to predict when the predicted bounding shape 118 has an error based on the bounding shape uncertainty 304. The bounding shape error detection function 306 may output the bounding shape error indication 308 based on the predictions of one or more models regarding whether the predicted bounding shape has an error.

[0086] The enclosure shape error indicator 308 can be output to another component of the system, which can use the enclosure shape error indicator 308 to adjust the operation of the system. For example, one or more controllers of the vehicle (e.g., hereinafter referred to as...). Figures 7A-7E One or more controllers 736 of the machine 700 under discussion may determine when to stop outputting certain operating commands (e.g., signals indicating commands) for controlling the vehicle based on the enclosure shape error indication 308, and hand over control of the vehicle to the driver. If the enclosure shape error indication 308 indicates (e.g., near the path of the machine 700) that the predicted enclosure shape 118 may include at least one error, one or more controllers of the vehicle may stop outputting certain operating commands and hand over control of the vehicle to the driver.

[0087] refer to Figure 4 , Figure 4 The illustrations are example data flow diagrams showing the component interconnections and data information flow of a missed object detection system 400 according to some embodiments of the present disclosure. Figure 4 As shown, the missing object detection system 400 may include one or more splicers 402 and one or more models 404 that generate missing object detection confidence values ​​406. It should be understood that the arrangements and other arrangements described herein are illustrative only. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or as alternatives to the arrangements and elements shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and location. The various functions described herein as being performed by entities can be implemented by hardware, firmware, and / or software. For example, various functions can be implemented by a processor that executes instructions stored in memory.

[0088] The missing object detection system 400 can receive feature representation 108, object presence probability 112, and (e.g., from 3D object detection system 100) uncertainty estimate 114 associated with the scene. The scene can be represented by feature representation 108 and associated with the features used to generate the scene as described above. Figure 1The sensor data 105 corresponds to the feature representation 108 discussed. The object presence probability 112 and the uncertainty estimate 114 correspond to each unit and each category of the feature representation 118 of the scene. In some embodiments, the missing object detection system 400 utilizes only a subset of the object presence probability 112 and the corresponding uncertainty estimate 114. For example, the subset of the object presence probability 112 and the corresponding uncertainty estimate 114 used by the missing object detection system 400 may correspond to units of the feature representation 108 in which the object presence probability 112 is less than or equal to a threshold, such that one or more bounding shape predictors 116 do not generate one or more predicted bounding shapes 118 for these units.

[0089] exist Figure 4 In the example shown, feature representation 108, object existence probability 112, and uncertainty estimate 114 can be provided to one or more concatenation units 402. One or more concatenation units 402 can concatenate feature representation 108, object existence probability 112, and uncertainty estimate 114 to generate one or more concatenation vectors 403. One or more concatenation vectors 403 can include high-dimensional vectors (e.g., tensors) that capture representations of feature representation 108, object existence probability 112, and uncertainty estimate 114. One or more concatenation vectors 403 can be provided as input (e.g., conditional) to one or more models 404.

[0090] One or more models 404 of the missing object detection system 400 can generate missing object detection confidence values ​​406 for units of feature representation 108 based on one or more concatenation vectors 403. The missing object detection confidence value 406 can indicate the probability that the evidence deep learning model 110 and one or more bounding shape predictors 116 may have missed detecting at least one object based on feature representation 108. One or more models 404 of the missing object detection system 400 may include one or more machine learning models trained to generate missing detection probabilities (e.g., indicating false negatives) based on feature representation 108, object presence probability 112, and uncertainty estimate 114 (e.g., as represented by one or more concatenation vectors 403). For example, the above-mentioned... Figure 1 The described evidence deep learning model 110 uses similar objectives, losses, and training processes to train one or more models 404, but one or more models 404 can receive object presence probability 112 and uncertainty estimate 114 as inputs, and can be trained using a subset of units of feature representation 108 with low values ​​(e.g., less than a threshold) of object presence probability 112.

[0091] One or more missing object detection indications 408 can be output to another component of the system, which can use one or more missing object detection indications 408 to adjust the operation of the system. For example, one or more controllers of a vehicle (e.g., hereinafter referred to as...) Figures 7A-7E One or more controllers 736 of the machine 700 under discussion may determine when to stop outputting certain operating commands (e.g., signals indicating commands) for controlling the vehicle based on one or more missed object detection indications 408, and hand over control of the vehicle to the driver. If one or more missed object detection indications 408 indicate that the detection of an object (e.g., near the path of the machine 700) may have been missed, one or more controllers of the vehicle may stop outputting certain operating commands and hand over control of the vehicle to the driver.

[0092] refer to Figure 5 , Figure 5 The illustration shows an example data flow diagram of an example automatic annotation system 500 according to some embodiments of the present disclosure. The automatic annotation system 500 may include one or more automatic annotation models 502, one or more scene annotators 504, one or more bounding shape annotators 506, and one or more missing object detection annotators 508. It should be understood that the arrangements and other arrangements described herein are illustrative by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or as alternatives to the arrangements and elements shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and location. The various functions described herein as being performed by entities can be implemented by hardware, firmware, and / or software. For example, various functions can be implemented by a processor that executes instructions stored in memory.

[0093] The automatic annotation system 500 can receive one or more predicted bounding shapes 118, OOD indicators 208, bounding shape error indicators 308, and one or more missing object detection indicators 408 as input, and the automatic annotation system 500 can generate one or more automatically annotated driving scenarios 510 based on these inputs. Although Figure 5 The automatic annotation system 500 is shown to receive data from the above-mentioned... Figures 2-4 The inputs to the OOD detection system 200, the bounding shape error detection system 300, and the missing object detection system 400 are described, but it should be understood that in some embodiments, the automatic annotation system 500 may include the OOD detection system 200, the bounding shape error detection system 300, and / or the missing object detection system 400.

[0094] One or more automatic labeling models 502 may include one or more machine learning models trained to generate one or more labels for one or more automatically labeled driving scenarios 510 associated with feature representations 108 based on one or more predicted bounding shapes 118. The one or more labels for one or more automatically labeled driving scenarios 510 generated by one or more automatic labeling models 502 may include one or more bounding shapes and one or more corresponding category labels to locate and classify objects in one or more automatically labeled driving scenarios 510. One or more automatic labeling models may generate one or more labels based at least on one or more predicted bounding shapes 118.

[0095] The automatic annotation system 500 may include one or more OOD scene annotators 504, which re-annotate one or more driving scenes based on the OOD indication 208 provided by the OOD detection system 200. For example, if the OOD indication 208 provided by the OOD detection system 200 indicates that the driving scene is OOD, then one or more OOD scene annotators 504 may annotate the driving scene as OOD. If the OOD indication 208 provided by the OOD detection system 200 indicates that the driving scene is within the distribution, then one or more OOD scene annotators 504 may not annotate the driving scene.

[0096] The automatic annotation system 500 may include one or more bounding shape annotators 506, which may re-annotate one or more bounding shapes in a driving scene to identify one or more bounding shape positioning errors. For example, if the bounding shape error indication 308 provided by the bounding shape error detection system 300 indicates that the predicted bounding shape 118 may include at least one error (e.g., positioning error), then one or more bounding shape annotators 506 may annotate the bounding shape in the driving scene corresponding to the predicted bounding shape 118 as having an error. If the bounding shape error indication 308 provided by the bounding shape error detection system 300 indicates that the predicted bounding shape of the driving scene may not include an error, then one or more bounding shape annotators 506 may not annotate the driving scene.

[0097] The automatic annotation system 500 may further include one or more missing object detection annotators 508, which can re-annotate parts of the driving scene based on one or more missing object detection indications 408 provided by the missing object detection system 400 to identify potentially missing objects in the driving scene. For example, if one or more missing object detection indications 408 provided by the missing object detection system 400 indicate that the detection of an object may have been missed, then one or more missing object detection annotators 508 can annotate the driving scene to identify parts of the driving scene where objects may have been missed. If one or more missing object detection indications 408 provided by the missing object detection system 400 indicate that no missing object detection has been detected, then one or more missing object detection annotators 508 may not annotate the driving scene.

[0098] The one or more automatically labeled driving scenes 510 output by the automatic labeling system 500 may include one or more labels generated by one or more automatic labeling models 502, one or more OOD scene labelers 504, one or more bounding shape labelers 506, and one or more missing object detection labelers 508. The one or more automatically labeled driving scenes 510 may include, but are not limited to, one or more RGB images, one or more IR images, one or more depth images, one or more point clouds, or other types of scenes corresponding to sensor data 105 from one or more sensors 102.

[0099] In addition to one or more automatically labeled driving scenarios 510, the automatic labeling system 500 may also output one or more verification required indicators 512. The one or more verification required indicators 512 can identify one or more specific labels generated by the automatic labeling system 500 that require (e.g., human) verification. If one or more OOD scene labelers 504, one or more bounding shape labelers 506, and / or one or more missing object detection labelers 508 generate one or more labels for automatically labeled driving scenarios 510, one or more verification required indicators 512 may be output. The one or more verification required indicators 512 may be specific to one or more specific labels generated by one or more OOD scene labelers 504, one or more bounding shape labelers 506, and / or one or more missing object detection labelers 508.

[0100] One or more automatically labeled driving scenarios 510 that are output without the required verification indicator 512 or that have been verified can be stored (e.g. in a data storage system) and retrieved from the data storage system by a machine learning model training system that can be used to train one or more models for a variety of different applications (e.g., autonomous vehicle navigation, etc.).

[0101] Now for reference Figure 6 , Figure 6 This is a flowchart illustrating a method 600 for 3D object detection according to some embodiments of this disclosure. Each block of the method 600 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (such as, but not limited to, the processors described herein) that execute instructions stored in one or more memories or memory systems. In some embodiments, the computer process can also be embodied as computer-usable instructions stored on a computer storage medium. The method can be provided by a standalone application, service, or managed service (alone or in combination with another managed service), application programming interface (API), and / or plug-in to another product, etc. Additionally, as an example, regarding Figures 1-5 The system described herein describes method 600. However, additionally or alternatively, the method may be performed by any single system or any combination of systems, including but not limited to the system described herein.

[0102] At box B602, method 600 includes generating a feature representation based at least on sensor data from one or more sensors in the environment. The sensor data may be captured by one or more sensors of the self-machine (e.g., one or more sensors 102) and may be related to the self-machine and / or the environment surrounding the self-machine. The one or more sensors may include, for example, one or more LiDAR sensors and / or one or more image sensors, which may be positioned and / or oriented to capture similar or different parts of the environment. The sensor data may include, but is not limited to, RGB image data, IR image data, depth image data, point cloud data, or other types of sensor data from one or more sensors. The feature representation (e.g., feature representation 108) may be generated based on the sensor data from one or more sensors using one or more models (e.g., one or more models 106). The one or more models may include one or more machine learning models (e.g., one or more encoders) and may include a single-stage or multi-stage process for generating the feature representation. The feature representation may include a representation of a set of features associated with detected objects in the environment. Feature representations can (e.g., using tensors) represent features in any number of viewpoints or spaces, and these features can be transformed into a single viewpoint or space (e.g., BEV viewpoint or BEV space).

[0103] At box B604, method 600 includes: generating object presence probabilities using a first model based at least on feature representations. The first model may include an evidence deep learning model (e.g., evidence deep learning module 110). Object presence probabilities for each unit (e.g., each BEV unit) and class of the feature representations may be generated. Object presence probabilities may be generated based on parameters (e.g., α and β) of the probability distributions (e.g., a Beta distribution) of the units and classes of the feature representations. One of the parameters of the probability distribution (e.g., α) may represent positive evidence, while the other parameter (e.g., β) may represent negative evidence. Object presence probabilities may include heatmap data indicating the probability that an object is located at a specific 3D coordinate. In addition to predicting the location of the object (e.g., where the object is located), object presence probabilities may also include class prediction (e.g., what type of object is detected). For example, object presence probabilities may include the probability that the center of an object of a particular class is located within a unit of the feature representation. In some embodiments, the object presence probability indicates the proportion of positive evidence (e.g., a measure of the amount of support collected from the data that favors or disfavors classifying a sample into a certain category) in the total evidence (e.g., a measure of the amount of support collected from the data that favors or disfavors classifying a sample into a certain category) representing the units and categories of the feature representation. The object presence probability can increase as positive evidence represents a larger portion of the total evidence.

[0104] At box B606, method 600 includes: generating an uncertainty estimate corresponding to the object presence probability using a first model, at least based on a feature representation. The first model may include an evidence deep learning model (e.g., evidence deep learning module 110) and may be the same model used to generate the object presence probability. Each uncertainty estimate (e.g., uncertainty estimate 114) among the generated uncertainty estimates corresponds to a corresponding generated object presence probability. For example, the uncertainty estimate corresponding to a particular object presence probability may be generated based on the same parameters (e.g., α and β) of the probability distributions (e.g., a Beta distribution) of the units and categories used to generate the feature representation of a particular object presence probability. In addition to location uncertainty (e.g., where the object is located), the uncertainty estimate corresponding to the object presence probability may also capture category uncertainty (e.g., what type of object is detected). For example, uncertainty estimates corresponding to the object presence probability for each unit and category of the feature representation may be generated. The uncertainty estimate may indicate the level of uncertainty in predicting the object presence probability, which may be inversely proportional to the total evidence. For example, the level of uncertainty indicated by the uncertainty estimate decreases as the total evidence increases.

[0105] At box B608, method 600 includes: outputting an indication of object presence probability and uncertainty estimation. The indication of object presence probability and uncertainty estimation may include heatmap data for each cell and category of the feature representation and the corresponding uncertainty estimation of the heatmap data. In some embodiments, the indication of object presence probability and uncertainty estimation may include output derived from the object presence probability and uncertainty estimation. For example, one or more bounding shapes (e.g., one or more predicted bounding shapes 118) predicted based on the object presence probability (e.g., using one or more bounding shape predictors 116) may be output. One or more OOD scenes, one or more bounding shapes with errors, and / or one or more indications of missing objects generated based on the uncertainty estimation may also be output. Uncertainty estimations may be aggregated or combined in different ways to perform different types of detection on scenes corresponding to feature representations. An automatically labeled driving scene (e.g., automatically labeled driving scene 510) including one or more predicted bounding shapes and one or more indicators (e.g., labels) may be output, which identify the scene as OOD, identify one or more bounding shapes with errors, and / or identify the location of potentially missing objects. At least a portion of an automatically labeled scenario can be identified as requiring (e.g., human) verification based on uncertainty estimates (e.g., exceeding a threshold).

[0106] The systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles connected to one or more trailers, aircraft, ships, shuttles (e.g., robotaxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones, and / or other types of vehicles. Furthermore, the systems and methods described herein can be used for a wide range of purposes, including, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets (e.g., NVIDIA's Omniverse), cloud computing, and / or any other suitable application.

[0107] The disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines, etc.), systems implemented using robots, aviation systems, medical systems, marine systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing language models (such as large language models (LLM), visual language models (VLM), and / or multimodal language models), systems using or deploying one or more inference microservices, systems including deploying one or more machine learning models in services or microservices and OS-level virtualization packages (e.g., containers), systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in data centers, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0108] Example autonomous or semi-autonomous machines

[0109] Figure 7AExamples of sensor positions with corresponding fields of view or sensing fields of view of autonomous or semi-autonomous vehicles 700A, autonomous mobile robots (AMRs) 700B, and humanoid robots 700C according to some embodiments of this disclosure. While three types of machines 700 are shown, this is not intended to be limiting, and the machines 700 described herein may include vehicles, cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police or emergency vehicles, ambulances, boats, construction vehicles, underwater vehicles, robots (e.g., AMRs, humanoid robots, robotic arms, end effectors, forklifts, etc.), drones, aircraft, vehicles coupled to trailers (e.g., semi-trailer tractors for hauling goods), and / or another type of vehicle or machine (e.g., driverless and / or vehicles or machines accommodating one or more passengers). In some cases, vehicle 700A, AMR 700B, humanoid robot 700C, and / or other machine types may be collectively referred to herein as machine 700.

[0110] Regarding Vehicle 700A, autonomous and semi-autonomous vehicles are typically described by automation levels, defined by the National Highway Traffic Safety Administration (NHTSA) (a division of the U.S. Department of Transportation) and the Society of Automotive Engineers (SAE) in their "Classification and Definition of Terms Related to Driving Automation Systems for Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 700 may have one or more functions that meet Level 3 through Level 5 of autonomous driving. For example, Vehicle 700 may be able to provide driver assistance (Level 1), partial automation (Level 2, Level 2+, Level 2++), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the specific implementation. The term “autonomy” as used herein can include any and / or all types of autonomy of Machine 700 or other machines, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, provision of auxiliary autonomy, semi-autonomy, primary autonomy or other specified autonomy.

[0111] about Figure 7AThe sensors and their respective fields of view (not shown for clarity) or sensing fields (not shown for clarity) are an example embodiment and are not intended to be limiting. Although not shown, each sensor may have a corresponding field of view (e.g., a 360-degree field of view around camera 768D, a 180-degree field of view around wide-angle camera 768B, a 360-degree sensing field of view of LiDAR sensor 764, etc.). For example, only a subset of the sensors shown may be included, additional sensors may be included, alternative sensors may be included, the number of each sensor mode may be different, the sensor modes may be different (e.g., LiDAR or RADAR may not be included, SONAR, thermal sensors, etc. may be included), and the sensor positions may differ from those shown on vehicles 700A, AMR 700b, and / or humanoid robots 700c, etc. For example, for vehicle 700A, the position, number, mode, and / or other sensor information may differ depending on the type (e.g., SUV, truck, car, robot, motorcycle, etc.), size (e.g., 18-wheeler, transport vehicle, small car, etc.), and associated functions (e.g., L2 vs. L5). Similarly, for the AMR 700B and / or the humanoid robot 700C, the shape, size, purpose, implementation method, model, etc., can determine the number and type of sensors used.

[0112] like Figure 7A As shown, autonomous or semi-autonomous vehicles 700A, AMR 700B, and humanoid robots 700C may include different sensor types, numbers, and locations. As a non-limiting example, vehicle 700A may include twelve cameras 768, such as a front wide-angle camera (e.g., 120-degree field of view (FOV)), a front telephoto camera (e.g., 30-degree FOV), a side-rear left camera (e.g., 70-degree FOV), a side-rear right camera (e.g., 70-degree FOV), a front fisheye camera (e.g., 200-degree FOV), a rear fisheye camera (e.g., 200-degree FOV), a left fisheye camera (e.g., 200-degree FOV), a right fisheye camera (e.g., 200-degree FOV), a front telephoto satellite camera (e.g., 30-degree FOV), a rear telephoto camera (e.g., 30-degree FOV), a cross-left camera (e.g., 120-degree FOV), and a cross-right camera (e.g., 120-degree FOV). In this embodiment, the camera 768 may use a Gigabit Multimedia Serial Link (GMSL) interface (e.g., GMSL2) as input / output (I / O).

[0113] In some embodiments, although Figure 7AAs not shown, vehicle 700A may include an in-cabin occupant and / or driver monitoring system, which may include various sensors. For example, in-cabin sensors may include various cameras 768, such as a driver monitoring camera (e.g., located in front of the driver's seat at a 55-degree FOV facing the driver's seat), a front occupant monitoring camera (e.g., located in front of the front occupant seat at a 190-degree FOV facing the front occupant seat), and a rear occupant monitoring camera (e.g., located in front of the rear occupant seat at a 190-degree FOV facing the rear occupant seat). Similar to external cameras 768, in embodiments, internal cameras 768 may use a GMSL (e.g., GMSL2) interface for I / O.

[0114] As another non-limiting example, vehicle 700A may also include nine RADAR sensors 760. For example, vehicle 700A may include a front center imaging RADAR sensor (e.g., 120-degree FOV or sensing field), a left front corner RADAR sensor (e.g., 160-degree FOV or sensing field), a right front corner RADAR sensor (e.g., 160-degree FOV or sensing field), a right rear corner RADAR sensor (e.g., 160-degree FOV or sensing field), a left RADAR sensor (e.g., 160-degree FOV or sensing field), a right RADAR sensor (e.g., 160-degree FOV or sensing field), a left rear RADAR sensor (e.g., 50-degree FOV or sensing field), and a right rear RADAR sensor (e.g., 50-degree FOV or sensing field). In embodiments, the RADAR sensors 760 may use an Ethernet interface as I / O.

[0115] As a non-limiting example, the vehicle 700A may also include twelve ultrasonic sensors 762. Figure 7A As shown, the ultrasonic sensor can be placed along the front and rear bumpers of vehicle 700A and along the sides of vehicle 700A, and can be used to detect objects (static and dynamic) close to vehicle 700A. In some embodiments, the ultrasonic sensor 762 can use a DS13 interface as I / O.

[0116] As a non-limiting example, the vehicle 700A may also include a LiDAR sensor 764, such as a front-center LiDAR sensor (e.g., a 120-degree horizontal FOV or sensing field and a 30-degree vertical FOV or sensing field). In some embodiments, such as when using additional or alternative LiDAR sensors, the LiDAR sensor may have different horizontal and vertical fields of view or sensing fields. For example, the LiDAR sensor 764 may include a 360-degree horizontal FOV or sensing field (e.g., located in a rotating LiDAR sensor) and a 90-degree vertical FOV or sensing field. In some embodiments, the LiDAR sensor 764 may use an Ethernet interface as I / O.

[0117] As a non-limiting example, the Autonomous Mobile Robot (AMR) 700B may include three LiDAR sensors 764. For example, the topmost LiDAR sensor 764 may include a beam or 3D LiDAR sensor (e.g., a 360-degree horizontal and 90-degree vertical FOV or sensing field), and the front and rear LiDAR sensors may include planar or 2D LiDAR sensors (e.g., a 180-degree horizontal FOV or sensing field).

[0118] As a non-limiting embodiment, the AMR 700B may further include eight cameras 768, such as a front stereo camera (e.g., 120-degree FOV), a rear stereo camera (e.g., 120-degree FOV), a left stereo camera (e.g., 120-degree FOV), a right stereo camera (e.g., 120-degree FOV), a front fisheye camera (e.g., 202-degree ± 3-degree FOV), a rear fisheye camera (e.g., 202-degree ± 3-degree FOV), a left fisheye camera (e.g., 202-degree ± 3-degree FOV), and a right fisheye camera (e.g., 202-degree ± 3-degree FOV).

[0119] The AMR 700B may also include a charging port, charging port contacts, status indicators, one or more (e.g., four) RGB LEDs, one or more IMU sensors 766, a magnetometer, and a barometer. The AMR 700B is capable of high-precision time synchronization between sensors using hardware timestamps and PTPs over Ethernet with sensor acquisition times of less than 10 microseconds. In embodiments, the AMR 700B provides simultaneous camera capture within 100 microseconds of a single hardware trigger on all cameras 768, and can write sensor captures to disk at a rate of 4 GB / s for writing packets (e.g., writing to ROSbags of the Robot Operating System (ROS)). Therefore, the AMR 700B is capable of running ROS (e.g., NVIDIA's IsaacROS), can be remotely operated (as described herein), can map the environment, and can navigate the environment using vision cameras 768, LiDAR 764, and / or other sensor types or modalities.

[0120] The humanoid robot 700C may include (as a non-limiting example) a LiDAR sensor 764. For example, the LiDAR sensor 764 may include a beam or a 3D LiDAR sensor (e.g., a 360-degree horizontal and 90-degree vertical FOV or sensing field), or it may include a planar or 2D LiDAR sensor (e.g., a 180-degree horizontal FOV or sensing field).

[0121] As a non-limiting embodiment, the humanoid robot 700C may also include four cameras 768, such as a front stereo camera (e.g., 120-degree FOV), a rear stereo camera (e.g., 120-degree FOV), a front fisheye camera (e.g., 202-degree ± 3-degree FOV), and a rear fisheye camera (e.g., 202-degree ± 3-degree FOV).

[0122] As a non-limiting embodiment, the humanoid robot 700C may also include four ultrasonic sensors 762, such as a left arm ultrasonic sensor, a right arm ultrasonic sensor, a left leg ultrasonic sensor, and a right leg ultrasonic sensor.

[0123] The humanoid robot 700C may also include any number of actuators, such as those allowing control and manipulation of joints. For example, the humanoid robot 700C may include actuators that allow for various degrees of freedom (DoF) depending on the design. In a non-limiting embodiment, the humanoid robot 700C may have a total of 40 degrees of freedom (DoF) (e.g., 6DoF x2 for arms, 6DoF x2 for hands, 6DoF x2 for legs, 2DoF for the torso, and 2DoF for the neck). Actuators can convert energy into physical motion, thereby allowing actions such as joint movement, locomotion, and grasping / manipulation. For example, motors and servos can be used to perform joint movements to control the rotation of joints in an arm or manipulator and allow for reaching, grasping, and manipulating objects. Locomotion can be achieved by moving through the environment using wheels, tracks, or other mobility devices (robot legs). Grasping and manipulation can be performed using end effectors or hands / fingers, which can be equipped with actuators to grasp objects, apply forces, and perform specific tasks. In some examples, the humanoid robot 700C may include position and orientation sensors, such as encoders, gyroscopes, etc., to determine the robot 700C's position in space, thereby enabling position determination and motion tracking. In embodiments, the humanoid robot 700C may include force and pressure sensors to detect environmental interactions, enabling the robot 700C to grasp objects with appropriate force and avoid obstacles along its path. Perception sensors (e.g., cameras, LiDAR, RADAR, ultrasound, sonar (SONAR), etc.) may be used in conjunction with tactile sensors to enable the robot 700C to perceive objects, shapes, and textures, and to know when to begin and stop touching (in conjunction with force sensors that regulate the force used during touching). As a non-limiting example, the humanoid robot 700C may have a height of approximately 1-2 meters (e.g., 1.7 meters or 5 feet 6 inches), a weight of 50-70 kilograms, be able to move at speeds of 8 km / h or higher, and be able to carry a payload of 20-100 kilograms, depending on the system design and requirements.

[0124] In an embodiment, the humanoid robot 700C may include a dialogue system—such as a dialogue system driven by a language model (e.g., LLM, VLM, MMLM, VLA, etc.)—to help understand the environment, reason, and communicate with humans, animals, devices, and / or other robots, and / or make planning, control, and navigation decisions. Therefore, in addition to performing various tasks, the humanoid robot 700C can also use onboard sensors, microphones, and speakers to understand speech, audio, and visual cues, while also being able to communicate with the environment.

[0125] Referring to camera 768 of machine 700, the camera type of camera 768 may include, but is not limited to, digital cameras applicable to components and / or systems of machine 700. For vehicle 700a implementation, camera 768 may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may have any image capture rate, such as 30 frames per second (fps), 60 fps, 120 fps, 240 fps, etc., depending on the embodiment. The camera may use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red colorless colorless colorless (RCCC) color filter array, a red colorless colorless blue (RCCB) color filter array, a red blue green colorless (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a colorless pixel camera (e.g., a camera with an array of RCCC, RCCB, and / or RBGC color filters) can be used to improve light sensitivity.

[0126] The field of view includes cameras (e.g., front-facing cameras) in the area in front of the machine 700, which can be used for surround view to help identify the path and obstacles ahead, and, with the help of one or more controllers 736 and / or control SoCs, provide information crucial for generating an occupancy grid and / or determining preferred machine movement, trajectory, and / or path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including lane departure warning (“LDW”), adaptive cruise control (“ACC”), and / or other functions such as traffic sign recognition.

[0127] Various cameras can be used in front-mounted configurations, including, for example, monocular camera platforms that include complementary metal-oxide-semiconductor (“CMOS”) color imagers. Another example could be a wide-angle camera 768B, which can be used to perceive objects entering the field of view from the periphery (e.g., pedestrians, warehouse vehicles, other robots, pedestrian traffic, or bicycles). Furthermore, any number of long-range cameras 768E (e.g., a pair of long-range stereo cameras) can be used for depth-based object detection, especially for objects for which neural networks have not yet been trained. The long-range camera 768E can also be used for object detection and classification, as well as basic object tracking.

[0128] Any number of stereo cameras 768A can also be included in front-mounted and / or other (e.g., rear-mounted) configurations. In at least one embodiment, one or more stereo cameras 768A may include an integrated control unit that includes a scalable processing unit that can provide programmable logic (“FPGA”) and a multi-core microprocessor with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the machine 700 environment, including distance estimates of midpoints in the image (e.g., parallax or depth images). Alternative stereo cameras 768A may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip that can measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 768A may be used in addition to those described herein, or as an alternative to those described herein. For example, in some embodiments, stereo depth estimation can be performed using a camera other than a stereo camera (e.g., two monocular cameras with at least partially overlapping fields of view).

[0129] Cameras with a field of view including portions of the side environment of machine 700 (e.g., side-view cameras) can be used, for example, for surround view, to provide information for creating and updating occupancy grids, and to generate side collision warnings and / or indicate to AMR 700B or humanoid robot 700C, for example, the presence of objects, features, and / or people on the side. For example, surround cameras 768D can be mounted on machine 700. Surround cameras 768D can include wide-angle cameras 768B, fisheye cameras, 360-degree cameras, etc. For example, four fisheye cameras can be mounted on the front, rear, and sides of machine 700. In an alternative arrangement, machine 700 can use three surround cameras 768D (e.g., left, right, and rear), and can utilize one or more other cameras (e.g., front-facing cameras) as a fourth surround-view camera.

[0130] A camera 768 (e.g., a rear-view camera) having a field of view including a portion of the environment behind the machine 700 can be used to understand objects, features, people, and / or other information behind the machine 700, such as for parking assistance, surround view, rear collision warning, planning, control, and navigation determination, and / or to create and update occupancy grids, BEV images representing the environment, height maps, etc. A wide variety of cameras 768 can be used, including but not limited to those also suitable for use as front cameras (e.g., long-range and / or mid-range cameras 768E, stereo cameras 768A, infrared cameras 768C, etc.), rear cameras, side cameras, downward cameras, upward cameras, and / or similar cameras 768, as described herein.

[0131] Similarly, for LiDAR sensor 764, RADAR sensor 760, ultrasonic sensor 762 and / or other sensor modes or types, the location and placement of the sensors and their corresponding fields of view or sensing fields can be determined based on the use case, implementation or design of the particular machine 700.

[0132] For example, machine 700 includes a RADAR sensor 760, which can be used by machine 700 for long-range object detection, even in dark and / or inclement weather conditions. In embodiments, the RADAR functional safety level may be ASIL B. RADAR sensor 760 can be controlled and accessed for object tracking data using CAN and / or bus 702 (e.g., for transmitting data generated by RADAR sensor 760), and in some examples, raw data can be accessed using Ethernet. Various types of RADAR sensors can be used. For example, but not limited to, RADAR sensor 760 can be suitable for front, rear, and side radar applications. In some examples, a pulse Doppler RADAR sensor is used.

[0133] The RADAR sensor 760 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range lateral coverage, etc. In some examples, long-range radar can be used for adaptive cruise control (ACC) functions. A long-range RADAR system can provide a wide field of view, for example, within a 250m range, achieved through two or more independent scans. The RADAR sensor 760 can help distinguish between static and moving objects; ADAS systems can use it for emergency braking assistance and forward collision warning, and robots can use it to detect dynamic objects in various environments (e.g., low-light or no-light environments). A long-range RADAR sensor can include a single static multimodal RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record the machine 700's surroundings at higher speeds while minimizing peripheral interference (e.g., traffic from adjacent lanes). The other two antennas expand the field of view, enabling rapid detection of objects entering or leaving the machine's direct path (e.g., lanes).

[0134] Mid-range RADAR systems can include, for example, a range of up to 960m (front) or 80m (rear), and a field of view of up to 42 degrees (front) or 150 degrees (rear). Short-range RADAR systems can include, but are not limited to, RADAR sensors designed to be mounted at both ends of a side surface (e.g., rear bumper), allowing continuous monitoring of blind spots behind and beside a machine (e.g., vehicle, robot, etc.) using two beams. Therefore, short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0135] Machine 700 may also include an ultrasonic sensor 762. The ultrasonic sensor 762 may be located at the front, rear, and / or sides of machine 700 and may be used to assist near-field perception, such as for parking assistance, collision avoidance (e.g., for robot parts), and / or to create and update occupancy grids, evidence grid maps (EGMs), height maps, BEV images, and / or other representations of objects and features in the machine 700 environment. Multiple ultrasonic sensors 762 may be used, and different ultrasonic sensors 762 may be used for different detection ranges (e.g., 2.5 m, 4 m). For example, the ultrasonic sensor 762 may operate at an ASIL B functional safety level.

[0136] Machine 700 may include a LiDAR sensor 764. The LiDAR sensor 764 can be used for object and feature detection, pedestrian and other robot detection, emergency braking, collision avoidance, simultaneous localization and mapping (SLAM), free space detection, and / or other functions. In embodiments, the LiDAR sensor 764 may be of functional safety level ASIL B. In some examples, machine 700 may include multiple LiDAR sensors 764 (e.g., two, four, six, etc.), which can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0137] In some examples, the LiDAR sensor 764 may be able to provide a 360-degree field of view of objects and their distances. For example, a commercially available LiDAR sensor 764 may have a advertised range of approximately 900m, an accuracy of 2cm-3cm, and support for 900Mbps Ethernet connectivity. In some examples, one or more non-protruding LiDAR sensors 764 may be used. In such examples, the LiDAR sensor 764 can be implemented as a small device that can be embedded in the front, rear, side, top, and / or corner of the machine 700. In such examples, the LiDAR sensor 764 can provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, with a range of up to 200m even for low-reflectivity objects. A front-mounted LiDAR sensor 764 can be configured with a horizontal field of view between 45 and 135 degrees.

[0138] In some examples, LiDAR technology, such as 3D flash LiDAR, can also be used. 3D flash LiDAR uses a laser flash as a transmission source, illuminating the environment around a vehicle up to approximately 200m away. The flash LiDAR unit includes a receiver that records the laser pulse transmission time and the reflected light on each pixel, which in turn corresponds to the distance from the vehicle to the object. Flash LiDAR allows for the generation of highly accurate and distortion-free images of the surrounding environment with each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of machine 700. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras with no moving parts other than a fan (e.g., non-scanning LiDAR devices). Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of 3D distance point clouds and co-registration intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, LiDAR sensor 764 may be less susceptible to motion blur, vibration, and / or shock.

[0139] Figure 7BThis is an illustration of the location of sensors and components in an example autonomous or semi-autonomous vehicle 700A (also referred to herein as "vehicle 700", "self-vehicle 700", "self-machine 700", or "machine 700") according to some embodiments of this disclosure. While vehicle 700A is shown in the figure, this is not intended to be limiting, and similar components and / or sensors may be included on any other machine type without departing from the scope of this disclosure. For example, similar sensors and / or components may be used in vehicles, cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police cars, ambulances, ships, construction vehicles, underwater vehicles, robots (e.g., AMRs, humanoid robots, robotic arms, end effectors, forklifts, etc.), drones, aircraft, vehicles coupled to trailers (e.g., semi-trailers for hauling goods), and / or another type of vehicle or machine (e.g., driverless and / or capable of accommodating one or more passengers).

[0140] Figure 7CThis is a block diagram of an example system architecture for a machine 700 (e.g., an autonomous or semi-autonomous vehicle 700A, an autonomous mobile robot (AMR) 700B, a humanoid robot 700C, and / or other types of machines) according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are presented as examples only. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of the arrangements shown, and some elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components or together with other components, and can be implemented in any suitable combination and location (e.g., on a local device, vehicle, or edge machine, in the field (e.g., a locally hosted server), in a remote location (e.g., in one or more computing or server devices in one or more data centers in the cloud) and / or other locations). The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, microcontroller, embedded processor, digital signal processor (DSP), image signal processor (ISP), physical processing unit (PPU), field-programmable gate array (FPGA), accelerators (e.g., deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural network accelerator (NNA) and / or neural processing unit (NPU), programmable vision accelerator (PVA), optical flow accelerator (OFA), etc.), application-specific integrated circuit (ASIC), data processing unit (DPU), quantum processor, etc.) that execute instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 7A to 7E Example machine 700, Figure 8 Example computing ecosystem 800, Figure 9 Example Generative Language Model System 900 and / or Figure 10 The example computing device 1000 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the operation.

[0141] Figure 7CEach component, feature, and system of machine 700 is shown connected via bus 702 (or referred to as "machine communication network 702" or simply "communication network 702"). Bus 702 may include a Controller Area Network (CAN) data interface (or referred to herein as "CAN bus"). CAN can be a network within machine 700 used to help control various features and functions of machine 700, such as brake actuation, acceleration, braking, steering, windshield wipers, etc. CAN bus can be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CANID). CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button position, and / or other vehicle status indicators. CAN bus may conform to ASIL B standard. In some embodiments, in addition to or as an alternative to the CAN bus, bus 702 may include FlexRay, embedded buses (e.g., SPI, I2C), local interconnect links (LIN), NVIDIA's NVLink, USB (2.0, 3.0 and above), radio frequency (RF), Ethernet (e.g., 10BASE / 100BASE, 1000BASE, 10G, etc.), and / or other communication protocols or functions. Furthermore, while a single line is used to represent bus 702, this is not limiting. For example, there can be any number of buses 702, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 702 may be used to perform different functions and / or for redundancy. For example, a first bus 702 may be used for a collision avoidance function, while a second bus 702 may be used for actuation control. In any example, each bus 702 may communicate with any component of machine 700, and two or more buses 702 may communicate with the same component. In some examples, each computer or computing engine within each SoC 704, each controller 736, and / or machine 700 can access the same input data (e.g., input from sensors in the machine 700) and can be connected to a common bus, such as the CAN bus.

[0142] Machine 700 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, batteries, side mirrors, and / or other parts of the vehicle or machine. Machine 700 may include a propulsion system 750, such as an internal combustion engine, a hybrid power plant, an all-electric motor, a hydrogen fuel cell engine, and / or another type of propulsion system. Propulsion system 750 may be connected to the drivetrain of machine 700, which may include a transmission to enable propulsion of machine 700. Propulsion system 750 may be controlled in response to a signal received from throttle / accelerator 752.

[0143] Steering system 754 may include a steering wheel and / or other steering mechanisms (e.g., remote steering and / or local steering) for steering machine 700 (e.g., along a desired path or route) while propulsion system 750 is in operation (e.g., when the vehicle is moving). Steering system 754 may receive signals from steering actuator 756. In some embodiments, a steering wheel or other steering mechanism may be omitted, for example, for machine 700 capable of fully automated (e.g., level 5) functionality.

[0144] The brake sensor system 746 can be used to operate the vehicle brakes in response to signals received from the brake actuator 748 and / or the brake sensor.

[0145] Machine 700 may include one or more controllers 736, such as those described herein. Figure 7AThe controller 736 is described. It can be used for a variety of functions and can be coupled to any of the various other components and systems of the machine 700. For example, the controller 736 can be used to control the machine 700, artificial intelligence performed on the machine 700, infotainment of the machine 700, etc. For example, one controller 736 can be used for some or all of the functions, or different controllers 736 can be used for different functions, for example, to ensure availability and safety separation between various controllers for different tasks. For example, the controller 736 can use a system-calculated plan (e.g., the path or trajectory of vehicle 700A or AMR 700B, or the movement, component trajectory, motion position or displacement, etc. of joints or components (e.g., manipulators, end effectors, limbs, hands, fingers, legs, feet, etc.) of humanoid robot 700C to control the machine 700 in the environment. In some cases, the controller 736 may include a proportional-integral-derivative (PID) controller, a fuzzy logic controller, a neural controller (e.g., a controller embodied as one or more neural networks), a force control controller, a programmable logic controller (PLC), and / or other types of controllers. For example, in the humanoid robot 700C, the controller 736 can act as the brain, responsible for analyzing sensor data, making decisions, and sending commands to the actuators. The controller 736 may include a low-level controller that handles basic motor control, ensuring accurate and precise movement of the individual joints and actuators. The controller 736 may also include a high-level controller to coordinate multiple actuators and sensors, plan complex movements, and adapt to constantly changing environments.

[0146] In some embodiments, controller 736 may include an artificial intelligence controller that can use AI algorithms (e.g., DNN, MLM, etc.) to learn, make decisions, and autonomously perform tasks of machine 700. In some embodiments, controller 736 may use a fixed open-loop control algorithm and may not adjust its actions based on the environment. In other embodiments, closed-loop control may be used, which incorporates a feedback mechanism to monitor the robot's performance and make necessary adjustments. In examples, controller 736 may implement reactive control to directly respond to sensor inputs, enabling rapid reflexive actions and real-time changes. Furthermore, in some examples, deliberate control may be implemented, using internal models and planning algorithms to generate advanced actions, which may be suitable for complex tasks requiring reasoning, decision-making, and long-term planning.

[0147] The controller 736 may include one or more system-on-chip (SoC) 704 ( Figure 7C and Figure 7DThe controller 736 may provide signals (e.g., signals representing commands or messages) to one or more components and / or systems of the machine 700, including CPUs, GPUs, accelerators, etc. Although the controller 736 is listed separately from the SoC 704, this is not intended to be limiting, and in some embodiments, one or more components of the SoC 704 may perform the operations of the controller 736. For example, the controller may send signals to operate machine brakes via one or more brake actuators 748, to operate steering system 754 via one or more steering actuators 756, to operate propulsion system 750 via one or more throttle / accelerators 752, and so on. The controller 736 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous or semi-autonomous navigation and movement and / or assist human operators using the machine 700. Controller 736 may include a first controller 736 for autonomous control and navigation functions, a second controller 736 for functional safety functions, a third controller 736 for artificial intelligence functions (e.g., computer vision), a fourth controller 736 for infotainment functions, a fifth controller 736 for redundancy in emergency situations, and / or other controllers. For example, hardware for safety monitoring and other safety functions (e.g., functional safety islands) may be discrete or partitioned (physically or by processing separation) relative to hardware for processing sensor data to make perception and vehicle control decisions. Similarly, hardware for controlling in-vehicle infotainment and / or in-cabin monitoring (e.g., controllers, SOCs, etc.) may be separate or independent from hardware for vehicle perception and control. In some examples, a single controller 736 may handle two or more of the above functions, two or more controllers 736 may handle a single function, and / or any combination thereof.

[0148] The controller 736 may provide signals for controlling one or more components and / or systems of the machine 700 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data can be received from, for example, but not limited to, Global Navigation Satellite System (“GNSS”) sensor 758 (e.g., Global Positioning System sensor), RADAR sensor 760, ultrasonic sensor 762, LiDAR sensor 764, Inertial Measurement Unit (IMU) sensor 766 (e.g., accelerometer, gyroscope, magnetic compass, magnetometer, etc.), microphone 796, camera 768 (e.g., stereo camera 768A, wide-angle camera 768B (e.g., fisheye camera), infrared camera 768C, surround camera 768D (e.g., 360-degree camera), long-range and / or medium-range camera 768E and / or other types of cameras), speed sensor 744 (e.g., for measuring the speed of machine 700), vibration sensor 742, steering sensor 740, braking sensor (e.g., as part of braking sensor system 746), actuators and / or other types of sensors. In some embodiments, sensor 102 may include at least one of sensors from machine 700 of the sensor types described above, and sensor data from these sensors from machine 700 may be provided to 3D object detection system 100, for example as input to model 106, to generate feature representation 108.

[0149] One or more controllers 736 may receive input (e.g., represented by input data) from the instrument panel 732 of the machine 700 and provide output (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 734 (e.g., screen, head-up display, mirror display, facial display, robot display, etc.), sound alarms, loudspeakers, speakers, and / or via other components of the machine 700. Outputs may include parameters such as machine speed, rate, time, and... Figure 7C The system may display map data corresponding to map 722 (e.g., from a navigation map, a standard definition (SD) map, a high definition (“HD”) map, etc.), location data (e.g., the location of machine 700, such as its location on map 722), direction, the location of other vehicles (e.g., occupancy map, elevation map, bird's-eye view (BEV) image, grid, etc.), information about objects perceived by the system and their states, system status information, and so on. For example, HMI display 734 may display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.), and / or information about driving maneuvers that the vehicle has performed, is performing, or will perform (e.g., changing lanes now, exiting from exit 34B in two miles, etc.).

[0150] Machine 700 may include one or more System-on-a-Chip (SoC) 704 (in Figure 7D (described in more detail below). SoC 704 may include CPU 706, GPU 708, processor 710, cache 712, accelerator 714, data storage 716, and / or other components and features. SoC 704 can be used to process and provide data for various operations, such as navigation, planning, reasoning, inference, perception, control, and / or actuation operations of machine 700 on various platforms and systems. For example, SoC 704 can process real-time perception data (e.g., from cameras, LiDAR, RADAR, ultrasound, etc.) and map data corresponding to one or more maps 722 (e.g., HD maps, SD maps, navigation maps, occupancy maps, etc.) to perform or assist in performing various operations of machine 700. When using maps and / or AI, maps and / or AI (e.g., model parameter updates, fine-tuning, etc.) are transmitted via network interface 724 from one or more servers (e.g., [unclear]). Figure 7E The server 778 (e.g., one or more servers in a cloud-based data center) is refreshed and / or updated.

[0151] Despite Figures 7A to 7E The SoC 704 is illustrated; however, additional or alternative components and / or architectures, such as multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiPs), field-programmable gate arrays (FPGAs), heterogeneous integration (HI), and single-board computers (SBCs), may be used without departing from the scope of this disclosure. For example, depending on the type of machine 700, the purpose of machine 700, the model of machine 700, and the capabilities required by machine 700, one or more SoC 704s and / or alternative architectures and / or components may be used to meet a particular implementation.

[0152] Machine 700 may include CPU 718 (e.g., a discrete CPU or dCPU) which may be coupled to SoC 704 via a high-speed interconnect (e.g., PCIe). CPU 718 may include, for example, an x86 processor. CPU 718 may be used to perform any of a variety of functions, including arbitrating potentially inconsistent results between ADAS sensors and SoC 704, and / or monitoring the status and health of controller 736 and / or infotainment SoC 730.

[0153] Machine 700 may include GPU 720 (e.g., a discrete GPU or dGPU) which may be coupled to SoC 704 via a high-speed interconnect (e.g., NVIDIA's NVLink). GPU 720 may provide additional artificial intelligence capabilities, such as by executing redundant and / or different neural networks, and may be used to train and / or update neural networks based on inputs from sensors (e.g., sensor data) from Machine 700.

[0154] The machine 700 may also include a network interface 724, which may include one or more wireless antennas 726 and / or modems (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 724 can be used to establish wireless connectivity with the cloud (e.g., with server 778 and / or other network devices) via the Internet, and with other vehicles and / or computing devices (e.g., passenger client devices). For communication with other vehicles, direct links and / or indirect links (e.g., across networks and via the Internet) can be established between the two vehicles. A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide the machine 700 with information about vehicles near the machine 700 (e.g., vehicles in front, to the side, and / or behind the machine 700). This functionality can be part of the machine 700's cooperative adaptive cruise control function.

[0155] Network interface 724 may include a SoC that provides modulation and demodulation functions and enables controller 736 to communicate over a wireless network. Network interface 724 may include a radio frequency (RF) front-end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. Frequency conversion can be performed by well-known processes and / or using superheterodyne processes. In some examples, the RF front-end functionality may be provided by a separate chip. For example, network interface 724 may be able to communicate via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multicarrier (“CDMA2000”), 5G, 6G, and / or other cellular and / or wireless communication standards. The wireless antenna 726 can also enable communication between objects in the environment (such as vehicles, mobile devices, etc.) using local area networks (such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc.) and / or low power wide area networks (“LPWAN”) (such as LoRaWAN, SigFox, etc.).

[0156] Machine 700 may also include data memory 728, which may include off-chip (e.g., outside of SoC 704) storage. Data memory 728 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, Flash, hard disk and / or other components and / or devices capable of storing at least one bit of data.

[0157] The machine 700 may also include a GNSS sensor 758. The GNSS sensor 758 (e.g., a GPS, an auxiliary GPS sensor, a differential GPS (DGPS) sensor, etc.) is used to assist in mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 758 can be used, such as, but not limited to, GPS sensors with a USB connector having an Ethernet-to-serial (RS-232) bridge.

[0158] Machine 700 may also include an IMU sensor 766. In some examples, the IMU sensor 766 may be located at the center of the rear axis of machine 700. The IMU sensor 766 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 766 may include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 766 may include an accelerometer, a gyroscope, and a magnetometer.

[0159] In some embodiments, the IMU sensor 766 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 766 can enable the machine 700 to estimate its heading by directly observing and correlating velocity changes from GPS to the IMU sensor 766, without requiring input from a magnetic sensor. In some examples, the IMU sensor 766 and the GNSS sensor 758 can be combined in a single integrated unit.

[0160] The vehicle may include one or more microphones 796 placed inside and / or around the machine 700. The microphones 796 can be used for emergency vehicle detection and identification, etc.

[0161] Machine 700 may also include vibration sensors 742. Vibration sensors 742 can measure vibrations of machine parts, such as the arm or leg of humanoid robot 700C, or the axle of vehicle 700A or AMR 700B. For example, changes in vibration may indicate changes in roads, walking, or traversable surfaces. In another example, when two or more vibration sensors 742 are used, differences between vibrations can be used to determine friction or slippage on surfaces (e.g., when the vibration difference is between an electrically driven shaft and a freely rotating axle).

[0162] Machine 700 may include ADAS system 738, for example, when machine 700 is vehicle 700A. In some examples, ADAS system 738 may include a dedicated SoC. ADAS system 738 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision or collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), blind spot monitoring (BSM), rear cross traffic warning (RCTW), pedestrian detection, driver monitoring, collision warning system (CWS), traffic sign recognition, speed limit detection, automatic parking, lane centering (LC), high beam safety system and / or other features and functions.

[0163] Machine 700 may also include an infotainment SoC 730 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include one or more discrete components, such as multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiP), heterogeneous integration (HI), single-board computers (SBCs), etc. The infotainment SoC 730 may include a combination of hardware and software that can be used to provide machine 700 with audio (e.g., music, personal digital assistant, navigation commands, news, radio, etc.), video (e.g., television, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., wireless, Wi-Fi, etc.) and / or information services (e.g., navigation systems, rear parking assist, radio data systems, vehicle-related information (e.g., fuel level, total driving distance, brake fluid level, fuel level, door opening / closing, air filter information, etc.). For example, an infotainment SoC 730 may be a radio, disk player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, Wi-Fi, steering wheel audio controls, hands-free voice control, head-up display (HUD), HMI display 734, telematics device, control panel (e.g., for controlling various components, features and / or systems and / or interacting with various components, features and / or systems) and / or other components. The infotainment SoC 730 may also be used to provide information to vehicle users (e.g., visual and / or auditory), such as information from ADAS system 738, autonomous driving information (e.g., planned vehicle maneuvers, trajectory), surrounding environment information (e.g., intersection information, vehicle information, road information, etc.) and / or other information.

[0164] The infotainment SoC 730 may include GPU functionality. The infotainment SoC 730 can communicate with other devices, systems, and / or components of the machine 700 via bus 702 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 730 may be coupled to a monitoring MCU, allowing the GPU of the infotainment system to perform some autonomous driving functions in the event of a failure of the main controller 736 (e.g., the main computer and / or backup computer of the machine 700). In such an example, the infotainment SoC 730 may place the machine 700 into a driver-safe stop mode, as described herein.

[0165] In some embodiments, the infotainment system may provide a digital or virtual assistant, which may be voice-only or may have visual components (e.g., in the form of a digital human or digital avatar). The assistant may provide basic functions such as sending text messages, adjusting vehicle settings, controlling music or video, navigation, etc., and / or more advanced functions, such as those supported by one or more language models (e.g., Large Language Model (LLM), Visual Language Model (VLM), Multimodal Language Model (MMLM), etc.). For example, the driver and / or occupants may interact with the assistant in a manner similar to how a user interacts with a language model, such as asking general questions, specific questions, requesting restaurants, gas stations, and / or other recommendations and / or locations, inquiring about vehicle functions or troubleshooting (e.g., asking for tire pressure information, oil change information, battery swap information, etc.). Therefore, the machine 700 (whether it is a vehicle 700A, AMR 700B, humanoid robot 700C, or other type of machine) may include locally stored language models and / or communicate with remotely hosted language models (e.g., via one or more APIs) to provide users of the machine 700 with more detailed and in-depth communication capabilities.

[0166] In some examples, the infotainment SoC 730, SoC 704, and / or another SoC or computing / processing system can perform in-cabin driver and / or occupant monitoring. For example, the computing system can perform facial recognition, and vehicle owner recognition can use data from cameras and / or other sensors to identify the presence of an authorized driver and / or owner of the machine 700. An always-on sensor processing engine can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in a security mode, to disable the vehicle when the owner leaves. In this way, SoC 704 can provide security against theft and / or carjacking.

[0167] In some embodiments, one or more neural networks running on another or a dedicated SoC (e.g., an in-vehicle infotainment or in-vehicle monitoring SoC) can be used to monitor in-cabin monitoring camera sensors. This other or dedicated SoC is configured to recognize in-cabin events and respond accordingly. The in-cabin system can activate cellular services and make phone calls via lip reading, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. The in-cabin system may also include one or more in-cabin AI agents or assistants that can interact with one or more LLMs, VLMs, MMLMs, etc., in the cloud using one or more APIs or plugins. For example, the in-cabin AI agents or assistants can provide directions, vehicle or machine feedback information, answer general questions, handle music / video and / or other requests, activate windows, doors, and / or other vehicle components, etc. Therefore, one or more dedicated SoCs and / or processor groups can be used to perform in-cabin infotainment and / or in-cabin monitoring (e.g., as an occupant monitoring system (OMS)) of the machine 700.

[0168] The device 700 may also include an instrument cluster 732 (e.g., a digital instrument cluster, electronic instrument cluster, digital dashboard, etc.). The instrument cluster 732 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 732 may include a set of instruments such as a speedometer, fuel level, fuel pressure, tachometer, odometer, turn signal indicator, gearshift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 730 and the instrument cluster 732. In other words, the instrument cluster 732 may be included as part of the infotainment SoC 730, and vice versa.

[0169] Figure 7D A computing system according to at least some embodiments of this disclosure (about Figure 7C A block diagram of an example architecture (a subset of the systems described). Although illustrated as SoC 704, this is not intended to be limiting, and the computing system may additionally or alternatively include multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiP), heterogeneous integration (HI), single-board computers (SBCs), and / or other components and / or architectures without departing from the scope of this disclosure.

[0170] The SoC 704 can be an end-to-end platform with a flexible architecture spanning automation levels 2-5, or it can be specifically designed for a particular automation level (e.g., a first SoC 704 for levels 2 to 2++, a second SoC 704 for level 3, a third SoC 704 for level 4, and so on), providing a comprehensive functional safety architecture that leverages and effectively utilizes computer vision, neural network inference, robot planning, control and navigation, ADAS technologies, and more, with diversity and redundancy to provide a flexible and reliable platform for driving or robot control software stacks, as well as deep learning tools. The SoC 704 can be faster, more reliable, and even more energy-efficient and space-saving than conventional systems. For example, when the accelerator 714 is used in conjunction with the CPU 706, GPU 708, and data repository 716, it can provide a fast and efficient platform for Level 2-5 autonomous vehicles, as well as for the safety planning, navigation, and control of AMR 700B, humanoid robot 700C, and / or other robot or machine types.

[0171] In some embodiments, for example, SoC 704 includes: a GPU 708 having 2000 or more cores (e.g., 2048 cores), 60 or more tensor cores (e.g., 64 tensor cores), and a maximum GPU frequency exceeding 1 GHz (e.g., 1.3 GHz); a CPU 706 including 10 or more cores (e.g., 12 cores), having 64-bit, 3 MB L2 and 6 MB L3 caches, and a maximum frequency of 2 GHz or more (e.g., 2.2 GHz); one or more deep learning accelerators (DLAs); a deep learning accelerator cluster (XNN); a neural network accelerator (NNA) or neural processing unit (NPU) 709 (e.g., two DLAs / XNNs / NNAs / NPUs 709); and a vision accelerator (e.g., a programmable vision accelerator (PVA)) 707. A single SoC 704 may be capable of achieving AI performance of 275 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGXOrin. The 64GB SoC meets these standards and achieves this level of performance.

[0172] Similarly, in an embodiment, SoC 704 includes: a GPU 708 having 1700 or more cores (e.g., 1792 cores), 50 or more tensor cores (e.g., 56 tensor cores), and a maximum GPU frequency exceeding 900MHz (e.g., 930MHz); a CPU 706 including 8 or more cores (e.g., 8 cores), having 64-bit, 2MB L2 and 4MB L3 cache memory, and a maximum frequency of 2GHz or more (e.g., 2.2GHz); one or more deep learning accelerators (DLAs); a deep learning accelerator cluster (XNN); a neural network accelerator (NNA) or neural processing unit (NPU) 709 (e.g., 2 DLAs / XNNs / NNAs / NPUs 709); and a vision accelerator (e.g., a programmable vision accelerator (PVA) 707), and a single SoC 704 may be capable of achieving AI performance of 200 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGXOrin 32GB SoC meets these standards and achieves such performance.

[0173] In some embodiments, for example, SoC 704 includes: a GPU 708 having 1,000 or more cores (e.g., 1,024 cores), 28 or more tensor cores (e.g., 32 tensor cores) and a maximum GPU frequency exceeding 900 MHz (e.g., 1,173 MHz); a CPU 706 including 8 or more cores (e.g., 8 cores), having 64-bit, 2 MB L2 and 4 MB L3 cache memory, and a maximum frequency of 2 GHz or more (e.g., 2 GHz); one or more deep learning accelerators (DLAs); a deep learning accelerator cluster (XNN); a neural network accelerator (NNA) or neural processing unit (NPU) 709 (e.g., one DLA / XNN / NNA / NPU 709); and a vision accelerator (e.g., a programmable vision accelerator (PVA) 707), and a single SoC 704 may be capable of achieving AI performance of 157 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGX Orin NX 16GB SoC meets these standards and achieves such performance.

[0174] In various embodiments, for example, the SoC 704 includes: a GPU 708 having 1000 or more cores (e.g., 1024 cores), 28 or more tensor cores (e.g., 32 tensor cores), and a maximum GPU frequency exceeding 900MHz (e.g., 1020MHz); and a CPU 706 including 6 or more cores (e.g., 6 cores), having 64-bit, 1.5MB L2 and 4MB L3 cache memory, and a maximum frequency of 1.5GHz or higher (e.g., 1.7GHz). A single SoC 704 can achieve AI performance of 67 trillion operations per second (TOPS). For example, NVIDIA's Jetson Orin Nano 8GB SoC meets these standards and achieves such performance.

[0175] SoC 704 may include one or more CPUs 706. In embodiments, CPU 706 may include CPU clusters or CPU complexes (also referred to herein as “CCPLEX”). CPU 706 may include multiple cores and / or (e.g., L2, L3) caches. For example, in some embodiments, CPU 706 may include twelve cores in a coherent multiprocessor configuration. In some embodiments, CPU 706 may include four dual-core clusters, each with a dedicated L2 cache (e.g., 3MB L2 cache). CPU 706 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, thereby allowing any combination of clusters of CPU 706 to be active at any given time.

[0176] The SoC 704 may include any type and number of GPUs 708. For example, an integrated GPU (also referred to herein as an "iGPU") may be used in some embodiments. The GPU 708 may be programmable and can be used efficiently for parallel workloads. In some examples, the GPU 708 may use an enhanced tensor instruction set. The GPU 708 may include one or more streaming microprocessors, wherein each streaming microprocessor may include a cache (e.g., an L1 cache with at least 96KB of storage), and two or more streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage). In some embodiments, the GPU 708 may include at least eight streaming microprocessors. The GPU 708 may use a computing application programming interface (API). Furthermore, the GPU 708 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0177] The GPU 708 can be power-optimized for optimal performance in automotive, robotics, and / or other embedded use cases. For example, the GPU 708 can be fabricated on FinFETs. However, this is not intended to be limiting, and the GPU 708 can be fabricated using other semiconductor fabrication methods or processes. Each streaming microprocessor can include multiple mixed-precision processing cores divided into multiple blocks. For example, but not limited to, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSOR COREs for deep learning matrix algorithms, one (e.g., L0) instruction cache, one warp scheduler, one dispatch unit, and / or one (e.g., 64KB) register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to efficiently perform workloads involving mixed-computation and addressing computations. Streaming microprocessors can include independent thread scheduling capabilities to enable finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors can also include combined L1 data caches and shared memory units to improve performance while simplifying programming.

[0178] The GPU 708 may include a high-bandwidth memory (HBM) and / or (e.g., 16GB) HBM2 memory subsystem to provide peak memory bandwidth of approximately 900GB / s in some examples. In some examples, synchronous graphics random access memory (SGRAM), such as graphics double data rate type 5 synchronous random access memory (GDDR5), may be used in addition to or as an alternative to HBM memory.

[0179] The GPU 708 may include unified memory technology, including access counters, to allow more accurate migration of memory pages to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, Address Translation Service (ATS) support may be used to allow the GPU 708 to directly access the CPU 706 page tables. In such examples, when a miss occurs in the GPU 708 Memory Management Unit (MMU), an address translation request can be transferred to the CPU 706. In response, the CPU 706 can look up the virtual-to-physical mapping of the address in its page tables and transfer the translation back to the GPU 708. Therefore, unified memory technology allows the memory of the CPU 706 and GPU 708 to use a single, unified virtual address space, simplifying GPU 708 programming and porting applications to the GPU 708.

[0180] SoC 704 may include any number of caches 712, including the caches described herein. For example, cache 712 may include L0 cache, L1 cache, L2 cache, L3 cache (e.g., available for CPU 706 and GPU 708 (e.g., connected to CPU 706 and GPU 708)), etc. Cache 712 may include write-back caches that can track the state of the lines, for example, by using one or more cache coherence protocols (e.g., MEI, MESI, MSI, etc.). According to embodiments, the cache (e.g., L3) may include 4MB or more, although smaller or larger cache sizes may be used.

[0181] SoC 704 may include one or more arithmetic logic units (ALUs) 765, which can be used to perform processing associated with various tasks or operations of machine 700, such as computer vision, machine learning or deep learning processing, world model management, etc. Furthermore, SoC 704 may include a floating-point unit (FPU) 767 or other mathematical coprocessors or digital coprocessors for performing mathematical operations within the system. For example, SoC 704 may include one or more FPUs 767, which are integrated as execution units within CPU 706 and / or GPU 708.

[0182] SoC 704 may include one or more accelerators 714 (e.g., hardware accelerators, software accelerators, or combinations thereof). For example, SoC 704 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. Large on-chip memory 715 (e.g., 4MB SRAM, 32GB and / or 64GB 256-bit LPDDR5 (204.8GB / s), 8GB and / or 16GB 128-bit LPDDR5 (102.4GB / s), and / or other memory types and sizes) enables the hardware acceleration cluster to accelerate neural network processing, converter processing, optical flow processing, vision processing, and / or other computations or processing. The hardware acceleration cluster can be used to supplement GPU 708 and offload some tasks from GPU 708 (e.g., freeing up more cycles of GPU 708 to perform other tasks). As an example, the accelerator 714 can be used for target workloads (e.g., perceptual, convolutional neural network (CNN), deep neural network (DNN), language model (LLM, VLM, MMLM, VLA, etc.), transformer model, diffusion model, encoder-only model, encoder-decoder model, etc., which are stable enough to be suitable for acceleration.

[0183] Accelerator 714 (e.g., a hardware acceleration cluster) may include a deep learning accelerator (DLA) 709 (also referred to herein as a “deep learning accelerator cluster (XNN) 709”, a “neural network accelerator (NNA) 709”, or a “neural processing unit (NPU) 709”). DLA 709 may include one or more tensor processing units (TPUs) 741, which may be configured to provide additional, for example, trillions of operations per second for deep learning applications and inference. TPU 741 may be an accelerator configured to perform image processing functions (e.g., for CNNs, RCNNs, DNNs, etc.) and optimized for them. DLA 709 may be further optimized for a specific set of neural network types and floating-point operations and inference. The DLA is designed to provide higher performance per millimeter than general-purpose GPUs and significantly outperforms CPUs. TPU 741 may perform several functions, including single-instance convolution functions, support for INT8, INT16, and FP16 data types for features and weights, and post-processor functions. Although TPU 741 is described as being included as part of DLA 709, this is not intended to be limiting, and TPU 741 may be included in additional or alternative accelerator 714 and / or other components, and / or may be included as a discrete processing component.

[0184] The DLA 709 can quickly and efficiently execute neural networks on processed or unprocessed data to achieve a variety of functions, including but not limited to: object and feature recognition and detection using data from one or more sensor modalities (e.g., vehicles, pedestrians, other robots, lane lines, road boundary lines, debris, potholes, boxes, warehouse items, etc.); distance estimation using data from one or more sensor modalities; emergency vehicle detection and identification using data from microphones and / or vision-based sensors; facial recognition; pick-up and place operations; maneuvering operations; occupant monitoring; vehicle owner identification; and / or other in-cabin operations using data from in-cabin cameras and / or other sensor types; and / or safety and / or safety-related events, to name just a few.

[0185] The DLA 709 can perform any function of the GPU 708, and by using inference accelerators, for example, designers can anchor either the DLA 709 or the GPU 708 for any function. For instance, designers can centralize the processing of DNNs and floating-point operations on the DLA 709, leaving other functions to the GPU 708 and / or other accelerators 714. The DLA 709 can be used to run any type of network to enhance control and security, including neural networks that, for example, output a confidence metric for each object detection.

[0186] Accelerator 714 (e.g., a hardware acceleration cluster) may include a programmable vision accelerator (PVA) 707, which may alternatively be referred to herein as a computer vision accelerator or generally as a vision accelerator. The PVA 707 may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), semi-autonomous driving, autonomous driving, robotics applications, safety and supervision applications, augmented reality (AR), virtual reality (VR), and / or mixed reality (MR) applications. The PVA 707 provides a balance between performance and flexibility. For example, each PVA 707 may include (e.g., but not limited to) any number of Reduced Instruction Set Computer (RISC) cores, Direct Memory Access (DMA) systems, Pixel Processing Engines (PPEs), Vector Processors or Vector Processing Units (VPUs), and / or other components. The PVA engine may include an Advanced Very Long Instruction Word (VLIW) or Single Instruction Multiple Data (SIMD) digital signal processor. The PVA 707 may be optimized for image processing and computer vision algorithm acceleration tasks. For example, the PVA707 offers excellent performance with extremely low power consumption and can be used asynchronously and concurrently as part of a heterogeneous computing pipeline with CPU 706, GPU 708 and / or other accelerators in systems (e.g., vehicles, robots, etc.).

[0187] The PVA 707 may include one or more (e.g., two) Vector Processing Subsystems (VPSs), each of which may include one or more Vector Processing Unit (VPU) cores, one or more Decoupled Lookup Units (DLUTs), one or more shared or vector memory (VMEMs), and one or more instruction caches (I-caches). The VPU core may be the main processing unit and may include a vector SIMD VLIW DSP 743 optimized for computer vision. The VPU core can fetch instructions via the I-cache and access data via the VMEM. The DLUT may include dedicated hardware components that enhance the efficiency of parallel lookup operations. For example, the DLUT allows parallel lookups using a single copy of the lookup table by performing these lookups in a decoupled pipeline independent of the main processor pipeline. By doing so, the DLUT can minimize or reduce memory usage and increase throughput while avoiding memory library conflicts associated with data, ultimately leading to improved overall system performance. The VPU VMEM can provide local data storage for the VPU, allowing for the efficient implementation of various image processing and computer vision algorithms. The VPU VMEM can support access from external VPS hosts, such as Direct Memory Access (DMA) and CPU 706 (e.g., an ARM Cortex-R5 processor), thereby facilitating data exchange with the CPU 706 and other system-level components. The VPU I-cache can provide instruction data to the VPU upon request, request missing instruction data from system memory, and / or maintain temporary instruction storage for the VPU. For each VPU task, the CPU 706 can configure the DMA system, optionally prefetching the VPU program into the VPU I-cache, and / or initiating each VPU-DMA pair to process the task. The PVA 707 may also include L2 SRAM memory shared between one or more (e.g., two) sets of VPS and DMA. In some embodiments, one or more (e.g., two) DMA devices are used to move data between external memory, PVA L2 memory, VMEM (e.g., one per VPS), CPU tightly coupled memory (TCM), DMA descriptor memory, and / or PVA-level configuration registers. In lightly loaded systems, two parallel DMA accesses to DRAM can achieve read / write bandwidth of up to 15 GB / s, while in heavily loaded systems, this bandwidth can reach up to 10 GB / s. Regarding compute capacity, INT8 gigabyte multiply-accumulate operations per second (GMAC) can be 2048 or greater, excluding DLUT. FP32 GMACs can include 32 per PVA instance.

[0188] The RISC core can interact with image sensors (e.g., the image sensor of any camera described herein), image signal processors, etc. Each RISC core may include any amount of memory. The RISC core can use any of a variety of protocols, depending on the implementation. In some examples, the RISC core can execute a real-time operating system (RTOS). The RISC core can be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, the RISC core may include an instruction cache and / or tightly coupled RAM.

[0189] The DMA system enables PVA 707 components to access system memory independently of the CPU 706. DMA can support any number of functions optimized for the PVA 707, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which may include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0190] A vector processor, or VPU, can be a programmable processor designed to efficiently and flexibly execute computer vision algorithms and provide signal processing capabilities. In some examples, the PVA 707 may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA 707 and may include one or more vector processing units (VPUs), one or more pixel processing engines (PPEs) (which may include a 2D layout of interconnected (e.g., for north, south, east, and west communication) processing elements), one or more instruction caches, and / or one or more shared or vector memories (e.g., VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can improve throughput and speed.

[0191] In some embodiments, each vector processor may include an instruction cache and may be coupled to dedicated memory. Therefore, in some examples, each vector processor may be configured to execute independently of other vector processors. In other examples, the vector processors included in a particular PVA 707 may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA 707 may execute the same computer vision algorithm, but for different regions of an image. In other examples, the vector processors included in a particular PVA 707 may execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on consecutive images or portions of an image. Among other things, the hardware acceleration cluster may include any number of PVA 707s, and each PVA may include any number of vector processors. Furthermore, the PVA 707 may include additional error correction code (ECC) memory to enhance overall system security.

[0192] Accelerators 714 (e.g., hardware accelerator clusters) have broad applications in autonomous and semi-autonomous machine control. PVA 707 can be a programmable vision accelerator used in critical processing stages in perception, robot understanding and reasoning, ADAS, semi-autonomous and autonomous vehicles, etc. The capabilities of PVA 707 are well-suited for algorithmic domains requiring predictable processing, featuring low power consumption and low latency. In other words, PVA 707 performs well in semi-intensive or intensive rule computation, even on small datasets requiring predictable runtime, low latency, and low power consumption. Therefore, in the context of autonomous vehicles and robotic platforms, PVA 707 is designed to run classic computer vision algorithms, as they are highly efficient in object detection and integer mathematical operations.

[0193] For example, according to one embodiment of this technology, the PVA 707 is used to perform computer stereo vision. In some examples, algorithms based on semi-global matching can be used, although this is not intended to be limiting. Many Level 3-5 autonomous driving applications require on-the-fly motion estimation / stereo matching (e.g., moving structures, pedestrian recognition, lane detection, etc.). The PVA 707 can perform computer stereo vision functions on input from two monocular cameras.

[0194] In some examples, the PVA 707 can be used to perform dense optical flow, providing processed RADAR data based on the raw RADAR data (e.g., using 4D Fast Fourier Transform). In other examples, the PVA 707 is used for time-of-flight depth processing, for example, providing processed time-of-flight data by processing the raw time-of-flight data.

[0195] While the VPU, DMA, RISC Core, VMEM, and decoupled coprocessors (e.g., DLUT) are described as being included within the PVA707, this is not intended to be limiting. In some embodiments, these components may be included in alternative or additional processing components and / or accelerators 714, and / or may be included as discrete components of the SoC 704 and / or other computing system architectures.

[0196] In some examples, SoC 704 may include a real-time ray tracing hardware accelerator (RTA) 751, which can be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) to generate real-time or near-real-time visualization simulations, for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for simulation of SONAR, RADAR, LiDAR, camera and / or other sensor modalities in a simulation, for general wave propagation simulation, for comparison with LiDAR data for localization, for generating realistic training data for training neural networks, and / or other functions and uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations. For example, machine 700 (or another machine or device) may perform simulations in a simulation environment, and one or more light transport simulation algorithms (e.g., ray tracing, path tracing, etc.) may be used to generate the simulation environment. Therefore, the ray tracing accelerator 751 and / or a ray tracing-optimized GPU 706 (e.g., NVIDIA's RTX GPU) may be used to accelerate these ray tracing algorithms.

[0197] Accelerator 714 (e.g., in a hardware acceleration cluster) may include one or more optical flow accelerators (OFAs) 711. For example, OFA 711 can be used to calculate optical flow and stereo disparity between sensor data frames (e.g., images). Optical flow can be accelerated on OFA 711 for purposes such as object detection and tracking, and / or for stereo depth estimation, where stereo disparity is calculated between stereo image frames (e.g., two or more frames captured using two or more image sensors with at least partially overlapping fields of view).

[0198] SoC 704 may include one or more Camera Serial Interfaces (CSI) 723s. ​​For example, CSI 723 may include a Mobile Industry Processor Interface (MIPI) Camera Serial Interface (CSI) for receiving video and input from cameras, a high-speed interface, and / or a video input block that can be used for camera and associated pixel input functions. SoC 704 may also include a software-controllable input / output controller and may be used to receive I / O signals not assigned to a specific role. For example, CSI 723 may include MIPI CSI-2 connectors, such as a 16-channel MIPI CSI-2 connector, D-PHY 2.1 (up to 40Gbps), and C-PHY 2.0 (up to 164Gbps) to support 16 virtual channels and 6 or more cameras; an 8-channel MIPI CSI-2 connector, D-PHY 2.1 (up to 20Gbps) to support 8 virtual channels and 4 or more cameras; and / or 2x MIPI CSI-2, 22-pin camera connectors, depending on the embodiment and implementation.

[0199] Accelerator 714 (e.g., a hardware acceleration cluster) may include an on-chip computer vision network (CVNOC) 763 and SRAM for providing high-bandwidth, low-latency SRAM to accelerator 714. In some examples, on-chip memory may include at least 4 MB of SRAM, such as, but not limited to, eight field-configurable memory blocks accessible by PVA 707, OFA 711, DLA 709, and / or other accelerator 714. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory 715 may be used. PVA 707, OFA 711, DLA 709, and / or other accelerator 714 may access memory via a backbone providing high-speed memory access to accelerator 714. The backbone may include an on-chip computer vision network that interconnects accelerator 714 to memory (e.g., using APB).

[0200] The CGNOC 763 may include an interface that determines whether the accelerator 714 provides a ready and valid signal before transmitting any control signals / addresses / data. Such an interface can provide separate stages and separate channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, although other standards and protocols can be used.

[0201] SoC 704 may include a data repository 716 and / or memory 715. The data repository 716 may be on-chip memory 715 of the SoC 704, which may store neural networks and / or other algorithms to be executed on the CPU 706, GPU 708, and / or one or more accelerators 714. In some examples, the capacity of the data repository 716 may be large enough to store multiple neural network instances for redundancy and security. The data repository 716 may include, for example, L2 and / or L3 cache 712. The memory 715 may include SRAM, LPDDR5, and / or other memory types. For example, the memory 715 may include 4MB SRAM, 32GB and / or 64GB 256-bit LPDDR5 (204.8GB / s), 8GB and / or 16GB 128-bit LPDDR5 (102.4GB / s), and / or other memory types and sizes. References to the data repository 716 may include references to the memory associated with the PVA 707, OFA 711, DLA 709 and / or other accelerators 714, as described herein.

[0202] Data repository 716 may include various storage types, such as eMMC, NVMe, etc. For example, SoC 704 may include storage in the form of an embedded multimedia card (eMMC) (e.g., 64GB eMMC 5.1) and / or an SD card slot, with external NVMExpress (NVMe) capability, for example, via M.2Key M. For example, data repository 716 and / or other storage may be accessed via, for example, NVMe using PCI Express (PCIe), RDMA, TCP, and / or other protocols.

[0203] SoC 704 may include one or more processors 710 (e.g., embedded processors). Processor 710 may include a boot and power management processor (BPMP) 753, which may be a dedicated processor and subsystem for handling boot power and management functions, as well as associated safety implementations. BPMP 753 may be part of the SoC 704 boot sequence and may provide runtime power management services. BPMP 753 may provide clock and voltage programming, assistance with system low-power state transitions, management of SoC 704 thermal and temperature sensors, and / or management of SoC 704 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and SoC 704 may use the ring oscillator to detect the temperature of CPU 706, GPU 708, accelerator 714, and / or other components. If the temperature is determined to exceed a threshold, BPMP 753 may enter a temperature fault routine and place SoC 704 into a lower power state and / or place machine 700 into a driver safety stop mode (e.g., safely stop machine 700).

[0204] The processor 710 may also include a set of embedded processors that can be used as the Audio Processing Engine (APE) 755. The APE 755 can be an audio subsystem capable of providing full hardware support for multi-channel audio through multiple interfaces, as well as a wide and flexible audio I / O interface. In some examples, the APE 755 is a dedicated processor core with a digital signal processor and dedicated RAM.

[0205] The processor 710 may also include an Always-On Processor Engine (AOPE) 757, which provides the necessary hardware features to support low-power sensor management and wake-up use cases. AOPE 757 may include a processor core, tightly coupled RAM, peripheral support (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0206] Processor 710 may also include a security processor 713 (or “security island 713”), which may include a security cluster engine comprising a dedicated processor or processor subsystem for handling security management for automotive, robotic, and / or other applications. Security processor 713 and / or the security cluster engine may include two or more processor cores, tightly coupled RAM, support for peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In secure mode, the two or more cores may operate in lockstep mode and function as a single core, with comparison logic to detect any differences in their operation. In some embodiments, security processor 713 may include discrete processors such that failure of other system components may not affect the performance and availability of security processor 713.

[0207] The processor 710 may also include a real-time or near-real-time sensor engine (SE) 759, which may include a dedicated processor subsystem for handling real-time or near-real-time camera, LiDAR, RADAR and / or other sensor modal management.

[0208] The processor 710 may also include one or more image signal processors (ISPs) 727, which may include high dynamic range signal processors and / or hardware engines as part of one or more sensor processing pipelines.

[0209] The processor 710 may include a video image synthesizer (VIC) 761, which may be a processing block (e.g., implemented on a microprocessor) that implements the video post-processing functions required by the video playback application to produce the final image of the player window. The VIC 761 can perform lens distortion correction on the wide-angle camera 768B, the surround camera 768D, the cabin monitoring camera sensor, and / or other camera sensors with distorted fields of view.

[0210] VIC 761 may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, when motion occurs in the video, noise reduction appropriately weights spatial information, thereby reducing the weight of information provided by adjacent frames. When an image or part of an image does not contain motion, temporal noise reduction performed by the video image synthesizer can use information from previous images to reduce noise in the current image.

[0211] The VIC 761 can also be configured to perform stereoscopic correction on input stereoscopic camera frames. The video image compositor can also be used for user interface compositing when the operating system desktop is in use and the GPU 708 does not need to continuously render new surfaces. Even when the GPU 708 is powered on and actively performing 3D rendering, the video image compositor can be used to offload the GPU 708 to improve performance and responsiveness.

[0212] SoC 704 may also include various peripheral interfaces for input / output (I / O) 725, for example, to enable communication with peripheral devices, audio codecs, power management and / or other devices. SoC 704 can be used to process data from cameras (e.g., via a gigabit multimedia serial link and / or Ethernet connection), from sensors (e.g., LiDAR sensor 764, RADAR sensor 760, etc., connected via Ethernet), from bus 702 (e.g., speed of machine 700, steering wheel position, etc.), and from GNSS sensor 758 (e.g., connected via Ethernet or CAN bus). SoC 704 may also include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine and can be used to free CPU 706 from routine data management tasks. In some embodiments, the SoC 704I / O 725 may include headers (e.g., 40-pin headers or 40-pin extension headers) supporting Universal Asynchronous Receiver / Transmitter (UART), Serial Peripheral Interface (SPI), Inter-Integrated Sound (I2S), Inter-Integrated Sound (I2C), Controller Area Network (CAN), Pulse Width Modulation (PWM), Digital Microphone Interface (DMIC), Digital Speaker Station (DSPK), General Purpose I / O (GPIO), etc.; automation headers (e.g., 12-pin automation headers); audio panel headers (e.g., 10-pin audio panel headers); Joint Test Action Group (JTAG) headers (e.g., 10-pin JTAG headers); fan headers (e.g., 4-pin fan headers); RTC battery spare connectors (e.g., 2-pin battery spare connectors); a microSD slot; a DC power jack; power, force, resume, and reset buttons; and one or more display connectors (e.g., DisplayPort (DP), such as DP 1.4A(+MST), eDP 1.41, HDMI). 2.1 and / or 4K30 multi-model DP1.2 (+MST) connectors) and / or other I / O 725 elements, components or features.

[0213] The SoC 704 may include machine intranetting capabilities using, for example, Ethernet (e.g., automotive Ethernet), SERDES, Controller Area Network (CAN), FlexRay, Local Interconnect Network (LIN), Low Voltage Differential Signaling (LVDS), Media-Oriented System Transport (MOST), another network type, and / or combinations thereof. For example, the SoC 704 may include RJ45 connectors with up to 10GbE, 1GbE connectors, and / or other network connector types.

[0214] SoC 704 may include one or more digital signal processors (DSPs) 743. For example, DSP 743 may include a dedicated or specialized microprocessor chip optimized for digital signal processing, such as audio signal processing, telecommunications, digital image processing, RADAR, SONAR, LiDAR and / or other sensor processing, speech recognition and / or other applications.

[0215] SoC 704 may include one or more video encoders 719 and / or one or more video decoders 721. For example, video encoder 719 may include a hardware-based (e.g., as part of GPU 708) video encoder (e.g., supporting H.264, H.265, etc., and conforming to HEVC standards, such as NVIDIA's NVENC) that can process image input (e.g., as YUV, RGB, etc.) to generate a video bitstream. Video decoder 721 may include a video decoder engine that can provide fully accelerated hardware video decoding capabilities (e.g., supporting decoding of various bitstream formats such as AV1, H.264, H.265, VP8, VP9, ​​MPEG-1, MPEG-2, MPEG-4, VC-1, etc., and conforming to HEVC standards, such as NVIDIA's NVDEC). In some examples, video decoder 721 may be hardware-based (e.g., as part of GPU 708).

[0216] SoC 704 may include one or more General Purpose Computing Acceleration Clusters (GCACs) 729. For example, GCACs 729 may include various processor types that can be used to accelerate computing, such as one or more Vector Microcode Processors (VMPs) 733, one or more Multi-Threaded Processing Clusters (MPCs) 731, one or more Programmable Macroarrays (PMAs) 735, and / or one or more other processor types. For example, GCAC 729 may include a PMA 735, two VMPs 733, and two MPCs 731.

[0217] SoC 704 may include one or more vector microcode processors (VMPs) 733. In embodiments, VMP 733 may include wide vector (Very Long Instruction Word (VLIW) and Single Instruction Multiple Data (SIMD)) machines that perform various operations, such as short integral-type operations common in computer vision and deep learning algorithms.

[0218] SoC 704 may include one or more multi-threaded processing clusters (MPCs) 731. MPCs 731 may include processing clusters that are more general-purpose than GPUs and more efficient than CPUs in some embodiments. For example, an MPC 731 may include a multi-threaded processor that allows multiple threads to share resources and execute instructions concurrently.

[0219] The SoC 704 may include one or more programmable macro arrays (PMAs) 735. The PMA 735 may include a coarse-grained reconfigurable architecture (CGRA) dataflow machine, which has a unique architecture that delivers powerful performance on intensive computer vision and deep learning algorithms that may not be achievable in classic digital signal processing (DSP) architectures.

[0220] The SoC 704 may include one or more Display Processing Units (DPUs) 745 for performing hardware-accelerated image processing. For example, the DPU 745 may retrieve pixel data from memory 715 and send it to display peripherals via a standard interface. Therefore, the DPU 745 can handle display processing and rendering for displays within and / or on the machine.

[0221] The SoC 704 may include one or more Application Processing Units (APUs) 739. For example, an APU 739 may include a quad-core or dual-core processor with 48KB / 32KB L1 cache (with parity and ECC) and a 1MB L2 cache with ECC. The APU 739 may support NEON instructions and single-precision and double-precision floating-point operations.

[0222] The SoC 704 may include one or more Real-Time Processing Units (RTPUs) 769. The RTPU 769 may include a dual-core processor with 32KB / 32KB L1 cache and a 256KB TCM with ECC. The RTPU 769 can support single-precision and double-precision floating-point operations.

[0223] SoC 704 may include one or more built-in self-test (BIST) components 737. For example, BIST component 737 may include a memory BIST (MBIST) for testing the system's memory and / or a logic BIST (LBIST) for testing the system's logic. BIST component 737 may include embedded logic for directly testing the system's logic and / or memory.

[0224] SoC 704 may include one or more dynamically reconfigurable processors (DRPs) 771. For example, DRP 771 can be used to accelerate various computational operations. For instance, in one embodiment, DRP 771 may be combined with a MAC unit to function as an AI accelerator. In another embodiment, DRP 771 can execute an application while dynamically switching the circuit connection configuration of the arithmetic unit (e.g., ALU) on the chip each operating clock according to what needs to be processed. Because it uses only the necessary arithmetic circuitry, DRP 771 can consume less power than a CPU and achieve higher speeds. Furthermore, compared to a CPU, which suffers from performance degradation due to frequent accesses to external memory for cache misses and other reasons, DRP 771 can pre-build the necessary data paths in the hardware, thereby reducing performance degradation and operating speed variations (jitter) caused by memory accesses. DRP 771 may include a dynamic loading function that switches circuit connection information each time the algorithm changes, enabling processing with limited hardware resources, even in robotics / automotive applications that require processing multiple algorithms.

[0225] In some embodiments, accelerator 714 may include an OpenCV accelerator for accelerating OpenCV processing, OpenCV being an open-source industry-standard library for computer vision processing. In some embodiments, the combination of one or more DRP 771s deployed as AI accelerators with an OpenCV accelerator can enhance AI computation and image processing algorithms, enabling complex and computationally intensive operations such as visual simultaneous localization and mapping (SLAM).

[0226] Compared to conventional systems, the techniques described herein, by providing CPU complexes, GPU complexes, and hardware acceleration clusters, allow multiple neural networks to be executed simultaneously (e.g., at least partially in parallel) and / or sequentially, and the results combined to achieve Level 2–5 autonomous driving capabilities and / or autonomous robot motion, control, planning, and / or navigation operations. Furthermore, since the SoC 704 can include various computing engines (e.g., processor 710, CPU 706, GPU 708, accelerator 714, etc.), tasks can be distributed among the computing engines, and in some cases, common-cause failures are avoided due to the discrete footprint of the computing engines. Additionally, since the SoC 704 can include a dedicated safety processor 713 (or safety island 713), critical safety or redundant operations can be performed without common-cause failures of the SoC 714's main processing components or computing engines. Due to these features, the underlying system of the SoC 704 and / or machine 700 may be able to meet higher safety levels—such as the Automotive Safety Integrity Level (ASIL) D of the ISO 26262 standard.

[0227] Figure 7E According to some embodiments of this disclosure, cloud-based servers (e.g., servers such as those described herein in a data center) and Figure 7A The following is a system diagram illustrating communication between an example autonomous or semi-autonomous vehicle or machine 700. System 776 may include server 778, network 790, and machine 700. Server 778 may include multiple GPUs 784(A)-784(H) (collectively referred to herein as GPU 784), switches 782(A)-782(H) (e.g., PCIe 4.0 / 5.0 switches, M.2 slots, Thunderbolt, USB4, NVIDIA's NVLink, NVIDIA's NVSwitch, GPUDirect RDMA, GPUDirect Storage, etc.), CPUs 780(A)-780(B) (collectively referred to herein as CPU 780), accelerators, and / or other processor types. GPU 784, CPU 780, and PCIe switches may interconnect with high-speed interconnects, such as, but not limited to, NVIDIA-developed NVLink interface 788 and / or PCIe connection 786. In some examples, the GPU 784 is connected via NVLink and / or NVSwitch SoC, and the GPU 784 and PCIe switch 782 are connected via PCIe interconnect. While the figure shows eight GPUs 784, two CPUs 780, and two PCIe switches, this is not limiting. According to embodiments, each server 778 may include any number of GPUs 784, CPUs 780, and / or PCIe switches. For example, each server 778 may include eight, sixteen, thirty-two, and / or more GPUs 784.

[0228] Server 778 may receive sensor data from network 790 and machines 700 that indicates information about new or previously unexplored locations, and / or sensor data indicating changes to previously seen / stored locations (e.g., unexpected or changed road conditions, such as recently started road construction). Server 778 may transmit neural network 792, updated neural network 792, map information 794, etc., including information about traffic and road conditions, to machines 700 via network 790. Updates to map information 794 may include updates to HD maps 722, SD maps, navigation maps, etc., such as information about construction sites, potholes, detours, floods, and / or other obstacles. In some examples, neural network 792, updated neural network 792, map information 794, and / or other information may come from new training and / or experience, represented in data received from any number of machines 700 in the environment, and / or based on training performed in a data center (e.g., using server 778 and / or other servers).

[0229] Server 778 can be used to train a machine learning model (e.g., a neural network) based on training data. Training data can be generated by machine 700, and / or can be generated in a simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is unlabeled and / or unprocessed (e.g., the neural network does not require supervised learning). Training can be performed according to any one or more machine learning techniques, including but not limited to the following categories: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component analysis and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, it can be used by machine 700 (e.g., transmitted to machine 700 via network 790), and / or the machine learning model can be used by server 778 to remotely monitor and / or control machine 700.

[0230] In some examples, server 778 can receive data from machine 700 and apply the data to state-of-the-art real-time neural networks for real-time intelligent inference. Server 778 may include a deep learning supercomputer powered by GPU 784 and / or a dedicated AI computer, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, server 778 may include a deep learning infrastructure in a data center using only CPU power.

[0231] The deep learning infrastructure of server 778 may be capable of rapid, real-time inference and can use this capability to assess and verify the health of the processor, software, and / or related hardware in machine 700. For example, the deep learning infrastructure may receive periodic updates from machine 700, such as image sequences and / or objects located by machine 700 in the image sequence (e.g., through computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them with the objects identified by machine 700. If the results do not match and the infrastructure concludes that the AI ​​in machine 700 has malfunctioned, server 778 can send a signal to machine 700 instructing the fail-safe computer of machine 700 to take over control, notify the occupants, and perform safety maneuvers or operations, such as slowing down, returning control to the driver, stopping, and / or pulling over / closing the vehicle.

[0232] For inference, the server 778 can include a GPU 784 and one or more programmable inference accelerators (such as NVIDIA's TensorRT). The combination of GPU-driven servers and inference acceleration enables real-time response. In other examples, such as in scenarios where performance is less critical, servers powered by CPUs, FPGAs, and other processors can be used for inference. The computing ecosystem for generating, training, and deploying AI.

[0233] Figure 8 This is a system diagram illustrating a three-computer ecosystem 800 according to at least some embodiments of the present disclosure, including a first computing system 802 for generating or creating artificial intelligence (AI) (e.g., AI training and validation data), a second computing system 804 for training the AI, and a third computing system 806 for deploying AI at the edge (which may include or correspond to...). Figures 7A-7E (SoC 704). For example, to develop and deploy materialized or physical AI, a three-computer ecosystem 800 can be used, comprising three accelerated computer systems to handle physical AI training, simulation, and runtime (e.g., edge deployment). These systems can use scalable, physical-based simulations of the Machine 700 and its world to generate training data and train multimodal base models (and / or other model types). By doing so, simulations of the Machine 700 can be performed at scale, allowing skills (e.g., robotic skills) to be improved, tested, and optimized in virtual worlds that simulate the laws of physics (e.g., using NVIDIA's OMNIVERSE), thereby helping to reduce the cost of real-world data acquisition and ensuring that the Machine 700 can operate safely in a controlled environment.

[0234] Computing system 804 (e.g., NVIDIA's DGX platform) can be used to train and fine-tune powerful foundational and generative AI models. Models such as general foundational models (e.g., NVIDIA's Project GR00T) can be used to enable robots and other machines 700 to understand natural language and mimic actions by observing human movements. Computing system 804 may include a platform that integrates software, infrastructure, and expertise into a modern, unified AI development and training solution. Computing system 804 may include individual computing devices 810 (e.g., NVIDIA's DGX B200, H200, etc.) and / or any number of computing devices 810 (e.g., NVIDIA's DGX SuperPOD) within data center infrastructure 812.

[0235] For example, a standalone computing device 810 may include GPUs (e.g., 8 GPUs with a total GPU memory of 1,440 GB) and CPUs (e.g., 2 CPUs with a total of 112 cores, 2.1 GHz or 4 GHz (with enhancements)), providing up to 72 petaFLOPS of training capability and 144 petaFLOPS of inference capability. The computing device 810 may include memory (e.g., 4 TB of memory) and storage (e.g., 2 x 1.9 TB NVMe M.2 OS storage and 8 x 3.84 TB NVMe U.2 internal storage). The computing device 810 may include various networking and network management components, such as OSFP ports (e.g., 4 OSFP ports) for servicing single-port intelligent host channel adapters (e.g., 8 single-port ConnextX-7 Virtual Protocol Interconnect (VPI)) providing up to 400 GB / s of Infiniband / Ethernet. The computing device 810 may also include, for example, a dual-port quad small pluggable (QSFFP) data processing unit (DPU) (e.g., two dual-port QSFP112 DPUs – such as NVIDIA’s BlueField-3 DPU) providing up to 400Gb / s of InfiniBand / Ethernet. The computing device 810 may include an onboard network interface card (NIC) (e.g., a 10Gb / s onboard NIC with RJ45), a dual-port Ethernet NIC (e.g., a 100GB / s dual-port Ethernet NIC), and / or a host board management controller (MBC) (e.g., with RJ45). In some embodiments, the NIC for the computing device 810 may include a SuperNIC (e.g., NVIDIA’s ConnectX-8 SuperNIC) to provide up to 800Gb / s of data throughput for in-network computing acceleration engines, thereby providing the performance and robust feature set required to support trillion-parameter-scale AI factories and scientific computing workloads. In other embodiments, computing device 810 may include a smart host channel adapter (HCA) (e.g., NVIDIA's ConnectX-7) to provide ultra-low latency, 400Gb / s throughput for computing acceleration engines within the network.

[0236] Data center infrastructure 812 may include: any number of computing devices 810, and an operating system (OS) (e.g., a DGX OS extension of a Linux distribution) for maximizing system uptime, security, and reliability; network / storage acceleration libraries and management for accelerating end-to-end infrastructure performance; cluster management for scaling and managing a single node (e.g., a computing device 810) to thousands of nodes; job scheduling and orchestration for ensuring the smooth execution of each developer's jobs; AI workflow management and machine learning operations (MLOps) for moving more models from prototypes to production; and enterprise software to accelerate developer success.

[0237] Computational systems 802 (e.g., NVIDIA's OVX servers) can provide development and simulation platforms for testing and optimizing physics-based AI using APIs and frameworks for simulation (e.g., NVIDIA's DriveSIM, ISAAC Sim, ISAAC Gym, ISAAC Lab, etc.). Computational systems 802 allow developers to use simulation frameworks to simulate and validate robot models, and / or generate large amounts of physics-based synthetic data to guide model training. Computational systems 802 can support learning frameworks that power robot reinforcement learning and imitation learning to accelerate robot policy training and refinement. For example, computational systems 802 can be used to generate any number of simulations 808, such as in NVIDIA's OMNIVERSE. Computational systems 802 can be used to optimize and accelerate the entire software stack, from training, fine-tuning, and deploying generative AI to supporting industrial digitization within content collaboration platforms (including APIs, software development kits (SDKs), and services). These platforms allow the integration of OpenUSD, ray tracing rendering technologies (e.g., NVIDIA's RTX), and generative physics-based AI into existing software tools and simulation workflows, such as for industrial and robotics use cases (e.g., NVIDIA's OMNIVERSE). Therefore, computing system 802 can host or support the native OpenUSD software platform, enabling enterprises to connect 3D pipelines and develop advanced real-time 3D applications for industrial digitalization. With powerful ray tracing-accelerated AI and graphics capabilities, computing system 802 delivers robust performance for workloads such as extended reality (XR), multi-user design collaboration, and digital twins. This allows for the creation of physically accurate models with high-fidelity ray tracing and path tracing material rendering, large-scale, AI-enabled simulations, and the generation of realistic 3D synthetic data for training. Computing system 802 can include a single computing device 814 (e.g., an NVIDIA OVX L40S server) and / or any number of computing devices 814 (e.g., NVIDIA OVX systems) within data center infrastructure 816.

[0238] Computing device 814 (which may include a server) may include CPUs (e.g., two CPUs, each with 32 cores) and GPUs (e.g., four or eight GPUs, each including 48GB GDDR6 with ECC memory, 864GB / s memory bandwidth, a PCIe Gen4x16: 64GB / s bidirectional interconnect interface, 18,176 CUDA cores, 142 ray tracing (RT) cores, and 568 Tensor cores). Computing device 814 may include various networking and network management components, such as Intelligent Host Channel Adapters (HCAs) (e.g., two or four single-port ConnextX-7s, each with 200Gb / s, providing up to 800Gb / s InfiniBand / Ethernet), one or more DPUs (e.g., dual-port QSFP112 DPUs, such as the NVIDIA BlueField-3 DPU), providing up to 400Gb / s InfiniBand / Ethernet. In some embodiments, the NIC for computing device 814 may include a SuperNIC (e.g., NVIDIA's ConnectX-8 SuperNIC) to provide up to 800Gb / s of data throughput for in-network compute acceleration engines, thereby providing the performance and robust feature set required to support trillion-parameter-scale AI factories and scientific computing workloads. In other embodiments, computing device 814 may include a Smart Host Channel Adapter (HCA) (e.g., NVIDIA's ConnectX-7) to provide ultra-low latency and 400Gb / s throughput for in-network compute acceleration engines. Computing device 814 may include host memory (e.g., 384Gb DDR5 ECC for four GPUs, or 768Gb DDR5 ECC for eight GPUs) and may include dual in-line memory module (DIMM) slots, host boot drives (e.g., 1TB NVMe), and / or host storage (e.g., 24TB NVMe).

[0239] Similar to data center infrastructure 812, data center infrastructure 816 allows any number of computing devices 814 to be combined into a cluster configuration based on a reference architecture.

[0240] Computing system 806 can be used to deploy trained AI models on a runtime computer (such as the SoC 704 described herein). For example, these computing systems 806 can be designed for compact onboard computing needs, including ensembles of models such as control policies, vision, and language models deployed on an energy-efficient onboard edge computing system 806. (See also: [link to document]). Figures 7A to 7E A more detailed description of the components, features, and capabilities of the computing system 806.

[0241] Example Generative Model

[0242] In at least some embodiments, language models such as Large Language Models (LLMs), Visual Language Models (VLMs), Multimodal Language Models (MMLMs), Visual Language Action (VLA) models, and / or other types of generative artificial intelligence (AI) can be implemented. These models may be able to understand, summarize, translate, and / or otherwise generate text (e.g., natural language text, code, etc.), images, videos, computer-aided design (CAD) assets, OMNIVERSE and / or METAVERSE file information (e.g., USD formats such as OpenUSD), and / or the like based on context provided in input prompts or queries. In embodiments, these language models may be considered "large" because they are trained on massive datasets and have architectures with a large number of learnable network parameters (weights and biases)—e.g., millions or billions of parameters. This disclosure allows for the implementation of LLM / VLM / MMLM, etc., for summarizing textual data, analyzing data (e.g., text, images, videos, etc.), extracting insights from data (e.g., text, images, videos, etc.), and generating new text / images / videos / , etc., in a user-specified style, tone, and / or format. In some embodiments, the LLM / VLM / MMLM, etc., of this disclosure may be specifically designed for text processing, while in other embodiments, multimodal LLMs may be implemented to accept, understand, and / or generate text and / or other types of content, such as images, audio (sound, synthesized speech, etc.), 2D and / or 3D data (e.g., USD format), and / or video. For example, a visual language model (VLM) or more specifically a multimodal language model (MMLM) may be implemented to accept images, videos, sensor data, audio, text, 3D designs (e.g., CAD), and / or other input data types and / or generate or output images, videos, audio, text, 3D designs, and / or other output data types.

[0243] Various types of LLM / VLM / MMLM / etc. architectures can be implemented in various embodiments. For example, different architectures can be implemented using different techniques to understand and generate outputs (e.g., text, audio, video, images, 2D and / or 3D design or asset data, etc.). In some embodiments, LLM / VLM / MMLM / etc. architectures (e.g., recurrent neural networks (RNNs) or long short-term memory networks (LSTMs)) can be used, while in other embodiments, converter architectures (e.g., architectures relying on self-attention and / or cross-attention (e.g., between contextual data and textual data) mechanisms) can be used to understand and recognize relationships between words or tokens and / or contextual data (e.g., other text, video, images, design data, USD, etc.). One or more generative processing pipelines including LLM / VLM / MMLM / etc. may also include one or more diffusion blocks (e.g., noise reduction blocks). The LLM / VLM / MMLM / etc. of this disclosure may include encoder and / or decoder blocks. For example, discriminative or encoder-only models (e.g., BERT (Bidirectional Encoder Representations from Transformers)) can be implemented for tasks involving language understanding (e.g., classification, sentiment analysis, question answering, and named entity recognition). As another example, generative or decoder-only models (e.g., GPT (Generative Pretrained Transformer)) can be implemented for tasks involving language and content generation (e.g., text completion, story generation, and dialogue generation). LLM / VLM / MMLM / etc., including encoder and decoder components (e.g., T5 (Text-to-Text Transformer)), can be implemented to understand and generate content, such as for translation and summarization. These examples are not intended to be limiting and any architecture type (including, but not limited to, those described herein) can be implemented depending on the specific implementation and the task performed using LLM / VLM / MMLM / etc.

[0244] In various embodiments, LLM / VLM / MMLM / etc. can be trained using unsupervised learning, whereby LLM / VLM / MMLM / etc. learns patterns from a large amount of unlabeled text / audio / video / image / design / USD / etc. data. Due to the extensive training, in these embodiments, the model may not require task-specific or domain-specific training. An LLM / VLM / MMLM / etc. extensively pre-trained on a large amount of unlabeled data can be referred to as a base model and can excel at various tasks, such as question answering, summarizing, filling in missing information, translation, and image / video / design / USD / data generation. Some LLM / VLM / MMLM / etc. can be customized for specific use cases using techniques such as cue tuning, fine-tuning, retrieval augmentation generation (RAG), adding adapters (e.g., custom neural networks and / or neural network layers to tune or adjust cues or labels to bias the language model towards a specific task or domain), and / or using optimization models for specific tasks and / or other fine-tuning or customization techniques within a specific domain.

[0245] In some embodiments, the LLM / VLM / MMLM / etc. disclosed herein can be implemented using various model alignment techniques. For example, in some embodiments, guardrails can be implemented to identify incorrect or unwanted inputs (e.g., prompts) and / or outputs of the model. In this process, the system can use guardrails and / or other model alignment techniques to prevent the processing of specific unwanted inputs using LLM / VLM / MMLM / etc., and / or to prevent the output or presentation of information generated by LLM / VLM / MMLM / etc. (e.g., displays, audio outputs, etc.). In some embodiments, one or more additional models (or layers thereof) can be implemented to identify problems with the model's inputs and / or outputs. For example, these "protective" models can be trained to identify "safe" or otherwise okay or desired inputs and / or outputs and / or "unsafe" or otherwise unwanted inputs and / or outputs for a particular application / implementation. Therefore, the LLM / VLM / MMLM / etc. disclosed herein are unlikely to output language / text / audio / video / design data / USD data / etc. that may be offensive, vulgar, inappropriate, insecure, out of scope, and / or unwanted for a particular application / implementation.

[0246] In some embodiments, an LLM / VLM / etc. can be configured or able to access or use one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc. For example, for certain tasks or operations where the model is not ideally suited, the model may have instructions for accessing one or more plugins (e.g., third-party plugins) to help process the current input (e.g., as a result of training, and / or based on instructions in a given prompt). In such an example, when at least part of the prompt relates to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs) to retrieve relevant information. Another example is that if at least part of the response requires mathematical computation, the model can access one or more mathematical plugins or APIs to help solve the problem, and then the response from the plugins and / or APIs can be used in the model's output. This process can be repeated (e.g., recursively) an arbitrary number of iterations, using any number of plugins and / or APIs, until a response to each query / question / request / process / action / etc. can be generated in response to the input prompt. Therefore, models can rely not only on their own knowledge gained from training on large datasets, but also on the expertise or optimized properties of one or more external resources (such as APIs, plugins, etc.).

[0247] In some embodiments, multiple language models (e.g., LLM / VLM / MMLM / etc., multiple instances of the same language model, and / or multiple hints provided to the same language model or instances of the same language model) can be implemented, executed, or accessed (e.g., using one or more plugins, user interfaces, APIs, databases, data stores, repositories, etc.) to provide output in response to the same query or in response to separate parts of a query. In at least one embodiment, the same input query and hints (e.g., a set of constraints, condition generators, etc.) can be provided to multiple language models (e.g., language models with different architectures, language models trained on different (e.g., updated) data corpora). In one or more embodiments, the language models can be different versions of the same base model. In one or more embodiments, at least one language model can be instantiated as multiple agents, for example, providing more than one hint to constrain, guide, or otherwise influence the style, content, or character of the provided output. In one or more exemplary non-limiting embodiments, the same language model can be required to provide output corresponding to different roles, perspectives, characters, or different knowledge bases, as defined by the provided hints.

[0248] In any such embodiment, the outputs of two or more (e.g., each) language models, two or more versions of at least one language model, two or more instantiated proxies of at least one language model, and / or provided to two or more prompts for at least one language model can be further processed, such as aggregated, compared, or filtered, or used to determine (and provide) a consensus response. In one or more embodiments, the output from one language model (or version, instance, or proxy) can be provided as input to another language model for further processing and / or validation. In one or more embodiments, the language model can be required to generate or otherwise obtain output about the input source material, wherein the output is associated with the input source material. This association may include, for example, generating captions or text portions embedded (e.g., as metadata) within the input source text or image. In one or more embodiments, the output of the language model can be used to determine the validity of the input source material for further processing or inclusion in a dataset. For example, the language model can be used to evaluate the presence (or absence) of a target word in a text portion or the presence (or absence) of an object in an image, wherein the text or image is annotated to indicate such presence (or absence). Alternatively, the determination from the language model can be used to determine whether the source material should be included in the curatorial dataset, for example, but not limited to this.

[0249] Figure 9 This is a block diagram of an example generative language model system 900 suitable for implementing at least some embodiments of the present disclosure. Figure 9 In the example shown, the generative language model system 900 includes a retrieval augmentation (RAG) component 992, an input processor 905, a tokenizer 910, an embedding component 920, a plug-in / API 995, and a generative language model (LM) 930 (which may include LLM, VLM, MMLM, VLA models, etc.).

[0250] At a high level, the input processor 905 can receive input 901, which includes text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasound, etc.), 3D design data, CAD data, generic scene descriptor (USD) data (e.g., OpenUSD, etc.), depending on the architecture of the generative LM 930 (e.g., LLM / VLM / MMLM, etc.). In some embodiments, input 901 includes plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, input 901 may include numerical sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., tabular format, JSON, or XML). In the generative LM In some implementations of 930 capable of handling multimodal input, input 901 can combine text (or text that may be omitted) with image data, audio data, video data, design data, USD data, and / or other types of input data (e.g., but not limited to the data described herein). Taking raw input text as an example, input processor 905 can prepare the raw input text in various ways. For example, input processor 905 can perform various types of text filtering to remove noise (e.g., special characters, punctuation marks, HTML tags, stop words, portions of images, portions of audio, etc.) from the relevant text content. In examples involving stop words (common words that often have little semantic meaning), input processor 905 can remove stop words to reduce noise and enable generative LM. 930 focuses on more meaningful content. Input processor 905 can apply text normalization, for example, by converting all characters to lowercase, removing accents, and / or handling special cases (such as abbreviations or shortenings) to ensure consistency (e.g., converting 1 / 4 to 1 / 4). Similarly, input processor 905 and / or post-processor can perform inverse text normalization (ITN) to convert plain language back to canonical or other forms (e.g., converting 1 / 4 to 1 / 4). These are just a few examples; other types of input and / or output processing can be applied.

[0251] In some embodiments, RAG component 992 (which may include one or more RAG models, and / or may be performed using generative LM 930 itself) may be used to retrieve additional information to be used as part of input 901 or a prompt. RAGs can be used to enhance input to LLM / VLM / MMLM / etc. with external knowledge to make the answer to a specific question or query or request more relevant, such as where specific knowledge is required. RAG component 992 may obtain this additional information from one or more external sources (e.g., basic information such as basic text / images / videos / audio / USD / CAD / etc.), which can then be fed along with the prompt to LLM / VLM / MMLM / etc. to improve the accuracy of the model's response or output.

[0252] For example, in some embodiments, in addition to the data retrieved using RAG component 992, input 901 may also be generated using query or model input (e.g., questions, requests, etc.). In some embodiments, input processor 905 may analyze input 901 and communicate with RAG component 992 (or in embodiments, RAG component 992 may be part of input processor 905) to identify relevant text and / or other data to provide to generative LM 930 as additional context or information source, typically from which to identify responses, answers, or outputs 990. For example, when the input indicates that a user is interested in the required tire pressure for a particular brand and model of vehicle, RAG component 992 may use a RAG model, for example, to perform a vector search in the embedding space to retrieve tire pressure information or its corresponding text from a digital (embedded) version of the owner's manual for that particular vehicle brand and model. Similarly, when a user revisits a chatbot associated with a specific product sale or service, the RAG component 992 can retrieve previously stored conversation history (or at least its summary) and provide the previous conversation history, along with the current inquiry / request, as part of the generative LM 930 as input 901.

[0253] RAG component 992 can use various RAG techniques. For example, it can use naive RAG ( The RAG (Regulatory Aggregator) is used to index and chunk documents and apply them to an embedding model to generate embeddings corresponding to chunks. User queries can also be applied to this embedding model and / or another embedding model of the RAG component 992, and the embeddings of chunks can be compared with the embeddings of the query to identify the most similar / relevant embeddings to the query. These most similar / relevant embeddings can be provided to the generative LM 930 to generate output.

[0254] In some embodiments, more advanced RAG techniques can be used. For example, chunks can undergo pre-retrieval processes (e.g., routing, rewriting, metadata analysis, expansion, etc.) before being passed to the embedding model. Furthermore, post-retrieval processes (e.g., re-ranking, hint compression, etc.) can be performed on the output of the embedding model before generating the final embedding, which is then used for comparison with the input query.

[0255] As a further example, modular RAG techniques can be used, such as those similar to Naive RAG and / or Advanced RAG, but also including features such as hybrid search, recursive retrieval and query engines, StepBack methods, subqueries and hypothetical document embeddings.

[0256] As another example, Graph RAG can use a knowledge graph as a source of context or factual information. Graph RAG can be implemented using a graph database as a source of contextual information sent to LLM / VLM / MMLM / etc. Instead of providing the model with chunks of data extracted from larger documents (which may result in a lack of context, factual accuracy, linguistic accuracy, etc.) (or anything other than providing the model with chunks of data extracted from larger documents), Graph RAG can also provide the model with structured entity information by combining structured entity text descriptions with their many attributes and relationships, thus giving the model deeper insights. In implementing Graph RAG, the systems and methods described herein use a graph as content storage and extract associated document chunks, requiring the LLM / VLM / MMLM / etc. to use them to respond. In such embodiments, the knowledge graph may include relevant textual content and metadata about the knowledge graph, and may also be integrated with a vector database. In some embodiments, Graph RAG can use the graph as a subject matter expert, where descriptions of concepts and entities associated with a query / hint can be extracted and passed to the model as semantic context. These descriptions may include relationships between concepts. In other examples, the graph can be used as a database where a portion of a query / hint can be mapped to a graph query, the graph query can be executed, and LLM / VLM / MMLM / etc. can aggregate the results. In such examples, the graph can store associated factual information and can be used for queries (natural language queries) to graph query tools (NL to graph query tools) and entity links. In some embodiments, the graph RAG (e.g., using a graph database) can be combined with standard (e.g., vector database) RAGs and / or other RAG types to benefit from a variety of approaches.

[0257] In any embodiment, the RAG component 992 can implement plugins, APIs, user interfaces, and / or other functions to perform RAG. For example, LLM / VLM / MMLM / etc. can use graph RAG plugins to run queries on knowledge graphs to extract relevant information to feed into the model, and can use standard or vector RAG plugins to run queries on vector databases. For example, the graph database can interact with the plugin's REST interface, thus decoupling the graph database from the vector database and / or the embedded model.

[0258] The tokenizer 910 can segment (e.g., processed) text data into smaller units (tags) for subsequent analysis and processing. Depending on the implementation, the tags can represent individual words, sub-words, characters, audio / video / images, etc. Word-based tokenization divides the text into individual words, treating each word as a separate tag. Sub-word tokenization breaks words down into smaller meaningful units (e.g., prefixes, suffixes, stems), enabling the generative LM 930 to understand morphological changes and process words outside the vocabulary more effectively. Character-based tokenization represents each character as a separate tag, enabling the generative LM 930 to process text at a fine-grained level. The choice of tokenization strategy can depend on factors such as the language being processed, the task at hand, and / or the characteristics of the training dataset. Therefore, the tokenizer 910 can transform (e.g., processed) text into a structured format according to the tokenization scheme implemented in a particular embodiment.

[0259] Embedding component 920 can use any known embedding technique to transform discrete tokens into semantically meaningful (e.g., dense, continuous vector) representations. For example, embedding component 920 can use pre-trained word embeddings (e.g., Word2Vec, GloVe, or FastText), one-hot encoding, Term Frequency-Inverse Document Frequency (TF-IDF) encoding, one or more embedding layers of a neural network, and / or others.

[0260] In some implementations where input 901 includes image data / video data, etc., input processor 905 may resize the data to a standard size compatible with the format of the corresponding input channel and / or normalize pixel values ​​to a common range (e.g., 0 to 1) to ensure consistent representation, and embedding component 920 may encode the image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations where input 901 includes audio data, input processor 905 may resample the audio file to a consistent sampling rate for uniform processing, and embedding component 920 may use any known technique to extract and encode audio features, such as in the form of a spectrogram (e.g., a Mel spectrogram). In some implementations where input 901 includes video data, input processor 905 may extract frames or apply resizing to extracted frames, and embedding component 920 may extract features such as optical flow embedding or video embedding and / or encode temporal information or frame sequences. In some implementations where input 901 includes multimodal data, the embedded component 920 can use techniques such as early fusion (stitching), late fusion (sequential processing), and attention-based fusion (e.g., self-attention, cross-attention) to fuse representations of different types of data (e.g., text, images, audio, data, video, design, etc.).

[0261] Other components of the generative LM 930 and / or generative LM system 900 may use different types of neural network architectures depending on the implementation scheme. For example, a transducer-based architecture (e.g., the architecture used in models such as GPT) may be implemented, and it may include a self-attention mechanism that weights the importance of different words or tokens in the input sequence and / or a feedforward network that processes the output of the self-attention layer, applying a nonlinear transformation to the input representation and extracting higher-level features. Some non-limiting example architectures include transducers (e.g., encoder-decoder, decoder-only, multimodal), RNNs, LSTMs, fusion models, diffusion models, cross-modal embedding models that learn a joint embedding space, graph neural networks (GNNs), hybrid architectures that combine different types of adversarial networks (e.g., generative adversarial networks or GANs or adversarial autoencoders (AAEs) for joint distribution learning, linear time series modeling using a selective state-space modeling (SSM) architecture (e.g., the Mamba LLM architecture), etc. Therefore, depending on the implementation scheme and architecture, the embedded component 920 can apply the encoded representation of the input 901 to the generative LM 930, and the generative LM 930 can process the encoded representation of the input 901 to generate an output 990, which may include response text and / or other types of data.

[0262] As described herein, in some embodiments, the generative LM 930 may be configured to access or use (or be able to access or use) plugins / APIs 995 (which may include one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc.). For example, for certain tasks or operations where the generative LM 930 is not ideally suited, the model may have instructions (e.g., as a result of training, and / or based on instructions in a given prompt, such as instructions retrieved using RAG component 992) to access one or more plugins / APIs 995 (e.g., third-party plugins) to help process the current input. In such an example, when at least part of the prompt is related to a restaurant or weather, the model may access one or more restaurant or weather plugins (e.g., via one or more APIs), sending at least part of the prompt associated with a particular plugin / API 995 to the plugin / API 995, which may process the information and return an answer to the generative LM 930, which may then use the response to generate output 990. This process can be repeated (e.g., recursively) an arbitrary number of iterations and repeated using any number of plugins / APIs 995 until an output 990 that resolves each query / question / request / process / action / etc. from input 901 is generated. Therefore, the model can rely not only on its own knowledge acquired from training on a large dataset and / or from data retrieved using the RAG component 992, but also on the expertise or optimized properties of one or more external resources (e.g., plugins / APIs 995).

[0263] In some embodiments, one or more converter engines (TEs) can be implemented. Converter engines can use micro-tensor scaling to optimize performance and accuracy, such as enabling 16-bit floating-point (FP16), 8-bit floating-point (FP8), and / or 4-bit floating-point (FP4) AI processing. For example, a converter engine can use 16-bit or 8-bit floating-point precision and 8-bit or 4-bit floating-point data formats, combined with software algorithms, to improve AI performance and capabilities. By reducing mathematical operations to 8 or 4 bits, TEs can train larger networks faster without compromising accuracy. For example, TEs can include libraries for accelerating converter models on processing devices such as GPUs to provide better performance in training and inference with lower memory utilization. When TEs are combined with other technologies, such as high-speed interconnects between nodes (e.g., using NVLink switches) and tensor cores (which enable mixed-precision computation, such as micro-scaling precision support), server clusters can be more capable of training massive networks (e.g., billions of parameters) at high speeds. Therefore, it can support tensor core precision for FP64, TF32, BF16, FP16, FP8, INT8, FP6 and FP4, as well as CUDA core precision for FP64, FP32, FP16 and BF16.

[0264] The LLM / VLM / MMLM / VLA and other architectures described herein are intended to be examples only, and other suitable architectures may be implemented within the scope of this disclosure.

[0265] Example computing device

[0266] Figure 10This is a block diagram of an example computing device 1000 suitable for implementing some embodiments of the present disclosure. In some embodiments, the computing device 1000 may be used to perform one or more functions of the 3D object detection system 100, OOD detection system 200, bounding shape error detection system 300, missing object detection system 400, and / or automatic annotation system 500 described herein. The computing device 1000 may include an interconnect system 1002 directly or indirectly coupled to the following devices: memory 1004, one or more central processing units (CPUs) 1006, one or more graphics processing units (GPUs) 1008, a communication interface 1010, input / output (I / O) ports 1012, input / output components 1014, a power supply 1016, one or more presentation components 1018 (e.g., one or more displays, one or more speakers, etc.), and one or more logic units 1020. In at least one embodiment, one or more computing devices 1000 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more of the GPUs 1008 may include one or more vGPUs, one or more of the CPUs 1006 may include one or more vCPUs, and / or one or more of the logic units 1020 may include one or more virtual logic units. Thus, one or more computing devices 1000 may include discrete components (e.g., a full GPU dedicated to computing device 1000), virtual components (e.g., a portion of a GPU dedicated to computing device 1000), or combinations thereof.

[0267] although Figure 10 The various blocks are shown as connected via interconnect system 1002 using lines, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 1018 (such as a display device) may be considered I / O component 1014 (e.g., if the display is a touchscreen). As another example, CPU 1006 and / or GPU 1008 may include memory (e.g., memory 1004 may represent a storage device other than the memory of GPU 1008, CPU 1006, and / or other components). Therefore, Figure 10 The computing devices described are for illustrative purposes only. No distinction is made between such categories as “workstation,” “server,” “laptop computer,” “desktop computer,” “tablet computer,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of… Figure 10 Within the scope of computing devices.

[0268] Interconnect system 1002 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 1002 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 1006 may be directly connected to memory 1004. Further, CPU 1006 may be directly connected to GPU 1008. In cases where there is a direct or point-to-point connection between components, interconnect system 1002 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required to be included in computing device 1000.

[0269] The memory 1004 may include any computer-readable medium from a variety of computer-readable media. The computer-readable medium may be any available medium accessible by the computing device 1000. The computer-readable medium may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, the computer-readable medium may include computer storage media and communication media.

[0270] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1004 may store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 1000. As used herein, computer storage media does not include the signal itself.

[0271] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and includes any information transmission medium. The term "modulated data signal" can refer to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the above should also be included within the scope of computer-readable media.

[0272] CPU 1006 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1000 to perform one or more of the methods and / or processes described herein. Each CPU 1006 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling numerous software threads simultaneously. CPU 1006 may include any type of processor and may include different types of processors depending on the type of computing device 1000 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 1000, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (such as math coprocessors), computing device 1000 may also include one or more CPUs 1006.

[0273] In addition to or in lieu of one or more CPUs 1006, one or more GPUs 1008 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1000 to perform one or more of the methods and / or processes described herein. One or more GPUs 1008 may be integrated GPUs (e.g., with one or more CPUs 1006) and / or one or more GPUs 1008 may be discrete GPUs. In embodiments, one or more GPUs 1008 may be coprocessors of one or more CPUs 1006. GPUs 1008 may be used by computing device 1000 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 1008 may be used for general-purpose computing on a GPU (GPGPU). GPUs 1008 may include hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. GPUs 1008 may produce pixel data of an output image in response to rendering commands (e.g., rendering commands received from CPUs 1006 via a host interface). GPU 1008 may include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 1004. GPU 1008 may include two or more GPUs operating in parallel (e.g., via a link). The link may be directly connected to the GPUs (e.g., using NVLINK) or may be connected to the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 1008 may produce pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for an analog image). Each GPU may include its own memory or may share memory with other GPUs. In some embodiments, one or more functions of the 3D object detection system 100, OOD detection system 200, bounding shape error detection system 300, missing object detection system 400, and / or automatic annotation system 500 described herein may be performed at least in part by CPU 1006 and / or GPU 1008.

[0274] In addition to or in lieu of CPU 1006 and / or GPU 1008, logic unit 1020 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1000 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 1006, one or more GPUs 1008, and / or one or more logic units 1020 may execute any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 1020 may be a portion of one or more CPUs 1006 and / or GPUs 1008 and / or integrated into one or more CPUs 1006 and / or GPUs 1008, and / or one or more logic units 1020 may be discrete components or otherwise external to CPUs 1006 and / or GPUs 1008. In embodiments, one or more of the logic units 1020 may be coprocessors of one or more of the CPU 1006 and / or one or more of the GPU 1008. In some embodiments, the logic units 1020 may perform at least in part one or more functions of the 3D object detection system 100, the OOD detection system 200, the bounding shape error detection system 300, the missing object detection system 400, and / or the automatic annotation system 500 described herein.

[0275] Examples of logic unit 1020 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree lateral unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), programmable vision accelerator (PVA) (which may include one or more direct memory access (DMA) systems), and one or more vision or vector processing units (VPUs). This includes one or more pixel processing engines (PPEs) (e.g., comprising a 2D array of processing elements, each of which communicates north, south, east, and west with one or more other processing elements in the array), one or more decoupled accelerators or units (e.g., decoupled lookup table (DLUT) accelerators or units), vision processing units (VPUs), optical flow accelerators (OFAs), field-programmable gate arrays (FPGAs), neuromorphic chips, quantum processing units (QPUs), associative processing units (APUs), arithmetic logic units (ALUs), application-specific integrated circuits (ASICs), floating-point units (FPUs), input / output (I / O) elements, peripheral component interconnects (PCIs) or fast peripheral component interconnects (PCIe) elements, etc.

[0276] The communication interface 1010 may include one or more receivers, transmitters, and / or transceivers enabling the computing device 1000 to communicate with other computing devices via an electronic communication network (including wired and / or wireless communication). The communication interface 1010 may include components and functions for enabling communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wirelessband), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 1020 and / or the communication interface 1010 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via interconnect system 1002 to one or more GPUs 1008 (e.g., memory of one or more GPUs 1008).

[0277] I / O port 1012 enables computing device 1000 to be logically coupled to other devices including I / O component 1014, one or more presentation components 1018, and / or other components, some of which may be built into (e.g., integrated into) computing device 1000. Illustrative I / O component 1014 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dish antennas, scanners, printers, wireless devices, etc. I / O component 1014 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by the user. In some cases, input may be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, pen recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of computing device 1000. Computing device 1000 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. Additionally, the computing device 1000 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables motion detection. In some examples, the computing device 1000 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0278] The power supply 1016 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 1016 may provide power to the computing device 1000 so that the components of the computing device 1000 can operate.

[0279] The presentation component 1018 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 1018 may receive data from other components (e.g., GPU 1008, CPU 1006, etc.) and output the data (e.g., as images, videos, sounds, etc.).

[0280] Example network environment

[0281] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 10It is implemented on one or more instances of one or more computing devices 1000—for example, each device may include similar components, features, and / or functions of one or more computing devices 1000. Furthermore, in the case of implementing back-end devices (e.g., servers, NAS, etc.), the back-end devices may be included as part of a data center, such as, but not limited to, those described herein.

[0282] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0283] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein for the server can be implemented on any number of client devices.

[0284] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or application may respectively include network-based service software or applications. In embodiments, one or more client devices may use network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs). The framework layer may be, but is not limited to, a free and open-source software network application framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0285] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers at the state, region, country, global, etc.). The core server may assign at least a portion of the functionality to the edge server if the connection to the user (e.g., a client device) is relatively close to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0286] One or more client devices may include the information described in this article. Figure 10 At least some of the components, features, and functions of one or more example computing devices 1000 described. By way of example and not limitation, the client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming equipment or system, entertainment system, vehicle computer system, embedded system controller, chat booth, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0287] Example Terms

[0288] Example Clause 1: One or more processors, including a processing circuitry system, the processing circuitry system being configured to: obtain a feature representation associated with sensor data obtained from one or more sensors in an environment of use; generate object presence probabilities using a first model and at least based on the feature representation, each of the object presence probabilities being generated at least based on parameters of a corresponding probability distribution; generate uncertainty estimates including category uncertainty and location uncertainty using the first model and at least based on the feature representation, each of the uncertainty estimates being associated with a corresponding object presence probability and being generated at least based on the parameters of the corresponding probability distribution used to generate the corresponding object presence probability; and output an indication of the object presence probabilities and the uncertainty estimates.

[0289] Example Clause 2. One or more processors as described in Example Clause 1, wherein the processing circuitry is further configured to: combine the uncertainty estimates to generate an aggregate uncertainty estimate of the scene associated with the feature representation; and output an indication of whether the scene associated with the feature representation is an out-of-distribution scene, at least based on the aggregate uncertainty estimate of the scene associated with the feature representation.

[0290] Example Clause 3. One or more processors as described in Example Clause 1 or Example Clause 2, wherein the processing circuitry is further configured to: control the storage of data from the one or more sensors based at least on the aggregate uncertainty estimate of the scene associated with the feature representation.

[0291] Example Clause 4. One or more processors as described in any one of Example Clauses 1-3, wherein the processing circuitry is further configured to: output an indication of whether the probability of the existence of the object needs to be verified, at least based on the uncertainty estimate.

[0292] Example Clause 5. One or more processors as described in any one of Example Clauses 1-4, wherein the processing circuitry is further configured to: generate a predicted bounding shape based at least on the probability of the object's existence; combine the uncertainty estimates associated with the predicted bounding shape to generate an aggregate uncertainty estimate of the predicted bounding shape; and output an indication of the presence of a positioning error of the predicted bounding shape based at least on the aggregate uncertainty estimate of the predicted bounding shape.

[0293] Example Clause 6. One or more processors as described in any one of Example Clauses 1-5, wherein the processing circuitry is further configured to: generate one or more confidence values ​​for the detection of missed objects using a second model based at least on the feature representation, a subset of the object presence probability, and a subset of the uncertainty estimate corresponding to the subset of the object presence probability.

[0294] Example Clause 7. One or more processors as described in any one of Example Clauses 1-6, wherein the subset of the object presence probability includes the object presence probability less than or equal to a threshold.

[0295] Example Clause 8. One or more processors as described in any one of Example Clauses 1-7, wherein the feature representation includes a feature representation corresponding to a bird's-eye view (BEV) of the environment.

[0296] Example Clause 9. One or more processors as described in any one of Example Clauses 1-8, wherein the processing circuitry is further configured to: automatically label one or more scenes associated with the feature representation to generate one or more automatically labeled scenes; and identify at least a portion of the one or more automatically labeled scenes as requiring verification, based at least on the uncertainty estimate.

[0297] Example Clause 10. One or more processors as described in any one of Example Clauses 1-9, wherein said one or more processors are included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more analog operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for using one or more Large Language Models (LLMs). A system for performing operations; a system for performing operations using one or more visual language models (VLMs); a system for performing operations using one or more multimodal language models (MMLMs); a system for performing operations using one or more visual-language-action (VLA) models; a system for using or deploying one or more inference microservices; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0298] Example Clause 11. A system comprising one or more processors configured to: generate, using a model, one or more object presence probabilities of a unit of feature representation associated with sensor data obtained from one or more sensors in an environment of use, the one or more object presence probabilities being generated at least based on parameters of a corresponding probability distribution; generate, using the model, one or more uncertainty estimates of the unit of feature representation, the one or more uncertainty estimates being associated with the corresponding object presence probabilities and being generated at least based on the parameters of the corresponding probability distribution used to generate the corresponding object presence probabilities; and output an indication of the one or more object presence probabilities and the one or more uncertainty estimates.

[0299] Example Clause 12. A system as described in Example Clause 11, wherein the model includes an evidence deep learning model.

[0300] Example Clause 13. A system as described in Example Clause 11 or Example Clause 12, wherein the one or more processors are further configured to: identify a scene associated with the feature representation as an out-of-distribution (OOD) scene based at least on an aggregation of the one or more uncertainty estimates.

[0301] Example Clause 14. The system of any one of Example Clauses 11-13, wherein the one or more processors are further configured to: generate a predicted bounding shape based at least on the existence probabilities of the one or more objects; aggregate the one or more uncertainty estimates associated with the predicted bounding shape to generate an aggregated uncertainty estimate of the predicted bounding shape; and identify the predicted bounding shape as having an error based at least on the aggregated uncertainty estimate of the predicted bounding shape.

[0302] Example Clause 15. The system as described in any one of Example Clauses 11-14, wherein the feature representation is generated based at least on data captured using at least one of a LiDAR sensor or an image sensor.

[0303] Example Clause 16. The system of any one of Example Clauses 11-15, wherein the one or more processors are further configured to: automatically label one or more scenes associated with the feature representation to generate one or more automatically labeled scenes based at least on the existence probabilities of the one or more objects; and identify at least a portion of the one or more automatically labeled scenes as requiring verification based at least on the one or more uncertainty estimates.

[0304] Example Clause 17. The system as described in any one of Example Clauses 11-16, wherein the one or more processors are further configured to: train an object detection model using the one or more automatically labeled scenarios that have been validated.

[0305] Example Clause 18. The system of any one of Example Clauses 11-17, wherein the one or more processors are further configured to: generate a representation of an enclosing shape based at least on the existence probabilities of the one or more objects; and perform one or more operations corresponding to the environment based at least on the representation of the enclosing shape or the one or more uncertainty estimates.

[0306] Example Clause 19. A system as described in any one of Example Clauses 11-18, wherein said system includes at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more analog operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more Large Language Models (LLMs). A system; a system for performing operations using one or more visual language models (VLMs); a system for performing operations using one or more multimodal language models (MMLMs); a system for performing operations using one or more visual-language-action (VLA) models; a system for using or deploying one or more inference microservices; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0307] Example Clause 20. A method comprising: using a model to generate an indication of an object presence probability of at least a portion of a feature representation associated with sensor data obtained using one or more sensors and an uncertainty estimate corresponding to the object presence probability, wherein the object presence probability and the uncertainty estimate are generated based at least on parameters of the probability distribution of the at least a portion of the feature representation and a category.

[0308] This disclosure can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program modules, which are executed by a computer or other machine (such as a personal data assistant or other handheld device). Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure can also be practiced in distributed computing environments, where tasks are performed by remote processing devices linked via a communication network.

[0309] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0310] This document describes in detail the subject matter of this disclosure to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the discloser has envisioned that the claimed subject matter may be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of the method employed, these terms should not be construed as suggesting any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. One or more processors, including a processing circuitry system, said processing circuitry system being used for: Obtain a feature representation associated with sensor data acquired by one or more sensors in the usage environment; The first model is used and the object existence probabilities are generated based at least on the feature representations, each of the object existence probabilities being generated based at least on the parameters of the corresponding probability distribution; Using the first model and at least based on the feature representation, uncertainty estimates including category uncertainty and location uncertainty are generated, each of the uncertainty estimates being associated with the probability of the existence of a corresponding object and being generated at least based on the parameters of the corresponding probability distribution used to generate the probability of the existence of the corresponding object; as well as Output an indication of the probability of the object's existence and the uncertainty estimate.

2. The processor of claim 1 or more, wherein the processing circuitry is further configured to: The uncertainty estimates are combined to generate an aggregate uncertainty estimate of the scene associated with the feature representation; and An indication of whether the scene associated with the feature representation is an out-of-distribution scene is output, based at least on the aggregate uncertainty estimate of the scene associated with the feature representation.

3. The processor of claim 2 or more, wherein the processing circuitry is further configured to: control the storage of data from the one or more sensors based at least on the aggregate uncertainty estimate of the scene associated with the feature representation.

4. One or more processors as claimed in claim 1, wherein the processing circuitry is further configured to: output an indication of whether the probability of the object's existence needs to be verified, at least based on the uncertainty estimate.

5. The processor of claim 1 or more, wherein the processing circuitry is further configured to: The predicted bounding shape is generated based at least on the probability of the existence of the object; The uncertainty estimates associated with the predicted bounding shape are combined to generate an aggregate uncertainty estimate of the predicted bounding shape; and The output indicates whether there is a positioning error in the predicted bounding shape, based at least on the aggregate uncertainty estimate of the predicted bounding shape.

6. The processor of claim 1 or more, wherein the processing circuitry is further configured to: generate one or more confidence values ​​for the detection of missed objects using a second model based at least on the feature representation, a subset of the object presence probability, and a subset of the uncertainty estimate corresponding to the subset of the object presence probability.

7. One or more processors as claimed in claim 6, wherein the subset of the object presence probability includes the object presence probability less than or equal to a threshold.

8. One or more processors as claimed in claim 1, wherein the feature representation includes a feature representation corresponding to a bird's-eye view (BEV) of the environment.

9. The processor of claim 1 or more, wherein the processing circuitry is further configured to: Automatic annotation of one or more scenes associated with the feature representation to generate one or more automatically annotated scenes; and At least a portion of the one or more automatically labeled scenarios is identified as requiring verification, based at least on the uncertainty estimate.

10. The processor of claim 1 or more, wherein the processor is included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models (MMLMs); A system for performing operations using one or more visual-language-action (VLA) models; A system for using or deploying one or more inference microservices; A system for performing one or more conversational AI operations; A system for generating synthetic data; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

11. A system comprising one or more processors, said one or more processors being used to: The model generates the probability of the existence of one or more objects, which is associated with the feature representation of one or more sensors obtained from the environment of use. The probability of the existence of one or more objects is generated based at least on the parameters of the corresponding probability distribution. The model is used to generate one or more uncertainty estimates for the unit representing the feature, the one or more uncertainty estimates being associated with the probability of the existence of the corresponding object and being generated at least based on the parameters of the corresponding probability distribution used to generate the probability of the existence of the corresponding object; and Output an indication of the probability of the existence of one or more objects and an estimate of one or more uncertainties.

12. The system of claim 11, wherein the model comprises an evidence deep learning model.

13. The system of claim 11, wherein the one or more processors are further configured to: identify a scene associated with the feature representation as an out-of-distribution (OOD) scene based at least on the aggregation of the one or more uncertainty estimates.

14. The system of claim 11, wherein the one or more processors are further configured to: The predicted bounding shape is generated based at least on the probability of the presence of one or more of the objects; Aggregate the one or more uncertainty estimates associated with the predicted bounding shape to generate an aggregated uncertainty estimate of the predicted bounding shape; and The predicted bounding shape is identified as having an error, at least based on the aggregate uncertainty estimate of the predicted bounding shape.

15. The system of claim 11, wherein the feature representation is generated based at least on data captured using at least one of a LiDAR sensor or an image sensor.

16. The system of claim 11, wherein the one or more processors are further configured to: Automatically label one or more scenes associated with the feature representation based at least on the existence probability of the one or more objects to generate one or more automatically labeled scenes; and At least a portion of the one or more automatically labeled scenarios is identified as requiring verification based on at least one or more uncertainty estimates.

17. The system of claim 16, wherein the one or more processors are further configured to: train an object detection model using the one or more automatically labeled scenarios that have been validated.

18. The system of claim 11, wherein the one or more processors are further configured to: A representation of the enclosing shape is generated based at least on the probability of the existence of one or more of the objects; and One or more operations corresponding to the environment are performed, at least based on the representation of the enclosing shape or the one or more uncertainty estimates.

19. The system of claim 11, wherein the system comprises at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models (MMLMs); A system for performing operations using one or more visual-language-action (VLA) models; A system for using or deploying one or more inference microservices; A system for performing one or more conversational AI operations; A system for generating synthetic data; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

20. A method comprising: The model generates an indication of the object presence probability of at least a portion of the feature representation associated with sensor data obtained using one or more sensors, and an uncertainty estimate corresponding to the object presence probability, wherein the object presence probability and the uncertainty estimate are generated based at least on parameters of the probability distribution of the at least a portion of the feature representation and the category.

Images