Iterative projection and evaluation of scenario states using end-to-end neural networks for autonomous and semi-autonomous systems and applications

Abstract

This disclosure relates to iterative projection and evaluation of scene states using end-to-end neural networks for autonomous and semi-autonomous systems and applications. In various examples, a modular (e.g., control, driving, software, soft / hardware) stack can evaluate a candidate trajectory by predicting a subsequent scene or world state that would result from using the candidate trajectory and quantifying a measure of quality, safety, and / or desirability of the predicted subsequent state. If the estimated quality, safety, and / or desirability of the subsequent state meets a specified threshold (e.g., the candidate trajectory is predicted to result in a desirable, safe, expected, and / or beneficial future state), the candidate trajectory can be used to control navigation. Otherwise, a disengagement from autonomous navigation, execution of a safety procedure, and / or a parameterized optimization can be used to generate an optimized trajectory using an optimized planner to iteratively refine a trajectory that is predicted to result in a most desirable state (or at least a state that exceeds a threshold desirability).

Description

Background Technology

[0001] Machine learning has proven extremely effective at learning complex input-output mappings. Early examples include character recognition or image classification (ImageNet), where traditional hand-crafted algorithms performed poorly, while machine learning proved far superior. In other words, optimizing the intrinsic values ​​of a parameterized machine using example input / output data pairs and simple numerical optimization algorithms yields better results than attempting to achieve the same goal manually. Over the past few decades, the amount of training data, the size of neural networks, and available computing power have steadily increased, continuously enhancing the complexity of problems solved by machine learning and the value provided by these solutions. A recent example is Large Language Models (LLMs), which essentially predict the most likely next word (or token) based on preceding words. Trained on massive amounts of data, this simple setup can become an extremely valuable information retrieval system, whose output is often virtually indistinguishable from human-generated text.

[0002] Moving into the realm of autonomous or semi-autonomous machines (e.g., autonomous vehicles (AVs)), these machines typically rely on perception data from sensors such as cameras, LiDAR sensors, ultrasonic sensors, and / or RADAR sensors to understand their surrounding three-dimensional (3D) environment, and on navigation (e.g., driving or control) strategies implemented using machine learning to make navigation decisions such as turning, accelerating, and braking in various scenarios (e.g., driving, control, or navigation). These components responsible for decision-making and execution are sometimes referred to as a stack (e.g., a control stack or driving stack in the context of autonomous or semi-autonomous driving). In early designs, machine learning was used to transform pixels into some hand-designed representation, typically bounding boxes of detected objects and polylines of detected environmental portions, such as road boundaries or lane lines. More recently, industry leaders have begun using end-to-end (E2E) learning, combined with large-scale pre-training. For example, end-to-end learning uses a single unified neural network or other machine learning model type / architecture to directly map raw sensor data (e.g., pixels, point clouds, etc.) to driving actions without breaking down or decomposing the decision-making process into modular, hand-designed stages. End-to-end systems may include multiple neural networks (or a stacked set of one or more neural network layers, each performing a specified function), such as an image encoder, followed by a parameterized neural planner (e.g., a learning planner), which typically outputs a target trajectory or path specifying or indicating the desired position, orientation, and speed of the vehicle, while a conventional controller translates this information into execution commands (e.g., throttle, brake, and steering). A common feature of conventional end-to-end systems is that they are trained using sensor-action or sensor-trajectory pairs (unlike detectors, which are typically trained using sensor-to-human specified outputs, such as bounding boxes, polylines, and / or other output representations).

[0003] End-to-end learning yields better results, likely because it avoids information loss (e.g., the model learns which information is crucial for producing accurate and precise output, rather than estimating which information might be more useful, as humans do). For example, human-designed representations (such as object bounding boxes) have inherent limitations; they provide too much information in most cases (e.g., driving decisions rarely require precise bounding boxes for every nearby vehicle) and too little information in some cases (e.g., a ladder extending from the pickup truck bed might enlarge the detected pickup truck's bounding box, making the truck appear larger than it actually is). Large-scale pre-training typically involves self-supervised learning on massive, publicly available datasets (often referred to as "internet-scale"), followed by domain-specific fine-tuning using smaller datasets.

[0004] Because end-to-end learning directly maps sensor data to actions (or corresponding trajectories or paths), existing end-to-end systems typically need to learn the mapping from all potentially relevant scenarios to corresponding actions (e.g., how to handle a paper bag blown into the street by the wind, rather than a wheel falling off a truck; or how to handle an open pothole, rather than a closed one). In other words, existing techniques for training end-to-end systems rely on building very long internal tables of scenario-action pairs, which often leads to unwanted actions for unknown scenarios—e.g., forcibly triggering detachment events, unnecessarily bypassing objects (e.g., paper bags), and so on. Given the sheer variety and scale of all possible scenarios that autonomous or semi-autonomous vehicles should be able to handle, learning all relevant scenarios may not be feasible, considering the practical constraints on data availability, memory capacity, and computational resources. At least, end-to-end learning alone does not appear to be the most efficient approach to solving this problem.

[0005] Therefore, improved technologies are needed to determine what actions autonomous or semi-autonomous machines (such as self-driving vehicles, robots, and drones) should take. Summary of the Invention

[0006] Embodiments of this disclosure relate to a modular control (e.g., driving) stack that employs end-to-end (E2E) learning, featuring the prediction and evaluation of scenario states for autonomous and semi-autonomous systems and applications. The disclosed systems and methods evaluate candidate trajectories in partial or full iterations by predicting the resulting subsequent scenario or world states and quantifying a measure of the quality or favorability of the predicted subsequent states. Therefore, this quality or favorability measure can be used to determine whether to use the candidate trajectory and / or whether to generate an optimized trajectory. In cases where the world state is unfavorable to the candidate trajectory, this information can be stored to perform updates to neural networks included in the stack (e.g., including E2E neural networks).

[0007] Unlike traditional systems, an existing stack of the self-generating machine, such as an end-to-end control stack (e.g., a control stack, a driving stack, etc., which may include hardware and / or software components), can be used to generate candidate trajectories. Deep neural networks (DNNs) (e.g., random neural networks) can be used to generate a latent representation of the estimated future scene state that would result from taking the candidate trajectory, and a state evaluation network can be used to evaluate the quality or advantage of the estimated future state. If the estimated future state of the scene meets a specified threshold (e.g., the candidate trajectory is predicted to lead to a favorable or beneficial future state), the candidate trajectory can be used to control the self-generating machine's navigation. In some embodiments, if the estimated future scene state does not meet the specified threshold, non-engaged autonomous or semi-autonomous features (e.g., for navigation) can be determined.

[0008] In some embodiments, if the estimated future scenario state does not meet a specified threshold, parameterized optimization can be used to generate an optimized trajectory by leveraging a state evaluation network as a cost function. For example, the parameterized planner, parameterized optimization, future state estimation, and state evaluation can be run in a loop as the optimal planner, iteratively refining the trajectory predicted to lead to the most desired state. In some embodiments, to accommodate the additional time required to run one or more iterations of the optimization loop, the ego machine can slow down to accommodate the longer processing time (similar to how a human slows down a vehicle when encountering an unknown situation). In some embodiments, scenarios involving the optimized planner (e.g., scenarios where candidate trajectories predicted by the existing control stack do not meet a specified threshold) and / or scenarios where the ego machine slows down can be used to generate training data to improve the existing control stack and / or other control stacks or planners.

[0009] Therefore, current technology can be used to control the navigation of autonomous or semi-autonomous machines, vehicles, robots and / or other object or machine types. Attached Figure Description

[0010] The following describes in detail, with reference to the accompanying drawings, the system and method for a modular (e.g., drive, control, etc.) stack for autonomous or semi-autonomous systems and applications, which can employ end-to-end learning, utilizing the projection and evaluation of scenario states, wherein: Figure 1 This is a data flow diagram illustrating an example control stack for predicting and evaluating future scenario states according to some embodiments of the present disclosure; Figure 2 This is a data flow diagram of an example control stack illustrating an optimized trajectory according to some embodiments of the present disclosure to result in a predicted favorable or beneficial future scenario state; Figure 3 This is a flowchart illustrating a method, according to some embodiments of the present disclosure, for determining whether to use candidate trajectories to control the navigation of an autonomous machine, based at least on measurements of estimated favorability of an estimated future world state; Figure 4A These are examples of sensor positions for, for example, autonomous or semi-autonomous machines, having corresponding fields of view or sensing fields, according to at least some embodiments of this disclosure. Figure 4B This is an illustration of examples of the locations of components and sensors on an autonomous or semi-autonomous vehicle according to at least some embodiments of this disclosure; Figure 4C This is a block diagram of an example system architecture for autonomous or semi-autonomous vehicles, robots, and / or other machine types according to at least some embodiments of this disclosure; Figure 4D This is a block diagram of an example architecture of a computing system (e.g., a system-on-a-chip (SoC)) according to at least some embodiments of the present disclosure; Figure 4E This is a system diagram of communication between a cloud-based server and example autonomous or semi-autonomous vehicles, robots and / or other machine types according to at least some embodiments of this disclosure; Figure 5 This is a system diagram illustrating three computer ecosystems according to at least some embodiments of the present disclosure, including a computing system for generating or creating artificial intelligence (AI) (e.g., AI training and validation data), a computing system for training artificial intelligence, and a computing system for deploying AI at the edge; Figure 6 This is a block diagram of an example computing system for generative artificial intelligence (AI) according to at least some embodiments of the present disclosure; and Figure 7 This is a block diagram of an example computing device according to at least some embodiments of the present disclosure. Detailed Implementation

[0011] Systems and methods related to modular (e.g., driving, control, software, hardware, hardware-software, etc.) stacks for autonomous and semi-autonomous systems and applications are disclosed. These stacks can employ end-to-end machine learning, utilizing the projection and evaluation of scene states. For example, candidate trajectories generated using end-to-end stacks, processing pipelines, processes, etc., can be used to predict the subsequent scene or world state that the candidate trajectory will result in. The predicted scene or world state can be evaluated and used to determine whether to use the candidate trajectory and / or whether to generate an optimized trajectory. This technology can be used to assist in the generation or can directly output control or navigation commands for autonomous or semi-autonomous vehicles, machines, robots, drones, ships, aircraft, and / or other object or machine types, and / or can be used to generate training data to improve end-to-end stacks, processing pipelines, processes, etc.

[0012] Although the description herein is primarily as a stack, control stack, or driving stack, this is not intended to be limiting, and the systems and methods described herein may alternatively be referred to as processing pipelines, processes, control stacks, driving stacks, software stacks, hardware / software stacks, etc., without departing from the scope of this disclosure.

[0013] Although this disclosure may relate to examples of autonomous or semi-autonomous vehicles, robots, and / or other machine types 400 (which may alternatively be referred to herein as "vehicle 400," "self-vehicle 400," "machine 400," "self-machine 400," "robot 400," and / or "self-robot 400"), examples of which relate to Figures 4A-4EThe description herein is intended to be restrictive. For example, the systems and methods described herein may be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms (e.g., autonomous mobile robots (AMRs), humanoid robots, robotic arms and / or end effectors), warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, spacecraft, ships, shuttles (e.g., robotaxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones, robots and / or other types of vehicles, robots or machines. Furthermore, while this disclosure may describe navigation control for autonomous driving, it is not intended to be limiting, and the systems and methods described herein can be used in augmented reality (AR), virtual reality (VR), mixed reality (MR), robotics, safety and supervision (e.g., smart cities), autonomous or semi-autonomous machine applications, industrial manufacturing, simulation, and / or any other technological space where navigation control can be used. In some embodiments, the systems, methods, and / or processes described herein can be used with... Figures 4A-4E Example machine 400 Figure 5 Example calculation of ecosystem 500, Figure 6 Example generative language model system 600 and / or Figure 7 The example computing device 700 uses components, features, and / or functions similar to those of other computing devices to perform the task.

[0014] In some embodiments, an existing control stack (e.g., an end-to-end control stack) can be used to generate candidate trajectories, and a deep neural network (DNN) (e.g., a random neural network) can be used to project a representation of the aggregated history of the scene and candidate trajectories onto a latent space, generating a prior distribution representing the estimated future state of the scene that would result from taking the candidate trajectory. In embodiments, the DNN may be referred to as a world-model DNN, a world-model neural network, or a world-model machine learning model. The most probable states can be sampled from the predicted distribution, and a state evaluation network can be used to evaluate the advantage, safety, and / or quality of the estimated future states. If the estimated future state of the scene meets a specified threshold (e.g., the candidate trajectory is predicted to lead to an advantageous, safe, and / or beneficial future state), the candidate trajectory (or at least its inclusion in a set of trajectories) can be used to control navigation. In some embodiments, if the estimated future state of the scene does not meet a specified threshold, it can be determined that autonomous or semi-autonomous navigation is not engaged (or disengaged from autonomous navigation). Disengagement may include complete handover to the human driver (e.g., disengaging all autonomous and / or semi-autonomous functions), executing safety procedures (e.g., slowing down to a stop, pulling over, etc.) at least until an updated safe, beneficial, or advantageous state is achieved, and / or disengaging or deactivating one or more autonomous or semi-autonomous features (e.g., disabling system-initiated lane change (SILC) or adaptive cruise control (ACC)) until an updated safe, beneficial, or advantageous state can be achieved.

[0015] In some embodiments, if the estimated future state of the scenario does not meet a specified threshold, parameterized optimization can be used to generate an optimized trajectory by leveraging a state evaluation network as a cost function. For example, if the candidate trajectory is generated using a parameterized planner in an existing control stack, a second instance of the parameterized planner can be optimized using the estimated advantage of the estimated future state (e.g., using gradients computed by backpropagating the estimated advantage of the estimated future state to the parameterized planner in the existing control stack). This second instance of the parameterized planner can be used to predict alternative candidate trajectories, which can be passed back to the world model DNN to predict alternative future states. The state evaluation network can be used to evaluate the advantage of the alternative future states, and a specified threshold can be applied to the estimated advantage of the alternative future states.

[0016] In another example (e.g., if the initial candidate trajectory is generated using a first parametric planner in the existing control stack, which differs from a second parametric planner used to predict alternative candidate trajectories), the second parametric planner can be used to predict alternative candidate trajectories based on the current state of the scenario (e.g., and a representation of its estimated advantage), which can be applied to a world model DNN to predict alternative future states, and a state evaluation network can be used to evaluate the advantage of the alternative future states. This scenario can be used to implement specialized network architectures for each purpose (e.g., one architecture to generate trajectories that are almost always correct, while another architecture to quickly generate alternative trajectories for cases where the first planner is incorrect—or at least generate trajectories below a threshold for safety, advantage, etc.). In some embodiments, a specified threshold can be used to test the estimated advantage of the alternative future states, and if the estimated advantage of the alternative future states of the scenario meets the specified threshold, the alternative candidate trajectory can be used to control navigation.

[0017] In some embodiments (e.g., if the estimated advantage of an alternative future state of a scenario generated using any of the techniques described above does not meet a specified threshold), gradients of the estimated advantage of the alternative future state can be computed with respect to the parameters of a second parameterized planner (e.g., by backpropagating the estimated advantage of the alternative future state back to the second parameterized planner). These gradients can be used to update the second parameterized planner, which can then be used to generate an improved modification of the alternative candidate trajectory. This improved candidate trajectory can be passed back to the world model DNN and the state evaluation network for subsequent evaluation. If adopting the optimized candidate trajectory results in an estimated future state of the scenario that meets a specified threshold (e.g., the optimized candidate trajectory is predicted to result in a favorable future state), the optimized candidate trajectory (or at least included in a set of trajectories) can be used to control navigation. If the estimated future state of the scenario does not meet the specified threshold, the optimization can be iterated in one or more additional passes of the loop, e.g., until a specified convergence criterion is met (e.g., until the optimized candidate trajectory is predicted to result in an estimated future state that meets the specified threshold).

[0018] Therefore, the parametric planner, parametric optimization, future state estimation, and / or state evaluation can run as an optimized planner in a loop, iteratively refining the trajectory predicted to lead to the most desired state—or at least to states exceeding one or more specified thresholds. In some embodiments, to accommodate the additional time required to run one or more iterations of the optimization loop, the vehicle (or other self-driving machine type) may slow down (e.g., as part of safety maneuvering) to accommodate the longer processing time (just as a human would slow down their vehicle when encountering an unknown situation).

[0019] In some embodiments, scenarios involving the optimized planner (e.g., candidate trajectories predicted by the existing control stack fail to meet a specified threshold) and / or machine deceleration scenarios can be used to generate training data to improve the existing control stack. For example, if a candidate trajectory predicted by the learned planner of the end-to-end control stack results in an estimated state that fails to meet a specified threshold, the vehicle (or other self-operated machine) may be in a state it was not trained to handle and is likely to remain in this state for several consecutive time slices (e.g., corresponding 10 Hz or 100 ms decisions). For example, if the vehicle encounters a tree branch on the road, and this scenario is not reflected in its training data, the vehicle may take 10, 20, or 50 time slices to move away from the branch in order to formulate a plan to bypass it, during which time the optimized planner can engage. Thus, the optimized trajectory and / or actions generated by the optimized planner, along with the corresponding sensor data, can be recorded, incorporated into the training dataset, and used to train the learned planner of the end-to-end control stack. In this way, the learned planner of the end-to-end control stack should learn to continuously generate better trajectories and actions over time, resulting in a more efficient system (e.g., fewer scenarios where the system decelerates), while simultaneously adapting in real-time to previously unseen scenarios using the stack's iterative loop. In this manner, the machine can safely navigate previously unseen or challenging scenarios using the iterative loop process, and the data can then be stored and subsequently used to fine-tune or update the stack, requiring less processing time and computation when encountering the same (or similar) scenario again.

[0020] Thus, this technology can be used to control navigation of autonomous vehicles, semi-autonomous vehicles, machines, robots, and / or other object or machine types. Estimating and evaluating the expected future states resulting from candidate trajectories and / or optimizing trajectories that produce favorable expected future states effectively constitutes an efficient decomposition that avoids both the limitations of manually designed representations and the need to develop very large scene-action tables. Therefore, this technology can be used to improve the accuracy, precision, and performance of the control stack, driving stack, software stack, hardware / software stack, end-to-end processing pipeline, etc., while simultaneously improving machine safety and reducing the number of disengagement events.

[0021] In some embodiments, the systems and methods described herein can be performed in a simulated environment (e.g., NVIDIA DriveSIM, ISAAC Sim, ISAAC Gym, ISAAC Lab, etc.) using simulated data (e.g., simulated environment data and simulated sensor data from simulated sensors of virtual or simulated vehicles, robots, or machines within the simulated environment). For example, simulated input data (e.g., map data, perception data, self-motion data, tactile data, and / or any other data described herein) can be used to determine trajectories or control actions, and this information can be used to perform operations associated with virtual machines in the simulated environment (e.g., to control the navigation of a simulated machine). These simulated operations can be used to test the performance of underlying algorithms, systems, and / or processes before deploying them to the real world. In some cases, simulations can be used to generate synthetic training data—e.g., representing simulated scenarios of using an optimized planner to control navigation in a simulated environment. The synthetic training data (in addition to real-world data, or as a substitute for real-world data) can then be used or processed to train and test various planners or other types of machine learning models.

[0022] In any example (such as where the simulation environment is used for testing, validation, training, etc.), the simulation environment and / or associated training data may be rendered or otherwise generated using one or more optical transport simulation algorithms (such as one or more ray tracing and / or path tracing algorithms). When using optical transport simulation, the simulation system may employ one or more dedicated ray tracing hardware accelerators and / or processors (e.g., NVIDIA's RTX or another real-time ray tracing GPU, such as a GPU including one or more ray tracing (RT) cores) optimized for performing real-time or near-real-time optical transport simulation operations in conjunction with one or more other processors of the system (e.g., GPUs, CPUs, accelerators, etc.). In some embodiments, the simulation environment and / or one or more of its objects, features, or components may be generated or managed in a 3D content collaboration platform (e.g., NVIDIA's OMNIVERSE) that can be optimized or suitable for industrial digitization, generative physics, artificial intelligence, and / or other use cases, applications, and / or services. For example, the content collaboration platform or system may include a system for managing objects, features, scenes, etc., within a simulation environment, digital environment, etc., using or developing generic scene descriptor (USD) data (e.g., OpenUSD). The platform can include realistic physics simulations (e.g., using NVIDIA's PhysX software development kit (SDK)) to simulate real-world physical phenomena and physical interactions with simulations hosted on the platform. The platform can integrate OpenUSD with ray tracing / path tracing / light transport simulations (e.g., NVIDIA's RTX rendering technology) into software tools and simulation workflows for building, training, deploying, and / or testing AI systems, such as systems for testing, validating, training (e.g., machine learning models, neural networks, etc.) and / or for other tasks related to motor vehicles, robots, other machine types, and / or other systems and applications. In some examples, the simulated environment can include digital twins of real-world environments, such as specific road segments, warehouses, data centers, airports, geographic areas, ocean areas, and / or any other real-world environment in which autonomous or semi-autonomous vehicles or machines can operate.

[0023] In some embodiments, teleoperation or remote control of vehicles, robots, and / or other machines can be performed using a remote control or teleoperation system. For example, the systems and methods described herein can be used to generate suggested trajectories and / or control actions, which can be included in a visualization or map of the environment to assist a remote operator in controlling an autonomous or semi-autonomous machine through the environment, or to provide waypoints or other indications for control or navigation. Therefore, a remote operator can use visual, auditory, textual, and / or other cues or indicators generated using the systems and methods described herein to use a teleoperation system to assist vehicles, robots, machines, etc., in navigating in a real-world environment.

[0024] In some embodiments, the systems and methods described herein can be deployed to control navigation in robotic applications. A robot or robotic system may include one or more onboard processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), hardware-based programmable vision accelerator (PVA) (which may include one or more vector processing units (VPU)), direct memory access (DMA) systems and / or pixel processing engines (PPE)), hardware-based optical flow accelerator (OFA), SoC, etc.), and memory and / or storage devices (e.g., for storing control algorithms, sensor data, one or more machine learning models, entertainment content, navigation data, user preferences, etc.). Robotic systems can use these processors to execute one or more machine learning models (e.g., language models, visual language models (VLM), large language models (LLM), visual-language-action (VLA) models, multimodal language models (MMLM), etc.) that enable the robotic system to autonomously or semi-autonomously perform complex tasks, such as interacting with and / or manipulating static and / or dynamic objects, or navigating the environment using sensors (e.g., cameras, LiDAR, RADAR, ultrasonic sensors, etc.). The system can use sensor fusion techniques to combine data from multiple sensors (e.g., cameras, infrared, LiDAR, RADAR, accelerometers) to create a comprehensive model of the robot's surrounding environment. This data can be processed locally on the robot or sent to a remote server for computationally intensive tasks such as 3D mapping or SLAM (simultaneous localization and mapping). In one or more embodiments, data from individual robots (e.g., sensor data, task status, or environmental conditions) can be uploaded to the cloud, where a centralized AI model can analyze optimized commands and distribute them across the entire fleet. In some embodiments, one or more machine learning models described herein (e.g., language models, VLM, VLA, LLM, MMLM, diffusion models, NeRF models, DNN, etc.) may be used to allow the robot to perceive and reason about its environment and / or communicate with one or more other robots and / or humans in the environment. In some embodiments, the robot may (e.g., using one or more network interface cards (NICs) and / or data processing units (DPUs)) communicate with one or more locally hosted servers / computing devices and / or with one or more remotely located servers / computing devices (e.g., in one or more data centers).

[0025] In some embodiments, the systems and methods described herein can be used to present a visual representation of generated trajectories or control actions via an in-vehicle infotainment (IVI) system or an in-cabin experience (IX) application. For example, an infotainment system within a vehicle (e.g., a car, truck, drone, construction machinery, robot, semi-autonomous vehicle, or autonomous vehicle) may include one or more in-vehicle processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), hardware-based programmable vision accelerator (PVA)), which may include one or more vector processing units (VPU), direct memory access (DMA) systems and / or pixel processing engines (PPE), hardware-based optical flow accelerators (OFA), SoCs, etc.) and memory and / or storage devices (e.g., for storing control algorithms, sensor data, and one or more machine learning models). The system can utilize these processors to execute one or more machine learning models (e.g., language models) to enable functions such as voice control, personalized media recommendations, dynamic navigation, and real-time communication with other services via network connectivity. In-vehicle infotainment systems can also use natural language processing (NLP) models to enable voice-based interaction. One or more machine learning models can be stored locally or accessed via one or more APIs connected to cloud services, enabling the system to process requests in real-time or near real-time.

[0026] In some examples, the machine learning models described herein (e.g., deep neural networks, language models, LLMs, VLMs, multimodal language models, visual-language-action (VLA) models, perceptual models, tracking models, fusion models, transformer models, diffusion models, encoder-only models, decoder-only models, encoder-decoder models, neural rendering field (NERF) models, etc.) can be packaged into microservices—such as inference microservices (e.g., NVIDIA NIM)—which may include containers (e.g., operating system (OS) level virtualization packages) that may include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model “engine.” For example, an inference microservice may include the container itself as well as the model (e.g., weights and biases). In some cases, such as when the machine learning model is small enough (e.g., with a small number of parameters), the model may be included within the container itself. In other examples (e.g., when the model is large), the model may be hosted / stored in the cloud (e.g., in a data center), and / or hosted locally and / or at the edge (e.g., on a local server or computing device, but outside the container). In these embodiments, the model may be accessed via one or more APIs (e.g., REST APIs). Therefore, in some embodiments, the machine learning models described herein can be deployed as inference microservices to accelerate model deployment on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, an optimized inference engine (e.g., execution software built using standardized AI model deployments, such as NVIDIA's Triton Inference Server), and / or one or more APIs for high-performance deep learning inference, which may include inference runtimes and model optimizations providing low latency and high throughput for production applications, such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). The machine learning models described herein may be included as part of a microservice along with an acceleration infrastructure capable of deployment with a single command and / or orchestrated and automatically scaled on the acceleration infrastructure using a container orchestration system (e.g., reaching data center scale on a single device). Therefore, the inference microservice may include a machine learning model (e.g., optimized for high-performance inference), inference runtime software for executing the machine learning model and providing output / response to inputs (e.g., user queries, prompts, etc.), and enterprise management software for providing health checks, authentication, and / or other monitoring. In some embodiments, the inference microservice may include software for in-situ replacement and / or updating of the machine learning model. During replacement or updating, the software performing the replacement / update may maintain the user configurations of the inference runtime software and the enterprise management software.

[0027] While this article may describe examples using machine learning models (such as neural networks), it is not intended to be limiting. For example, but not limited to, any of the various machine learning models and / or neural networks described herein may include any type of machine learning model, such as those using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoder neural networks, artificial neural networks (ANN), convolutional neural networks (CNN), recurrent neural networks (RNN), perceptrons, long short-term memory (LSTM) networks, multilayer perceptron (MLP) networks, deep stacked networks (DSN), generative pre-trained (GPT) models or networks, feedforward networks, radial basis function ANNs, self-organizing maps (SOM), Kohonen maps, Hopfield networks, Boltzmann machines, deep belief neural networks, deconvolutional neural networks, generative adversarial networks (GAN)), liquid machines, modular neural networks, sequence-to-sequence models, networks using transformer architectures, state-space models (SSM) (e.g., networks using Mamba architectures (e.g., Mamba-1, Mamba...)). 2, etc.), networks using selective state-space models, networks using structured state-space sequence models, etc.), diffusion models (e.g., diffusion probability models, fraction-based generative models, etc.), neural radiation field (NeRF) models, Gaussian sputtering models, Kolmogorov-Arnold networks (KAN), models with encoder-only architectures, models with decoder-only architectures, models with encoder-decoder architectures, generative machine learning models, language models, large language models (LLM), visual language models (VLM), multimodal language models (MMLM), large action models (LAM), visual-language-action (VLA) models, etc., and / or other types of machine learning models.

[0028] The systems and methods described herein may be used, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, spacecraft, ships, shuttles (e.g., robotaxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones, and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, by way of example but not limited to: machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and supervision, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets (e.g., NVIDIA's Omniverse), cloud computing and / or any other suitable application.

[0029] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines, etc.), systems implemented using robots, aviation systems, medical systems, marine systems, smart zone monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing language models (such as large language models (LLM), visual language models (VLM), visual-language-action (VLA) models, and / or multimodal language models), systems using or deploying one or more inference microservices, systems containing one or more machine learning models deployed in services or microservices and OS-level virtualization packages (e.g., containers), systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0030] refer to Figure 1 , Figure 1This is a data flow diagram illustrating an example control stack 100 according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely examples. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to the arrangements, components, features, and elements shown (e.g., machines, interfaces, functions, sequences, functional groups, etc.), and some elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be deployed in any suitable combination and at any suitable location (e.g., on a local device, vehicle, or machine at the edge; locally deployed—(e.g., on a locally hosted server located remotely)—e.g., on one or more computing or server devices in one or more data centers in the cloud; and / or other locations). The various functions performed by the entities described herein can be implemented through hardware, firmware, and / or software. For example, one or more processors (e.g., one or more central processing units (CPUs), one or more graphics processing units (GPUs), microprocessors, microcontrollers, embedded processors, digital signal processors (DSPs), image signal processors (ISPs), physical processing units (PPUs), field-programmable gate arrays (FPGAs), one or more accelerators (e.g., deep learning accelerators (DLAs), deep learning accelerator clusters (XNNs), neural network accelerators (NNAs) and / or neural processing units (NPUs), programmable vision accelerators (PVAs), optical flow accelerators (OFAs), etc.), application-specific integrated circuits (ASICs), data processing units (DPUs), quantum processors, etc.) can be used to execute instructions stored in memory to achieve various functions. In some embodiments, the systems, methods, and / or processes described herein can be used with... Figures 4A-4E Example machine 400 Figure 5 Example calculation of ecosystem 500, Figure 6 Example generative language model system 600 and / or Figure 7 Example computing device 700 uses components, features, and / or functions similar to those of other computing devices to perform the operation.

[0031] Typically, the control stack 100 can be deployed on a real or simulated self-machine (e.g., Figures 4A-4E In an autonomous or semi-autonomous vehicle, robot, or machine (400), or otherwise accessible to or used by the self-machine for control of the self-machine. Although described herein primarily as a stack, control stack, or driving stack, this is not intended to be limiting, and the systems and methods described herein may alternatively be referred to as processing pipelines, processes, control stacks, driving stacks, software stacks, hardware / software stacks, etc., without departing from the scope of this disclosure.

[0032] exist Figure 1 In the example shown, control stack 100 includes an end-to-end (E2E) control stack consisting of a perceptual encoder 110, a state aggregation network 120, and a neural planner 140. This control stack can be used to generate candidate trajectories 145 and / or control actions from sensor data 105. Future state estimation network 150 (also known as a world model neural network) can project a representation of the aggregated history of the scene, along with the candidate trajectories 145 and / or control actions, into a latent space and generate a prior distribution representing the estimated future states of the scene that would result from taking the candidate trajectory 145 and / or control action. Therefore, state estimator 170 can sample predicted future states 160 from the predicted distribution and generate a representation of the quality 180 (or advantage) of the predicted future state 160. Therefore, the self-machine's control component 190 can compare the quality 180 of the predicted future state 160 with a specified threshold. If the quality 180 of the predicted future state 160 meets the specified threshold (e.g., the candidate trajectory 145 and / or control action is predicted to lead to a favorable or beneficial future state), then the control component 190 can use (or at least consider) the candidate trajectory 145 and / or control action to control the self-machine's navigation. The control stack 100 can operate at any suitable frame rate (e.g., 30 fps, 45 fps, 60 fps, etc.). For example, one or more frames of sensor data 105 can be generated for each time slice, and the control stack 100 can use one or more frames of sensor data 105 for each time slice to generate trajectories and / or control actions, and can use the generated trajectories and / or control actions to control the self-machine (whether at the same frame rate as the generated sensor data 105 or at some other frame rate).

[0033] Typically, the control stack 100 can use sensor data 105 from any number and / or any type of sensor, such as, but not limited to, one or more cameras, LiDAR sensors, RADAR sensors, ultrasonic sensors, and / or other sensor types, as described below. Figures 4A-4EThe sensor 105 is described in the autonomous or semi-autonomous machine 400, and the control stack 100 can process the sensor data 105 into a format acceptable to the perception encoder 110. In some embodiments, the sensor can be deployed to the autonomous machine and can be used to generate one or more frames of sensor data 105 representing the 3D environment in which the autonomous machine is navigating. For example, the sensor data 105 may include one or more images from one or more cameras (e.g., N images from N cameras (e.g., surround-view cameras), or stitched images), and can be stacked, connected, or otherwise combined in any suitable manner. Additionally or alternatively, in some embodiments where sensor data 105 includes a representation of measured 3D points (e.g., a LiDAR or RADAR point cloud), control stack 100 may accumulate the measured 3D points (e.g., over a specified number of time slices, LiDAR or RADAR spins, etc.), transform the measured 3D points to a single coordinate system (e.g., centered on the origin of the ego machine's equipment coordinate system), perform self-motion compensation on the measured 3D points (e.g., compensate to the latest known position of the ego machine), and / or project the measured 3D points to form a projected image representing any suitable view of the 3D environment (e.g., a perspective view, an orthographic graph) having any number of channels (e.g., a single-channel image, a multi-channel image, or a tensor) representing any characteristic of sensor data 105 (e.g., the projected position of the measured 3D points, one or more reflection characteristics, image data such as pixel color, etc.). For example, (accumulated, self-motion compensated) LiDAR point clouds can be projected to form a perspective or top view (e.g., bird's-eye view (BEV)) representation (e.g., elevation map) of the projected 3D positions of the measured 3D points, and a LiDAR distance image with any number of channels (e.g., storing reflection characteristics such as intensity, elevation, or distance profiles). In some embodiments, different sensors (whether of the same or different types) can be used to generate sensor data 105 of different modalities (e.g., camera images, LiDAR distance images, etc.) that have the same (e.g., perspective) view of the 3D environment in a common image space, and sensor data 105 from different sensors and / or sensor modalities can be stored in different channels of a multi-channel image or tensor. These are intended only as examples, and any suitable representation of the 3D environment and / or the state of the self-machine can be used as at least a portion of the sensor data 105.

[0034] Typically, the perceptual encoder 110 can be implemented using any type of machine learning model or neural network (e.g., backbone network, feature extractor, etc.), such as the machine learning model or neural network described herein. For example, the perceptual encoder 110 may include, but is not limited to, a feature extractor (e.g., a deep neural network (DNN), encoder / decoder, etc.) that includes convolutional layers, transformer layers, pooling layers, and / or other types of layers, and the perceptual encoder 110 can be used to extract a latent representation of the state of the scene represented by the sensor data 105 (e.g., a scene embedding representing a 3D environment).

[0035] The state aggregation network 120 can accept the latent state of the scene as input and generate an encoded state history 125. Typically, the state aggregation network 120 can be implemented using any known machine learning model or neural network, such as a gated recurrent unit (GRU), recurrent neural network (RNN), long short-term memory (LSTM), transformer, temporal convolutional network (TCN), or a model based on another sequence that can be used as a memory mechanism. By sequentially operating the state aggregation network 120, the state aggregation network 120 can deterministically and efficiently aggregate or accumulate previous latent states (time-sequence) using the hidden states of the state aggregation network 120. t Previously, the memory mechanism was sequentially updated in each time slice based on the latent representation of the sensor data 105 extracted by the perceptual encoder 110 for the current time slice and the network's previous hidden states. Thus, the encoded state history 125 can be provided to the neural planner 140 and the future state estimation network 150.

[0036] Typically, the neural planner 140 can be implemented using any type of machine learning model or neural network, such as the machine learning model or neural network described herein, and can be used to predict or generate candidate trajectories 145 and / or control actions from an encoded state history 125. For example, the neural planner 140 can predict any suitable representation of the candidate trajectory 145 (e.g., a sequence of future waypoints with associated timestamps, positions, velocities, and / or accelerations) and / or candidate control actions or execution commands (e.g., steering angle, acceleration or throttle, braking pressure, advanced navigation decisions such as lane changing, merging, or stopping at an intersection, motor or actuator control for robot motion, grasping, manipulation (e.g., using an end effector, robotic arm, etc.). The neural planner 140 may additionally or alternatively accept other types of input, such as route and / or map data 130 (e.g., a planned navigation route, a map of the environment, etc.). Figure 4CHigh-resolution (“HD”) maps 422 or other map information 494), representations of detected features of airworthy surfaces (e.g., rasterized lane maps); advanced route guidance (e.g., change lanes as soon as possible), etc.), the self-state of the self-machine (e.g., detected position, orientation, speed, self-motion, etc.), the history of actions taken by the self-machine, and / or others.

[0037] For example, the path planner of control stack 100 can generate planned navigation routes using any known technique (e.g., any path planning algorithm) and can represent the planned navigation routes in a variety of ways (e.g., a rasterized binary mask of a view (e.g., a top view) visualizing the planned trajectory; a sequence of 2D waypoints; a list, array, queue, or other representation of a sequence of navigation route commands (e.g., turn right after 100 feet, go straight for 30 miles, turn left at the next traffic light), which can be encoded using one-hot encoding, embedding vectors, RNNs for processing order of disposal, positional encoding, and transformers, etc.), and can use a feature extractor (e.g., using any type of machine learning model or neural network implementation, such as the machine learning model or neural network described herein) to generate corresponding embeddings. In some embodiments, the state detection component of control stack 100 can generate representations of one or more states of the ego machine. For example, the velocity of the ego machine can be estimated (e.g., using a velocity sensor, Global Positioning System (GPS), etc.) and encoded as a velocity embedding. In some embodiments, a representation of the self-movement of the self-machine (e.g., velocity, relative pose) can be detected (e.g., using a velocity sensor or GPS estimation) or generated (e.g., by employing differences between successively detected self-poses) and a corresponding embedding can be generated using a corresponding feature extractor (e.g., using any type of machine learning model or neural network implementation, such as the machine learning model or neural network described herein). These are just a few examples, and additional or alternative feature extraction can be performed on other components of the detected self-state (e.g., represented by GPS data, IMU data, and / or other means).

[0038] Therefore, the neural planner 140 can accept a representation of encoded state history 125, route and / or map data 130, the self-state of the ego machine, and / or other types of input, and can generate candidate trajectories 145 and / or control actions from these inputs using any known techniques. Although Figure 1The illustrated embodiments include a neural planner, but the navigation strategy can be implemented using any known technique, such as techniques that apply one or more human-defined or established rules or heuristics to make navigation decisions (classical planners), techniques that use neural networks to predict or generate planned trajectories or actions (neural planners), techniques that use conventional controllers to implement classical control algorithms (e.g., model predictive control (MPC)), some combination of the above techniques (e.g., a neural or classical planner that predicts actions, then uses a classical control algorithm to optimize the predicted actions), and / or other techniques. For example, the navigation strategy can use a neural planner 140 or a classical planner to predict candidate trajectory 145 (e.g., instead of directly predicting acceleration and steering), while the control component 190 can use MPC to consider the dynamic characteristics of the ego machine and use candidate trajectory 145 as a reference to track or guide its control actions. Thus, a representation of candidate trajectory 145 and / or control actions can be provided. Figure 1 The next branch is used to predict and evaluate the quality of the state that would result from taking the candidate trajectory 145 and / or control actions.

[0039] More specifically, an evaluation network (e.g., a trajectory) including a future state estimation network 150 and a state evaluator 170 can be used to predict and evaluate the quality of the state resulting from taking the candidate trajectory 145 and / or control action. The following discussion will use the evaluation of the candidate trajectory 145 and / or control action predicted by the neural planner 140 as an example, but other variations are also possible. For example, the upper branch of control stack 100 and / or control stack 200 may include a classical planner that generates the candidate trajectory, while the lower branch may evaluate the candidate trajectory generated by the classical planner. In some embodiments, a classical planner or a neural planner may generate a trajectory that a classical control algorithm can use to generate an optimized control action, and the lower branch may evaluate the trajectory and / or control action. In some embodiments, a classical planner or a neural planner may generate a trajectory, the lower branch may evaluate the trajectory, and the control component 190 may implement a classical control algorithm to generate an optimized control action based on the trajectory. These are just a few examples, and other variations are also considered within the scope of this disclosure.

[0040] exist Figure 1In the example shown, the future state estimation network 150 (or world model neural network) takes a representation of candidate trajectories 145 (and / or control actions) and an encoded state history 125 as input, and projects this input into a latent representation of the predicted future state 160 of the scene. Typically, the future state estimation network 150 can be implemented using any type of neural network or other machine learning model, such as those described herein. For example, the future state estimation network 150 can include generative DNNs, such as one or more stochastic neural networks (e.g., Bayesian neural networks, variational autoencoders (VAEs)), which predict a state distribution (or world model) that models a range of possible (estimated future) scene or world states. More specifically, the future state estimation network 150 can project a representation of the predicted future state 160 of the scene from... t -1 to t The system models a stochastic latent space world transition conditioned on encoded state history 125, candidate trajectories 145 and / or control actions, and / or other inputs (e.g., routes and / or map data 130, the self-state of the self-machine, the history of actions taken by the self-machine, etc.). Therefore, the future state estimation network 150 can predict the parameters of a distribution of possible latent states (e.g., a Gaussian distribution), from which the state estimator 170 can sample to generate prior (estimated future) latent state samples representing the estimated future states. For example, the future state estimation network 150 can predict a multidimensional latent distribution parameterized by the mean vector and covariance (or variance) of each latent unit, and the state estimator 170 can identify one or more most probable states for each latent unit. For a single-mode Gaussian distribution, the state estimator 170 can sample the most probable states by selecting the predicted mean as the most probable latent state. For more complex or multimodal distributions, the state estimator 170 can use any known mode-seeking technique to identify the most likely state, and can use any suitable technique (e.g., averaging) to combine the potential representations of the most likely identified states.

[0041] Therefore, the latent representation of the most likely state can be used as the predicted future state 160 and evaluated by the state evaluator 170 to generate a representation of the quality 180 (or advantage) of the future state. Typically, the state evaluator 170 can be implemented using any type of machine learning model or neural network, such as those described herein. For example, the state evaluator 170 can use any known regression or classification technique to generate a measurement of the predicted quality, safety, advantage, or other state, indicating the acceptability or desirability of the future state. By way of non-limiting example, the state evaluator 170 can regress a floating-point scalar value (e.g., normalize to a range of values, such as 0 to 10, where 0 represents the most likely state and 10 represents the least likely state), or it can classify the predicted future state 160 into one of several discrete bins representing different measurements of quality or advantage (e.g., by generating a classification score for each bin and taking the measurement of quality or advantage corresponding to the bin with the highest classification score). Thus, the state evaluator 170 can extract the measurement results of the quality 180 (e.g., quality, advantage, expectation, safety or other measure) of the predicted future state 160 and provide it to the control component 190.

[0042] Therefore, control component 190 can compare the quality 180 of the predicted future state 160 with a specified threshold. If the quality 180 of the predicted future state 160 meets the specified threshold (e.g., candidate trajectory 145 is predicted to lead to a favorable or beneficial future state), control component 190 can use candidate trajectory 145 and / or control actions to control the navigation of the autonomous machine. In some embodiments, control component 190 can use control actions predicted by neural planner 140 (e.g., steering angle, acceleration or throttle, braking pressure, advanced navigation decisions such as lane changing, merging, or stopping at an intersection) as execution commands. In some embodiments, control component 190 can use a conventional controller to translate candidate trajectory 145 predicted by neural planner 140 into corresponding control actions, and these control actions can be used as execution commands.

[0043] In some embodiments, if the quality 180 of the predicted future state 160 does not meet a specified threshold, the control component 190 may decide not to engage autonomous navigation (e.g., if the assessment is associated with a request to switch to autonomous navigation), or may decide to disengage autonomous navigation (e.g., if autonomous navigation is engaged). If disengaging autonomous navigation, the control component 190 may trigger one or more alerts (auditory, visual, or tactile) to draw the operator's attention and may collaborate with a driver or operator monitoring system to assess the operator's attention using any known technology. If the operator remains alert and ready to take control, the control component 190 may transfer control to the operator (e.g., by gradually disabling autonomous functions). If the operator is inattentive or unresponsive, the control component 190 may adjust one or more operator assistance features, such as adaptive cruise control (e.g., increasing following distance or reducing vehicle speed) or lane keeping assist (e.g., more proactively correcting lane departure), and / or trigger ADAS to perform one or more safety interventions (e.g., automatic braking, emergency steering, automatic pull-over, etc.).

[0044] Therefore, the control stack 100 can be deployed within the self-machine and / or otherwise used to control the navigation of the self-machine. For example, the control stack 100 can run in real time on the self-machine's on-chip system (which may correspond to...). Figure 4C On SoC 404 (e.g., automotive SoCs, such as NVIDIA's DRIVE Orin™ SoC, which can be used as a central computer for in-vehicle computing). Other types of SoCs that can be used for robots or self-contained machines include AI-optimized SoCs (e.g., NVIDIA Jetson series), robotic SoCs (e.g., for robotic applications such as drones, service robots, or industrial robots), industrial SoCs (e.g., for industrial robots or automation systems for factory automation, robotics, or control systems), drone and unmanned aerial vehicle (UAV) SoCs (e.g., for handling flight control, navigation, and / or real-time video processing) and / or others.

[0045] In embodiments where the control stack 100 includes one or more machine learning models (e.g., an end-to-end control stack consisting of a perceptual encoder 110, a state aggregation network, and a neural planner 140), any known technique can be used to train the machine learning models. For example, the perceptual encoder 110 can be trained using any number of auxiliary tasks that generate training signals (loss functions) (e.g., image classification, object detection, or end-to-end learning), and / or the perceptual encoder 110 can be self-supervised (e.g., implemented using any type of autoencoder that learns to reproduce the input through a compact latent representation). The end-to-end control stack can be trained using behavioral cloning, imitation learning, and / or reinforcement learning (combined with expert demonstration) to minimize planning loss to ensure smooth trajectory generation, add perturbations that intentionally place the ego machine in a bad state, and provide representations of how the ego machine should recover, and / or other methods. The future state estimation network 150 can be self-supervised trained using any number of time-recorded (e.g., video) sequences of sensor data 105. For example, since world model neural networks may be more efficient to train in feature space, one or more auxiliary feature extractors can be used to extract latent representation sequences from sensor data 105, and the extracted sequences can be used to self-supervised train the future state estimation network 150. The state estimator 170 can be trained using training data representing world scene states labeled with corresponding measures of quality or advantage (e.g., good states are labeled with ground truth classifications or regressions representing high-quality or advantageous states, and bad states are labeled with ground truth classifications or regressions representing low-quality or disadvantageous states). For example, almost any state of human navigation (e.g., driving a vehicle) without causing an accident can be labeled a good state. An accident and the short period preceding it can be labeled an undesirable state. Additionally or alternatively, human detachment from deployed autonomous navigation can be labeled a bad state. In some embodiments, to address the possibility of uneven distribution of good and bad states represented in the training data, techniques such as using an energy-based model can be used to train the state evaluator 170 to distinguish between observed and unobserved states (e.g., by assigning lower energy values ​​to seen states and higher energy values ​​to unobserved or unlikely states), and to learn that any observed state is good and any unobserved state is bad. Therefore, the control stack 100 can be trained to generate and evaluate candidate trajectories and / or control actions, and to control navigation using the trajectories and / or control actions predicted to lead to the most favorable state.

[0046] Turn now Figure 2 , Figure 2This is a data flow diagram illustrating an example control stack 200 with an optimized trajectory according to some embodiments of the present disclosure to achieve a predicted favorable or beneficial future scenario state. Although generally described herein as a stack, control stack, or driving stack, this is not intended to be limiting, and the systems and methods described herein may instead be referred to as processing pipelines, processes, control stacks, driving stacks, software stacks, hardware / software stacks, etc., without departing from the scope of this disclosure.

[0047] Figure 1 and Figure 2 Components with the same reference number can operate similarly. However, in Figure 2 If control component 190 determines that the quality 180 (or advantage) of the predicted future state 160 does not meet a specified threshold, control component 190 may initiate one or more iterations of parametric optimization to generate an optimized trajectory using state evaluator 170 as a cost function. For example, control component 190 may iteratively refine the trajectory predicted to reach the most desired (or optimized) state by looping through a parametric planner (e.g., neural planner 210), parametric optimization of neural planner 210, future state estimation using future state estimation network 150, and state evaluation operation using state evaluator 170 as the optimal planner. Thus, this loop can effectively operate as an auxiliary or alternative control stack when candidate trajectories 145 generated using the existing or primary control stack and / or control actions fail to meet the specified threshold.

[0048] For example, in some embodiments (e.g., if neural planner 210 and neural planner 140 are instances of the same neural network), control component 190 can optimize neural planner 210 using the quality 180 (or advantage) of the predicted future state 160 (e.g., using a gradient computed by backpropagating the quality 180 (or advantage) of the predicted future state 160 to neural planner 210), and control component 190 can trigger a pass via an updated version of neural planner 210, future state estimation network 150, and state evaluator 170. In some embodiments (e.g., if neural planner 210 and neural planner 140 are different networks), control component 190 can trigger a pass via neural planner 210, future state estimation network 150, and state evaluator 170 without initial optimization of neural planner 210.

[0049] In any scenario, control component 190 can use neural planner 210 to predict candidate trajectories 220 and / or control actions. Typically, neural planner 210 can be implemented using any type of machine learning model or neural network, such as those described herein, and can be used to predict or generate candidate trajectories 220 and / or control actions from encoded state history 125. For example, neural planner 210 can predict any suitable representation of candidate trajectory 220 (e.g., a sequence of future waypoints with associated timestamps, positions, speeds, and / or accelerations) and / or candidate control actions or commands to be executed (e.g., steering angle, acceleration or throttle, braking pressure, advanced navigation decisions such as changing lanes, merging lanes, or stopping at an intersection). Neural planner 210 can accept inputs such as encoded state history 125, route and / or map data 130, the self-state of the ego machine, the history of actions taken by the ego machine, and / or others. Therefore, neural planner 210 can use any known techniques to generate candidate trajectories 220 and / or control actions from these inputs.

[0050] Therefore, control component 190 can use neural planner 210 to predict candidate trajectory 220 and / or control actions, use future state estimation network 150 to generate a predicted future state 160 that the candidate trajectory 220 would result in (e.g., replacing the candidate trajectory 145 from the first pass with the candidate trajectory 220 from subsequent passes via future state estimation network 150), and use state evaluator 170 to extract a measurement of the quality 180 (or advantage) of the predicted future state 160. If control component 190 determines that the quality 180 of the predicted future state 160 of the scenario resulting from taking the optimized candidate trajectory 220 satisfies a specified convergence criterion (e.g., the candidate trajectory 220 and / or control actions are predicted to result in a predicted future state 160 with a quality 180 that satisfies a specified threshold), then control component 190 can use the optimized candidate trajectory 220 to control navigation. If not, the control component 190 can use any known optimization process to modify the parameters of the neural planner 210 to generate improved candidate trajectories 220 using the state evaluator 170 as the cost function.

[0051] For example, control component 190 can use any known technique (e.g., backpropagation) to compute the gradient of the quality 180 of the predicted future state 160 of the scene relative to the parameters of neural planner 210, and use these gradients to update the parameters of neural planner 210. Control component 190 can then use the updated neural planner 210 to generate an improved modification of candidate trajectory 220, and pass this improved candidate trajectory 220 to future state estimation network 150 and state evaluator 170 for further evaluation. If control component 190 determines that taking this optimized candidate trajectory 220 results in a quality 180 of the predicted future state 160 of the scene that meets a specified threshold (e.g., the optimized candidate trajectory 220 is predicted to lead to a favorable future state), then control component 190 can use the optimized candidate trajectory 220 to control navigation. If the control component 190 determines that the quality 180 of the predicted future state 160 of the scenario does not meet a specified threshold, the control component 190 may iterate the optimization process in one or more additional passes of the loop, for example, until a specified convergence criterion is met (e.g., until the optimized candidate trajectory 220 and / or the control action is predicted as the predicted future state 160 that causes the quality 180 to meet the specified threshold).

[0052] Therefore, control component 190 can iteratively optimize neural planner 210 and the resulting candidate trajectories 220 and / or control actions until an optimized trajectory and / or control action is found that produces a predicted future state 160 with quality 180 satisfying a specified threshold. Thus, control component 190 can use the optimized trajectory and / or control action to control the navigation of the autonomous machine. In some embodiments, control component 190 can use control actions predicted by neural planner 210 (e.g., steering angle, acceleration or throttle, braking pressure, advanced navigation decisions such as lane changing, merging, or stopping at an intersection) as execution commands. In some embodiments, control component 190 can use a conventional controller to translate the optimized trajectory predicted by neural planner 210 into corresponding control actions and use these control actions as execution commands. In some embodiments, to accommodate the additional time required to run one or more iterations of the optimization loop, control component 190 can adjust one or more driver assistance features, such as adaptive cruise control (e.g., increasing following distance or decreasing vehicle speed), to accommodate the longer processing time (similar to how a human slows down a vehicle when encountering an unknown situation).

[0053] Therefore, the control stack 200 can be deployed within the self-machine and / or otherwise used to control the navigation of the self-machine. For example, the control stack 200 can run in real time on the self-machine's on-chip system (which may correspond to...). Figure 4COn SoC 404 (e.g., automotive SoCs, such as NVIDIA's DRIVE Orin™ SoC, which can be used as a central computer for in-vehicle computing). Other types of SoCs that can be used for robots or self-contained machines include AI-optimized SoCs (e.g., the NVIDIA Jetson series), robotic SoCs (e.g., for robotic applications such as drones, service robots, or industrial robots), industrial SoCs (e.g., for industrial robots or automation systems for factory automation, robotics, or control systems), drone and unmanned aerial vehicle (UAV) SoCs (e.g., for handling flight control, navigation, and / or real-time video processing) and / or others.

[0054] In some embodiments, scenarios in which the neural planner 210 participates (e.g., scenarios where candidate trajectories 145 predicted by neural planner 140 or other existing control stacks fail to meet a specified threshold) and / or scenarios where the ego machine decelerates can be used to generate training data to improve neural planner 210 or other existing planners or control stacks. For example, if candidate trajectories 145 predicted by neural planner 140 result in a predicted future state 160 where quality 180 fails to meet a specified threshold, the vehicle (or other ego machine) may be in a state it was not trained to handle and is likely to remain in this state for several consecutive time slices (e.g., corresponding 10 Hz or 100 ms decisions). Therefore, control component 190 can record optimized trajectories and / or control actions generated by neural planner 210, along with corresponding sensor data 105, include them in a training dataset, and use this training dataset (or a training dataset that another training engine can use) to train (e.g., fine-tune) neural planner 140, an existing control stack, or another planner or control stack. Thus, the neural planner 140 and / or the end-to-end control stack should be able to learn to generate better trajectories and actions over time, resulting in a more efficient control stack 200 (e.g., fewer decelerations).

[0055] Now for reference Figure 3 Each block of the method 300 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be performed using one or more processors (e.g., but not limited to, the processors described herein) that execute instructions stored in one or more memories or memory systems. In some embodiments, the computer process may also be embodied as computer-usable instructions stored on a computer storage medium. The method may be provided by a standalone application, service, or managed service (used independently or in combination with another managed service), application programming interface (API), and / or plug-in to another product, etc. Furthermore, regarding... Figure 1-2Method 300 is described by way of example. However, this method may also be additionally or alternatively performed by any system or any combination of systems (including, but not limited to, the systems described herein).

[0056] Figure 3 This is a flowchart illustrating a method 300 according to some embodiments of the present disclosure, which is used to determine whether to use a candidate trajectory to control the navigation of an ego machine, based at least on a measurement of the estimated advantage of an estimated future world state. Method 300 may include at block B302 generating a candidate trajectory using a control stack associated with the ego machine. For example, a control stack associated with the ego machine may be used... Figure 1 or Figure 2 An end-to-end control stack consisting of a perception encoder 110, a state aggregation network 120, and a neural planner 140 is used to generate candidate trajectories 145 from sensor data 105. In some embodiments, a classical planner can be used to generate candidate trajectories.

[0057] Method 300 at box B304 may include: generating a measure of the estimated advantage of the estimated future world state corresponding to the candidate trajectory, based at least on the representation of the candidate trajectory processed using one or more neural networks. For example, it may include... Figure 1 or Figure 2 The future state estimation network 150 and the state evaluator 170 (e.g., trajectory) evaluation network can be used to predict and evaluate the quality, advantage, safety, acceptability, and desirability of the state resulting from taking a candidate trajectory 145 predicted by the neural planner 140 and / or control action. In some embodiments, the upper branch of the control stack 100 and / or control stack 200 may include a classical (e.g., rule-based, heuristic, etc.) planner that generates candidate trajectories, while the lower branch evaluates the candidate trajector generated by the classical planner. In some embodiments, a classical planner or neural planner can generate a trajectory that a classical control algorithm can use to generate an optimized control action, while the lower branch evaluates the trajectory and / or the optimized control action.

[0058] Method 300, at box B306, may include: determining whether to use the candidate trajectory to control the navigation of the ego machine, based at least on a measurement of the estimated advantage of the estimated future world state. For example, Figure 1 or Figure 2 The control component 190 can compare the quality 180 of the predicted future state 160 with specified thresholds. If the quality 180 of the predicted future state 160 meets one or more specified thresholds (e.g., candidate trajectory 145 is predicted to lead to a favorable, safe, desirable, or beneficial future state), the control component 190 can use the candidate trajectory 145 and / or control actions to control the navigation of the self-machine.

[0059] The systems and methods described herein can be used, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, spacecraft, ships, shuttles (e.g., robotaxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones, and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, as examples but not limited to: machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and supervision, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets (e.g., NVIDIA's Omniverse), cloud computing and / or any other suitable application.

[0060] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines, etc.), systems implemented using robots, aviation systems, medical systems, marine systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing language models (such as large language models (LLM), visual language models (VLM), and / or multimodal language models), systems using or deploying one or more inference microservices, systems containing one or more machine learning models deployed in services or microservices and OS-level virtualization packages (e.g., containers), systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0061] Example autonomous or semi-autonomous machines Figure 4AExamples of sensor locations with corresponding fields of view or sensing fields of view of autonomous or semi-autonomous vehicles 400a, autonomous mobile robots (AMRs) 400b, and humanoid robots 400c, according to some embodiments of this disclosure. While three types of machines 400 are shown, this is not intended to be limiting, and the machines 400 described herein may include vehicles, cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police or emergency vehicles, ambulances, boats, construction vehicles, underwater vehicles, robots (e.g., AMRs, humanoid robots, robotic arms, end effectors, forklifts, etc.), drones, aircraft, vehicles coupled to trailers (e.g., semi-trailer tractors for hauling goods), and / or another type of vehicle or machine (e.g., driverless and / or vehicles or machines accommodating one or more passengers). In some cases, vehicle 400a, AMR 400b, humanoid robot 400c, and / or other machine types may be collectively referred to herein as machine 400.

[0062] Regarding Vehicle 400A, autonomous and semi-autonomous vehicles are typically described by automation levels, defined by the National Highway Traffic Safety Administration (NHTSA) (a division of the U.S. Department of Transportation) and the Society of Automotive Engineers (SAE) in their "Classification and Definition of Terms Related to Driving Automation Systems for Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 400 may have one or more functions conforming to Levels 3-5 of autonomous driving. For example, Vehicle 400 may be able to provide driver assistance (Level 1), partial automation (Level 2, 2+, 2++), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the specific implementation. The term “autonomy” as used herein can include any and / or all types of autonomy of Machine 400 or other machines, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, provision of auxiliary autonomy, semi-autonomy, primary autonomy or other specified autonomy.

[0063] about Figure 4AThe sensors and their respective fields of view (not shown for clarity) or sensing fields (not shown for clarity) are an example embodiment and are not intended to be limiting. Although not shown, each sensor may have a corresponding field of view (e.g., a 360-degree field of view around camera 468D, a 180-degree field of view of wide-angle camera 470, a 360-degree sensing field of view of LiDAR sensor 464, etc.). For example, only a subset of the sensors shown may be included, additional sensors may be included, alternative sensors may be included, the number of each sensor mode may be different, the sensor modes may be different (e.g., LiDAR or RADAR may not be included, SONAR, thermal sensors, etc. may be included), and the sensor locations may differ from those shown on vehicles 400a, AMR 400b, and / or humanoid robots 400c, etc. For example, for vehicle 400a, the location, number, modes, and / or other sensor information may differ depending on the type (e.g., SUV, truck, car, robot, motorcycle, etc.), size (e.g., 18-wheeler, transport vehicle, small car, etc.), and associated functions (e.g., L2 vs. L5). Similarly, for the AMR 400b and / or the humanoid robot 400c, the shape, size, purpose, implementation method, model, etc., can determine the number and type of sensors used.

[0064] like Figure 4A As shown, autonomous or semi-autonomous vehicles 400A, AMR 400B, and humanoid robots 400C may include different sensor types, numbers, and locations. As a non-limiting example, vehicle 400A may include twelve cameras 468, such as a front wide-angle camera (e.g., 120-degree field of view (FOV)), a front telephoto camera (e.g., 30-degree FOV), a rear-side left camera (e.g., 70-degree FOV), a rear-side right camera (e.g., 70-degree FOV), a front fisheye camera (e.g., 200-degree FOV), a rear fisheye camera (e.g., 200-degree FOV), a left fisheye camera (e.g., 200-degree FOV), a right fisheye camera (e.g., 200-degree FOV), a front telephoto satellite camera (e.g., 30-degree FOV), a rear telephoto camera (e.g., 30-degree FOV), a cross-left camera (e.g., 120-degree FOV), and a cross-right camera (e.g., 120-degree FOV). In this embodiment, camera 468 may use a gigabit multimedia serial link (GMSL) interface (e.g., GMSL2) as input / output (I / O).

[0065] In some embodiments, although Figure 4AAs not shown, vehicle 400A may include an in-cabin occupant and / or driver monitoring system, which may include various sensors. For example, in-cabin sensors may include various cameras 468, such as a driver monitoring camera (e.g., located in front of the driver's seat at a 55-degree FOV facing the driver's seat), a front occupant monitoring camera (e.g., located in front of the front occupant seat at a 190-degree FOV facing the front occupant seat), and a rear occupant monitoring camera (e.g., located in front of the rear occupant seat at a 190-degree FOV facing the rear occupant seat). Similar to external cameras 468, in embodiments, internal cameras 468 may use a GMSL (e.g., GMSL2) interface for I / O.

[0066] As another non-limiting example, vehicle 400A may also include nine RADAR sensors 460. For example, vehicle 400A may include a front center imaging RADAR sensor (e.g., 120-degree FOV or sensing field), a left front corner RADAR sensor (e.g., 160-degree FOV or sensing field), a right front corner RADAR sensor (e.g., 160-degree FOV or sensing field), a right rear corner RADAR sensor (e.g., 160-degree FOV or sensing field), a left RADAR sensor (e.g., 160-degree FOV or sensing field), a right RADAR sensor (e.g., 160-degree FOV or sensing field), a left rear RADAR sensor (e.g., 50-degree FOV or sensing field), and a right rear RADAR sensor (e.g., 50-degree FOV or sensing field). In embodiments, the RADAR sensors 460 may use an Ethernet interface as I / O.

[0067] As a non-limiting example, the vehicle 400A may also include twelve ultrasonic sensors 462. Figure 4A As shown, the ultrasonic sensor can be placed along the front and rear bumpers of vehicle 400A and along the sides of vehicle 400A, and can be used to detect objects (static and dynamic) close to vehicle 400A. In some embodiments, the ultrasonic sensor 462 can use a DS13 interface as I / O.

[0068] As a non-limiting example, the vehicle 400A may also include a LiDAR sensor 464, such as a front-center LiDAR sensor (e.g., a 120-degree horizontal FOV or sensing field and a 30-degree vertical FOV or sensing field). In some embodiments, such as when using additional or alternative LiDAR sensors, the LiDAR sensor may have different horizontal and vertical fields of view or sensing fields. For example, the LiDAR sensor 464 may include a 360-degree horizontal FOV or sensing field (e.g., located in a rotating LiDAR sensor) and a 90-degree vertical FOV or sensing field. In some embodiments, the LiDAR sensor 464 may use an Ethernet interface as I / O.

[0069] As a non-limiting example, the autonomous mobile robot (AMR) 400B may include three LiDAR sensors 464. For example, the topmost LiDAR sensor 464 may include a beam or 3D LiDAR sensor (e.g., a 360-degree horizontal and 90-degree vertical FOV or sensing field), and the front and rear LiDAR sensors may include planar or 2D LiDAR sensors (e.g., a 180-degree horizontal FOV or sensing field).

[0070] As a non-limiting embodiment, the AMR 400B may further include eight cameras 468, such as a front stereo camera (e.g., 120-degree FOV), a rear stereo camera (e.g., 120-degree FOV), a left stereo camera (e.g., 120-degree FOV), a right stereo camera (e.g., 120-degree FOV), a front fisheye camera (e.g., 202-degree ± 3-degree FOV), a rear fisheye camera (e.g., 202-degree ± 3-degree FOV), a left fisheye camera (e.g., 202-degree ± 3-degree FOV), and a right fisheye camera (e.g., 202-degree ± 3-degree FOV).

[0071] The AMR 400B may also include a charging port, charging port contacts, status indicators, one or more (e.g., four) RGB LEDs, one or more IMU sensors 466, a magnetometer, and a barometer. The AMR 400B is capable of high-precision time synchronization between sensors using hardware timestamps and PTPs over Ethernet with sensor acquisition times of less than 10 microseconds. In an embodiment, the AMR 400B provides simultaneous camera capture within 100 microseconds of a single hardware trigger on all cameras 468, and can write sensor captures to disk at a rate of 4 GB / s for writing packets (e.g., writing to ROSbags of the Robot Operating System (ROS)). Therefore, the AMR 400B is capable of running ROS (e.g., NVIDIA's Isaac ROS), can be remotely operated (as described herein), can map the environment, and can navigate the environment using vision cameras 468, LiDAR 464, and / or other sensor types or modalities.

[0072] The humanoid robot 400C may include (as a non-limiting example) a LiDAR sensor 464. For example, the LiDAR sensor 464 may include a beam or a 3D LiDAR sensor (e.g., a 360-degree horizontal and 90-degree vertical FOV or sensing field), or it may include a planar or 2D LiDAR sensor (e.g., a 180-degree horizontal FOV or sensing field).

[0073] As a non-limiting embodiment, the humanoid robot 400C may also include four cameras 468, such as a front stereo camera (e.g., 120-degree FOV), a rear stereo camera (e.g., 120-degree FOV), a front fisheye camera (e.g., 202-degree ± 3-degree FOV), and a rear fisheye camera (e.g., 202-degree ± 3-degree FOV).

[0074] As a non-limiting embodiment, the humanoid robot 400C may also include four ultrasonic sensors 462, such as a left arm ultrasonic sensor, a right arm ultrasonic sensor, a left leg ultrasonic sensor, and a right leg ultrasonic sensor.

[0075] The humanoid robot 400C may also include any number of actuators, such as those allowing control and manipulation of joints. For example, the humanoid robot 400C may include actuators that allow for various degrees of freedom (DoF) depending on the design. In a non-limiting embodiment, the humanoid robot 400C may have a total of 40 degrees of freedom (DoF) (e.g., 6DoF x2 for arms, 6DoF x2 for hands, 6DoF x2 for legs, 2DoF for the torso, and 2DoF for the neck). Actuators can convert energy into physical motion, thereby allowing actions such as joint movement, locomotion, and grasping / manipulation. For example, motors and servos can be used to perform joint movements to control the rotation of joints in an arm or manipulator and allow for reaching, grasping, and manipulating objects. Locomotion can be achieved by moving through the environment using wheels, tracks, or other locomotion devices (robot legs). Grasping and manipulation can be performed using end effectors or hands / fingers, which can be equipped with actuators to grasp objects, apply forces, and perform specific tasks. In some examples, the humanoid robot 400C may include position and orientation sensors, such as encoders, gyroscopes, etc., to determine the robot 400C's position in space, thereby enabling position determination and motion tracking. In embodiments, the humanoid robot 400C may include force and pressure sensors to detect environmental interactions, enabling the robot 400C to grasp objects with appropriate force and avoid obstacles along its path. Perception sensors (e.g., cameras, LiDAR, RADAR, ultrasound, sonar (SONAR), etc.) may be used in conjunction with tactile sensors to enable the robot 400C to perceive objects, shapes, and textures, and to know when to begin and stop touching (in conjunction with force sensors that regulate the force used during touching). As a non-limiting example, the humanoid robot 400C may have a height of approximately 1-2 meters (e.g., 1.7 meters or 5 feet 6 inches), a weight of 50-70 kilograms, be able to move at speeds of 8 km / h or higher, and be able to carry a payload of 20-100 kilograms, depending on the system design and requirements.

[0076] In embodiments, the humanoid robot 400C may include a dialogue system—such as a dialogue system driven by a language model (e.g., LLM, VLM, MMLM, VLA, etc.)—to help understand the environment, reason, and communicate with humans, animals, devices, and / or other robots, and / or make planning, control, and navigation decisions. Therefore, in addition to performing various tasks, the humanoid robot 400C can also use onboard sensors, microphones, and speakers to understand speech, audio, and visual cues, while also being able to communicate with the environment.

[0077] Referring to a camera 468 of one or more machines 400, the camera type of camera 468 may include, but is not limited to, a digital camera suitable for components and / or systems of machine 400. For the vehicle 400a implementation, camera 468 may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may have any image capture rate, such as 30 frames per second (fps), 60 fps, 120 fps, 240 fps, etc., depending on the embodiment. The camera may use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red colorless colorless colorless (RCCC) color filter array, a red colorless colorless blue (RCCB) color filter array, a red blue green colorless (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a colorless pixel camera (e.g., a camera with an array of RCCC, RCCB, and / or RBGC color filters) can be used to improve light sensitivity.

[0078] The field of view includes cameras (e.g., front-facing cameras) in the area in front of the machine 400, which can be used for surround view to help identify the path and obstacles ahead, and, with the help of one or more controllers 436 and / or control SoCs, provide information crucial for generating an occupancy grid and / or determining preferred machine movement, trajectory, and / or path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including lane departure warning (“LDW”), adaptive cruise control (“ACC”), and / or other functions such as traffic sign recognition.

[0079] Various cameras can be used in front-end configurations, including, for example, monocular camera platforms that include complementary metal-oxide-semiconductor (“CMOS”) color imagers. Another example could be a wide-angle camera 468B, which can be used to perceive objects entering the field of view from the periphery (e.g., pedestrians, warehouse vehicles, other robots, pedestrian traffic, or bicycles). Furthermore, any number of long-range cameras 468E (e.g., a pair of long-range stereo cameras) can be used for depth-based object detection, especially for objects for which neural networks have not yet been trained. Long-range cameras 468E can also be used for object detection and classification, as well as basic object tracking.

[0080] Any number of stereo cameras 468A may also be included in front-mounted and / or other (e.g., rear-mounted) configurations. In at least one embodiment, one or more stereo cameras 468A may include an integrated control unit comprising a scalable processing unit that may provide programmable logic (“FPGA”) and a multi-core microprocessor with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the machine 400 environment, including distance estimates of midpoints in the image (e.g., parallax or depth images). Alternative stereo cameras 468A may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip that can measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 468A may be used in addition to those described herein, or as an alternative to those described herein. For example, in some embodiments, cameras other than stereo cameras (e.g., two monocular cameras with at least partially overlapping fields of view) may be used to perform stereo depth estimation.

[0081] Cameras with a field of view including portions of the side environment of machine 400 (e.g., side-view cameras) can be used, for example, for surround view, providing information for creating and updating occupancy grids, and generating side collision warnings and / or indicating to AMR 400B or humanoid robot 400C, for example, the presence of objects, features, and / or people on the side. For example, surround cameras 468D can be mounted on machine 400. Surround cameras 468D can include wide-angle cameras 468B, fisheye cameras, 360-degree cameras, etc. For example, four fisheye cameras can be mounted on the front, rear, and sides of machine 400. In an alternative arrangement, machine 400 can use three surround cameras 468D (e.g., left, right, and rear), and can utilize one or more other cameras (e.g., front-facing cameras) as a fourth surround camera.

[0082] A camera 468 (e.g., a rear-view camera) having a field of view including a portion of the environment behind the machine 400 can be used to understand objects, features, people, and / or other information behind the machine 400, such as for parking assistance, surround view, rear collision warning, planning, control, and navigation determination, and / or to create and update occupancy grids, BEV images representing the environment, height maps, etc. A wide variety of cameras 465 can be used, including but not limited to those also suitable for use as front cameras (e.g., long-range and / or mid-range cameras 468E, stereo cameras 468A, infrared cameras 468C, etc.), rear cameras, side cameras, downward cameras, upward cameras, and / or similar cameras 468, as described herein.

[0083] Similarly, for LiDAR sensor 464, RADAR sensor 460, ultrasonic sensor 462 and / or other sensor modes or types, the location and placement of the sensors and their corresponding fields of view or sensing fields can be determined based on the use case, implementation or design of the particular machine 400.

[0084] For example, machine 400 includes a RADAR sensor 460, which can be used by machine 400 for long-range object detection, even in dark and / or inclement weather conditions. In embodiments, the RADAR functional safety level may be ASIL B. RADAR sensor 460 can be controlled and accessed for object tracking data using CAN and / or bus 402 (e.g., for transmitting data generated by RADAR sensor 460), and in some examples, raw data can be accessed using Ethernet. Various types of RADAR sensors can be used. For example, but not limited to, RADAR sensor 460 can be suitable for front, rear, and side radar applications. In some examples, a pulse Doppler RADAR sensor is used.

[0085] The RADAR sensor 460 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range lateral coverage, etc. In some examples, long-range radar can be used for adaptive cruise control (ACC) functions. A long-range RADAR system can provide a wide field of view, for example, within a 250m range, achieved through two or more independent scans. The RADAR sensor 460 can help distinguish between static and moving objects; ADAS systems can use it for emergency braking assistance and forward collision warning, and robots can use it to detect dynamic objects in various environments (e.g., low-light or no-light environments). A long-range RADAR sensor can include a single static multimodal RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record the machine 400's surroundings at higher speeds while minimizing peripheral interference (e.g., traffic from adjacent lanes). The other two antennas expand the field of view, enabling rapid detection of objects entering or leaving the machine's direct path (e.g., lanes).

[0086] Mid-range RADAR systems can include, for example, a range of up to 460m (front) or 50m (rear), and a field of view of up to 42 degrees (front) or 150 degrees (rear). Short-range RADAR systems can include, but are not limited to, RADAR sensors designed to be mounted at both ends of a side surface (e.g., rear bumper), allowing continuous monitoring of blind spots behind and beside a machine 400 (e.g., vehicle, robot, etc.) using two beams. Therefore, short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0087] Machine 400 may also include an ultrasonic sensor 462. The ultrasonic sensor 462 may be located at the front, rear, and / or sides of machine 400 and may be used to assist near-field sensing, such as for parking assistance, collision avoidance (e.g., for robot parts), and / or to create and update occupancy grids, evidence mesh maps (EGMs), height maps, BEV images, and / or other representations of objects and features in the machine 400 environment. Multiple ultrasonic sensors 462 may be used, and different ultrasonic sensors 462 may be used for different detection ranges (e.g., 2.5 m, 4 m). For example, the ultrasonic sensor 462 may operate at an ASIL B functional safety level.

[0088] Machine 400 may include LiDAR sensor 464. LiDAR sensor 464 may be used for object and feature detection, pedestrian and other robot detection, emergency braking, collision avoidance, simultaneous localization and mapping (SLAM), free space detection, and / or other functions. In embodiments, one or more LiDAR sensors 464 may be of functional safety level ASIL B. In some examples, machine 400 may include multiple LiDAR sensors 464 (e.g., two, four, six, etc.), which may use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0089] In some examples, the LiDAR sensor 464 may be able to provide a list of objects and their distances within a 360-degree field of view. For example, a commercially available LiDAR sensor 464 may have a advertised range of approximately 900m, an accuracy of 2cm-3cm, and support for a 400Mbps Ethernet connection. In some examples, one or more non-protruding LiDAR sensors 464 may be used. In such examples, one or more LiDAR sensors 464 may be implemented as small devices that can be embedded in the front, rear, sides, top, and / or corners of a machine 400. In such examples, one or more LiDAR sensors 464 may provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, with a range of up to 200m even for low-reflectivity objects. One or more front-mounted LiDAR sensors 464 may be configured with a horizontal field of view between 45 and 135 degrees.

[0090] In some examples, LiDAR technology, such as 3D flash LiDAR, can also be used. 3D flash LiDAR uses a laser flash as a transmission source, illuminating the environment around a vehicle up to approximately 200m away. The flash LiDAR unit includes a receiver that records the laser pulse transmission time and the reflected light on each pixel, which in turn corresponds to the distance from the vehicle to the object. Flash LiDAR allows for the generation of highly accurate and distortion-free images of the surrounding environment with each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of machine 400. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras with no moving parts other than a fan (e.g., non-scanning LiDAR devices). Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of 3D distance point clouds and co-registration intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, LiDAR sensor 464 may be less susceptible to motion blur, vibration, and / or shock.

[0091] Figure 4BThis is an illustration of the location of sensors and components of an example autonomous or semi-autonomous vehicle 400A (also referred to herein as "vehicle 400", "self-vehicle 400", "self-machine 400", or "machine 400") according to some embodiments of this disclosure. While vehicle 400A is shown in the figure, this is not intended to be limiting, and similar components and / or sensors may be included on any other machine type without departing from the scope of this disclosure. For example, similar sensors and / or components may be used in vehicles, cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police cars, ambulances, ships, construction vehicles, underwater vehicles, robots (e.g., AMRs, humanoid robots, robotic arms, end effectors, forklifts, etc.), drones, aircraft, vehicles coupled to trailers (e.g., semi-trailers for hauling goods), and / or another type of vehicle or machine (e.g., driverless and / or capable of accommodating one or more passengers).

[0092] Figure 4CThis is a block diagram of an example system architecture for a machine 400 (e.g., an autonomous or semi-autonomous vehicle 400A, an autonomous mobile robot (AMR) 400B, a humanoid robot 400C, and / or other types of machines) according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are presented as examples only. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of the arrangements shown, and some elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components or together with other components, and can be implemented in any suitable combination and location (e.g., on a local device, vehicle, or edge machine, in the field (e.g., a locally hosted server), in a remote location (e.g., in one or more computing or server devices in one or more data centers in the cloud) and / or other locations). The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, microcontroller, embedded processor, digital signal processor (DSP), image signal processor (ISP), physical processing unit (PPU), field-programmable gate array (FPGA), accelerators (e.g., deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural network accelerator (NNA) and / or neural processing unit (NPU), programmable vision accelerator (PVA), optical flow accelerator (OFA), etc.), application-specific integrated circuit (ASIC), data processing unit (DPU), quantum processor, etc.) executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 4A to 4E Example machine 400 Figure 5 Example calculation of ecosystem 500, Figure 6 Example Generative Language Model System 600 and / or Figure 7 The example computing device 700 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the task.

[0093] Figure 4CEach component, feature, and system of machine 400 is shown connected via bus 402 (or referred to as "machine communication network 402" or simply "communication network 402"). Bus 402 may include a Controller Area Network (CAN) data interface (or referred to herein as "CAN bus"). CAN can be a network within machine 400 used to help control various features and functions of machine 400, such as brake actuation, acceleration, braking, steering, windshield wipers, etc. CAN bus can be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CANID). CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button position, and / or other vehicle status indicators. CAN bus may conform to ASIL B standard. In some embodiments, in addition to or as an alternative to the CAN bus, bus 402 may include FlexRay, embedded buses (e.g., SPI, I2C), local interconnect links (LIN), NVIDIA's NVLink, USB (2.0, 3.0 and above), radio frequency (RF), Ethernet (e.g., 10BASE / 100BASE, 1000BASE, 10G, etc.), and / or other communication protocols or functions. Furthermore, while a single line is used to represent bus 402, this is not limiting. For example, there can be any number of buses 402, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 402 may be used to perform different functions and / or for redundancy. For example, a first bus 402 may be used for a collision avoidance function, while a second bus 402 may be used for actuation control. In any example, each bus 402 may communicate with any component of machine 400, and two or more buses 402 may communicate with the same component. In some examples, each computer or computing engine within each SoC 404, each controller 436, and / or machine 400 can access the same input data (e.g., input from sensors in the machine 400) and can be connected to a common bus, such as a CAN bus.

[0094] Machine 400 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, batteries, side mirrors, and / or other parts of the vehicle or machine. Machine 400 may include a propulsion system 450, such as an internal combustion engine, a hybrid power plant, an all-electric motor, a hydrogen fuel cell engine, and / or another type of propulsion system. Propulsion system 450 may be connected to the drivetrain of machine 400, which may include a transmission to enable propulsion of machine 400. Propulsion system 450 may be controlled in response to a signal received from throttle / accelerator 452.

[0095] Steering system 454 may include a steering wheel and / or other steering mechanisms (e.g., remote steering and / or local steering) for steering machine 400 (e.g., along a desired path or route) while propulsion system 450 is in operation (e.g., when the vehicle is traveling). Steering system 454 may receive signals from steering actuator 456. In some embodiments, a steering wheel or other steering mechanism may be omitted, for example, for machine 400 capable of fully automated (e.g., level 5) functionality.

[0096] The brake sensor system 446 can be used to operate the vehicle brakes in response to signals received from the brake actuator 448 and / or the brake sensor.

[0097] Machine 400 may include one or more controllers 436, such as those described herein. Figure 4AThe controller 436 is described. It can be used for a variety of functions and can be coupled to any of the various other components and systems of the machine 400. For example, controller 436 can be used to control the machine 400, artificial intelligence performed on the machine 400, infotainment on the machine 400, etc. For example, one controller 436 can be used for some or all of the functions, or different controllers 436 can be used for different functions, for example, to ensure availability and safety separation between various controllers for different tasks. For example, controller 436 can use a system-calculated plan (e.g., the path or trajectory of vehicle 400A or AMR 400B, or the movement, component trajectory, motion position or displacement, etc. of joints or components of humanoid robot 400C (e.g., manipulators, end effectors, limbs, hands, fingers, legs, feet, etc.) to control the machine 400 in the environment. In some cases, controller 436 may include a proportional-integral-derivative (PID) controller, a fuzzy logic controller, a neural controller (e.g., a controller embodied as one or more neural networks), a force control controller, a programmable logic controller (PLC), and / or other types of controllers. For example, in the humanoid robot 400C, the controller 436 can act as the brain, responsible for analyzing sensor data, making decisions, and sending commands to the actuators. The controller 436 may include a low-level controller that handles basic motor control, ensuring accurate and precise movement of the individual joints and actuators. The controller 436 may also include a high-level controller to coordinate multiple actuators and sensors, plan complex movements, and adapt to changing environments.

[0098] In some embodiments, controller 436 may include an artificial intelligence controller that can use AI algorithms (e.g., DNN, MLM, etc.) to learn, make decisions, and autonomously perform tasks of machine 400. In some embodiments, controller 436 may use a fixed open-loop control algorithm and not adjust its actions according to the environment. In other embodiments, closed-loop control may be used, which incorporates a feedback mechanism to monitor the robot's performance and make necessary adjustments. In examples, controller 436 may implement reactive control to directly respond to sensor inputs, enabling rapid reflexive actions and real-time changes. Furthermore, in some examples, deliberate control may be implemented, using internal models and planning algorithms to generate advanced actions, which may be suitable for complex tasks requiring reasoning, decision-making, and long-term planning.

[0099] Controller 436 may include one or more system-on-chip (SoC) 404 ( Figure 4C and Figure 4DController 436 may provide signals (e.g., signals representing commands or messages) to one or more components and / or systems of machine 400, including CPUs, GPUs, and accelerators. Although controller 436 is listed separately from SoC 404, this is not intended to be limiting, and in some embodiments, one or more components of SoC 404 may perform the operations of controller 436. For example, the controller may send signals to operate machine brakes via one or more brake actuators 445, to operate steering system 454 via one or more steering actuators 456, to operate propulsion system 450 via one or more throttle / accelerators 452, and so on. One or more controllers 436 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous or semi-autonomous navigation and movement and / or assist human operators using machine 400. One or more controllers 436 may include a first controller 436 for autonomous control and navigation functions, a second controller 436 for functional safety functions, a third controller 436 for artificial intelligence functions (e.g., computer vision), a fourth controller 436 for infotainment functions, a fifth controller 436 for redundancy in emergency situations, and / or other controllers. For example, hardware for safety monitoring and other safety functions (e.g., functional safety islands) may be discrete or partitioned (physically or by processing separation) relative to hardware for processing sensor data to make perception and vehicle control decisions. Similarly, hardware for controlling in-vehicle infotainment and / or in-cabin monitoring (e.g., controllers, SOCs, etc.) may be separate or independent from hardware for vehicle perception and control. In some examples, a single controller 436 may handle two or more of the above functions, two or more controllers 436 may handle a single function, and / or any combination thereof.

[0100] Controller 436 may provide signals for controlling one or more components and / or systems of machine 400 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example, but not limited to, one or more Global Navigation Satellite System (“GNSS”) sensors 458 (e.g., Global Positioning System sensors), one or more RADAR sensors 460, one or more ultrasonic sensors 462, one or more LiDAR sensors 464, one or more Inertial Measurement Unit (IMU) sensors 466 (e.g., one or more accelerometers, one or more gyroscopes, one or more magnetic compasses, one or more magnetometers, etc.), one or more microphones 496, one or more cameras 468 (e.g., one or more... Stereo camera 468A, one or more wide-angle cameras 465B (e.g., fisheye cameras), one or more infrared cameras 468C, one or more surround cameras 468D (e.g., 360-degree cameras), one or more long-range and / or medium-range cameras 468E and / or other types of cameras, speed sensor 444 (e.g., for measuring the speed of machine 400), one or more vibration sensors 442, one or more steering sensors 440, one or more braking sensors (e.g., as part of braking sensor system 446), actuators and / or other types of sensors receive the sensor.

[0101] One or more controllers 436 may receive input (e.g., represented by input data) from the instrument panel 432 of the machine 400 and provide output (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 434 (e.g., screen, head-up display, mirror display, facial display, robot display, etc.), sound alarms, loudspeakers, speakers, and / or other components of the machine 400. Outputs may include parameters such as machine speed, rate, time, and... Figure 4C The system may display map data corresponding to map 422 (e.g., from a navigation map, a standard definition (SD) map, a high definition (“HD”) map, etc.), location data (e.g., the location of machine 400, such as its location on map 422), direction, the location of other vehicles (e.g., occupancy map, elevation map, bird's-eye view (BEV) image, grid, etc.), information about objects perceived by the system and their states, system status information, and so on. For example, one or more HMI displays 434 may display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.), and / or information about driving maneuvers that the vehicle has performed, is performing, or will perform (e.g., changing lanes now, exiting from exit 34B in two miles, etc.).

[0102] Machine 400 may include one or more System-on-a-Chip (SoC) 404 (in Figure 4D (described in more detail below). SoC 404 may include one or more CPUs 406, one or more GPUs 408, one or more processors 410, one or more caches 412, one or more accelerators 414, one or more data repositories 416, and / or other components and features. One or more SoCs 404 can be used to process and provide data for various operations, such as navigation, planning, reasoning, inference, perception, control, and / or actuation operations of machine 400 on various platforms and systems. For example, SoC 404 can process real-time perception data (e.g., from cameras, LiDAR, RADAR, ultrasound, etc.) and map data corresponding to one or more maps 422 (e.g., HD maps, SD maps, navigation maps, occupancy maps, etc.) to perform or assist in performing various operations of machine 400. When using maps and / or AI, maps and / or AI (e.g., model parameter updates, fine-tuning, etc.) are transmitted via network interface 424 from one or more servers (e.g., [server name missing]). Figure 4E Refresh and / or update the server 478 (e.g., one or more servers in a cloud-based data center).

[0103] Despite Figures 4A to 4E SoC 404 is shown, however, additional or alternative components and / or architectures, such as multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiPs), field-programmable gate arrays (FPGAs), heterogeneous integration (HI), and single-board computers (SBCs), may be used without departing from the scope of this disclosure. For example, depending on the type of machine 400, the purpose of machine 400, the model of machine 400, and the capabilities required by machine 400, one or more SoCs 404 and / or alternative architectures and / or components may be used to meet a particular implementation.

[0104] Machine 400 may include one or more CPUs 418 (e.g., discrete CPUs or dCPUs) that may be coupled to SoC 404 via a high-speed interconnect (e.g., PCIe). The one or more CPUs 418 may include, for example, x86 processors. The one or more CPUs 418 may be used to perform any of a variety of functions, including arbitrating potentially inconsistent results between ADAS sensors and SoC 404, and / or monitoring the status and health of controller 436 and / or infotainment SoC 430.

[0105] Machine 400 may include one or more GPUs 420 (e.g., discrete GPUs or dGPUs) that may be coupled to one or more SoCs 404 via a high-speed interconnect (e.g., NVIDIA's NVLink). One or more GPUs 420 may provide additional artificial intelligence capabilities, such as by executing redundant and / or different neural networks, and may be used to train and / or update neural networks based on inputs from sensors (e.g., sensor data) from machine 400.

[0106] The machine 400 may also include one or more network interfaces 424, which may include one or more wireless antennas 426 and / or modems (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 424 can be used to establish wireless connectivity with the cloud (e.g., with one or more servers 478 and / or other network devices) via the Internet, and with other vehicles and / or computing devices (e.g., passenger client devices). For communication with other vehicles, direct links and / or indirect links (e.g., across networks and via the Internet) can be established between the two vehicles. A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide the machine 400 with information about vehicles near the machine 400 (e.g., vehicles in front, to the side, and / or behind the machine 400). This functionality can be part of the machine 400's cooperative adaptive cruise control function.

[0107] Network interface 424 may include a System-on-Chip (SoC) that provides modulation and demodulation functions and enables one or more controllers 436 to communicate over a wireless network. Network interface 424 may include a radio frequency (RF) front-end for up-conversion from baseband to radio frequency (RF) and down-conversion from RF to baseband. Frequency conversion can be performed by well-known processes and / or using superheterodyne processes. In some examples, the RF front-end functionality may be provided by a separate chip. For example, network interface 424 may be able to communicate via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multicarrier (“CDMA2000”), 5G, 6G, and / or other cellular and / or wireless communication standards. One or more wireless antennas 426 can also enable communication between objects in the environment (such as vehicles, mobile devices, etc.) using local area networks (such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc.) and / or low power wide area networks (“LPWAN”) (such as LoRaWAN, SigFox, etc.).

[0108] Machine 400 may also include one or more data memories 428, which may include off-chip (e.g., storage outside of one or more SoCs 404) storage. One or more data memories 428 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, Flash, hard disk and / or other components and / or devices capable of storing at least one bit of data.

[0109] The machine 400 may also include one or more GNSS sensors 458. One or more GNSS sensors 458 (e.g., GPS, assisted GPS sensors, differential GPS (DGPS) sensors, etc.) are used to assist mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 458 can be used, such as, but not limited to, GPS with a USB connector having an Ethernet-to-serial (RS-232) bridge.

[0110] Machine 400 may also include one or more IMU sensors 466. In some examples, one or more IMU sensors 466 may be located at the center of the rear axis of machine 400. One or more IMU sensors 466 may include, for example, but not limited to, one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more magnetic compasses, and / or other sensor types. In some examples, such as in a six-axis application, one or more IMU sensors 466 may include accelerometers and gyroscopes, while in a nine-axis application, one or more IMU sensors 466 may include accelerometers, gyroscopes, and magnetometers.

[0111] In some embodiments, one or more IMU sensors 466 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, one or more IMU sensors 466 can enable the machine 400 to estimate heading by directly observing and correlating velocity changes from GPS to one or more IMU sensors 466, without requiring input from a magnetic sensor. In some examples, IMU sensors 466 and one or more GNSS sensors 458 can be combined in a single integrated unit.

[0112] The vehicle may include one or more microphones 496 placed inside and / or around the machine 400. The one or more microphones 496 may be used for emergency vehicle detection and identification, etc.

[0113] Machine 400 may also include one or more vibration sensors 442. One or more vibration sensors 442 can measure vibrations of machine parts, such as the arm or leg of humanoid robot 400C, or the axle of vehicle 400A or AMR 400B. For example, changes in vibration may indicate changes in roads, walking, or traversable surfaces. In another example, when two or more vibration sensors 442 are used, differences between vibrations can be used to determine friction or slippage on surfaces (e.g., when the vibration difference is between an electrically driven shaft and a freely rotating axle).

[0114] Machine 400 may include ADAS system 438, for example, when machine 400 is vehicle 400A. In some examples, ADAS system 438 may include a dedicated SoC. ADAS system 438 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision or collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), blind spot monitoring (BSM), rear cross traffic warning (RCTW), pedestrian detection, driver monitoring, collision warning system (CWS), traffic sign recognition, speed limit detection, automatic parking, lane centering (LC), high beam safety system and / or other features and functions.

[0115] Machine 400 may also include an infotainment SoC 430 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include one or more discrete components, such as multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiP), heterogeneous integration (HI), single-board computers (SBCs), etc. The infotainment SoC 430 may include a combination of hardware and software that can be used to provide the machine 400 with audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., television, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., wireless, Wi-Fi, etc.) and / or information services (e.g., navigation systems, rear parking assist, radio data systems, vehicle-related information (e.g., fuel level, total driving distance, brake fluid level, fuel level, door opening / closing, air filter information, etc.). For example, an infotainment SoC 430 may be a radio, disk player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, Wi-Fi, steering wheel audio controls, hands-free voice controls, head-up display (HUD), HMI display 434, telematics device, control panel (e.g., for controlling various components, features and / or systems and / or interacting with various components, features and / or systems) and / or other components. The infotainment SoC 430 may also be used to provide information to vehicle users (e.g., visual and / or auditory), such as information from ADAS system 438, autonomous driving information (e.g., planned vehicle maneuvers, trajectory), surrounding environment information (e.g., intersection information, vehicle information, road information, etc.) and / or other information.

[0116] The infotainment SoC 430 may include GPU functionality. The infotainment SoC 430 can communicate with other devices, systems, and / or components of the machine 400 via bus 402 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 430 may be coupled to a monitoring MCU, enabling the GPU of the infotainment system to perform some autonomous driving functions in the event of a failure of the main controller 436 (e.g., the main computer and / or backup computer of the machine 400). In such an example, the infotainment SoC 430 may place the machine 400 into a driver-safe stop mode, as described herein.

[0117] In some embodiments, the infotainment system may provide a digital or virtual assistant, which may be voice-only or may have visual components (e.g., in the form of a digital human or digital avatar). The assistant may provide basic functions such as sending text messages, adjusting vehicle settings, controlling music or video, navigation, etc., and / or more advanced functions, such as those supported by one or more language models (e.g., Large Language Model (LLM), Visual Language Model (VLM), Multimodal Language Model (MMLM), etc.). For example, the driver and / or occupants may interact with the assistant in a manner similar to how a user interacts with a language model, such as asking general questions, specific questions, requesting restaurants, gas stations, and / or other recommendations and / or locations, inquiring about vehicle functions or troubleshooting (e.g., asking for tire pressure information, oil change information, battery swap information, etc.). Therefore, the machine 400 (whether it is a vehicle 400A, AMR 400B, humanoid robot 400C, or other type of machine) may include locally stored language models and / or communicate with remotely hosted language models (e.g., via one or more APIs) to provide users of the machine 400 with more detailed and in-depth communication capabilities.

[0118] In some examples, the infotainment SoC 430, SoC 404, and / or another SoC or computing / processing system can perform in-cabin driver and / or occupant monitoring. For example, the computing system can perform facial recognition, and vehicle owner recognition can use data from cameras and / or other sensors to identify the presence of an authorized driver and / or owner of the machine 400. An always-on sensor processing engine can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in a security mode, to disable the vehicle when the owner leaves. In this way, one or more SoCs 404 can provide security against theft and / or hijacking.

[0119] In some embodiments, one or more neural networks running on another or a dedicated SoC (e.g., an in-vehicle infotainment or in-vehicle monitoring SoC) can be used to monitor in-cabin monitoring camera sensors. This other or dedicated SoC is configured to recognize in-cabin events and respond accordingly. The in-cabin system can activate cellular services and make phone calls via lip reading, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. The in-cabin system may also include one or more in-cabin AI agents or assistants that can interact with one or more LLMs, VLMs, MMLMs, etc., in the cloud using one or more APIs or plugins. For example, the in-cabin AI agents or assistants can provide directions, vehicle or machine feedback information, answer general questions, handle music / video and / or other requests, activate windows, doors, and / or other vehicle components, etc. Therefore, one or more dedicated SoCs and / or processor groups can be used to perform in-cabin infotainment and / or in-cabin monitoring (e.g., as an occupant monitoring system (OMS)) of the machine 400.

[0120] Machine 400 may also include an instrument panel 432 (e.g., a digital instrument panel, electronic instrument panel, digital dashboard, etc.). Instrument panel 432 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). Instrument panel 432 may include a set of instruments such as a speedometer, fuel level, fuel pressure, tachometer, odometer, turn signal indicator, gearshift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 430 and instrument panel 432. In other words, instrument panel 432 may be included as part of infotainment SoC 430, and vice versa.

[0121] Figure 4D A computing system according to at least some embodiments of this disclosure (about Figure 4C A block diagram of an example architecture (a subset of the systems described). Although illustrated as SoC 404, this is not intended to be limiting, and the computing system may additionally or alternatively include multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiP), heterogeneous integration (HI), single-board computers (SBCs), and / or other components and / or architectures without departing from the scope of this disclosure.

[0122] SoC 404 can be an end-to-end platform with a flexible architecture spanning automation levels 2-5, or SoC 404 can be specifically designed for a particular automation level (e.g., the first SoC 404 for levels 2 to 2++, the second SoC 404 for level 3, the third SoC 404 for level 4, and so on), thereby providing a comprehensive functional safety architecture that leverages and effectively utilizes computer vision, neural network inference, robot planning, control and navigation, ADAS technologies, etc., with diversity and redundancy to provide a platform for a flexible and reliable driving or robot control software stack, as well as deep learning tools. One or more SoC 404s can be faster, more reliable, and even more energy-efficient and space-saving than traditional systems. For example, when one or more accelerators 414 are used in conjunction with one or more CPUs 406, one or more GPUs 408, and one or more data repositories 416, they can provide a fast and efficient platform for Level 2-5 autonomous vehicles, as well as for the safety planning, navigation, and control of AMR 400B, humanoid robot 400C, and / or other robot or machine types.

[0123] In some embodiments, for example, SoC 404 includes: a GPU 408 having 2000 or more cores (e.g., 2048 cores), 60 or more tensor cores (e.g., 64 tensor cores), and a maximum GPU frequency exceeding 1 GHz (e.g., 1.3 GHz); a CPU 406 including 10 or more cores (e.g., 12 cores), having 64-bit, 3 MB L2 and 6 MB L3 caches, and a maximum frequency of 2 GHz or more (e.g., 2.2 GHz); one or more deep learning accelerators (DLAs); one or more deep learning accelerator clusters (XNNs); one or more neural network accelerators (NNAs) or one or more neural processing units (NPUs) 409 (e.g., 2 DLAs / XNNs / NNAs / NPUs 409); and a vision accelerator (e.g., a programmable vision accelerator (PVA)) 407. A single SoC 404 may be capable of achieving AI performance of 275 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGX. The Orin 64GB SoC meets these standards and achieves this level of performance.

[0124] Similarly, in the embodiments, one or more SoCs 404 include: a GPU 408 having 1700 or more cores (e.g., 1792 cores), 50 or more tensor cores (e.g., 56 tensor cores) and a maximum GPU frequency exceeding 900MHz (e.g., 930MHz); a CPU 406 including 8 or more cores (e.g., 8 cores), having 64-bit, 2MB L2 and 4MB L3 cache memory, and a maximum frequency of 2GHz or more (e.g., 2.2GHz); one or more deep learning accelerators (DLAs); deep learning accelerator clusters (XNNs); neural network accelerators (NNAs) or one or more neural processing units (NPUs) 409 (e.g., 2 DLAs / XNNs / NNAs / NPUs 409), and a vision accelerator (e.g., a programmable vision accelerator (PVA) 407), and a single SoC 404 may be capable of achieving AI performance of 200 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGX Orin 32GB SoC meets these standards and achieves such performance.

[0125] In some embodiments, for example, one or more SoCs 404 include: a GPU 408 having 1,000 or more cores (e.g., 1,024 cores), 28 or more tensor cores (e.g., 32 tensor cores) and a maximum GPU frequency exceeding 900 MHz (e.g., 1,173 MHz); a CPU 406 including 8 or more cores (e.g., 8 cores), having 64-bit, 2 MB L2 and 4 MB L3 cache memory, and a maximum frequency of 2 GHz or more (e.g., 2 GHz); one or more deep learning accelerators (DLAs); a deep learning accelerator cluster (XNN); a neural network accelerator (NNA) or neural processing unit (NPU) 409 (e.g., one DLA / XNN / NNA / NPU 409); and a vision accelerator (e.g., a programmable vision accelerator (PVA) 407), with a single SoC 404 capable of achieving AI performance of 157 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGX Orin NX 16GB SoC meets these standards and achieves such performance.

[0126] In various embodiments, for example, one or more SoCs 404 include: a GPU 408 having 1000 or more cores (e.g., 1024 cores), 28 or more tensor cores (e.g., 32 tensor cores), and a maximum GPU frequency exceeding 900MHz (e.g., 1020MHz); and a CPU 406 including 6 or more cores (e.g., 6 cores), having 64-bit, 1.5MB L2 and 4MB L3 cache memory, and a maximum frequency of 1.5GHz or higher (e.g., 1.7GHz), with a single SoC 404 capable of achieving AI performance of 67 trillion operations per second (TOPS). For example, NVIDIA's Jetson OrinNano 5GB SoC meets these standards and achieves such performance.

[0127] One or more SoCs 404 may include one or more CPUs 406. In embodiments, one or more CPUs 406 may include CPU clusters or CPU complexes (also referred to herein as “CCPLEX”). One or more CPUs 406 may include multiple cores and / or (e.g., L2, L3) caches. For example, in some embodiments, one or more CPUs 406 may include twelve cores in a coherent multiprocessor configuration. In some embodiments, one or more CPUs 406 may include four dual-core clusters, each with a dedicated L2 cache (e.g., 3MB L2 cache). One or more CPUs 406 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, thereby allowing any combination of clusters of CPUs 406 to be active at any given time.

[0128] One or more SoCs 404 may include any type and number of GPUs 408. For example, an integrated GPU (also referred to herein as an "iGPU") may be used in some embodiments. One or more GPUs 408 may be programmable and can be used efficiently for parallel workloads. In some examples, one or more GPUs 408 may use an enhanced tensor instruction set. One or more GPUs 408 may include one or more streaming microprocessors, wherein each streaming microprocessor may include a cache (e.g., an L1 cache with at least 96KB of storage), and two or more streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage). In some embodiments, one or more GPUs 408 may include at least eight streaming microprocessors. One or more GPUs 408 may use a computation application programming interface (API). Furthermore, one or more GPUs 408 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0129] One or more GPUs 408 can be power-optimized for optimal performance in automotive, robotics, and / or other embedded use cases. For example, one or more GPUs 408 can be fabricated on FinFETs. However, this is not intended to limit, and one or more GPUs 408 can be fabricated using other semiconductor fabrication or manufacturing processes. Each streaming microprocessor can contain multiple mixed-precision processing cores divided into multiple blocks. For example, but not limited to, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSOR COREs for deep learning matrix algorithms, one (e.g., L0) instruction cache, one warp scheduler, one dispatch unit, and / or one (e.g., 64KB) register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to efficiently perform workloads involving mixed computation and addressing computation. Streaming microprocessors can include independent thread scheduling capabilities to enable finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors can also include combined L1 data caches and shared memory units to improve performance while simplifying programming.

[0130] One or more GPUs 408 may include high-bandwidth memory (HBM) and / or (e.g., 16GB) HBM2 memory subsystems to provide peak memory bandwidth of approximately 900GB / s in some examples. In some examples, synchronous graphics random access memory (SGRAM), such as graphics double data rate type 5 synchronous random access memory (GDDR5), may be used in addition to or as an alternative to HBM memory.

[0131] One or more GPUs 408 may include unified memory technology, including access counters, to allow more accurate migration of memory pages to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, Address Translation Service (ATS) support may be used to allow one or more GPUs 408 to directly access the page tables of one or more CPUs 406. In such examples, when a memory management unit (MMU) miss occurs on one or more GPUs 408, an address translation request may be transferred to one or more CPUs 406. In response, one or more CPUs 406 may look up the virtual-to-physical mapping of the address in their page tables and transfer the translation back to one or more GPUs 408. Therefore, unified memory technology allows the memory of one or more CPUs 406 and one or more GPUs 408 to use a single, unified virtual address space, thereby simplifying the programming of one or more GPUs 408 and the porting of applications to one or more GPUs 408.

[0132] One or more SoCs 404 may include any number of caches 412, including the caches described herein. For example, cache 412 may include L0 cache, L1 cache, L2 cache, L3 cache (e.g., available for one or more CPUs 406 and one or more GPUs 408 (e.g., connected to both one or more CPUs 406 and one or more GPUs 408)), etc. One or more caches 412 may include write-back caches that can track the state of the line, for example, by using one or more cache coherence protocols (e.g., MEI, MESI, MSI, etc.). According to embodiments, the cache (e.g., L3) may include 4MB or more, although smaller or larger cache sizes may be used.

[0133] One or more SoCs 404 may include one or more arithmetic logic units (ALUs) 465, which can be used to perform processing related to various tasks or operations of machine 400, such as computer vision, machine learning or deep learning processing, world model management, etc. Furthermore, one or more SoCs 404 may include one or more floating-point units (FPUs) 467 or other mathematical coprocessors or digital coprocessor types for performing mathematical operations within the system. For example, one or more SoCs 404 may include one or more FPUs 467, which are integrated as execution units within one or more CPUs 406 and / or one or more GPUs 408.

[0134] One or more SoCs 404 may include one or more accelerators 414 (e.g., hardware accelerators, software accelerators, or combinations thereof). For example, one or more SoCs 404 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. Large on-chip memory 415 (e.g., 4MB SRAM, 32GB and / or 64GB 256-bit LPDDR5 (204.8GB / s), 8GB and / or 16GB 128-bit LPDDR5 (102.4GB / s), and / or other memory types and sizes) enables the hardware acceleration cluster to accelerate neural network processing, converter processing, optical flow processing, vision processing, and / or other computations or processing. The hardware acceleration cluster can be used to supplement one or more GPUs 408 and offload some tasks from one or more GPUs 408 (e.g., freeing up more cycles from one or more GPUs 408 to perform other tasks). As an example, one or more accelerators 414 can be used for target workloads (e.g., perceptual, convolutional neural network (CNN), deep neural network (DNN), language model (LLM, VLM, MMLM, VLA, etc.), transformer model, diffusion model, encoder-only model, encoder-decoder model, etc., which are stable enough to be suitable for acceleration.

[0135] One or more accelerators 414 (e.g., hardware acceleration clusters) may include one or more deep learning accelerators (DLAs) 409 (also referred to herein as “Deep Learning Accelerator Cluster (XNN) 409”, “Neural Network Accelerator (NNA) 409”, or “Neural Processing Unit (NPU) 409”). One or more DLAs 409 may include one or more tensor processing units (TPUs) 441, which may be configured to provide additional, for example, trillions of operations per second for deep learning applications and inference. TPUs 441 may be accelerators configured to perform image processing functions (e.g., for CNNs, RCNNs, DNNs, etc.) and optimized for them. One or more DLAs 409 may be further optimized for a specific set of neural network types and floating-point operations and inference. DLAs are designed to provide higher performance per millimeter than general-purpose GPUs and significantly outperform CPUs. One or more TPUs 441 may perform several functions, including single-instance convolution functions, support for INT8, INT16, and FP16 data types for features and weights, and post-processor functions. Although one or more TPUs 441 are described as being included as part of one or more DLAs 409, this is not intended to be limiting, and one or more TPUs 441 may be included in an additional or alternative accelerator 414 and / or other components, and / or may be included as a discrete processing component.

[0136] One or more DLA 409s can rapidly and efficiently perform neural network operations on processed or unprocessed data to achieve a variety of functions, including but not limited to: object and feature recognition and detection (e.g., vehicles, pedestrians, other robots, lane lines, road boundary lines, debris, potholes, boxes, warehouse items, etc.) using data from one or more sensor modalities; distance estimation using data from one or more sensor modalities; emergency vehicle detection and identification using data from microphones and / or vision-based sensors; facial recognition; pick-up and place operations; maneuvering operations; occupant monitoring; vehicle owner identification; and / or other in-cabin operations using data from in-cabin cameras and / or other sensor types; and / or safety and / or safety-related events, to name just a few.

[0137] One or more DLAs 409 can perform any function of one or more GPUs 408, and by using inference accelerators, for example, designers can anchor one or more DLAs 409 or one or more GPUs 408 for any function. For example, designers can centralize the processing of DNN and floating-point operations on one or more DLAs 409, while leaving other functions to one or more GPUs 408 and / or other accelerators 414. One or more DLAs 409 can be used to run any type of network to enhance control and security, including, for example, neural networks that output a confidence metric for each object detection.

[0138] One or more accelerators 414 (e.g., a hardware acceleration cluster) may include one or more programmable vision accelerators (PVAs) 407, which may alternatively be referred to herein as computer vision accelerators or generally as vision accelerators. One or more PVAs 407 may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), semi-autonomous driving, autonomous driving, robotics applications, safety and supervision applications, augmented reality (AR), virtual reality (VR), and / or mixed reality (MR) applications, etc. One or more PVAs 407 may provide a balance between performance and flexibility. For example, each PVA 407 may include (e.g., but not limited to) any number of Reduced Instruction Set Computer (RISC) cores, Direct Memory Access (DMA) systems, Pixel Processing Engines (PPEs), Vector Processors or Vector Processing Units (VPUs), and / or other components. The PVA engine may include an Advanced Very Long Instruction Word (VLIW) or Single Instruction Multiple Data (SIMD) digital signal processor. One or more PVAs 407 may be optimized for image processing and computer vision algorithm acceleration tasks. For example, one or more PVA407s offer excellent performance with extremely low power consumption and can be used asynchronously and concurrently as part of a heterogeneous computing pipeline with one or more CPU 406s, one or more GPU 408s, and / or other accelerators in a system (e.g., vehicles, robots, etc.).

[0139] One or more PVA 407s may include one or more (e.g., two) Vector Processing Subsystems (VPSs), each of which may include one or more Vector Processing Unit (VPU) cores, one or more Decoupled Lookup Units (DLUTs), one or more shared or vector memory (VMEMs), and one or more instruction caches (I-caches). The VPU core may be the main processing unit and may include a vector SIMD VLIWDSP 443 optimized for computer vision. The VPU core can fetch instructions via the I-cache and access data via the VMEM. The DLUT may include dedicated hardware components that enhance the efficiency of parallel lookup operations. For example, the DLUT allows parallel lookups using a single copy of the lookup table by performing these lookups in a decoupled pipeline independent of the main processor pipeline. By doing so, the DLUT can minimize or reduce memory usage and increase throughput while avoiding data-related memory library conflicts, ultimately leading to improved overall system performance. The VPU VMEM can provide local data storage for the VPU, allowing for the efficient implementation of various image processing and computer vision algorithms. The VPU VMEM can support access from an external VPS host, such as Direct Memory Access (DMA), and one or more CPUs 406 (e.g., ARM Cortex-R5 processors), thereby facilitating data exchange with one or more CPUs 406 and other system-level components. The VPU I-cache can provide instruction data to the VPU upon request, request missing instruction data from system memory, and / or maintain temporary instruction storage for the VPU. For each VPU task, one or more CPUs 406 can configure the DMA system, optionally prefetching the VPU program into the VPU I-cache, and / or initiating each VPU-DMA pair to process the task. One or more PVAs 407 may also include L2 SRAM memory shared between the VPS and one or more (e.g., two) sets of DMAs. In some embodiments, one or more (e.g., two) DMA devices are used to move data between external memory, PVA L2 memory, VMEM (e.g., one per VPS), CPU tightly coupled memory (TCM), DMA descriptor memory, and / or PVA-level configuration registers. In lightly loaded systems, two parallel DMA accesses to DRAM can achieve read / write bandwidth of up to 15 GB / s, while in heavily loaded systems, this bandwidth can reach up to 10 GB / s. Regarding compute capacity, INT8 gigabyte multiply-accumulate operations per second (GMAC) can be 2048 or greater, excluding DLUT. FP32 GMACs can be 32 per PVA instance.

[0140] The RISC core can interact with image sensors (e.g., the image sensor of any camera described herein), image signal processors, etc. Each RISC core may include any amount of memory. The RISC core can use any of a variety of protocols, depending on the implementation. In some examples, the RISC core can execute a real-time operating system (RTOS). The RISC core can be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, the RISC core may include an instruction cache and / or tightly coupled RAM.

[0141] The DMA system enables one or more components of PVA 407 to access system memory independently of one or more CPUs 406. DMA can support any number of functions optimized for one or more PVA 407s, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which may include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0142] A vector processor, or VPU, can be a programmable processor designed to efficiently and flexibly execute computer vision algorithms and provide signal processing capabilities. In some examples, one or more PVA 407s may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as a main processing engine for one or more PVA 407s and may include one or more vector processing units (VPUs), one or more pixel processing engines (PPEs) (which may include a 2D layout of interconnected (e.g., for north, south, east, and west communication) processing elements), one or more instruction caches, and / or one or more shared or vector memories (e.g., VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can improve throughput and speed.

[0143] In some embodiments, each vector processor may include an instruction cache and may be coupled to dedicated memory. Therefore, in some examples, each vector processor may be configured to execute independently of other vector processors. In other examples, vector processors included in one or more specific PVA 407s may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA 407 may execute the same computer vision algorithm, but for different regions of an image. In other examples, vector processors included in one or more specific PVA 407s may execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on consecutive images or portions of an image. Among other things, the hardware acceleration cluster may include any number of PVA 407s, and each PVA may include any number of vector processors. Furthermore, one or more PVA 407s may include additional error correction code (ECC) memory to enhance overall system security.

[0144] One or more accelerators 414 (e.g., a cluster of hardware accelerators) have broad applications for autonomous and semi-autonomous machine control. One or more PVAs 407 can be programmable vision accelerators used in critical processing stages such as perception, robot understanding and reasoning, ADAS, semi-autonomous and autonomous vehicles. The capabilities of PVAs 407 are well-suited for algorithmic domains requiring predictable processing with low power consumption and low latency. In other words, one or more PVAs 407 perform well in semi-intensive or intensive rule computation, even on small datasets requiring predictable runtime, low latency, and low power consumption. Therefore, in the context of autonomous vehicles and robotic platforms, PVAs 407 are designed to run classic computer vision algorithms, as they are highly efficient in object detection and integer mathematical operations.

[0145] For example, according to one embodiment of this technology, PVA 407 is used to perform computer stereo vision. In some examples, semi-global matching-based algorithms can be used, although this is not intended to limit it. Many Level 3-5 autonomous driving applications require on-the-fly motion estimation / stereo matching (e.g., moving structures, pedestrian recognition, lane detection, etc.). PVA 407 can perform computer stereo vision functions on input from two monocular cameras.

[0146] In some examples, one or more PVA 407s may be used to perform dense optical flow, providing processed RADAR based on the raw RADAR data (e.g., using 4D Fast Fourier Transform). In other examples, one or more PVA 407s may be used for time-of-flight depth processing, for example, providing processed time-of-flight data by processing the raw time-of-flight data.

[0147] While one or more VPUs, one or more DMAs, one or more RISC Cores, one or more VMEMs, and one or more decoupled coprocessors (e.g., one or more DLUTs) are described as being contained within one or more PVAs 407, this is not intended to be limiting. In some embodiments, these components may be included in alternative or additional processing components and / or one or more accelerators 414, and / or may be included as discrete components of one or more SoCs 404 and / or one or more other computing system architectures.

[0148] In some examples, one or more SoCs 404 may include a real-time ray tracing hardware accelerator (RTA) 451, which can be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) to generate real-time or near-real-time visualization simulations, for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for simulation of SONAR, RADAR, LiDAR, camera and / or other sensor modalities in a simulation, for general wave propagation simulation, for comparison with LiDAR data for localization, for generating realistic training data for training neural networks, and / or other functions and uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations. For example, machine 400 (or another machine or device) may perform simulations in a simulation environment, and one or more light transport simulation algorithms (e.g., ray tracing, path tracing, etc.) may be used to generate the simulation environment. Therefore, the ray tracing accelerator 451 and / or a ray tracing-optimized GPU 408 (e.g., NVIDIA's RTX GPU) may be used to accelerate these ray tracing algorithms.

[0149] One or more accelerators 414 (e.g., in a hardware acceleration cluster) may include one or more optical flow accelerators (OFAs) 411. For example, one or more OFAs 411 may be used to calculate optical flow and stereo disparity between sensor data frames (e.g., images). Optical flow may be accelerated on one or more OFAs 411 for purposes such as object detection and tracking, and / or for stereo depth estimation, where stereo disparity is calculated between stereo image frames (e.g., two or more frames captured using two or more image sensors with at least partially overlapping fields of view).

[0150] One or more SoCs 404 may include one or more Camera Serial Interfaces (CSIs) 423. For example, a CSI 423 may include a Mobile Industry Processor Interface (MIPI) Camera Serial Interface (CSI) for receiving video and input from cameras, a high-speed interface, and / or a video input block that can be used for camera and associated pixel input functions. One or more SoCs 404 may also include software-controllable input / output controllers and may be used to receive I / O signals not assigned to a specific role. For example, a CSI 423 may include MIPI CSI-2 connectors, such as a 16-channel MIPI CSI-2 connector, D-PHY 2.1 (up to 40Gbps), and C-PHY 2.0 (up to 164Gbps) to support 16 virtual channels and 6 or more cameras; a 5-channel MIPI CSI-2 connector, D-PHY 2.1 (up to 20Gbps) to support 5 virtual channels and 4 or more cameras; and / or 2x MIPI CSI-2, 22-pin camera connectors, depending on the embodiment and implementation.

[0151] One or more accelerators 414 (e.g., a hardware acceleration cluster) may include an on-chip computer vision network (CVNOC) 463 and SRAM for providing high-bandwidth, low-latency SRAM to one or more accelerators 414. In some examples, the on-chip memory may include at least 4 MB of SRAM, for example, but not limited to, consisting of eight field-configurable memory blocks accessible by PVA 407, OFA 411, DLA 409, and / or one or more other accelerators 414. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory 415 may be used. PVA 407, OFA 411, DLA 409, and / or one or more other accelerators 414 may access the memory via a backbone providing high-speed memory access to one or more accelerators 414. The backbone may include an on-chip computer vision network that interconnects one or more accelerators 414 to the memory (e.g., using an APB).

[0152] CVNOC 463 may include an interface that determines whether one or more accelerators 414 provide ready and valid signals before transmitting any control signals / addresses / data. Such an interface can provide separate stages and separate channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface may conform to ISO 26262 or IEC 61508 standards, although other standards and protocols may be used.

[0153] One or more SoCs 404 may include one or more data repositories 416 and / or memories 415. One or more data repositories 416 may be on-chip memories 415 of one or more SoCs 404, which may store neural networks and / or other algorithms to be executed on one or more CPUs 406, one or more GPUs 408, and / or one or more accelerators 414. In some examples, the capacity of one or more data repositories 416 may be large enough to store multiple neural network instances for redundancy and security. One or more data repositories 416 may include, for example, one or more L2 and / or L3 caches 412. One or more memories 415 may include SRAM, LPDDR5, and / or other memory types. For example, one or more memories 415 may include 4MB SRAM, 32GB and / or 64GB 256-bit LPDDR5 (204.8GB / s), 8GB and / or 16GB 128-bit LPDDR5 (102.4GB / s), and / or other memory types and sizes. References to one or more data repositories 416 may include references to memories associated with PVA 407, OFA 411, DLA 409 and / or one or more other accelerators 414, as described herein.

[0154] One or more data repositories 416 may include various storage types, such as eMMC, NVMe, etc. For example, one or more SoCs 404 may include storage in the form of embedded multimedia cards (eMMC) (e.g., 64GB eMMC 5.1) and / or SD card slots with external NVM Express (NVMe) capability, for example, via M.2 Key M. For example, one or more data repositories 416 and / or other storage may be accessed via, for example, NVMe using PCI Express (PCIe), RDMA, TCP, and / or other protocols.

[0155] One or more SoCs 404 may include one or more processors 410 (e.g., embedded processors). One or more processors 410 may include a boot and power management processor (BPMP) 453, which may be a dedicated processor and subsystem for handling boot power and management functions, as well as related security implementations. BPMP 453 may be part of the boot sequence of one or more SoCs 404 and may provide runtime power management services. BPMP 453 may provide clock and voltage programming, assistance with low-power state transitions of the system, management of thermal and temperature sensors of one or more SoCs 404, and / or management of power states of one or more SoCs 404. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and one or more SoCs 404 may use the ring oscillator to detect the temperature of one or more CPUs 406, one or more GPUs 408, one or more accelerators 414, and / or other components. If the temperature is determined to exceed the threshold, the BPMP 453 may enter the temperature fault routine and place one or more SoCs 404s into a lower power state and / or place the machine 400 into a driver safety stop mode (e.g., to safely stop the machine 400).

[0156] One or more processors 410 may also include a set of embedded processors that can be used as an audio processing engine (APE) 455. The APE 455 may be an audio subsystem capable of providing full hardware support for multi-channel audio through multiple interfaces, as well as a wide and flexible audio I / O interface. In some examples, the APE 455 is a dedicated processor core with a digital signal processor and dedicated RAM.

[0157] One or more processors 410 may also include an Always-On Processor Engine (AOPE) 457, which can provide the necessary hardware features to support low-power sensor management and wake-up use cases. AOPE 457 may include a processor core, tightly coupled RAM, peripheral support (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0158] One or more processors 410 may also include one or more security processors 413 (or "security islands 413"), which may include a security cluster engine comprising dedicated processors or processor subsystems for handling security management for automotive, robotic, and / or other applications. One or more security processors 413 and / or the security cluster engine may include two or more processor cores, tightly coupled RAM, support for peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In security mode, the two or more cores may operate in lockstep mode and as a single core, with comparison logic to detect any differences in their operation. In some embodiments, one or more security processors 413 may include discrete processors such that failure of other system components does not affect the performance and availability of the security processor 413.

[0159] One or more processors 410 may also include a real-time or near-real-time sensor engine (SE) 459, which may include a dedicated processor subsystem for handling real-time or near-real-time camera, LiDAR, RADAR and / or other sensor modal management.

[0160] One or more processors 410 may also include one or more image signal processors (ISPs) 427, which may include high dynamic range signal processors and / or hardware engines as part of one or more sensor processing pipelines.

[0161] One or more processors 410 may include a video image synthesizer (VIC) 461, which may be a processing block (e.g., implemented on a microprocessor) that implements the video post-processing functions required by the video playback application to produce the final image of the player window. VIC 461 may perform lens distortion correction on one or more wide-angle cameras 468B, one or more surround cameras 468D, cabin monitoring camera sensors, and / or other camera sensors with distorted fields of view.

[0162] VIC 461 may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, when motion occurs in the video, noise reduction appropriately weights spatial information, thereby reducing the weight of information provided by adjacent frames. When an image or part of an image does not contain motion, temporal noise reduction performed by the video image synthesizer can use information from previous images to reduce noise in the current image.

[0163] The VIC 461 can also be configured to perform stereoscopic correction on input stereoscopic camera frames. The video image compositor can also be used for user interface compositing when the operating system desktop is in use and one or more GPUs 408 do not need to continuously render new surfaces. Even when one or more GPUs 408 are powered on and actively performing 3D rendering, the video image compositor can be used to offload one or more GPUs 408 to improve performance and responsiveness.

[0164] One or more SoCs 404 may also include various peripheral interfaces for input / output (I / O) 425, for example, to enable communication with peripheral devices, audio codecs, power management and / or other devices. One or more SoCs 404 may be used to process data from one or more cameras (e.g., via gigabit multimedia serial links and / or Ethernet connections), data from one or more sensors (e.g., one or more LiDAR sensors 464, one or more RADAR sensors 460, etc., connected via Ethernet), data from bus 402 (e.g., speed of machine 400, steering wheel position, etc.), and data from one or more GNSS sensors 458 (e.g., connected via Ethernet or CAN bus). One or more SoCs 404 may also include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine and may be used to free one or more CPUs 406 from routine data management tasks. In some embodiments, one or more SoC 404 I / O 425s may include headers (e.g., 40-pin headers or 40-pin extension headers) supporting Universal Asynchronous Receiver / Transmitter (UART), Serial Peripheral Interface (SPI), Inter-Integrated Sound (I2S), Inter-Integrated Sound (I2C), Controller Area Network (CAN), Pulse Width Modulation (PWM), Digital Microphone Interface (DMIC), Digital Speaker Station (DSPK), General Purpose I / O (GPIO), etc., such as automation headers (e.g., 12-pin automation headers), audio panel headers (e.g., 10-pin audio panel headers), Joint Test Action Group (JTAG) headers (e.g., 10-pin JTAG headers), fan headers (e.g., 4-pin fan headers), RTC battery spare connectors (e.g., 2-pin battery spare connectors), microSD slots, DC power jacks, power, force, resume, and reset buttons, and one or more display connectors (e.g., DisplayPort (DP), such as DP 1.4A (+MST), eDP1.41, HDMI 2.1, and / or 4K30+ DP models). 1.2 (+MST) connector) and / or other I / O 525 elements, components or features.

[0165] One or more SoCs 404 may include machine intranetting capabilities using, for example, Ethernet (e.g., automotive Ethernet), SERDES, Controller Area Network (CAN), FlexRay, Local Interconnect Network (LIN), Low Voltage Differential Signaling (LVDS), Media-Oriented System Transport (MOST), another network type, and / or combinations thereof. For example, one or more SoCs 404 may include RJ45 connectors with up to 10GbE, 1GbE connectors, and / or other network connector types.

[0166] One or more SoCs 404 may include one or more digital signal processors (DSPs) 443. For example, one or more DSPs 443 may include dedicated or specialized microprocessor chips optimized for digital signal processing, such as audio signal processing, telecommunications, digital image processing, RADAR, SONAR, LiDAR and / or other sensor processing, speech recognition and / or other applications.

[0167] One or more SoCs 404 may include one or more video encoders 419 and / or one or more video decoders 421. For example, one or more video encoders 419 may include hardware-based (e.g., as part of one or more GPUs 408) video encoders (e.g., supporting H.264, H.265, etc., and conforming to the HEVC standard, such as NVIDIA's NVENC) that can process image input (e.g., as YUV, RGB, etc.) to generate video bitstreams. One or more video decoders 421 may include a video decoder engine that can provide fully accelerated hardware video decoding capabilities (e.g., supporting decoding of various bitstream formats such as AV1, H.264, H.265, VP8, VP9, ​​MPEG-1, MPEG-2, MPEG-4, VC-1, etc., and conforming to the HEVC standard, such as NVIDIA's NVDEC). In some examples, one or more video decoders 421 may be hardware-based (e.g., as part of one or more GPUs 408).

[0168] One or more SoCs 404 may include one or more General Purpose Computing Acceleration Clusters (GCACs) 429. For example, one or more GCACs 429 may include various processor types that can be used to accelerate computing, such as one or more Vector Microcode Processors (VMPs) 433, one or more Multi-Threaded Processing Clusters (MPCs) 431, one or more Programmable Macroarrays (PMAs) 435, and / or one or more other processor types. For example, one or more GCACs 429 may include a PMA 435, two VMPs 433, and two MPCs 431.

[0169] One or more SoCs 404 may include one or more vector microcode processors (VMPs) 433. In embodiments, one or more VMPs 433 may include wide vector (Very Long Instruction Word (VLIW) and Single Instruction Multiple Data (SIMD)) machines that perform various operations, such as short integral-type operations common in computer vision and deep learning algorithms.

[0170] One or more SoCs 404 may include one or more multi-threaded processing clusters (MPCs) 431. One or more MPCs 431 may include processing clusters that are more general-purpose than GPUs and more efficient than CPUs in embodiments. For example, one or more MPCs 431 may include multi-threaded processors that allow multiple threads to share resources and execute instructions concurrently.

[0171] One or more SoCs 404 may include one or more programmable macro arrays (PMAs) 435. One or more PMAs 435 may include coarse-grained reconfigurable architecture (CGRA) dataflow machines, which have a unique architecture that can deliver powerful performance on intensive computer vision and deep learning algorithms that may not be achievable in classic digital signal processing (DSP) architectures.

[0172] One or more SoCs 404 may include one or more display processing units (DPUs) 445 for performing hardware-accelerated image processing. For example, one or more DPUs 445 may retrieve pixel data from memory 415 and send it to display peripherals via a standard interface. Thus, one or more DPUs 445 may handle display processing and rendering for displays within and / or on the machine.

[0173] One or more SoCs 404 may include one or more application processing units (APUs) 439. For example, one or more APUs 439 may include a quad-core or dual-core processor with 48KB / 32KB L1 cache (with parity and ECC) and a 1MB L2 cache with ECC. One or more APUs 439 may support NEON instructions and single-precision and double-precision floating-point operations.

[0174] One or more SoCs 404 may include one or more Real-Time Processing Units (RTPUs) 469. One or more RTPUs 469 may include a dual-core processor with 32KB / 32KB L1 cache and a 256KB TCM with ECC. One or more RTPUs 469 may support single-precision and double-precision floating-point operations.

[0175] One or more SoCs 404 may include one or more Built-in Self-Test (BIST) components 437. For example, BIST component 437 may include a Memory BIST (MBIST) for testing the system's memory and / or a Logic BIST (LBIST) for testing the system's logic. BIST component 437 may include embedded logic for directly testing the system's logic and / or memory.

[0176] One or more SoCs 404 may include one or more dynamically reconfigurable processors (DRPs) 471. For example, one or more DRPs 471 may be used to accelerate various computational operations. For example, in one embodiment, one or more DRPs 471 may be combined with a MAC unit to function as an AI accelerator. In one embodiment, one or more DRPs 471 may execute an application while dynamically switching the circuit connection configuration of the arithmetic unit (e.g., ALU) on the chip each operating clock according to what needs to be processed. By using only the necessary arithmetic circuitry, one or more DRPs 471 may consume less power than a CPU and can achieve higher speeds. Furthermore, compared to a CPU, which suffers from performance degradation due to frequent accesses to external memory due to cache misses and other reasons, one or more DRPs 471 can pre-build the necessary data paths in the hardware, thereby reducing performance degradation and operating speed variations (jitter) caused by memory accesses. One or more DRP471s may include a dynamic loading function that switches circuit connection information each time the algorithm changes, enabling processing with limited hardware resources, even in robotic / automotive applications that require processing multiple algorithms.

[0177] In some embodiments, one or more accelerators 414 may include OpenCV accelerators for accelerating OpenCV processing, OpenCV being an open-source industry-standard library for computer vision processing. In some embodiments, the combination of one or more DRPs 471 deployed as AI accelerators with OpenCV accelerators can enhance AI computation and image processing algorithms to enable complex and computationally intensive operations such as visual simultaneous localization and mapping (SLAM).

[0178] Compared to traditional systems, the techniques described herein, by providing CPU complexes, GPU complexes, and hardware acceleration clusters, allow multiple neural networks to be executed simultaneously (e.g., at least partially in parallel) and / or sequentially, and the results can be combined to achieve Level 2-5 autonomous driving capabilities and / or autonomous robot motion, control, planning, and / or navigation operations. Furthermore, since one or more SoCs 404 can include various computing engines (e.g., processors 410, CPUs 406, one or more GPUs 408, one or more accelerators 414, etc.), tasks can be distributed among the computing engines, and in some cases, common-cause failures are avoided due to the discrete footprint of the computing engines. Moreover, since one or more SoCs 404 can include one or more dedicated safety processors 413 (or safety islands 413), critical safety or redundant operations can be performed without common-cause failures of the main processing components or computing engines of one or more SoCs 404. Because of these features, the underlying systems of one or more SoC 404 and / or Machine 400 are able to meet higher security levels—such as Automotive Safety Integrity Level (ASIL) D of the ISO 26262 standard.

[0179] Figure 4E According to some embodiments of this disclosure, one or more cloud-based servers (e.g., servers such as those described herein in a data center) and Figure 4AThe following is a system diagram illustrating communication between an example autonomous or semi-autonomous vehicle or machine 400. System 476 may include one or more servers 478, one or more networks 490, and one or more machines 400. One or more servers 478 may include multiple GPUs 484(A)-484(H) (collectively referred to herein as GPU484), switches 482(A)-482(H) (e.g., PCIe 4.0 / 5.0 switches, M.2 slots, Thunderbolt, USB4, NVIDIA's NVLink, NVIDIA's NVSwitch, GPUDirect RDMA, GPUDirect Storage, etc.), CPUs 480(A)-480(B) (collectively referred to herein as CPU 480), accelerators, and / or other processor types. GPU 484, CPU 480, and PCIe switches may interconnect with high-speed interconnects, such as, but not limited to, NVIDIA-developed NVLink interfaces 488 and / or PCIe connections 486. In some examples, the GPU 484 is connected via NVLink and / or NVSwitch SoC, and the GPU 484 and PCIe switch 482 are connected via PCIe interconnect. While the figure shows eight GPUs 484, two CPUs 480, and two PCIe switches, this is not limiting. According to embodiments, each server 478 may include any number of GPUs 484, CPUs 480, and / or PCIe switches. For example, each of one or more servers 478 may include eight, sixteen, thirty-two, and / or more GPUs 484.

[0180] One or more servers 478 may receive sensor data indicating information about new or previously unexplored locations, and / or sensor data indicating changes to previously seen / stored locations (e.g., unexpected or changed road conditions, such as recently started road construction), via one or more networks 490 and one or more machines 400. One or more servers 478 may transmit neural network 492, updated neural network 492, map information 494, etc., including information about traffic and road conditions, to one or more machines 400 via one or more networks 490. Updates to map information 494 may include updates to HD maps 422, SD maps, navigation maps, etc., such as information about construction sites, potholes, detours, floods, and / or other obstacles. In some examples, neural network 492, updated neural network 492, map information 494 and / or other information may come from new training and / or experience, which is represented in data received from any number of machines 400 in the environment, and / or based on training performed in a data center (e.g., using one or more servers 478 and / or other servers).

[0181] One or more servers 478 can be used to train machine learning models (e.g., neural networks) based on training data. Training data can be generated by one or more machines 400, and / or can be generated in simulations (e.g., using a game engine). In some examples, the training data is labeled (e.g., neural networks benefit from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is unlabeled and / or unprocessed (e.g., neural networks do not require supervised learning). Training can be performed according to any one or more machine learning techniques, including but not limited to the following categories: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component analysis and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, it can be used by one or more machines 400 (e.g., transmitted to one or more machines 400 via one or more networks 490, and / or the machine learning model can be used by one or more servers 478 to remotely monitor and / or control one or more machines 400).

[0182] In some examples, one or more servers 478 may receive data from one or more machines 400 and apply the data to state-of-the-art real-time neural networks for real-time intelligent inference. One or more servers 478 may include deep learning supercomputers and / or dedicated AI computers powered by one or more GPUs 484, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, one or more servers 478 may include deep learning infrastructure in a data center using only CPUs.

[0183] The deep learning infrastructure of one or more servers 478 may be capable of rapid, real-time inference and can use this capability to assess and verify the health of the processors, software, and / or associated hardware in machine 400. For example, the deep learning infrastructure may receive periodic updates from machine 400, such as image sequences and / or objects located by machine 400 in the image sequence (e.g., through computer vision and / or other machine learning object classification techniques). The deep learning infrastructure may run its own neural network to identify objects and compare them with the objects identified by machine 400. If the results do not match and the infrastructure concludes that the AI ​​in machine 400 has malfunctioned, one or more servers 478 may signal to machine 400 that the fail-safe computer of machine 400 take over control, notify the occupants, and perform safety maneuvers or operations, such as slowing down, returning control to the driver, stopping, and / or pulling over / closing off.

[0184] For inference, one or more servers 478 may include one or more GPUs 484 and one or more programmable inference accelerators (such as NVIDIA's TensorRT). The combination of GPU-driven servers and inference acceleration enables real-time response. In other examples, such as in scenarios where performance is less critical, servers driven by CPUs, FPGAs, and other processors can be used for inference.

[0185] Computational ecosystem for generating, training, and deploying AI Figure 5 This is a system diagram illustrating a three-computer ecosystem 500 according to at least some embodiments of the present disclosure, including a first computing system 502 for generating or creating artificial intelligence (AI) (e.g., AI training and validation data), a second computing system 504 for training the AI, and a third computing system 506 for deploying AI at the edge (which may include or correspond to...). Figures 4A to 4EOne or more SoCs 404). For example, to develop and deploy materialized or physical AI, a three-computer ecosystem 500 can be used, comprising three accelerated computer systems to handle physical AI training, simulation, and runtime (e.g., edge deployment). These systems can use scalable, physical-based simulations of one or more machines 400 and their worlds to generate training data and train multimodal base models (and / or other model types). By doing so, simulations of one or more machines 400 can be performed at scale, allowing skills (e.g., robotic skills) to be improved, tested, and optimized in virtual worlds that simulate the laws of physics (e.g., using NVIDIA's OMNIVERSE), thereby helping to reduce the cost of real-world data acquisition and ensuring that one or more machines 400 can operate safely in a controlled environment.

[0186] Computing system 504 (e.g., NVIDIA's DGX platform) can be used to train and fine-tune powerful foundational and generative AI models. Models such as general foundational models (e.g., NVIDIA's Project GR00T) can be used to enable robots and one or more other machines 400 to understand natural language and mimic actions by observing human movements. Computing system 504 may include a platform that integrates software, infrastructure, and expertise into a modern, unified AI development and training solution. Computing system 504 may include individual computing devices 510 (e.g., NVIDIA's DGX B200, H200, etc.) and / or any number of computing devices 510 (e.g., NVIDIA's DGX SuperPOD) within data center infrastructure 512.

[0187] For example, a standalone computing device 510 may include GPUs (e.g., 8 GPUs with a total GPU memory of 1,440 GB) and CPUs (e.g., 2 CPUs with a total of 112 cores, 2.1 GHz or 4 GHz (with enhancements)), providing up to 72 petaFLOPS of training capability and 144 petaFLOPS of inference capability. The computing device 510 may include memory (e.g., 4 TB of memory) and storage (e.g., 2 x 1.9 TB NVMe M.2 OS storage and 8 x 3.84 TB NVMe U.2 internal storage). The computing device 510 may include various networking and network management components, such as OSFP ports (e.g., 4 OSFP ports) for servicing single-port intelligent host channel adapters (e.g., 8 single-port ConnextX-7 Virtual Protocol Interconnect (VPI)), providing up to 400 GB / s of Infiniband / Ethernet. Computing device 510 may also include, for example, a dual-port quad small pluggable (QSFFP) data processing unit (DPU) (e.g., two dual-port QSFP112 DPUs – such as NVIDIA’s BlueField-3DPU) providing up to 400Gb / s of InfiniBand / Ethernet. One or more computing devices 510 may include onboard network interface cards (NICs) (e.g., a 10Gb / s onboard NIC with RJ45), dual-port Ethernet NICs (e.g., a 100GB / s dual-port Ethernet NIC), and / or host board management controllers (MBCs) (e.g., with RJ45). In some embodiments, the NICs for one or more computing devices 510 may include SuperNICs (e.g., NVIDIA’s ConnectX-8 SuperNIC) to provide up to 800Gb / s of data throughput for in-network computing acceleration engines, thereby providing the performance and powerful feature set required to support trillion-parameter-scale AI factories and scientific computing workloads. In other embodiments, one or more computing devices 510 may include intelligent host channel adapters (HCAs) (e.g., NVIDIA's ConnectX-7) to provide ultra-low latency, 400Gb / s throughput for computing acceleration engines within the network.

[0188] Data center infrastructure 512 may include: any number of computing devices 510, and an operating system (OS) (e.g., a DGX OS extension of a Linux distribution) for maximizing system uptime, security, and reliability; network / storage acceleration libraries and management for accelerating end-to-end infrastructure performance; cluster management for scaling and managing a single node (e.g., a computing device 510) to thousands of nodes; job scheduling and orchestration to ensure the smooth execution of each developer's jobs; AI workflow management and machine learning operations (MLOps) for moving more models from prototypes to production; and enterprise software to accelerate developer success.

[0189] Computational System 502 (e.g., NVIDIA's OVX server) can provide a development and simulation platform for testing and optimizing physics-based AI using APIs and frameworks for simulation (e.g., NVIDIA's DriveSIM, ISAAC Sim, ISAAC Gym, ISAAC Lab, etc.). Computational System 502 allows developers to use simulation frameworks to simulate and validate robot models, and / or generate large amounts of physics-based synthetic data to guide model training. Computational System 502 can support learning frameworks that power robot reinforcement learning and imitation learning to accelerate robot policy training and refinement. For example, Computational System 502 can be used to generate any number of simulations 508, such as in NVIDIA's OMNIVERSE. Computational System 502 can be used to optimize and accelerate the entire software stack, from training, fine-tuning, and deploying generative AI to supporting industrial digitization within content collaboration platforms (including APIs, software development kits (SDKs), and services). These platforms allow the integration of OpenUSD, ray tracing rendering technologies (e.g., NVIDIA's RTX), and generative physics-based AI into existing software tools and simulation workflows, such as for industrial and robotics use cases (e.g., NVIDIA's OMNIVERSE). Therefore, computing system 502 can host or support the native OpenUSD software platform, enabling enterprises to connect 3D pipelines and develop advanced real-time 3D applications for industrial digitalization. With powerful ray tracing-accelerated AI and graphics capabilities, computing system 502 delivers robust performance for workloads such as extended reality (XR), multi-user design collaboration, and digital twins. This allows for the creation of physically accurate models with high-fidelity ray tracing and path tracing material rendering, large-scale, AI-enabled simulations, and the generation of realistic 3D synthetic data for training. Computing system 502 can include a single computing device 514 (e.g., an NVIDIA OVX L40S server) and / or any number of computing devices 514 (e.g., NVIDIA OVX systems) within data center infrastructure 516.

[0190] One or more computing devices 514 (which may include servers) may include CPUs (e.g., 2 CPUs, each with 32 cores) and GPUs (e.g., 4 or 8 GPUs, each including 48GB GDDR6 with ECC memory, 864GB / s memory bandwidth, PCIe Gen4x16: 64GB / s bidirectional interconnect interface, 18,176 CUDA cores, 142 ray tracing (RT) cores, and 568 Tensor cores). Computing devices 514 may include various networking and network management components, such as Intelligent Host Channel Adapters (HCAs) (e.g., 2 or 4 single-port ConnextX-7s, each with 200Gb / s, providing up to 800Gb / s InfiniBand / Ethernet), and one or more DPUs (e.g., dual-port QSFP112 DPUs, such as the NVIDIA BlueField-3 DPU), providing up to 400Gb / s InfiniBand / Ethernet. In some embodiments, the NIC for computing device 514 may include a SuperNIC (e.g., NVIDIA's ConnectX-8 SuperNIC) to provide up to 800Gb / s of data throughput for the network-in-network compute acceleration engine, thereby providing the performance and robust feature set required to support trillion-parameter-scale AI factories and scientific computing workloads. In other embodiments, one or more computing devices 514 may include a Smart Host Channel Adapter (HCA) (e.g., NVIDIA's ConnectX-7) to provide ultra-low latency and 400Gb / s throughput for the network-in-network compute acceleration engine. One or more computing devices 514 may include host memory (e.g., 384Gb DDR5 ECC for four GPUs, or 768Gb DDR5 ECC for eight GPUs) and may include dual in-line memory module (DIMM) slots, host boot drives (e.g., 1TB NVMe), and / or host storage (e.g., 24TB NVMe).

[0191] Similar to data center infrastructure 512, data center infrastructure 516 can allow any number of computing devices 514 to be combined into a cluster configuration according to a reference architecture.

[0192] Computing system 506 can be used to deploy trained AI models on runtime computers, such as one or more SoCs 404 described herein. For example, these computing systems 506 can be designed for compact onboard computing needs, including ensembles of models such as control policies, vision, and language models deployed on an energy-efficient onboard edge computing system 506. (See also: [link to document]) Figures 4A to 4EA more detailed description of the components, features, and capabilities of computing system 506.

[0193] Example Generative Model In at least some embodiments, language models such as Large Language Models (LLM), Visual Language Models (VLM), Multimodal Language Models (MMLM), Visual Language Action (VLA) models, and / or other types of generative artificial intelligence (AI) can be implemented. These models may be able to understand, summarize, translate, and / or otherwise generate text (e.g., natural language text, code, etc.), images, videos, computer-aided design (CAD) assets, OMNIVERSE and / or METAVERSE file information (e.g., USD formats such as OpenUSD), and / or the like, based on context provided in input prompts or queries. In embodiments, these language models may be considered "large" because they are trained on massive datasets and have architectures with a large number of learnable network parameters (weights and biases)—e.g., millions or billions of parameters. This disclosure allows for the implementation of LLM / VLM / MMLM, etc., for summarizing textual data, analyzing data (e.g., text, images, videos, etc.), extracting insights from data (e.g., text, images, videos, etc.), and generating new text / images / videos / , etc., in a user-specified style, tone, and / or format. In some embodiments, the LLM / VLM / MMLM, etc., of this disclosure may be specifically designed for text processing, while in other embodiments, multimodal LLMs may be implemented to accept, understand, and / or generate text and / or other types of content, such as images, audio (sound, synthesized speech, etc.), 2D and / or 3D data (e.g., USD format), and / or video. For example, a visual language model (VLM) or more specifically a multimodal language model (MMLM) may be implemented to accept images, videos, sensor data, audio, text, 3D designs (e.g., CAD), and / or other input data types and / or generate or output images, videos, audio, text, 3D designs, and / or other output data types.

[0194] Various types of LLM / VLM / MMLM / etc. architectures can be implemented in various embodiments. For example, different architectures can be implemented using different techniques to understand and generate outputs (e.g., text, audio, video, images, 2D and / or 3D design or asset data, etc.). In some embodiments, LLM / VLM / MMLM / etc. architectures (e.g., recurrent neural networks (RNNs) or long short-term memory networks (LSTMs)) can be used, while in other embodiments, converter architectures (e.g., architectures relying on self-attention and / or cross-attention (e.g., between contextual data and textual data) mechanisms) can be used to understand and recognize relationships between words or tokens and / or contextual data (e.g., other text, video, images, design data, USD, etc.). One or more generative processing pipelines including LLM / VLM / MMLM / etc. may also include one or more diffusion blocks (e.g., noise reduction blocks). The LLM / VLM / MMLM / etc. of this disclosure may include encoder and / or decoder blocks. For example, discriminative or encoder-only models (e.g., BERT (Bidirectional Encoder Representations from Transformers)) can be implemented for tasks involving language understanding (e.g., classification, sentiment analysis, question answering, and named entity recognition). As another example, generative or decoder-only models (e.g., GPT (Generative Pretrained Transformer)) can be implemented for tasks involving language and content generation (e.g., text completion, story generation, and dialogue generation). LLM / VLM / MMLM / etc., including encoder and decoder components (e.g., T5 (Text-to-Text Transformer)), can be implemented to understand and generate content, such as for translation and summarization. These examples are not intended to be limiting and any architecture type (including, but not limited to, those described herein) can be implemented depending on the specific implementation and the task performed using LLM / VLM / MMLM / etc.

[0195] In various embodiments, LLM / VLM / MMLM / etc. can be trained using unsupervised learning, where LLM / VLM / MMLM / etc. learns patterns from a large amount of unlabeled text / audio / video / image / design / USD / etc. data. Due to the extensive training, in these embodiments, the model may not require task-specific or domain-specific training. An LLM / VLM / MMLM / etc. extensively pre-trained on a large amount of unlabeled data can be referred to as a base model and can excel at various tasks, such as question answering, summarizing, filling in missing information, translation, and image / video / design / USD / data generation. Some LLM / VLM / MMLM / etc. can be customized for specific use cases using techniques such as cue tuning, fine-tuning, retrieval augmentation generation (RAG), adding adapters (e.g., custom neural networks and / or neural network layers to tune or adjust cues or labels to bias the language model towards a specific task or domain), and / or using optimization models for specific tasks and / or other fine-tuning or customization techniques for specific domains.

[0196] In some embodiments, the LLM / VLM / MMLM / etc. disclosed herein can be implemented using various model alignment techniques. For example, in some embodiments, guardrails can be implemented to identify incorrect or unwanted inputs (e.g., prompts) and / or outputs of the model. In this process, the system can use guardrails and / or other model alignment techniques to prevent the processing of specific unwanted inputs using LLM / VLM / MMLM / etc., and / or to prevent the output or presentation of information generated by LLM / VLM / MMLM / etc. (e.g., displays, audio outputs, etc.). In some embodiments, one or more additional models (or layers thereof) can be implemented to identify problems with the model's inputs and / or outputs. For example, these "protective" models can be trained to identify "safe" or otherwise okay or desired inputs and / or outputs and / or "unsafe" or otherwise unwanted inputs and / or outputs for a particular application / implementation. Therefore, the LLM / VLM / MMLM / etc. disclosed herein are unlikely to output language / text / audio / video / design data / USD data / etc. that may be offensive, vulgar, inappropriate, insecure, out of scope, and / or unwanted for a particular application / implementation.

[0197] In some embodiments, an LLM / VLM / etc. can be configured or able to access or use one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc. For example, for certain tasks or operations where the model is not ideally suited, the model may have instructions for accessing one or more plugins (e.g., third-party plugins) to help process the current input (e.g., as a result of training, and / or based on instructions in a given prompt). In such an example, when at least part of the prompt relates to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs) to retrieve relevant information. Another example is that if at least part of the response requires mathematical computation, the model can access one or more mathematical plugins or APIs to help solve the problem, and then the response from the plugins and / or APIs can be used in the model's output. This process can be repeated (e.g., recursively) an arbitrary number of iterations, using any number of plugins and / or APIs, until a response to each query / question / request / process / operation / etc. can be generated in response to the input prompt. Therefore, models can rely not only on their own knowledge gained from training on large datasets, but also on the expertise or optimized properties of one or more external resources (such as APIs, plugins, etc.).

[0198] In some embodiments, multiple language models (e.g., LLM / VLM / MMLM / etc., multiple instances of the same language model, and / or multiple hints provided to the same language model or instances of the same language model) can be implemented, executed, or accessed (e.g., using one or more plugins, user interfaces, APIs, databases, data stores, repositories, etc.) to provide output in response to the same query or in response to separate parts of a query. In at least one embodiment, the same input query and hints (e.g., a set of constraints, condition generators, etc.) can be provided to multiple language models (e.g., language models with different architectures, language models trained on different (e.g., updated) data corpora). In one or more embodiments, the language models can be different versions of the same base model. In one or more embodiments, at least one language model can be instantiated as multiple agents, for example, providing more than one hint to constrain, guide, or otherwise influence the style, content, or character of the provided output. In one or more exemplary non-limiting embodiments, the same language model can be required to provide output corresponding to different roles, perspectives, characters, or different knowledge bases, as defined by the provided hints.

[0199] In any such embodiment, the outputs of two or more (e.g., each) language models, two or more versions of at least one language model, two or more instantiated proxies of at least one language model, and / or provided to two or more prompts for at least one language model can be further processed, such as aggregated, compared, or filtered, or used to determine (and provide) a consensus response. In one or more embodiments, the output from one language model (or version, instance, or proxy) can be provided as input to another language model for further processing and / or validation. In one or more embodiments, the language model can be required to generate or otherwise obtain output about the input source material, wherein the output is associated with the input source material. Such association can include, for example, generating captions or text portions embedded (e.g., as metadata) within the input source text or image. In one or more embodiments, the output of the language model can be used to determine the validity of the input source material for further processing or inclusion in a dataset. For example, the language model can be used to evaluate the presence (or absence) of a target word in a text portion or the presence (or absence) of an object in an image, wherein the text or image is annotated to indicate such presence (or absence). Alternatively, the determination from the language model can be used to determine whether the source material should be included in a curatorial dataset, for example, but not limited to this.

[0200] Figure 6 This is a block diagram of an example generative language model system 600 suitable for implementing at least some embodiments of the present disclosure. Figure 6 In the example shown, the generative language model system 600 includes a retrieval enhancement generation (RAG) component 692, an input processor 605, a tokenizer 610, an embedding component 620, a plugin / API 695, and a generative language model (LM) 630 (which may include LLM, VLM, MMLM, VLA model, etc.).

[0201] At a high level, the input processor 605 can receive input 601, which includes text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasound, etc.), 3D design data, CAD data, generic scene descriptor (USD) data (e.g., OpenUSD, etc.), depending on the architecture of the generative LM 630 (e.g., LLM / VLM / MMLM, etc.). In some embodiments, input 601 includes plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, input 601 may include numerical sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., tabular format, JSON, or XML). In the generative LM In some implementations of 630 capable of handling multimodal input, input 601 can combine text (or text that may be omitted) with image data, audio data, video data, design data, USD data, and / or other types of input data (e.g., but not limited to the data described herein). Taking raw input text as an example, input processor 605 can prepare the raw input text in various ways. For example, input processor 605 can perform various types of text filtering to remove noise (e.g., special characters, punctuation marks, HTML tags, stop words, portions of images, portions of audio, etc.) from the relevant text content. In examples involving stop words (common words that often have little semantic meaning), input processor 605 can remove stop words to reduce noise and enable generative LM. 630 focuses on more meaningful content. Input processor 605 can apply text normalization, for example, by converting all characters to lowercase, removing accents, and / or handling special cases (such as abbreviations or abbreviations) to ensure consistency (e.g., converting ¼ to ¼). Similarly, input processor 605 and / or post-processor can perform inverse text normalization (ITN) to convert plain language back to canonical or other forms (e.g., converting ¼ to ¼). These are just a few examples; other types of input and / or output processing can be applied.

[0202] In some embodiments, RAG component 692 (which may include one or more RAG models, and / or may be performed using generative LM 630 itself) may be used to retrieve additional information to be used as part of input 601 or a prompt. RAGs can be used to enhance input to LLM / VLM / MMLM / etc. with external knowledge to make the answer to a specific question or query or request more relevant, for example, where specific knowledge is required. RAG component 692 may obtain this additional information (e.g., basic information such as basic text / images / videos / audio / USD / CAD / etc.) from one or more external sources, and then feed it along with the prompt to LLM / VLM / MMLM / etc. to improve the accuracy of the model's response or output.

[0203] For example, in some embodiments, in addition to the data retrieved using RAG component 692, input 601 may also be generated using query or model inputs (e.g., questions, requests, etc.). In some embodiments, input processor 605 may analyze input 601 and communicate with RAG component 692 (or in embodiments, RAG component 792 may be part of input processor 705) to identify relevant text and / or other data to provide to generative LM 630 as additional context or information sources, typically from which responses, answers, or outputs 690 are identified. For example, when the input indicates that a user is interested in the required tire pressure for a particular brand and model of vehicle, RAG component 692 may use a RAG model, for example, to perform a vector search in the embedding space to retrieve tire pressure information or its corresponding text from a digital (embedded) version of the owner's manual for that particular brand and model of vehicle. Similarly, when a user revisits a chatbot related to a particular product for sale or service, RAG component 692 may retrieve previously stored conversation history (or at least its summary) and provide the previous conversation history along with the current inquiry / request as part of input 601 to generative LM 630.

[0204] RAG component 692 can use various RAG techniques. For example, naïve RAG can be used, where documents are indexed, chunked, and applied to an embedding model to generate embeddings corresponding to chunks. User queries can also be applied to this embedding model and / or another embedding model of RAG component 692, and the embeddings of chunks can be compared with the embeddings of the query to identify the most similar / relevant embeddings to the query. These most similar / relevant embeddings can be provided to generative LM 630 to generate output.

[0205] In some embodiments, more advanced RAG techniques can be used. For example, chunks can undergo pre-retrieval processes (e.g., routing, rewriting, metadata analysis, expansion, etc.) before being passed to the embedding model. Furthermore, post-retrieval processes (e.g., re-ranking, hint compression, etc.) can be performed on the output of the embedding model before generating the final embedding, which is then used for comparison with the input query.

[0206] As a further example, modular RAG techniques can be used, such as those similar to Naive RAG and / or Advanced RAG, but also including features such as hybrid search, recursive retrieval and query engines, StepBack methods, subqueries and hypothetical document embeddings.

[0207] As another example, Graph RAG can use a knowledge graph as a source of context or factual information. Graph RAG can be implemented using a graph database as a source of contextual information sent to LLM / VLM / MMLM / etc. Instead of providing the model with data chunks extracted from larger documents (which can lead to a lack of context, factual accuracy, linguistic accuracy, etc.) (or anything other than providing the model with data chunks extracted from larger documents), Graph RAG can also provide the model with structured entity information by combining structured entity text descriptions with their many attributes and relationships, thus giving the model deeper insights. In implementing Graph RAG, the systems and methods described herein use graphs as content stores and extract relevant document chunks, requiring LLM / VLM / MMLM / etc. to use them to answer questions. In such embodiments, the knowledge graph may contain relevant textual content and metadata about the knowledge graph, or it may be integrated with a vector database. In some embodiments, Graph RAG can use the graph as a subject matter expert, where descriptions of concepts and entities relevant to the query / hint can be extracted and passed to the model as semantic context. These descriptions may include relationships between concepts. In other examples, the graph can be used as a database where a portion of a query / hint can be mapped to a graph query, the graph query can be executed, and LLM / VLM / MMLM / etc. can summarize the results. In such examples, the graph can store relevant factual information and can be used for queries (natural language queries) and entity links to graph query tools (NL to graph query tools). In some embodiments, the graph RAG (e.g., using a graph database) can be combined with standard (e.g., vector database) RAGs and / or other RAG types to benefit from a variety of approaches.

[0208] In any embodiment, RAG component 692 can implement plugins, APIs, user interfaces, and / or other functions to perform RAG. For example, LLM / VLM / MMLM / etc. can use graph RAG plugins to run queries on knowledge graphs to extract relevant information to feed into the model, and can use standard or vector RAG plugins to run queries on vector databases. For example, the graph database can interact with the plugin's REST interface, thus decoupling the graph database from the vector database and / or the embedded model.

[0209] The tokenizer 610 can segment (e.g., processed) text data into smaller units (tags) for subsequent analysis and processing. Depending on the implementation, the tags can represent individual words, sub-words, characters, audio / video / images, etc. Word-based tokenization divides the text into individual words, treating each word as a separate tag. Sub-word tokenization breaks words down into smaller meaningful units (e.g., prefixes, suffixes, stems), enabling the generative LM 630 to understand morphological changes and process words outside the vocabulary more effectively. Character-based tokenization represents each character as a separate tag, enabling the generative LM 630 to process text at a fine-grained level. The choice of tokenization strategy can depend on factors such as the language being processed, the task at hand, and / or the characteristics of the training dataset. Therefore, the tokenizer 610 can transform (e.g., processed) text into a structured format according to the tokenization scheme implemented in a particular embodiment.

[0210] Embedding component 620 can use any known embedding technique to transform discrete tokens into semantically meaningful (e.g., dense, continuous vector) representations. For example, embedding component 620 can use pre-trained word embeddings (e.g., Word2Vec, GloVe, or FastText), one-hot encoding, Term Frequency-Inverse Document Frequency (TF-IDF) encoding, one or more embedding layers of a neural network, and / or others.

[0211] In some implementations where input 601 includes image data / video data, etc., input processor 605 may resize the data to a standard size compatible with the format of the corresponding input channel and / or normalize pixel values ​​to a common range (e.g., 0 to 1) to ensure consistent representation, and embedding component 620 may encode the image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations where input 601 includes audio data, input processor 605 may resample the audio file to a consistent sampling rate for uniform processing, and embedding component 620 may use any known technique to extract and encode audio features, such as in the form of a spectrogram (e.g., a Mel spectrogram). In some implementations where input 601 includes video data, input processor 605 may extract frames or apply resizing to extracted frames, and embedding component 620 may extract features such as optical flow embedding or video embedding and / or encode temporal information or frame sequences. In some implementations of input 601 that include multimodal data, the embedded component 720 can use techniques such as early fusion (concatenation), late fusion (sequential processing), and attention-based fusion (e.g., self-attention, cross-attention) to fuse representations of different types of data (e.g., text, images, audio, USD, video, design, etc.).

[0212] Other components of the generative LM 630 and / or generative LM system 600 may use different types of neural network architectures depending on the implementation scheme. For example, a transducer-based architecture (such as the one used in models like GPT) may be implemented, and it may include a self-attention mechanism that weights the importance of different words or tokens in the input sequence and / or a feedforward network that processes the output of the self-attention layer, applying a nonlinear transformation to the input representation and extracting higher-level features. Some non-limiting example architectures include transducers (e.g., encoder-decoder, decoder-only, multimodal), RNNs, LSTMs, fusion models, diffusion models, cross-modal embedding models that learn a joint embedding space, graph neural networks (GNNs), hybrid architectures that combine different types of adversarial networks (such as generative adversarial networks or GANs or adversarial autoencoders (AAEs) for joint distribution learning, linear time series modeling using a selective state-space modeling (SSM) architecture (e.g., the Mamba LLM architecture), etc. Therefore, depending on the implementation scheme and architecture, the embedded component 620 can apply the encoded representation of the input 601 to the generative LM 630, and the generative LM 630 can process the encoded representation of the input 601 to generate an output 690, which may include response text and / or other types of data.

[0213] As described herein, in some embodiments, the generative LM 630 may be configured to access or use (or be able to access or use) plugins / APIs 695 (which may include one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc.). For example, for certain tasks or operations where the generative LM 630 is not ideally suited, the model may have instructions (e.g., as a result of training, and / or based on instructions in a given prompt, such as instructions retrieved using RAG component 692) to access one or more plugins / APIs 695 (e.g., third-party plugins) to help process the current input. In such an example, when at least part of the prompt is related to a restaurant or weather, the model may access one or more restaurant or weather plugins (e.g., via one or more APIs), sending at least part of the prompt related to a particular plugin / API 695 to the plugin / API 695, which can process the information and return an answer to the generative LM 630, which can then use the response to generate output 690. This process can be repeated (e.g., recursively) an arbitrary number of iterations and repeated using any number of plugins / APIs 695 until an output 690 that resolves each query / question / request / process / action / etc. from input 601 is generated. Therefore, one or more models can rely not only on their own knowledge acquired from training on a large dataset and / or from data retrieved using the RAG component 692, but also on the expertise or optimized properties of one or more external resources (e.g., plugins / APIs 695).

[0214] In some embodiments, one or more converter engines (TEs) can be implemented. Converter engines can use micro-tensor scaling to optimize performance and accuracy, such as enabling 16-bit floating-point (FP16), 8-bit floating-point (FP8), and / or 4-bit floating-point (FP4) AI processing. For example, a converter engine can use 16-bit or 8-bit floating-point precision and 8-bit or 4-bit floating-point data formats, combined with software algorithms, to improve AI performance and capabilities. By reducing mathematical operations to 8 or 4 bits, TEs can train larger networks faster without compromising accuracy. For example, TEs can include libraries for accelerating converter models on processing devices such as GPUs to provide better performance in training and inference with lower memory utilization. When TEs are combined with other technologies, such as high-speed interconnects between nodes (e.g., using switches, such as NVLink switches) and tensor cores (which enable mixed-precision computation, such as micro-scaling precision support), server clusters can be more capable of training massive networks (e.g., billions of parameters) at high speeds. Therefore, it can support tensor core precision for FP64, TF32, BF16, FP16, FP8, INT8, FP6 and FP4, as well as CUDA core precision for FP64, FP32, FP16 and BF16.

[0215] The LLM / VLM / MMLM / VLA and other architectures described herein are intended to be examples only, and other suitable architectures may be implemented within the scope of this disclosure.

[0216] Example computing device Figure 7This is a block diagram of an example computing device 700 suitable for implementing some embodiments of the present disclosure. The computing device 700 may include an interconnect system 702 directly or indirectly coupled to: a memory 704, one or more central processing units (CPUs) 706, one or more graphics processing units (GPUs) 708, a communication interface 710, input / output (I / O) ports 712, input / output components 714, a power supply 716, one or more presentation components 718 (e.g., one or more displays, one or more speakers, etc.), and one or more logic units 720. In at least one embodiment, one or more computing devices 700 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 708 may include one or more vGPUs, one or more CPUs 706 may include one or more vCPUs, and / or one or more logic units 720 may include one or more virtual logic units. Thus, one or more computing devices 700 may include discrete components (e.g., a full GPU dedicated to computing device 700), virtual components (e.g., a portion of the GPU dedicated to computing device 700), or a combination thereof.

[0217] although Figure 7 The various blocks are shown as connected via interconnect system 702 using lines, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 718 (such as a display device) may be considered I / O component 714 (e.g., if the display is a touchscreen). As another example, CPU 706 and / or GPU 708 may include memory (e.g., memory 704 may represent a storage device other than the memory of GPU 708, CPU 706, and / or other components). Therefore, Figure 7 The computing devices described are for illustrative purposes only. No distinction is made between such categories as “workstation,” “server,” “laptop computer,” “desktop computer,” “tablet computer,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of this category. Figure 7 Within the scope of computing devices.

[0218] Interconnect system 702 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 702 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 706 may be directly connected to memory 704. Further, CPU 706 may be directly connected to GPU 708. In cases where there is a direct or point-to-point connection between components, interconnect system 702 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required to be included in computing device 700.

[0219] The memory 704 may include any computer-readable medium from a variety of computer-readable media. The computer-readable medium may be any available medium accessible by the computing device 700. The computer-readable medium may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, the computer-readable medium may include computer storage media and communication media.

[0220] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 704 may store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 700. As used herein, computer storage media does not include the signal itself.

[0221] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and includes any information transmission medium. The term "modulated data signal" can refer to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the above should also be included within the scope of computer-readable media.

[0222] One or more CPUs 706 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 700 to perform one or more of the methods and / or processes described herein. Each CPU 706 may contain one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling numerous software threads simultaneously. CPUs 706 may contain any type of processor and may contain different types of processors depending on the type of computing device 700 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 700, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (such as math coprocessors), computing device 700 may also include one or more CPUs 706.

[0223] In addition to or in lieu of one or more CPUs 706, one or more GPUs 708 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 700 to perform one or more of the methods and / or processes described herein. One or more GPUs 708 may be integrated GPUs (e.g., with one or more CPUs 706) and / or one or more GPUs 708 may be discrete GPUs. In embodiments, one or more GPUs 708 may be coprocessors of one or more CPUs 706. One or more GPUs 708 may be used by computing device 700 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 708 may be used for general-purpose computing on a GPU (GPGPU). One or more GPUs 708 may contain hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. One or more GPUs 708 may produce pixel data of an output image in response to rendering commands (e.g., rendering commands received via a host interface from one or more CPUs 706). One or more GPUs 708 may include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 704. One or more GPUs 708 may include two or more GPUs operating in parallel (e.g., via a link). The link may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 708 may produce pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for an analog image). Each GPU may contain its own memory or may share memory with other GPUs.

[0224] In addition to or in lieu of one or more CPUs 706 and / or one or more GPUs 708, one or more logic units 720 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 700 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 706, one or more GPUs 708, and / or one or more logic units 720 may perform any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 720 may be a portion of one or more CPUs 706 and / or GPUs 708 and / or integrated into one or more CPUs 706 and / or GPUs 708, and / or one or more logic units 720 may be discrete components or otherwise external to CPUs 706 and / or GPUs 708. In an embodiment, one or more of the logic units 720 may be coprocessors of one or more of the CPU 706 and / or one or more of the GPU 708.

[0225] Examples of one or more logic units 720 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree lateral unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), programmable vision accelerator (PVA) (which may include one or more direct memory access (DMA) systems), and one or more vision or vector processing units (VPUs). PU), one or more pixel processing engines (PPE) (e.g., comprising a 2D array of processing elements, each of which communicates north, south, east, and west with one or more other processing elements in the array), one or more decoupled accelerators or units (e.g., decoupled lookup table (DLUT) accelerators or units), visual processing units (VPU), optical flow accelerators (OFA), field-programmable gate arrays (FPGA), neuromorphic chips, quantum processing units (QPU), associative processing units (APU), arithmetic logic units (ALU), application-specific integrated circuits (ASIC), floating-point units (FPU), input / output (I / O) elements, peripheral component interconnects (PCI) or fast peripheral component interconnects (PCIe) elements, etc.

[0226] The communication interface 710 may include one or more receivers, transmitters, and / or transceivers enabling the computing device 800 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). The communication interface 710 may include components and functions for implementing communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wirelessband), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, one or more logic units 720 and / or the communication interface 710 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via interconnect system 702 to one or more GPUs 708 (e.g., memory of one or more GPUs 708).

[0227] I / O port 712 enables computing device 700 to be logically coupled to other devices including I / O component 714, one or more presentation components 718, and / or other components, some of which may be built into (e.g., integrated into) computing device 700. Illustrative I / O component 714 includes microphone, mouse, keyboard, joystick, gamepad, game controller, dish satellite dish, scanner, printer, wireless device, etc. I / O component 714 can provide a natural user interface (NUI) for processing air gestures, voice, or other physiological input generated by the user. In some cases, input may be transmitted to appropriate network elements for further processing. NUI can implement any combination of voice recognition, pen recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of computing device 700. Computing device 700 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. Additionally, the computing device 700 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables motion detection. In some examples, the computing device 700 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0228] The power supply 716 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 716 may provide power to the computing device 700 so that the components of the computing device 700 can operate.

[0229] One or more presentation components 718 may include displays (e.g., monitors, touchscreens, television screens, head-up displays (HUDs), other display types, or combinations thereof), speakers, and / or other presentation components. Presentation component 718 may receive data from other components (e.g., one or more GPUs 708, one or more CPUs 706, etc.) and output said data (e.g., as images, videos, sounds, etc.).

[0230] Example network environment A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 7 It is implemented on one or more instances of one or more computing devices 700—for example, each device may include similar components, features, and / or functions of one or more computing devices 700. Furthermore, in the case of implementing back-end devices (e.g., servers, NAS, etc.), the back-end devices may be included as part of a data center, such as, but not limited to, those described herein.

[0231] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. In cases where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0232] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein for the server can be implemented on any number of client devices.

[0233] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or application may respectively include network-based service software or applications. In embodiments, one or more client devices may use network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs). The framework layer may be, but is not limited to, a free and open-source software network application framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0234] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers at the state, region, country, global, etc.). If the connection to a user (e.g., a client device) is relatively close to an edge server, the core server can assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0235] One or more client devices may include the information described in this article. Figure 7 At least some of the components, features, and functions of one or more example computing devices 700 described. By way of example and not limitation, the client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming equipment or system, entertainment system, vehicle computer system, embedded system controller, chat booth, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0236] Example Terms Clause 1. One or more processors, including processing circuitry, said processing circuitry being used to: generate candidate trajectories associated with a self-machine.

[0237] Clause 2. One or more processors as described in Clause 1, wherein the processing circuitry is further configured to: calculate an estimated advantage of the future world state corresponding to the candidate trajectory, based at least on the representation of the candidate trajectory processed using one or more neural networks.

[0238] Clause 3. One or more processors as described in Clause 1 or 2, wherein said processing circuitry is further configured to: determine, at least based on the estimated advantage of the future world state, whether to use the candidate trajectory to control the navigation of the ego machine.

[0239] Clause 4. One or more processors as described in any one of Clauses 1, 2 or 3, wherein the processing circuitry is further configured to: generate a predicted distribution of future world states corresponding to the candidate trajectories, at least based on processing the representation of the candidate trajectories using a world model neural network in one or more neural networks; and sample the future world states from the predicted distribution.

[0240] Clause 5. One or more processors as described in any one of Clauses 1, 2 or 3, wherein said processing circuitry is further configured to: determine at least one of the following based at least on the estimated advantage of the future world state not meeting a specified threshold: disengage autonomous navigation, disengage autonomous navigation, disable one or more autonomous or semi-autonomous features of the self-machine, or execute a security procedure associated with the self-machine.

[0241] Clause 6. One or more processors as described in any one of Clauses 1, 2 or 3, wherein said processing circuitry is further configured to: generate an alternative trajectory using an optimization process, at least based on the fact that the estimated advantage of the future world state does not meet a specified threshold, the alternative trajectory optimizing the corresponding estimated advantage of an alternative future world state corresponding to the alternative trajectory.

[0242] Clause 7. One or more processors as described in any one of Clauses 1, 2 or 3, wherein the processing circuitry is further configured to: perform one or more iterations of an optimization process that predicts alternative candidate trajectories, and use the one or more neural networks to generate corresponding estimated favorability of alternative future world states corresponding to the alternative candidate trajectories.

[0243] Clause 8. One or more processors as described in any one of Clauses 1, 2 or 3, wherein the processing circuitry is further configured to: reduce the speed of the ego machine in response to determining that an optimization process is used to generate an alternative trajectory, the alternative trajectory optimizing the corresponding estimated advantage of an alternative future world state corresponding to the alternative trajectory.

[0244] Clause 9. One or more processors as described in any one of Clauses 1, 2 or 3, wherein the processing circuitry is further configured to: include representations of sensor data and corresponding alternative trajectories in a training dataset, the alternative trajectories being generated at least based on optimizing the corresponding estimated favorability of alternative future world states corresponding to the alternative trajectories.

[0245] Clause 10. One or more processors as described in any one of Clauses 1, 2 or 3, wherein the processing circuitry is further configured to: perform at least one or more iterations of training a neural planner based on an alternative trajectory, the alternative trajectory being generated at least based on optimizing the corresponding estimated favorability of an alternative future world state corresponding to the alternative trajectory.

[0246] Clause 11. One or more processors as described in any one of Clauses 1, 2, or 3, wherein said one or more processors are located in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more analog operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language model LLMs. Systems for performing operations; systems for performing operations using one or more visual language models (VLMs); systems for performing operations using one or more multimodal language models (MMLMs); systems for performing operations using one or more visual-language-action (VLA) models; systems for using or deploying one or more inference microservices; systems for performing one or more conversational AI operations; systems for generating synthetic data; systems for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; systems containing one or more virtual machines (VMs); systems implemented at least partially in a data center; or systems implemented at least partially using cloud computing resources.

[0247] Clause 12. A method comprising: evaluating, based on at least one or more neural networks, candidate trajectories generated using a processing pipeline associated with a self-machine, and generating an estimate of the quality, security, or predictability of a future scene state generated at least based on said candidate trajectories.

[0248] Clause 13. The method as described in Clause 12 further comprises: determining, at least based on the estimated quality, safety, or predictability of the future scenario state, whether to use the candidate trajectory to control the navigation of the ego machine.

[0249] Clause 14. The method as described in Clause 12 or 13 further comprises: generating a predicted distribution of future scene states corresponding to the candidate trajectories, at least based on processing the representations of the candidate trajectories using a world model neural network in one or more of the neural networks; and sampling the estimated future scene states from the predicted distribution.

[0250] Clause 15. The method as described in Clause 12 or 13 further comprises: determining, based at least on the fact that the estimated quality, security, or expectancy of the future scenario state does not meet a specified threshold, at least one of the following: not engaging autonomous navigation, disengaging from autonomous navigation, disabling one or more autonomous or semi-autonomous features of the self-machine, or executing a security procedure associated with the self-machine.

[0251] Clause 16. The method as described in Clause 12 or 13 further comprises: determining, based at least on the fact that a measurement of the estimated quality of the estimated future scene state does not meet a specified threshold, to use an optimization process to generate an alternative trajectory, the alternative trajectory optimizing a corresponding measurement of the estimated quality of an alternative estimated future scene state corresponding to the candidate trajectory.

[0252] Clause 17. The method as described in Clause 12 or 13 further comprises: performing one or more iterations of an optimization process that predicts alternative candidate trajectories, and using the one or more neural networks to generate corresponding estimates of the quality, safety, or predictability of alternative future scenario states corresponding to the alternative candidate trajectories.

[0253] Clause 18. The method as described in Clause 12 or 13, wherein the method is performed by at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulated operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more visual language models (VLMs); a system for performing operations using one or more multimodal language models (MMLMs); a system for performing operations using one or more visual-language-action (VLA) models; a system for using or deploying one or more inference microservices; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system comprising one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0254] Clause 19. An autonomous or semi-autonomous machine comprising: one or more system-on-a-chip (SoC) units, each including one or more central processing units (CPUs), one or more graphics processing units (GPUs), and one or more hardware accelerators.

[0255] Clause 20. An autonomous or semi-autonomous machine as described in Clause 19, comprising: one or more sensors having a field of view or sensing field located outside the autonomous or semi-autonomous machine, wherein the autonomous or semi-autonomous machine navigates according to a selected trajectory, the trajectory being selected from at least one candidate trajectory based on at least one candidate trajectory satisfying a quality, desirability, or security threshold associated with a future world state, the at least one candidate trajectory being generated based at least on end-to-end neural network processing of sensor data obtained using the one or more sensors, the future world state being generated using the at least one candidate trajectory.

[0256] Clause 21. An autonomous or semi-autonomous machine as described in Clause 19 or 20, wherein a single SoC in one or more SoCs is capable of achieving an AI performance of 275 trillion operations per second (TOPS).

[0257] Clause 22. An autonomous or semi-autonomous machine as described in Clause 19 or 20, wherein the one or more hardware accelerators include at least one of a vision accelerator, a ray tracing accelerator, an optical flow accelerator, or a deep learning accelerator.

[0258] Clause 23. Autonomous or semi-autonomous machines as described in Clause 19 or 20, wherein the autonomous or semi-autonomous machines include vehicles, automobiles, trucks, robots, humanoid robots, autonomous mobile robots (AMRs), warehouse vehicles, drones, ships, or aircraft.

[0259] Clause 24. An autonomous or semi-autonomous machine as described in Clause 19 or 20, wherein the autonomous or semi-autonomous machine includes or uses at least one of the following: a control system for the autonomous or semi-autonomous machine; a perception system for the autonomous or semi-autonomous machine; a system for performing one or more simulated operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language model LLMs. Systems for performing operations using one or more visual language models (VLMs); systems for performing operations using one or more multimodal language models (MMLMs); systems for performing operations using one or more visual-language-action (VLA) models; systems for using or deploying one or more inference microservices; systems for performing one or more conversational AI operations; systems for generating synthetic data; systems for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; systems containing one or more virtual machines (VMs); systems implemented at least partially in a data center; or systems implemented at least partially using cloud computing resources.

[0260] This disclosure can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions, such as program modules, that are executed by a computer or other machine, such as a personal data assistant or other handheld device. Typically, a program module, which includes routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure can also be practiced in distributed computing environments, where tasks are performed by remote processing devices linked via a communication network.

[0261] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0262] This document describes in detail the subject matter of this disclosure to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the discloser has envisioned that the claimed subject matter may be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of the method employed, these terms should not be construed as suggesting any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. One or more processors, including processing circuitry, said processing circuitry being used to: Generate candidate trajectories associated with the self-machine; Based at least on the representations of the candidate trajectories processed using one or more neural networks, an estimated advantage of the future world state corresponding to the candidate trajectories is calculated; and Based at least on the estimated favorableness of the future world state, it is determined whether to use the candidate trajectory to control the navigation of the ego machine.

2. The processor of claim 1 or more, wherein the processing circuitry is further configured to: generate a predicted distribution of future world states corresponding to the candidate trajectories, at least based on processing the representation of the candidate trajectories using a world model neural network in the one or more neural networks; and sample the future world states from the predicted distribution.

3. One or more processors as claimed in claim 1, wherein the processing circuitry is further configured to: determine at least one of the following based at least on the estimated advantage of the future world state not meeting a specified threshold: not engaging autonomous navigation, disengaging from autonomous navigation, disabling one or more autonomous or semi-autonomous features of the ego machine, or executing a security procedure associated with the ego machine.

4. One or more processors as claimed in claim 1, wherein the processing circuitry is further configured to: generate an alternative trajectory using an optimization process, based at least on the fact that the estimated advantage of the future world state does not meet a specified threshold, the alternative trajectory optimizing the corresponding estimated advantage of an alternative future world state corresponding to the alternative trajectory.

5. The processor of claim 1 or more, wherein the processing circuitry is further configured to: perform one or more iterations of an optimization process, the optimization process predicting alternative candidate trajectories, and using the one or more neural networks to generate corresponding estimated favorability of alternative future world states corresponding to the alternative candidate trajectories.

6. One or more processors as claimed in claim 1, wherein the processing circuitry is further configured to: reduce the speed of the ego machine in response to determining that an optimization process is used to generate an alternative trajectory, the alternative trajectory optimizing the corresponding estimated advantage of an alternative future world state.

7. One or more processors as claimed in claim 1, wherein the processing circuitry is further configured to: include representations of sensor data and corresponding alternative trajectories in a training dataset, the alternative trajectories being generated at least based on optimizing the corresponding estimated favorability of alternative future world states corresponding to the alternative trajectories.

8. One or more processors as claimed in claim 1, wherein the processing circuitry is further configured to: perform at least one or more iterations of training a neural planner based on an alternative trajectory, the alternative trajectory being generated at least based on optimizing the corresponding estimated favorability of an alternative future world state corresponding to the alternative trajectory.

9. The one or more processors as claimed in claim 1, wherein the one or more processors are located in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models (MMLMs); A system for performing operations using one or more visual-language-action (VLA) models; A system for using or deploying one or more inference microservices; A system for performing one or more conversational AI operations; A system for generating synthetic data; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

10. A method comprising: The quality, safety, or predictability of the estimated future scene state generated based on at least one or more neural networks are evaluated using a processing pipeline associated with the self-machine. as well as Based at least on the estimated quality, safety, or predictability of the future scenario state, determine whether to use the candidate trajectory to control the navigation of the ego machine.

11. The method of claim 10, further comprising: At least based on processing the representation of the candidate trajectory using the world model neural network in one or more of the neural networks, a predicted distribution of the future scene state corresponding to the candidate trajectory is generated; And sample the estimated future scenario state from the predicted distribution.

12. The method of claim 10, further comprising: If the estimated quality, security, or expectancy of the future scenario state does not meet a specified threshold, determine at least one of the following: disengage autonomous navigation, detach from autonomous navigation, disable one or more autonomous or semi-autonomous features of the self-machine, or execute a security procedure associated with the self-machine.

13. The method of claim 10, further comprising: If the measurement result of the estimated quality of the estimated future scene state does not meet a specified threshold, it is determined that an optimization process will be used to generate an alternative trajectory, which optimizes the corresponding measurement result of the estimated quality of the alternative estimated future scene state corresponding to the candidate trajectory.

14. The method of claim 10, further comprising: The optimization process is performed in one or more iterations, the optimization process predicts alternative candidate trajectories, and uses the one or more neural networks to generate corresponding estimates of the quality, safety, or predictability of alternative future scenario states corresponding to the alternative candidate trajectories.

15. The method of claim 10, wherein the method is performed by at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models (MMLMs); A system for performing operations using one or more visual-language-action (VLA) models; A system for using or deploying one or more inference microservices; A system for performing one or more conversational AI operations; A system for generating synthetic data; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

16. An autonomous or semi-autonomous machine, comprising: One or more system-on-a-chip (SoC) units, each including one or more central processing units (CPUs), one or more graphics processing units (GPUs), and one or more hardware accelerators; as well as One or more sensors having a field of view or sensing field located outside the autonomous or semi-autonomous machine. The autonomous or semi-autonomous machine navigates according to a selected trajectory, which is selected from at least one candidate trajectory based on at least one candidate trajectory satisfying a quality, expectancy, or security threshold associated with a future world state. The at least one candidate trajectory is generated based at least on sensor data obtained using one or more sensors through end-to-end neural network processing. The future world state is generated using the at least one candidate trajectory.

17. The autonomous or semi-autonomous machine of claim 16, wherein a single SoC in one or more SoCs is capable of achieving an AI performance of 275 trillion operations per second (TOPS).

18. The autonomous or semi-autonomous machine of claim 16, wherein the one or more hardware accelerators include at least one of a vision accelerator, a ray tracing accelerator, an optical flow accelerator, or a deep learning accelerator.

19. The autonomous or semi-autonomous machine as claimed in claim 16, wherein the autonomous or semi-autonomous machine includes vehicles, automobiles, trucks, robots, humanoid robots, autonomous mobile robots (AMRs), warehouse vehicles, drones, ships, or aircraft.

20. The autonomous or semi-autonomous machine of claim 16, wherein the autonomous or semi-autonomous machine comprises or uses at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models (MMLMs); A system for performing operations using one or more visual-language-action (VLA) models; A system for using or deploying one or more inference microservices; A system for performing one or more conversational AI operations; A system for generating synthetic data; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

Images