Generalizable movement model for robotic systems

Abstract

The present disclosure relates to generalizable movement models for robotic systems. In various examples, a generalizable movement model can receive a state and an identifier of a robot and generate an action for the robot based on the state and the identifier. The state can identify a location, an environment, and a navigation goal for the robot, while the identifier can indicate a type of the robot. The generalizable movement model can utilize the identifier to generate an action for the robot to reach the navigation goal from its current location based on the type of the robot (e.g., a humanoid robot, a quadruped robot, or a wheeled robot). The generalizable movement model can be a distilled combination of a plurality of robot type-specific models and can utilize the identifier to mimic type-specific actions output by the plurality of robot type-specific models for communication to the robot for movement.

Description

[0001] Cross-references to related applications This application claims the benefit of U.S. Provisional Application No. 63 / 759,693, filed on February 18, 2025, the entire contents of which are incorporated herein by reference. Background Technology

[0002] Mobility models generate and send commands to move the robot. Because each robot form has unique kinematic constraints and complexities, mobility models can be customized for specific robot forms. Such models generate a significant amount of repetitive work and data requirements for each different robot form and are specifically tailored to one form. Summary of the Invention

[0003] Embodiments of this disclosure relate to generalizable mobility models for robotic systems. The disclosed systems and methods are applicable to a wide variety of robots, regardless of their form. The systems and methods of this disclosure distill form-specific policies into a single model that can use form-specific encoding to mimic the action output of each form-specific policy, thereby achieving scalable and adaptive mobility for the robotic system. Each form-specific policy can be updated from a baseline imitation learning model, for example, using residual reinforcement learning, to adapt the output action of each policy to a specific robot form.

[0004] Unlike conventional systems, the systems and methods disclosed herein include a system that may include one or more processors. The one or more processors can determine a robot's state and identifier, the identifier indicating a robot type corresponding to the robot among a plurality of robot types. The one or more processors can process the state and identifier based at least on a generalist motion strategy to generate at least one action for the robot, wherein the generalist motion strategy is generated using a combination of a basic motion strategy corresponding to the plurality of robot types and one or more specialist motion strategies corresponding to different robot types among the plurality of robot types. The one or more processors can cause the robot to move according to the at least one action.

[0005] In various embodiments, a base motion policy is updated using imitation learning and a world model. This base motion policy receives at least one state of the robot as input and outputs a base motion for moving the robot. One or more specialist motion policies can be updated using residual reinforcement learning. These specialist motion policies receive at least one state of the robot as input and output specialist motions for moving the robot. One or more specialist motion policies can be updated at least based on the base motion policy. The specialist motions are combinations of base motions and residual motions, where the residual motions are used to adapt the base motions to the robot type. To generate a generalist motion policy, the combination of the base motion policy and one or more specialist motion policies can be distilled.

[0006] In various embodiments, to generate a generalist motion policy, one or more processors may utilize one or more specialist motion policies to generate multiple specialist actions for the robot by inputting multiple states into one or more specialist motion policies. One or more processors may utilize each of the one or more specialist motion policies to generate multiple normal distributions of the multiple specialist actions. One or more processors may combine the multiple normal distributions. One or more processors may distill the combination of multiple normal distributions into a generalist motion policy by at least minimizing the divergence of the combination of the multiple normal distributions. The state may include at least one of the robot's environment, the velocity of each joint of the robot, or the robot's objective. The identifier may correspond to an embedding in an embedding space that identifies the robot as one of the multiple robot types.

[0007] In various embodiments, the at least one action may include a plurality of speed commands, each speed command corresponding to a joint of the robot. To perform the at least one action, each of the plurality of speed commands may be mapped to a corresponding joint of the robot. The plurality of robot types may include at least humanoid robots, autonomous mobile robots (AMRs), wheeled robots, warehouse vehicles or machines, or quadruped robots.

[0008] In various embodiments, the one or more processors are included in at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more analog operations; a system for performing one or more digital twin operations; a system for performing one or more optical transmission simulations; a system for performing collaborative content creation of 3D assets; a system for performing one or more wireless cellular transmissions using a wireless cellular network; a system for providing one or more cloud gaming applications; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing one or more conversational AI operations; a system for performing operations using one or more Large Language Models (LLMs). Systems for performing operations using one or more visual language models (VLMs), systems for performing operations using one or more multimodal language models (MMLMs), systems for performing operations using one or more visual-language-action (VLA) models, systems for performing one or more conversational AI operations, systems for performing one or more synthetic data generation operations, systems for presenting at least one of virtual reality content, augmented reality content, or mixed reality content, systems that use or deploy one or more inference microservices, systems that include deploying one or more machine learning models and operating system-level virtualization packages (e.g., containers) within services or microservices, systems that include one or more virtual machines (VMs), systems that are at least partially implemented in a data center, or systems that are at least partially implemented using cloud computing resources.

[0009] The systems and methods disclosed herein may include a method. This method may include: using one or more processors to determine a state and embedding corresponding to a robot, the state being determined using a world model, and the embedding indicating the type of robot. This method may include: utilizing one or more processors and at least based on motion policies trained for deployment on multiple robot types, generating multiple commands for various joints of the robot, the state and the embedding being processed using the motion policies to generate the multiple commands. This method may include: utilizing one or more processors to transmit the multiple commands to various joints of the robot to guide and move the robot.

[0010] In various embodiments, the action policy may include a distillation combination of multiple robot type-specific action policies, wherein each of the multiple robot type-specific action policies corresponds to one of the multiple robot types. To generate the action policy, the method may include: generating multiple normal distributions of multiple robot type-specific actions using each of the multiple robot type-specific action policies by one or more processors, wherein multiple states are input into each of the multiple robot type-specific action policies, and each of the multiple robot type-specific action policies outputs multiple robot type-specific actions. The method may include: combining the multiple normal distributions by one or more processors. The method may include: distilling the combination of multiple normal distributions into an action policy by one or more processors at least by minimizing the divergence of the combination of the multiple normal distributions.

[0011] In various embodiments, multiple robot type-specific action policies are updated using reinforcement learning, and one or more processors are further configured to: during reinforcement learning, record the inputs and outputs of each of the multiple robot type-specific action policies, the inputs and outputs being used to generate multiple normal distributions. Before generating the multiple normal distributions, the weights of the multiple robot type-specific action policies may be updated to converge and freeze, the multiple robot type-specific action policies being updated using a base model and a world model updated using imitation learning, wherein the weights of the base model are frozen before updating the multiple robot type-specific action policies.

[0012] The systems and methods disclosed herein may include one or more processors. The one or more processors may include a processing circuitry system configured to: perform one or more control operations associated with a robot based at least on one or more actions generated using a generalist motion strategy, the generalist motion strategy generating the one or more actions if: (i) while conditioned on using an embedding indicating a robot type corresponding to the robot among a plurality of robot types, and (ii) at least based on the generalist motion strategy, processing state information corresponding to the robot and the robot's environment.

[0013] In various embodiments, the generalist motion policy is trained using a teacher dataset generated using motion outputs and latent states corresponding to specialist motion policies associated with various robot types. The embedding may include one-hot morphological encoding indicating the robot type. State information may be represented at least using a world model. The one or more control operations may correspond to one or more joints, actuators, or motors of the robot.

[0014] In various embodiments, the one or more processors are included in at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more analog operations; a system for performing one or more digital twin operations; a system for performing one or more optical transmission simulations; a system for performing collaborative content creation of 3D assets; a system for performing one or more wireless cellular transmissions using a wireless cellular network; a system for providing one or more cloud gaming applications; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing one or more conversational AI operations; a system for performing operations using one or more Large Language Models (LLMs). Systems for performing operations using one or more visual language models (VLMs), systems for performing operations using one or more multimodal language models (MMLMs), systems for performing operations using one or more visual-language-action (VLA) models, systems for performing one or more conversational AI operations, systems for performing one or more synthetic data generation operations, systems for presenting at least one of virtual reality content, augmented reality content, or mixed reality content, systems that use or deploy one or more inference microservices, systems that include deploying one or more machine learning models and operating system-level virtualization packages (e.g., containers) within services or microservices, systems that include one or more virtual machines (VMs), systems that are at least partially implemented in a data center, or systems that are at least partially implemented using cloud computing resources. Attached Figure Description

[0015] The present system and method for a generalizable mobility model for a robotic system will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is an example system for generating generalizable mobility models for robotic systems, according to at least some embodiments of this disclosure; Figure 2 Based on at least some embodiments of this disclosure Figure 1 Example talent strategy generator; Figure 3 Based on at least some embodiments of this disclosure Figure 1 Example of a generic action strategy generator; Figure 4 Based on at least some embodiments of this disclosure Figure 1 Example images generated from simulations of the system; Figure 5 The illustrations depict at least some embodiments according to the present disclosure. Figure 1A generalizable mobility model can generate actions for various types of robots; Figure 6 The illustrations depict at least some embodiments of the present disclosure for updating Figure 1 The system's strategy includes a simulated environment for multiple robots; Figure 7 The illustrations depict at least some embodiments according to the present disclosure. Figure 1 The experimental setup of the system; Figure 8 The illustrations depict at least some embodiments according to the present disclosure. Figure 7 The results of the experimental setup; Figure 9 The illustrations depict at least some embodiments according to the present disclosure. Figure 1 The system's example base policy generator; Figure 10A The illustrations depict at least some embodiments according to the present disclosure. Figure 9 An example of a generative world model of a system; Figure 10B The illustrations depict at least some embodiments according to the present disclosure. Figure 10A Example estimator models; Figure 11A-11C The illustrations depict at least some embodiments according to the present disclosure. Figure 10A Various processing operations for generative world models; Figure 12 This is a flowchart of an example method for generating basic actions according to at least some embodiments of the present disclosure; Figure 13 Based on at least some embodiments of this disclosure Figure 9 Example data generation pipeline for the system; Figure 14 The illustrations depict at least some embodiments according to the present disclosure. Figure 13 Example simulation data for the data generation pipeline; Figure 15 This is a flowchart of an example method for generating simulated data according to at least some embodiments of the present disclosure; Figure 16A This is a flowchart of an example method for generating actions using a generalizable mobile model and transmitting the actions to a robot, according to at least some embodiments of this disclosure; Figure 16B This is a flowchart of an example method for a robot to move according to commands provided by a generalizable mobility model, according to at least some embodiments of the present disclosure; Figure 17A Examples of sensor locations having a corresponding field of view or sensing field, such as an autonomous or semi-autonomous machine, according to at least some embodiments of this disclosure; Figure 17B This is an illustration of examples of the locations of components and sensors on an autonomous or semi-autonomous vehicle according to at least some embodiments of this disclosure; Figure 17C This is a block diagram of an example system architecture for autonomous or semi-autonomous vehicles, robots, and / or other machine types according to at least some embodiments of this disclosure; Figure 17D This is a block diagram of an example architecture of a computing system (e.g., a system-on-a-chip (SoC)) according to at least some embodiments of the present disclosure; Figure 17E This is a system diagram of communication between a cloud-based server and example autonomous or semi-autonomous vehicles, robots and / or other machine types according to at least some embodiments of this disclosure; Figure 18 This is a system diagram illustrating three computer ecosystems according to at least some embodiments of the present disclosure, including a computing system for generating or creating artificial intelligence (AI) (e.g., AI training and validation data), a computing system for training artificial intelligence, and a computing system for deploying AI at the edge; Figure 19 This is a block diagram of an example computing system for generative artificial intelligence (AI) according to at least some embodiments of the present disclosure; and Figure 20 This is a block diagram of an example computing device according to at least some embodiments of the present disclosure. Detailed Implementation

[0016] Systems and methods related to generalizable mobility models for robotic systems are disclosed. Although this disclosure may relate to examples of autonomous or semi-autonomous vehicles, robots, and / or other machine types 1700 (which may alternatively be referred to herein as "vehicle 1700," "self-vehicle 1700," "machine 1700," "self-machine 1700," "robot 1700," and / or "self-robot 1700"), examples are related to... Figures 17A-17EThe description herein is provided, but is not intended to be limiting. For example, the systems and methods described herein may be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms (e.g., autonomous mobile robots (AMRs), humanoid robots, robotic arms and / or end effectors), warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, water vehicles, shuttle vehicles (e.g., robotaxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones and / or other vehicle, robot or machine types. Furthermore, while this disclosure may describe mobility models for robotic systems, it is not intended to be limiting, and the systems and methods described herein may be used in augmented reality (AR), virtual reality (VR), mixed reality (MR), robotics, safety and supervision (e.g., smart cities), autonomous or semi-autonomous machine applications, industrial manufacturing, simulation and / or any other technical field where mobility models can be used. In some embodiments, the systems, methods, and / or processes described herein can be used with Figures 17A-17E Example machine 1700 Figure 18 Example calculation of ecosystem 1800, Figure 19 Example generative language model systems 1900 and / or Figure 20 The example computing device 2000 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the operation.

[0017] Robotics has made significant progress in both industry and everyday life, driving the demand for collaborative robots capable of handling increasingly complex tasks. However, developing robust navigation strategies across robot types remains challenging due to the numerous differences in morphological features, kinematics, joints, controls, and sensor configurations across diverse robot platforms. These differences complicate the creation of general strategies that are both robust and adaptable to real-world environments. Classical (e.g., decomposed or modular) navigation stacks perform well on specific robots (e.g., wheeled platforms) but often require significant readjustment or redevelopment when applied to new robot types with different sensor suites and physical constraints. This reliance on per-robot optimization has fueled interest in end-to-end learning approaches, particularly for scaling across multiple robots.

[0018] Imitation learning (IL) can leverage existing expert demonstrations and teacher strategies. While IL has an intuitive appeal, it can be susceptible to covariate bias, where the policy encounters out-of-distribution states not seen during the demonstration. Although advances in machine learning architectures and data augmentation techniques have helped mitigate these issues, adding more robot-specific factors increases data requirements and training complexity. Generating high-quality demonstrations for complex modalities (e.g., humanoid robots) further complicates pure IL approaches.

[0019] Reinforcement learning (RL) offers an alternative approach to acquiring type-specific policies, particularly for tasks such as motion. However, navigational RL remains constrained by the vast search space and scarce rewards in the natural environment. Residual RL addresses these issues by refining pre-trained policies in a data-driven manner, resulting in faster convergence and higher stability. Meanwhile, emerging vision-language-action (VLA) models have shown potential across robot types, but typically operate within a low-dimensional action space based on waypoints or in open-loop planning phases, thus performing poorly for platforms with higher-dimensional dynamic characteristics.

[0020] To at least address the aforementioned shortcomings, this disclosure relates to systems and methods for deploying a single, generalizable mobility model (e.g., a generalist policy, framework) to guide robots with different morphologies (e.g., embodiments, types, platforms) such as wheeled robots, humanoid robots, or quadruped robots. Conventional approaches have attempted to provide a single policy that considers the kinematic and dynamic constraints of each robot type, but struggle to cope with wide variability, resulting in poor performance, excessively long model training times, or inefficiencies (e.g., the need to retrain the model for each robot). These approaches may include: training a single network across multiple tasks or robot morphologies, transfer learning, using a separate reinforcement learning (RL) model for each morphology, fine-tuning feedback controllers, or relying solely on imitation learning (IL). However, these existing methods are time-consuming, data-intensive, and have limited task performance, thus failing to adapt to the constraints of each robot morphology. Furthermore, conventional generalist models may not perform as well as models dedicated to each robot morphology without significant retraining overhead.

[0021] The systems and methods of this disclosure can be built upon an IL model and refined using residual RL (e.g., residual RL networks, models) for each robot form. Residual RL can add to or modify (e.g., modify or refine) the robot's actions (e.g., action policies) generated by the IL model, and can preserve the robot's navigation knowledge in the IL model while adapting to robot form-specific constraints. For example, the systems and methods of this disclosure can achieve obstacle avoidance while providing stable walking for humanoid robots and leg coordination for quadruped robots. The IL model can include a world model (e.g., an environment model, a simulation model, etc.) to provide the residual RL model with perception and state representation of the robot's environment. Refining the IL model using residual RL can reduce training time and complexity compared to existing general models. To maintain the diversity of control across different robot forms, the systems and methods can distill form-specific knowledge into a single model by utilizing the complete Gaussian parameter set for each robot form. The IL model can be refined using residual RL for each robot morphology and then unified into a single generalizable model, thereby avoiding the complexity of joint multimorphic training and improving data and time efficiency compared to continuous multimorphic training.

[0022] To deploy this single, generalizable model across different robot forms, the single model can receive a one-hot form encoding (e.g., an embedding) that indicates the robot's type. Based on this embedding, the single model can select robot form-specific actions and behaviors. The single model can receive policy states (e.g., the robot's position, location, etc.) from the world model and combine these policy states with the embedding to guide and move the robot.

[0023] The system and method disclosed herein provide a general action policy that inherits the expertise of each specialist, enabling seamless adaptation to various robot types. The framework and resulting action policy described herein utilize imitation learning, residual reinforcement learning, and policy distillation. An initial IL step provides a strong baseline, while residual RL rapidly adapts the model to the dynamic characteristics of each robot type, and the final distillation step merges the specialized policies into a single generalist policy. The system and method described herein enable efficient scaling to various robot types and environments.

[0024] The systems and methods of this disclosure are applicable to a variety of technical fields, such as robot control and mobility, autonomous vehicles or machines, semi-autonomous vehicles or machines, or any other device with autonomous mobility. For example, the systems and methods of this disclosure can be refined using residual RL for use with vehicles, aircraft, watercraft, bicycles, airplanes, and / or any other type of vehicle, as well as humanoid robots, tracked robots, modular robots, soft robots, industrial robots, service robots, medical robots, and / or any other type of robot. In various embodiments, the model can control one or more robots in parallel (e.g., simultaneously). The model can send instructions to multiple robots of different types to navigate in their environment.

[0025] In some embodiments, the systems and methods described herein can be performed using simulated data (e.g., simulated environment data and simulated sensor data of simulated sensors of virtual or simulated vehicles, robots, or machines in the simulated environment) in a simulated environment (e.g., NVIDIA's DriveSIM, ISAAC Sim, ISAAC Gym, ISAAC Lab, etc.). For example, simulated input data (e.g., map data, perception data, self-motion data, tactile data, and / or any other data described herein) can be used to determine the environment and navigation objectives for various types of robots in the simulation. These environments and objectives can be used to update type-specific mobility models, etc., and this information can be used to perform virtual machine-related operations in the simulated environment. For example, the simulation may include a simulation controller for performing actions on the robot in the simulation. These simulated operations can be used to test and update the performance of underlying algorithms, systems, and / or processes before deploying them to the real world. In some cases, the simulation can be used to generate synthetic training data in the simulation, such as simulating robot movement, different environments, etc. This synthetic training data can then be used or processed (as an addition to or replacement of real-world data) to update the mobility model until the weights of the mobility model converge, at which point the mobility model is distilled into a single generalizable mobility model.

[0026] In any example, such as when the simulation environment is used for testing, validation, training, etc., one or more optical transport simulation algorithms (such as one or more ray tracing and / or path tracing algorithms) may be used to render or otherwise generate the simulation environment and / or associated training data. When using optical transport simulation, the simulation system may employ one or more dedicated ray tracing hardware accelerators and / or processors (e.g., NVIDIA's RTX, or other real-time ray tracing GPUs, such as GPUs including one or more ray tracing (RT) cores) optimized for performing real-time or near-real-time optical transport simulation operations in conjunction with one or more other processors in the system (e.g., GPUs, CPUs, accelerators, etc.). In some embodiments, the simulation environment and / or one or more of its objects, features, or components may be generated or managed within a three-dimensional (3D) content collaboration platform (e.g., NVIDIA's OMNIVERSE), which may be optimized for or adapted to industrial digitization, generative physics, artificial intelligence, and / or other use cases, applications, and / or services. For example, a content collaboration platform or system may include a system for managing objects, features, scenes, etc., in simulated environments, digital environments, etc., using or developing common scene descriptors (USD) (such as OpenUSD). The platform may include realistic physics simulations (e.g., using NVIDIA's PhysX software development kit (SDK)) to simulate real physics and physical interactions with simulations hosted on the platform. The platform may integrate OpenUSD with ray tracing / path tracing / light transport simulations (such as NVIDIA's RTX rendering technology) into software tools and simulation workflows for building, training, deploying, and / or testing AI systems—such as systems for testing, validating, training (e.g., machine learning models, neural networks, etc.) and / or other tasks related to automobiles, robots, other machine types, and / or other systems and applications. In some examples, the simulated environment may include a digital twin of a real-world environment, such as a specific road segment, warehouse, data center, airport, geographic area, ocean area, and / or any other real-world environment in which autonomous or semi-autonomous vehicles or machines can operate. Simulated robots may be placed in the simulated environment and provided with navigational targets within it. The motion model can generate actions for the simulated robot to reach navigation targets in the simulated environment, and the movement results of the simulated robot can be recorded and used to update the motion model.

[0027] In some embodiments, a remote control or teleoperation system can be used to teleoperate or remotely control vehicles, robots, and / or other machines. For example, the systems and methods described herein can be used to send movement commands to robots, which can be included in the visualization or mapping of the environment to assist a remote operator in controlling an autonomous or semi-autonomous machine through the environment or to provide waypoints or other control or navigation instructions. Therefore, a remote operator can use visual, auditory, textual, and / or other cues or indicators generated using the systems and methods described herein to assist in navigating vehicles, robots, machines, etc., in a real-world environment using a teleoperation system.

[0028] In some embodiments, the systems and methods described herein can be deployed in robotic applications. For example, a robot or robotic system may include one or more onboard processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), hardware-based programmable vision accelerator (PVA) (which may include one or more vector processing units (VPU), direct memory access (DMA) system and / or pixel processing engine (PPE), hardware-based optical flow accelerator (OFA), SoC, etc.)) and memory and / or storage devices (e.g., for storing control algorithms, sensor data and one or more machine learning models). Robotic systems can use these processors to execute one or more machine learning models (e.g., language models, visual language models (VLM), large language models (LLM), visual-language-action (VLA) models, multimodal language models (MMLM), etc.) that enable them to perform complex tasks autonomously or semi-autonomously, such as interacting with and / or manipulating static and / or dynamic objects, or navigating in their environment using sensors such as cameras, LiDAR, RADAR, ultrasonic sensors, etc. For example, the robotic system can execute at least one of a type-specific movement model or a single generalizable movement model to move autonomously or semi-autonomously in its environment. The system can use sensor fusion techniques to combine data from multiple sensors (e.g., cameras, infrared cameras, LiDAR, RADAR, accelerometers) to create a comprehensive model of the robot's surroundings. This data can be processed locally on the robot or sent to a remote server for more computationally intensive tasks such as 3D mapping or SLAM (simultaneous localization and mapping). In one or more embodiments, data from individual robots (e.g., sensor data, mission status, or environmental conditions) can be uploaded to the cloud, where a centralized AI model can analyze optimized commands and distribute them across the entire fleet. In some embodiments, the machine learning models described herein (e.g., language models, VLM, VLA, LLM, MMLM, diffusion models, NeRF models, DNN, etc.) can be used to enable robots to perceive and reason about their environment and / or communicate with one or more other robots and / or people in the environment. In some embodiments, robots can communicate with one or more locally hosted servers / computing devices and / or with one or more remote servers / computing devices (e.g., located in one or more data centers) using one or more network interface cards (NICs) and / or data processing units (DPUs).

[0029] In some embodiments, the systems and methods described herein can be deployed in in-vehicle infotainment (IVI) systems or cockpit experience (IX) applications. For example, an infotainment system within a vehicle (e.g., a car, truck, drone, construction equipment, robot, semi-autonomous vehicle, or autonomous vehicle) may include one or more onboard processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), hardware-based programmable vision accelerator (PVA) (which may include one or more vector processing units (VPU), direct memory access (DMA) systems, and / or pixel processing engines (PPE)), hardware-based optical flow accelerator (OFA), SoC, etc.) and memory and / or storage devices (e.g., for storing control algorithms, sensor data, and one or more machine learning models) and memory and / or storage devices (e.g., for storing entertainment content, navigation data, and user preferences). The system can use these processors to execute one or more machine learning models (e.g., language models) to achieve features such as voice control, personalized media recommendations, dynamic navigation, and real-time communication with other services via network connectivity. For example, mobility models can be applied to vehicles and executed by the system to implement dynamic navigation. In-vehicle infotainment systems can also use natural language processing (NLP) models to enable voice-based interaction. One or more machine learning models can be stored locally or accessed via one or more APIs connected to cloud services, enabling the system to process requests in real-time or near real-time.

[0030] In some examples, the machine learning models described herein (e.g., deep neural networks, language models, LLMs, VLMs, multimodal language models, visual-language-action (VLA) models, perceptual models, tracking models, fusion models, transformer models, diffusion models, encoder-only models, decoder-only models, encoder-decoder models, neural rendering field (NERF) models, etc.) can be packaged as microservices (e.g., inference microservices, such as NVIDIA NIM), which may include containers (e.g., operating system (OS) level virtualization packages) that may include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model "engine". For example, an inference microservice may include the container itself and the model (e.g., weights and biases). In some cases, such as when the machine learning model is small enough (e.g., has a sufficiently small number of parameters), the model may be included within the container itself. In other examples (e.g., when the model is large), the model may be hosted / stored in the cloud (e.g., in a data center) and / or hosted on-premises and / or at the edge (e.g., on a local server or computing device, but outside the container). In these embodiments, the model can be accessed via one or more APIs, such as REST APIs. Therefore, in some embodiments, the machine learning models described herein can be deployed as inference microservices to accelerate model deployment on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, an optimized inference engine (e.g., execution software built using standardized AI model deployments, such as NVIDIA's Triton Inference Server, and / or one or more APIs for high-performance deep learning inference, which may include inference runtimes and model optimizations for providing low latency and high throughput for production applications, such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). The machine learning models described herein, along with the accelerated infrastructure, can be included as part of a microservice, capable of deployment with a single command and / or orchestrated and automatically scaled on the accelerated infrastructure using a container orchestration system (e.g., reaching data center scale on a single device). Therefore, inference microservices may include machine learning models (e.g., models optimized for high-performance inference), inference runtime software for executing the machine learning models and providing outputs / responses to inputs (e.g., user queries, prompts, etc.), and enterprise management software for providing health checks, identity verification, and / or other monitoring. In some embodiments, inference microservices may include software for in-situ replacement and / or updating of the machine learning models.When replacing or updating, the software performing the replacement / update can maintain the user configurations of the inference runtime software and the enterprise management software.

[0031] While examples of machine learning models (such as neural networks) are described in this document, this is not limiting. For example, but not limited to, any of the various machine learning models and / or neural networks described herein can include any type of machine learning model, such as those using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoder neural networks, artificial neural networks (ANN), convolutional neural networks (CNN), recurrent neural networks (RNN), perceptrons, long short-term memory (LSTM) networks, multilayer perceptron (MLP) networks, deep stacked networks (DSN), generative pre-trained (GPT) models or networks, feedforward networks, radial basis function ANNs, self-organizing maps (SOM). Kohonen mapping, Hopfield network, Boltzmann machine, deep belief neural network, deconvolutional neural network, generative adversarial network (GAN), imitation learning model, world model, reinforcement learning model, residual reinforcement learning model, liquid machine, modular neural network, liquid machine, sequence-to-sequence model, network using transformer architecture, state space model (SSM) (e.g., networks using Mamba architecture (e.g., Mamba-1, Mamba2, etc.), networks using selective state space model, networks using structured state space sequence model, etc.), diffusion model (e.g., diffusion probability model, fraction-based generative model, etc.), neural radiation field (NeRF) model, Gaussian sputtering. One or more machine learning models and / or other types of machine learning models, including splat models, Kolmogorov-Arnold networks (KAN), models with encoder-only architectures, models with decoder-only architectures, models with encoder-decoder architectures, generative machine learning models, language models, large language models (LLM), visual language models (VLM), multimodal language models (MMLM), large action models (LAM), visual-language-action (VLA) models, etc.

[0032] The systems and methods described herein may be used, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, boats, shuttle vehicles (e.g., robotaxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones, and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, such as, but not limited to, machine control, machine localization, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or participant simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing), collaborative content creation of 3D assets (NVIDIA's Omniverse), cloud computing, and / or any other suitable application.

[0033] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aerial systems, medical systems, marine systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing analog operations, systems for performing digital twin operations, systems using edge devices, systems implementing language models (such as Large Language Models (LLM), Visual Language Models (VLM), Visual-Language-Action (VLA) models, and / or multimodal language models), and systems using or deploying one or more Systems that include inference microservices, systems that include deploying one or more machine learning models in services or microservices and OS-level virtualization packages (e.g., containers), systems that perform one or more wireless cellular transmissions using wireless cellular networks, systems that include one or more virtual machines (VMs), systems that perform synthetic data generation operations, systems that are at least partially implemented in a data center, systems that perform conversational AI operations, systems that perform optical transmission simulations, systems that perform collaborative content creation of 3D assets, systems that perform generative AI operations, systems that are at least partially implemented using cloud computing resources, and / or other types of systems.

[0034] refer to Figure 1 , Figure 1This is an example system 100 for generating a single generalizable mobile model according to some embodiments of this disclosure. It should be understood that the arrangements and other arrangements described herein are merely illustrative. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to or in place of the arrangements, components, features, and elements shown, and some elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and at any suitable location (e.g., on a local device, vehicle, or edge machine, locally deployed (e.g., a locally hosted server), remotely (e.g., in one or more computing or server devices in one or more data centers in the cloud), and / or elsewhere). The various functions performed by the entities described herein can be implemented by hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, microcontroller, embedded processor, digital signal processor (DSP), image signal processor (ISP), physical processing unit (PPU), field-programmable gate array (FPGA), accelerators (e.g., deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural network accelerator (NNA) and / or neural processing unit (NPU), programmable vision accelerator (PVA), optical flow accelerator (OFA), etc.), application-specific integrated circuits (ASIC), data processing unit (DPU), quantum processor, etc.) that execute instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 17A-17E Example machine 1700 Figure 18 Example calculation of ecosystem 1800, Figure 19 Example generative language model systems 1900 and / or Figure 20 The example computing device 2000 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the operation.

[0035] System 100 can generate strategy π θ (For example, a general motion policy 130) that can generate actions to provide point-to-point navigation to different robot types (e.g., robot type 304, morphology, avatar, etc.), each characterized by different kinematic and dynamic properties. At time step t, the robot (e.g., robot 1700) can observe the following state: (1) , Among them, I tThis represents the current camera input (e.g., a red-green-blue (RGB) image, image 202), v t It is the measured speed, r t Provides route or goal-related information (e.g., coded waypoints or global planning), 'e' is an avatar (e.g., type, form, etc.) embedding (e.g., indicator, code, identifier, type embedding 306, one-hot form code, etc.) used to specify the robot's form. 'e' may remain constant during the execution of actions by a single robot, but may vary depending on the type of robot. For robots of the same type, the type embedding may be the same.

[0036] Strategy π θ x can be t Mapping to speed command u t = ( v t , ω t (e.g., commands, control operations, etc.) This speed command can then be transmitted to the robot's controller (e.g., a lower-level controller, controller 220) for joint-level actuation (e.g., actuating each joint, actuator, or motor of the robot). The transient dynamics of the environment. This can depend on both the type of robot and external factors in the environment (such as obstacles). A reward function R(·) can be defined to encourage the generation of actions that can efficiently and collision-free reach the goal. The goal can be used to maximize the expected discounted return, defined as follows: (2)

[0037] The system and method disclosed herein can generate a single strategy π that utilizes type embedding e. θ This allows robot types to share knowledge while adapting to different morphological constraints.

[0038] System 100 may include at least one basic policy generator 102. Basic policy generator 102 can generate basic policies 104 (e.g., models, first action policies, action policies, first policies, motion policies, etc.). Basic policy 104 may be a machine learning model (e.g., neural networks, etc.), such as a multilayer perceptron (MLP). Basic policy 104 can generate actions for multiple robots of different types (e.g., morphologies, avatars, etc.). Robot types may include, but are not limited to, humanoid robots, quadruped robots, wheeled robots, tracked robots, snake-like robots, hexapod robots, aerial robots, underwater robots, or swarm robots. Basic policy generator 102 may apply imitation learning (IL) to obtain a universal navigation baseline (e.g., a universal policy, etc.) applicable to environmental states and actions for various robot types.

[0039] To generate the base policy 104, the base policy generator 102 may include at least one data generator 106. The data generator 106 may generate states for the robot (e.g., state 204, simulated state, etc.). These states (e.g., The basic policy 104 may include at least one of the following: position in the environment (e.g., the robot's position in the simulation), velocity of each joint, navigation target (e.g., the robot's goal), route, position of each joint of the robot, environment in which the robot is located, or type of robot. The basic policy 104 may receive a state as input and output a basic action to move at least one of a plurality of robots.

[0040] Data generator 106 can generate different environments in the simulation, such as offices, warehouses, factories, construction sites, roads, and / or outdoor spaces, and the simulated environment may contain various obstacles and components. Data generator 106 can generate different positions of the robot in the simulated environment. Data generator 106 may include at least one motion database 108, which may include one or more motions for the base policy 104 to generate base actions based on these motions. For example, motion database 108 may include one or more routes or navigation goals that the robot wants to achieve and follow. Information (such as data) generated by data generator 106 can be used to generate and update base policy 104. The motions included in motion database 108 may include the robot's current actions or states. For example, these actions may indicate the speed of at least one of the robot's joints, actuators, or motors, and these speeds can be used to generate the robot's actions.

[0041] The basic policy generator 102 may include at least one world model 110 (e.g., environment representation, cognition, prediction model, world tokenizer, etc.). The world model 110 can receive the robot's environment, state, and actions from the data generator 106 and generate a representation of the environment (e.g., latent space, etc.) to capture the dynamic characteristics of the environment and predict transitions in latent states. Latent states can be internal representations of the robot; for example, they can be encodings or vectors of respect for the environment layout, the positions of objects in the environment, and the robot's location (e.g., position) within the environment. Specifically, the world model can predict transitions in the latent space using the following formula: (3) , Among them, s t It is a latent state (e.g., a learned latent state), o t These are the raw observation results (e.g., red, green, and blue (RGB) images, robot speed, etc.). f ϕ Update the potential state based on previous actions. g ψ Attempt to rebuild or predict o t . s t It can encapsulate dynamic characteristics and constraints of the environment (e.g., environmental information).

[0042] World model 110 can encode the robot's state and generate codes indicating the robot's state (e.g., policy tag 208, tag, policy state, etc.). For example, data generator 106 can provide world model 110 with actions and robot states from action database 108, and the world model can encode the robot's state and generate codes indicating the robot's state. The robot's actions can include at least the robot's navigation goal. World model 110 can use both the robot's actions and states to predict the robot's transitions in a simulated environment. For example, the robot may be in a motion state, and world model 110 can predict the robot's transitions and further movements based on this motion state. In various embodiments, the simulated environment may include moving objects, and world model 110 can predict the transitions of moving objects in the environment. World model 110 can generalize out-of-distribution states of the robot (e.g., states the robot has not encountered during training) by predicting future observations (e.g., RGB images) and potential transitions. Therefore, world model 110 can generate codes indicating both the robot's state and out-of-distribution states, thereby providing a robust encoded representation. World model 110 may be an autoregressive world model (e.g., predicting transitions sequentially). In various embodiments, world model 110 may generate latent state and route (e.g., action) codes for the robot, and may combine the latent state and route codes into a single code indicating the state of the robot.

[0043] Data generator 106 may include an expert policy database 112, which may include one or more expert policies (e.g., teacher policies, supervisor policies, goal policies, reference policies, classic navigation stacks, etc.) that can serve as truth values ​​for the robot. For example, an expert policy can generate a truth or ideal action for the robot given its state. A truth action may refer to the best or optimal route the robot should take to reach a given goal in a given environment, as well as the velocity of each joint of the robot. The expert policy may be applicable to standard mobile robots, such as wheeled robots. Expert policy database 112 may include one or more teacher datasets (e.g., training datasets, truth values, etc.). The teacher datasets can at least map output actions to the robot's latent states. The teacher datasets can also map output actions and latent states to the type of robot.

[0044] The basic policy generator 102 may include at least one basic policy updater 114 (e.g., a basic policy trainer, etc.). The basic policy updater 114 may receive at least one expert policy from the expert policy database 112 and may input encodings from the world model 110 into the expert policy to generate expert actions. The weights of the world model 110 may be updated to minimize reconstruction and prediction losses using the expert actions. For example, the weights of the world model 110 may be updated to minimize losses on an expert demonstration (e.g., action) dataset. (4) , in These are expert (e.g., teacher) actions generated by the expert policy. For example, while the base policy updater 114 is updating the base policy 104 based on the expert actions generated by the expert policy, the base policy updater 114 can provide these expert actions to the world model 110 to update the weights of the world model 110. The world model 110 can be updated to generate the robot's potential states better (e.g., more accurately or precisely).

[0045] The basic policy updater 114 may include policies (e.g., first policy, first action policy, etc.), and imitation learning can be used to update the policy. (For example, strategy) (This could be an imitation learning model). For example, a strategy. Expert policies can be observed and replicated (e.g., imitated), thereby generating a mapping from encoded to generated expert actions. In various embodiments, the base policy updater 114 can begin updating the policy using imitation learning after the weights of the world model 110 have converged. .Strategy Can receive s t (e.g., encoding of robot state) and route information r t (e.g., navigation target, route, route 206) to predict u t (For example, the robot's movements), u t It can be represented as: (5) .

[0046] Strategy Output u t The difference between the expert action and the expert policy output can be minimized in the following way: (6) , Where ℓ(·) can be a simple regression loss. The basic policy updater 114 can receive encoding from the world model 110 and provide the encoding to the policy. and strategies It can output u t Route information can be received from data generator 106. Once output, the basic policy updater 114 can determine the loss and update the policy according to equation (6). The weights. Once the strategy... Once the weights converge, the basic policy updater 114 can generate a basic policy 104, which can be an updated (e.g., trained) policy. .

[0047] Therefore, the base policy 104 can be a general navigation policy based on imitation learning (e.g., common-sense navigation) for various types of robots. For example, instead of being refined to be specific to one type of robot, the base policy 104 can provide a navigation and motion baseline to enable the robot to maneuver in its environment while avoiding obstacles (e.g., obstacle avoidance) to reach a navigation target. The base policy 104 can be a velocity prediction motion model. For example, the base policy 104 can generate base actions (e.g., base action 212) that include the velocities of each joint of the robot that cause it to move, thus producing the robot's motion. The base actions output by the base policy 104 can indicate predicted transitions of the robot in its environment, such as predictions of the robot's current orientation and joint movements. For example, the base policy 104 can generate base actions based on robot movements indicated by states generated by the world model 110. The base policy updater 114 can integrate the world model 110 and the base policy 104 to generate actions for different robots, and the base policy 104 can also combine latent states. t And route information to generate actions (e.g., navigation commands, etc.).

[0048] System 100 may include at least one specialist policy generator 116 (e.g., type-specific policy, second action policy generator, etc.). After the weights of at least one of the base policy 104 or world model 110 converge, specialist policy generator 116 may receive base policy 104 from base policy generator 102 and update (e.g., refine, adjust, etc.) base policy 104 for a specific type of robot. For example, specialist policy generator 116 may output multiple specialist policies (e.g., type-specific policy, second action policy, residual reinforcement learning model, etc.) using base policy 104. Before specialist policy generator 116 receives base policy 104, the weights of at least one of the base policy 104 or world model 110 may be frozen. Each specialist policy may correspond to a type of robot. Specialist policy generator 116 may apply residual RL to base policy 104 and progressively refine base policy 104 into specialist policies. Because the specialist policy generator 116 uses the base policy 104 updated based on imitation learning to generate specialist policies, it can efficiently integrate type-specific constraints and sensor flows with fewer interactions compared to generating specialist policies without using the base policy 104 updated based on imitation learning. Utilizing residual RL, the specialist policy generator 116 can generate specialist policies to address robot-type-specific kinematics, sensor configurations, and constraints that the base policy 104 may not capture.

[0049] To generate a specialist strategy, the specialist strategy generator 116 may include at least one input generator 118. The input generator 118 may generate at least one state of the robot in the simulation (e.g., As described above, this state may include at least the following information: camera inputs simulating the robot, measured velocities (e.g., for each joint), route information, and a type embedding indicating the robot's type. This state may also include, but is not limited to, the robot's position in the environment. This state may include at least the robot's potential state and route information. For example, input generator 118 may input this state into world model 110, and world model 110 may generate encodings that at least indicate the robot's potential state and route information.

[0050] Specialty policy generator 116 may include at least one residual RL updater 120. Residual RL updater 120 may receive states from input generator 118 and generate and update specialty policies 122 (e.g., robot type-specific action policies, second policies, etc.) using a base policy 104. Specialty policies 122 may be machine learning models (e.g., neural networks, etc.), such as multilayer perceptrons (MLPs). Each specialty policy 122 may correspond to one of multiple robot types. For example, residual RL updater 120 may receive a base policy 104 and use the base policy 104 as the basis for each specialty policy 122, such as a first specialty policy 122A and a second specialty policy 122B. For example, the first specialty policy 122A may correspond to a quadruped robot and output actions for the quadruped robot. For example, the second specialty policy 122B may correspond to a humanoid robot and output actions for the humanoid robot. Each specialist policy 122 can receive at least one state of one of the multiple robots as input and output specialist actions (e.g., a second action, a robot type-specific action, etc.) to move at least one of the multiple robots. The specialist policy 122 can be a residual RL model, and the specialist actions can be output using residual RL. The specialist policy 122 can be tailored for a specific robot type.

[0051] Specialized policy 122 can be based on base policy 104. For example, specialized policy 122 can share the same feature extraction layer as base policy 104, and the weights of base policy 104 can be copied to specialized policy 122, and the final output layer of specialized policy 122 can have weights reinitialized (e.g., reset) from base policy 104. Specialized policy generator 116's reinitialization of only the weights of the final output layer ensures training stability and focuses updates for each specialized policy 122 on residual reinforcement learning updates and corrections to base policy 104 necessary to achieve type-specific performance of specialized policy 122. By leveraging base policy 104, specialized policy generator 116 can reduce problems such as sparse rewards and sample complexity while accelerating the convergence of the weights for each specialized policy 122.

[0052] To refine at least one specialist policy 122 for a specific robot type (e.g., updating the weights of the final output layer), a residual RL updater 120 can receive a state from an input generator 118 that indicates the type embedding and apply that state to the specialist policy 122 to update the weights of the specialist policy 122 (e.g., the weights of the final output layer). Each specialist policy 122 can receive a state as input and output a specialist (e.g., residual) action (e.g., specialist action 214) using residual RL. The specialist policy 122 can be a residual RL model. The specialist policy 122 can output a specialist action that refines (e.g., adapts, modifies) the base action output by the base policy 104 to suit the corresponding robot type. For example, let This is the basic output of the basic policy 104 (e.g., basic action, first action, etc.). A residual policy can be introduced. (For example, Specialist Strategy 122), its output is (For example, the fourth action, a specialized action). The final action (for example, a combination of actions, the second action) can be: (7) .

[0053] In various embodiments, hybrid or gating mechanisms (e.g., weighted sums) may be used. For example, the final action could be a weighted sum of a basic action and a specialist action. π res Its function is to adapt the basic strategy 104 to the subtle differences in the dynamics, kinematics, and constraints of a specific robot type. For example, It can Adapt to the corresponding type of robot, thereby The specific dynamics, kinematics, and constraints of the corresponding type of robot are adapted. In various embodiments, the base policy 104 outputs base actions in parallel with the specialist actions output by specialist policy 122.

[0054] Input generator 118 can generate one or more states, which can be input into specialist policy 122. Specialist policy 122 can output specialist actions, which can be combined with basic actions output by base policy 104 to form a final action (e.g., ...). The final action (e.g., a combination of the first and second actions) can be transmitted to the simulated robot to make the robot move, and the result of the simulated robot can be recorded and used to update the weights of the specialist policy 122. Each specialist policy 122 can correspond to a different type of robot and can be trained independently. Therefore, each specialist policy 122 can generate specialist actions for a type of robot. In various embodiments, the specialist policies 122 can be updated in parallel. In various embodiments, the specialist policy generator 116 includes a world model 110 that can receive states from the input generator 118 and generate codes indicating those states. In this case, the codes are input into the base policy 104 and the specialist policy 122 to output the base action and the specialist action, respectively.

[0055] System 100 may include at least one general action policy generator 124. Once the weights of each specialist policy 122 converge, input generator 118 may provide additional state to each specialist policy 122 and record at least the inputs and outputs of each specialist policy 122. For example, residual RL updater 120 may record at least the latent state and route information, avatar identifiers, and the mean and variance of the Gaussian action distribution of specialist actions from world model 110 (e.g., included in the encoding). Residual RL updater 120 may log each record to the specialist dataset 126 of each corresponding specialist policy 122. For example, a first specialist policy 122A may have a corresponding first specialist dataset 126A, and a second specialist policy 122B may have a corresponding second specialist dataset 126B. Residual RL updater 120 may record the specialist actions generated by each specialist policy 122 to the corresponding specialist dataset 126 in response to the state provided by input generator 118. In various embodiments, the residual RL updater 120 may record the talent dataset 126 when updating the weights of the talent policy 122.

[0056] A general action policy generator 124 can generate at least one general action policy 130 (e.g., a third action policy, etc.) through a combination of a base policy 104 and a specialist policy 122. For example, the general action policy generator 124 may include at least one distiller 128 (e.g., a compressor, condenser, extractor, etc.). The general action policy 130 may be a machine learning model (e.g., a neural network, etc.), such as a multilayer perceptron (MLP). The general action policy generator 124 can input a specialist dataset 126 into the distiller 128, which can distill the specialist dataset 126 into the general action policy 130. By distilling the specialist dataset 126 into the general action policy 130, the distiller 128 can maintain the potential processing pipeline provided by the specialist policy generator 116, while further adjusting the output action based on type embedding. The distiller 128 can use at least one computing system (e.g., NVIDIA's DGX system) to implement an simulation platform (e.g., NVIDIA's Omniverse System for Modular Operations, OSMO) to generate the general action policy 130.

[0057] The general motion strategy 130 can be applied to various types of robots (e.g., multiple types of robots) while maintaining the performance of the specialized strategy 122 for each type of robot. For example, the general motion strategy 130 can receive the robot's state and identifiers (e.g., type embedding, avatar embedding, encoding, etc.) and generate motions for the robot. These motions can then be transmitted to a controller to move the robot based on them. The general motion strategy 130 can generate these motions based on the state and identifiers.

[0058] The actions output by the general motion strategy 130 may include multiple speed commands, and each speed command may correspond to a joint of the robot. Therefore, in order to perform the action on the robot, the general motion strategy 130 may map at least one speed command to a corresponding joint of the robot. In various embodiments, at least one of the following may be employed: the action may include the mapping, or the controller may map speed commands to corresponding joints. Speed ​​commands may include at least one of an identifier of the corresponding joint, an association with the corresponding joint, or a correspondence. Speed ​​commands may be transmitted to each joint to guide and move the robot.

[0059] refer to Figure 2 , Figure 2This is an example system 200 for updating a talent strategy 122 according to some embodiments of this disclosure. It should be understood that the arrangements and other arrangements described herein are merely illustrative. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of the arrangements, components, features, and elements shown, and some elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and at any suitable location (e.g., on a local device, vehicle, or edge machine, locally deployed (e.g., a locally hosted server), remotely (e.g., in one or more computing or server devices in one or more data centers in the cloud), and / or elsewhere). The various functions performed by the entities described herein may be implemented by hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, microcontroller, embedded processor, digital signal processor (DSP), image signal processor (ISP), physical processing unit (PPU), field-programmable gate array (FPGA), accelerators (e.g., deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural network accelerator (NNA) and / or neural processing unit (NPU), programmable vision accelerator (PVA), optical flow accelerator (OFA), etc.), application-specific integrated circuits (ASIC), data processing unit (DPU), quantum processor, etc.) that execute instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 17A-17E Example machine 1700 Figure 18 Example calculation of ecosystem 1800, Figure 19 Example generative language model systems 1900 and / or Figure 20 The example computing device 2000 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the operation.

[0060] System 200 may be included in system 100. For example, system 200 may be an example of talent strategy generator 116. In various embodiments, talent strategy generator 116 may be system 200 or include system 200.

[0061] To update the weights of the final output layer of specialist policy 122 (e.g., to refine specialist policy 122 for each robot type), system 200 may include an input generator 118. Input generator 118 may include multiple inputs to be provided to world model 110. For example, input generator 118 may include at least one image 202 (e.g., raw observations of the robot), at least one state 204 of the robot, and at least one route 206 of the robot (e.g., route information, navigation target, etc.). The robot's state 204 may include, but is not limited to, the robot's current position and joint velocities in the simulated environment. Image 202 may correspond to state 204, and input generator 118 may generate various combinations of route 206 and state 204 to be provided to world model 110. For example, image 202 may include the viewpoint of the robot in its current state 204.

[0062] Before updating the weights of specialist policy 122, base policy generator 102 can update and freeze the weights of world model 110 and base policy 104. In system 200, the weights of world model 110 and base policy 104 can be frozen. World model 110 can receive at least one of image 202, state 204, or route 206 from input generator 118 and output policy label 208 (e.g., encoding, latent state). World model 110 can determine the robot's latent state based on at least one of image 202, state 204, or route 206, and encode that latent state to generate policy label 208. Policy label 208 can at least indicate the robot's latent state (e.g., state). World model 110 can determine the robot's latent state, encode the robot's route 206 provided by input generator 118, and generate policy label 208 including the latent state and route encoding.

[0063] System 200 may include at least one basic motion generator 210. Basic motion generator 210 may include a world model 110 and a basic policy 104. World model 110 may provide policy tags 208, indicating state and route encoding, as input to basic policy 104. Basic policy 104 may use policy tags 208 to generate basic actions 212 (e.g., nominal actions, first actions, etc.). Policy tags 208 may be input into basic policy 104, and basic policy 104 may output basic actions 212. Basic actions 212 may serve as navigation baselines for various types of robots, such as for obstacle avoidance, and may be applied to different types of robots. Basic actions 212 may be represented as... ,in Basic movement 212, Basic Strategy 104 The strategy marker 208 may be a sign indicating at least one of image 202, status 204, or route 206.

[0064] System 200 may include at least one specialist policy 122, and specialist policy 122 may receive policy tags 208 from world model 110. In various embodiments, specialist policy 122 may receive policy tags 208 in parallel or sequentially from a base policy 104 that receives policy tags 208. Specialist policy 122 may use policy tags 208 to generate specialist actions 214 (e.g., residual actions, correction terms, adaptive actions, etc.). Specialist policy 122 may utilize residual reinforcement learning to generate specialist actions 214, and specialist actions 214 may be directed to a type of robot. For example, system 200 may include multiple specialist policies 122, each for a different type of robot (e.g., corresponding to a different type of robot). System 200 may update multiple specialist policies 122 in parallel or sequentially. System 200 may include and update one specialist policy 122 for each type of robot. Specialist action 214 may be represented as... ,in It is a specialist action 214. It is the specialist strategy 122. It can be policy label 208, which indicates at least one of image 202, state 204, or route 206. Specialist policy 122 can be a residual reinforcement learning policy (e.g., a model).

[0065] System 200 may include at least one speed command generator 216 (e.g., speed motion generator, etc.). Speed ​​command generator 216 may receive basic motions 212 from basic strategy 104 and specialized motions 214 from specialized strategy 122. Speed ​​command generator 216 may combine basic motions 212 and specialized motions 214 to generate speed motions 218 (e.g., speed commands, actions, third actions, speeds per joint, etc.). For example, speed command generator 216 may... The basic motion 212 and the specialized motion 214 are combined. In various embodiments, the velocity command generator 216 combines the basic motion 212 and the specialized motion 214 using a weighted sum to generate a velocity motion 218. The velocity motion 218 may include the velocity of each joint of the robot, and the mapping of each velocity to the corresponding joint of the robot. The velocity motion 218 may be a combination of the basic motion 212 and the specialized motion 214.

[0066] System 200 may include at least one controller 220. Controller 220 may be a controller for a simulated robot in the simulation. Controller 220 can control the robot's joints, thereby controlling the robot's movement. A velocity command generator 216 can send a velocity action 218 to controller 220. Controller 220 can execute the velocity action 218 on the simulated robot in the simulation and cause the simulated robot to move according to the velocity action 218. Controller 220 can send velocity actions 218 for different types of robots. Controller 220 can record the robot's movement as it moves according to the velocity action 218, and can record the robot's starting and ending positions, as well as the path taken by the robot. Controller 220 can record the robot's interaction with the simulated environment. Controller 220 can generate and output a result 222 indicating the robot's movement. Result 222 may include, but is not limited to, the robot's starting position, ending position, path, and contact with other objects in the simulated environment.

[0067] In various embodiments, controller 220 may be pre-trained to map velocity commands to robot joints. For example, controller 220 receives velocity action 218 and is configured to map velocity action 218 to corresponding joints of the robot. In this case, velocity command generator 216 may not generate velocity action 218 that includes a mapping of velocity action 218 to robot joints.

[0068] System 200 may include at least one result evaluator 224 (e.g., a result analyzer, etc.). Result evaluator 224 may receive results 222 from controller 220. Result evaluator 224 may generate a reward 226 based on at least one of the following (but not limited to): progress to a target (e.g., navigation target, path information), collision avoidance (e.g., contact with other objects), or target completion (e.g., reaching the navigation target). Progress to the target may be a positive reward proportional to the distance between the simulated robot and the target, indicated in at least one of the route information or status. Collision avoidance may be a negative reward (e.g., penalty) for contact with an object (e.g., collision) or other dangerous actions (e.g., robot joint pinching). Target completion may be a positive reward for reaching the navigation target, greater than the reward for progress to the target. Result evaluator 224 may calculate (e.g., determine, generate, etc.) the reward 226 based on, for example, the sum or weighted sum of progress to the target, collision avoidance, and target completion.

[0069] The result evaluator 224 can generate observation results 228. Observation results 228 may include, but are not limited to: the efficiency of the route taken by the robot to reach the target, the robot's speed, the complexity of the environment (e.g., the number of objects, stairs, etc.) or the number of moving objects in the environment, such as other simulated robots. Observation results 228 may include the robot's next state (e.g., x) after the completion of velocity action 218. t+1 ).

[0070] The result evaluator 224 can send rewards 226 to the specialist policy 122, which can be used to update the weights of a layer of the specialist policy 122 (e.g., the final output layer). In some embodiments, the result evaluator 224 can provide rewards 226 to the input generator 118, which in turn can provide rewards 226 to the specialist policy 122. The weights of the specialist policy 122 can be adjusted to maximize positive rewards (e.g., goal completion) and reduce negative rewards (e.g., collision avoidance). The weights of the specialist policy 122 can be updated using proximal policy optimization (PPO). For example, the weights of the specialist policy 122 can be updated using gradients (e.g., gradient-based methods). In various embodiments, the result evaluator 224 can update the weights of the specialist policy 122 based on the complexity of the policy label 208. For example, outcome evaluator 224 can determine the difficulty of at least one of the environment, state complexity, etc. encoded in policy label 208, and weight the reward 226 according to the complexity of policy label 208, thereby updating the weight of specialist policy 122.

[0071] System 200 may include at least one data logger 230 (e.g., a memory, etc.). Data logger 230 may include at least one database, array, table, or other storage medium, or may be communicatively coupled to a storage medium. Data logger 230 may record at least one of the following: policy tag 208, speed action 218, reward 226, or observation result 228 output by world model 110. In various embodiments, data logger 230 may also record at least one of image 202, state 204, route 206, basic action 212, specialist action 214, or result 222. Data logger 230 may record and associate each output of system 200. For example, data logger 230 may record policy tag 208, speed action 218, observation result 228, reward 226, and the association between each of policy tag 208, speed action 218, observation result 228, and reward 226. Specifically, data logger 230 can record the association between policy tag 208 and at least one of basic action 212, specialist action 214, speed action 218, result 222, reward 226, or observation result 228. Data logger 230 can record the inputs (e.g., policy tag 208) and outputs (e.g., speed action 218 and result 222) of basic policy 104 and specialist policy 122.

[0072] During the updating of specialist policies 122, data logger 230 may record the input and output distributions of each specialist policy 122. For example, data logger 230 may record at least one of the latent state and route encodings of the world model output (e.g., in policy label 208), the robot's type identifier e, and the mean and variance of the Gaussian action distribution used for PPO (e.g., updating the weights of specialist policy 122). Each specialist policy 122 may output a normal distribution (e.g., a state-action distribution) of specialist actions 214. In various embodiments, the type identifier e is indicated in policy label 208 and is a part of policy label 208. Data logger 230 may record the input and output distributions to generate a specialist dataset 126 corresponding to each specialist policy 122.

[0073] In various embodiments, system 200 updates specialist policies 122 in parallel. In this case, based on type identifier e, system 200 provides policy tags 208 to specialist policies 122 corresponding to robot types with type identifier e. Therefore, input generator 118 can generate states 204 with different type identifiers, and specialist policies 122 can be updated in parallel. In various embodiments, specialist policies 122 are updated sequentially, and input generator 118 can generate the same type identifier for state 204 until the weights of specialist policies 122 corresponding to the same type identifier converge. After convergence, input generator 118 can generate different type identifiers for state 204 to update another specialist policy 122 corresponding to a different robot type.

[0074] In various embodiments, the input generator 118 provides an image 202, a state 204, and a route 206 after the weights of the specialist strategy 122 converge to record the velocity action 218 and the result 222 corresponding to each specialist strategy 122 for different types of robots.

[0075] Data logger 230 can record the inputs and outputs of system 200 for each iteration. For example, input generator 118 can provide image 202, state 204, and route 206 until the weights of specialist policies 122 (e.g., each specialist policy 122) converge. Each iteration of system 200 may include: receiving x t Then obtain from basic strategy 104 Obtained from the talent strategy 122 Perform combined actions (e.g., velocity action 218) in the simulation. t Observe the next state x t+1 And the reward R, and update using gradient-based methods. At the same time, maintain It has been frozen.

[0076] In various embodiments, system 200 may continuously update the specialist strategies 122 until the weights of each specialist strategy 122 converge. For example, system 200 may be automated and may include, for example, an input generator 118 that generates at least one of an image 202, a state 204, or a route 206 after receiving a reward 226 from the result evaluator 224. To this end, system 200 may include or implement at least one simulation platform (e.g., NVIDIA's ISAAC Lab, OSMO, Omniverse Extensions (OVX)) for generating at least one of the robot's image 202, state 204, or route 206, and includes a controller 220 for executing velocity actions 218 and recording the results 222.

[0077] refer to Figure 3 , Figure 3 This is an example system 300 for generating and updating a general action policy 130 according to some embodiments of this disclosure. It should be understood that the arrangements and other arrangements described herein are merely illustrative. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to or in place of the arrangements, components, features, and elements shown, and some elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and at any suitable location (e.g., on a local device, vehicle, or edge machine, locally deployed (e.g., a locally hosted server), remotely (e.g., in one or more computing or server devices in one or more data centers in the cloud), and / or elsewhere). The various functions performed by the entities described herein may be implemented by hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, microcontroller, embedded processor, digital signal processor (DSP), image signal processor (ISP), physical processing unit (PPU), field-programmable gate array (FPGA), accelerators (e.g., deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural network accelerator (NNA) and / or neural processing unit (NPU), programmable vision accelerator (PVA), optical flow accelerator (OFA), etc.), application-specific integrated circuits (ASIC), data processing unit (DPU), quantum processor, etc.) that execute instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 17A-17E Example machine 1700 Figure 18 Example calculation of ecosystem 1800, Figure 19 Example generative language model systems 1900 and / or Figure 20 The example computing device 2000 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the operation.

[0078] System 300 may be included in system 100. For example, system 300 may be an example of a general action policy generator 124. In various embodiments, the general action policy generator 124 may be system 300 or include system 300. System 300 may receive output from system 200, which may be an example of a specialist policy generator 116.

[0079] System 300 may include specialist datasets 126 for different robot types, which may be merged into a single general motion policy 130 by distiller 128. Specialist datasets 126 may be provided by specialist policy generator 116. General motion policy 130 may capture the collective knowledge of all specialist policies 122 and utilize type embedding to generate actions based on robot type, as will be further described below.

[0080] To generate a general action policy 130, the distiller 128 can combine and distill the specialist dataset 126 into the general action policy 130. For example, It can be used for the first i Specialized strategies 122 for each type of robot. Each specialized strategy 122 can generate a normal distribution of actions (e.g., at least one of specialized action 214 or velocity action 218). (e.g., multiple normal distributions), where z is the latent state and route encoding. The general action policy 130 can be defined as... Given z and the avatar embedding e, it can output To match the normal distribution of each specialist strategy 122, the Kullback-Leibler (KL) divergence can be minimized by the following equation: (8) in It comes from the first i A dataset of state-action distributions recorded by a specialist policy 122 (e.g., the first specialist dataset 126A), while e i It is the corresponding type embedding. i It can be with the first i The type embedding corresponding to the robot type of the specialist policy 122. Therefore, the combination of normal distributions (e.g., specialist dataset 126) output by the specialist policy 122 can be distilled into a general action policy 130 at least by minimizing the divergence of multiple normal distribution combinations.

[0081] In various embodiments, the weights of the specialist policy 122 are updated to converge and freeze before generating a normal distribution. In other embodiments, the normal distribution, as well as the inputs and outputs of each specialist policy 122, are recorded while the weights of the specialist policy 122 are being adjusted and converged.

[0082] The general motion policy 130 can maintain the same underlying processing pipeline as the basic policy 104 and the specialist policy 122, and further condition it with type embedding before generating the final action. For example, the general motion policy 130 can maintain the expert actions emulated by the basic policy 104, while adapting the actions according to the robot type.

[0083] like Figure 3 As shown, system 300 may include an input generator 118 that can provide at least one of an image 202, a state 204, or a route 206 (but is not limited thereto) to world model 110. World model 110 can receive at least one of the image 202, state 204, or route 206 (but is not limited thereto) and output a policy tag 208, as shown in reference. Figure 2 As described.

[0084] System 300 may include at least one type encoder 302. Type encoder 302 may receive robot types 304 (e.g., multiple robot types) and generate type embeddings 306 (e.g., embedding e, identifier, etc.) based on robot types 304. Robot types 304 may be associated with states 204 generated by input generator 118. Robot types 304 may include multiple robot types associated with each specialty policy 122. For example, robot types 304 may include each robot type 304 corresponding to specialty policy 122. As another example, in response to a first specialty policy 122A corresponding to a quadruped robot and a second specialty policy 122B corresponding to a humanoid robot, robot types 304 may include at least quadruped robots and humanoid robots. Robot types 304 may be a database of robot types 304 corresponding to specialty policies 122 or specialty datasets 126. In various embodiments, type encoder 302 may randomly select (e.g., extract, etc.) robot types 304 in parallel or sequentially with input generator 118 to provide input to world model 110. In some embodiments, state 204 includes type embedding 306.

[0085] Type encoder 302 can generate type embedding 306 based on robot type 304. Type embedding 306 may include an embedding e associated with each specialty dataset 126. Type embedding 306 may correspond to robot type 304, and for robots of the same robot type 304, type embedding 306 may be the same. To generate type embedding 306, type encoder 302 may include, for example, a table or guide. For example, type encoder 302 may include a table that maps robot type 304 to type embedding 306. In various embodiments, robot type 304 may correspond to state 204, and type encoder 302 may generate type embedding 306 based on state 204 generated by input generator 118.

[0086] The general motion policy 130 can receive policy tags 208 and type embeddings 306 from the world model 110 and type encoder 302, respectively. The general motion policy 130 can generate actions using policy tags 208 and type embeddings 306. The general motion policy 130 can maintain potential processing (e.g., potential processing of the world model 110 and the base policy 104) and adjust based on the type embeddings 306 to generate actions. For example, the general motion policy 130 can be adjusted using the type embeddings 306 to generate decisions based on robot type 304. The action output by the general motion policy 130 can be a base action 212, modified from a specialized action selected using the type embeddings 306, and this action can be tuned for specific kinematics and constraints of robot type 304 associated with the type embeddings 306. The general motion policy 130 can generate actions including the base action based on policy tags 208 and refinements of that base action based on the type embeddings 306. The general motion policy 130 can be used to generate motions for any robot type 304 associated with the specialist dataset 126, and the generated motions can be tuned using type embedding 306. The general motion policy 130 can mimic specialist motions 214 corresponding to robot type 304 by using type embedding 306.

[0087] Figure 4 Image 402A is depicted, which can serve as an example of image 202 in the simulation. Image 402A can be input into world model 110, and world model 110 can use image 402A to determine potential states, generate route codes, and generate policy labels 208, which can include both potential states and route codes.

[0088] Figure 5 Examples of different robot types 304 are depicted. For example... Figure 5 As shown, robot type 304 may include at least humanoid robots and quadruped robots.

[0089] Figure 6 State 204 is depicted, where the robot's simulated environment includes multiple robots. In this case, world model 110 can predict transitions of multiple robots in the environment, thereby generating policy labels 208. Actions output by at least one of the base policy 104, specialist policy 122, or general action policy 130 can reflect predictions of robot movement. For example, using the predicted movements of multiple robots, general action policy 130 can generate actions to avoid multiple robots in the environment. Specialist policy 122 can be updated with a scenario including multiple robots, allowing the weights of specialist policy 122 to be optimized for scenarios with multiple moving objects.

[0090] Figure 7 Experimental setup 700 depicts an example residual RL update environment for specialist policy 122. This environment may include multiple tiled regions that can extend in parallel to accelerate sampling and updating for each specialist policy 122. This environment can be generated by a simulator such as NVIDIA's ISAAC Lab and can be a scalable visual RL environment. The tasks provided to the simulated robot (e.g., actions from action database 108) may include point-to-point indoor navigation in a wide variety of environments (e.g., warehouse, office, laboratory, etc.). In experimental setup 700, wheeled, humanoid, and quadrupedal robot types 304 are evaluated to cover a broad range of physical characteristics. In the initial imitation learning phase (e.g., base policy generator 102), a pre-trained base policy 104 for standard wheeled navigation is used. The weights of the base policy 104 are frozen for subsequent RL refinement. In the RL phase (e.g., specialist policy generator 116), each robot type 304 is trained in a uniform environment that includes randomized obstacle layouts and target placement. Each training round (e.g., iteration) may include up to 256 time steps. The robot's movement may terminate early upon a collision or upon reaching its maximum length (e.g., 256 time steps). In this experimental setup, approximately 320 trajectories per robot type 304 are recorded from specialist policy 122. Each trajectory can be recorded for 128 steps, resulting in approximately 40,000 frames per robot type 304. Distillation (e.g., by distiller 128) can then be performed by matching the output distribution of each specialist policy 122.

[0091] Figure 8 Depicting Figure 7 Table 800 shows the results of the experimental setup 700. Table 800 includes results based on success rate (SR) and weighted time to travel (WTT). SR can indicate the fraction of a run that reaches the target without collisions or timeouts (e.g., maximum time steps). WTT can indicate the average time or number of steps to reach the target successfully (e.g., target completion) and can be weighted by SR. As shown in Table 800, the general motion strategy 130 outperforms both the basic strategy 104 and the specialized strategy 122 across different robot types 304 and different simulation environments.

[0092] refer to Figure 9-1 0, Figure 9-10 is an example system 900 for generating and updating a basic policy 104 according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of the arrangements, components, features, and elements shown, and certain elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and at any suitable location (e.g., on a local device, vehicle, or edge machine, locally deployed (e.g., a locally hosted server), remotely (e.g., in one or more computing or server devices in one or more data centers in the cloud), and / or elsewhere). The various functions performed by the entities described herein may be implemented by hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, microcontroller, embedded processor, digital signal processor (DSP), image signal processor (ISP), physical processing unit (PPU), field-programmable gate array (FPGA), accelerators (e.g., deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural network accelerator (NNA) and / or neural processing unit (NPU), programmable vision accelerator (PVA), optical flow accelerator (OFA), etc.), application-specific integrated circuits (ASIC), data processing unit (DPU), quantum processor, etc.) that execute instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 17A-17E Example machine 1700 Figure 18 Example calculation of ecosystem 1800, Figure 19 Example generative language model systems 1900 and / or Figure 20 The components, features, and / or functions of the example computing device 2000 are similar to those of other components, features, and / or functions to perform the same function. Detailed descriptions (including drawings and examples) of U.S. Patent Application No. 18 / 921,863 (filed October 21, 2024) are incorporated herein by reference in their entirety.

[0093] Figure 9A system 900 for performing end-to-end navigation according to various embodiments is illustrated. The system 900 includes a data generation pipeline 902, a training engine 904, and an execution engine 906. System 900 may be included in system 100. For example, a base policy generator 102 may include or may be system 900. System 900 may be an example of base policy generator 102. As discussed herein, the data generation pipeline 902 generates synthetic data that can be used to train, evaluate, test, and / or otherwise operate a multimodal generative world model 942, other types of machine learning models (which may be used by an autonomous mobile robot (AMR) (e.g., robot 1700) and / or other types of machines to perform tasks), machine hardware configurations, and / or other components of the machine. Generative world model 942 may correspond to world model 110. Generative world model 942 may be an example of world model 110. In some embodiments, generative world model 942 is an example of base action generator 210. The data generation pipeline 902 may be an example of the data generator 106 and / or the input generator 118. The training engine 904 and the execution engine 906 may be examples of the basic policy updater 114.

[0094] In some embodiments, the data generation pipeline 902 generates synthetic datasets for training (e.g., updating), evaluating, and / or testing a multimodal generative world model 942, other types of machine learning models that can be used by AMR and / or other machine types to perform tasks, the machine's hardware configuration, and / or other components of the machine. As discussed herein, the data generation pipeline 902 can be configured and / or customized with various parameters to generate data that captures different scenarios related to navigation and / or other types of tasks performed by the machine in the environment. For example, at least one of the data generator 106 or the input generator 118 may include the data generation pipeline 902 for generating at least actions, images 202, states 204, and routes 206 to update the base policy 104 and the specialist policy 122, respectively.

[0095] In some embodiments, training engine 904 and execution engine 906 may include functionality for training and executing a multimodal generative world model 942 to perform end-to-end navigation and / or other tasks for AMR and / or other types of machines. The multimodal generative world model 942 may directly map inputs (e.g., but not limited to camera images (e.g., image 202), velocities, global guidance, and / or robot states (e.g., state 204)) to multimodal outputs, such as (e.g., but not limited to) semantic segmentation, paths, and / or navigation commands (e.g., basic actions 212). These multimodal outputs can then be used to perform machine navigation, generate machine-associated future state predictions, simulate machine operation, and / or perform other machine-related tasks.

[0096] The data generation pipeline 902 may include at least one simulator 922 that generates multiple sets of simulation data 932(1)-932(X) (each set of simulation data is individually referred to as simulation data 932 herein, where X may be a number (e.g., an integer)). For example, simulator 922 may perform physical simulations of various environments surrounding AMR and / or other types of machines. During these physical simulations, simulator 922 may generate simulation data 932, which includes (but is not limited to) rendered images of the environment surrounding the machine (e.g., visualizations from the perspective and / or bird's-eye view of one or more cameras on the machine), semantic labels associated with the images (e.g., segmentation maps, detected objects, bounding shapes, etc.), the state of the machine (e.g., position, heading, velocity, etc.) and / or occupancy maps of free space and / or occupied space in the environment.

[0097] The data generation pipeline 902 may include at least one target generator 924 that determines a set of targets 934(1)-934(Y) associated with the simulation data 932 (each target is individually referred to as target 934 herein, where Y may be a number). For example, the target generator 924 may generate target locations within a given occupancy map output by the simulator 922 in order to navigate to those locations in the appropriate environment.

[0098] The data generation pipeline 902 may include at least one planner 926 that generates various commands 936(1)-936(Z) that can cause the machine to take (e.g., receive) one or more corresponding actions based on simulation data 932 and / or target 934. Z may be a number. For example, the planner 926 may generate commands 936 that include (but are not limited to) linear velocities and / or angular velocities that cause the machine to move toward a target 934 from the target generator 924 while avoiding obstacles in the environment represented by simulation data 932 from the simulator 922.

[0099] At least one set of commands 936 output by planner 926 can be sent to simulator 922. Simulator 922 can update the machine state, rendered image, semantic labels, occupancy map, and / or other simulation data 932 based on the commands 936 provided by planner 926. Simulator 922 can then send some or all of the updated simulation data 932 to planner 926, so that planner 926 can generate a new set of commands 936 based on the updated simulation data 932 and the corresponding target 934 received from target generator 924. This process can be repeated until target 934 is reached, a certain number of time steps have been executed in a given simulation, and / or other conditions indicating the end of the simulation are met.

[0100] The data generation pipeline 902 may include at least one data logger 928 that aggregates simulation data 932, targets 934, commands 936, and / or other data generated by the simulator 922, target generator 924, and planner 926 into multiple records 938(1)-938(N) (each record is individually referred to as record 938 herein, where N may be a number). For example, the data logger 928 may record data from the simulator 922, target generator 924, and / or planner 926 into record 938 in the order in which the corresponding events occur in the corresponding simulation (e.g., in time steps, simulation data “frames” 932, and / or other discrete-time representations). The data logger 928 may also, or alternatively, downsample and / or resample (e.g., based on space and / or time) some or all of the data in record 938 to reduce and / or modify the size of the recorded data.

[0101] At least one post-processor 930 in the data generation pipeline 902 can adapt simulated data 932, targets 934, commands 936, records 938, and / or other data generated by other components of the data generation pipeline 902 to various machine learning models and / or use cases. For example, post-processor 930 can resample, compress, format, and / or otherwise transform data in a given recordset 938 into a form that can be used to train and / or evaluate machine learning models, hardware configurations, and / or other components of one or more machines. Each recordset 938 post-processed for a given purpose and / or in some way can be stored in one or more datasets 940(1)-940(K) (each individually referred to as dataset 940 herein, where K may be a number) for subsequent retrieval and use.

[0102] In some embodiments, the data generation pipeline 902 can be configured and / or customized through different types of configuration parameters. For example, configuration parameters may include a unique name and / or identifier for a given scenario (e.g., a combination of specific environments, machines, objectives, strategies, etc.) in which data will be generated and collected. Configuration parameters can also be used to customize the type of environment and / or machine to be simulated, the objective, the planner type, the type of data to be recorded, the data recording frequency, and / or the way the recorded data is converted into a format suitable for training and / or evaluating machine learning models and / or other components of the machine. Different sets of configuration parameters can be used to launch different instances of the data generation pipeline (e.g., in parallel on multiple nodes of a distributed system) to generate data capturing different scenarios related to navigation and / or other types of tasks performed by the machine in the environment.

[0103] Training engine 904 can update one or more machine learning models 916 using training data 908, which originates from one or more datasets 940 generated by data generation pipeline 902, data collected by the machine in real-world environments, simulations performed using, for example, simulator 922, and / or other data sources. Machine learning models 916 can be included in expert policy database 112. Figure 9 As shown, training data 908 may include training state data 912 representing the state associated with the machine and / or the environment. Training data 908 may include an action database 108. For example, training state data 912 may include sensor data captured by virtual and / or real sensors on the machine (e.g., images, lidar, radar, audio data, ultrasonic data, inertial measurement unit (IMU) data, etc.), representations of the machine's surrounding environment (e.g., visualizations, semantic segmentation, point clouds, meshes, environment types, environment descriptions, scene descriptions using generic scene descriptor (USD) data (e.g., OpenUSD), the machine's linear velocity and / or angular velocity, the machine type and / or machine model associated with the machine, global guidance associated with the machine's navigation and / or other tasks or objectives 934, and / or other information that can be used to characterize the machine's state and / or the machine's surrounding environment.

[0104] Training data 908 may include training action data 910, which represents the actions the machine is to perform in the environment. For example, training action data 910 may include "truth" actions, "teacher" action policies, commands, routes, trajectories, paths, and / or other indications of actions the machine will perform during perception, planning, control, prediction, navigation, manipulation, and / or other tasks. For example, training action data 910 may include an expert policy database 112.

[0105] In some embodiments, the machine learning model 916 can be used to perform and / or guide tasks performed by a machine. For example, the machine learning model 916 may include: tree-based models such as decision trees, random forests, and gradient boosting trees; feedforward neural networks, convolutional neural networks (CNNs), recurrent neural networks (RNNs), residual neural networks, long short-term memory networks (LSTMs), graph neural networks, transformer neural networks, diffusion models, generative adversarial networks (GANs), language models (large language models (LLMs), visual language models (VLMs), multimodal language models (MMLMs), etc.), neural rendering field (NeRF) models, and / or other types of neural networks; and / or support vector machines (SVMs), logistic regression models, hierarchical models, ensemble models, Bayesian networks, Naive Bayes classifiers, and / or other types of model architectures. The machine learning model 916 may also or alternatively include rules, filters, heuristics, logic programming, semantic networks, search techniques, named entity recognition techniques, and / or other symbolic models. Each machine learning model can be used to generate embeddings, semantic segmentation, reconstruction and / or prediction of sensor data, classification outputs, security alerts, trajectories, commands, and / or other outputs related to one or more corresponding tasks.

[0106] During the training of the machine learning model 916, the training engine 904 can input at least one training state data 912 into the machine learning model 916. The training engine 904 can process the input training state data 912 using the model parameters 914 of the machine learning model 916 (e.g., neural network weights) and obtain a training output 918, which includes predictions from one or more layers, blocks, or components of the machine learning model 916 associated with the training state data 912. The training engine 904 can use the training output 918, the training state data 912, and / or the training action data 910 to compute one or more losses 920. The training engine 904 can use training techniques (e.g., gradient descent and backpropagation) to iteratively update the model parameters 914 of the machine learning model 916 in a manner that reduces the losses 920.

[0107] In one or more embodiments, the execution engine 906 may use at least one trained machine learning model 916 to implement a generative world model 942, which can be used to perform end-to-end navigation and / or other tasks for the machine 960 in real-world, simulated, digital twin, and / or other types of environments. For example, the machine 960 may include a quadruped robot, a humanoid robot, a differential drive system, an Ackerman drive system, a warehouse robot, a delivery robot, a forklift, and / or other types of AMRs. The machine 960 may also, or alternatively, include autonomous or semi-autonomous vehicles, drones, submarines, water vehicles, and / or other types of vehicles with navigation capabilities. The generative world model 942 may be deployed on the machine 960 using a runtime platform (such as NVIDIA's TensorRT) for real-time inference, which accelerates and optimizes performance through quantization, layer and tension fusion, kernel tuning, GPU-based execution, streaming audio and / or video, and / or concurrent execution.

[0108] The generative world model 942 can operate on inputs such as (but not limited to) sensor data 964 from machine 960 (e.g., camera images, LiDAR data, radar data, audio data, velocity, state, etc.), task-related global guidance 966 (e.g., path, trajectory, route, destination, etc.), and / or other representations of machine 960 and / or its surrounding environment. Given these inputs, the generative world model 942 can generate embedded features 944, historical records 946, states 948, action policies 950, and / or outputs 952 related to machine 960 and / or its surrounding environment. The embedded features 944, historical records 946, states 948, action policies 950, and / or outputs 952 generated by the generative world model 942 can be additionally used to determine one or more actions 962 to be performed by machine 960 during task execution. At least one action policy 950 may correspond to a base policy 104.

[0109] Figures 10A-10B Based on the various embodiments Figure 9 A more detailed explanation of the generative world model 942 in [the text is missing]. Figure 10A As shown, the generative world model 942 includes an observation module 1022 (e.g., an observer), a prediction module 1024 (e.g., a predictor), a decoding module 1026 (e.g., a decoder), and an action policy module 1028 (e.g., an action policy generator). Each of these components will be described in more detail in this paper.

[0110] The observation module 1022 can iteratively generate and / or update a set of states 948(1)-948(10) based on observation results in the form of sensor data 964(1)-964(2) received from the machine 960. Within the observation module 1022, the first encoder 1002 can convert the first type of sensor data 964(1) into a first set of embedded features 944(1), and the second encoder 1004 can convert the second type of sensor data 964(2) into a second set of embedded features 944(2).

[0111] In one or more embodiments, encoder 1002 may convert sensor data 964(1) associated with the current time step t in the form of one or more images (e.g., from one or more cameras on machine 960, one or more cameras outside machine 960, visualizations generated by combining multiple camera views of the environment around machine 960, etc.) into vectors, matrices and / or another embedded feature set 944(1) in a lower-dimensional latent space. The encoder 1004 can convert sensor data 964(2) associated with the machine at the same time step in the form of one or more machine states (e.g., machine type, machine model, linear velocity, angular velocity, position, orientation, configuration, etc.) into another vector, matrix, and / or another set of embedded features 944(1) in a different lower-dimensional latent space. .

[0112] The feature compressor 1006 can convert embedded features 944(1)-944(2) into a third set of embedded features 944(10) associated with the same time step. For example, feature compressor 1006 may include a neural network and / or other type of machine learning model that transforms two embedded feature sets 944(1)-944(2) into new vectors, matrices, and / or alternative representations of embedded feature 944(10) in a latent space different from the latent space of embedded features 944(1)-944(2). In another example, feature compressor 1006 may generate embedded feature 944(10) as a concatenation, summation, averaging, and / or other aggregation or combination of embedded features 944(1)-944(2).

[0113] The posterior estimator 1012 in the observation module 1022 can generate a set of states 948(1)-948(2), which represent the state at the current time step. t The world surrounding the machine. For example... Figure 10A As shown, the posterior estimator 1012 can generate the first state 948 (1) based on the input. The input may include at least one of the following: (i) an embedded feature set 944 (10) from the feature compressor; (ii) in the previous time step t -1 One or more actions performed by a machine 962(1) ; or (iii) the historical record of potential states up to the previous time step 946(1) During a certain number of initial time steps when executing the generative world model 942, state 948(1) may be generated without action 962(1) and historical record 946(1) due to a lack of information related to any previous time step. After state 948(1) is generated, it may be spliced ​​with and / or otherwise combined with historical record 946(1) by splicer 1009 until the previous time step (e.g., in response to historical record 946(1) being available), thereby generating a second potential state 948(2). This second potential state can be associated with the current time step and capture the "world" around the machine up to the current time step.

[0114] Decoding module 1026 may include a set of decoders 1008 and 1010 that convert latent state 948(2) into a set of multimodal outputs 952. More specifically, decoder 1008 may convert state 948(2) into a first output 952(1) corresponding to the reconstruction of image-based sensor data 964(1). Decoder 1010 may convert state 948(2) into a second output 952(2) corresponding to semantic segmentation of image-based sensor data 964(1). These outputs 952(1)-952(2) may be used to train components of generative world model 942 and / or perform other tasks, as will be discussed in detail below.

[0115] The motion strategy module 1028 may include an encoder 1020 that converts routes, trajectories, paths, headings and / or other global guidance 966 associated with the task to be performed by the machine into an embedded feature set 944 (4) of the current time step. These embedded features 944 (4) and the state 948 (2) at the same time step can be input into the self-attention module 1018. The self-attention module 1018 can convert the input embedded features 944 (4) and state 948 (2) into a fusion strategy state 948 (10). The fusion policy state 948 (10) can be decoded by a neural network (or other type of machine learning model) that implements the basic action policy 104 (e.g., action policy 950) into one or more actions 962 (2) at the current time step.

[0116] The prediction module 1024 may include a prior estimator 1014, which generates the next time step after the current time step based on the input. t +1 state 948 (4) The input may include at least one of the following: (i) the history of the current time step 946 (2) Or (ii) one or more actions associated with the current time step 962(2) (For example, as generated by the action strategy module 1028). The history record 946(2) can be generated by the gated loop unit (GRU) 1016 based on the history record 946(1) including the history up to the previous time step. and state 948(1) The input is generated. The prior estimator 1014 can correspond to the posterior estimator 1012, such that the prior estimator 1014 and the posterior estimator 1012 have the same model architecture. State 948 (4) can be combined (e.g., spliced) by the splicer 1009 with the history 946 (2) to produce the latent state 948 (5). The latent state is associated with the next time step and represents a prediction of the “future” world around the machine at the next time step. State 948 (5) can be used to train the multimodal generative world model 942, (e.g., using decoders 1008 and / or 1010 in the decoding module 1026) to be decoded into a corresponding output (not shown) associated with the next time step, and / or to perform other tasks associated with the next time step.

[0117] The prediction process associated with prediction module 1024 can be targeted at t +1 followed by an additional future time step t +2、 t +10 and so on are repeated. For example, state 948(4) And historical records 946 (2) It can be processed by GRU 1016 to generate an updated history for the next time step 946(2). Potential state 948 (5) Alternatively, the action policy module 1028 can be used for processing to generate a new fusion policy state for the next time step. Then, the new fusion policy state can be transformed into a new set of actions for the next time step. Furthermore, the updated history and new action sets can be used to generate future time steps. t +2 new state and the corresponding potential states Then, the decoding module 1026 can decode the potential state into a value corresponding to a future time step.t +2 output 952. This process can be repeated to generate additional predictions for each subsequent future time step by utilizing the state 948 associated with the previous time step, the historical record 946 up to the previous time step, and the action 962 associated with the previous time step.

[0118] In one or more embodiments, the operation of the generative world model 942 can be represented as a partially observable Markov decision process (POMDP), which models probabilistic belief states and solves the decision-making problem through staggered observations and actions. This POMDP can be composed of tuples. Definition, where This represents the state space associated with one or more states 948. This represents the action space associated with one or more actions 962, and This is the observation space associated with sensor data 964. Transition function. It can respond to the action being taken. a From state Transition to state The probability is modeled. Observation function This can represent the action being applied. a And transition to state Later observed o The probability of reward function. It can be defined in the state s Next action a The reward, and It is a discount factor. The solution to POMDP can include the optimal policy. This strategy maximizes the expected cumulative reward. ,in and These represent the time of machine 960. t The state and actions.

[0119] In some embodiments, the observation module 1022 and the prediction module 1024 can learn a transformation function for model prediction. and the observation function used for observation correction The action policy module 1028 aims to approximate the optimal policy by mimicking a near-optimal one. Teacher strategies are used to solve POMDP.

[0120] More specifically, the prior estimator 1014 can learn the state transition by modeling a given state 948(4) as a normal distribution with diagonal covariance: (9) Historical transitions are represented as follows: (10)

[0121] The posterior estimator 1012 can capture both state transitions and observation corrections, and its corresponding state 948(1) is also estimated as a normal distribution with diagonal covariance: (11) in This represents embedded feature 944 (10) generated by encoders 1002 and 1004 and feature compressor 1006 from sensor data 964 and / or other input observations. Historical record 946 (1) and state 948(1) They can be pieced together to form a 1-D potential state 948(2). This 1-D latent state can be used for multi-task decoding.

[0122] In one or more embodiments, the transformation learned by the prior estimator 1014 and the posterior estimator 1012, and represented by equations 9-11, can be modeled using a neural network. For example, It can be implemented as a GRU 1016, and in the prior estimator 1014 and the posterior estimator 1012 This can include multilayer perceptrons (MLPs). This article will combine... Figure 10B The prior estimator 1014 and the posterior estimator 1012 are discussed in more detail.

[0123] Figure 10B Further illustrations are provided according to various embodiments. Figure 10A The estimator model in question is 1042. More specifically, Figure 10B The diagram illustrates the model architecture of estimator model 1042, which can correspond to... Figure 10A The posterior estimator 1012 and / or prior estimator 1014 in the model.

[0124] In response to the estimator model 1042 corresponding to the posterior estimator 1012, one or more actions 962(1) associated with the previous time step can be processed by the MLP 1044. To generate higher-dimensional feature states. The feature states output by MLP 1044, historical records 946(1) And the embedded feature 944(10) of the current time step t. It can be input into the normal distribution model 1046 in the posterior estimator 1012.

[0125] The normal distribution model 1046 can include an MLP, which estimates the mean 1048 of the current time step. Sum of standard deviation 1050 Sampler 1052 can sample from a normal distribution with a mean of 1048 and a standard deviation of 1050 to generate the corresponding state 948 for the current time step. As described in this article, status 948 It can be compared with the historical record 946 (1) Combine them to generate the corresponding potential state 948(2). .

[0126] In response to the estimator model 1042 corresponding to the prior estimator 1014, one or more actions 962(1) associated with the current time step. (For example, the action policy module 1028 based on the potential state 948(2) received from the decoding module 1026) (The determined state) can be processed by the MLP 1044 to generate a higher-dimensional feature state. The feature state output by the MLP 1044 and the historical data up to the current time step are then considered. 946(2) can be input into the normal distribution model 1046. Since the observations of future time steps are not available, the embedded features of the next time step can be omitted. As input to the normal distribution model 1046. The normal distribution model 1046 can generate the mean 1048 for the next time step. Sum of standard deviation 1050 Furthermore, sampler 1052 can sample from the corresponding distribution to generate the corresponding state 948 for the next time step. This process can be repeated for additional time steps following the next time step.

[0127] Encoder 1002 may correspond to a machine learning model that generates an embedded feature set 944(1) for one or more input images included in sensor data 964(1). For example, encoder 1002 may include a visual transformer (ViT) (or other type of machine learning model) trained using self-supervised techniques. A one-dimensional vector corresponding to the embedded feature 944(1) can be generated by concatenating the class labels generated by ViT from the input images with a set of average pooled patch labels generated by ViT from the input images. .

[0128] The encoder may correspond to a machine learning model that generates different embedded feature sets 944(2) for one or more machine states included in the sensor data 964(2). For example, the encoder 1004 may include a fully connected neural network (or other type of machine learning model) that converts another representation of the linear velocity, angular velocity, and / or machine state included in the sensor data 964(2) into another vector corresponding to the embedded features 944(2). .

[0129] Feature compressor 1006 may include neural network layers and / or operations that concatenate and / or otherwise combine two embedded feature sets 944(1) and 944(2) into a third vector corresponding to a third embedded feature set 944(10). These embedded features 944(10) may include time steps t Potential representations of associated observations.

[0130] In some embodiments, decoders 1008 and 1010 may generate decoded outputs 952(1) and 952(2), respectively, to ensure consistency with potential state 948(2). The associated latent spatial capture provides information that machine 960 can use to perform navigation and / or other tasks. For example, decoder 1008 may include a diffusion model (or other type of machine learning model) that reconstructs one or more input images included in sensor data 964 (1). The denoising process of the diffusion model may be in latent state 948 (2). As a condition, the mean squared error (MSE) and / or the difference between the input image and the corresponding reconstructed result output by the diffusion model can be used to train the decoder 1008, posterior estimator 1012, feature compressor 1006, and / or encoder 1002 in an end-to-end manner.

[0131] In another example, decoder 1010 may include a generative adversarial network (GAN) (or other type of machine learning model) that is capable of processing latent states 948(2). The semantic segmentation is converted into a semantic segmentation included in output 952 (2). The semantic segmentation may correspond to one or more images included in sensor data 964 (1), a perspective view associated with machine 960, and / or another representation of the environment surrounding machine 960. The cross-entropy loss (or another measure of the difference between the output semantic segmentation and the ground truth semantic segmentation of the corresponding environment) may be computed pixel-by-pixel at each upsampled resolution of the decoder 1010 output. The computed loss may then be used to train decoder 1010, posterior estimator 1012, feature compressor 1006, encoder 1002, and / or encoder 1004 in an end-to-end manner.

[0132] In one or more embodiments, the Kullback-Leibler (KL) divergence between the prior distribution output by the prior estimator 1014 and the corresponding posterior distribution output by the posterior estimator 1012 can be calculated (e.g., for the same time step). The KL divergence can be used to train the prior estimator 1014 (e.g., update the weights of the prior estimator 1014) to match the prior distribution with the posterior distribution, thereby enabling the generative world model 942 to predict future states consistent with the observed data.

[0133] As described herein, action policy module 1028 can use latent state 948(2). The encoded representation of global guidance 966 is used to generate one or more actions 962(2). To incorporate route information, the global route included in global guidance 966 can be transformed to the local reference frame of machine 960 and truncated into regional route segments near machine 960. These regional route segments can then be represented as a tensor containing a series of route poses with x and y positions.

[0134] Encoder 1020 may include VectorNet (or other types of machine learning models) that transform the tensor into a vector corresponding to the embedded features 944(4). These embedded features 944 (4) can capture route information associated with global guidance 966, while providing the flexibility to encode additional attributes (e.g., final destination markers) that enable machine 960 to perform navigation and / or other tasks.

[0135] Next, the self-attention module 1018 can convert the latent state 948(2). And embedded features 944(4) Merge into strategy state 948 (10) Then, the policy state 948 (10) can be decoded by an MLP (or other type of machine learning model) implementing one or more action policies 950 as: one or more actions 962 (2). These actions specify the linear and angular velocities in the x, y, and z directions, and / or the navigation path. The navigation path includes five path poses in the local reference frame of machine 960. The MLP can be trained using L1 loss (or other differential measure), which is computed between action 962 (2) and the corresponding action output by the teacher action policy (not shown) to make the MLP mimic the teacher action policy.

[0136] In one embodiment, the training of the generative world model 942 can be divided into multiple phases. In the first training phase, the action policy module 1028 can be omitted, and actions from the teacher's action policy are used to train the observation module 1022, prediction module 1024, and decoding module 1026 using the corresponding loss. After the training of the observation module 1022, prediction module 1024, and decoding module 1026 is completed (e.g., after a certain number of training steps, iterations, batches, and / or cycles have been performed; the parameters of the machine learning models in the observation module 1022, prediction module 1024, and decoding module 1026 have converged; the loss is below a threshold; and / or other conditions are met), during the second training phase, the action policy module 1028 can be trained end-to-end with the observation module 1022, prediction module 1024, and decoding module 1026.

[0137] Although Figure 10A The generative world model 942 is shown to process two types of sensor data 964(1)-964(2) (e.g., images and robot state) and generate two types of outputs 952(1)-952(2) (e.g., image and semantic segmentation), but it will be understood that the generative world model 942 is capable of operating with a variety of types and / or combinations of inputs. For example, sensor data 964 associated with the environment surrounding machine 960 may include (but is not limited to) images, depth maps, point clouds, meshes, audio data, temperature data, weather data, traffic data, and / or proximity data. In another example, sensor data 964 associated with the state of machine 960 may include (but is not limited to) accelerometer data, gyroscope data, odometry data, log data, performance data, event data, and / or error data collected by machine 960. Each type of sensor data 964 can be converted into a corresponding embedded feature set by a different encoder. The individual embedded feature sets can then be further aggregated, combined, and / or otherwise processed to produce a potential representation of the observations at the corresponding time step.

[0138] In another example, various components included in the decoding module 1026 and / or action strategy module 1028 may generate different types of outputs 952 based on the corresponding latent state 948 generated by the observation module 1022 and / or prediction module 1024. These outputs 952 may include (but are not limited to) reconstruction results of images, depth maps, point clouds, and / or other sensor data 964 used to generate the latent state 948. These outputs may also, or alternatively, include (but are not limited to) semantic segmentation, detected objects and / or instances, bounding shapes, occupancy maps, paths, trajectories, linear velocities and / or angular velocities, obstacle avoidance and / or collision avoidance actions, fault handling actions, and / or other predictions and / or actions associated with the sensor data 964.

[0139] Figure 11A The illustrations show the relationship between various embodiments and Figure 9 The generative world model 942 is associated with a set of example inputs and outputs. More specifically, Figure 11A Example sensor data 964(1), outputs 952(1)-952(2) and action 962 associated with three different time steps 1102, 1104 and 1106 are illustrated.

[0140] Sensor data 964(1) includes images captured by cameras on machine 960 at each time step 1102, 1104, and 1106. For example, each image may be captured by an AMR corresponding to machine 960 while the robot is navigating in a warehouse environment.

[0141] Outputs 952(1) and 952(2) include the image reconstruction results and the semantic segmentation results associated with these images at the same time steps 1102, 1104 and 1106, respectively. As described herein, outputs 952(1)-952(2) can be generated by the decoder included in the generative world model 942 based on the latent state 948 representing the sensor data 964 associated with time steps 1102, 1104 and 1106.

[0142] Action 962 includes representations of linear and angular velocities at time steps 1102, 1104, and 1106, which can be sent as commands to machine 960 during a navigation task. The magnitude of the linear velocity is depicted in the left-hand bar chart, and the magnitude and direction of the angular velocity are depicted in the right-hand bar chart.

[0143] Figure 11B The illustrations show the relationship between various embodiments and Figure 9A set of example inputs and outputs associated with the generative world model 942. The inputs include global guidance 966, which is in the form of a route to be taken by machine 960. Global guidance 966 can be specified in the context of a bird's-eye view 1112 of the environment around machine 960, a map of the environment around machine 960, and / or other representations of the environment around machine 960 (e.g., in response to the availability of such representations).

[0144] Given global guidance 966 and sensor data 964 including a camera view of machine 960 at a given time step, generative world model 942 can generate an action 962 to be performed at that time step. Action 962 may include linear velocity and / or angular velocity, path, trajectory, and / or other motion indicators associated with machine 960. Figure 11B As shown, the path corresponding to action 962 differs slightly from the global guidance 966. Therefore, global guidance 966 can be used to inform navigation tasks performed using the generative world model 942 without requiring the navigation task to strictly follow the specified route.

[0145] Figure 11C The illustrations show the relationship between various embodiments and Figure 9 The generative world model 942 is associated with a set of example inputs and outputs. More specifically, Figure 11C Example sensor data 964 and outputs 952(1)-952(3) associated with three different environments 1122, 1124, and 1126 surrounding machine 960 are shown. Sensor data 964 includes images of environments 1122, 1124, and 1126 (e.g., images captured by a camera on machine 960). Output 952(1) includes a semantic segmentation generated by decoding a latent state 948 associated with a time step of 0.2 seconds after the time the corresponding image was captured. Output 952(2) includes a semantic segmentation generated by a latent state 948 associated with a time step of 1 second after the time the corresponding image was captured. Output 952(3) includes a semantic segmentation generated by a latent state 948 associated with a time step of 2 seconds after the time the corresponding image was captured. These latent states 948 may be generated by a priori estimator 1014 based on sensor data 964 received from, for example, machine 960, as a representation of the “future” world surrounding machine 960. In semantic segmentation, different regions can represent accessible surfaces, fences, pallets, forklifts, signs, and / or other types of objects depicted in images.

[0146] The latent state 948 representing a future time step and the corresponding decoded output 952 can be used to train the various components of the generative world model 942. After training, the observation module 1022 and the action policy module 1028 can be used by the machine 960 to perform inference during a given task (e.g., navigation) based on sensor data 964 received from the machine 960 corresponding to the machine 960's observations. The latent state 948 generated by the prediction module 1024 and / or the corresponding output 952 of the future time step can be used to perform tasks such as (but not limited to) running simulations, performing safety checks (e.g., detecting and responding to potential hazards), and / or clarifying and / or interpreting the predictions generated by the generative world model 942 and / or the behavior of the machine 960.

[0147] It should be understood that the arrangements and other arrangements described herein are merely illustrative. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of the arrangements, components, features, and elements shown, and certain elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and at any suitable location (e.g., on a local device, vehicle, or edge machine; locally deployed (e.g., a locally hosted server); remotely located (e.g., in one or more computing or server devices in one or more data centers in the cloud); and / or in other locations). The various functions performed by the entities described herein may be implemented by hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, microcontroller, embedded processor, digital signal processor (DSP), image signal processor (ISP), physical processing unit (PPU), field-programmable gate array (FPGA), accelerators (e.g., deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural network accelerator (NNA) and / or neural processing unit (NPU), programmable vision accelerator (PVA), optical flow accelerator (OFA), etc.), application-specific integrated circuits (ASIC), data processing unit (DPU), quantum processor, etc.) that execute instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 17A-17E Example machine 1700 Figure 18 Example calculation of ecosystem 1800, Figure 19 Example generative language model systems 1900 and / or Figure 20The example computing device 2000 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the operation.

[0148] For reference Figure 12 Each block of the method 1200 described herein may include a computational process, which may be executed using any combination of hardware, firmware, and / or software. For example, various functions may be implemented by a processor executing instructions stored in memory. The method may also be embodied as computer-usable instructions stored on a computer storage medium. The method may be provided by a standalone application, service, or managed service (independently or in combination with another managed service), or a plug-in to another product, to name just a few. Furthermore, this document provides examples regarding... Figures 1-3 or Figure 9 The system described herein describes method 1200. However, this method may be performed additionally or alternatively by any system or any combination of systems, including but not limited to the systems described herein. Furthermore, without departing from the scope of this disclosure, the operations in method 1200 may be omitted, repeated, and / or performed in any order.

[0149] Figure 12 A flowchart of a method 1200 for performing end-to-end navigation using a generative world model, according to various embodiments, is shown. Figure 12 As shown, method 1200 begins with operation 1202, in which training engine 904 can determine training state data and training action data associated with one or more machines in one or more environments. For example, training engine 904 may receive training state data and training actions from data generation pipeline 902, one or more datasets collected from real-world machines interacting with real-world environments, one or more synthetic datasets, and / or other data sources. Training state data may characterize the machine and / or the environment surrounding the machine. Training action data may include the machine's ground truth action policy, actions the machine is to perform based on the corresponding state data, and / or other indications of the behavior required by the machine when performing one or more tasks.

[0150] In operation 1204, training engine 904 can generate training outputs (e.g., training outputs 918) associated with one or more tasks the machine will perform in the environment, based on training state data (e.g., training state data 912), via a generative world model (e.g., generative world model 942). For example, training engine 904 can input training state data into the generative world model. Training engine 904 can also use the generative world model to generate embedded features, states, decoded outputs, actions, and / or other training outputs based on the input training state data.

[0151] In operation 1206, training engine 904 can train the generative world model based on one or more losses (e.g., loss 920), which are calculated using training state data, training action data (e.g., training action data 910), and / or training outputs. Continuing the example above, training engine 904 can calculate L1 loss, MSE, cross-entropy loss, and / or other measures of the difference between the decoded outputs and / or actions and their corresponding ground truth values. Training engine 904 can also, or alternatively, calculate KL divergence and / or other measures of the difference between the posterior distribution associated with the state of the posterior estimator output in the generative world model and the prior distribution associated with the state of the prior estimator output in the generative world model. Training engine 904 can also further update the parameters of the individual components of the generative world model based on the corresponding losses.

[0152] In various embodiments, the training engine 904 can train the generative world model in multiple training phases. During a first training phase, the training engine 904 can use one or more losses to train the neural networks and / or other machine learning models included in the observation module, decoding module, and / or prediction module of the generative world model. After the first training phase is completed, the training engine 904 can execute a second training phase, which uses appropriate losses to train the action policy module, as well as the observation, decoding, and prediction modules of the generative world model in an end-to-end manner.

[0153] In operation 1208, execution engine 906 can transform a set of sensor inputs received by the machine into embedded feature sets via one or more encoders (e.g., encoders 1002, 1004, 1020) included in a trained generative world model. For example, execution engine 906 can use different encoders to transform each type of sensor input into a corresponding embedded feature set in a lower-dimensional latent space. Execution engine 906 can also use a feature compressor to aggregate and / or otherwise combine multiple embedded feature sets corresponding to various types of sensor inputs into a single embedded feature set representing all observations made by the machine at the current time step.

[0154] In operation 1210, execution engine 906 may generate one or more states based on embedded features, a history of previous states, and / or a set of previous actions by executing a posterior estimator (e.g., posterior estimator 1012) included in a trained generative world model. For example, execution engine 906 may initially (e.g., during each time step included within a certain number of initial time steps) only transform embedded features into latent states.

[0155] In operation 1212, execution engine 906 transforms one or more states into a set of predictions. Continuing the example above, execution engine 906 can use the decoding module in a trained generative world model to transform latent states into reconstructed images, point clouds, and / or other representations of the machine's surrounding environment. Execution engine 906 can also, or alternatively, use the decoding module and / or action policy module to transform latent states into semantic segmentation, a set of actions, and / or another type of prediction associated with the machine and / or environment.

[0156] In operation 1214, execution engine 906 can cause a machine (e.g., robot 1700) to perform a set of actions based on the prediction results. Continuing the example above, execution engine 906 can send the predicted actions as commands associated with linear velocity, angular velocity, and / or other types of motion (e.g., forward motion, backward motion, left turn, right turn, etc.) to the machine. The machine can execute the sent commands, thereby driving the machine to perform a task.

[0157] In operation 1216, execution engine 906 can determine whether to continue using the machine and / or the trained generative world model to perform the task. For example, execution engine 906 can determine that the task should continue using the machine and / or the trained generative world model if the navigation (or other type of) task has not been completed and / or a certain amount of time has elapsed since the task was assigned to the machine. When execution engine 906 determines that the task should continue, it repeats operations 1208, 1210, 1212, and 1214 to generate additional states, predictions, and / or actions for subsequent time steps. After a certain number of time steps, execution engine 906 can perform operation 1210 by generating one or more states associated with the current time step using the history of previous states up to the previous time step, a set of previous actions associated with the previous time step, and an embedded feature set associated with the current time step. Execution engine 906 can perform operation 1216 after a certain number of time steps and / or at other frequencies to determine whether to continue the task. The Execution Engine 906 can use generative world models and machines to perform tasks until the task is completed, the task has "timed out," and / or other conditions are met.

[0158] Figure 13 Based on the various embodiments Figure 9 A more detailed description of the data generation pipeline 902 is provided below. As described herein, the data generation pipeline 902 can generate synthetic data that can be used to train, evaluate, test, simulate and / or otherwise operate the generative world model 942, other types of machine learning models that can be used by AMR and / or other types of machines to perform tasks, the hardware configuration of the machine, and / or other components of the machine.

[0159] In the data generation pipeline 902, simulator 922 generates and / or updates simulation data 932 related to one or more machines and / or one or more environments surrounding the machines. For example... Figure 13 As shown, the simulation data 932 may include (but is not limited to) occupancy figure 1312, odometer value 1314, image 1316, semantic label 1318 and / or enclosing shape 1320 (e.g., box, square, rectangle, polygon, etc.).

[0160] Occupancy map 1312 may include representations of vacant and occupied spaces in the environment. For example, occupancy map 1312 associated with a given simulation may include a two-dimensional (2D) and / or three-dimensional (3D) grid representing the environment surrounding the machine. In the grid, each cell may be associated with a binary value indicating whether the corresponding spatial area is occupied (e.g., occupied by obstacles, objects, etc.). Each cell may also be associated with, or alternatively with, the probability that the corresponding spatial area is occupied. Each cell may also be associated with, or alternatively with, a numerical “cost” used to quantify the difficulty of moving within the corresponding spatial area. Occupancy map 1312 may also include, or alternatively replace, point clouds, meshes, and / or other representations of “occupied spaces” in the environment.

[0161] The odometer value 1314 may include numerical values ​​associated with the motion of the machine. For example, the odometer value 1314 of a machine in a given simulation may indicate the distance traveled by the machine, the machine's position, the machine's heading, the machine's linear velocity and / or angular velocity, the machine's linear acceleration and / or angular acceleration, and / or other information that can be used to derive and / or estimate the machine's position and / or orientation in the corresponding environment.

[0162] Image 1316 may include a visual representation of the environment surrounding the machine. For example, image 1316 may depict the environment from the perspective of cameras and / or other sensor modalities on the machine. Image 1316 may also, or alternatively, include a bird's-eye view of the environment, a perspective view of the environment, a 10130-degree visualization of the environment, and / or other depictions of the environment outside the machine and / or outside the individual cameras on the machine. Image 1316 may include color values ​​per pixel, depth values, normal values, motion vectors (e.g., motion vectors between consecutive frames of a video), LiDAR intensity values, and / or other types of information that can be used to characterize the environment.

[0163] Semantic labels 1318 may include indications of categories, objects, and / or other attributes that help in understanding the environment. For example, semantic labels 1318 may include semantic segmentation, which labels individual pixels in image 1316, points in a point cloud, polygons in a mesh, and / or other representations of the environment with corresponding categories. Semantic labels 1318 may also, or alternatively, identify objects, object instances, and / or other entities found in the various images, point clouds, meshes, and / or other representations of the environment.

[0164] The bounding shape 1320 may include a representation of the position and / or size of an object in the environment. For example, the bounding shape 1320 may include a rectangular outline of an object in image 1316. The bounding shape 1320 may also, or alternatively, include a parallelepiped outline of an object in a point cloud and / or other 10D representation of the environment. Each bounding shape may be associated with a class label, instance, and / or other indications of the corresponding object.

[0165] In one or more embodiments, simulator 922 may generate at least a portion of simulation data 932 using physical simulation and / or realistic rendering of the machine and / or environment. These physical simulations and / or realistic renderings may be performed using a physically based virtual environment (e.g., NVIDIA ISAAC Sim (NVIDIA ISAAC Sim™, NVIDIA ISAAC Gym™, and / or NVIDIA Drive Sim™, all registered trademarks of NVIDIA Corporation)) built on the NVIDIA Omniverse platform (NVIDIA Omniverse Sim™ is a registered trademark of NVIDIA Corporation). The simulation environment may support loading robot models (e.g., quadruped robots, humanoid robots, differential drive systems, Ackerman drive systems, forklifts, etc.) and / or sensors (e.g., cameras, LiDAR, IMUs, etc.), randomization of the environment and / or environmental properties (e.g., lighting, reflection, color, position, etc.), adding objects to the environment, and / or specifying the physical, material, and / or collision properties of objects.

[0166] As described herein, target generator 924 can identify one or more targets 934 associated with simulation data 932. For example, target generator 924 can generate a target location in a given occupancy map output by simulator 922 so that navigation to that target location is possible within the appropriate environment.

[0167] In some embodiments, the target generator 924 may generate some or all of the targets 934 based on corresponding target parameters 1302. For example, target parameters 1302 may specify that navigation-based targets will be randomly sampled from the airworthiness free space within the occupancy map 1312 generated by the simulator 922. Target parameters 1302 may also, or alternatively, specify one or more regions of the occupancy map 1312 from which targets 934 are preferentially sampled, and / or specify attributes of these regions (e.g., regions with more "details" and / or obstacles). The target generator 924 may use these target parameters 1302 to sample targets 934 more frequently from the corresponding regions, thereby increasing the coverage of tasks associated with these regions in the synthetic data.

[0168] Planner 926 may use one or more strategies 1304 to generate commands 936 that instruct the machine in the simulation executed by simulator 922 to perform actions related to target 934. For example, planner 926 may implement and / or execute action strategies 1304 that generate commands 936 based on target 934 from target generator 924 and odometer values ​​1314 and / or other information from simulator 922. Each strategy may include a planning stack, a teacher strategy, and / or generate commands 936 to operate other components of the machine based on the machine's state and / or the environment surrounding the machine. These commands 936 may include (but are not limited to) linear and / or angular velocities, trajectories, paths, and / or other motion instructions to move the machine toward a target 934 from target generator 924 while avoiding obstacles in the corresponding simulated environment.

[0169] Each set of commands 936 output by planner 926 can be sent to simulator 922, which updates simulation data 932 based on the corresponding actions. For example, planner 926 can generate a given set of commands 936 based on simulation data 932 associated with a given time step in the simulation. These commands 936 can be sent to simulator 922, which generates updated simulation data 932 for the next time step. The simulation data 932 for the next time step reflects changes in the machine and / or environment after the machine executes the actions corresponding to the commands 936. Simulator 922 can then send some or all of the updated simulation data 932 to planner 926, allowing planner 926 to generate a new set of commands 936 based on the updated simulation data 932 and the corresponding target 934 from target generator 924. This process can be repeated until target 934 is reached, a certain number of time steps have been executed in the simulation, and / or other conditions indicating the end of the simulation are met.

[0170] Data logger 928 can aggregate simulation data 932, target 934, command 936, and / or other data generated by simulator 922, target generator 924, and planner 926 into record 938 of events associated with the corresponding time step. For example, simulator 922, target generator 924, and / or planner 926 may include nodes implementing publishers and / or associated with them in a publish-subscribe messaging system (e.g., Robot Operating System (ROS)). Each publisher can publish messages and / or events associated with the corresponding component of data generation pipeline 902 (e.g., simulator 922, target generator 924, planner 926, etc.) to one or more topics. Data logger 928 may include nodes implementing subscribers to these topics in the publish-subscribe messaging system and / or associated with them. Each subscriber can receive messages from one or more corresponding topics. Data logger 928 can log data from received messages by preprocessing data 1306 and storing the preprocessed data in record 938.

[0171] In one or more embodiments, preprocessing 1306 may include: determining the order in which data and / or events occur in a given simulation; generating records 938 that are published across a time interval and / or at a frequency; downsampling some or all of the recorded data; and / or other data processing operations associated with data from simulator 922, target generator 924, and / or planner 926. For example, data logger 928 may synchronize data published at different frequencies by simulator 922, target generator 924, and planner 926 by associating the published data with individual “frames” of time, time interval, time step, and / or other discrete-time metrics in each simulation. Data logger 928 may also store the data associated with each discrete-time metric in one or more records corresponding to that time metric. In another example, data logger 928 may downsample image 1316, semantic tags 1318, bounding shapes 1320, and / or other high-resolution data from simulator 922 before storing the data in record 938. In the third example, data logger 928 may store records 938 associated with a given scenario (e.g., a combination of a specific environment, machine, objective, strategy, simulation, etc.) along with a path and / or directory corresponding to that scenario. Data logger 928 may also, or alternatively, associate each record 938 with a unique identifier and / or name of the corresponding scenario.

[0172] In some embodiments, data logger 928 may generate visualizations and / or charts of the data in record 938 when record 938 is created. For example, data logger 928 may output images 1316, semantic labels 1318, enclosing shapes 1320, occupancy maps 1312, odometer values ​​1314, and / or other visual representations of simulated data 932 in a graphical user interface. Data logger 928 may also, or alternatively, output "pins," routes, and / or other representations of targets 934 and / or guidance information associated with targets 934 in the corresponding occupancy map 1312, a bird's-eye view of the simulated environment, and / or other visual depictions of the environment. Data logger 928 may also, or alternatively, output commands 936 and / or paths, trajectories, and / or other visual representations of actions performed based on commands 936 as overlays on images 1316, maps, and / or other representations of the environment. This output information allows users to visually review the recorded data, determine whether the recorded data accurately reflects the corresponding scenario, and / or determine whether the recorded data can be used for various use cases and / or applications.

[0173] Post-processor 930 can perform post-processing 1308, which adapts records 938 and / or other data generated by other components of the data generation pipeline 902 to various machine learning models and / or use cases. For example, post-processor 930 can resample, compress, smooth, format, and / or otherwise transform data in a given set of records 938 into a form (e.g., file format, schema, etc.) that can be used to train, test, and / or evaluate machine learning models, hardware configurations, digital twins, and / or other components of physical and / or virtualized machines. Post-processor 930 can also, or alternatively, store each set of records 938 that has been post-processed for a given purpose and / or in a certain way in one or more corresponding datasets 940.

[0174] In some embodiments, post-processor 930 may generate and store metadata associated with the data recorded in dataset 940. For example, post-processor 930 may store the following in association with a dataset for a given scenario: the number of instances of object types (e.g., forklifts, shelves, people, etc.) in the scenario; the time interval between consecutive frames represented by records 938 in the dataset; the distance traveled by the machine in the scenario; the distribution of actions performed by the machine; and / or other metrics and / or statistics associated with the simulated operation of the machine in the scenario. In another example, post-processor 930 may specify target parameters 1302, strategies 1304, preprocessing 1306 and / or postprocessing 1308 techniques, and / or other types of configuration parameters 1322 used to generate the dataset in the metadata of a given dataset.

[0175] In some embodiments, some or all components of the data generation pipeline 902 may be configured and / or customized via configuration parameters 1322 provided by the control module 1310. For example, configuration parameters 1322 may specify the machine type and / or model, the machine's initial posture, the scene, one or more objects in the scene, object attributes, and / or other information that may be used by the simulator 922 for simulation. Configuration parameters 1322 may also, or alternatively, include target parameters 1302 for controlling the target generator 924 to generate targets 934. These target parameters 1302 may specify the type of target 934 to be generated (e.g., location-based target, task, etc.), the sampling technique used to generate the target 934, the regions within the environment from which the target 934 is preferentially sampled, the attributes of the regions within the environment from which the target 934 is preferentially sampled, the weights and / or other importance measures associated with sampling the target 934 from the various regions of the environment, the time at which one or more new targets 934 will be sampled (e.g., after one or more existing targets 934 have been reached), and / or other parameters that may be used to control and / or modify the target generator 924 in generating targets 934. Configuration parameter 1322 may also or alternatively include: specific strategies 1304 that the planner 926 will use to generate commands 936, behavioral attributes associated with these strategies 1304 (e.g., the level of initiative and / or conservatism when performing tasks and / or achieving goals; the type of commands 936 to be generated; minimum, maximum, and / or valid values ​​associated with commands 936; etc.), text-based and / or code-based instructions for strategies 1304, the platform and / or framework for implementing strategies 1304, and / or other information that may be used to implement strategies 1304 and / or generate commands 936. Configuration parameter 1322 may also or alternatively include parameters related to the publication and / or subscription of topics by components of the data generation pipeline 902. Configuration parameter 1322 may also or alternatively include identifiers, paths, recording frequency, downsampling parameters, resampling parameters, file formats, modes, visualization types, and / or other information that may be used to perform preprocessing 1306 and / or postprocessing 1308 associated with data in records 938 and / or datasets 940.

[0176] Configuration parameters 1322 can be defined and / or updated using various techniques. For example, configuration parameters 1322 can be provided by one or more users via one or more configuration files, application programming interfaces (APIs), user interfaces, and / or other mechanisms. Some or all of the configuration parameters 1322 can also be randomly generated (e.g., by sampling from the distribution, range, and / or set of valid configuration parameters 1322). Some or all of the configuration parameters 1322 can also be generated and / or updated using machine learning, optimization, and / or search techniques (e.g., to increase the coverage of dataset 940 to the environment and / or scene and / or generate synthetic data relevant to a specific environment and / or scene). Control module 1310 can send configuration parameters 1322 to simulator 922, target generator 924, planner 926, data logger 928, and / or post-processor 930. The control module 1310 can also, or alternatively, use the corresponding configuration parameters 1322 to configure the operation of the simulator 922, the target generator 924, the planner 926, the data logger 928, and / or the post-processor 930.

[0177] In some embodiments, configuration parameter 1322 may include a unique name and / or identifier for a given scenario (e.g., a combination of a particular environment, machine, objective, policy, etc.) in which data will be generated and collected. Configuration parameter 1322 may also be used to customize and / or randomize the type of environment and / or machine to be simulated, the type of objective, the type of policy, the type of data to be recorded, the frequency at which data is recorded, and / or the way the recorded data is converted into a format suitable for training and / or evaluating machine learning models and / or other components of the machine.

[0178] In some embodiments, different sets of configuration parameters 1322 can be used to launch different instances of the data generation pipeline 902 to generate data depicting different scenarios related to navigation and / or other types of tasks performed by the machine in the environment. For example, NVIDIA’s One-system-to-many-others (OSMO) workflow can be used to launch multiple instances of the data generation pipeline 902 in parallel on multiple nodes of a cloud computing system. Each instance can be used to generate and / or collect simulation data 932, objectives 934, commands 936, records 938, and / or datasets 940 associated with a given scenario and / or a set of scenarios. The number of instances of the data generation pipeline 902 and / or the number of nodes on which a given instance of the data generation pipeline 902 is deployed can be scaled to accommodate: requirements and / or preferences associated with the amount of synthetic data to be generated; applications and / or use cases associated with the synthetic data; coverage of the environment, machine, policy 1304, objectives 934, and / or scenarios associated with the synthetic data; and / or other factors. Additional OSMO workflows can also be used to launch pipelines that are used to train, test, and / or evaluate machine learning models, policies, hardware configurations, software stacks, twins, and / or other components or representations of machines using the generated simulation data 932, objectives 934, commands 936, records 938, and / or datasets 940.

[0179] Figure 14 The various embodiments are illustrated by Figure 9 The example synthesized data generated by pipeline 902 in the data generation pipeline. For example... Figure 14 As shown, the synthetic data includes two images 1316(1)-1316(2) that depict the warehouse environment surrounding the machine at a given time step in the simulation. Image 1316(1) includes a perspective view of the environment from the rear of the machine, and image 1316(2) includes a view from a camera on the machine. These images 1316(1)-1316(2) can be rendered by simulator 922 based on a 10D scene representing the environment, an odometer value 1314 associated with the machine at that time step, and / or other simulation data 932.

[0180] The synthesized data may include a set of commands 936 associated with the same time step. These commands 936 may include linear velocity (its magnitude is depicted in the left bar) and angular velocity (its magnitude and direction are depicted in the right bar). These commands 936 can be used to update the state of the robot and / or the environment in the simulation. The updated state can then be used to generate a new image 1316, other simulation data 932, and / or commands 936 for the next time step in the simulation.

[0181] It should be understood that the arrangements and other arrangements described herein are merely illustrative. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of the arrangements, components, features, and elements shown, and certain elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and at any suitable location (e.g., on a local device, vehicle, or edge machine; locally deployed (e.g., a locally hosted server); remotely located (e.g., in one or more computing or server devices in one or more data centers in the cloud); and / or in other locations). The various functions performed by the entities described herein may be implemented by hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, microcontroller, embedded processor, digital signal processor (DSP), image signal processor (ISP), physical processing unit (PPU), field-programmable gate array (FPGA), accelerators (e.g., deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural network accelerator (NNA) and / or neural processing unit (NPU), programmable vision accelerator (PVA), optical flow accelerator (OFA), etc.), application-specific integrated circuits (ASIC), data processing unit (DPU), quantum processor, etc.) that execute instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 17A-17E Example machine 1700 Figure 18 Example calculation of ecosystem 1800, Figure 19 Example generative language model systems 1900 and / or Figure 20 The example computing device 2000 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the operation.

[0182] Now for reference Figure 15 Each block of the method 1500 described herein may include a computational process, which may be executed using any combination of hardware, firmware, and / or software. For example, various functions may be implemented by a processor executing instructions stored in memory. The method may also be embodied as computer-usable instructions stored on a computer storage medium. The method may be provided by a standalone application, service, or managed service (independently or in combination with another managed service), or a plug-in to another product, to name just a few. Furthermore, as an example, regarding... Figure 9The system described herein describes method 1500. However, these methods may be additionally or alternatively implemented by any system or any combination of systems, including but not limited to the systems described herein.

[0183] Figure 15 A flowchart illustrating a method 1500 for generating synthetic data associated with machines in an environment, according to various embodiments, is shown. Figure 15 As shown, method 1500 may begin with operation 1502, where data generation pipeline 902 receives configuration parameters associated with the generation of synthetic data. For example, data generation pipeline 902 may receive configuration parameters via one or more configuration files, API calls, and / or user interfaces. These configuration parameters can be used to configure and / or customize the generation of synthetic data. For example, configuration parameters include a unique name and / or identifier for a given scenario (e.g., a combination of specific environments, machines, objectives, policies, etc.) in which data will be generated and collected. These parameters can also be used to customize the type of environment and / or machine to be simulated, the objectives, policies, the type of data to be recorded, the frequency at which data is recorded, and / or the way the recorded data is converted into a format suitable for training and / or evaluating machine learning models and / or other components of the machine.

[0184] In operation 1504, the data generation pipeline 902 can initialize one or more simulations using a set of attributes associated with the machine and / or the environment in which the machine operates. For example, the data generation pipeline 902 can use configuration parameters to determine and / or randomize the machine type, machine model, and / or the machine's initial pose in the environment. The data generation pipeline 902 can also, or alternatively, obtain, generate, and / or randomize a 10D scene corresponding to the environment and / or an occupancy map of that 10D scene. The data generation pipeline 902 can also, or alternatively, add one or more objects to the 10D scene and / or set the objects' physical, material, and / or collision properties.

[0185] In operation 1506, data generation pipeline 902 can determine targets associated with the machine's operation in the environment. For example, data generation pipeline 902 can generate navigation-based targets by sampling locations within the environment that the machine wants to navigate to from unoccupied space in the environment. This sampling can be preferentially performed on certain areas of the environment specified in the configuration parameters and / or certain areas with attributes specified in the configuration parameters.

[0186] In operation 1508, the data generation pipeline 902 can generate simulated data depicting the operation of the machine in the environment via one or more simulations. For example, the data generation pipeline 902 can render one or more images of the environment from the perspectives of one or more cameras on the machine, one or more locations outside the machine, and / or other viewpoints. The data generation pipeline 902 can also, or alternatively, generate point clouds, IMU measurements, and / or other sensor measurements associated with sensors on the machine.

[0187] In operation 1510, the data generation pipeline 902 can determine one or more commands to be issued to the machine based on simulated data and / or objectives, via a machine-specific strategy. For example, the data generation pipeline 902 can input simulated data and / or objectives into a planning stack, a neural network, and / or other components implementing the strategy. Given the input data, the component can generate commands specifying the linear velocity and / or angular velocity of the machine. The component can also, or alternatively, generate one or more command distributions sampling one or more commands from it.

[0188] In operation 1512, data generation pipeline 902 can store simulation data and one or more commands in one or more data records. For example, data generation pipeline 902 can associate the simulation data generated in operation 1508 and the commands generated in operation 1510 with the same time step and / or "frame" in the simulation. Data generation pipeline 902 can also record simulation data and one or more commands into one or more data records associated with the time step and / or frame.

[0189] In operation 1514, the data generation pipeline 902 can determine whether to continue generating synthetic data. For example, the data generation pipeline 902 can determine to continue generating synthetic data until the machine reaches a target, one or more simulations have run a certain number of time steps, and / or another condition is met. If the data generation pipeline 902 determines to continue generating synthetic data, then the data generation pipeline 902 performs operation 1516, where the data generation pipeline updates the simulation data based on one or more commands. For example, the data generation pipeline 902 can update the machine's position, heading, speed, and / or another state to reflect the machine's execution of commands. The data generation pipeline 902 can also, or alternatively, generate new image and / or sensor data reflecting the updated machine state.

[0190] Data generation pipeline 902 may repeat operation 1510 to generate new commands based on updated simulation data. Similarly, data generation pipeline 902 may repeat operation 1512 to store updated simulation data and one or more commands in one or more additional data records. For example, data generation pipeline 902 may store updated simulation data and one or more commands in association with new (e.g., incremental) time steps and / or frames. After a given simulation dataset and command set have been stored in one or more data records, data generation pipeline 902 repeats operation 1514 to determine whether to continue generating synthetic data.

[0191] After data generation pipeline 902 determines in operation 1514 to stop generating synthetic data, it may perform operation 1516, in which it stores and / or formats one or more data records into one or more datasets. For example, data generation pipeline 902 may generate different datasets for each use case and / or application associated with the synthetic data. Within a given dataset, data generation pipeline 902 may resample, format, and / or otherwise post-process the corresponding data records to adapt them to the appropriate use case and / or application. The data generation pipeline 902 can then provide this dataset for training, testing, and / or evaluating machine learning models, hardware configurations, policies, digital twins, and / or other components and / or representations of machines in various environments.

[0192] See now Figure 16A Each block of the method 1600 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., but not limited to, the processors described herein) that execute instructions stored in one or more memories or memory systems. In some embodiments, the computer process may also be embodied as computer-usable instructions stored on a computer storage medium. These methods may be provided by standalone applications, services, or managed services (independently or in combination with another managed service), application programming interfaces (APIs), and / or plug-ins to another product, to name a few. Furthermore, this document provides examples regarding... Figures 1-3 or Figure 9 Method 1600 is described. However, these methods may be additionally or alternatively implemented by any system or any combination of systems, including but not limited to the systems described herein.

[0193] Figure 16AThis illustrates a method 1600 for generating at least one action and transmitting it to a robot according to some embodiments of the present disclosure. At block 1602, method 1600 may include: determining the state (e.g., policy tag 208) and identifier (e.g., type embedding 306) of a robot (e.g., robot 1700). The state may be determined using a world model (e.g., world model 110), and the identifier may indicate the type of the robot (e.g., robot type 304). The state of the robot may include at least one of the robot's environment, the velocity of each joint of the robot, or the robot's goal. For robots of the same type, the identifier may be the same. The identifier may include a unique morphological code indicating the robot type. The identifier may correspond to an embedding in an embedding space used to identify the robot as a robot type among a plurality of robot types. The robot type may include at least one of a humanoid robot, a wheeled robot, or a quadruped robot.

[0194] At block 1604, method 1600 may include: generating at least one action for the robot. The action may be generated by processing states and identifiers through a general action policy (e.g., general action policy 130). States and identifiers may be input into the action policy, and the action policy may output actions based on states and embeddings. The general action policy may be generated using a combination of a base action policy (e.g., base policy 104) corresponding to multiple robot types and one or more specialized action policies (e.g., specialized policy 122) corresponding to different robot types. The base action policy may be updated using imitation learning and a world model, and the base action policy is used to receive at least one state of the robot as input and output a basic action for moving the robot. One or more specialized action policies may be updated using residual reinforcement learning, and one or more specialized action policies are used to receive at least one state of the robot as input and output specialized actions for moving the robot. One or more specialized action policies may be updated at least based on the base action policy, and the specialized action is a combination of the base action and a residual action used to adapt the base action to the robot type.

[0195] In various embodiments, to generate a generalist action policy, a combination of a base action policy and one or more specialist action policies can be distilled. To generate a generalist action policy, method 1600 may include generating multiple specialist actions (e.g., specialist action 214) for the robot using each of the one or more specialist action policies. Specialist actions can be generated using residual reinforcement learning. Residual reinforcement learning may include at least one reward (e.g., reward 226), and method 1600 may further include generating the at least one reward based on the outcome of at least one robot type-specific action. The outcome may include at least one of the following: progress toward at least one objective, collision avoidance, and completion of at least one objective, the at least one reward being used to update the multiple robot type-specific action policies.

[0196] In various embodiments, method 1600 may include: generating multiple normal distributions of multiple specialist actions using each of one or more specialist action policies. Method 1600 may include: combining the multiple normal distributions. Method 1600 may include: distilling the combination of multiple normal distributions into a generalist action policy by at least minimizing the divergence of the combination of the multiple normal distributions.

[0197] At block 1606, method 1600 may include: causing the robot to move according to an action. The transmission of the action can guide and move the robot. In various embodiments, at least one action includes a plurality of speed commands, each speed command corresponding to a joint of the robot. To perform at least one action, each of the plurality of speed commands can be mapped to a corresponding joint of the robot. The plurality of robot types may include at least humanoid robots, autonomous mobile robots (AMRs), wheeled robots, warehouse vehicles or machines, or quadruped robots. In various embodiments, the action is transmitted to at least one or more joints, actuators, or motors of the robot.

[0198] In various embodiments, moving the robot may include causing the robot to perform actions. In various embodiments, method 1600 may include causing one or more control operations associated with the robot to be performed based on at least one or more actions generated using a generalist motion strategy. The generalist motion strategy may generate the one or more actions when: (i) conditioned by an embedding indicating a robot type corresponding to the robot among a plurality of robot types, and (ii) when state information corresponding to the robot and the robot's environment is processed at least based on the generalist motion strategy.

[0199] In various embodiments, the generalist motion policy is trained using a teacher dataset generated using motion outputs and latent states corresponding to specialist motion policies associated with various robot types. The embedding may include one-hot morphological encoding indicating the robot type. State information may be represented at least using a world model. One or more control operations may correspond to one or more joints, actuators, or motors of the robot.

[0200] Figure 16B This is a flowchart illustrating a method 1650 for a robot to receive commands and move according to commands, according to some embodiments of the present disclosure. At block 1652, method 1650 may include: receiving a plurality of first commands (e.g., actions output by a general motion strategy 130). The first commands may be received by a robot (e.g., robot 1700), and each first command may be mapped to at least one of the robot's joints, actuators, or motors. The first commands may be generated by a general motion strategy, which may be generated based on a combination of a base motion strategy corresponding to a plurality of robots and one or more specialized motion strategies. The general motion strategy may correspond to a plurality of robots, and one or more specialized motion strategies may be for a plurality of robots of one type. The general motion strategy may generate actions based on the robot's state and an identifier. The identifier may at least indicate the type of robot, and the state may indicate at least one of the robot's environment, the velocity of each joint of the robot, or the robot's goal. The identifier may be an embedding, and may be the same for robots of the same type. The identifier may correspond to an embedding in an embedding space used to identify the robot as a robot type among a plurality of robot types. After generating the motion, the generalist motion strategy can transmit the motion to the robot, which can be multiple first commands. The motion can include multiple first commands, which can be velocity commands, and each first command can correspond to at least one of the robot's joints, actuators, or motors.

[0201] In various embodiments, imitation learning and world models are used to update a base action policy. The base action policy may receive at least one state of at least one of a plurality of robots as input and output a base action to move at least one of the robots. Multiple specialist action policies may be updated using residual reinforcement learning. Multiple specialist action policies may receive at least one state of at least one of a plurality of robots as input and output specialist actions to move at least one of the robots. Multiple specialist action policies may be updated based on the base action policy. Specialist actions may be combinations of base actions and residual actions, whereby the residual actions are used to adapt the base action to the type of at least one of the robots. To generate a generalist action policy, a combination of the base action policy and one or more specialist action policies may be distilled.

[0202] In various embodiments, the type of at least one of the multiple robots may include at least a humanoid robot, an autonomous mobile robot (AMR), a wheeled robot, a warehouse vehicle or machine, or a quadruped robot. To generate a generalist motion policy, method 1650 may include: generating multiple specialized actions for at least one of the multiple robots by inputting multiple states into the multiple specialized motion policies. Method 1650 may include: generating multiple normal distributions of the multiple specialized actions using each of the multiple specialized motion policies. Method 1600 may include: combining the multiple normal distributions and distilling the combination of the multiple normal distributions into a generalist motion policy, at least by minimizing the divergence of the combination of the multiple normal distributions.

[0203] At block 1654, method 1650 may include: moving according to a first command. The robot may receive the first command and move according to the first command. For example, to perform the action, method 1650 may include: mapping each first command to a corresponding joint of the robot, and each joint of the robot may move according to the corresponding first command.

[0204] At block 1656, method 1650 may include: transmitting the result of the movement (e.g., result 222). As the robot moves according to the first command, at least one of the controller or the robot may record the robot's movement, and once the movement according to the first command is completed, at least one of the controller or the robot may transmit the result of the movement to, for example, system 100. The result may include at least one of the following: goal completion, robot progress along the route according to the first command, and collision status.

[0205] At block 1658, method 1650 can receive multiple second commands. The third action policy can generate the second commands and the robot's next state based on the results. The robot's next state can be the result of the robot moving according to the first command (e.g., the final position).

[0206] The systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles connected to one or more trailers, aircraft, ships, shuttles (e.g., robotaxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones, and / or other types of vehicles. Furthermore, the systems and methods described herein can be used for a wide range of purposes, including, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets (e.g., NVIDIA's Omniverse), cloud computing, and / or any other suitable applications.

[0207] The disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines, etc.), systems implemented using robots, aviation systems, medical systems, marine systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing analog operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing language models (such as Large Language Model (LLM), Visual Language Model (VLM), Visual-Language-Action (VLA) models, and / or Multimodal Language Model (MMLM)), and systems using or deploying a Systems comprising one or more inference microservices, systems including those deploying one or more machine learning models in services or microservices and OS-level virtualization packages (e.g., containers), systems for performing one or more wireless cellular transmissions using wireless cellular networks, systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems at least partially implemented in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems at least partially implemented using cloud computing resources, and / or other types of systems.

[0208] Example autonomous or semi-autonomous machines Figure 17A Examples of sensor positions with corresponding fields of view or sensing fields of view of autonomous or semi-autonomous vehicles 1700a, autonomous mobile robots (AMRs) 1700b, and humanoid robots 1700c, according to some embodiments of this disclosure. While three types of machines 1700 are shown, this is not intended to be limiting, and the machines 1700 described herein may include vehicles, cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police or emergency vehicles, ambulances, boats, construction vehicles, underwater vehicles, robots (e.g., AMRs, humanoid robots, robotic arms, end effectors, forklifts, etc.), drones, aircraft, vehicles coupled to trailers (e.g., semi-trailer tractors for hauling goods), and / or another type of vehicle or machine (e.g., driverless and / or vehicles or machines accommodating one or more passengers). In some cases, vehicles 1700a, AMRs 1700b, humanoid robots 1700c, and / or other machine types may be collectively referred to herein as machines 1700.

[0209] Regarding Vehicle 1700A, autonomous and semi-autonomous vehicles are typically described by automation levels, defined by the National Highway Traffic Safety Administration (NHTSA) (a division of the U.S. Department of Transportation) and the Society of Automotive Engineers (SAE) in their "Classification and Definition of Terms Related to Driving Automation Systems for Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 1700 may have one or more functions that meet Level 3 through Level 5 of autonomous driving. For example, Vehicle 1700 may be able to provide driver assistance (Level 1), partial automation (Level 2, Level 2+, Level 2++), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the specific implementation. The term “autonomy” as used herein can include any and / or all types of autonomy of Machine 1700 or other machines, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, provision of auxiliary autonomy, semi-autonomy, primary autonomy or other specified autonomy.

[0210] about Figure 17A The sensors and their respective fields of view (not shown for clarity) or sensing fields (not shown for clarity) are an example embodiment and are not intended to be limiting. Although not shown, each sensor may have a corresponding field of view (e.g., a 360-degree field of view around camera 1768D, a 180-degree field of view around wide-angle camera 1768B, a 360-degree sensing field of view of LiDAR sensor 1764, etc.). For example, only a subset of the sensors shown may be included, additional sensors may be included, alternative sensors may be included, the number of each sensor mode may be different, the sensor modes may be different (e.g., LiDAR or RADAR may not be included, SONAR, thermal sensors, etc. may be included), and the sensor positions may differ from those shown on vehicles 1700a, AMR 1700b, and / or humanoid robots 1700c, etc. For example, for vehicle 1700a, the location, number, modality, and / or other sensor information may differ depending on type (e.g., SUV, truck, car, robot, motorcycle, etc.), size (e.g., 18-wheeler, transport vehicle, small car, etc.), and associated functions (e.g., L2 vs. L5). Similarly, for AMR 1700b and / or humanoid robot 1700c, shape, size, purpose, implementation, model, etc., can determine the number and type of sensors used.

[0211] like Figure 11AAs shown, autonomous or semi-autonomous vehicles 1700A, AMR 1700B, and humanoid robots 1700C may include different sensor types, numbers, and locations. As a non-limiting example, vehicle 1700A may include twelve cameras 1768, such as a front wide-angle camera (e.g., 120-degree field of view (FOV)), a front telephoto camera (e.g., 30-degree FOV), a rear-side left camera (e.g., 70-degree FOV), a rear-side right camera (e.g., 70-degree FOV), a front fisheye camera (e.g., 200-degree FOV), a rear fisheye camera (e.g., 200-degree FOV), a left fisheye camera (e.g., 200-degree FOV), a right fisheye camera (e.g., 200-degree FOV), a front telephoto satellite camera (e.g., 30-degree FOV), a rear telephoto camera (e.g., 30-degree FOV), a cross-left camera (e.g., 120-degree FOV), and a cross-right camera (e.g., 120-degree FOV). In this embodiment, the camera 1768 may use a Gigabit Multimedia Serial Link (GMSL) interface (e.g., GMSL2) as input / output (I / O).

[0212] In some embodiments, although Figure 17A As not shown, vehicle 1700A may include an in-cabin occupant and / or driver monitoring system, which may include various sensors. For example, in-cabin sensors may include various cameras 1768, such as a driver monitoring camera (e.g., located in front of the driver's seat and facing the driver's seat at a 55-degree FOV), a front occupant monitoring camera (e.g., located in front of the front occupant seat and facing the front occupant seat at a 190-degree FOV), and a rear occupant monitoring camera (e.g., located in front of the rear occupant seat and facing the rear occupant seat at a 190-degree FOV). Similar to external cameras 1768, in embodiments, internal cameras 1768 may use a GMSL (e.g., GMSL2) interface for I / O.

[0213] As another non-limiting example, vehicle 1700A may also include nine RADAR sensors 1760. For example, vehicle 1700A may include a front center imaging RADAR sensor (e.g., 120-degree FOV or sensing field), a left front corner RADAR sensor (e.g., 160-degree FOV or sensing field), a right front corner RADAR sensor (e.g., 160-degree FOV or sensing field), a right rear corner RADAR sensor (e.g., 160-degree FOV or sensing field), a left RADAR sensor (e.g., 160-degree FOV or sensing field), a right RADAR sensor (e.g., 160-degree FOV or sensing field), a left rear RADAR sensor (e.g., 50-degree FOV or sensing field), and a right rear RADAR sensor (e.g., 50-degree FOV or sensing field). In embodiments, the RADAR sensors 1760 may use an Ethernet interface as I / O.

[0214] As a non-limiting example, the vehicle 1700A may also include twelve ultrasonic sensors 1762. Figure 17A As shown, the ultrasonic sensor can be placed along the front and rear bumpers of vehicle 1700A and along the sides of vehicle 1700A, and can be used to detect objects (static and dynamic) close to vehicle 1700A. In some embodiments, the ultrasonic sensor 1762 can use a DS13 interface as I / O.

[0215] As a non-limiting example, vehicle 1700A may also include a LiDAR sensor 1764, such as a front-center LiDAR sensor (e.g., a 120-degree horizontal FOV or sensing field and a 30-degree vertical FOV or sensing field). In some embodiments, such as when using additional or alternative LiDAR sensors, the LiDAR sensor may have different horizontal and vertical fields of view or sensing fields. For example, LiDAR sensor 1764 may include a 360-degree horizontal FOV or sensing field (e.g., located in a rotating LiDAR sensor) and a 90-degree vertical FOV or sensing field. In some embodiments, LiDAR sensor 1764 may use an Ethernet interface as I / O.

[0216] As a non-limiting example, the Autonomous Mobile Robot (AMR) 1700B may include three LiDAR sensors 1764. For example, the topmost LiDAR sensor 1764 may include a beam or 3D LiDAR sensor (e.g., a 360-degree horizontal and 90-degree vertical FOV or sensing field), and the front and rear LiDAR sensors may include planar or 2D LiDAR sensors (e.g., a 180-degree horizontal FOV or sensing field).

[0217] As a non-limiting embodiment, the AMR 1700B may further include eight cameras 1768, such as a front stereo camera (e.g., 120-degree FOV), a rear stereo camera (e.g., 120-degree FOV), a left stereo camera (e.g., 120-degree FOV), a right stereo camera (e.g., 120-degree FOV), a front fisheye camera (e.g., 202-degree ± 3-degree FOV), a rear fisheye camera (e.g., 202-degree ± 3-degree FOV), a left fisheye camera (e.g., 202-degree ± 3-degree FOV), and a right fisheye camera (e.g., 202-degree ± 3-degree FOV).

[0218] The AMR 1700B may also include a charging port, charging port contacts, status indicators, one or more (e.g., four) RGB LEDs, one or more IMU sensors 1766, a magnetometer, and a barometer. The AMR 1700B is capable of high-precision time synchronization between sensors using hardware timestamps and PTPs over Ethernet with sensor acquisition times of less than 10 microseconds. In an embodiment, the AMR 1700B provides simultaneous camera capture within 100 microseconds of a single hardware trigger on all cameras 1768, and can write sensor captures to disk at a rate of 4 GB / s for writing packets (e.g., writing to ROSbags of the Robot Operating System (ROS)). Therefore, the AMR 1700B is capable of running ROS (e.g., NVIDIA's ISAAC ROS), can be remotely operated (as described herein), can map the environment, and can navigate the environment using vision cameras 1768, LiDAR 1764, and / or other sensor types or modalities.

[0219] The humanoid robot 1700C may include (as a non-limiting example) a LiDAR sensor 1764. For example, the LiDAR sensor 1764 may include a beam or a 3D LiDAR sensor (e.g., a 360-degree horizontal and 90-degree vertical FOV or sensing field), or it may include a planar or 2D LiDAR sensor (e.g., a 180-degree horizontal FOV or sensing field).

[0220] As a non-limiting embodiment, the humanoid robot 1700C may also include four cameras 1768, such as a front stereo camera (e.g., 120-degree FOV), a rear stereo camera (e.g., 120-degree FOV), a front fisheye camera (e.g., 202-degree ± 3-degree FOV), and a rear fisheye camera (e.g., 202-degree ± 3-degree FOV).

[0221] As a non-limiting embodiment, the humanoid robot 1700C may also include four ultrasonic sensors 1762, such as a left arm ultrasonic sensor, a right arm ultrasonic sensor, a left leg ultrasonic sensor, and a right leg ultrasonic sensor.

[0222] The humanoid robot 1700C may also include any number of actuators, such as those allowing control and manipulation of joints. For example, the humanoid robot 1700C may include actuators that allow for various degrees of freedom (DoF) depending on the design. In a non-limiting embodiment, the humanoid robot 1700C may have a total of 40 degrees of freedom (DoF) (e.g., 6DoF x2 for arms, 6DoF x2 for hands, 6DoF x2 for legs, 2DoF for the torso, and 2DoF for the neck). Actuators can convert energy into physical motion, thereby allowing actions such as joint movement, locomotion, and grasping / manipulation. For example, motors and servos can be used to perform joint movements to control the rotation of joints in an arm or manipulator and allow for reaching, grasping, and manipulating objects. Locomotion can be achieved by moving through the environment using wheels, tracks, or other mobility devices (robot legs). Grasping and manipulation can be performed using end effectors or hands / fingers, which can be equipped with actuators to grasp objects, apply forces, and perform specific tasks. In some examples, the humanoid robot 1700C may include position and orientation sensors, such as encoders, gyroscopes, etc., to determine the robot 1700C's position in space, thereby enabling position determination and motion tracking. In embodiments, the humanoid robot 1700C may include force and pressure sensors to detect environmental interactions, enabling the robot 1700C to grasp objects with appropriate force and avoid obstacles along its path. Perception sensors (e.g., cameras, LiDAR, RADAR, ultrasound, sonar (SONAR), etc.) may be used in conjunction with tactile sensors to enable the robot 1700C to perceive objects, shapes, and textures, and to understand when to begin and stop touching (along with force sensors that regulate the force used during touching). As a non-limiting example, the humanoid robot 1700C may have a height of approximately 1-2 meters (e.g., 1.7 meters or 5 feet 6 inches), a weight of 50-70 kilograms, be able to move at speeds of 8 km / h or higher, and be able to carry a payload of 20-100 kilograms, depending on the system design and requirements.

[0223] In embodiments, the humanoid robot 1700C may include a dialogue system—such as a dialogue system driven by a language model (e.g., LLM, VLM, MMLM, VLA, etc.)—to help understand the environment, reason, and communicate with humans, animals, devices, and / or other robots, and / or make planning, control, and navigation decisions. Therefore, in addition to performing various tasks, the humanoid robot 1700C can also use onboard sensors, microphones, and speakers to understand speech, audio, and visual cues, while also being able to communicate with the environment.

[0224] Referring to camera 1768 of machine 1700, the camera type of camera 1768 may include, but is not limited to, digital cameras suitable for components and / or systems of machine 1700. For vehicle 1700a implementation, camera 1768 may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may have any image capture rate, such as 30 frames per second (fps), 60 fps, 120 fps, 240 fps, etc., depending on the embodiment. The camera may use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-neutral-neutral-colorless (RCCC) color filter array, a red-neutral-neutral-blue (RCCB) color filter array, a red-blue-green-neutral (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a colorless pixel camera (e.g., a camera with an array of RCCC, RCCB, and / or RBGC color filters) can be used to improve light sensitivity.

[0225] The field of view includes cameras (e.g., front-facing cameras) in front of the machine 1700, which can be used for surround view to help identify the path and obstacles ahead, and, with the help of one or more controllers 1736 and / or control SoCs, provide information crucial for generating an occupancy grid and / or determining preferred machine movement, trajectory, and / or path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including lane departure warning (“LDW”), adaptive cruise control (“ACC”), and / or other functions such as traffic sign recognition.

[0226] Various cameras can be used in front-mounted configurations, including, for example, monocular camera platforms that include complementary metal-oxide-semiconductor (“CMOS”) color imagers. Another example could be a wide-angle camera 1768B, which can be used to perceive objects entering the field of view from the periphery (e.g., pedestrians, warehouse vehicles, other robots, pedestrian traffic, or bicycles). Furthermore, any number of long-range cameras 1768E (e.g., a pair of long-range stereo cameras) can be used for depth-based object detection, especially for objects for which neural networks have not yet been trained. Long-range cameras 1768E can also be used for object detection and classification, as well as basic object tracking.

[0227] Any number of stereo cameras 1768A can also be included in front-mounted and / or other (e.g., rear-mounted) configurations. In at least one embodiment, one or more stereo cameras 1768A may include an integrated control unit that includes a scalable processing unit that can provide programmable logic (“FPGA”) and a multi-core microprocessor with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the machine 1700 environment, including distance estimates of midpoints in the image (e.g., parallax or depth images). Alternative stereo cameras 1768A may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip that can measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1768A may be used in addition to those described herein, or as an alternative to those described herein. For example, in some embodiments, a camera other than a stereo camera (e.g., two monocular cameras with at least partially overlapping fields of view) can be used to perform stereo depth estimation.

[0228] Cameras with a field of view including portions of the side environment of machine 1700 (e.g., side-view cameras) can be used, for example, for surround view, providing information for creating and updating occupancy grids, and generating side collision warnings and / or indicating to AMR 1700B or humanoid robot 1700C, for example, the presence of objects, features, and / or people on the side. For example, a surround camera 1768D can be mounted on machine 1700. The surround camera 1768D can include a wide-angle camera 1768B, a fisheye camera, a 360-degree camera, etc. For example, four fisheye cameras can be mounted on the front, rear, and sides of machine 1700. In an alternative arrangement, machine 1700 can use three surround cameras 1768D (e.g., left, right, and rear), and can utilize one or more other cameras (e.g., a front-facing camera) as a fourth surround-view camera.

[0229] A camera 1768 (e.g., a rear-view camera) having a field of view including a portion of the environment behind the machine 1700 can be used to understand objects, features, people, and / or other information behind the machine 1700, such as for parking assistance, surround view, rear collision warning, planning, control, and navigation determination, and / or to create and update occupancy grids, BEV images representing the environment, height maps, etc. A wide variety of cameras 1768 can be used, including but not limited to those also suitable for use as front cameras (e.g., long-range and / or mid-range cameras 1768E, stereo cameras 1768A, infrared cameras 1768C, etc.), rear cameras, side cameras, downward cameras, upward cameras, and / or similar cameras 1768, as described herein.

[0230] Similarly, for LiDAR sensor 1764, RADAR sensor 1760, ultrasonic sensor 1762 and / or other sensor modes or types, the location and placement of the sensors and their corresponding fields of view or sensing fields can be determined based on the use case, implementation or design of the particular machine 1700.

[0231] For example, machine 1700 includes a RADAR sensor 1760, which can be used by machine 1700 for long-range object detection, even in dark and / or inclement weather conditions. In embodiments, the RADAR functional safety level can be ASIL B. RADAR sensor 1760 can be controlled and accessed for object tracking data using CAN and / or bus 1702 (e.g., for transmitting data generated by RADAR sensor 1760), and in some examples, raw data can be accessed using Ethernet. Various types of RADAR sensors can be used. For example, but not limited to, RADAR sensor 1760 can be suitable for front, rear, and side radar applications. In some examples, a pulse Doppler RADAR sensor is used.

[0232] The RADAR sensor 1760 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range lateral coverage, etc. In some examples, long-range radar can be used for adaptive cruise control (ACC) functions. Long-range RADAR systems can provide a wide field of view, for example, within a 250m range, achieved through two or more independent scans. The RADAR sensor 1760 can help distinguish between static and moving objects; ADAS systems can use it for emergency braking assistance and forward collision warning, and robots can use it to detect dynamic objects in various environments (e.g., low-light or no-light environments). Long-range RADAR sensors can include a single static multimodal RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record the machine 1700's surroundings at higher speeds while minimizing peripheral interference (e.g., traffic from adjacent lanes). The other two antennas expand the field of view, enabling rapid detection of objects entering or leaving the machine's direct path (e.g., lanes).

[0233] Mid-range RADAR systems can include, for example, a range of up to 960m (front) or 80m (rear), and a field of view of up to 42 degrees (front) or 150 degrees (rear). Short-range RADAR systems can include, but are not limited to, RADAR sensors designed to be mounted at both ends of a side surface (e.g., rear bumper), allowing continuous monitoring of blind spots behind and beside a machine (e.g., vehicle, robot, etc.) using two beams. Therefore, short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0234] Machine 1700 may also include an ultrasonic sensor 1762. The ultrasonic sensor 1762 may be located at the front, rear, and / or sides of machine 1700 and may be used to assist near-field sensing, such as for parking assistance, collision avoidance (e.g., for robot parts), and / or to create and update occupancy grids, evidence grid maps (EGMs), height maps, BEV images, and / or other representations of objects and features in the machine 1700 environment. Multiple ultrasonic sensors 1762 may be used, and different ultrasonic sensors 1762 may be used for different detection ranges (e.g., 2.5 m, 4 m). For example, the ultrasonic sensor 1762 may operate at a functional safety level of ASIL B.

[0235] Machine 1700 may include a LiDAR sensor 1764. The LiDAR sensor 1764 can be used for object and feature detection, pedestrian and other robot detection, emergency braking, collision avoidance, simultaneous localization and mapping (SLAM), free space detection, and / or other functions. In embodiments, the LiDAR sensor 1764 may be of functional safety level ASIL B. In some examples, machine 1700 may include multiple LiDAR sensors 1764 (e.g., two, four, six, etc.), which can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0236] In some examples, the LiDAR sensor 1764 may be able to provide a 360-degree field of view of objects and their distances. For example, a commercially available LiDAR sensor 1764 may have a advertised range of approximately 1700m, an accuracy of 2cm-3cm, and support for 1700Mbps Ethernet connectivity. In some examples, one or more non-protruding LiDAR sensors 1764 may be used. In such examples, the LiDAR sensor 1764 can be implemented as a small device that can be embedded in the front, rear, side, top, and / or corner of the machine 1700. In such examples, the LiDAR sensor 1764 can provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, with a range of up to 200m even for low-reflectivity objects. The front-mounted LiDAR sensor 1764 can be configured with a horizontal field of view between 45 and 135 degrees.

[0237] In some examples, LiDAR technology, such as 3D flash LiDAR, can also be used. 3D flash LiDAR uses a laser flash as a transmission source, illuminating the environment around a vehicle up to approximately 200m away. The flash LiDAR unit includes a receiver that records the laser pulse transmission time and the reflected light on each pixel, which in turn corresponds to the distance from the vehicle to the object. Flash LiDAR allows for the generation of highly accurate and distortion-free images of the surrounding environment with each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of the machine 1700. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras with no moving parts other than a fan (e.g., non-scanning LiDAR devices). Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of 3D distance point clouds and co-registration intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 1764 may be less susceptible to motion blur, vibration, and / or shock.

[0238] Figure 17BThis is an illustration of the location of sensors and components of an example autonomous or semi-autonomous vehicle 1700A (also referred to herein as "vehicle 1700", "self-vehicle 1700", "self-machine 1700", or "machine 1700") according to some embodiments of this disclosure. While vehicle 1700A is shown in the figure, this is not intended to be limiting, and similar components and / or sensors may be included on any other machine type without departing from the scope of this disclosure. For example, similar sensors and / or components may be used in vehicles, cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police cars, ambulances, ships, construction vehicles, underwater vehicles, robots (e.g., AMRs, humanoid robots, robotic arms, end effectors, forklifts, etc.), drones, aircraft, vehicles coupled to trailers (e.g., semi-trailers for hauling goods), and / or another type of vehicle or machine (e.g., driverless and / or capable of accommodating one or more passengers).

[0239] Figure 17CThis is a block diagram of an example system architecture for a machine 1700 (e.g., an autonomous or semi-autonomous vehicle 1700A, an autonomous mobile robot (AMR) 1700B, a humanoid robot 1700C, and / or other types of machines) according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are presented as examples only. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of the arrangements shown, and some elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components or together with other components, and can be implemented in any suitable combination and location (e.g., on a local device, vehicle, or edge machine, in the field (e.g., a locally hosted server), in a remote location (e.g., in one or more computing or server devices in one or more data centers in the cloud) and / or other locations). The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, microcontroller, embedded processor, digital signal processor (DSP), image signal processor (ISP), physical processing unit (PPU), field-programmable gate array (FPGA), accelerators (e.g., deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural network accelerator (NNA) and / or neural processing unit (NPU), programmable vision accelerator (PVA), optical flow accelerator (OFA), etc.), application-specific integrated circuit (ASIC), data processing unit (DPU), quantum processor, etc.) executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 17A to 17E Example machine 1700 Figure 18 Example computational ecosystem 1800, Figure 19 Example generative language modeling system 1900 and / or Figure 20 The example computing device 2000 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the same task.

[0240] Figure 17CEach component, feature, and system of machine 1700 is shown connected via bus 1702 (or referred to as "machine communication network 1702" or simply "communication network 1702"). Bus 1702 may include a Controller Area Network (CAN) data interface (or referred to herein as "CAN bus"). CAN can be a network within machine 1700 used to help control various features and functions of machine 1700, such as brake actuation, acceleration, braking, steering, windshield wipers, etc. CAN bus can be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button position, and / or other vehicle status indicators. CAN bus may conform to ASIL B standard. In some embodiments, in addition to or as an alternative to the CAN bus, bus 1702 may include FlexRay, embedded buses (e.g., SPI, I2C), Local Interconnect Link (LIN), NVIDIA's NVLink, Ultra Accelerator Link (UALink), USB (2.0, 3.0 and above), radio frequency (RF), Ethernet (e.g., 10BASE / 100BASE, 1000BASE, 10G, etc.), and / or other communication protocols or functions. Furthermore, while a single line is used to represent bus 1702, this is not limiting. For example, there can be any number of buses 1702, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 1702 may be used to perform different functions and / or for redundancy. For example, a first bus 1702 may be used for collision avoidance, while a second bus 1702 may be used for actuation control. In any example, each bus 1702 can communicate with any component of machine 1700, and two or more buses 1702 can communicate with the same component. In some examples, each SoC 1704, each controller 1736, and / or each computer or computing engine within machine 1700 can access the same input data (e.g., input from sensors of machine 1700) and can be connected to a common bus, such as a CAN bus.

[0241] Machine 1700 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, batteries, side mirrors, and / or other parts of the vehicle or machine. Machine 1700 may include a propulsion system 1750, such as an internal combustion engine, a hybrid power plant, an all-electric motor, a hydrogen fuel cell engine, and / or another type of propulsion system. Propulsion system 1750 may be connected to the drivetrain of machine 1700, which may include a transmission to enable propulsion of machine 1700. Propulsion system 1750 may be controlled in response to a signal received from throttle / accelerator 1752.

[0242] Steering system 1754 may include a steering wheel and / or other steering mechanisms (e.g., remote steering and / or local steering) for steering machine 1700 (e.g., along a desired path or route) while propulsion system 1750 is in operation (e.g., when the vehicle is traveling). Steering system 1754 may receive signals from steering actuator 1756. In some embodiments, a steering wheel or other steering mechanism may be omitted, for example, for machine 1700 capable of fully automated (e.g., level 5) functionality.

[0243] The brake sensor system 1746 can be used to operate the vehicle brakes in response to signals received from the brake actuator 1748 and / or the brake sensor.

[0244] Machine 1700 may include one or more controllers 1736, such as those described herein. Figure 17AThe controller 1736 is described. It can be used for a variety of functions and can be coupled to any of the various other components and systems of the machine 1700. For example, the controller 1736 can be used to control the machine 1700, artificial intelligence performed on the machine 1700, infotainment on the machine 1700, etc. For example, one controller 1736 can be used for some or all of the functions, or different controllers 1736 can be used for different functions, for example, to ensure availability and safety separation between various controllers for different tasks. For example, the controller 1736 can use a system-calculated plan (e.g., the path or trajectory of vehicle 1700A or AMR 1700B, or the movement, component trajectory, motion position or displacement, etc. of joints or components (e.g., manipulators, end effectors, limbs, hands, fingers, legs, feet, etc.) of humanoid robot 1700C to control the machine 1700 in the environment. In some cases, the controller 1736 may include a proportional-integral-derivative (PID) controller, a fuzzy logic controller, a neural controller (e.g., a controller embodied as one or more neural networks), a force control controller, a programmable logic controller (PLC), and / or other types of controllers. For example, in the humanoid robot 1700C, the controller 1736 can act as the brain, responsible for analyzing sensor data, making decisions, and sending commands to the actuators. The controller 1736 may include a low-level controller that handles basic motor control, ensuring accurate and precise movement of the individual joints and actuators. The controller 1736 may also include a high-level controller to coordinate multiple actuators and sensors, plan complex movements, and adapt to constantly changing environments.

[0245] In some embodiments, controller 1736 may include an artificial intelligence controller that can use AI algorithms (e.g., DNN, MLM, etc.) to learn, make decisions, and autonomously perform tasks of machine 1700. In some embodiments, controller 1736 may use a fixed open-loop control algorithm and not adjust its actions according to the environment. In other embodiments, closed-loop control may be used, which incorporates a feedback mechanism to monitor the robot's performance and make necessary adjustments. In examples, controller 1736 may implement reactive control to directly respond to sensor inputs, enabling rapid reflexive actions and real-time changes. Furthermore, in some examples, deliberate control may be implemented, using internal models and planning algorithms to generate advanced actions, which may be suitable for complex tasks requiring reasoning, decision-making, and long-term planning.

[0246] The controller 1736 may include one or more system-on-chip (SoC) 1704 ( Figure 17C and Figure 17DThe controller 1736 may provide signals (e.g., signals representing commands or messages) to one or more components and / or systems of the machine 1700, including components such as CPUs, GPUs, and accelerators. Although the controller 1736 is listed separately from the SoC 1704, this is not intended to be limiting, and in some embodiments, one or more components of the SoC 1704 may perform the operations of the controller 1736. For example, the controller may send signals to operate machine brakes via one or more brake actuators 1748, to operate steering system 1754 via one or more steering actuators 1756, to operate propulsion system 1750 via one or more throttle / accelerators 1752, and so on. The controller 1736 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous or semi-autonomous navigation and movement and / or assist human operators using the machine 1700. Controller 1736 may include a first controller 1736 for autonomous control and navigation functions, a second controller 1736 for functional safety functions, a third controller 1736 for artificial intelligence functions (e.g., computer vision), a fourth controller 1736 for infotainment functions, a fifth controller 1736 for redundancy in emergency situations, and / or other controllers. For example, hardware for safety monitoring and other safety functions (e.g., functional safety islands) may be discrete or partitioned (physically or by processing separation) relative to hardware for processing sensor data to make perception and vehicle control decisions. Similarly, hardware for controlling in-vehicle infotainment and / or in-cabin monitoring (e.g., controllers, SOCs, etc.) may be separate or independent from hardware for vehicle perception and control. In some examples, a single controller 1736 may handle two or more of the above functions, two or more controllers 1736 may handle a single function, and / or any combination thereof.

[0247] The controller 1736 may provide signals for controlling one or more components and / or systems of the machine 1700 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data can be received from, for example but not limited to, Global Navigation Satellite System (“GNSS”) sensor 1758 (e.g., Global Positioning System sensor), RADAR sensor 1760, ultrasonic sensor 1762, LiDAR sensor 1764, Inertial Measurement Unit (IMU) sensor 1766 (e.g., accelerometer, gyroscope, magnetic compass, magnetometer, etc.), microphone 1796, camera 1768 (e.g., stereo camera 1768A, wide-angle camera 1768B (e.g., fisheye camera), infrared camera 1768C, surround camera 1768D (e.g., 360-degree camera), long-range and / or medium-range camera 1768E and / or other types of cameras), speed sensor 1744 (e.g., for measuring the speed of machine 1700), vibration sensor 1742, steering sensor 1740, braking sensor (e.g., as part of braking sensor system 1746), actuators and / or other types of sensors.

[0248] One or more controllers 1736 may receive input (e.g., represented by input data) from the instrument panel 1732 of the machine 1700 and provide output (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 1734 (e.g., screen, head-up display, mirror display, face display, robot display, etc.), sound alarms, loudspeakers, speakers, and / or via other components of the machine 1700. Outputs may include parameters such as machine speed, rate, time, and... Figure 17C The system can display map data corresponding to map 1722 (e.g., from a navigation map, a standard definition (SD) map, a high definition (“HD”) map, etc.), location data (e.g., the location of machine 1700, such as its location on map 1722), direction, the location of other vehicles (e.g., occupancy map, elevation map, bird's-eye view (BEV) image, grid, etc.), information about objects perceived by the system and their status, system status information, and so on. For example, HMI display 1734 can display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.), and / or information about driving maneuvers that the vehicle has performed, is performing, or will perform (e.g., changing lanes now, exiting from exit 34B in two miles, etc.).

[0249] Machine 1700 may include one or more System-on-a-Chip (SoC) 1704 (in Figure 17D(described in more detail below). SoC 1704 may include CPU 1706, GPU 1708, processor 1710, cache 1712, accelerator 1714, data storage 1716, and / or other components and features. SoC 1704 can be used to process and provide data for various operations, such as navigation, planning, reasoning, inference, perception, control, and / or actuation operations of machine 1700 on various platforms and systems. For example, SoC 1704 can process real-time perception data (e.g., from cameras, LiDAR, RADAR, ultrasound, etc.) and map data corresponding to one or more maps 1722 (e.g., HD maps, SD maps, navigation maps, occupancy maps, etc.) to perform or assist in performing various operations of machine 1700. When using maps and / or AI, maps and / or AI (e.g., model parameter updates, fine-tuning, etc.) are transmitted via network interface 1724 from one or more servers (e.g., [unclear]). Figure 17E Refresh and / or update the server 1778 (e.g., one or more servers in a cloud-based data center).

[0250] Despite Figures 17A to 17E The SoC 1704 is illustrated; however, additional or alternative components and / or architectures, such as multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiPs), field-programmable gate arrays (FPGAs), heterogeneous integration (HI), and single-board computers (SBCs), may be used without departing from the scope of this disclosure. For example, depending on the type of machine 1700, the purpose of machine 1700, the model of machine 1700, and the capabilities required by machine 1700, one or more SoC 1704s and / or alternative architectures and / or components may be used to meet a particular implementation.

[0251] Machine 1700 may include CPU 1718 (e.g., a discrete CPU or dCPU) which may be coupled to SoC 1704 via a high-speed interconnect (e.g., PCIe). CPU 1718 may include, for example, an x86 processor. CPU 1718 may be used to perform any of a variety of functions, including arbitrating potentially inconsistent results between ADAS sensors and SoC 1704, and / or monitoring the status and health of controller 1736 and / or infotainment SoC 1730.

[0252] Machine 1700 may include GPU 1720 (e.g., a discrete GPU or dGPU) which may be coupled to SoC 1704 via a high-speed interconnect (e.g., NVIDIA's NVLink, Ultra Accelerator Link (UALink), etc.). GPU 1720 may provide additional artificial intelligence capabilities, such as by executing redundant and / or different neural networks, and may be used to train and / or update neural networks based on inputs from sensors (e.g., sensor data) from Machine 1700.

[0253] The machine 1700 may also include a network interface 1724, which may include one or more wireless antennas 1726 and / or modems (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 1724 can be used to establish wireless connectivity with the cloud (e.g., with server 1778 and / or other network devices) via the Internet, and with other vehicles and / or computing devices (e.g., passenger client devices). For communication with other vehicles, direct links and / or indirect links (e.g., across networks and via the Internet) can be established between the two vehicles. A vehicle-to-vehicle communication link can be used to provide a direct link. The vehicle-to-vehicle communication link can provide the machine 1700 with information about vehicles near the machine 1700 (e.g., vehicles in front, to the side, and / or behind the machine 1700). This functionality can be part of the machine 1700's cooperative adaptive cruise control function.

[0254] Network interface 1724 may include a System-on-Chip (SoC) that provides modulation and demodulation functions and enables controller 1736 to communicate over a wireless network. Network interface 1724 may include a radio frequency (RF) front-end for up-conversion from baseband to radio frequency (RF) and down-conversion from RF to baseband. Frequency conversion can be performed through well-known processes and / or using superheterodyne processes. In some examples, the RF front-end functionality may be provided by a separate chip. For example, network interface 1724 may be able to communicate via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multicarrier (“CDMA2000”), 5G, 6G, and / or other cellular and / or wireless communication standards. The wireless antenna 1726 can also enable communication between objects in the environment (such as vehicles, mobile devices, etc.) using local area networks (such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc.) and / or low power wide area networks (“LPWAN”) (such as LoRaWAN, SigFox, etc.).

[0255] Machine 1700 may also include data memory 1728, which may include off-chip (e.g., outside of SoC 1704) storage. Data memory 1728 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, Flash, hard disk and / or other components and / or devices capable of storing at least one bit of data.

[0256] The machine 1700 may also include a GNSS sensor 1758. The GNSS sensor 1758 (e.g., a GPS, an auxiliary GPS sensor, a differential GPS (DGPS) sensor, etc.) is used to assist in mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 1758 can be used, such as, but not limited to, GPS sensors with a USB connector having an Ethernet-to-serial (RS-232) bridge.

[0257] Machine 1700 may also include IMU sensor 1766. In some examples, IMU sensor 1766 may be located at the center of the rear axis of machine 1700. IMU sensor 1766 may include, for example, but not limited to, accelerometers, magnetometers, gyroscopes, magnetic compasses, and / or other sensor types. In some examples, such as in a six-axis application, IMU sensor 1766 may include accelerometers and gyroscopes, while in a nine-axis application, IMU sensor 1766 may include accelerometers, gyroscopes, and magnetometers.

[0258] In some embodiments, the IMU sensor 1766 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 1766 can enable the machine 1700 to estimate heading by directly observing and correlating velocity changes from GPS to the IMU sensor 1766, without requiring input from a magnetic sensor. In some examples, the IMU sensor 1766 and the GNSS sensor 1758 can be combined in a single integrated unit.

[0259] The vehicle may include one or more microphones 1796 placed inside and / or around the machine 1700. The microphones 1796 may be used for emergency vehicle detection and identification, etc.

[0260] Machine 1700 may also include vibration sensor 1742. Vibration sensor 1742 can measure vibrations of machine parts, such as the arm or leg of humanoid robot 1700C, or the axle of vehicle 1700A or AMR 1700B. For example, changes in vibration may indicate changes in roads, walking, or traversable surfaces. In another example, when two or more vibration sensors 1742 are used, differences between vibrations can be used to determine friction or slippage on surfaces (e.g., when the vibration difference is between an electrically driven shaft and a freely rotating axle).

[0261] Machine 1700 may include ADAS system 1738, for example, when machine 1700 is vehicle 1700A. In some examples, ADAS system 1738 may include a dedicated SoC. ADAS system 1738 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision or collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), blind spot monitoring (BSM), rear cross traffic warning (RCTW), pedestrian detection, driver monitoring, collision warning system (CWS), traffic sign recognition, speed limit detection, automatic parking, lane centering (LC), high beam safety system and / or other features and functions.

[0262] Machine 1700 may also include an infotainment SoC 1730 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include one or more discrete components, such as multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiP), heterogeneous integration (HI), single-board computers (SBCs), etc. The infotainment SoC 1730 may include a combination of hardware and software that can be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., television, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., wireless, Wi-Fi, etc.), and / or information services (e.g., navigation systems, rear parking assist, radio data systems, vehicle-related information (e.g., fuel level, total mileage, brake fluid level, fuel level, door opening / closing, air filter information, etc.) to the machine 1700. For example, the infotainment SoC 1730 may be a radio, disk player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, Wi-Fi, steering wheel audio controls, hands-free voice control, head-up display (HUD), HMI display 1734, telematics device, control panel (e.g., for controlling various components, features, and / or systems and / or interacting with various components, features, and / or systems), and / or other components. 1730 can also be used to provide information to vehicle users (e.g., visual and / or auditory), such as information from ADAS system 1738, autonomous driving information (e.g., planned vehicle maneuvers, trajectories), surrounding environment information (e.g., intersection information, vehicle information, road information, etc.) and / or other information.

[0263] The infotainment SoC 1730 may include GPU functionality. The infotainment SoC 1730 can communicate with other devices, systems, and / or components of the machine 1700 via bus 1702 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 1730 may be coupled to a monitoring MCU, allowing the GPU of the infotainment system to perform some autonomous driving functions in the event of a failure of the main controller 1736 (e.g., the main computer and / or backup computer of the machine 1700). In such an example, the infotainment SoC 1730 may place the machine 1700 into a driver-safe stop mode, as described herein.

[0264] In some embodiments, the infotainment system may provide a digital or virtual assistant, which may be voice-only or may have visual components (e.g., in the form of a digital human or digital avatar). The assistant may provide basic functions such as sending text messages, adjusting vehicle settings, controlling music or video, navigation, etc., and / or provide more advanced functions, such as those supported by one or more language models (e.g., large language model (LLM), visual language model (VLM), multimodal language model (MMLM), etc.). For example, the driver and / or occupants may interact with the assistant in a manner similar to how a user interacts with a language model, such as asking general questions, specific questions, requesting restaurants, gas stations, and / or other recommendations and / or locations, inquiring about vehicle functions or troubleshooting (e.g., asking for tire pressure information, oil change information, battery swap information, etc.). Therefore, the Machine 1700 (whether it is a vehicle 1700A, AMR 1700B, humanoid robot 1700C, or other type of machine) may include a locally stored language model and / or communicate with a remotely hosted language model (e.g., via one or more APIs) to provide users of the Machine 1700 with more detailed and in-depth communication capabilities.

[0265] In some examples, the infotainment SoC 1730, SoC 1704, and / or another SoC or computing / processing system can perform in-cabin driver and / or occupant monitoring. For example, the computing system can perform facial recognition, and vehicle owner recognition can use data from cameras and / or other sensors to identify the presence of an authorized driver and / or owner of the machine 1700. An always-on sensor processing engine can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in security mode, to disable the vehicle when the owner leaves. In this way, SoC 1704 can provide security against theft and / or carjacking.

[0266] In some embodiments, one or more neural networks running on another or a dedicated SoC (e.g., an in-vehicle infotainment or in-vehicle monitoring SoC) can be used to monitor in-cabin monitoring camera sensors. This other or dedicated SoC is configured to recognize in-cabin events and respond accordingly. The in-cabin system can activate cellular services and make phone calls by lip reading, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. The in-cabin system may also include one or more in-cabin AI agents or assistants that can interact with one or more LLMs, VLMs, MMLMs, etc., in the cloud using one or more APIs or plugins. For example, the in-cabin AI agents or assistants can provide directions, vehicle or machine feedback information, answer general questions, handle music / video and / or other requests, activate windows, doors, and / or other vehicle components, etc. Therefore, one or more dedicated SoCs and / or processor groups can be used to perform in-cabin infotainment and / or in-cabin monitoring (e.g., as an occupant monitoring system (OMS)) for Machine 1700.

[0267] Machine 1700 may also include instrument panel 1732 (e.g., digital instrument panel, electronic instrument panel, digital dashboard, etc.). Instrument panel 1732 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). Instrument panel 1732 may include a set of instruments such as speedometer, fuel level, oil pressure, tachometer, odometer, turn signal indicator, gearshift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting control, safety system control, navigation information, etc. In some examples, information may be displayed and / or shared between infotainment SoC 1730 and instrument panel 1732. In other words, instrument panel 1732 may be included as part of infotainment SoC 1730, and vice versa.

[0268] Figure 17D A computing system according to at least some embodiments of this disclosure (about Figure 17C A block diagram of an example architecture (a subset of the systems described). Although illustrated as SoC 1704, this is not intended to be limiting, and the computing system may additionally or alternatively include multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiP), heterogeneous integration (HI), single-board computers (SBCs), and / or other components and / or architectures without departing from the scope of this disclosure.

[0269] The SoC 1704 can be an end-to-end platform with a flexible architecture spanning automation levels 2-5, or it can be specifically designed for a particular automation level (e.g., a first SoC 1704 for levels 2 to 2++, a second for level 3, a third for level 4, and so on), providing a comprehensive functional safety architecture that leverages and effectively utilizes computer vision, neural network inference, robot planning, control and navigation, ADAS technologies, and more. This architecture offers versatility and redundancy to provide a flexible and reliable platform for driving or robot control software stacks, as well as deep learning tools. The SoC 1704 can be faster, more reliable, and even more energy-efficient and space-saving than conventional systems. For example, when the accelerator 1714 is used in conjunction with the CPU 1706, GPU 1708, and data storage 1716, it can provide a fast and efficient platform for Level 2-5 autonomous vehicles, as well as for the safety planning, navigation, and control of AMR 1700B, humanoid robot 1700C, and / or other robot or machine types.

[0270] In some embodiments, for example, SoC 1704 includes: a GPU 1708 having 2000 or more cores (e.g., 2048 cores), 60 or more tensor cores (e.g., 64 tensor cores), and a maximum GPU frequency exceeding 1 GHz (e.g., 1.3 GHz); a CPU 1706 including 10 or more cores (e.g., 12 cores), having 64-bit, 3 MB L2 and 6 MB L3 caches, and a maximum frequency of 2 GHz or more (e.g., 2.2 GHz); one or more deep learning accelerators (DLAs); deep learning accelerator clusters (XNNs); neural network accelerators (NNAs) or neural processing units (NPUs) 1709 (e.g., two DLAs / XNNs / NNAs / NPUs 1709); and a vision accelerator (e.g., a programmable vision accelerator (PVA)) 1707. A single SoC 1704 may be capable of achieving AI performance of 275 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGX Orin. The 64GB SoC meets these standards and achieves this level of performance.

[0271] Similarly, in an embodiment, SoC 1704 includes: a GPU 1708 having 1700 or more cores (e.g., 1792 cores), 50 or more tensor cores (e.g., 56 tensor cores), and a maximum GPU frequency exceeding 900MHz (e.g., 930MHz); a CPU 1706 including 8 or more cores (e.g., 8 cores), having 64-bit, 2MB L2 and 4MB L3 cache memory, and a maximum frequency of 2GHz or more (e.g., 2.2GHz); one or more deep learning accelerators (DLAs); a deep learning accelerator cluster (XNN); a neural network accelerator (NNA) or neural processing unit (NPU) 1709 (e.g., 2 DLAs / XNNs / NNAs / NPUs 1709); and a vision accelerator (e.g., a programmable vision accelerator (PVA) 1707). A single SoC 1704 may be capable of achieving AI performance of 200 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGX Orin 32GB SoC meets these standards and achieves such performance.

[0272] In some embodiments, such as SoC 1704, the SoC 1704 may include: a GPU 1708 having 1,000 or more cores (e.g., 1,024 cores), 28 or more tensor cores (e.g., 32 tensor cores) and a maximum GPU frequency exceeding 900 MHz (e.g., 1,173 MHz); a CPU 1706 having 8 or more cores (e.g., 8 cores), 64-bit, 2 MB L2 and 4 MB L3 cache memory, and a maximum frequency of 2 GHz or more (e.g., 2 GHz); one or more deep learning accelerators (DLAs); a deep learning accelerator cluster (XNN); a neural network accelerator (NNA) or neural processing unit (NPU) 1709 (e.g., one DLA / XNN / NNA / NPU 1709); and a vision accelerator (e.g., a programmable vision accelerator (PVA) 1707). A single SoC 1704 may be capable of achieving AI performance of 157 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGX Orin NX 16GB SoC meets these standards and achieves such performance.

[0273] In various embodiments, for example, the SoC 1704 includes: a GPU 1708 having 1000 or more cores (e.g., 1024 cores), 28 or more tensor cores (e.g., 32 tensor cores), and a maximum GPU frequency exceeding 900MHz (e.g., 1020MHz); and a CPU 1706 including 6 or more cores (e.g., 6 cores), having 64-bit, 1.5MB L2 and 4MB L3 cache memory, and a maximum frequency of 1.5GHz or higher (e.g., 1.7GHz). A single SoC 1704 can achieve AI performance of 67 trillion operations per second (TOPS). For example, NVIDIA's Jetson Orin Nano 8GB SoC meets these standards and achieves such performance.

[0274] SoC 1704 may include one or more CPUs 1706. In embodiments, CPU 1706 may include CPU clusters or CPU complexes (also referred to herein as “CCPLEX”). CPU 1706 may include multiple cores and / or (e.g., L2, L3) caches. For example, in some embodiments, CPU 1706 may include twelve cores in a coherent multiprocessor configuration. In some embodiments, CPU 1706 may include four dual-core clusters, each with a dedicated L2 cache (e.g., 3MB L2 cache). CPU 1706 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, thereby allowing any combination of CPU 1706 clusters to be active at any given time.

[0275] The SoC 1704 may include any type and number of GPUs 1708. For example, in some embodiments, an integrated GPU (also referred to herein as an "iGPU") may be used. The GPU 1708 may be programmable and can be used efficiently for parallel workloads. In some examples, the GPU 1708 may use an enhanced tensor instruction set. The GPU 1708 may include one or more streaming microprocessors, wherein each streaming microprocessor may include a cache (e.g., an L1 cache with at least 96KB of storage), and two or more streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage). In some embodiments, the GPU 1708 may include at least eight streaming microprocessors. The GPU 1708 may use a computing application programming interface (API). Furthermore, the GPU 1708 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0276] The GPU 1708 can be power-optimized for optimal performance in automotive, robotics, and / or other embedded use cases. For example, the GPU 1708 can be fabricated on FinFETs. However, this is not intended to be limiting, and the GPU 1708 can be fabricated using other semiconductor fabrication or manufacturing processes. Each streaming microprocessor can include multiple mixed-precision processing cores divided into multiple blocks. For example, but not limited to, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSORCOREs for deep learning matrix algorithms, one (e.g., L0) instruction cache, one warp scheduler, one dispatch unit, and / or one (e.g., 64KB) register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to efficiently perform workloads involving mixed computation and addressing computation. Streaming microprocessors can include independent thread scheduling capabilities to enable finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors can also include combined L1 data caches and shared memory units to improve performance while simplifying programming.

[0277] The GPU 1708 may include high-bandwidth memory (HBM) and / or (e.g., 16GB) HBM2 memory subsystems to provide peak memory bandwidth of approximately 900GB / s in some examples. In some examples, synchronous graphics random access memory (SGRAM), such as graphics double data rate type 5 synchronous random access memory (GDDR5), may be used in addition to or as an alternative to HBM memory.

[0278] The GPU 1708 may include unified memory technology, including access counters, to allow more accurate migration of memory pages to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, Address Translation Service (ATS) support may be used to allow the GPU 1708 to directly access the CPU 1706 page tables. In such examples, when a miss occurs in the GPU 1708 Memory Management Unit (MMU), an address translation request can be transferred to the CPU 1706. In response, the CPU 1706 can look up the virtual-to-physical mapping of the address in its page tables and transfer the translation back to the GPU 1708. Therefore, unified memory technology allows the memory of the CPU 1706 and GPU 1708 to use a single, unified virtual address space, thereby simplifying GPU 1708 programming and porting applications to the GPU 1708.

[0279] SoC 1704 may include any number of caches 1712, including the caches described herein. For example, cache 1712 may include L0 cache, L1 cache, L2 cache, L3 cache (e.g., available for CPU 1706 and GPU 1708 (e.g., connected to CPU 1706 and GPU 1708)), etc. Cache 1712 may include write-back caches that can track the state of the lines, for example, by using one or more cache coherence protocols (e.g., MEI, MESI, MSI, etc.). According to embodiments, the cache (e.g., L3) may include 4MB or more, although smaller or larger cache sizes may be used.

[0280] SoC 1704 may include one or more arithmetic logic units (ALUs) 1765, which can be used to perform processing associated with various tasks or operations of machine 1700, such as computer vision, machine learning or deep learning processing, world model management, etc. Additionally, SoC 1704 may include a floating-point unit (FPU) 1767 or other mathematical coprocessors or digital coprocessors for performing mathematical operations within the system. For example, SoC 1704 may include one or more FPUs 1767, which are integrated as execution units within CPU 1706 and / or GPU 1708.

[0281] The SoC 1704 may include one or more accelerators 1714 (e.g., hardware accelerators, software accelerators, or combinations thereof). For example, the SoC 1704 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. Large on-chip memory 1715 (e.g., 4MB SRAM, 32GB and / or 64GB 256-bit LPDDR5 (204.8GB / s), 8GB and / or 16GB 128-bit LPDDR5 (102.4GB / s), and / or other memory types and sizes) enables the hardware acceleration cluster to accelerate neural network processing, transformer processing, optical flow processing, vision processing, and / or other computations or processing. The hardware acceleration cluster can be used to supplement the GPU 1708 and offload some tasks from the GPU 1708 (e.g., freeing up more cycles of the GPU 1708 to perform other tasks). As an example, the accelerator 1714 can be used for target workloads (e.g., perceptual, convolutional neural network (CNN), deep neural network (DNN), language models (LLM, VLM, MMLM, VLA, etc.), transformer models, diffusion models, encoder-only models, encoder-decoder models, etc., which are stable enough to be suitable for acceleration.

[0282] Accelerator 1714 (e.g., a hardware acceleration cluster) may include a Deep Learning Accelerator (DLA) 1709 (also referred to herein as “Deep Learning Accelerator Cluster (XNN) 1709”, “Neural Network Accelerator (NNA) 1709”, or “Neural Processing Unit (NPU) 1709”). DLA 1709 may include one or more Tensor Processing Units (TPUs) 1741, which may be configured to provide additional, for example, trillions of operations per second for deep learning applications and inference. TPU 1741 may be an accelerator configured to perform image processing functions (e.g., for CNNs, RCNNs, DNNs, etc.) and optimized for them. DLA 1709 may be further optimized for a specific set of neural network types and floating-point operations and inference. The DLA is designed to provide higher performance per millimeter than general-purpose GPUs and significantly outperforms CPUs. TPU 1741 can perform several functions, including single-instance convolution functions, support for INT8, INT16, and FP16 data types for features and weights, and post-processor functions. Although TPU 1741 is described as being included as part of DLA 1709, this is not intended to be limiting, and TPU 1741 may be included in additional or alternative accelerator 1714 and / or other components, and / or may be included as a discrete processing component.

[0283] The DLA 1709 can quickly and efficiently execute neural networks on processed or unprocessed data to achieve a variety of functions, including but not limited to: object and feature recognition and detection using data from one or more sensor modalities (e.g., vehicles, pedestrians, other robots, lane lines, road boundary lines, debris, potholes, boxes, warehouse items, etc.); distance estimation using data from one or more sensor modalities; emergency vehicle detection and identification using data from microphones and / or vision-based sensors; facial recognition; pick-up and place operations; maneuvering operations; occupant monitoring; vehicle owner identification; and / or other in-cabin operations using data from in-cabin cameras and / or other sensor types; and / or safety and / or safety-related events, to name a few.

[0284] The DLA 1709 can perform any function of the GPU 1708, and by using inference accelerators, for example, designers can anchor either the DLA 1709 or the GPU 1708 for any function. For instance, designers can centralize the processing of DNNs and floating-point operations on the DLA 1709, leaving other functions to the GPU 1708 and / or other accelerators 1714. The DLA 1709 can be used to run any type of network to enhance control and security, including neural networks that, for example, output a confidence metric for each object detection.

[0285] Accelerator 1714 (e.g., a hardware acceleration cluster) may include a programmable vision accelerator (PVA) 1707, which may alternatively be referred to herein as a computer vision accelerator or generally as a vision accelerator. The PVA 1707 may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), semi-autonomous driving, autonomous driving, robotics applications, safety and supervision applications, augmented reality (AR), virtual reality (VR), and / or mixed reality (MR) applications. The PVA 1707 provides a balance between performance and flexibility. For example, each PVA 1707 may include (e.g., but not limited to) any number of Reduced Instruction Set Computer (RISC) cores, Direct Memory Access (DMA) systems, Pixel Processing Engines (PPEs), Vector Processors or Vector Processing Units (VPUs), and / or other components. The PVA engine may include an Advanced Very Long Instruction Word (VLIW) or Single Instruction Multiple Data (SIMD) digital signal processor. The PVA 1707 may be optimized for image processing and computer vision algorithm acceleration tasks. For example, the PVA 1707 offers excellent performance with extremely low power consumption and can be used asynchronously and concurrently as part of a heterogeneous computing pipeline with CPU 1706, GPU 1708 and / or other accelerators in systems such as vehicles, robots and so on.

[0286] The PVA 1707 may include one or more (e.g., two) Vector Processing Subsystems (VPSs), each of which may include one or more Vector Processing Unit (VPU) cores, one or more Decoupled Lookup Units (DLUTs), one or more shared or vector memory (VMEMs), and one or more instruction caches (I-caches). The VPU core may be the main processing unit and may include a vector SIMD VLIW DSP 1743 optimized for computer vision. The VPU core can fetch instructions via the I-cache and access data via the VMEM. The DLUT may include dedicated hardware components that enhance the efficiency of parallel lookup operations. For example, the DLUT allows parallel lookups using a single copy of the lookup table by performing these lookups in a decoupled pipeline independent of the main processor pipeline. By doing so, the DLUT can minimize or reduce memory usage and increase throughput while avoiding data-related memory library conflicts, ultimately leading to improved overall system performance. The VPU VMEM can provide local data storage for the VPU, allowing for the efficient implementation of various image processing and computer vision algorithms. The VPU VMEM can support access from external VPS hosts, such as Direct Memory Access (DMA) and the CPU1706 (e.g., an ARM Cortex-R5 processor), thereby facilitating data exchange with the CPU1706 and other system-level components. The VPU I-cache can provide instruction data to the VPU on request, request missing instruction data from system memory, and / or maintain temporary instruction storage for the VPU. For each VPU task, the CPU1706 can configure the DMA system, optionally prefetching the VPU program into the VPU I-cache, and / or initiating each VPU-DMA pair to process the task. The PVA 1707 may also include L2 SRAM memory shared between one or more (e.g., two) sets of VPS and DMA. In some embodiments, one or more (e.g., two) DMA devices are used to move data between external memory, PVA L2 memory, VMEM (e.g., one per VPS), CPU tightly coupled memory (TCM), DMA descriptor memory, and / or PVA-level configuration registers. In lightly loaded systems, two parallel DMA accesses to DRAM can achieve read / write bandwidth of up to 15 GB / s, while in heavily loaded systems, this bandwidth can reach up to 10 GB / s. Regarding compute capacity, INT8 gigabyte multiply-accumulate operations per second (GMAC) can be 2048 or greater, excluding DLUT. FP32 GMACs can be 32 per PVA instance.

[0287] The RISC core can interact with image sensors (e.g., the image sensor of any camera described herein), image signal processors, etc. Each RISC core may include any amount of memory. The RISC core can use any of a variety of protocols, depending on the implementation. In some examples, the RISC core can execute a real-time operating system (RTOS). The RISC core can be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, the RISC core may include an instruction cache and / or tightly coupled RAM.

[0288] The DMA system enables PVA 1707 components to access system memory independently of the CPU 1706. DMA can support any number of functions optimized for the PVA 1707, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which may include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0289] A vector processor, or VPU, can be a programmable processor designed to efficiently and flexibly execute computer vision algorithms and provide signal processing capabilities. In some examples, the PVA 1707 may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA 1707 and may include one or more vector processing units (VPUs), one or more pixel processing engines (PPEs) (which may include a 2D layout of interconnected (e.g., for north, south, east, and west communication) processing elements), one or more instruction caches, and / or one or more shared or vector memories (e.g., VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can improve throughput and speed.

[0290] In some embodiments, each vector processor may include an instruction cache and may be coupled to dedicated memory. Therefore, in some examples, each vector processor may be configured to execute independently of other vector processors. In other examples, the vector processors included in a particular PVA 1707 may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA 1707 may execute the same computer vision algorithm, but for different regions of an image. In other examples, the vector processors included in a particular PVA 1707 may execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on consecutive images or portions of an image. Among other things, the hardware acceleration cluster may include any number of PVA 1707s, and each PVA may include any number of vector processors. Furthermore, the PVA 1707 may include additional error correction code (ECC) memory to enhance overall system security.

[0291] Accelerator 1714 (e.g., hardware accelerator clusters) has broad applications for autonomous and semi-autonomous machine control. PVA 1707 can be a programmable vision accelerator used in critical processing stages in perception, robot understanding and reasoning, ADAS, semi-autonomous and autonomous vehicles, etc. The capabilities of PVA 1707 are well-suited for algorithmic domains requiring predictable processing, featuring low power consumption and low latency. In other words, PVA 1707 performs well in semi-intensive or intensive rule computation, even on small datasets requiring predictable runtime, low latency, and low power consumption. Therefore, in the context of autonomous vehicles and robotic platforms, PVA 1707 is designed to run classic computer vision algorithms, as they are highly efficient in object detection and integer mathematical operations.

[0292] For example, according to one embodiment of this technology, the PVA 1707 is used to perform computer stereo vision. In some examples, semi-global matching-based algorithms can be used, although this is not intended to be limiting. Many Level 3-5 autonomous driving applications require on-the-fly motion estimation / stereo matching (e.g., moving structures, pedestrian recognition, lane detection, etc.). The PVA 1707 can perform computer stereo vision functions on input from two monocular cameras.

[0293] In some examples, the PVA 1707 can be used to perform dense optical flow, providing processed RADAR data based on the raw RADAR data (e.g., using 4D Fast Fourier Transform). In other examples, the PVA 1707 is used for time-of-flight depth processing, for example, providing processed time-of-flight data by processing the raw time-of-flight data.

[0294] While the VPU, DMA, RISC Core, VMEM, and decoupled coprocessors (e.g., DLUT) are described as being included within the PVA1707, this does not imply limitation. In some embodiments, these components may be included in alternative or additional processing components and / or accelerators 1714, and / or may be included as discrete components of the SoC 1704 and / or other computing system architectures.

[0295] In some examples, SoC 1704 may include a real-time ray tracing hardware accelerator (RTA) 1751, which can be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) to generate real-time or near-real-time visualization simulations, for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for simulation of SONAR, RADAR, LiDAR, camera and / or other sensor modalities in a simulation, for general wave propagation simulation, for comparison with LiDAR data for localization, for generating realistic training data for training neural networks, and / or other functions and uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations. For example, machine 1700 (or another machine or device) may perform simulations in a simulation environment, and one or more light transport simulation algorithms (e.g., ray tracing, path tracing, etc.) may be used to generate the simulation environment. Therefore, the ray tracing accelerator 1751 and / or a ray tracing-optimized GPU 1706 (e.g., NVIDIA's RTX GPU) may be used to accelerate these ray tracing algorithms.

[0296] Accelerator 1714 (e.g., in a hardware acceleration cluster) may include one or more optical flow accelerators (OFAs) 1711. For example, OFA 1711 can be used to calculate optical flow and stereo disparity between sensor data frames (e.g., images). Optical flow can be accelerated on OFA 1711 for purposes such as object detection and tracking, and / or for stereo depth estimation, where stereo disparity is calculated between stereo image frames (e.g., two or more frames captured using two or more image sensors with at least partially overlapping fields of view).

[0297] SoC 1704 may include one or more Camera Serial Interfaces (CSIs) 1723. For example, CSI 1723 may include a Mobile Industry Processor Interface (MIPI) Camera Serial Interface (CSI) for receiving video and input from cameras, a high-speed interface, and / or a video input block for camera and associated pixel input functions. SoC 1704 may also include a software-controllable input / output controller and may be used to receive I / O signals not assigned to a specific role. For example, CSI 1723 may include MIPI CSI-2 connectors, such as a 16-channel MIPI CSI-2 connector, D-PHY 2.1 (up to 40Gbps), and C-PHY 2.0 (up to 164Gbps) to support 16 virtual channels and 6 or more cameras; an 8-channel MIPI CSI-2 connector, D-PHY 2.1 (up to 20Gbps) to support 8 virtual channels and 4 or more cameras; and / or 2x MIPI CSI-2, 22-pin camera connectors, depending on the embodiment and implementation.

[0298] Accelerator 1714 (e.g., a hardware acceleration cluster) may include an on-chip computer vision network (CVNOC) 1763 and SRAM for providing high-bandwidth, low-latency SRAM to accelerator 1714. In some examples, on-chip memory may include at least 4 MB of SRAM, such as, but not limited to, eight field-configurable memory blocks accessible by PVA 1707, OFA1711, DLA 1709, and / or other accelerator 1714. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory 1715 may be used. PVA 1707, OFA1711, DLA 1709, and / or other accelerator 1714 may access memory via a backbone providing high-speed memory access to accelerator 1714. The backbone may include an on-chip computer vision network that interconnects accelerator 1714 to memory (e.g., using APB).

[0299] CGNOC 1763 may include an interface that determines whether the accelerator 1714 provides ready and valid signals before transmitting any control signals / addresses / data. Such an interface can provide separate stages and separate channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, although other standards and protocols may be used.

[0300] SoC 1704 may include a data repository 1716 and / or memory 1715. The data repository 1716 may be on-chip memory 1715 of SoC 1704, which may store neural networks and / or other algorithms to be executed on CPU 1706, GPU 1708, and / or one or more accelerators 1714. In some examples, the capacity of the data repository 1716 may be large enough to store multiple neural network instances for redundancy and security. The data repository 1716 may include, for example, L2 and / or L3 cache 1712. The memory 1715 may include SRAM, LPDDR5, and / or other memory types. For example, the memory 1715 may include 4MB SRAM, 32GB and / or 64GB 256-bit LPDDR5 (204.8GB / s), 8GB and / or 16GB 128-bit LPDDR5 (102.4GB / s), and / or other memory types and sizes. References to data repository 1716 may include references to memories associated with PVA 1707, OFA 1711, DLA 1709 and / or other accelerators 1714, as described herein.

[0301] Data repository 1716 may include various storage types, such as eMMC, NVMe, etc. For example, SoC 1704 may include storage in the form of an embedded multimedia card (eMMC) (e.g., 64GB eMMC 5.1) and / or an SD card slot, with external NVM Express (NVMe) capability, for example, via M.2 Key M. For example, data repository 1716 and / or other storage may be accessed via, for example, NVMe using PCI Express (PCIe), RDMA, TCP, and / or other protocols.

[0302] The SoC 1704 may include one or more processors 1710 (e.g., embedded processors). The processor 1710 may include a boot and power management processor (BPMP) 1753, which may be a dedicated processor and subsystem for handling boot power and management functions, as well as associated security implementations. The BPMP 1753 may be part of the SoC 1704 boot sequence and may provide runtime power management services. The BPMP 1753 may provide clock and voltage programming, assistance with low-power state transitions of the system, management of the SoC 1704 thermal and temperature sensors, and / or management of the SoC 1704 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and the SoC 1704 may use the ring oscillator to detect the temperature of the CPU 1706, GPU 1708, accelerator 1714, and / or other components. If the temperature is determined to exceed the threshold, the BPMP 1753 may enter the temperature fault routine and put the SoC 1704 into a lower power state and / or put the machine 1700 into a driver safety stop mode (e.g., to safely stop the machine 1700).

[0303] The processor 1710 may also include a set of embedded processors that can be used as the Audio Processing Engine (APE) 1755. The APE 1755 can be an audio subsystem capable of providing full hardware support for multi-channel audio through multiple interfaces, as well as a wide and flexible audio I / O interface. In some examples, the APE 1755 is a dedicated processor core with a digital signal processor and dedicated RAM.

[0304] The processor 1710 may also include an always-on processor engine (AOPE) 1757, which can provide the necessary hardware features to support low-power sensor management and wake-up use cases. The AOPE 1757 may include a processor core, tightly coupled RAM, support for peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0305] Processor 1710 may also include a security processor 1713 (or "security island 1713"), which may include a security cluster engine comprising a dedicated processor or processor subsystem for handling security management for automotive, robotic, and / or other applications. Security processor 1713 and / or the security cluster engine may include two or more processor cores, tightly coupled RAM, support for peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In security mode, the two or more cores may operate in lockstep mode and as a single core, with comparison logic to detect any differences in their operation. In some embodiments, security processor 1713 may include discrete processors such that failure of other system components may not affect the performance and availability of security processor 1713.

[0306] The processor 1710 may also include a real-time or near-real-time sensor engine (SE) 1759, which may include a dedicated processor subsystem for handling real-time or near-real-time camera, LiDAR, RADAR and / or other sensor modal management.

[0307] The processor 1710 may also include one or more image signal processors (ISPs) 1727, which may include high dynamic range signal processors and / or hardware engines as part of one or more sensor processing pipelines.

[0308] Processor 1710 may include a video image synthesizer (VIC) 1761, which may be a processing block (e.g., implemented on a microprocessor) that implements the video post-processing functions required by the video playback application to produce the final image of the player window. VIC 1761 can perform lens distortion correction on wide-angle camera 1768B, surround camera 1768D, cabin monitoring camera sensor and / or other camera sensors with distorted field of view.

[0309] VIC 1761 may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, when motion occurs in the video, noise reduction appropriately weights spatial information, thereby reducing the weight of information provided by adjacent frames. When an image or part of an image does not contain motion, temporal noise reduction performed by the video image synthesizer can use information from previous images to reduce noise in the current image.

[0310] The VIC 1761 can also be configured to perform stereoscopic correction on input stereoscopic camera frames. The video image compositor can also be used for user interface compositing when the operating system desktop is in use and the GPU 1708 does not need to continuously render new surfaces. Even when the GPU 1708 is powered on and actively performing 3D rendering, the video image compositor can be used to offload the GPU 1708 to improve performance and responsiveness.

[0311] The SoC 1704 may also include various peripheral interfaces for input / output (I / O) 1725, for example, to enable communication with peripheral devices, audio codecs, power management and / or other devices. The SoC 1704 can be used to process data from cameras (e.g., via a gigabit multimedia serial link and / or Ethernet connection), from sensors (e.g., LiDAR sensor 1764, RADAR sensor 1760, etc., connected via Ethernet), from bus 1702 (e.g., speed of machine 1700, steering wheel position, etc.), and from GNSS sensor 1758 (e.g., connected via Ethernet or CAN bus). The SoC 1704 may also include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine and can be used to free the CPU 1706 from routine data management tasks. In some embodiments, the SoC 1704 I / O 1725 may include headers (e.g., 40-pin headers or 40-pin extension headers) supporting Universal Asynchronous Receiver / Transmitter (UART), Serial Peripheral Interface (SPI), Inter-Integrated Sound (I2S), Inter-Integrated Sound (I2C), Controller Area Network (CAN), Pulse Width Modulation (PWM), Digital Microphone Interface (DMIC), Digital Speaker Station (DSPK), General Purpose I / O (GPIO), etc., such as automation headers (e.g., 12-pin automation headers), audio panel headers (e.g., 10-pin audio panel headers), Joint Test Action Group (JTAG) headers (e.g., 10-pin JTAG headers), fan headers (e.g., 4-pin fan headers), RTC battery spare connectors (e.g., 2-pin battery spare connectors), microSD slot, DC power jack, power, force, resume, and reset buttons, and one or more display connectors (e.g., DisplayPort (DP), such as DP 1.4A (+MST), eDP 1.41, HDMI 2.1, and / or 4K30 multi-model DP). 1.2 (+MST) connector) and / or other I / O 1725 elements, components or features.

[0312] The SoC 1704 may include machine intranetting capabilities using, for example, Ethernet (e.g., automotive Ethernet), SERDES, Controller Area Network (CAN), FlexRay, Local Interconnect Network (LIN), Low Voltage Differential Signaling (LVDS), Media-Oriented System Transport (MOST), another network type, and / or combinations thereof. For example, the SoC 1704 may include RJ45 connectors with up to 10GbE, 1GbE connectors, and / or other network connector types.

[0313] The SoC 1704 may include one or more digital signal processors (DSPs) 1743. For example, the DSP 1743 may include a dedicated or specialized microprocessor chip optimized for digital signal processing, such as audio signal processing, telecommunications, digital image processing, RADAR, SONAR, LiDAR and / or other sensor processing, speech recognition and / or other applications.

[0314] SoC 1704 may include one or more video encoders 1719 and / or one or more video decoders 1721. For example, video encoder 1719 may include a hardware-based (e.g., as part of GPU 1708) video encoder (e.g., supporting H.264, H.265, etc., and conforming to HEVC standards, such as NVIDIA's NVENC) that can process image input (e.g., as YUV, RGB, etc.) to generate a video bitstream. Video decoder 1721 may include a video decoder engine that provides fully accelerated hardware video decoding capabilities (e.g., supporting decoding of various bitstream formats such as AV1, H.264, H.265, VP8, VP9, ​​MPEG-1, MPEG-2, MPEG-4, VC-1, etc., and conforming to HEVC standards, such as NVIDIA's NVDEC). In some examples, video decoder 1721 may be hardware-based (e.g., as part of GPU 1708).

[0315] The SoC 1704 may include one or more General Purpose Computing Acceleration Clusters (GCACs) 1729. For example, the GCAC 1729 may include various processor types that can be used for accelerated computing, such as one or more Vector Microcode Processors (VMPs) 1733, one or more Multi-Threaded Processing Clusters (MPCs) 1731, one or more Programmable Macroarrays (PMAs) 1735, and / or one or more other processor types. For example, the GCAC 1729 may include a PMA 1735, two VMPs 1733, and two MPCs 1731.

[0316] The SoC 1704 may include one or more vector microcode processors (VMPs) 1733. In embodiments, the VMP 1733 may include a wide vector (Very Long Instruction Word (VLIW) and Single Instruction Multiple Data (SIMD)) machine that performs a variety of operations, such as short integral-type operations common in computer vision and deep learning algorithms.

[0317] SoC 1704 may include one or more multi-threaded processing clusters (MPCs) 1731. MPCs 1731 may include processing clusters that are more general-purpose than GPUs and more efficient than CPUs in some embodiments. For example, an MPC 1731 may include a multi-threaded processor that allows multiple threads to share resources and execute instructions concurrently.

[0318] The SoC 1704 may include one or more programmable macro arrays (PMAs) 1735. The PMA 1735 may include a coarse-grained reconfigurable architecture (CGRA) dataflow machine, which has a unique architecture that delivers powerful performance on intensive computer vision and deep learning algorithms that may not be achievable in classic digital signal processing (DSP) architectures.

[0319] The SoC 1704 may include one or more Display Processing Units (DPUs) 1745 for performing hardware-accelerated image processing. For example, the DPU 1745 may retrieve pixel data from memory 1715 and send it to a display peripheral device via a standard interface. Therefore, the DPU 1745 can handle display processing and rendering for displays within and / or on the machine.

[0320] The SoC 1704 may include one or more Application Processing Units (APUs) 1739. For example, the APU 1739 may include a quad-core or dual-core processor with 48KB / 32KB L1 cache (with parity and ECC) and a 1MB L2 cache with ECC. The APU 1739 may support NEON instructions and single-precision and double-precision floating-point operations.

[0321] The SoC 1704 may include one or more Real-Time Processing Units (RTPUs) 1769. The RTPU 1769 may include a dual-core processor with 32KB / 32KB L1 cache and a 256KB TCM with ECC. The RTPU 1769 can support single-precision and double-precision floating-point operations.

[0322] SoC 1704 may include one or more built-in self-test (BIST) components 1737. For example, BIST component 1737 may include a memory BIST (MBIST) for testing the system's memory and / or a logic BIST (LBIST) for testing the system's logic. BIST component 1737 may include embedded logic for directly testing the system's logic and / or memory.

[0323] SoC 1704 may include one or more dynamically reconfigurable processors (DRPs) 1771. For example, DRP 1771 can be used to accelerate various computational operations. For instance, in one embodiment, DRP 1771 may be combined with a MAC unit to function as an AI accelerator. In another embodiment, DRP 1771 can execute an application while dynamically switching the circuit connection configuration of the arithmetic unit (e.g., ALU) on the chip each operating clock according to what needs to be processed. Because it uses only the necessary arithmetic circuitry, DRP 1771 can consume less power than a CPU and achieve higher speeds. Furthermore, compared to a CPU, which suffers from performance degradation due to frequent accesses to external memory for cache misses and other reasons, DRP 1771 can pre-build the necessary data paths in the hardware, thereby reducing performance degradation and operating speed variations (jitter) caused by memory accesses. DRP 1771 may include a dynamic loading function that switches circuit connection information each time the algorithm changes, enabling processing with limited hardware resources, even in robotics / automotive applications that require processing multiple algorithms.

[0324] In some embodiments, accelerator 1714 may include an OpenCV accelerator for accelerating OpenCV processing, OpenCV being an open-source industry-standard library for computer vision processing. In some embodiments, the combination of one or more DRP 1771s deployed as AI accelerators with an OpenCV accelerator can enhance AI computation and image processing algorithms to enable complex and computationally intensive operations such as visual simultaneous localization and mapping (SLAM).

[0325] Compared to conventional systems, the techniques described herein, by providing CPU complexes, GPU complexes, and hardware acceleration clusters, allow multiple neural networks to be executed simultaneously (e.g., at least partially in parallel) and / or sequentially, and the results combined to achieve Level 2–5 autonomous driving capabilities and / or autonomous robot motion, control, planning, and / or navigation operations. Furthermore, since the SoC 1704 can include various computing engines (e.g., processor 1710, CPU 1706, GPU 1708, accelerator 1714, etc.), tasks can be distributed among the computing engines, and in some cases, common-cause failures are avoided due to the discrete footprint of the computing engines. Additionally, since the SoC 1704 can include a dedicated safety processor 1713 (or safety island 1713), critical safety or redundant operations can be performed without common-cause failures of the SoC 1714's main processing components or computing engines. Due to these features, the underlying systems of the SoC 1704 and / or machine 1700 may be able to meet higher safety levels—such as the Automotive Safety Integrity Level (ASIL) D of the ISO 26262 standard.

[0326] Figure 17E According to some embodiments of this disclosure, cloud-based servers (e.g., servers such as those described herein in a data center) and Figure 17A The following is a system diagram illustrating communication between an example autonomous or semi-autonomous vehicle or machine 1700. System 1776 may include server 1778, network 1790, and machine 1700. Server 1778 may include multiple GPUs 1784(A)-1784(H) (collectively referred to herein as GPU 1784), switches 1782(A)-1782(H) (e.g., PCIe 4.0 / 5.0 switches, M.2 slots, Thunderbolt, USB4, NVIDIA's NVLink, NVIDIA's NVSwitch, GPUDirectRDMA, GPUDirect Storage, etc.), CPUs 1780(A)-1780(B) (collectively referred to herein as CPU 1780), accelerators, and / or other processor types. GPU 1784, CPU 1780, and PCIe switches may interconnect with high-speed interconnects, such as, but not limited to, NVIDIA-developed NVLink interface 1788 and / or PCIe connection 1786 and / or Ultra Accelerator Link (UALink). In some examples, the GPU 1784 is connected via NVLink and / or NVSwitch SoC, and the GPU 1784 and PCIe switch 1782 are connected via PCIe interconnect. While the figure shows eight GPUs 1784, two CPUs 1780, and two PCIe switches, this is not limiting. According to embodiments, each server 1778 may include any number of GPUs 1784, CPUs 1780, and / or PCIe switches. For example, each server 1778 may include eight, sixteen, thirty-two, and / or more GPUs 1784.

[0327] Server 1778 may receive sensor data indicating information about new or previously unexplored locations, and / or sensor data indicating changes to previously seen / stored locations (e.g., unexpected or changed road conditions, such as recently started road construction), via network 1790 and from machine 1700. Server 1778 may transmit neural network 1792, updated neural network 1792, map information 1794, etc., including information about traffic and road conditions, to machine 1700 via network 1790. Updates to map information 1794 may include updates to HD maps 1722, SD maps, navigation maps, etc., such as information about construction sites, potholes, detours, floods, and / or other obstacles. In some examples, neural network 1792, updated neural network 1792, map information 1794, and / or other information may come from new training and / or experience, represented in data received from any number of machines 1700 in the environment, and / or based on training performed in a data center (e.g., using server 1778 and / or other servers).

[0328] Server 1778 can be used to train machine learning models (e.g., neural networks) based on training data. Training data can be generated by machine 1700 and / or can be generated in simulations (e.g., using a game engine). In some examples, the training data is labeled (e.g., the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is unlabeled and / or unprocessed (e.g., the neural network does not require supervised learning). Training can be performed according to any one or more machine learning techniques, including but not limited to the following categories: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component analysis and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, it can be used by machine 1700 (e.g., transmitted to machine 1700 via network 1790), and / or the machine learning model can be used by server 1778 to remotely monitor and / or control machine 1700.

[0329] In some examples, server 1778 can receive data from machine 1700 and apply the data to a state-of-the-art real-time neural network for real-time intelligent inference. Server 1778 may include a deep learning supercomputer and / or a dedicated AI computer powered by GPU 1784, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, server 1778 may include a deep learning infrastructure in a data center using only CPU power.

[0330] The deep learning infrastructure of server 1778 may be capable of rapid, real-time inference and can use this capability to assess and verify the health of the processor, software, and / or related hardware in machine 1700. For example, the deep learning infrastructure may receive periodic updates from machine 1700, such as image sequences and / or objects located by machine 1700 in the image sequence (e.g., through computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them with the objects identified by machine 1700. If the results do not match and the infrastructure concludes that the AI ​​in machine 1700 has malfunctioned, server 1778 may send a signal to machine 1700 instructing the fail-safe computer of machine 1700 to take over control, notify the occupants, and perform safety maneuvers or operations, such as slowing down, returning control to the driver, stopping, and / or pulling over / closing the vehicle.

[0331] For inference, server 1778 may include GPU 1784 and one or more programmable inference accelerators (such as NVIDIA's TensorRT). The combination of GPU-driven servers and inference acceleration enables real-time response. In other examples, such as in scenarios where performance is less critical, servers driven by CPUs, FPGAs, and other processors can be used for inference.

[0332] Computational ecosystem for generating, training, and deploying AI Figure 18 This is a system diagram illustrating a three-computer ecosystem 1800 according to at least some embodiments of the present disclosure, including a first computing system 1802 for generating or creating artificial intelligence (AI) (e.g., AI training and validation data), a second computing system 1804 for training the AI, and a third computing system 1806 for deploying AI at the edge (which may include or correspond to...). Figures 17A to 17E (The SoC 1704). For example, to develop and deploy materializ...

Claims

1. A system comprising one or more processors, said one or more processors being used to: Determine the robot's state and identifier, the identifier indicating the robot type corresponding to the robot among a plurality of robot types; The robot processes the state and the identifier based at least on a generalist motion strategy to generate at least one action, wherein the generalist motion strategy is generated by a combination of a basic motion strategy corresponding to the plurality of robot types and one or more specialized motion strategies corresponding to different robot types among the plurality of robot types; and The robot moves according to the at least one action.

2. The system according to claim 1, wherein: The basic action policy is updated using imitation learning and a world model. This policy receives at least one state of the robot as input and outputs a basic action for moving the robot. The one or more specialist action policies are updated using residual reinforcement learning, and the one or more specialist action policies are used to receive at least one state of the robot as input and output specialist actions for moving the robot.

3. The system of claim 2, wherein the one or more specialist motion strategies are updated at least based on the basic motion strategy, the specialist motion being a combination of the basic motion and residual motions, the residual motions being used to adapt the basic motions to the robot type of the robot.

4. The system of claim 1, wherein, in order to generate the generalist action strategy, the combination of the basic action strategy and the one or more specialist action strategies is distilled.

5. The system of claim 4, wherein, in order to generate the generalist action strategy, the one or more processors are configured to: Using the one or more specialist action strategies, multiple specialist actions are generated for the robot by inputting multiple states into the one or more specialist action strategies; Using each of the one or more specialist action strategies, generate multiple normal distributions of the multiple specialist actions; Combining the multiple normal distributions; as well as The combination of the plurality of normal distributions is distilled into the generalist action strategy by at least minimizing the divergence of the combination of the plurality of normal distributions.

6. The system of claim 1, wherein the state includes at least one of the robot's environment, the speed of each joint of the robot, or the robot's objective.

7. The system of claim 1, wherein the identifier corresponds to an embedding in an embedding space, the embedding identifying the robot as the robot type among the plurality of robot types.

8. The system according to claim 1, wherein: The at least one action includes multiple speed commands, each speed command corresponding to a joint of the robot, and In order to perform the at least one action, each of the plurality of speed commands is mapped to a corresponding joint of the robot.

9. The system of claim 1, wherein the plurality of robot types include at least a humanoid robot, an autonomous mobile robot (AMR), a wheeled robot, a warehouse vehicle or machine, or a quadruped robot.

10. The system of claim 1, wherein the one or more processors are included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing one or more optical transmission simulations; A system for collaborative content creation of 3D assets; A system for performing one or more wireless cellular transmissions using a wireless cellular network; A system that provides one or more cloud gaming applications; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing one or more conversational AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models (MMLMs); A system for performing operations using one or more visual-language-action (VLA) models; A system for performing one or more conversational AI operations; A system for performing one or more synthetic data generation operations; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system that uses or deploys one or more inference microservices; A system that includes a service or microservice in which one or more machine learning models are deployed, along with an operating system-level virtualization package (e.g., a container); A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

11. A method comprising: One or more processors are used to determine the state and embedding corresponding to the robot, the state being determined using a world model, and the embedding indicating the type of the robot; Using the one or more processors and at least based on motion policies trained for deployment on multiple robot types, multiple commands for each joint of the robot are generated, wherein the state and the embedding are processed using the motion policies to generate the multiple commands; as well as Using one or more processors, the plurality of commands are transmitted to the respective joints of the robot to guide and move the robot.

12. The method of claim 11, wherein the motion strategy comprises a distillation combination of a plurality of robot type-specific motion strategies, wherein each of the plurality of robot type-specific motion strategies corresponds to one of the plurality of robot types; In order to generate the action strategy, the method further includes: The one or more processors use each of the plurality of robot type-specific action strategies to generate multiple normal distributions of multiple robot type-specific actions, wherein multiple states are input into each of the plurality of robot type-specific action strategies, and each of the plurality of robot type-specific action strategies outputs the plurality of robot type-specific actions; The plurality of normal distributions are combined by one or more processors; as well as The combination of the plurality of normal distributions is distilled into the action policy by one or more processors at least by minimizing the divergence of the combination of the plurality of normal distributions.

13. The method of claim 12, wherein the plurality of robot type-specific action policies are updated using reinforcement learning, and the one or more processors are further configured to: during the reinforcement learning, record the input and output of each of the plurality of robot type-specific action policies, the input and the output being used to generate the plurality of normal distributions.

14. The method of claim 12, wherein before generating the plurality of normal distributions, the weights of the plurality of robot type-specific action policies are updated to converge and frozen, the plurality of robot type-specific action policies being updated using a base model updated using imitation learning and the world model, wherein the weights of the base model are frozen before updating the plurality of robot type-specific action policies.

15. One or more processors, including a processing circuitry system, said processing circuitry system being used for: This enables the execution of one or more control operations associated with a robot based on at least one or more actions generated using a generalist action strategy, which generates the one or more actions when: (i) while using an embedding of a robot type corresponding to the robot among a plurality of robot types as a condition, and (ii) at least based on the generalist action strategy processing state information corresponding to the robot and the robot's environment.

16. The processor of claim 15, wherein the generalist action policy is trained using a teacher dataset generated using action outputs and latent states corresponding to specialist action policies associated with each of the plurality of robot types.

17. One or more processors according to claim 15, wherein the embedding includes a uniquely heated morphological code indicating the robot type.

18. One or more processors according to claim 15, wherein the state information is represented using at least a world model.

19. The processor of claim 18, wherein the one or more control operations correspond to one or more joints, actuators or motors of the robot.

20. The processor of claim 15, wherein the processor is included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing one or more optical transmission simulations; A system for collaborative content creation of 3D assets; A system for performing one or more wireless cellular transmissions using a wireless cellular network; A system that provides one or more cloud gaming applications; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing one or more conversational AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models (MMLMs); A system for performing operations using one or more visual-language-action (VLA) models; A system for performing one or more conversational AI operations; A system for performing one or more synthetic data generation operations; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system that uses or deploys one or more inference microservices; A system that includes a service or microservice in which one or more machine learning models are deployed, along with an operating system-level virtualization package (e.g., a container); A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

Images