Target object and feature mining

Abstract

This disclosure relates to target objects and feature mining. The performance of autonomous and / or semi-autonomous machines benefits from the many different scenarios that the learning machine (ego machine) may encounter, particularly scenarios related to other machines (agent machines) encountered in the environment. By tracking varying parameters of relative movement between the agent machine, the ego machine, and the agent machine, as well as varying parameters of ground and weather conditions and traffic conditions, more complex scenarios can be created than those achievable by existing methods, thereby improving the training of one or more machine learning systems on the ego machine. Depending on the aspect, traffic scenarios involving occlusion can be provided, which prevent the ego machine from recognizing and reacting to conditions in its path. These conditions may involve pedestrians, bicycles, or one or more agent machines.

Description

Background Technology

[0001] The performance of autonomous and semi-autonomous vehicles, robots, and / or other machines benefits from training one or more machine learning systems on the machine in many different scenarios (e.g., traffic, warehouses, etc.), particularly scenarios related to objects encountered during navigation or use. It would be desirable to design sufficient scenarios for the database and make queries to the database more readily available.

[0002] Therefore, there is a need for more efficient and expressive scene mining frameworks to train vehicles, robots, and / or other machines to improve the effectiveness of planning, control, navigation, and / or other tasks performed by these machines. Attached Figure Description

[0003] The system and method for target object and feature mining will be described in detail below with reference to the accompanying drawings, wherein:

[0004] Figure 1 A computing device configured to implement one or more aspects of the various embodiments is shown;

[0005] Figure 2 The illustrations show various embodiments including Figure 1 A scenario development system for data generation pipelines, training engines, and execution engines;

[0006] Figure 3A-3P Different scenarios according to various embodiments are described;

[0007] Figure 4A Example input and output objects associated with a set of filtering functions according to various embodiments are shown;

[0008] Figure 4B Aspects of the disclosed system according to various embodiments are illustrated;

[0009] Figure 4C Example mining scenarios according to various embodiments are shown;

[0010] Figure 5 A flowchart illustrating a method for mining scenarios to train one or more machine learning models to define additional scenarios, according to various embodiments;

[0011] Figure 6 According to various embodiments Figure 2 A more detailed diagram of the data generation pipeline;

[0012] Figure 7 The various embodiments are illustrated by Figure 2 A set of example scenarios for defining the data generation pipeline;

[0013] Figure 8 A flowchart is shown for a method for generating a scene associated with vehicles in an environment, according to various embodiments;

[0014] Figure 9A Examples of sensor locations having a corresponding field of view or sensing field, such as that of an autonomous or semi-autonomous machine, according to at least some embodiments of this disclosure;

[0015] Figure 9B This is an illustration of examples of the locations of components and sensors on an autonomous or semi-autonomous vehicle according to at least some embodiments of this disclosure;

[0016] Figure 9C This is a block diagram of an example system architecture for autonomous or semi-autonomous vehicles, robots, and / or other machine types according to at least some embodiments of this disclosure;

[0017] Figure 9D This is a block diagram of an example architecture of a computing system (e.g., a system-on-a-chip (SoC)) according to at least some embodiments of the present disclosure;

[0018] Figure 9E This is a system diagram of communication between a cloud-based server and example autonomous or semi-autonomous vehicles, robots and / or other machine types according to at least some embodiments of this disclosure;

[0019] Figure 10 This is a system diagram illustrating three computer ecosystems according to at least some embodiments of the present disclosure, including a computing system for generating or creating artificial intelligence (AI) (e.g., AI training and validation data), a computing system for training artificial intelligence, and a computing system for deploying AI at the edge;

[0020] Figure 11 This is a block diagram of an example computing system for generative artificial intelligence (AI) according to at least some embodiments of the present disclosure; and

[0021] Figure 12 This is a block diagram of an example computing device according to at least some embodiments of the present disclosure. Detailed Implementation

[0022] The disclosed systems and methods related to object tracking facilitate the creation of more complex scenarios than existing methods can achieve. Depending on the aspect, traffic scenarios involving multiple agents (which may be referred to herein as agent machines, agent robots, target vehicles, target robots, actors, objects, features, or obstacles) and occlusion that prevents self-operated machines (with multiple sensors) from recognizing and reacting to conditions inside and around the machine. When multiple agents are involved, the disclosed techniques enable the tracking and tracing of all of these agents.

[0023] Image retrieval systems based on Visual Language Models (VLMs) excel at capturing open-vocabulary semantics, combinatorial relationships between multiple agents, and environmental signals. In the field of autonomous driving, vehicle dynamics knowledge (including dynamic interactions between multiple agents) is essential for achieving safe and successful autonomous driving. The systems and methods described in this paper facilitate the capture of vehicle dynamics, the trajectories of dynamic objects, and the dynamic interactions between multiple agents. In this way, the described systems and methods bridge the gap between VLMs and autonomous driving scenarios.

[0024] For example, a VLM-based image retrieval system can capture pedestrians crossing the street while walking their dogs in the rain. The described system and method add information about the ego machine's orientation, velocity, acceleration, and / or distance to the intersection where the pedestrian is walking. As another example, a VLM-based image retrieval system can capture pedestrians walking on a sidewalk at night as the ego machine reverses into a street along a driving lane. Again, here, the described system and method add information about the ego machine's orientation, velocity, acceleration, and / or distance to the pedestrian. The described system and method enable VLM-based image retrieval systems to capture rare, complex, or unexpected driving scenarios that may not be well represented by basic driving scenarios. Capturing these driving scenarios facilitates the training of one or more machine learning models that will provide instructions to the ego machine.

[0025] Systems and methods related to target objects and feature mining are disclosed for developing a large number of scenarios (examples of which are provided in this paper) to train one or more machine learning systems of autonomous and / or semi-autonomous vehicles, robots, and / or other machine types. Aspects of the disclosed systems and methods enable the creation of many different scenarios from a basic scene. For example, by describing scene aspects such as the type and number of objects; the relative distance, velocity, and acceleration between the ego machine and the agent machine and / or objects; information about occlusion; and / or the spatiotemporal relationship between the ego machine and objects, parameters of each of these scene aspects can be varied to generate a large number of scenarios.

[0026] In some respects, the scenario may involve not only agent machines or robots as objects, but also occlusion, which may consist of one or more agent machines or robots hidden by one or more other agent machines or robots, or one or more pedestrians that may be hidden by one or more agent machines, robots or other objects.

[0027] In other respects, the disclosed systems and methods provide a unified data interface to enable data mining from diverse sources. These sources can include offline sensing and tracking systems; online object detection and tracking; and human 4DGT (ground truth) annotation systems. The unified data interface can query databases to download data (e.g., proxy trajectories and attributes) and label specific data with segmentation tags that have one or more specific characteristics (e.g., attributes related to road and weather conditions). Once query results are created, they can be cached for faster retrieval than repeatedly querying the database itself.

[0028] The disclosed systems and methods bridge the gap between basic visual language models and the fields of autonomous vehicles or robotics, enabling wider application of visual language models in these fields. Target object and feature mining can specify parameters such as ego velocity and distance between the ego and the target agent. These parameters are typically inaccurate for the direct output of large visual language models.

[0029] As described in this article, a scene can be viewed as a collection of object spans, where an object span is the start and end time of a given object. For example, in a scene involving multiple objects (such as trucks) parked back-to-back, the scene could include object spans to represent each truck. A scene can also include video from a camera on a self-contained machine.

[0030] In some embodiments, focusing on fragments of an object rather than the entire object may be more efficient, since for a given scene, only a portion of the object's trajectory may be of interest. Furthermore, a single object may appear multiple times within a given scene category, making it desirable to create multiple scenes for that object.

[0031] Although this disclosure may relate to examples of autonomous or semi-autonomous vehicles, robots, and / or other machine types 900 (which may alternatively be referred to herein as "vehicle 900," "self-vehicle 900," "machine 900," "self-machine 900," "robot 900," and / or "self-robot 900"), examples of which relate to Figures 9A to 9EThe description is provided, but it is not intended to be a limitation. For example, the systems and methods described herein can be used, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robot platforms (e.g., autonomous mobile robots (AMRs), humanoid robots, robotic arms and / or end effectors, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, spacecraft, ships, shuttles (e.g., robotaxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones and / or other vehicle, robot or machine types. Furthermore, while this disclosure may describe target objects and feature mining, this is not intended to be limiting, and the systems and methods described herein can be used in augmented reality (AR), virtual reality (VR), mixed reality (MR), robotics, security and supervision (e.g., smart cities), autonomous or semi-autonomous machine applications, industrial manufacturing, simulation, and / or any other technological space where target objects and feature mining can be used. In some embodiments, the systems, methods, and / or processes described herein can be used with... Figures 9A to 9E Example machine 900, Figure 10 Example calculation of ecosystem 1000 Figure 11 Example generative language model system 1100 and / or Figure 12 The example computing device 1200 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the same task.

[0032] Furthermore, while this disclosure describes example roads and powered vehicles traveling on these roads, the environments that this disclosure can affect are not limited to these. For example, the systems and methods described herein can be used in environments other than roads (including, but not limited to, aviation or waterways). The road environment referred to herein is not limited to the road environment shown in the figures. Any type of road or street (including, but not limited to, one-way residential streets, residential streets with one or more opposite lanes, one-way urban streets, city streets with one or more opposite lanes, highways, expressways, motor vehicle lanes, and / or other surfaces on which powered vehicles can travel).

[0033] As discussed in this paper, self-navigating machines tend to use complex integrations of multiple navigation modules related to perception, planning, control, prediction, safety, and / or other tasks. These navigation modules have several drawbacks, including (but not limited to) inter-module error propagation leading to degraded navigation performance, a lack of holistic understanding interfering with the ability to make contextually aware decisions, significant redesign and / or adaptation of individual modules to adapt the navigation system to new tasks and / or environments, and / or redundancy and / or sequential processing between modules negatively impacting the use of the navigation system in real-time and / or time-sensitive applications.

[0034] The number of scenarios a self-generating machine might encounter during its journey is virtually infinite. Training one or more machine learning systems involves getting as close as possible to this infinite number of scenarios. Inputting a wide range of scenarios with extensive variations into one or more machine learning systems helps them recognize newly encountered scenarios as having one or more features of the training scenarios.

[0035] To address the aforementioned limitations, according to one aspect, the disclosed systems and methods train and execute one or more machine learning systems by generating a large number of scenarios in which the ego machine can operate relative to one or more agent machines or machines, and / or one or more objects, and / or one or more occlusions (which prevent the ego machine from recognizing objects in front of it). These scenarios depend on inputs such as (but not limited to) camera images, other sensor data types (e.g., RADAR, LiDAR, ultrasound, etc.), velocity, and / or global guidance for the output, such as (but not limited to) semantic segmentation, path and / or navigation commands.

[0036] The disclosed systems and methods include a data generation pipeline that defines scenarios for training one or more machine learning models, other types of machine learning models that can be used by a self-machine and / or other machine types to perform tasks, the hardware configuration of the machine, and / or other components of the machine. The data generation pipeline may include a simulator that performs various types of simulations related to the navigation of a machine (e.g., a self-machine) within an environment (e.g., a road). Data generated by the simulator based on the simulation includes (but is not limited to) rendered images of the environment surrounding the machine (e.g., from the perspective of one or more cameras and / or a bird's-eye view visualization on the machine), semantic labels associated with the images (e.g., segmentation maps, detected objects, bounding shapes, etc.), the state of the machine (e.g., position, heading, speed, etc.), and / or maps of free and / or occupied space within the environment. The data generation pipeline may include a scene generator that determines targets within the map, such as (but not limited to) target locations navigated to within the environment.

[0037] The data generation pipeline may include a planner that generates commands to instruct the machine to take objective-related actions, such as linear and / or angular velocities that move the machine toward the objective. These commands are sent to a simulator, which updates the machine's state, rendered images, semantic labels, maps, and / or other data based on the actions. The simulator may send some or all of the updated data to the planner, allowing the planner to generate new commands based on the updated data and the scene from the scene generator. This process may be repeated until the machine completes the scene, has executed a certain number of time steps (object span), and / or meets another condition indicating the end of the simulation.

[0038] The data generation pipeline may include a recorder that records and synchronizes data output by other components at time steps. For example, the recorder may record data from other components in the order in which corresponding events occur in the simulation. The recorder may also downsample some or all of the data (e.g., on a spatial and / or temporal basis) to reduce the size of the recorded data.

[0039] Post-processors in the data generation pipeline can adapt the generated data to various machine learning models and / or use cases. For example, post-processors can resample, compress, format, and / or otherwise transform the generated data into a form that can be used to train and / or evaluate machine learning models, hardware configurations, and / or other components of the machine.

[0040] A data generation pipeline can be configured and / or customized via one or more sets of configuration parameters. For example, configuration parameters may include a specific name and / or identifier for a given scenario (e.g., a combination of specific environments, machines, scenarios, etc.) in which data will be generated and collected. Configuration parameters can also be used to customize the environment and / or scenario to be generated, the type of planner, the type of data to be recorded, the frequency of data recording, and / or the way the recorded data is converted into a format suitable for training and / or evaluating machine learning models and / or another component of the machine. Different sets of configuration parameters can be used to launch different instances of the data generation pipeline (e.g., in parallel on multiple nodes of a distributed system) to generate data capturing different scenarios related to navigation and / or other types of tasks performed by machines in the environment.

[0041] Figure 1This is a block diagram illustrating a computing system 100 configured to implement one or more aspects of at least one embodiment. In at least one embodiment, the computing system 100 may include any type of computing device, including but not limited to server machines, server platforms, desktop computers, laptops, handheld / mobile devices, digital kiosks, in-vehicle infotainment systems, smart speakers or displays, televisions, and / or wearable devices. In at least one embodiment, the computing system 100 is a server machine (or a combination of server machines) operating in a data center or cloud computing environment that provides scalable computing resources as a service over a network.

[0042] It should be noted that the computing device described herein is illustrative, and any other technically feasible configuration falls within the scope of this disclosure. For example, multiple instances of the data generation engine 122, training engine 124, and execution engine 126 may execute on a set of nodes in a distributed and / or cloud computing system to implement the functionality of the computing device 100. In another example, the data generation engine 122 and training engine 124 may execute on various hardware sets, device types, or environments to adapt the data generation engine 122 or training engine 124 to different use cases or applications. In a third example, the data generation engine 122 and training engine 124 may execute on different computing devices and / or different sets of computing devices.

[0043] In one embodiment, computing device 100 may include, but is not limited to, interconnect (bus) 112 connecting one or more processors 102, input / output (I / O) device interfaces 104 coupled to one or more input / output (I / O) devices 108, memory 116, storage 114, and network interface 106. Processor 102 may be any suitable processor implemented as a central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), artificial intelligence (AI) accelerator, any other type of processing unit, or combination of different processing units, such as a CPU configured to cooperate with a GPU. Typically, processor 102 may be any technically feasible hardware unit capable of processing data and / or executing software applications. Furthermore, in the context of this disclosure, the computing elements shown in computing device 100 may correspond to a physical computing system (e.g., a system in a data center) or may be a virtual computing instance executed in a computing cloud.

[0044] I / O device 108 includes devices capable of providing input, such as a keyboard, mouse, touchscreen, etc., and devices capable of providing output, such as a display device. Furthermore, I / O device 108 may include devices capable of both receiving input and providing output, such as a touchscreen, a Universal Serial Bus (USB) port, etc. I / O device 108 can be configured to receive various types of input from end users (e.g., designers) of computing device 100 and also to provide various types of output to end users of computing device 100, such as displayed digital images, digital video, or text. In some embodiments, one or more I / O devices 108 may be configured to couple computing device 100 to network 110.

[0045] Network 110 can be any technically feasible type of communication network that allows data exchange between computing device 100 and external entities or devices (such as a web server or another networked computing device). For example, network 110 may include a wide area network (WAN), a local area network (LAN), a wireless (WiFi) network, and / or the Internet.

[0046] Storage 114 may include non-volatile storage for applications and data, and may include fixed or removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other magnetic, optical, or solid-state storage devices. Data generation engine 122, training engine 124, and execution engine 126 may be stored in storage 114 and loaded into memory 116 when executed.

[0047] Memory 116 may include random access memory (RAM) modules, flash memory cells, or any other type of memory cell or a combination thereof. One or more processors 102, I / O device interfaces 104, and network interfaces 106 may be configured to read data from and write data to memory 116. Memory 116 may include various software programs executable by one or more processors 102 and application data associated with said software programs, including a data generation engine 122, a training engine 124, and an execution engine 126.

[0048] In some embodiments, training engine 124 trains one or more machine learning systems to define additional scenarios. In some embodiments, training engine 124 and execution engine 126 include functions for training and executing one or more machine learning systems to enable the vehicle to recognize a large number of scenarios and respond appropriately to them, thereby navigating roads safely and successfully. One or more machine learning systems directly or otherwise map inputs (e.g., (but not limited to) camera images, speed, global guidance, and / or vehicle state) to multimodal outputs (e.g., (but not limited to) semantic segmentation, paths, and / or navigation commands). These multimodal outputs can then be used to perform vehicle navigation, generate predictions of future states associated with the vehicle, simulate vehicle operation, and / or perform other vehicle-related tasks. Training engine 124 and execution engine 126 are described in further detail herein.

[0049] In some embodiments, the data generation pipeline 122 defines scenarios for training, evaluating, and / or testing vehicles, robots, or other machines, other types of machine learning models that can be used by various machine types to perform tasks, vehicle hardware configurations, and / or other components of the vehicle. As discussed herein, the data generation pipeline 122 can be configured and / or customized via various parameters to generate data capturing different scenarios related to navigation and / or other types of tasks performed by machines in the environment. This document will discuss... Figure 2 and Figures 6-8 The data generation pipeline 122 is described in further detail.

[0050] It should be understood that Figure 1The arrangements described herein, as well as other arrangements, are merely illustrative. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to or in place of the arrangements shown, and certain elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components or together with other components, and can be implemented in any suitable combination and location (e.g., on a machine at a local device, vehicle, or edge; in the field (e.g., a locally hosted server); remotely (e.g., in one or more computing or server devices in one or more data centers in the cloud; and / or elsewhere). The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, one or more processors (e.g., central processing units (CPUs)) that execute instructions stored in memory can be used. Various functionalities can be performed using graphics processing units (GPUs), microprocessors, microcontrollers, embedded processors, digital signal processors (DSPs), image signal processors (ISPs), physical processing units (PPUs), field-programmable gate arrays (FPGAs), accelerators (e.g., deep learning accelerators (DLAs), deep learning accelerator clusters (XNNs), neural network accelerators (NNAs) and / or neural processing units (NPUs), programmable vision accelerators (PVAs), optical flow accelerators (OFAs), etc.), application-specific integrated circuits (ASICs), data processing units (DPUs), quantum processors, etc. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 9A-9E Example machine 900, Figure 10 Example calculation of ecosystem 1000 Figure 11 Example generative language model system 1100 and / or Figure 12 The example computing device 1200 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the same task.

[0051] Figure 2 A system for performing end-to-end navigation according to various embodiments is shown, the system comprising: Figure 1 The data generation pipeline 122, training engine 124, and execution engine 126 are described herein. As discussed herein, the data generation pipeline 122 defines scenarios that can be used to train, evaluate, test, and / or otherwise operate the multimodal navigation system 242, other types of machine learning models that can be used by various machine types to perform tasks, machine hardware configurations, and / or other components of the machine.

[0052] The data generation pipeline 122 includes a simulator 216 that generates multiple sets of simulation data 232(1)-232(X) (each of which is individually referred to herein as simulation data 232). For example, the simulator 216 can perform physical simulations of various environments surrounding various types of machines. During these physical simulations, the simulator 216 can generate simulation data 232, which includes (but is not limited to) rendered images of the environment surrounding the machine (e.g., from the perspective of one or more cameras and / or bird's-eye view visualizations on the machine), semantic labels associated with the images (e.g., segmentation maps, detected objects, bounding shapes, etc.), the state of the machine (e.g., position, heading, speed, etc.) and / or maps of free and / or occupied space within the environment.

[0053] The data generation pipeline 122 includes a scene definer 218 that determines a set of scenes 234(1)-234(Y) associated with the simulation data 232 (each scene is individually referred to herein as scene 234). For example, the scene definer 218 can define a route for a vehicle to navigate in traffic in the presence of one or more objects or occlusions in the environment within a given map output by the simulator 216. The environment is not limited to specific weather or daylight conditions, but can include a variety of conditions that may affect the vehicle's navigation (including, but not limited to, dawn, daytime, dusk, night, sun, clouds, temperature, wind, fog, rain, freezing rain, snow, sleet and / or hail).

[0054] The data generation pipeline 122 includes a planner 220 that generates various commands 236(1)-236(Z) that cause the machine to take one or more corresponding actions based on simulation data 232 and / or scene 234. For example, the planner 220 may generate commands 236 that include (but are not limited to) linear and / or angular velocities that cause the vehicle to move along a scene 234 from scene definer 218 while avoiding objects in the environment represented by simulation data 232 from simulator 216 or from the scene 234 itself.

[0055] Each set of commands 236 output by planner 220 can be sent to simulator 216, which updates the machine's state, rendered images, semantic labels, maps, and / or other simulation data 232 based on the actions. Simulator 216 can then send partially or fully updated simulation data 232 to planner 220, allowing planner 220 to generate a new set of commands 236 based on the updated simulation data 232 and the corresponding scenario 234 from scenario definer 218. This process can be repeated until scenario 234 is reached, a certain number of time steps or iterations have been performed in a given simulation, and / or another condition indicating the end of the simulation is met.

[0056] The data generation pipeline 122 also includes a data logger 222 that aggregates simulation data 232, scenarios 234, commands 236, and / or other data generated by simulator 216, scenario definer 218, and planner 220 into multiple records 238(1)-238(N) (each record is individually referred to herein as record 238). For example, the data logger 222 may log data from simulator 216, scenario definer 218, and / or planner 220 in record 238 in the order in which the corresponding events occur in the corresponding simulation (e.g., in time steps, “frames” of simulation data 232, and / or other discrete time representations). The data logger 222 may also, or alternatively, downsample and / or resample some or all of the data in record 238 (e.g., on a spatial and / or temporal basis) to reduce and / or modify the size of the recorded data.

[0057] The post-processor 224 in the data generation pipeline 122 adapts the simulated data 232, scenarios 234, commands 236, records 238, and / or other data generated by other components of the data generation pipeline 122 to various machine learning models and / or use cases. For example, the post-processor 224 may resample, compress, format, and / or otherwise transform the data in a given set of records 238 into a form that can be used to train and / or evaluate machine learning models, hardware configurations, and / or other components of one or more machines. Each set of records 238 that is post-processed for a given purpose and / or in a certain way may be stored in one or more datasets 240(1)-240(K) (each dataset is individually referred to herein as dataset 240) for subsequent retrieval and use.

[0058] In some embodiments, the data generation pipeline 122 is configured and / or customized via different types of configuration parameters. For example, configuration parameters may include a specific name and / or identifier for a given scenario (e.g., a combination of specific environments, machines, scenarios, etc.) in which data will be generated and collected. Configuration parameters can also be used to customize the environment and / or scenario, the type of planner, the type of data to be recorded, the frequency of data recording, and / or the way the recorded data is converted into a format suitable for training and / or evaluating machine learning models and / or another component of the machine. Different sets of configuration parameters can be used to launch different instances of the data generation pipeline (e.g., in parallel on multiple nodes of a distributed system) to generate data capturing different scenarios related to navigation and / or other types of tasks performed by machines in the environment. Figures 6 to 8 A more detailed description is provided for data generation pipeline 122.

[0059] Training engine 124 uses training data 200 to train one or more machine learning models 208, said training data 200 being derived from one or more datasets 240 generated by data generation pipeline 122, data collected by machines in real-world environments, simulations using, for example, simulator 216, and / or other data sources. In some embodiments, training engine 124 trains one or more machine learning models 208 to define additional scenarios. In some embodiments, dataset 240 may include a large number of scenarios described herein. Figure 2 As shown, training data 200 includes training state data 204 representing the state associated with the machine and / or the environment. For example, training state data 204 may include sensor data captured by virtual and / or real sensors on the machine (e.g., images, LiDAR, RADAR, audio data, ultrasonic data, inertial measurement unit (IMU) data, etc.), representations of the environment around the machine (e.g., visualizations, semantic segmentation, point clouds, meshes, environment types, environment descriptions, scene descriptions using generic scene descriptor (USD) data (e.g., OpenUSD), linear and / or angular velocities of the machine, machine type and / or machine model associated with the machine, global guidance associated with navigation and / or other tasks or scenarios 234 of the machine, and / or other information that can be used to characterize the state of the machine and / or the environment around the machine.

[0060] Training data 200 also includes training motion data 202 representing actions the machine will perform in the environment. For example, training motion data 202 may include "truth" actions, action policies, commands, routes, trajectories, linear velocities, angular velocities, paths, and / or other indications of actions to be performed during perception, planning, control, prediction, navigation, manipulation, and / or other tasks using the machine.

[0061] In some embodiments, the machine learning model 208 is used to perform and / or guide tasks using a machine. While examples of using machine learning models (e.g., neural networks) are described herein, this is not limiting. For example, but not limited to, the machine learning model 208 may include any of the various machine learning models and / or neural networks described herein, such as those using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoder neural networks, artificial neural networks (ANN), convolutional neural networks (CNN), recurrent neural networks (RNN), perceptrons, long short-term memory (LSTM) networks, multilayer perceptron (MLP) networks, deep stacked networks (DSN), generative pre-trained (GPT) models or networks, feedforward networks, radial basis function ANNs, self-organizing maps (SOM), Kohonen maps, Hopfield networks, Boltzmann machines, deep belief neural networks, deconvolutional neural networks, generative adversarial networks (GANs), liquid machines, modular neural networks, sequence-to-sequence... Models, networks using transducer architectures, state-space models (SSM) (e.g., networks using Mamba architectures (e.g., Mamba-1, Mamba2, etc.), networks using selective state-space models, networks using structured state-space sequence models, etc.), diffusion models (e.g., diffusion probability models, fraction-based generative models, etc.), neural radiation field (NeRF) models, Gaussian sputtering models, Kolmogorov-Arnold networks (KAN), models with encoder-only architectures, models with decoder-only architectures, models with encoder-decoder architectures, generative machine learning models, language models, large language models (LLM), visual language models (VLM), multimodal language models (MMLM), large action models (LAM), visual-language-action (VLA) models, etc.) and / or other types of machine learning models.

[0062] Machine learning model 208 may also or alternatively include rules, filters, heuristics, logic programming, semantic networks, search techniques, named entity recognition techniques and / or other symbolic models. Each machine learning model can be used to generate embeddings, semantic segmentation, reconstruction and / or prediction, classification outputs, security alerts, trajectories, commands, and / or other outputs related to one or more corresponding tasks from sensor data.

[0063] During the training of machine learning model 208, training engine 124 feeds some or all of the training state data 204 into machine learning model 208. Training engine 124 processes the input training state data 204 using model parameters 206 (e.g., neural network weights) of machine learning model 208 and obtains training output 210 from one or more layers, blocks, or components of machine learning model 208, including predictions associated with the training state data 204. Training engine 124 uses training output 210, training state data 204, and / or training action data 202 to compute one or more losses 254. Training engine 124 then iteratively updates the model parameters 206 of machine learning model 208 using training techniques (e.g., gradient descent and backpropagation) in a manner that reduces the losses 254.

[0064] In one or more embodiments, execution engine 126 uses partially or fully trained machine learning models 208 to implement navigation system 242, which can be used to perform navigation and / or other tasks for machine 260 in real-world, simulated, digital twin, and / or other types of environments. Machine 260 may include autonomous or semi-autonomous vehicles, drones, submarines, ships, and / or other types of vehicles with navigation capabilities. Navigation system 242 may be deployed on machine 260 for real-time inference using a runtime platform (such as NVIDIA's TensorRT) that uses quantization, layer and tension fusion, kernel tuning, GPU-based execution, streaming audio and / or video, and / or concurrent execution to accelerate and optimize performance.

[0065] The navigation system 242 operates on inputs such as (but not limited to) sensor data 262 from machine 260 (e.g., camera images, LiDAR data, RADAR data, audio data, speed, status, etc.), task-related route data 264 (e.g., path, trajectory, route, destination, etc.), speed data 266 (e.g., relative speed, acceleration, and / or deceleration of the ego machine or one or more agent machines), and / or other representations of machine 260 and / or the environment surrounding machine 260.

[0066] In some embodiments, different scenes can be created from different combinations of filtering functions across an object span, each filtering function being a building block for creating a scene to be mined. Multiple filtering functions can be joined in various ways to produce one or more output objects from the input objects, thereby generating additional scenes. An exemplary (but not exhaustive) list of filtering functions may include:

[0067] • Filter object spans to determine if a span meets one or more criteria, then delete or retain the entire span. An exemplary, but not exhaustive, list of filters may include:

[0068] o Object heading

[0069] o-dimensionality (span size)

[0070] average object speed

[0071] average self-speed

[0072] o Object classes (including, for example, one or a group of pedestrians or cyclists who may be pushing or pulling another object (for pedestrians, pushing a stroller, and for cyclists, pulling a trailer); or one or a group of powered vehicles, including, for example, cars, buses, trains, trams, small or large trucks (with or without trailers), motorcycles or other powered wheeled vehicles)

[0073] o duration

[0074] o Objects (e.g., combinations of one or more filtering criteria listed in this article)

[0075] o Self-direction of travel, for example, a curved trajectory or a straight line, an object, or the distance traveled, which may include deviations from the trajectory.

[0076] o Distance traveled by self or proxy machine

[0077] Satisfaction level of the relationship (if satisfied, keep the span; otherwise, discard the span).

[0078] o Occlusion (where the ego machine may be blocked or occluded by other objects, such as one or more parked proxy machines, or multiple proxy machines moving in front of the ego machine)

[0079] o-copy (duplicates)

[0080] o Overtaking lines (one of which can define the separation between the ego machine and the agent machine, and in some cases may include lane dividers)

[0081] The overtaking polygon (where the polygon, whether convex or non-convex, defines a portion of the environment in which the vehicle is traveling); the overtaking polygon means that the vehicle is traveling through multiple polygons.

[0082] o Self-lateral speed

[0083] • Trim the object span based on specified criteria to retain portions of the span. An exemplary, but not exhaustive, list of trimming criteria may include one or more of the filtering criteria listed herein, as well as:

[0084] o Self-vertical speed

[0085] o Self-trajectory curvature (whether the self-machine moves left or right)

[0086] o overlaps with another span

[0087] o is obstructed

[0088] o one or more time intervals

[0089] o-span, for example, the time overlap between the source object span and the target object span.

[0090] o The visible span of the front-facing camera of the self-driving machine

[0091] • Extend the object span by time (e.g., if the span duration is too short) or by the distance the object travels (in the forward or backward direction).

[0092] • Aggregate or cluster multiple object spans. For example, when multiple object spans overlap in time, they can be aggregated into a single span or clustered into a list of spans. As another example, for multiple lists of spans, aggregate the spans, one for each list, that geometrically overlap each other.

[0093] In some embodiments, a scenario can be compiled using some or all of the following list of information (this list is exemplary but by no means exhaustive):

[0094] • List of objects;

[0095] • Tracking information for each object, including dimensions, location, orientation, and position relative to its own machine;

[0096] • Scene start time and end time;

[0097] • Scene start scenario and scene end scenario, each of which can be derived from an index corresponding to the frame number of the sensor (e.g., LiDAR or RADAR) on the self-machine;

[0098] • Ticket content used to enable scenes to be manually annotated;

[0099] • Links to different cameras with different fields of view (e.g., left, right, front, and rear) on different parts of the self-machine.

[0100] In some embodiments, the object-centric mining library can be compiled using one or more of the following based on different scenario descriptions:

[0101] Object class

[0102] Distance between the self-machine and one or more objects

[0103] • The speed of self-machines with one or more objects

[0104] • The relative velocity of the self-machine and one or more objects

[0105] • Acceleration of a self-machine and one or more objects

[0106] • Relative acceleration between the self-machine and one or more objects

[0107] • Obstructing information

[0108] • Trajectory curvature

[0109] Spatial relationships between the self-machine and one or more objects

[0110] In the following Figures 3A to 3P In the discussion of the different scenarios shown, those skilled in the art will understand that these scenarios are exemplary and that many variations of these scenarios can be explored. For ease of understanding, different units of measurement may be used in different sets of scenarios. The described embodiments are not limited to specific units of measurement. Sometimes the metric system may be used; sometimes the imperial system may be used.

[0111] Furthermore, to expand the scene set, lighting, weather, and road conditions can be altered, allowing the ego machine and one or more agent machines to operate in sunny, cloudy, rainy, snowy, windy, or foggy weather. The ego machine and one or more agent machines can operate at night, during the day, at dawn, or at dusk. Roads can be dry or slippery (e.g., due to rain, snow, or ice). Darkness (including lack of road lighting), slippery roads, strong winds, and / or foggy conditions can cause the ego machine and one or more agent machines to operate slower than under normal conditions.

[0112] Agent machines can be of different types. A non-limiting list of agent machine types can include passenger vehicles, such as cars, shuttle buses, pickup trucks, vans, minivans, limousines, SUVs, buses, trams, ambulances, shuttle buses, electric or motorized bicycles, motorcycles, fire engines, police cars, ambulances, boats, engineering vehicles, underwater vehicles, robotic vehicles, drones, aircraft, semi-trailers, or semi-trailers coupled to one or more trailers. Changing the agent machine type can expand the range of scenarios. Furthermore, autonomous machines can be of different types, as described herein with respect to agent machines. A non-limiting list of autonomous machine types can include passenger vehicles, such as cars, shuttle buses, pickup trucks, vans, minivans, limousines, SUVs, buses, trams, ambulances, shuttle buses, electric or motorized bicycles, motorcycles, fire engines, police cars, ambulances, boats, engineering vehicles, underwater vehicles, robotic vehicles, drones, aircraft, semi-trailer trucks, or semi-trailer trucks coupled to one or more trailers.

[0113] Those skilled in the art will understand that the different scenarios described are exemplary, not exhaustive. Variations can be derived from the scenario descriptions in this article.

[0114] constant linear velocity

[0115] In this set of scenarios, such as Figure 3A The autonomous machine and the agent machine are depicted to be in the same lane or in adjacent lanes. In one example scenario, the autonomous machine may be traveling at a specific speed in a specific direction, maintaining a specific distance from the agent machine. Changing the distance between the vehicles and / or the speed of the vehicles will increase the number of such scenarios.

[0116] Lane change / cut-in

[0117] In this set of scenarios, such as Figure 3B The depicted scenario shows an agent machine changing lanes in front of an autonomous machine to enter the same lane. The distance in front of the autonomous machine (where the agent machine cuts into the autonomous machine's lane) can be varied from relatively mild maneuvers at a greater distance to relatively aggressive maneuvers at a closer distance. Aggressive maneuvers can simulate the agent machine's movement in traffic congestion. Figure 3B In this diagram, the length of the diagonal line between the agent machine and the ego machine indicates the aggressiveness of the agent machine's approach. A shorter line indicates a more aggressive approach, while a longer line indicates a less aggressive (more gradual) approach.

[0118] The number of this type of scenario can be increased by changing the relative speed between the agent machine and the ego machine, and / or the distance between the ego machine and the agent machine at the time of the cut, and / or the relative trajectory between the agent machine and the ego machine (e.g., a straight road, a gently curved road, a steeply curved road).

[0119] In some embodiments, one agent machine can cut in front of another agent machine, where the relative speed, relative distance, and relative trajectory of the two agent machines can be mined, similar to an autonomous machine / agent machine cut-in scenario, to further expand the number of this type of scenario. Acceleration during overtaking by lighter agent machines

[0120] In this set of scenarios, such as Figure 3C The scenario depicts a proxy machine (e.g., a motorcycle) overtaking a heavier autonomous machine (e.g., a truck or van) at varying relative speeds, with the proxy machine accelerating at different rates to overtake the autonomous machine from an adjacent lane. The autonomous machine can be stationary or in motion. The proxy machine can begin stationary and can be at different distances from the autonomous machine at the start of the maneuver. As discussed herein, altering the absolute and relative speeds of the autonomous and proxy machines, and / or their relative trajectories, can increase the number of such scenarios.

[0121] Constant speed on curved roads

[0122] In this set of scenarios, such as Figure 3D As depicted, when an ego machine trails an agent machine, the curvature of the ego machine's trajectory can vary, as can its velocity as it traverses the curve. The number of such scenarios can be increased by altering the tightness of the curve, the velocities of entering, traversing, and leaving the curve, the different initial and relative velocities between the ego and agent machines, and the initial and / or relative distances between them.

[0123] Oncoming vulnerable road users (VRUs)

[0124] In this set of scenarios, such as Figure 3E The described agent machine can travel in the opposite direction to the ego machine and can be considered weaker relative to the ego machine. For example, the ego machine could be a car or truck, while the agent machine could be a motorcycle or bicycle. Changing the speed of the ego machine and agent machine traveling in opposite directions and / or changing the relative positions of the ego machine and agent machine in their respective lanes can increase the number of scenarios of this type.

[0125] Hard deceleration

[0126] In this set of scenarios, such as Figure 3FThe scenario depicts a proxy machine traveling in the same lane in front of the ego machine at different speeds and / or at varying distances from each other. The proxy machine may decelerate at different rates, representing different braking maneuvers, from slow to normal to rapid deceleration. The objective is to avoid collisions between the ego machine and the proxy machine. In some embodiments, it may be assumed that there is sufficient distance between the ego machine and the proxy machine to ensure the safety of the ego machine. Changing the speeds of the ego machine and the proxy machine, their relative distance, and / or the degree of braking of the proxy machine can increase the number of scenarios of this type.

[0127] Stagnation at 90 degrees

[0128] In this set of scenarios, such as Figure 3G The depicted scenario involves the self-machine and the agent machine stopping at the start, in different lanes, and at different relative distances. As the self-machine and the agent machine move forward, the agent machine can accelerate or decelerate to run alongside the self-machine, allowing it to be observed at a 90-degree angle relative to the self-machine. Changing the absolute and / or relative speeds of the self-machine and the agent machine, and / or the absolute and / or relative positions of the vehicles in their respective lanes, can increase the frequency of this type of scenario.

[0129] Behind a stationary target

[0130] In this set of scenarios, such as Figure 3H As depicted, varying gaps may exist between the ego machine and the stationary agent machine. In some embodiments, the agent machine may stop to allow pedestrians or bicycles to cross. The agent machine may be a motorcycle, car, truck, and / or van, but is not limited to these. Automatic emergency braking (AEB) may be invoked in some scenarios. Changing the speed of the ego machine and its distance from the agent machine at the start of the maneuver can increase the number of such scenarios. Acceleration during the period when the heavier proxy machine is surpassed

[0131] In this set of scenarios, such as Figure 3I As depicted, the surrogate machine can overtake the autonomous machine at varying relative speeds. The autonomous machine can be stationary or moving. The surrogate machine can initiate overtaking at different distances behind the autonomous machine. When initiating overtaking, the surrogate machine may or may not be in the same lane as the autonomous machine. The surrogate machine can accelerate relative to the autonomous machine at different rates. Changing the distance, lane, acceleration, and / or relative speed between the autonomous and surrogate machines before and during maneuvering increases the number of scenarios of this type.

[0132] Obscured scene

[0133] In this set of scenarios, such as Figure 3JThe scenario depicts an autonomous machine following one or more agent machines (which are objects) in a given lane (self-lane). One of the agent machines can occlude another agent machine, causing the autonomous machine to see only one of them. The number of such scenarios can be increased by changing the distance between the autonomous machine and one of the agent machines, the distance between the autonomous machine and one or more agent machines, and / or the speed of the autonomous machine relative to one or more agent machines. Active Lane Change Assist (ALCA) scenario

[0134] In this set of scenarios, such as Figure 3K As depicted, the surrogate machine may be positioned behind the ego machine and may pass from one side or the other. The surrogate machine may initiate its passing maneuver at different distances behind the ego machine, with different relative speeds, and may cut in to the left or right at different distances behind the ego machine. Changing these different distances, different relative speeds, and / or passing directions (left or right) can increase the number of scenarios of this type. In some embodiments, ALCA may include adaptive cruise control (ACC) and / or active steering assist (ASA).

[0135] Self-turning case

[0136] In this set of scenarios, such as Figure 3L The depicted ego machine may be turning left or right. The curvature of the ego machine's trajectory can be used to determine changes in the ego machine's orientation. In some embodiments, the ego machine may perform its turn if there is one or more agent machines nearby. Changing the curvature, direction, and / or length of the curve can increase the number of such scenarios.

[0137] Stop-and-go traffic

[0138] In this set of scenarios, such as Figure 3MThe scenario depicts a self-machine surrounded by several agent machines, which can be cars, trucks, buses, vans, motorcycles, or other road vehicles. This scenario can be viewed as a traffic jam or congestion on a road or highway. Depending on the scenario, the number of agent machines can range from three to nine or more. The movement of the self-machine relative to the agent machines can vary. For example, the self-machine can move at the same speed as the surrounding agent machines; it can accelerate to overtake an agent machine; it can decelerate to increase the distance between itself and the agent machines; or it can change lanes to cut in front of or cut away from an agent machine. One or more agent machines (which can also be called obstacle vehicles because they surround the self-machine and restrict its movement) can also accelerate or decelerate, or change lanes to cut in front of or cut away from the self-machine. Changing the relative speed, acceleration, and lane occupancy (including cutting in and / or cutting away) of the self-machine increases the number of scenarios of this type. Similarly, changing the relative speed, acceleration, and lane occupancy (including cutting in and / or cutting away) of agent machines increases the number of scenarios of this type.

[0139] Multiple objects in adjacent lanes

[0140] In this set of scenarios, such as Figure 3N The depicted scenario involves an autonomous machine traveling in lanes on a road with more than one lane in the direction of travel. An agent machine may be positioned ahead of the autonomous machine, in the same or a different lane, at varying distances. The autonomous machine may move at the same speed as the agent machine, or it may move faster or slower. Changing relative distance, relative speed, acceleration, and / or lane occupancy can increase the number of scenarios of this type.

[0141] Parked vehicles

[0142] In this set of scenarios, such as Figure 3O The scenario depicts maneuvering commencing at a specific distance from and / or at a specific speed to a stationary (parked) vehicle. The autonomous machine may approach one or more parked vehicles, which may vary between cars, vans, trucks, etc. The vehicles may be parked back-to-back. In some embodiments, the nearest target may be a desired target type, such as a truck. The number of scenarios of this type can be increased by varying the initial distance at which the autonomous machine begins maneuvering to approach the parked vehicle, and by varying the speed at which the autonomous machine approaches the target vehicle.

[0143] Passing through VRU in an obstructed state

[0144] In this set of scenarios, such as Figure 3PThe depicted scenario suggests that the self-machine may be moving along a road. Pedestrians, bicycles, or other vehicles (as the case) can be target types that pass in front of the self-machine. The self-machine may be blocked by one or more other agent machines that are moving in the same direction as the self-machine but obstruct (block) its line of sight. The initial distance between the self-machine and the target types may differ. The absolute and relative velocities and / or accelerations of the self-machine and one or more agent machines may differ. Changing the initial distance, absolute and / or relative velocities, and / or absolute and / or relative accelerations can increase the number of scenarios of this type.

[0145] In all the scenarios discussed in this article, the speeds of the self-driving machine and the target vehicle can vary depending on the road type (e.g., 25 mph on residential streets, 35 mph on secondary roads, 40 or 45 mph on main roads, etc.). Highway or expressway speeds can be considered, but it's important to remember that the likelihood of a target crossing in front of the self-driving machine on these roads is extremely low.

[0146] Figure 4A The document presents non-limiting examples of compilation scenarios using the filtering features discussed herein, including but not limited to... Figures 3A to 3P The scenario is illustrated. Depending on the scenario, and / or the relative orientation, acceleration and / or velocity of the ego machine and the surrogate machine, and / or the distance between the ego machine and the surrogate machine, and / or the location of the occlusion, in some embodiments, different combinations of filtering, pruning, aggregation, and expansion can pass the input object span 405 through the filter function sequence FilterA 410, FilterB 420, FilterC 430, FilterD 440, FilterE 450, FilterF 460, FilterG 470, FilterH 480, and FilterK 490 to produce an output object span 495. In some embodiments, the output object span 495 is a scenario that can have varying ego machine, surrogate machine, and occlusion parameters as discussed herein to mine many different scenarios. In some embodiments, the output object span 495 can be fed back as the input object span 405 to produce different output object spans 495' after filtering. Changing the number and / or type of filters, and / or their combinations, and / or the sequence of operations can increase the number of scenarios that can be created.

[0147] Figure 4A This is not intended to limit how filtering functions are ordered or combined, or how many filtering functions are used to produce output objects spanning 495. Different combinations of filtering functions and the content discussed in this article... Figure 5The parameter variations discussed can generate a very large number of scenarios that can be used to train one or more machine learning systems on a self-machine to enable the self-machine to navigate correctly through many different combinations of proxy machines and / or objects and / or occlusions.

[0148] In some embodiments, the user can describe the desired scenario, and the system can follow... Figure 4A The lines depicted in the image guide users to fill in the appropriate mining function templates. Additionally or alternatively, users can review the list of existing scenarios and select scenarios that are the same as or closely match the scenario they are looking for.

[0149] Figures 3A to 3P The scenarios depicted are provided as examples to enable users to create their own scenarios. In some embodiments, users can specify the type of scenario they wish to mine, and the system can provide combinations and / or sequences of filtering functions to produce the desired output object span. In some embodiments, users can identify the desired filtering function. A knowledge graph can be generated to link the user's identification to similar or identical scenarios that have already been defined. Additionally or alternatively, users can search for specific scenarios or filtering functions. The knowledge graph can link these searches to previously generated scenarios and / or previous combinations of filtering functions to return the scenarios the user expects.

[0150] Figure 4B This is a high-level diagram depicting aspects of the disclosed system according to various embodiments. A layered stack of functions is shown here. The top layer 412 is where users design one or more data mining functions to cover their desired scenarios. An Application Programming Interface (API) enables users to write data mining functions. Below this layer is layer 422, which is a Software Development Kit (SDK) that users can use to create data mining functions. Layer 432 is a Unified Data Interface (UDI) that obtains data from the lower layers to provide to the SDK, facilitating the development of desired data mining functions by users. As its name suggests, the UDI provides an interface to the SDK to provide data, thereby enabling the querying of segment tags 442.

[0151] In some embodiments, UDI can query a cloud database to download the appropriate data. The download process can be slow, so UDI can provide facilities for locally caching queries so that subsequent calls to these queries can be performed more quickly.

[0152] In some embodiments, a segment label may have three components: the segment's start time; the segment's end time; and the label type. Segment labels can mark specific segments of collected data with specific attributes. In some embodiments, the label type can represent many different data types of interest, such as road conditions (wet, dry, icy); weather conditions (sunshine, rain, snow, sleet, freezing rain, hail, wind); lighting conditions (daytime, nighttime, dawn, dusk, cloudy); lane counts (number of lanes in any direction of the road); and / or even data such as the name of the city where the road is located, to which the excavation scenario can be applied.

[0153] UDI can also enable queries for agent trajector trajector 452 to provide data to the SDK. Here, "agent" can represent a traffic item, such as an autonomous machine, or a car, van, truck, motorcycle, bicycle, and / or pedestrian. "Agent trajectory" can include several operational and contextual characteristics of the agent, including but not limited to latitude, longitude, route, speed, acceleration, relative position to the autonomous machine, and / or timestamp. "Agent attributes" can include agent type (including but not limited to the traffic items mentioned above), and physical measurements of each agent (e.g., length, width, height).

[0154] The agent trajectory and attributes 452 can come from one or more sources. One such source is offline object detection data 462. In some embodiments, this can represent an offline multi-sensor-based 3D object detection and tracking system. The algorithm providing this data may be more accurate for object detection and tracking than an algorithm running on an autonomous machine, which could be a LiDAR-based object detection and tracking algorithm.

[0155] Another source for agent trajectories and attributes 452 could be online object detection data 472. In some embodiments, this could represent an online object detection and tracking output system, such as a system running online on the ego machine. Using this system, agent type and trajectory information can be extracted from system logs on the ego machine.

[0156] A third source of proxy trajectories and attributes can be a 4DGT manual annotation system 482. This system can be used to place manually provided labels onto 3DGT data that has already been automatically annotated at an early stage. In some embodiments, 3D can represent three physical dimensions, such as a planar dimension (2D) plus depth. In some embodiments, 4D can represent three physical dimensions plus time as a fourth dimension.

[0157] Figure 4CAn exemplary, non-limiting sequence of operations for mining a scene or set of scenes is illustrated. In operation 417, a scene can be defined. Items that can define a scene include interactions between the ego machine and the agent machine; the presence of objects or occlusions; the configuration of sensors on the ego machine; weather conditions; and / or lighting conditions. Other potential items are the required amount of data. In some embodiments, there are at least three data sources: automatically labeled data; manually labeled data; and data accumulated during the operation of the ego machine (this data may be referred to as “record while driving” (RWD) data). In some embodiments, RWD data may be provided as video clips. In some embodiments, automatic labeling may include LiDAR-based automatic labeling. In some embodiments, the automatically labeled data may be 3DGT (ground truth) data. In some embodiments, manually labeled data may include providing manually generated labels to the automatically labeled data. In some embodiments, the manually labeled data may be 4DGT data. In some embodiments, manual labeling may involve providing a bounding box around the vehicle.

[0158] In operation 427, the scene defined in operation 417 can be initially mined. In some embodiments, this initial mining can be performed using object trajectories automatically annotated by 3DGT, modifying the object trajectories to provide additional scenes that the machine might encounter. In operation 437, the data mined in operation 427 can be reviewed to identify appropriate annotation requirements in operation 447. In some embodiments, there may be several types of labels. One type is bounding boxes, which can be placed around occlusions and / or objects (e.g., pedestrians and cyclists) by agent machines and / or non-agent machines. For bounding box labels, in operation 487, a scene annotation ticket can be submitted for the data with bounding boxes placed around it. The annotated data is then output in operation 497.

[0159] Another type of label is the cropped label or 3DGT automatic label mentioned in this article. Since the data has already been labeled, the data with that label will be output in operation 497. If the labeling requirement is for 3DGT, a 4DGT labeling ticket will be submitted in operation 457. In operation 467, it is determined whether the identity (ID) of each agent machine is required. If so, the mined scene is further mined using 4DGT data mining in operation 477. In some embodiments, 4DGT data mining can be applied to all mined scenes. The 4DGT-mined scene can then be output in operation 497. Finally, if the ID of each agent machine is not required, the data is output in operation 497.

[0160] It should be understood that the arrangements and other arrangements described herein are presented as examples only. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to the arrangements, components, features, and elements shown, and certain elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components or together with other components, and can be implemented in any suitable combination and location (e.g., on a local device, vehicle, or edge machine; in the field (e.g., a locally hosted server); remotely (e.g., in one or more computing or server devices in one or more data centers in the cloud) and / or other locations). The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be performed using one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU)), microprocessors, microcontrollers, embedded processors, digital signal processors (DSPs), image signal processors (ISPs), physical processing units (PPUs), field-programmable gate arrays (FPGAs), accelerators (e.g., deep learning accelerators (DLAs), deep learning accelerator clusters (XNNs), neural network accelerators (NNAs) and / or neural processing units (NPUs), programmable vision accelerators (PVAs), optical flow accelerators (OFAs), etc.), application-specific integrated circuits (ASICs), data processing units (DPUs), quantum processors, etc. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 9A to 9E Example machine 900, Figure 10 Example calculation of ecosystem 1000, Figure 11 Example generative language model system 1100, and / or Figure 12 The example computing device 1200 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the operation.

[0161] Now for reference Figure 5 Each block of the method 500 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. The method can also be embodied as computer-usable instructions stored on a computer storage medium. The method can be provided by a standalone application, service, or managed service (independently or in combination with another managed service), or a plug-in to another product, to name a few. Furthermore, method 500 is illustrated by way of example regarding... Figures 1 to 2The method is described in the context of a system. However, the method may be performed additionally or alternatively by any system or combination of systems, including but not limited to the system described herein. Furthermore, the operations in method 500 may be omitted, repeated, and / or performed in any order without departing from the scope of this disclosure.

[0162] Figure 5 A flowchart of method 500 according to various embodiments is shown, which is used to generate multiple scenes using target objects and feature mining, and to train one or more machine learning systems to define additional scenes.

[0163] like Figure 5 As shown, method 500 begins with operation 502, where data generation pipeline 122 receives trajectory and type data from one or more agent machines. For example, the agent machines can... Figures 3A to 3P The agent can proceed along one of the indicated trajectories, or it can proceed along different trajectories. The agent machine type can vary among many different types of powered road vehicles, from motorcycles to cars to vans to pickup trucks to trucks to trailers. In some embodiments, the data generation pipeline receives type data for one or more targets, such as pedestrians or cyclists. Simulator 216 can use this data to generate simulation data for the scene. Additionally or alternatively, the scene generator in data generation pipeline 122 can directly use the trajectory and type data to define the scene.

[0164] In operation 504, the scene definer 218 in the data generation pipeline 122 uses trajectory and type data from the agent machine and sensor data from the ego machine to define a scene describing traffic interactions between the ego machine and one or more agent machines.

[0165] In operation 506, one or more parameters of the traffic interaction between the ego machine and the agent machine are changed. In operation 508, this change can be used as simulation data to define a scene, or it can be used directly to define the scene. In operation 510, parameter changes continue until all parameter changes are completed, or until the desired number of scenes have been defined. When the parameter changes are complete, in operation 512, the defined scene is compiled. In operation 514, one or more machine learning systems can be trained on multiple scenes. One purpose of training one or more machine learning systems is to enable the generation of additional scenes. Another purpose of training one or more machine learning systems is to enable the ego machine to safely interact with one or more agent machines and safely handle objects or occlusions.

[0166] In operation 516, if more trajectory and type data are available, the process returns to operation 502 to compile multiple additional scenarios and train one or more machine learning systems on those scenarios.

[0167] When execution engine 126 determines that the task should continue, it generates additional states, predictions, and / or actions for subsequent time steps. After a certain number of time steps, execution engine 126 can use the history of previous states up to the previous time step, a set of previous actions associated with the previous time step, and a set of embedded features associated with the current time step to generate one or more states associated with the current time step. Furthermore, after a certain number of time steps and / or according to another frequency, execution engine 126 can determine whether to continue task execution. Therefore, execution engine 126 uses navigation system 242 and machine 260 to execute the task until the task is completed, the task "times out," and / or another condition is met.

[0168] Figure 6 According to various embodiments Figure 2 A more detailed illustration of the data generation pipeline 122 is provided. As discussed herein, the data generation pipeline 122 defines scenarios that can be used to train, evaluate, test, simulate and / or otherwise operate the navigation system 242, other types of machine learning models that can be used by AMR and / or other machine types to perform tasks, the hardware configuration of the machine, and / or other components of the machine.

[0169] Within the data generation pipeline 122, simulator 216 generates and / or updates simulation data 232 relating to the ego machine, one or more agent machines, and one or more environments surrounding the occlusion and / or vehicle. For example... Figure 6 As shown, the simulation data 232 may include (but is not limited to) a map 612, odometer readings 614, an image 616, semantic labels 618 and / or enclosing shapes 620 (e.g., boxes, squares, rectangles, polygons, etc.).

[0170] Map 612 may include a representation of the roads the vehicle is traversing. For example, a map associated with a given simulation may include a two-dimensional (2D) and / or three-dimensional (3D) grid representing the space around the ego machine. Within the grid, each cell may be associated with a binary value indicating whether the corresponding spatial area is occupied (e.g., occupied by an object, agent machine, etc.). Each cell may also be associated with, or alternatively with, the probability that the corresponding spatial area is occupied. Each cell may also be associated with, or alternatively with, a numerical “cost” that quantifies the difficulty of moving within the corresponding spatial area. Map 612 may also include, or alternatively include, point clouds, grids, and / or other representations of “occupied space” around the ego machine, and / or be replaced by, point clouds, grids, and / or other representations of “occupied space” around the ego machine.

[0171] Odometry value 614 may include numerical values ​​associated with the motion of the self-machine. For example, odometer value 614 for a given self-machine in a simulation may indicate the distance traveled by the self-machine, the position of the self-machine, the heading of the self-machine, the linear velocity and / or angular velocity of the self-machine, the linear acceleration and / or angular acceleration of the machine, and / or other information that can be used to derive and / or estimate the position and / or orientation of the self-machine within the corresponding environment (e.g., road, entrance ramp, exit ramp, or lane).

[0172] Image 616 may include a visual representation of the environment surrounding the self-machine. For example, image 616 may depict the space surrounding the self-machine from the perspective of cameras and / or other sensor modalities on the self-machine. Image 616 may also or alternatively include a bird's-eye view of the environment, a perspective view of the environment, a 360-degree visualization of the environment, and / or other depictions of the environment outside the self-machine and / or the various cameras on the self-machine. Image 616 may include per-pixel color values, depth values, normal values, motion vectors (e.g., between consecutive video frames), LiDAR intensity values, and / or other types of information that can be used to characterize the environment.

[0173] Semantic labels 618 include indications of classes, objects, and / or other attributes that aid in understanding the environment. For example, semantic labels 618 may include semantic segmentation, which labels individual pixels within image 616, points within point clouds, polygons within meshes, and / or other representations of the environment with appropriate classes. Semantic labels 618 may also, or alternatively, identify objects, object instances, and / or other entities found in individual images, point clouds, meshes, and / or other representations of the environment. In some embodiments, labels may be generated from one or more of LiDAR-based automatic annotations (e.g., 3DGT automatic annotations), manual annotations (e.g., 4DGT (ground truth) data), or path labels.

[0174] Bounding shape 620 includes a representation of the position and / or size of objects within the environment (e.g., proxy machines and / or occlusions). For example, bounding shape 620 may include a rectangular outline of an object within image 616. Bounding shape 620 may also, or alternatively, include a parallelepiped outline of an object within a point cloud and / or other 3D representation of the environment. Each bounding shape may be associated with a class label, instance, and / or other indications of the corresponding object.

[0175] In one or more embodiments, simulator 216 uses physical simulation of the machine and / or environment and / or photorealistic rendering to generate at least a portion of simulation data 232. These physical simulations and / or photorealistic renderings can be performed using a physically-based virtual environment, such as NVIDIA Omniverse Simulator.TM NVIDIA Isaac Sim (NVIDIA Isaac Sim is a registered trademark of NVIDIA Corporation) is built on the platform. TM NVIDIAIsaacGym TM and / or NVIDIA Drive Sim TM These are registered trademarks of NVIDIA Corporation. Simulation environments can support the loading of robot models (e.g., quadruped robots, humanoid robots, differential drive systems, Ackerman drive systems, forklifts, etc.) and / or sensors (e.g., cameras, LiDAR, IMU, etc.), randomization of environment and / or environmental attributes (e.g., lighting, reflection, color, position, etc.), adding objects to the environment, and / or specification of the physical, material, and / or collision properties of objects.

[0176] As discussed herein, scene definer 218 identifies one or more scenes 234 associated with simulation data 232. For example, scene definer 218 may generate target locations to be navigated to or avoided in the corresponding environment within a given map output by simulator 216.

[0177] In some embodiments, the scene definer 218 defines some or all of the scenes 234 based on corresponding scene parameters 602. For example, scene parameters 602 may specify that navigation-based targets will be randomly sampled from navigable free space within a map 612 generated by the simulator 216. Scene parameters 602 may also, or alternatively, specify one or more regions within the map 612 from which scenes 234 should be sampled preferentially, and / or the nature of these regions (e.g., regions with more “details” and / or objects). The scene definer 218 can use these scene parameters 602 to sample scenes 234 more frequently from the corresponding regions, thereby increasing the coverage of tasks associated with regions in the scene.

[0178] Planner 220 uses one or more strategies 604 to generate commands 236 that instruct the machine in the simulation, executed by simulator 216, to perform actions related to scene 234. For example, planner 220 may implement and / or execute strategies 604 that generate commands 236 based on scene 234 from scene definer 218 and odometry values ​​614 and / or other information from simulator 216. Each strategy may include a planning stack, a strategy, and / or another component that generates commands 236 for operating the machine based on the machine's state and / or the environment surrounding the machine. These commands 236 may be (but are not limited to) linear and / or angular velocities, trajectories, paths, and / or another motion indication that directs the machine toward a scene 234 from scene definer 218 while avoiding objects in the corresponding simulated environment.

[0179] Each set of commands 236 output by planner 220 can be sent to simulator 216, which updates simulation data 232 based on the corresponding actions. For example, planner 220 can generate a given set of commands 236 based on simulation data 232 associated with a given time step in the simulation. These commands 236 can be transmitted to simulator 216, which generates updated simulation data 232 for the next time step. This simulation data 232 for the next time step can reflect changes in the machine and / or environment after the machine executes the actions corresponding to the commands 236. Simulator 216 can then send some or all of the updated simulation data 232 to planner 220, allowing planner 220 to generate a new set of commands 236 based on the updated simulation data 232 and the corresponding scenario 234 from scenario definer 218. This process can be repeated until scenario 234 is reached, a certain number of time steps have been executed in the simulation, and / or another condition indicating the end of the simulation is met.

[0180] Data logger 222 aggregates simulation data 232, scenarios 234, commands 236, and / or other data generated by simulator 216, scenario definer 218, and planner 220 into a record 238 of events associated with the corresponding time step. For example, simulator 216, scenario definer 218, and / or planner 220 may include nodes implementing publishers and / or be associated with such nodes in a publish-subscribe messaging system (e.g., Robot Operating System (ROS)). Each publisher may publish messages and / or events associated with a corresponding component of data generation pipeline 122 (e.g., simulator 216, scenario definer 218, planner 220, etc.) to one or more topics. Data logger 222 may include nodes implementing subscribers to these topics within the publish-subscribe messaging system and / or be associated with such nodes. Each subscriber may receive messages from one or more corresponding topics. Data logger 222 can record data from received messages by preprocessing data 606 and storing the preprocessed data in record 238.

[0181] In one or more embodiments, preprocessing 606 includes: determining the order in which data and / or events occur in a given simulation; generating records 238 spanning a time interval and / or at a frequency; downsampling data from some or all of the records; and / or other data processing operations associated with data from simulator 216, scene definer 218, and / or planner 220. For example, data logger 222 can synchronize data published at different frequencies by simulator 216, scene definer 218, and planner 220 by associating published data with individual time “frames,” time intervals, time steps, and / or other discrete-time measures in each simulation. Data logger 222 can also store data associated with each discrete-time measure in one or more records corresponding to that time measure. In another example, data logger 222 can downsample image 616, semantic tags 618, bounding shape 620, and / or other high-resolution data from simulator 216 before storing the data in record 238. In the third example, data logger 222 may store records 238 associated with a given scenario (e.g., a combination of a particular environment, machine, scenario, strategy, simulation, etc.) along with a path and / or directory corresponding to that scenario. Data logger 222 may also, or alternatively, associate individual records 238 with a unique identifier and / or name corresponding to the scenario.

[0182] In some embodiments, when creating record 238, data logger 222 generates visualizations and / or charts of the data in record 238. For example, data logger 222 may output image 616, semantic label 618, bounding shape 620, map 612, odometer value 614, and / or other visual representations of simulation data 232 in a graphical user interface. Data logger 222 may also or alternatively output "map pins," other representations of routes and / or scenes 234, and / or guidance associated with scenes 234 in the corresponding map 612, bird's-eye views of the environment in the simulation, and / or other visual depictions of the environment. Data logger 222 may also or alternatively output paths, trajectories, and / or other visual representations of commands 236, and / or actions performed based on commands 236, as an overlay on image 616, map, and / or other representations of the environment. The information output allows users to visually review the recorded data, determine whether the recorded data accurately reflects the corresponding scene, and / or determine whether the recorded data can be used for various use cases and / or applications.

[0183] Postprocessor 224 performs postprocessing 608, which adapts records 238 and / or other data generated by other components of the data generation pipeline 122 to various machine learning models and / or use cases. For example, postprocessor 224 may resample, compress, smooth, format, and / or otherwise transform data in a given set of records 238 into a form (e.g., file format, schema, etc.) that can be used to train, test, and / or evaluate machine learning models, hardware configurations, digital twins, and / or other components of physical and / or virtualized machines. Postprocessor 224 may also, or alternatively, store each set of records 238 that has been postprocessed for a given purpose and / or in a certain way in one or more corresponding datasets 240.

[0184] In some embodiments, post-processor 224 generates and stores metadata associated with the data recorded in dataset 240. For example, post-processor 224 may store the number of instances of object types (e.g., forklifts, shelves, people, etc.) in the scene, the time interval between consecutive frames represented by records 238 in the dataset, the distance covered by machines in the scene, the distribution of actions performed by machines, and / or other metrics and / or statistics associated with the simulated operation of machines in the scene, in association with the dataset for a given scene. In another example, post-processor 224 may specify scene parameters 602, strategy 604, preprocessing 606 and / or postprocessing 608 techniques, and / or other types of configuration parameters 622 used to generate the dataset in the metadata of the given dataset.

[0185] In some embodiments, some or all components of the data generation pipeline 122 are configured and / or customized via configuration parameters 622 provided by the control module 610. For example, configuration parameters 622 may specify the machine type and / or model, the machine's initial posture, the scene, one or more objects within the scene, object attributes, and / or other information that the simulator 216 can use for simulation. Configuration parameters 622 may also, or alternatively, include scene parameters 602 for controlling the generation of scene 234 by the scene definer 218. These scene parameters 602 can specify the type of scene 234 to be generated (e.g., location-based target, task, etc.), the sampling technique used to define scene 234, the regions within the environment from which scene 234 is to be sampled preferentially, the attributes of the regions within the environment from which scene 234 is to be sampled preferentially, the weights and / or other importance measures associated with sampling scene 234 from each region within the environment, the time to sample one or more new scenes 234 (e.g., after one or more existing scenes 234 have been reached), and / or other parameters that can be used to control and / or modify the scene 234 generated by scene definer 218. Configuration parameter 622 may also or alternatively include the specific strategies 604 that the planner 220 will use when generating commands 236, the behavioral characteristics associated with these strategies 604 (e.g., the level of aggressiveness and / or conservatism of the execution scenario; the type of commands 236 to be generated; the minimum, maximum, and / or valid values ​​associated with commands 236; etc.), text-based and / or code-based instructions for strategies 604, the platform and / or framework for implementing strategies 604, and / or other information that may be used to implement strategies 604 and / or generate commands 236. Configuration parameter 622 may also or alternatively include parameters related to the publication and / or subscription of topics by components of the data generation pipeline 122. Configuration parameter 622 may also or alternatively include identifiers, paths, recording frequencies, downsampling parameters, resampling parameters, file formats, graphs, visualization types, and / or other information that may be used to perform preprocessing 606 and / or postprocessing 608 associated with the data in records 238 and / or dataset 240.

[0186] Various techniques can be used to define and / or update configuration parameters 622. For example, configuration parameters 622 can be provided by one or more users via one or more configuration files, application programming interfaces (APIs), user interfaces, and / or other mechanisms. Some or all of the configuration parameters 622 can also be randomly generated (e.g., by sampling from the distribution, range, and / or set of valid configuration parameters 622). Some or all of the configuration parameters 622 can also be generated and / or updated using machine learning, optimization, and / or search techniques (e.g., to increase the coverage of dataset 240 to environments and / or scenes and / or define scenes relevant to a specific environment). Control module 610 can transmit configuration parameters 622 to simulator 216, scene definer 218, planner 220, data logger 222, and / or post-processor 224. Control module 610 can also or alternatively use the corresponding configuration parameters 622 to configure the operation of simulator 216, scene definer 218, planner 220, data logger 222, and / or post-processor 224.

[0187] In some embodiments, configuration parameter 622 includes a unique name and / or identifier for a given scenario (e.g., a combination of a specific environment, ego machine, proxy machine, occlusion, etc.) in which data will be generated and collected. Configuration parameter 622 can also be used to customize and / or randomize the type of environment and / or machine to be simulated, the scenario, the type of data to be recorded, the frequency of data recording, and / or the way the recorded data is converted into a format suitable for training and / or evaluating machine learning models and / or other components of the machine.

[0188] In some embodiments, different sets of configuration parameters 622 are used to launch different instances of the data generation pipeline 122 to generate data depicting different scenarios related to navigation and / or other types of tasks performed by machines in the environment. For example, multiple instances of the data generation pipeline 122 can be launched in parallel on multiple nodes of a cloud computing system using an NVIDIA One-system-to-many-others (OSMO) workflow. Each instance can be used to generate and / or collect simulation data 232, scenarios 234, commands 236, records 238, and / or datasets 240 associated with a given scenario and / or set of scenarios. The number of instances of the data generation pipeline 122 and / or the number of nodes on which a given instance of the data generation pipeline 122 is deployed can be scaled to suit requirements and / or preferences associated with the number of scenarios to be generated; the applications and / or use cases associated with the scenarios; and the coverage of environment, machines, policy 604, scenario 234, and / or other factors. Additional OSMO workflows can also be used to launch pipelines for training, testing, and / or evaluating machine learning models, policies, hardware configurations, software stacks, twins, and / or other components or representations of machines using the generated simulation data 232, scenarios 234, commands 236, records 238, and / or datasets 240.

[0189] Figure 7 The various embodiments are illustrated by Figure 2 The data generation pipeline 122 generates sample data for creating the scene. For example... Figure 7 As shown, the data includes image 616(1), which depicts the road environment surrounding the autonomous machine at a given time step in the simulation. Image 616(1) includes a perspective view of the environment from a camera in front of the autonomous machine. Simulator 216 may render image 616(1) based on a 3D scene representing the environment, an odometer value 614 associated with the machine at that time step, and / or other simulation data 232.

[0190] The scenario may also include a set of commands 236 associated with the same time step. These commands 236 ( Figure 2 This can include trajectory, linear velocity, and / or angular velocity. These commands 236 can be used to update the state of the machine and / or environment in the simulation. The updated state can then be used to generate new images 616, other simulation data 232, and / or commands 236 for the next time step in the simulation.

[0191] Those skilled in the art will understand that the arrangements and other configurations described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and can be implemented in any suitable combination and location. The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can use... Figures 9A to 9E Example machine 900, Figure 10 Example calculation of ecosystem 1000 Figure 11 Example generative language model system 1100 and / or Figure 12 The example computing device 1200 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the same task.

[0192] Now for reference Figure 8 Each block of the method 800 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. These methods can also be embodied as computer-usable instructions stored on a computer storage medium. These methods can be provided by standalone applications, services, or managed services (independently or in combination with another managed service) or plug-ins to another product, to name a few. Furthermore, method 800 is illustrated by way of example regarding... Figure 1 and Figure 6 The systems described herein are as follows. However, these methods may be additionally or alternatively implemented by any system or combination of systems, including but not limited to the systems described herein.

[0193] Figure 8 A flowchart of a method 800 for generating a scene associated with machines in an environment, according to various embodiments, is shown. Figure 8As shown, method 800 begins with operation 802, where data generation pipeline 122 receives configuration parameters associated with scene generation. For example, data generation pipeline 122 may receive configuration parameters via one or more configuration files, API calls, and / or user interfaces. Configuration parameters can be used to configure and / or customize scene generation. For example, configuration parameters include specific names and / or identifiers for a given combination of conditions (e.g., a combination of specific environments, machines, scenes, etc.) under which data will be generated and collected. These parameters can also be used to customize the type of environment and / or machine to be simulated, the scene, the type of data to be recorded, the frequency of data recording, and / or the way the recorded data is converted into a format suitable for training and / or evaluating machine learning models and / or other components of the machine. Figure 5 In the flowchart, parameters and other data vary for a specific scenario to define other scenarios used to train one or more machine learning models.

[0194] In operation 804, the data generation pipeline 122 initializes one or more simulations using a set of properties associated with the machine and / or the environment in which the machine operates. For example, the data generation pipeline 122 may use configuration parameters to determine the machine type, machine model, and / or the machine's location in the environment. The data generation pipeline 122 may also, or alternatively, acquire, generate, and / or randomize data (e.g., trajectory and type data), parameters of traffic interactions between machines (e.g., ego machines and agent machines), and / or a 3D scene corresponding to the environment and / or a map of that 3D scene. The data generation pipeline 122 may also, or alternatively, add one or more objects to the 3D scene and / or set the physical, material, and / or collision properties of one or more objects.

[0195] In operation 806, data generation pipeline 122 determines the scenario associated with the machine's operation in the environment. For example, data generation pipeline 122 can define a navigation-based scenario by sampling locations the machine needs to navigate to in the environment from unoccupied space within the environment. This sampling can be preferentially performed on certain areas within the environment specified in the configuration parameters and / or certain areas having the properties specified in the configuration parameters.

[0196] In operation 808, the data generation pipeline 122 generates simulated data depicting the operation of the machine in the environment via one or more simulations. For example, the data generation pipeline 122 can render one or more images of the environment from one or more cameras on the machine, one or more locations outside the machine, and / or other viewpoints. The data generation pipeline 122 can also, or alternatively, generate point clouds, IMU measurements, and / or other sensor measurements associated with sensors on the machine.

[0197] In operation 810, the data generation pipeline 122 determines one or more commands for the machine based on simulated data and / or a scenario, via a machine-specific strategy. For example, the data generation pipeline 122 may input simulated data and / or a scenario into a planning stack, a neural network, and / or another component implementing the strategy. Given the input data, the component may generate commands specifying the linear velocity and / or angular velocity of the machine. The component may also, or alternatively, generate one or more command distributions sampled from one or more commands.

[0198] In operation 812, the data generation pipeline 122 stores simulation data and commands in one or more data records. For example, the data generation pipeline 122 may associate simulation data generated in operation 808 and commands generated in operation 810 with the same time step and / or "frame" in the simulation. The data generation pipeline 122 may also record simulation data and commands in one or more data records associated with that time step and / or frame.

[0199] In operation 814, the data generation pipeline 122 determines whether to continue scene generation. For example, the data generation pipeline 122 may determine that scene generation will continue until there are sufficient scenes to train one or more machine learning models that will operate the machine, the simulation has run a certain number of time steps, and / or another condition is met. If the data generation pipeline 122 determines that scene generation will continue, then the data generation pipeline 122 performs operation 816, where the data generation pipeline updates the simulation data based on commands. For example, the data generation pipeline 122 may update the machine's position, heading, speed, and / or other states to reflect the machine's execution of commands. The data generation pipeline 122 may also, or alternatively, generate new image and / or sensor data reflecting the updated machine states.

[0200] Then, data generation pipeline 122 repeats operation 810 to generate new commands based on the updated simulation data. Similarly, data generation pipeline 122 repeats operation 812 to store the updated simulation data and commands in one or more additional data records. For example, data generation pipeline 122 may store the updated simulation data and commands in association with new (e.g., incremental) time steps and / or frames. After a given set of simulation data and commands has been stored in one or more data records, data generation pipeline 122 repeats operation 814 to determine whether to continue generating the scene.

[0201] After data generation pipeline 122 determines in operation 814 that scenario generation should be stopped, data generation pipeline 122 performs operation 816, wherein data generation pipeline 122 stores and / or formats data records within one or more datasets. For example, data generation pipeline 122 can generate different datasets for each use case and / or application associated with a scenario. Within a given dataset, data generation pipeline 122 can resample, format, and / or otherwise post-process the corresponding data records to adapt the data records to the corresponding use case and / or application. Data generation pipeline 122 can then provide the datasets for training, testing, and / or evaluating machine learning models, hardware configurations, policies, digital twins, and / or other components and / or representations of machines in various environments.

[0202] The systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, spacecraft, ships, shuttle buses (e.g., robotaxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones, and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, including, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and supervision, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or participant simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets (e.g., NVIDIA's Omniverse), cloud computing, and / or any other suitable application.

[0203] The disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines, etc.), systems implemented using robots, aviation systems, medical systems, rowing systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing language models (e.g., large language models (LLM), visual language models (VLM), visual-language-action (VLA) models, and / or multimodal language models), systems using or deploying one or more inference microservices, systems containing one or more machine learning models deployed in services or microservices and OS-level virtualization packages (e.g., containers), systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0204] Example autonomous or semi-autonomous machines

[0205] Figure 9A Examples of sensor locations with corresponding fields of view or sensing fields of view of an autonomous or semi-autonomous vehicle 900A, an autonomous mobile robot (AMR) 900B, and a humanoid robot 900C, according to some embodiments of this disclosure. While three types of machines 900 are shown, this is not intended to be limiting, and the machines 900 described herein may include vehicles, cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police or emergency vehicles, ambulances, boats, construction vehicles, underwater vehicles, robots (e.g., AMRs, humanoid robots, robotic arms, end effectors, forklifts, etc.), drones, aircraft, vehicles coupled to trailers (e.g., semi-trailer tractors for hauling goods), and / or another type of vehicle or machine (e.g., driverless and / or vehicles or machines accommodating one or more passengers). In some cases, vehicle 900A, AMR 900B, humanoid robot 900C, and / or other machine types may be collectively referred to herein as machine 900.

[0206] Regarding Vehicle 900A, autonomous and semi-autonomous vehicles are typically described by automation levels, defined by the National Highway Traffic Safety Administration (NHTSA) (a division of the U.S. Department of Transportation) and the Society of Automotive Engineers (SAE) in their "Classification and Definition of Terms Related to Driving Automation Systems for Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 900 may have one or more functions that meet Level 3 through Level 5 of autonomous driving. For example, Vehicle 900 may be able to provide driver assistance (Level 1), partial automation (Level 2, Level 2+, Level 2++), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the specific implementation. The term “autonomy” as used herein can include any and / or all types of autonomy of Machine 900 or other machines, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, provision of auxiliary autonomy, semi-autonomy, primary autonomy or other specified autonomy.

[0207] about Figure 9A The sensors and their respective fields of view (not shown for clarity) or sensing fields (not shown for clarity) are an example embodiment and are not intended to be limiting. Although not shown, each sensor may have a corresponding field of view (e.g., a 360-degree field of view around camera 968D, a 180-degree field of view around wide-angle camera 968B, a 360-degree sensing field of view of LiDAR sensor 964, etc.). For example, only a subset of the sensors shown may be included, additional sensors may be included, alternative sensors may be included, the number of each sensor mode may be different, the sensor modes may be different (e.g., LiDAR or RADAR may not be included, SONAR, thermal sensors, etc. may be included), and the sensor locations may differ from those shown on vehicles 900a, AMR 900b, and / or humanoid robots 900c, etc. For example, for vehicle 900a, the location, number, modes, and / or other sensor information may differ depending on the type (e.g., SUV, truck, car, robot, motorcycle, etc.), size (e.g., 18-wheeler, transport vehicle, small car, etc.), and associated functions (e.g., L2 vs. L5). Similarly, for the AMR 900b and / or the humanoid robot 900c, the shape, size, purpose, implementation method, model, etc., can determine the number and type of sensors used.

[0208] like Figure 9AAs shown, autonomous or semi-autonomous vehicles 900A, AMR 900B, and humanoid robots 900C may include different sensor types, numbers, and locations. As a non-limiting example, vehicle 900A may include twelve cameras 968, such as a front wide-angle camera (e.g., 120-degree field of view (FOV)), a front telephoto camera (e.g., 30-degree FOV), a rear-side left camera (e.g., 70-degree FOV), a rear-side right camera (e.g., 70-degree FOV), a front fisheye camera (e.g., 200-degree FOV), a rear fisheye camera (e.g., 200-degree FOV), a left fisheye camera (e.g., 200-degree FOV), a right fisheye camera (e.g., 200-degree FOV), a front telephoto satellite camera (e.g., 30-degree FOV), a rear telephoto camera (e.g., 30-degree FOV), a cross-left camera (e.g., 120-degree FOV), and a cross-right camera (e.g., 120-degree FOV). In this embodiment, the camera 968 may use a Gigabit Multimedia Serial Link (GMSL) interface (e.g., GMSL2) as input / output (I / O).

[0209] In some embodiments, although Figure 9A As not shown, vehicle 900A may include an in-cabin occupant and / or driver monitoring system, which may include various sensors. For example, in-cabin sensors may include various cameras 968, such as a driver monitoring camera (e.g., located in front of the driver's seat and facing the driver's seat at a 55-degree FOV), a front occupant monitoring camera (e.g., located in front of the front occupant seat and facing the front occupant seat at a 190-degree FOV), and a rear occupant monitoring camera (e.g., located in front of the rear occupant seat and facing the rear occupant seat at a 190-degree FOV). Similar to external cameras 968, in embodiments, internal cameras 968 may use a GMSL (e.g., GMSL2) interface for I / O.

[0210] As another non-limiting example, the vehicle 900A may also include nine RADAR sensors 960. For example, the vehicle 900A may include a front center imaging RADAR sensor (e.g., 120-degree FOV or sensing field), a left front corner RADAR sensor (e.g., 160-degree FOV or sensing field), a right front corner RADAR sensor (e.g., 160-degree FOV or sensing field), a right rear corner RADAR sensor (e.g., 160-degree FOV or sensing field), a left RADAR sensor (e.g., 160-degree FOV or sensing field), a right RADAR sensor (e.g., 160-degree FOV or sensing field), a left rear RADAR sensor (e.g., 50-degree FOV or sensing field), and a right rear RADAR sensor (e.g., 50-degree FOV or sensing field). In embodiments, the RADAR sensors 960 may use an Ethernet interface as I / O.

[0211] As a non-limiting example, the vehicle 900A may also include twelve ultrasonic sensors 962. Figure 9A As shown, the ultrasonic sensor can be placed along the front and rear bumpers of vehicle 900A and along the sides of vehicle 900A, and can be used to detect objects (static and dynamic) close to vehicle 900A. In some embodiments, the ultrasonic sensor 962 can use a DS13 interface as I / O.

[0212] As a non-limiting example, the vehicle 900A may also include a LiDAR sensor 964, such as a front-center LiDAR sensor (e.g., a 120-degree horizontal FOV or sensing field and a 30-degree vertical FOV or sensing field). In some embodiments, such as when using additional or alternative LiDAR sensors, the LiDAR sensors may have different horizontal and vertical fields of view or sensing fields. For example, the LiDAR sensor 964 may include a 360-degree horizontal FOV or sensing field (e.g., located in a rotating LiDAR sensor) and a 90-degree vertical FOV or sensing field. In some embodiments, the LiDAR sensor 964 may use an Ethernet interface as I / O.

[0213] As a non-limiting example, the Autonomous Mobile Robot (AMR) 900B may include three LiDAR sensors 964. For example, the topmost LiDAR sensor 964 may include a beam or 3D LiDAR sensor (e.g., a 360-degree horizontal and 90-degree vertical FOV or sensing field), and the front and rear LiDAR sensors may include planar or 2D LiDAR sensors (e.g., a 180-degree horizontal FOV or sensing field).

[0214] As a non-limiting embodiment, the AMR 900B may further include eight cameras 968, such as a front stereo camera (e.g., 120-degree FOV), a rear stereo camera (e.g., 120-degree FOV), a left stereo camera (e.g., 120-degree FOV), a right stereo camera (e.g., 120-degree FOV), a front fisheye camera (e.g., 202-degree ± 3-degree FOV), a rear fisheye camera (e.g., 202-degree ± 3-degree FOV), a left fisheye camera (e.g., 202-degree ± 3-degree FOV), and a right fisheye camera (e.g., 202-degree ± 3-degree FOV).

[0215] The AMR 900B may also include a charging port, charging port contacts, status indicators, one or more (e.g., four) RGB LEDs, one or more IMU sensors 966, a magnetometer, and a barometer. The AMR 900B is capable of high-precision time synchronization between sensors using hardware timestamps and PTPs over Ethernet with sensor acquisition times of less than 10 microseconds. In an embodiment, the AMR 900B provides simultaneous camera capture within 100 microseconds of a single hardware trigger on all cameras 968, and can write sensor captures to disk at a rate of 4 GB / s for writing packets (e.g., writing to ROSbags of the Robot Operating System (ROS)). Therefore, the AMR 900B is capable of running ROS (e.g., NVIDIA's IsaacROS), can be remotely operated (as described herein), can map the environment, and can navigate the environment using vision cameras 968, LiDAR 964, and / or other sensor types or modalities.

[0216] The humanoid robot 900C may include (as a non-limiting example) a LiDAR sensor 964. For example, the LiDAR sensor 964 may include a beam or a 3D LiDAR sensor (e.g., a 360-degree horizontal and 90-degree vertical FOV or sensing field), or it may include a planar or 2D LiDAR sensor (e.g., a 180-degree horizontal FOV or sensing field).

[0217] As a non-limiting embodiment, the humanoid robot 900C may also include four cameras 968, such as a front stereo camera (e.g., 120-degree FOV), a rear stereo camera (e.g., 120-degree FOV), a front fisheye camera (e.g., 202-degree ± 3-degree FOV), and a rear fisheye camera (e.g., 202-degree ± 3-degree FOV).

[0218] As a non-limiting embodiment, the humanoid robot 900C may also include four ultrasonic sensors 962, such as a left arm ultrasonic sensor, a right arm ultrasonic sensor, a left leg ultrasonic sensor, and a right leg ultrasonic sensor.

[0219] The humanoid robot 900C may also include any number of actuators, such as those allowing control and manipulation of joints. For example, the humanoid robot 900C may include actuators that allow for various degrees of freedom (DoF) depending on the design. In a non-limiting embodiment, the humanoid robot 900C may have a total of 40 degrees of freedom (DoF) (e.g., 6DoF x2 for arms, 6DoF x2 for hands, 6DoF x2 for legs, 2DoF for the torso, and 2DoF for the neck). Actuators can convert energy into physical motion, thereby allowing actions such as joint movement, locomotion, and grasping / manipulation. For example, motors and servos can be used to perform joint movements to control the rotation of joints in an arm or manipulator and allow for reaching, grasping, and manipulating objects. Locomotion can be achieved by moving through the environment using wheels, tracks, or other mobility devices (robot legs). Grasping and manipulation can be performed using end effectors or hands / fingers, which can be equipped with actuators to grasp objects, apply forces, and perform specific tasks. In some examples, the humanoid robot 900C may include position and orientation sensors, such as encoders, gyroscopes, etc., to determine the robot 900C's position in space, thereby enabling position determination and motion tracking. In embodiments, the humanoid robot 900C may include force and pressure sensors to detect environmental interactions, enabling the robot 900C to grasp objects with appropriate force and avoid obstacles along its path. Perception sensors (e.g., cameras, LiDAR, RADAR, ultrasound, sonar (SONAR), etc.) may be used in conjunction with tactile sensors to enable the robot 900C to perceive objects, shapes, and textures, and to know when to begin and stop touching (along with force sensors that regulate the force used during touching). As a non-limiting example, the humanoid robot 900C may have a height of approximately 1-2 meters (e.g., 1.7 meters or 5 feet 6 inches), a weight of 50-70 kilograms, be able to move at speeds of 8 km / h or higher, and be able to carry a payload of 20-100 kilograms, depending on the system design and requirements.

[0220] In an embodiment, the humanoid robot 900C may include a dialogue system—such as a dialogue system driven by a language model (e.g., LLM, VLM, MMLM, VLA, etc.)—to help understand the environment, reason, and communicate with humans, animals, devices, and / or other robots, and / or make planning, control, and navigation decisions. Therefore, in addition to performing various tasks, the humanoid robot 900C can also use onboard sensors, microphones, and speakers to understand speech, audio, and visual cues, while also being able to communicate with the environment.

[0221] Referring to camera 968 of machine 900, the camera type of camera 968 may include, but is not limited to, digital cameras applicable to components and / or systems of machine 900. For vehicle 900a implementation, camera 968 may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may have any image capture rate, such as 30 frames per second (fps), 60 fps, 120 fps, 240 fps, etc., depending on the embodiment. The camera may use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-neutral-neutral-colorless (RCCC) color filter array, a red-neutral-neutral-blue (RCCB) color filter array, a red-blue-green-neutral (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a colorless pixel camera (e.g., a camera with an array of RCCC, RCCB, and / or RBGC color filters) can be used to improve light sensitivity.

[0222] The field of view includes cameras (e.g., front-facing cameras) in front of the machine 900, which can be used for surround view to help identify the path and obstacles ahead, and, with the help of one or more controllers 936 and / or control SoCs, provide information crucial for generating an occupancy grid and / or determining preferred machine movement, trajectory, and / or path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including lane departure warning (“LDW”), adaptive cruise control (“ACC”), and / or other functions such as traffic sign recognition.

[0223] Various cameras can be used in front-mounted configurations, including, for example, monocular camera platforms that include complementary metal-oxide-semiconductor (“CMOS”) color imagers. Another example could be a wide-angle camera 968B, which can be used to perceive objects entering the field of view from the periphery (e.g., pedestrians, warehouse vehicles, other robots, pedestrian traffic, or bicycles). Furthermore, any number of long-range cameras 968E (e.g., a pair of long-range stereo cameras) can be used for depth-based object detection, especially for objects for which neural networks have not yet been trained. The long-range camera 968E can also be used for object detection and classification, as well as basic object tracking.

[0224] Any number of stereo cameras 968A can also be included in front-mounted and / or other (e.g., rear-mounted) configurations. In at least one embodiment, one or more stereo cameras 968A may include an integrated control unit comprising a scalable processing unit that can provide programmable logic (“FPGA”) and a multi-core microprocessor with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the machine 900 environment, including distance estimates of midpoints in the image (e.g., parallax or depth images). Alternative stereo cameras 968A may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip that can measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 968A may be used in addition to those described herein, or as an alternative to those described herein. For example, in some embodiments, stereo depth estimation can be performed using a camera other than a stereo camera (e.g., two monocular cameras with at least partially overlapping fields of view).

[0225] Cameras with a field of view including portions of the machine 900's side environment (e.g., side-view cameras) can be used, for example, for surround view, to provide information for creating and updating occupancy grids, and to generate side collision warnings and / or indicate to the AMR 900B or humanoid robot 900C, for example, the presence of objects, features, and / or people on the side. For example, a surround camera 968D can be mounted on the machine 900. The surround camera 968D can include a wide-angle camera 968B, a fisheye camera, a 360-degree camera, etc. For example, four fisheye cameras can be mounted on the front, rear, and sides of the machine 900. In an alternative arrangement, the machine 900 can use three surround cameras 968D (e.g., left, right, and rear), and can utilize one or more other cameras (e.g., a front-facing camera) as a fourth surround-view camera.

[0226] A camera 968 (e.g., a rear-view camera) having a field of view including a portion of the environment behind the machine 900 can be used to understand objects, features, people, and / or other information behind the machine 900, such as for parking assistance, surround view, rear collision warning, planning, control, and navigation determination, and / or to create and update occupancy grids, BEV images representing the environment, height maps, etc. A wide variety of cameras 968 can be used, including but not limited to those also suitable for use as front cameras (e.g., long-range and / or mid-range cameras 968E, stereo cameras 968A, infrared cameras 968C, etc.), rear cameras, side cameras, downward cameras, upward cameras, and / or similar cameras 968, as described herein.

[0227] Similarly, for LiDAR sensor 964, RADAR sensor 960, ultrasonic sensor 962 and / or other sensor modes or types, the location and placement of the sensors, as well as their corresponding fields of view or sensing fields, can be determined based on the use case, implementation, or design of the specific machine 900.

[0228] For example, machine 900 includes a RADAR sensor 960, which can be used by machine 900 for long-range object detection, even in dark and / or inclement weather conditions. In embodiments, the RADAR functional safety level may be ASIL B. RADAR sensor 960 can be controlled and accessed for object tracking data using CAN and / or bus 902 (e.g., for transmitting data generated by RADAR sensor 960), and in some examples, raw data can be accessed using Ethernet. Various types of RADAR sensors can be used. For example, but not limited to, RADAR sensor 960 can be suitable for front, rear, and side radar applications. In some examples, a pulse Doppler RADAR sensor is used.

[0229] The RADAR sensor 960 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range lateral coverage, etc. In some examples, long-range radar can be used for adaptive cruise control (ACC) functions. A long-range RADAR system can provide a wide field of view, for example, within a 250m range, achieved through two or more independent scans. The RADAR sensor 960 can help distinguish between static and moving objects; ADAS systems can use it for emergency braking assistance and forward collision warning, and robots can use it to detect dynamic objects in various environments (e.g., low-light or no-light environments). A long-range RADAR sensor can include a single static multimodal RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record the machine 900's surroundings at higher speeds while minimizing peripheral interference (e.g., traffic from adjacent lanes). The other two antennas expand the field of view, enabling rapid detection of objects entering or leaving the machine's direct path (e.g., lanes).

[0230] Mid-range RADAR systems can include, for example, a range of up to 960m (front) or 80m (rear), and a field of view of up to 42 degrees (front) or 150 degrees (rear). Short-range RADAR systems can include, but are not limited to, RADAR sensors designed to be mounted at both ends of a side surface (e.g., rear bumper), allowing continuous monitoring of blind spots behind and beside a machine (e.g., a vehicle, robot, etc.) using two beams. Therefore, short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0231] Machine 900 may also include ultrasonic sensors 962. Ultrasonic sensors 962 may be located at the front, rear, and / or sides of machine 900 and may be used to assist near-field sensing, such as for parking assistance, collision avoidance (e.g., for robot parts), and / or to create and update occupancy grids, evidence mesh maps (EGMs), height maps, BEV images, and / or other representations of objects and features in the machine 900 environment. Multiple ultrasonic sensors 962 may be used, and different ultrasonic sensors 962 may be used for different detection ranges (e.g., 2.5 m, 4 m). For example, ultrasonic sensors 962 may operate at ASIL B functional safety level.

[0232] Machine 900 may include a LiDAR sensor 964. The LiDAR sensor 964 can be used for object and feature detection, pedestrian and other robot detection, emergency braking, collision avoidance, simultaneous localization and mapping (SLAM), free space detection, and / or other functions. In embodiments, the LiDAR sensor 964 may be of functional safety level ASIL B. In some examples, machine 900 may include multiple LiDAR sensors 964 (e.g., two, four, six, etc.), which can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0233] In some examples, the LiDAR sensor 964 may be able to provide a 360-degree field of view of objects and a list of their distances. For example, a commercially available LiDAR sensor 964 may have a advertised range of approximately 900m, an accuracy of 2cm-3cm, and support for 900Mbps Ethernet connectivity. In some examples, one or more non-protruding LiDAR sensors 964 may be used. In such examples, the LiDAR sensor 964 can be implemented as a small device that can be embedded in the front, rear, side, top, and / or corner of a machine 900. In such examples, the LiDAR sensor 964 can provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, with a range of up to 200m even for low-reflectivity objects. A front-mounted LiDAR sensor 964 can be configured with a horizontal field of view between 45 and 135 degrees.

[0234] In some examples, LiDAR technology, such as 3D flash LiDAR, can also be used. 3D flash LiDAR uses a laser flash as a transmission source, illuminating the environment around a vehicle up to approximately 200m away. The flash LiDAR unit includes a receiver that records the laser pulse transmission time and the reflected light on each pixel, which in turn corresponds to the distance from the vehicle to the object. Flash LiDAR allows for the generation of highly accurate and distortion-free images of the surrounding environment with each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of machine 900. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras with no moving parts other than a fan (e.g., non-scanning LiDAR devices). Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of 3D distance point clouds and co-registration intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, LiDAR sensor 964 may be less susceptible to motion blur, vibration, and / or shock.

[0235] Figure 9B This is an illustration of the location of sensors and components in an example autonomous or semi-autonomous vehicle 900A (also referred to herein as "vehicle 900", "self-vehicle 900", "self-machine 900", or "machine 900") according to some embodiments of this disclosure. While vehicle 900A is shown in the figure, this is not intended to be limiting, and similar components and / or sensors may be included on any other machine type without departing from the scope of this disclosure. For example, similar sensors and / or components may be used in vehicles, cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police cars, ambulances, ships, construction vehicles, underwater vehicles, robots (e.g., AMRs, humanoid robots, robotic arms, end effectors, forklifts, etc.), drones, aircraft, vehicles coupled to trailers (e.g., semi-trailers for hauling goods), and / or another type of vehicle or machine (e.g., driverless and / or capable of accommodating one or more passengers).

[0236] Figure 9CThis is a block diagram of an example system architecture for a machine 900 (e.g., an autonomous or semi-autonomous vehicle 900A, an autonomous mobile robot (AMR) 900B, a humanoid robot 900C, and / or other types of machines) according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are presented as examples only. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of the arrangements shown, and some elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components or together with other components, and can be implemented in any suitable combination and location (e.g., on a local device, vehicle, or edge machine, in the field (e.g., a locally hosted server), in a remote location (e.g., in one or more computing or server devices in one or more data centers in the cloud) and / or other locations). The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, microcontroller, embedded processor, digital signal processor (DSP), image signal processor (ISP), physical processing unit (PPU), field-programmable gate array (FPGA), accelerators (e.g., deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural network accelerator (NNA) and / or neural processing unit (NPU), programmable vision accelerator (PVA), optical flow accelerator (OFA), etc.), application-specific integrated circuit (ASIC), data processing unit (DPU), quantum processor, etc.) that execute instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 9A to 9E Example machine 900, Figure 10 Example calculation of ecosystem 1000, Figure 11 Example generative language model system 1100 and / or Figure 12 The example computing device 1200 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the same task.

[0237] Figure 9CEach component, feature, and system of machine 900 is shown connected via bus 902 (or referred to as "machine communication network 902" or simply "communication network 902"). Bus 902 may include a Controller Area Network (CAN) data interface (or referred to herein as "CAN bus"). CAN can be a network within machine 900 used to help control various features and functions of machine 900, such as brake actuation, acceleration, braking, steering, windshield wipers, etc. CAN bus can be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CANID). CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button position, and / or other vehicle status indicators. CAN bus may conform to ASIL B standard. In some embodiments, in addition to or as an alternative to the CAN bus, bus 902 may include FlexRay, embedded buses (e.g., SPI, I2C), local interconnect links (LIN), NVIDIA's NVLink, USB (2.0, 3.0 and above), radio frequency (RF), Ethernet (e.g., 10BASE / 100BASE, 1000BASE, 10G, etc.), and / or other communication protocols or functions. Furthermore, while a single line is used to represent bus 902, this is not limiting. For example, there can be any number of buses 902, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 902 may be used to perform different functions and / or for redundancy. For example, a first bus 902 may be used for collision avoidance, while a second bus 902 may be used for actuation control. In any example, each bus 902 may communicate with any component of machine 900, and two or more buses 902 may communicate with the same component. In some examples, each computer or computing engine within each SoC 904, each controller 936, and / or machine 900 can access the same input data (e.g., input from sensors in the machine 900) and can be connected to a common bus, such as the CAN bus.

[0238] Machine 900 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, batteries, side mirrors, and / or other parts of the vehicle or machine. Machine 900 may include a propulsion system 950, such as an internal combustion engine, a hybrid power plant, an all-electric motor, a hydrogen fuel cell engine, and / or another type of propulsion system. Propulsion system 950 may be connected to the drivetrain of machine 900, which may include a transmission to enable propulsion of machine 900. Propulsion system 950 may be controlled in response to a signal received from throttle / accelerator 952.

[0239] Steering system 954 may include a steering wheel and / or other steering mechanisms (e.g., remote steering and / or local steering) for steering machine 900 (e.g., along a desired path or route) while propulsion system 950 is in operation (e.g., when the vehicle is moving). Steering system 954 may receive signals from steering actuator 956. In some embodiments, a steering wheel or other steering mechanism may be omitted, for example, for machine 900 capable of fully automated (e.g., level 5) functionality.

[0240] The brake sensor system 946 can be used to operate the vehicle brakes in response to signals received from the brake actuator 948 and / or the brake sensor.

[0241] Machine 900 may include one or more controllers 936, such as those described herein. Figure 9AThe controller 936 is described. It can be used for a variety of functions and can be coupled to any of the various other components and systems of the machine 900. For example, the controller 936 can be used to control the machine 900, artificial intelligence performed on the machine 900, infotainment of the machine 900, etc. For example, one controller 936 can be used for some or all of the functions, or different controllers 936 can be used for different functions, for example, to ensure availability and safety separation between various controllers for different tasks. For example, the controller 936 can use a system-calculated plan (e.g., the path or trajectory of vehicle 900A or AMR 900B, or the movement, component trajectory, motion position or displacement, etc. of joints or components (e.g., manipulators, end effectors, limbs, hands, fingers, legs, feet, etc.) of humanoid robot 900C to control the machine 900 in the environment. In some cases, the controller 936 may include a proportional-integral-derivative (PID) controller, a fuzzy logic controller, a neural controller (e.g., a controller embodied as one or more neural networks), a force control controller, a programmable logic controller (PLC), and / or other types of controllers. For example, in the humanoid robot 900C, the controller 936 can act as the brain, responsible for analyzing sensor data, making decisions, and sending commands to the actuators. The controller 936 may include a low-level controller that handles basic motor control, ensuring accurate and precise movement of the individual joints and actuators. The controller 936 may also include a high-level controller to coordinate multiple actuators and sensors, plan complex movements, and adapt to constantly changing environments.

[0242] In some embodiments, controller 936 may include an artificial intelligence controller that can use AI algorithms (e.g., DNN, MLM, etc.) to learn, make decisions, and autonomously perform tasks of machine 900. In some embodiments, controller 936 may use a fixed open-loop control algorithm and not adjust its actions according to the environment. In other embodiments, closed-loop control may be used, which incorporates a feedback mechanism to monitor the robot's performance and make necessary adjustments. In examples, controller 936 may implement reactive control to directly respond to sensor inputs, enabling rapid reflexive actions and real-time changes. Furthermore, in some examples, deliberate control may be implemented, using internal models and planning algorithms to generate advanced actions, which may be suitable for complex tasks requiring reasoning, decision-making, and long-term planning.

[0243] The controller 936 may include one or more system-on-chip (SoC) 904 ( Figure 9C and Figure 9DThe controller 936 may provide signals (e.g., signals representing commands or messages) to one or more components and / or systems of the machine 900, including CPUs, GPUs, accelerators, etc. Although the controller 936 is listed separately from the SoC 904, this is not intended to be limiting, and in some embodiments, one or more components of the SoC 904 may perform the operations of the controller 936. For example, the controller may send signals to operate machine brakes via one or more brake actuators 948, to operate steering system 954 via one or more steering actuators 956, to operate propulsion system 950 via one or more throttle / accelerators 952, and so on. The controller 936 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous or semi-autonomous navigation and movement and / or assist human operators using the machine 900. Controller 936 may include a first controller 936 for autonomous control and navigation functions, a second controller 936 for functional safety functions, a third controller 936 for artificial intelligence functions (e.g., computer vision), a fourth controller 936 for infotainment functions, a fifth controller 936 for redundancy in emergency situations, and / or other controllers. For example, hardware for safety monitoring and other safety functions (e.g., functional safety islands) may be discrete or partitioned (physically or by processing separation) relative to hardware for processing sensor data to make perception and vehicle control decisions. Similarly, hardware for controlling in-vehicle infotainment and / or in-cabin monitoring (e.g., controllers, SOCs, etc.) may be separate or independent from hardware for vehicle perception and control. In some examples, a single controller 936 may handle two or more of the above functions, two or more controllers 936 may handle a single function, and / or any combination thereof.

[0244] The controller 936 may provide signals for controlling one or more components and / or systems of the machine 900 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data can be received from, for example, but not limited to, Global Navigation Satellite System (“GNSS”) sensor 958 (e.g., Global Positioning System sensor), RADAR sensor 960, ultrasonic sensor 962, LiDAR sensor 964, Inertial Measurement Unit (IMU) sensor 966 (e.g., accelerometer, gyroscope, magnetic compass, magnetometer, etc.), microphone 996, camera 968 (e.g., stereo camera 968A, wide-angle camera 968B (e.g., fisheye camera), infrared camera 968C, surround camera 968D (e.g., 360-degree camera), long-range and / or medium-range camera 968E and / or other types of cameras), speed sensor 944 (e.g., for measuring the speed of machine 900), vibration sensor 942, steering sensor 940, braking sensor (e.g., as part of braking sensor system 946), actuators and / or other types of sensors.

[0245] One or more controllers 936 may receive input (e.g., represented by input data) from the instrument panel 932 of the machine 900 and provide output (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 934 (e.g., screen, head-up display, mirror display, facial display, robot display, etc.), sound alarms, loudspeakers, speakers, and / or via other components of the machine 900. Outputs may include parameters such as machine speed, rate, time, and... Figure 9C The system may display map data corresponding to map 922 (e.g., from a navigation map, a standard definition (SD) map, a high definition (“HD”) map, etc.), location data (e.g., the location of machine 900, such as its location on map 922), direction, the location of other vehicles (e.g., occupancy map, elevation map, bird's-eye view (BEV) image, grid, etc.), information about objects perceived by the system and their status, system status information, and so on. For example, HMI display 934 may display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.), and / or information about driving maneuvers that the vehicle has performed, is performing, or will perform (e.g., changing lanes now, exiting from exit 34B in two miles, etc.).

[0246] Machine 900 may include one or more System-on-a-Chip (SoC) 904 (in Figure 9D(described in more detail below). SoC 904 may include CPU 906, GPU 908, processor 910, cache 912, accelerator 914, data storage 916, and / or other components and features. SoC 904 can be used to process and provide data for various operations, such as navigation, planning, reasoning, inference, perception, control, and / or actuation operations of machine 900 on various platforms and systems. For example, SoC 904 can process real-time perception data (e.g., from cameras, LiDAR, RADAR, ultrasound, etc.) and map data corresponding to one or more maps 922 (e.g., HD maps, SD maps, navigation maps, occupancy maps, etc.) to perform or assist in performing various operations of machine 900. When using maps and / or AI, maps and / or AI (e.g., model parameter updates, fine-tuning, etc.) are transmitted via network interface 924 from one or more servers (e.g., [server name missing]). Figure 9E The server 978 (e.g., one or more servers in a cloud-based data center) is refreshed and / or updated.

[0247] Despite Figures 9A to 9E The SoC 904 is shown, however, additional or alternative components and / or architectures, such as multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiPs), field-programmable gate arrays (FPGAs), heterogeneous integration (HI), and single-board computers (SBCs), may be used without departing from the scope of this disclosure. For example, depending on the type of machine 900, the purpose of machine 900, the model of machine 900, and the capabilities required by machine 900, one or more SoC 904s and / or alternative architectures and / or components may be used to meet a particular implementation.

[0248] Machine 900 may include CPU 918 (e.g., a discrete CPU or dCPU) which may be coupled to SoC 904 via a high-speed interconnect (e.g., PCIe). CPU 918 may include, for example, an x86 processor. CPU 918 may be used to perform any of a variety of functions, including arbitrating potentially inconsistent results between ADAS sensors and SoC 904, and / or monitoring the status and health of controller 936 and / or infotainment SoC 930.

[0249] Machine 900 may include GPU 920 (e.g., a discrete GPU or dGPU) which may be coupled to SoC 904 via a high-speed interconnect (e.g., NVIDIA's NVLink). GPU 920 may provide additional artificial intelligence capabilities, such as by executing redundant and / or different neural networks, and may be used to train and / or update neural networks based on inputs from sensors (e.g., sensor data) from Machine 900.

[0250] The machine 900 may also include a network interface 924, which may include one or more wireless antennas 926 and / or modems (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 924 can be used to establish wireless connectivity with the cloud (e.g., with server 978 and / or other network devices) via the Internet, and with other vehicles and / or computing devices (e.g., passenger client devices). For communication with other vehicles, direct links and / or indirect links (e.g., across networks and via the Internet) can be established between the two vehicles. A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide the machine 900 with information about vehicles near the machine 900 (e.g., vehicles in front, to the side, and / or behind the machine 900). This functionality can be part of the machine 900's cooperative adaptive cruise control function.

[0251] Network interface 924 may include a SoC that provides modulation and demodulation functions and enables controller 936 to communicate over a wireless network. Network interface 924 may include a radio frequency (RF) front-end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. Frequency conversion can be performed by well-known processes and / or using superheterodyne processes. In some examples, the RF front-end functionality may be provided by a separate chip. For example, network interface 924 may be able to communicate via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multicarrier (“CDMA2000”), 5G, 6G, and / or other cellular and / or wireless communication standards. The wireless antenna 926 can also enable communication between objects in the environment (such as vehicles, mobile devices, etc.) using local area networks (such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc.) and / or low power wide area networks (“LPWAN”) (such as LoRaWAN, SigFox, etc.).

[0252] Machine 900 may also include data memory 928, which may include off-chip (e.g., outside of SoC 904) storage. Data memory 928 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, Flash, hard disk and / or other components and / or devices capable of storing at least one bit of data.

[0253] The machine 900 may also include a GNSS sensor 958. The GNSS sensor 958 (e.g., a GPS, an auxiliary GPS sensor, a differential GPS (DGPS) sensor, etc.) is used to assist in mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 958 can be used, such as, but not limited to, GPS with a USB connector having an Ethernet-to-serial (RS-232) bridge.

[0254] Machine 900 may also include an IMU sensor 966. In some examples, the IMU sensor 966 may be located at the center of the rear axis of machine 900. The IMU sensor 966 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 966 may include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 966 may include an accelerometer, a gyroscope, and a magnetometer.

[0255] In some embodiments, the IMU sensor 966 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 966 can enable the machine 900 to estimate heading by directly observing and correlating velocity changes from GPS to the IMU sensor 966, without requiring input from a magnetic sensor. In some examples, the IMU sensor 966 and the GNSS sensor 958 can be combined in a single integrated unit.

[0256] The vehicle may include one or more microphones 996 placed inside and / or around the machine 900. The microphones 996 can be used for emergency vehicle detection and identification, etc.

[0257] Machine 900 may also include vibration sensors 942. Vibration sensors 942 can measure vibrations of machine parts, such as the arm or leg of humanoid robot 900C, or the axle of vehicle 900A or AMR900B. For example, changes in vibration may indicate changes in roads, walking, or traversable surfaces. In another example, when two or more vibration sensors 942 are used, differences between vibrations can be used to determine friction or slippage on surfaces (e.g., when the vibration difference is between an electrically driven shaft and a freely rotating axle).

[0258] Machine 900 may include ADAS system 938, for example, when machine 900 is vehicle 900A. In some examples, ADAS system 938 may include a dedicated SoC. ADAS system 938 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision or collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), blind spot monitoring (BSM), rear cross traffic warning (RCTW), pedestrian detection, driver monitoring, collision warning system (CWS), traffic sign recognition, speed limit detection, automatic parking, lane centering (LC), high beam safety system and / or other features and functions.

[0259] Machine 900 may also include an infotainment SoC 930 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include one or more discrete components, such as multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiP), heterogeneous integration (HI), single-board computers (SBCs), etc. The infotainment SoC 930 may include a combination of hardware and software that can be used to provide the machine 900 with audio (e.g., music, personal digital assistant, navigation commands, news, radio, etc.), video (e.g., television, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., wireless, Wi-Fi, etc.) and / or information services (e.g., navigation systems, rear parking assist, radio data systems, vehicle-related information (e.g., fuel level, total mileage, brake fluid level, fuel level, door opening / closing, air filter information, etc.). For example, an infotainment SoC 930 can be a radio, disk player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, Wi-Fi, steering wheel audio controls, hands-free voice controls, head-up display (HUD), HMI display 934, telematics device, control panel (e.g., for controlling various components, features and / or systems and / or interacting with various components, features and / or systems) and / or other components. The infotainment SoC 930 can also be used to provide information to vehicle users (e.g., visual and / or auditory), such as information from ADAS system 938, autonomous driving information (e.g., planned vehicle maneuvers, trajectory), surrounding environment information (e.g., intersection information, vehicle information, road information, etc.) and / or other information.

[0260] The infotainment SoC 930 may include GPU functionality. The infotainment SoC 930 can communicate with other devices, systems, and / or components of the machine 900 via bus 902 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 930 may be coupled to a monitoring MCU, allowing the GPU of the infotainment system to perform some autonomous driving functions in the event of a failure of the main controller 936 (e.g., the main computer and / or backup computer of the machine 900). In such an example, the infotainment SoC 930 may place the machine 900 into a driver-safe stop mode, as described herein.

[0261] In some embodiments, the infotainment system may provide a digital or virtual assistant, which may be voice-only or may have visual components (e.g., in the form of a digital human or digital avatar). The assistant may provide basic functions such as sending text messages, adjusting vehicle settings, controlling music or video, navigation, etc., and / or more advanced functions, such as those supported by one or more language models (e.g., Large Language Model (LLM), Visual Language Model (VLM), Multimodal Language Model (MMLM), etc.). For example, the driver and / or occupants may interact with the assistant in a manner similar to how a user interacts with a language model, such as asking general questions, specific questions, requesting restaurants, gas stations, and / or other recommendations and / or locations, inquiring about vehicle functions or troubleshooting (e.g., asking for tire pressure information, oil change information, battery swap information, etc.). Therefore, the machine 900 (whether it is a vehicle 900A, AMR 900B, humanoid robot 900C, or other type of machine) may include locally stored language models and / or communicate with remotely hosted language models (e.g., via one or more APIs) to provide users of the machine 900 with more detailed and in-depth communication capabilities.

[0262] In some examples, the infotainment SoC 930, SoC 904, and / or another SoC or computing / processing system can perform in-cabin driver and / or occupant monitoring. For example, the computing system can perform facial recognition, and vehicle owner recognition can use data from cameras and / or other sensors to identify the presence of an authorized driver and / or owner of the machine 900. An always-on sensor processing engine can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in a security mode, to disable the vehicle when the owner leaves. In this way, SoC 904 can provide security against theft and / or hijacking.

[0263] In some embodiments, one or more neural networks running on another or a dedicated SoC (e.g., an in-vehicle infotainment or in-vehicle monitoring SoC) can be used to monitor in-cabin monitoring camera sensors. This other or dedicated SoC is configured to recognize in-cabin events and respond accordingly. The in-cabin system can activate cellular services and make phone calls by lip-reading, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. The in-cabin system may also include one or more in-cabin AI agents or assistants that can interact with one or more LLMs, VLMs, MMLMs, etc., in the cloud using one or more APIs or plugins. For example, the in-cabin AI agents or assistants can provide directions, vehicle or machine feedback information, answer general questions, handle music / video and / or other requests, activate windows, doors, and / or other vehicle components, etc. Therefore, one or more dedicated SoCs and / or processor groups can be used to perform in-cabin infotainment and / or in-cabin monitoring (e.g., as an Occupant Monitoring System (OMS)) of Machine 900.

[0264] The device 900 may also include an instrument cluster 932 (e.g., a digital instrument cluster, electronic instrument cluster, digital dashboard, etc.). The instrument cluster 932 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 932 may include a set of instruments such as a speedometer, fuel level, fuel pressure, tachometer, odometer, turn signal indicator, gearshift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 930 and the instrument cluster 932. In other words, the instrument cluster 932 may be included as part of the infotainment SoC 930, and vice versa.

[0265] Figure 9D A computing system according to at least some embodiments of this disclosure (about Figure 9C A block diagram of an example architecture (a subset of the systems described). Although illustrated as a SoC 904, this is not intended to be limiting, and the computing system may additionally or alternatively include multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiP), heterogeneous integration (HI), single-board computers (SBCs), and / or other components and / or architectures without departing from the scope of this disclosure.

[0266] The SoC 904 can be an end-to-end platform with a flexible architecture spanning automation levels 2-5, or it can be specifically designed for a particular automation level (e.g., a first SoC 904 for levels 2 to 2++, a second SoC 904 for level 3, a third SoC 904 for level 4, and so on), providing a comprehensive functional safety architecture that leverages and effectively utilizes computer vision, neural network inference, robot planning, control and navigation, ADAS technologies, and more, with diversity and redundancy, to provide a flexible and reliable platform for driving or robot control software stacks, as well as deep learning tools. The SoC 904 can be faster, more reliable, and even more energy-efficient and space-saving than traditional systems. For example, when the accelerator 914 is used in conjunction with the CPU 906, GPU 908, and data repository 916, it can provide a fast and efficient platform for Level 2-5 autonomous vehicles, as well as for the safety planning, navigation, and control of the AMR900B, humanoid robot 900C, and / or other robot or machine types.

[0267] In some embodiments, for example, SoC 904 includes: a GPU 908 having 2000 or more cores (e.g., 2048 cores), 60 or more tensor cores (e.g., 64 tensor cores), and a maximum GPU frequency exceeding 1 GHz (e.g., 1.3 GHz); a CPU 906 including 10 or more cores (e.g., 12 cores), having 64-bit, 3 MB L2 and 6 MB L3 caches, and a maximum frequency of 2 GHz or more (e.g., 2.2 GHz); one or more deep learning accelerators (DLAs); a deep learning accelerator cluster (XNN); a neural network accelerator (NNA) or neural processing unit (NPU) 909 (e.g., two DLAs / XNNs / NNAs / NPUs 909); and a vision accelerator (e.g., a programmable vision accelerator (PVA)) 907. A single SoC 904 may be capable of achieving AI performance of 275 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGXOrin The 64GB SoC meets these standards and achieves this level of performance.

[0268] Similarly, in an embodiment, the SoC 904 includes: a GPU 908 having 1700 or more cores (e.g., 1792 cores), 50 or more tensor cores (e.g., 56 tensor cores), and a maximum GPU frequency exceeding 900MHz (e.g., 930MHz); a CPU 906 including 8 or more cores (e.g., 8 cores), having 64-bit, 2MB L2 and 4MB L3 cache memory, and a maximum frequency of 2GHz or more (e.g., 2.2GHz); one or more deep learning accelerators (DLAs); a deep learning accelerator cluster (XNN); a neural network accelerator (NNA) or neural processing unit (NPU) 909 (e.g., 2 DLAs / XNNs / NNAs / NPUs 909s); and a vision accelerator (e.g., a programmable vision accelerator (PVA) 907). A single SoC 904 may be capable of achieving AI performance of 200 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGX Orin32GB SoC meets these standards and achieves such performance.

[0269] In some embodiments, for example, SoC 904 includes: a GPU 908 having 1,000 or more cores (e.g., 1,024 cores), 28 or more tensor cores (e.g., 32 tensor cores) and a maximum GPU frequency exceeding 900 MHz (e.g., 1,173 MHz); a CPU 906 including 8 or more cores (e.g., 8 cores), having 64-bit, 2 MB L2 and 4 MB L3 cache memory, and a maximum frequency of 2 GHz or more (e.g., 2 GHz); one or more deep learning accelerators (DLAs); a deep learning accelerator cluster (XNN); a neural network accelerator (NNA) or neural processing unit (NPU) 909 (e.g., one DLA / XNN / NNA / NPU 909); and a vision accelerator (e.g., a programmable vision accelerator (PVA) 907), and a single SoC 904 may be capable of achieving AI performance of 157 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGX Orin NX 16GB SoC meets these standards and achieves such performance.

[0270] In various embodiments, for example, the SoC 904 includes: a GPU 908 having 1000 or more cores (e.g., 1024 cores), 28 or more tensor cores (e.g., 32 tensor cores), and a maximum GPU frequency exceeding 900MHz (e.g., 1020MHz); and a CPU 906 including 6 or more cores (e.g., 6 cores), with 64-bit, 1.5MB L2 and 4MB L3 cache memory, and a maximum frequency of 1.5GHz or higher (e.g., 1.7GHz). A single SoC 904 can achieve AI performance of 67 trillion operations per second (TOPS). For example, NVIDIA's Jetson Orin Nano 8GB SoC meets these standards and achieves such performance.

[0271] SoC 904 may include one or more CPUs 906. In embodiments, CPU 906 may include CPU clusters or CPU complexes (also referred to herein as “CCPLEX”). CPU 906 may include multiple cores and / or (e.g., L2, L3) caches. For example, in some embodiments, CPU 906 may include twelve cores in a coherent multiprocessor configuration. In some embodiments, CPU 906 may include four dual-core clusters, each with a dedicated L2 cache (e.g., 3MB L2 cache). CPU 906 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, thereby allowing any combination of CPU 906 clusters to be active at any given time.

[0272] The SoC 904 may include any type and number of GPUs 908. For example, an integrated GPU (also referred to herein as an "iGPU") may be used in some embodiments. The GPU 908 may be programmable and can be used efficiently for parallel workloads. In some examples, the GPU 908 may use an enhanced tensor instruction set. The GPU 908 may include one or more streaming microprocessors, wherein each streaming microprocessor may include a cache (e.g., an L1 cache with at least 96KB of storage), and two or more streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage). In some embodiments, the GPU 908 may include at least eight streaming microprocessors. The GPU 908 may use a computing application programming interface (API). Furthermore, the GPU 908 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0273] The GPU 908 can be power-optimized for optimal performance in automotive, robotics, and / or other embedded use cases. For example, the GPU 908 can be fabricated on FinFETs. However, this is not intended to limit it, and the GPU 908 can be fabricated using other semiconductor fabrication methods or processes. Each streaming microprocessor can contain multiple mixed-precision processing cores divided into multiple blocks. For example, but not limited to, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSOR COREs for deep learning matrix algorithms, one (e.g., L0) instruction cache, one warp scheduler, one dispatch unit, and / or one (e.g., 64KB) register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to efficiently perform workloads involving mixed-computation and addressing computations. Streaming microprocessors can include independent thread scheduling capabilities to enable finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors can also include combined L1 data caches and shared memory units to improve performance while simplifying programming.

[0274] The GPU 908 may include a high-bandwidth memory (HBM) and / or (e.g., 16GB) HBM2 memory subsystem to provide peak memory bandwidth of approximately 900GB / s in some examples. In some examples, synchronous graphics random access memory (SGRAM), such as graphics double data rate type 5 synchronous random access memory (GDDR5), may be used in addition to or as an alternative to HBM memory.

[0275] The GPU 908 may include unified memory technology, including access counters, to allow more accurate migration of memory pages to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, Address Translation Service (ATS) support may be used to allow the GPU 908 to directly access the CPU 906 page tables. In such examples, when a miss occurs in the GPU 908 Memory Management Unit (MMU), an address translation request can be transferred to the CPU 906. In response, the CPU 906 can look up the virtual-to-physical mapping of the address in its page tables and transfer the translation back to the GPU 908. Therefore, unified memory technology allows the memory of the CPU 906 and GPU 908 to use a single, unified virtual address space, simplifying GPU 908 programming and porting applications to the GPU 908.

[0276] The SoC 904 may include any number of caches 912, including the caches described herein. For example, cache 912 may include L0 cache, L1 cache, L2 cache, L3 cache (e.g., available for CPU 906 and GPU 908 (e.g., connected to CPU 906 and GPU 908)), etc. Cache 912 may include write-back caches that can track the state of the lines, for example, by using one or more cache coherence protocols (e.g., MEI, MESI, MSI, etc.). According to embodiments, the cache (e.g., L3) may include 4MB or more, although smaller or larger cache sizes may be used.

[0277] SoC 904 may include one or more arithmetic logic units (ALUs) 965, which can be used to perform processing related to various tasks or operations of machine 900, such as computer vision, machine learning or deep learning processing, world model management, etc. Furthermore, SoC 904 may include a floating-point unit (FPU) 967 or other mathematical coprocessors or digital coprocessors for performing mathematical operations within the system. For example, SoC 104 may include one or more FPUs 967, which are integrated as execution units within CPU 906 and / or GPU 908.

[0278] The SoC 904 may include one or more accelerators 914 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC 904 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. Large on-chip memory 915 (e.g., 4MB SRAM, 32GB and / or 64GB 256-bit LPDDR5 (204.8GB / s), 8GB and / or 16GB 128-bit LPDDR5 (102.4GB / s), and / or other memory types and sizes) enables the hardware acceleration cluster to accelerate neural network processing, converter processing, optical flow processing, vision processing, and / or other computations or processing. The hardware acceleration cluster can be used to supplement the GPU 908 and offload some tasks from the GPU 908 (e.g., freeing up more cycles of the GPU 908 to perform other tasks). As an example, the Accelerator 914 can be used for target workloads (e.g., perceptual, convolutional neural network (CNN), deep neural network (DNN), language model (LLM, VLM, MMLM, VLA, etc.), transformer model, diffusion model, encoder-only model, encoder-decoder model, etc., which are stable enough to be suitable for acceleration.

[0279] Accelerator 914 (e.g., a hardware acceleration cluster) may include a Deep Learning Accelerator (DLA) 909 (also referred to herein as a “Deep Learning Accelerator Cluster (XNN) 909”, a “Neural Network Accelerator (NNA) 909”, or a “Neural Processing Unit (NPU) 909”). DLA 909 may include one or more Tensor Processing Units (TPUs) 941, which may be configured to provide additional, for example, trillions of operations per second for deep learning applications and inference. TPU 941 may be an accelerator configured to perform image processing functions (e.g., for CNNs, RCNNs, DNNs, etc.) and optimized for them. DLA 909 may be further optimized for a specific set of neural network types and floating-point operations and inference. The DLA is designed to provide higher performance per millimeter than general-purpose GPUs and significantly outperforms CPUs. TPU 941 may perform several functions, including single-instance convolution functions, support for INT8, INT16, and FP16 data types for features and weights, and post-processor functions. Although TPU 941 is described as being included as part of DLA 909, this is not intended to be limiting, and TPU 941 may be included in additional or alternative accelerator 914 and / or other components, and / or may be included as a discrete processing component.

[0280] The DLA 909 can quickly and efficiently execute neural networks on processed or unprocessed data to achieve a variety of functions, including but not limited to: object and feature recognition and detection using data from one or more sensor modalities (e.g., vehicles, pedestrians, other robots, lane lines, road boundary lines, debris, potholes, boxes, warehouse items, etc.); distance estimation using data from one or more sensor modalities; emergency vehicle detection and identification using data from microphones and / or vision-based sensors; facial recognition; pick-up and place operations; maneuvering operations; occupant monitoring; vehicle owner identification; and / or other in-cabin operations using data from in-cabin cameras and / or other sensor types; and / or safety and / or safety-related events, to name a few.

[0281] The DLA 909 can perform any function of the GPU 908, and by using inference accelerators, for example, designers can anchor either the DLA 909 or the GPU 908 for any function. For instance, designers can centralize the processing of DNNs and floating-point operations on the DLA 909, leaving other functions to the GPU 908 and / or other accelerators 914. The DLA 909 can be used to run any type of network to enhance control and security, including neural networks that, for example, output a confidence metric for each object detection.

[0282] Accelerator 914 (e.g., a hardware acceleration cluster) may include a programmable vision accelerator (PVA) 907, which may alternatively be referred to herein as a computer vision accelerator or generally as a vision accelerator. The PVA 907 may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), semi-autonomous driving, autonomous driving, robotics applications, safety and supervision applications, augmented reality (AR), virtual reality (VR), and / or mixed reality (MR) applications. The PVA 907 provides a balance between performance and flexibility. For example, each PVA 907 may include (e.g., but not limited to) any number of Reduced Instruction Set Computer (RISC) cores, Direct Memory Access (DMA) systems, Pixel Processing Engines (PPEs), Vector Processors or Vector Processing Units (VPUs), and / or other components. The PVA engine may include an Advanced Very Long Instruction Word (VLIW) or Single Instruction Multiple Data (SIMD) digital signal processor. The PVA 907 may be optimized for image processing and computer vision algorithm acceleration tasks. For example, the PVA907 offers excellent performance with extremely low power consumption and can be used asynchronously and concurrently as part of a heterogeneous computing pipeline with CPU 906, GPU 908 and / or other accelerators in a system (e.g., vehicles, robots, etc.).

[0283] The PVA 907 may include one or more (e.g., two) Vector Processing Subsystems (VPSs), each of which may include one or more Vector Processing Unit (VPU) cores, one or more Decoupled Lookup Units (DLUTs), one or more shared or vector memory (VMEMs), and one or more instruction caches (I-caches). The VPU core may be the main processing unit and may include a vector SIMD VLIW DSP 943 optimized for computer vision. The VPU core can fetch instructions via the I-cache and access data via the VMEM. The DLUT may include dedicated hardware components that enhance the efficiency of parallel lookup operations. For example, the DLUT allows parallel lookups using a single copy of the lookup table by performing these lookups in a decoupled pipeline independent of the main processor pipeline. By doing so, the DLUT can minimize or reduce memory usage and increase throughput while avoiding data-related memory library conflicts, ultimately leading to improved overall system performance. The VPU VMEM can provide local data storage for the VPU, allowing for the efficient implementation of various image processing and computer vision algorithms. The VPU VMEM can support access from external VPS hosts, such as Direct Memory Access (DMA) and the CPU 906 (e.g., an ARM Cortex-R5 processor), thereby facilitating data exchange with the CPU 906 and other system-level components. The VPU I-cache can provide instruction data to the VPU on request, request missing instruction data from system memory, and / or maintain temporary instruction storage for the VPU. For each VPU task, the CPU 906 can configure the DMA system, optionally prefetching the VPU program into the VPU I-cache, and / or initiating each VPU-DMA pair to process the task. The PVA 907 may also include L2 SRAM memory shared between one or more (e.g., two) sets of VPS and DMA. In some embodiments, one or more (e.g., two) DMA devices are used to move data between external memory, PVA L2 memory, VMEM (e.g., one per VPS), CPU tightly coupled memory (TCM), DMA descriptor memory, and / or PVA-level configuration registers. In lightly loaded systems, two parallel DMA accesses to DRAM can achieve read / write bandwidth of up to 15 GB / s, while in heavily loaded systems, this bandwidth can reach up to 10 GB / s. Regarding compute capacity, INT8 gigabyte multiply-accumulate operations per second (GMAC) can be 2048 or greater, excluding DLUT. FP32 GMACs can be 32 per PVA instance.

[0284] The RISC core can interact with image sensors (e.g., the image sensor of any camera described herein), image signal processors, etc. Each RISC core may include any amount of memory. The RISC core can use any of a variety of protocols, depending on the implementation. In some examples, the RISC core can execute a real-time operating system (RTOS). The RISC core can be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, the RISC core may include an instruction cache and / or tightly coupled RAM.

[0285] The DMA system enables PVA 907 components to access system memory independently of the CPU 906. DMA can support any number of functions optimized for the PVA 907, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which may include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0286] A vector processor, or VPU, can be a programmable processor designed to efficiently and flexibly execute computer vision algorithms and provide signal processing capabilities. In some examples, the PVA907 may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA 907 and may include one or more vector processing units (VPUs), one or more pixel processing engines (PPEs) (which may include a 2D layout of interconnected (e.g., for north, south, east, and west communication) processing elements), one or more instruction caches, and / or one or more shared or vector memories (e.g., VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can improve throughput and speed.

[0287] In some embodiments, each vector processor may include an instruction cache and may be coupled to dedicated memory. Therefore, in some examples, each vector processor may be configured to execute independently of other vector processors. In other examples, the vector processors included in a particular PVA 907 may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA 907 may execute the same computer vision algorithm, but for different regions of an image. In other examples, the vector processors included in a particular PVA 907 may execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on consecutive images or portions of an image. Among other things, the hardware acceleration cluster may include any number of PVA 907s, and each PVA may include any number of vector processors. Furthermore, the PVA 907 may include additional error correction code (ECC) memory to enhance overall system security.

[0288] Accelerators 914 (e.g., hardware accelerator clusters) have broad applications in autonomous and semi-autonomous machine control. PVA 907 can be a programmable vision accelerator used in critical processing stages in perception, robot understanding and reasoning, ADAS, semi-autonomous and autonomous vehicles, etc. The PVA 907's capabilities are well-suited for algorithmic domains requiring predictable processing, featuring low power consumption and low latency. In other words, the PVA 907 performs well in semi-intensive or intensive rule computation, even on small datasets requiring predictable runtime, low latency, and low power consumption. Therefore, in the context of autonomous vehicles and robotic platforms, the PVA 907 is designed to run classic computer vision algorithms, as they are highly efficient in object detection and integer mathematical operations.

[0289] For example, according to one embodiment of this technology, the PVA 907 is used to perform computer stereo vision. In some examples, a semi-global matching-based algorithm can be used, although this is not intended to limit it. Many Level 3-5 autonomous driving applications require on-the-fly motion estimation / stereo matching (e.g., moving structures, pedestrian recognition, lane detection, etc.). The PVA 907 can perform computer stereo vision functions on input from two monocular cameras.

[0290] In some examples, the PVA 907 can be used to perform dense optical flow, providing processed RADAR data based on the raw RADAR data (e.g., using 4D Fast Fourier Transform). In other examples, the PVA 907 is used for time-of-flight depth processing, for example, providing processed time-of-flight data by processing the raw time-of-flight data.

[0291] While the VPU, DMA, RISC Core, VMEM, and decoupled coprocessors (e.g., DLUT) are described as being included within the PVA907, this does not imply limitation. In some embodiments, these components may be included in alternative or additional processing components and / or accelerator 914, and / or may be included as discrete components of the SoC 904 and / or other computing system architectures.

[0292] In some examples, SoC 904 may include a real-time ray tracing hardware accelerator (RTA) 951, which can be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) to generate real-time or near-real-time visualization simulations, for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for simulation of SONAR, RADAR, LiDAR, camera and / or other sensor modalities in a simulation, for general wave propagation simulation, for comparison with LiDAR data for localization, for generating realistic training data for training neural networks, and / or other functions and uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations. For example, machine 900 (or another machine or device) may perform simulations in a simulation environment, and one or more light transport simulation algorithms (e.g., ray tracing, path tracing, etc.) may be used to generate the simulation environment. Therefore, the ray tracing accelerator 951 and / or a ray tracing-optimized GPU 906 (e.g., NVIDIA's RTX GPU) may be used to accelerate these ray tracing algorithms.

[0293] Accelerator 914 (e.g., in a hardware acceleration cluster) may include one or more optical flow accelerators (OFAs) 911. For example, OFA 911 can be used to calculate optical flow and stereo disparity between sensor data frames (e.g., images). Optical flow can be accelerated on OFA 911 for purposes such as object detection and tracking, and / or for stereo depth estimation, where stereo disparity is calculated between stereo image frames (e.g., two or more frames captured using two or more image sensors with at least partially overlapping fields of view).

[0294] SoC 904 may include one or more Camera Serial Interfaces (CSI) 923s. For example, CSI 923 may include a Mobile Industry Processor Interface (MIPI) Camera Serial Interface (CSI) for receiving video and input from cameras, a high-speed interface, and / or a video input block available for camera and associated pixel input functions. SoC 904 may also include a software-controllable input / output controller and may be used to receive I / O signals not assigned to a specific role. For example, CSI 923 may include MIPI CSI-2 connectors, such as a 16-channel MIPI CSI-2 connector, D-PHY 2.1 (up to 40Gbps), and C-PHY 2.0 (up to 164Gbps) to support 16 virtual channels and 6 or more cameras; an 8-channel MIPI CSI-2 connector, D-PHY 2.1 (up to 20Gbps) to support 8 virtual channels and 4 or more cameras; and / or 2x MIPI CSI-2, 22-pin camera connectors, depending on the embodiment and implementation.

[0295] Accelerator 914 (e.g., a hardware acceleration cluster) may include an on-chip computer vision network (CVNOC) 963 and SRAM for providing high-bandwidth, low-latency SRAM to accelerator 914. In some examples, on-chip memory may include at least 4 MB of SRAM, such as, but not limited to, eight field-configurable memory blocks accessible by PVA 907, OFA 911, DLA 909, and / or other accelerator 914. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory 915 may be used. PVA 907, OFA 911, DLA 909, and / or other accelerator 914 may access memory via a backbone providing high-speed memory access to accelerator 914. The backbone may include an on-chip computer vision network that interconnects accelerator 914 to memory (e.g., using APB).

[0296] CVNOC 963 may include an interface that determines whether the accelerator 914 provides ready and valid signals before transmitting any control signals / addresses / data. Such an interface can provide separate stages and separate channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, although other standards and protocols can be used.

[0297] SoC 904 may include a data repository 916 and / or memory 915. The data repository 916 may be on-chip memory 915 of the SoC 904, which may store neural networks and / or other algorithms to be executed on the CPU 906, GPU 908, and / or one or more accelerators 914. In some examples, the capacity of the data repository 916 may be large enough to store multiple neural network instances for redundancy and security. The data repository 916 may include, for example, L2 and / or L3 cache 912. The memory 915 may include SRAM, LPDDR5, and / or other memory types. For example, the memory 915 may include 4MB SRAM, 32GB and / or 64GB 256-bit LPDDR5 (204.8GB / s), 8GB and / or 16GB 128-bit LPDDR5 (102.4GB / s), and / or other memory types and sizes. References to the data repository 916 may include references to the memory associated with the PVA 907, OFA 911, DLA 909 and / or other accelerators 914, as described herein.

[0298] Data repository 916 can include various storage types, such as eMMC, NVMe, etc. For example, SoC 904 can include storage in the form of an embedded multimedia card (eMMC) (e.g., 64GB eMMC5.1) and / or an SD card slot, with external NVMExpress (NVMe) capability, for example, via M.2 Key M. For example, data repository 916 and / or other storage can be accessed via, for example, NVMe using PCI Express (PCIe), RDMA, TCP, and / or other protocols.

[0299] SoC 904 may include one or more processors 910 (e.g., embedded processors). Processor 910 may include a boot and power management processor (BPMP) 953, which may be a dedicated processor and subsystem for handling boot power and management functions, as well as associated safety implementations. BPMP 953 may be part of the SoC 904 boot sequence and may provide runtime power management services. BPMP 953 may provide clock and voltage programming, assistance with system low-power state transitions, management of SoC 904 thermal and temperature sensors, and / or management of SoC 904 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and SoC 904 may use the ring oscillator to detect the temperature of CPU 906, GPU 908, accelerator 914, and / or other components. If the temperature is determined to exceed a threshold, BPMP 953 may enter a temperature fault routine and place SoC 904 into a lower power state and / or place machine 900 into a driver safety stop mode (e.g., safely stop machine 900).

[0300] The processor 910 may also include a set of embedded processors that can be used as the Audio Processing Engine (APE) 955. The APE 955 can be an audio subsystem capable of providing full hardware support for multi-channel audio through multiple interfaces, as well as a wide and flexible audio I / O interface. In some examples, the APE 955 is a dedicated processor core with a digital signal processor and dedicated RAM.

[0301] The processor 910 may also include an Always-On Processor Engine (AOPE) 957, which provides the necessary hardware features to support low-power sensor management and wake-up use cases. AOPE 957 may include a processor core, tightly coupled RAM, peripheral support (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0302] Processor 910 may also include a security processor 913 (or “security island 913”), which may include a security cluster engine comprising a dedicated processor or processor subsystem for handling security management for automotive, robotic, and / or other applications. Security processor 913 and / or security cluster engine may include two or more processor cores, tightly coupled RAM, support for peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In security mode, the two or more cores may operate in lockstep mode and as a single core, with comparison logic to detect any differences in their operation. In some embodiments, security processor 913 may include discrete processors such that failure of other system components may not affect the performance and availability of security processor 913.

[0303] The processor 910 may also include a real-time or near-real-time sensor engine (SE) 959, which may include a dedicated processor subsystem for handling real-time or near-real-time camera, LiDAR, RADAR and / or other sensor modal management.

[0304] The processor 910 may also include one or more image signal processors (ISPs) 927, which may include high dynamic range signal processors and / or hardware engines as part of one or more sensor processing pipelines.

[0305] Processor 910 may include a video image synthesizer (VIC) 961, which may be a processing block (e.g., implemented on a microprocessor) that implements the video post-processing functions required by the video playback application to produce the final image of the player window. VIC 961 can perform lens distortion correction on wide-angle camera 968B, surround camera 968D, cabin monitoring camera sensor and / or other camera sensors with distorted field of view.

[0306] VIC 961 may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, when motion occurs in the video, noise reduction appropriately weights spatial information, thereby reducing the weight of information provided by adjacent frames. When an image or part of an image does not contain motion, temporal noise reduction performed by the video image synthesizer can use information from previous images to reduce noise in the current image.

[0307] The VIC 961 can also be configured to perform stereoscopic correction on input stereoscopic camera frames. The video image compositor can also be used for user interface compositing when the operating system desktop is in use and the GPU 908 does not need to continuously render new surfaces. Even when the GPU 908 is powered on and actively performing 3D rendering, the video image compositor can be used to offload the GPU 908 to improve performance and responsiveness.

[0308] The SoC 904 may also include various peripheral interfaces for input / output (I / O) 925, for example, to enable communication with peripheral devices, audio codecs, power management and / or other devices. The SoC 904 can be used to process data from cameras (e.g., via a gigabit multimedia serial link and / or Ethernet connection), from sensors (e.g., LiDAR sensor 964, RADAR sensor 960, etc., connected via Ethernet), from bus 902 (e.g., speed of machine 900, steering wheel position, etc.), and from GNSS sensor 958 (e.g., connected via Ethernet or CAN bus). The SoC 904 may also include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine and can be used to free the CPU 906 from routine data management tasks. In some embodiments, the SoC 904I / O 925 may include headers (e.g., 40-pin headers or 40-pin extension headers) supporting Universal Asynchronous Receiver / Transmitter (UART), Serial Peripheral Interface (SPI), Inter-Integrated Sound (I2S), Inter-Integrated Sound (I2C), Controller Area Network (CAN), Pulse Width Modulation (PWM), Digital Microphone Interface (DMIC), Digital Speaker Station (DSPK), General Purpose I / O (GPIO), etc.; automation headers (e.g., 12-pin automation headers); audio panel headers (e.g., 10-pin audio panel headers); Joint Test Action Group (JTAG) headers (e.g., 10-pin JTAG headers); fan headers (e.g., 4-pin fan headers); RTC battery spare connectors (e.g., 2-pin battery spare connectors); a microSD slot; a DC power jack; power, force, resume, and reset buttons; and one or more display connectors (e.g., DisplayPort (DP), such as DP 1.4A(+MST), eDP 1.41, HDMI). 2.1 and / or 4K30 multi-model DP1.2 (+MST) connectors) and / or other I / O 925 elements, components or features.

[0309] The SoC 904 may include machine intranetting capabilities using, for example, Ethernet (e.g., automotive Ethernet), SERDES, Controller Area Network (CAN), FlexRay, Local Interconnect Network (LIN), Low Voltage Differential Signaling (LVDS), Media-Oriented System Transport (MOST), another network type, and / or combinations thereof. For example, the SoC 904 may include RJ45 connectors with up to 10GbE, 1GbE connectors, and / or other network connector types.

[0310] So104 may include one or more digital signal processors (DSPs) 943. For example, the DSP 943 may include a dedicated or specialized microprocessor chip optimized for digital signal processing, such as audio signal processing, telecommunications, digital image processing, RADAR, SONAR, LiDAR and / or other sensor processing, speech recognition and / or other applications.

[0311] SoC 904 may include one or more video encoders 919 and / or one or more video decoders 921. For example, video encoder 919 may include a hardware-based (e.g., as part of GPU 908) video encoder (e.g., supporting H.264, H.265, etc., and conforming to HEVC standards, such as NVIDIA's NVENC) that can process image input (e.g., as YUV, RGB, etc.) to generate a video bitstream. Video decoder 921 may include a video decoder engine that provides fully accelerated hardware video decoding capabilities (e.g., supporting decoding of various bitstream formats such as AV1, H.264, H.265, VP8, VP9, ​​MPEG-1, MPEG-2, MPEG-4, VC-1, etc., and conforming to HEVC standards, such as NVIDIA's NVDEC). In some examples, video decoder 921 may be hardware-based (e.g., as part of GPU 908).

[0312] The SoC 904 may include one or more General Purpose Computing Acceleration Clusters (GCACs) 929s. For example, a GCAC 929 may include various processor types that can be used to accelerate computing, such as one or more Vector Microcode Processors (VMPs) 933s, one or more Multi-Threaded Processing Clusters (MPCs) 931s, one or more Programmable Macroarrays (PMAs) 935s, and / or one or more other processor types. For example, a GCAC 929 may include a PMA 935, two VMPs 933s, and two MPCs 931s.

[0313] The SoC 904 may include one or more vector microcode processors (VMPs) 933. In an embodiment, the VMP 933 may include a wide vector (Very Long Instruction Word (VLIW) and Single Instruction Multiple Data (SIMD)) machine that performs a variety of operations, such as short integral-type operations common in computer vision and deep learning algorithms.

[0314] SoC 904 may include one or more multi-threaded processing clusters (MPCs) 931. MPCs 931 may include processing clusters that are more general-purpose than GPUs and more efficient than CPUs in some embodiments. For example, an MPC 931 may include a multi-threaded processor that allows multiple threads to share resources and execute instructions concurrently.

[0315] The SoC 904 may include one or more programmable macro arrays (PMAs) 935. The PMA 935 may include a coarse-grained reconfigurable architecture (CGRA) dataflow machine, which has a unique architecture that delivers powerful performance on intensive computer vision and deep learning algorithms that may not be achievable in classic digital signal processing (DSP) architectures.

[0316] The SoC 904 may include one or more Display Processing Units (DPUs) 945 for performing hardware-accelerated image processing. For example, the DPU 945 may retrieve pixel data from memory 915 and send it to display peripherals via a standard interface. Therefore, the DPU 945 can handle display processing and rendering for displays within and / or on the machine.

[0317] The SoC 904 may include one or more Application Processing Units (APUs) 939. For example, an APU 939 may include a quad-core or dual-core processor with 48KB / 32KB L1 cache (with parity and ECC) and a 1MB L2 cache with ECC. The APU 939 may support NEON instructions and single-precision and double-precision floating-point operations.

[0318] The SoC 904 may include one or more Real-Time Processing Units (RTPUs) 969. The RTPU 969 may include a dual-core processor with 32KB / 32KB L1 cache and 256KBTCM with ECC. The RTPU 969 can support single-precision and double-precision floating-point operations.

[0319] SoC 904 may include one or more built-in self-test (BIST) components 937. For example, BIST component 937 may include a memory BIST (MBIST) for testing the system's memory and / or a logic BIST (LBIST) for testing the system's logic. BIST component 937 may include embedded logic for directly testing the system's logic and / or memory.

[0320] SoC 904 may include one or more dynamically reconfigurable processors (DRPs) 971. For example, DRPs 971 can be used to accelerate various computational operations. For instance, in one embodiment, a DRP 971 may be combined with a MAC unit to function as an AI accelerator. In another embodiment, a DRP 971 may execute an application while dynamically switching the circuit connection configuration of the arithmetic unit (e.g., ALU) on the chip each operating clock, depending on what needs to be processed. By using only the necessary arithmetic circuitry, the DRP 971 can consume less power than a CPU and achieve higher speeds. Furthermore, compared to a CPU, which suffers from performance degradation due to frequent accesses to external memory caused by cache misses and other reasons, the DRP 971 can pre-build the necessary data paths in the hardware, thereby reducing performance degradation and operating speed variations (jitter) caused by memory accesses. The DRP 971 may include a dynamic loading function that switches circuit connection information each time the algorithm changes, enabling processing with limited hardware resources, even in robotics / automotive applications that require processing multiple algorithms.

[0321] In some embodiments, accelerator 914 may include an OpenCV accelerator for accelerating OpenCV processing, OpenCV being an open-source industry-standard library for computer vision processing. In some embodiments, the combination of one or more DRP 971 accelerators deployed as AI accelerators with OpenCV accelerators can enhance AI computation and image processing algorithms to enable complex and computationally intensive operations such as visual simultaneous localization and mapping (SLAM).

[0322] Compared to traditional systems, the techniques described herein, by providing CPU complexes, GPU complexes, and hardware acceleration clusters, allow multiple neural networks to be executed simultaneously (e.g., at least partially in parallel) and / or sequentially, and the results combined to achieve Level 2-5 autonomous driving capabilities and / or autonomous robot motion, control, planning, and / or navigation operations. Furthermore, since the SoC 904 can include various computing engines (e.g., processor 910, CPU 906, GPU 908, accelerator 914, etc.), tasks can be distributed among the computing engines, and in some cases, common-cause failures are avoided due to the discrete footprint of the computing engines. Additionally, since the SoC 904 can include a dedicated safety processor 913 (or safety island 913), critical safety or redundant operations can be performed without the common-cause failures of the SoC 914's main processing components or computing engines. Due to these features, the underlying system of the SoC 904 and / or machine 900 may be able to meet higher safety levels—such as the Automotive Safety Integrity Level (ASIL) D of the ISO 26262 standard.

[0323] Figure 9E According to some embodiments of this disclosure, cloud-based servers (e.g., servers such as those described herein in a data center) and Figure 9A The following is a system diagram illustrating communication between an example autonomous or semi-autonomous vehicle or machine 900. System 976 may include server 978, network 990, and machine 900. Server 978 may include multiple GPUs 984(A)-984(H) (collectively referred to herein as GPU 984), switches 982(A)-982(H) (e.g., PCIe 4.0 / 5.0 switches, M.2 slots, Thunderbolt, USB4, NVIDIA's NVLink, NVIDIA's NVSwitch, GPUDirect RDMA, GPUDirect Storage, etc.), CPUs 980(A)-980(B) (collectively referred to herein as CPU 980), accelerators, and / or other processor types. GPUs 984, CPUs 980, and PCIe switches may interconnect with high-speed interconnects, such as, but not limited to, NVIDIA-developed NVLink interface 988 and / or PCIe connection 986. In some examples, the GPU 984 is connected via NVLink and / or NVSwitch SoC, and the GPU 984 and PCIe switch 982 are connected via PCIe interconnect. While the figure shows eight GPU 984s, two CPU 980s, and two PCIe switches, this is not limiting. According to embodiments, each server 978 may include any number of GPU 984s, CPU 980s, and / or PCIe switches. For example, each server 978 may include eight, sixteen, thirty-two, and / or more GPU 984s.

[0324] Server 978 may receive sensor data from network 990 and machines 900 that indicates information about new or previously unexplored locations, and / or sensor data indicating changes to previously seen / stored locations (e.g., unexpected or changed road conditions, such as recently started road construction). Server 978 may transmit neural network 992, updated neural network 992, map information 994, etc., including information about traffic and road conditions, to machines 900 via network 990. Updates to map information 994 may include updates to HD maps 922, SD maps, navigation maps, etc., such as information about construction sites, potholes, detours, floods, and / or other obstacles. In some examples, neural network 992, updated neural network 992, map information 994, and / or other information may come from new training and / or experience, represented in data received from any number of machines 900 in the environment, and / or based on training performed in a data center (e.g., using server 978 and / or other servers).

[0325] Server 978 can be used to train machine learning models (e.g., neural networks) based on training data. Training data can be generated by machine 900 and / or generated in simulations (e.g., using a game engine). In some examples, the training data is labeled (e.g., neural networks benefit from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is unlabeled and / or unprocessed (e.g., neural networks do not require supervised learning). Training can be performed using any one or more machine learning techniques, including but not limited to: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component analysis and clustering analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, it can be used by machine 900 (e.g., transmitted to machine 900 via network 990), and / or the machine learning model can be used by server 978 to remotely monitor and / or control machine 900.

[0326] In some examples, server 978 can receive data from machine 900 and apply the data to a state-of-the-art real-time neural network for real-time intelligent inference. Server 978 may include a deep learning supercomputer powered by GPU 984 and / or a dedicated AI computer, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, server 978 may include a deep learning infrastructure in a data center that uses only CPU power.

[0327] The deep learning infrastructure of server 978 may be capable of rapid, real-time inference and can use this capability to assess and verify the health of the processor, software, and / or related hardware in machine 900. For example, the deep learning infrastructure may receive periodic updates from machine 900, such as image sequences and / or objects located by machine 900 in the image sequence (e.g., through computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them with the objects identified by machine 900. If the results do not match and the infrastructure concludes that the AI ​​in machine 900 has malfunctioned, server 978 can send a signal to machine 900 instructing its fail-safe computer to take over control, notify the occupants, and perform safety maneuvers or operations, such as slowing down, returning control to the driver, stopping, and / or pulling over / closing the circuit.

[0328] For inference, the server 978 can include a GPU 984 and one or more programmable inference accelerators (such as NVIDIA's TensorRT). The combination of GPU-driven servers and inference acceleration enables real-time responses. In other examples, such as in scenarios where performance is less critical, servers powered by CPUs, FPGAs, and other processors can be used for inference. The computing ecosystem for generating, training, and deploying AI.

[0329] Figure 10 This is a system diagram illustrating a three-computer ecosystem 1000 according to at least some embodiments of the present disclosure, including a first computing system 1002 for generating or creating artificial intelligence (AI) (e.g., AI training and validation data), a second computing system 1004 for training the AI, and a third computing system 1006 for deploying AI at the edge (which may include or correspond to...). Figures 9A to 9E (The SoC 904). For example, to develop and deploy materialized or physical AI, a three-computer ecosystem 1000 can be used, comprising three accelerated computer systems to handle physical AI training, simulation, and runtime (e.g., edge deployment). These systems can use scalable, physical-based simulations of the Machine 900 and its world to generate training data and train multimodal base models (and / or other model types). By doing so, simulations of the Machine 900 can be performed at scale, allowing skills (e.g., robotic skills) to be improved, tested, and optimized in virtual worlds that simulate the laws of physics (e.g., using NVIDIA's OMNIVERSE), thereby helping to reduce the cost of real-world data acquisition and ensuring that the Machine 900 can operate safely in a controlled environment.

[0330] Computing system 1004 (e.g., NVIDIA's DGX platform) can be used to train and fine-tune powerful foundational and generative AI models. Models such as general foundational models (e.g., NVIDIA's Project GR00T) can be used to enable robots and other machines 900 to understand natural language and mimic actions by observing human movements. Computing system 1004 may include a platform that integrates software, infrastructure, and expertise into a modern, unified AI development and training solution. Computing system 1004 may include individual computing devices 1010 (e.g., NVIDIA's DGX B200, H200, etc.) and / or any number of computing devices 1010 (e.g., NVIDIA's DGX SuperPOD) within data center infrastructure 1012.

[0331] For example, a standalone computing device 1010 may include GPUs (e.g., 8 GPUs with a total GPU memory of 1,440 GB) and CPUs (e.g., 2 CPUs with a total of 112 cores, 2.1 GHz or 4 GHz (with enhancements)), providing up to 72 petaFLOPS of training capability and 144 petaFLOPS of inference capability. The computing device 1010 may include memory (e.g., 4 TB of memory) and storage (e.g., 2 x 1.9 TB NVMe M.2 OS storage and 8 x 3.84 TB NVMe U.2 internal storage). The computing device 1010 may include various networking and network management components, such as OSFP ports (e.g., 4 OSFP ports) for servicing single-port intelligent host channel adapters (e.g., 8 single-port ConnextX-7 Virtual Protocol Interconnect (VPI)), providing up to 400 GB / s of Infiniband / Ethernet. The computing device 1010 may also include, for example, a dual-port quad small pluggable (QSFFP) data processing unit (DPU) (e.g., two dual-port QSFP112 DPUs – such as NVIDIA’s BlueField-3DPU) providing up to 400Gb / s of InfiniBand / Ethernet. The computing device 1010 may include an onboard network interface card (NIC) (e.g., a 10Gb / s onboard NIC with RJ45), a dual-port Ethernet NIC (e.g., a 100GB / s dual-port Ethernet NIC), and / or a host board management controller (MBC) (e.g., with RJ45). In some embodiments, the NIC for the computing device 1010 may include a SuperNIC (e.g., NVIDIA’s ConnectX-8 SuperNIC) to provide up to 800Gb / s of data throughput for in-network computing acceleration engines, thereby providing the performance and robust feature set required to support trillion-parameter-scale AI factories and scientific computing workloads. In other embodiments, computing device 1010 may include a smart host channel adapter (HCA) (e.g., NVIDIA's ConnectX-7) to provide ultra-low latency and 400Gb / s throughput for computing acceleration engines within the network.

[0332] Data center infrastructure 1012 may include: any number of computing devices 1010, and an operating system (OS) (e.g., a DGX OS extension of a Linux distribution) for maximizing system uptime, security, and reliability; network / storage acceleration libraries and management for accelerating end-to-end infrastructure performance; cluster management for scaling and managing a single node (e.g., a computing device 1010) to thousands of nodes; job scheduling and orchestration to ensure the smooth execution of each developer's jobs; AI workflow management and machine learning operations (MLOps) for moving more models from prototypes to production; and enterprise software to accelerate developer success.

[0333] Computational system 1002 (e.g., NVIDIA's OVX server) can provide a development and simulation platform for testing and optimizing physics-based AI using APIs and frameworks for simulation (e.g., NVIDIA's DriveSIM, ISAAC Sim, ISAAC Gym, ISAAC Lab, etc.). Computational system 1002 allows developers to use simulation frameworks to simulate and validate robot models, and / or generate large amounts of physics-based synthetic data to guide model training. Computational system 1002 can support learning frameworks that power robot reinforcement learning and imitation learning to accelerate robot policy training and refinement. For example, computational system 1002 can be used to generate any number of simulations 1008, such as in NVIDIA's OMNIVERSE. The Computing System 1002 can be used to optimize and accelerate the entire software stack, from training, fine-tuning, and deploying generative AI to supporting industrial digitization within content collaboration platforms (including APIs, software development kits (SDKs), and services). These platforms allow the integration of OpenUSD, ray tracing rendering technologies (such as NVIDIA's RTX), and generative physics AI into existing software tools and simulation workflows, such as those for industrial and robotics use cases (such as NVIDIA's OMNIVERSE). Therefore, the Computing System 1002 can host or support native OpenUSD software platforms, enabling enterprises to connect 3D pipelines and develop advanced real-time 3D applications for industrial digitization. With powerful ray tracing-accelerated AI and graphics capabilities, the Computing System 1002 delivers robust performance for workloads such as extended reality (XR), multi-user design collaboration, and digital twins. This allows for the creation of physically accurate models with high-fidelity ray tracing and path-tracing material rendering, large-scale, AI-enabled simulation operations, and the generation of realistic 3D synthetic data for training. The computing system 1002 may include a single computing device 1014 (e.g., an NVIDIA OVX L40S server) and / or any number of computing devices 1014 (e.g., NVIDIA OVX systems) in the data center infrastructure 1016.

[0334] Computing device 1014 (which may include a server) may include CPUs (e.g., two CPUs, each with 32 cores) and GPUs (e.g., four or eight GPUs, each including 48GB GDDR6 with ECC memory, 864GB / s memory bandwidth, a PCIeGen4x16:64GB / s bidirectional interconnect interface, 18,176 CUDA cores, 142 ray tracing (RT) cores, and 568 Tensor cores). Computing device 1014 may include various networking and network management components, such as Intelligent Host Channel Adapters (HCAs) (e.g., two or four single-port ConnextX-7s, each with 200Gb / s, providing up to 800Gb / s InfiniBand / Ethernet), one or more DPUs (e.g., dual-port QSFP112 DPUs, such as NVIDIA BlueField-3 DPUs), providing up to 400Gb / s InfiniBand / Ethernet. In some embodiments, the NIC for computing device 1014 may include a SuperNIC (e.g., NVIDIA's ConnectX-8 SuperNIC) to provide up to 800Gb / s of data throughput for in-network compute acceleration engines, thereby providing the performance and robust feature set required to support trillion-parameter-scale AI factories and scientific computing workloads. In other embodiments, computing device 1014 may include a Smart Host Channel Adapter (HCA) (e.g., NVIDIA's ConnectX-7) to provide ultra-low latency and 400Gb / s throughput for in-network compute acceleration engines. Computing device 1014 may include host memory (e.g., 384Gb DDR5 ECC for four GPUs, or 768Gb DDR5 ECC for eight GPUs) and may include dual in-line memory module (DIMM) slots, host boot drives (e.g., 1TB NVMe), and / or host storage (e.g., 24TB NVMe).

[0335] Similar to data center infrastructure 1012, data center infrastructure 1016 can allow any number of computing devices 1014 to be combined into a cluster configuration according to a reference architecture.

[0336] Computing system 1006 can be used to deploy trained AI models on a runtime computer (such as the SoC 904 described herein). For example, these computing systems 1006 can be designed for compact onboard computing needs, including ensembles of models such as control policies, vision, and language models deployed on an energy-efficient onboard edge computing system 1006. (See also: [link to document]). Figures 9A to 9E A more detailed description of the components, features, and capabilities of the computing system 1006.

[0337] Example Generative Model

[0338] In at least some embodiments, language models such as Large Language Models (LLMs), Visual Language Models (VLMs), Multimodal Language Models (MMLMs), Visual Language Action (VLA) models, and / or other types of generative artificial intelligence (AI) can be implemented. These models may be able to understand, summarize, translate, and / or otherwise generate text (e.g., natural language text, code, etc.), images, videos, computer-aided design (CAD) assets, OMNIVERSE and / or METAVERSE file information (e.g., USD formats such as OpenUSD), and / or the like based on context provided in input prompts or queries. In embodiments, these language models may be considered "large" because they are trained on massive datasets and have architectures with a large number of learnable network parameters (weights and biases)—e.g., millions or billions of parameters. This disclosure allows for the implementation of LLM / VLM / MMLM, etc., for summarizing textual data, analyzing data (e.g., text, images, videos, etc.), extracting insights from data (e.g., text, images, videos, etc.), and generating new text / images / videos / , etc., in a user-specified style, tone, and / or format. In some embodiments, the LLM / VLM / MMLM, etc., of this disclosure may be specifically designed for text processing, while in other embodiments, multimodal LLMs may be implemented to accept, understand, and / or generate text and / or other types of content, such as images, audio (sound, synthesized speech, etc.), 2D and / or 3D data (e.g., USD format), and / or video. For example, a visual language model (VLM) or more specifically a multimodal language model (MMLM) may be implemented to accept images, videos, sensor data, audio, text, 3D designs (e.g., CAD), and / or other input data types and / or generate or output images, videos, audio, text, 3D designs, and / or other output data types.

[0339] Various types of LLM / VLM / MMLM / etc. architectures can be implemented in various embodiments. For example, different architectures can be implemented using different techniques to understand and generate outputs (e.g., text, audio, video, images, 2D and / or 3D design or asset data, etc.). In some embodiments, LLM / VLM / MMLM / etc. architectures (e.g., recurrent neural networks (RNNs) or long short-term memory networks (LSTMs)) can be used, while in other embodiments, converter architectures (e.g., architectures relying on self-attention and / or cross-attention (e.g., between contextual data and textual data) mechanisms) can be used to understand and recognize relationships between words or tokens and / or contextual data (e.g., other text, video, images, design data, USD, etc.). One or more generative processing pipelines including LLM / VLM / MMLM / etc. may also include one or more diffusion blocks (e.g., noise reduction blocks). The LLM / VLM / MMLM / etc. of this disclosure may include encoder and / or decoder blocks. For example, discriminative or encoder-only models (e.g., BERT (Bidirectional Encoder Representations from Transformers)) can be implemented for tasks involving language understanding (e.g., classification, sentiment analysis, question answering, and named entity recognition). As another example, generative or decoder-only models (e.g., GPT (Generative Pretrained Transformer)) can be implemented for tasks involving language and content generation (e.g., text completion, story generation, and dialogue generation). LLM / VLM / MMLM / etc., including encoder and decoder components (e.g., T5 (Text-to-Text Transformer)), can be implemented to understand and generate content, such as for translation and summarization. These examples are not intended to be limiting and any architecture type (including, but not limited to, those described herein) can be implemented depending on the specific implementation and the task performed using LLM / VLM / MMLM / etc.

[0340] In various embodiments, LLM / VLM / MMLM / etc. can be trained using unsupervised learning, whereby LLM / VLM / MMLM / etc. learns patterns from a large amount of unlabeled text / audio / video / image / design / USD / etc. data. Due to the extensive training, in these embodiments, the model may not require task-specific or domain-specific training. An LLM / VLM / MMLM / etc. extensively pre-trained on a large amount of unlabeled data can be referred to as a base model and can excel at various tasks, such as question answering, summarizing, filling in missing information, translation, and image / video / design / USD / data generation. Some LLM / VLM / MMLM / etc. can be customized for specific use cases using techniques such as cue tuning, fine-tuning, retrieval augmentation generation (RAG), adding adapters (e.g., custom neural networks and / or neural network layers to tune or adjust cues or labels to bias the language model towards a specific task or domain), and / or using optimization models for specific tasks and / or other fine-tuning or customization techniques within a specific domain.

[0341] In some embodiments, the LLM / VLM / MMLM / etc. disclosed herein can be implemented using various model alignment techniques. For example, in some embodiments, guardrails can be implemented to identify incorrect or unwanted inputs (e.g., prompts) and / or outputs of the model. In this process, the system can use guardrails and / or other model alignment techniques to prevent the processing of specific unwanted inputs using LLM / VLM / MMLM / etc., and / or to prevent the output or presentation of information generated by LLM / VLM / MMLM / etc. (e.g., displays, audio outputs, etc.). In some embodiments, one or more additional models (or layers thereof) can be implemented to identify problems with the model's inputs and / or outputs. For example, these "protective" models can be trained to identify "safe" or otherwise okay or desired inputs and / or outputs and / or "unsafe" or otherwise unwanted inputs and / or outputs for a particular application / implementation. Therefore, the LLM / VLM / MMLM / etc. disclosed herein are unlikely to output language / text / audio / video / design data / USD data / etc. that may be offensive, vulgar, inappropriate, insecure, out of scope, and / or unwanted for a particular application / implementation.

[0342] In some embodiments, an LLM / VLM / etc. can be configured or able to access or use one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc. For example, for certain tasks or operations where the model is not ideally suited, the model may have instructions for accessing one or more plugins (e.g., third-party plugins) to help process the current input (e.g., as a result of training, and / or based on instructions in a given prompt). In such an example, when at least part of the prompt relates to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs) to retrieve relevant information. Another example is that if at least part of the response requires mathematical computation, the model can access one or more mathematical plugins or APIs to help solve the problem, and then the response from the plugins and / or APIs can be used in the model's output. This process can be repeated (e.g., recursively) an arbitrary number of iterations, using any number of plugins and / or APIs, until a response to each query / question / request / process / action / etc. can be generated in response to the input prompt. Therefore, models can rely not only on their own knowledge gained from training on large datasets, but also on the expertise or optimized properties of one or more external resources (such as APIs, plugins, etc.).

[0343] In some embodiments, multiple language models (e.g., LLM / VLM / MMLM / etc., multiple instances of the same language model, and / or multiple hints provided to the same language model or instances of the same language model) can be implemented, executed, or accessed (e.g., using one or more plugins, user interfaces, APIs, databases, data stores, repositories, etc.) to provide output in response to the same query or in response to separate parts of a query. In at least one embodiment, the same input query and hints (e.g., a set of constraints, condition generators, etc.) can be provided to multiple language models (e.g., language models with different architectures, language models trained on different (e.g., updated) data corpora). In one or more embodiments, the language models can be different versions of the same base model. In one or more embodiments, at least one language model can be instantiated as multiple agents, for example, providing more than one hint to constrain, guide, or otherwise influence the style, content, or character of the provided output. In one or more exemplary non-limiting embodiments, the same language model can be required to provide output corresponding to different roles, perspectives, characters, or different knowledge bases, as defined by the provided hints.

[0344] In any such embodiment, the outputs of two or more (e.g., each) language models, two or more versions of at least one language model, two or more instantiated proxies of at least one language model, and / or provided to two or more prompts for at least one language model can be further processed, such as aggregated, compared, or filtered, or used to determine (and provide) a consensus response. In one or more embodiments, the output from one language model (or version, instance, or proxy) can be provided as input to another language model for further processing and / or validation. In one or more embodiments, the language model can be required to generate or otherwise obtain output about the input source material, wherein the output is associated with the input source material. This association may include, for example, generating captions or text portions embedded (e.g., as metadata) within the input source text or image. In one or more embodiments, the output of the language model can be used to determine the validity of the input source material for further processing or inclusion in a dataset. For example, the language model can be used to evaluate the presence (or absence) of a target word in a text portion or the presence (or absence) of an object in an image, wherein the text or image is annotated to indicate such presence (or absence). Alternatively, the determination from the language model can be used to determine whether the source material should be included in the curatorial dataset, for example, but not limited to this.

[0345] Figure 11 This is a block diagram of an example generative language model system 1100 suitable for implementing at least some embodiments of the present disclosure. Figure 11 In the example shown, the generative language model system 1100 includes a retrieval augmentation generation (RAG) component 1192, an input processor 1105, a tokenizer 1110, an embedding component 1120, a plugin / API 1195, and a generative language model (LM) 1130 (which may include LLM, VLM, MMLM, VLA models, etc.).

[0346] At a high level, the input processor 1105 can receive input 1101, which includes text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasound, etc.), 3D design data, CAD data, generic scene descriptor (USD) data (e.g., OpenUSD, etc.), depending on the architecture of the generative LM 1130 (e.g., LLM / VLM / MMLM, etc.). In some embodiments, input 1101 includes plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, input 1101 may include number sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., tabular format, JSON, or XML). In some implementations where the generative LM 1130 is capable of handling multimodal input, input 1101 may include text (or may omit text) along with image data, audio data, video data, design data, USD data, etc. The input processor 1105 can prepare the raw input text in various ways, taking raw input text as an example. For example, the input processor 1105 can perform various types of text filtering to remove noise (e.g., special characters, punctuation marks, HTML tags, stop words, parts of images, parts of audio, etc.) from the relevant text content. In the example involving stop words (common words that often have little semantic meaning), the input processor 1105 can remove stop words to reduce noise and make the generative LM more efficient. 1130 focuses on more meaningful content. Input processor 1105 can apply text normalization, for example, by converting all characters to lowercase, removing accents, and / or handling special cases (such as abbreviations or abbreviations) to ensure consistency (e.g., converting 1 / 4 to 1 / 4). Similarly, input processor 1105 and / or post-processor can perform inverse text normalization (ITN) to convert plain language back to canonical or other forms (e.g., converting 1 / 4 to 1 / 4). These are just a few examples; other types of input and / or output processing can be applied.

[0347] In some embodiments, RAG component 1192 (which may include one or more RAG models, and / or may be performed using generative LM 1130 itself) may be used to retrieve additional information to be used as part of input 1101 or a prompt. RAGs can be used to enhance input to LLM / VLM / MMLM / etc. with external knowledge to make the answer to a specific question or query or request more relevant, such as where specific knowledge is required. RAG component 1192 may obtain this additional information (e.g., basic information such as basic text / images / videos / audio / USD / CAD / etc.) from one or more external sources, and then feed it along with the prompt to LLM / VLM / MMLM / etc. to improve the accuracy of the model's response or output.

[0348] For example, in some embodiments, in addition to the data retrieved using RAG component 1192, input 1101 may also be generated using query or model inputs (e.g., questions, requests, etc.). In some embodiments, input processor 1105 may analyze input 1101 and communicate with RAG component 1192 (or in embodiments, RAG component 1192 may be part of input processor 1105) to identify relevant text and / or other data to provide to generative LM 1130 as additional context or information sources, typically from which responses, answers, or outputs 1190 are identified. For example, when the input indicates that a user is interested in the required tire pressure for a particular brand and model of vehicle, RAG component 1192 may use a RAG model, for example, to perform a vector search in the embedding space to retrieve tire pressure information or its corresponding text from a digital (embedded) version of the owner's manual for that particular vehicle brand and model. Similarly, when a user revisits the chatbot related to a specific product sale or service, the RAG component 1192 can retrieve previously stored conversation history (or at least its summary) and provide the previous conversation history, along with the current inquiry / request, as part of the generative LM 1130 as input 1101.

[0349] RAG component 1192 can use various RAG technologies. For example, it can use naive RAG ( The document is indexed, chunked, and applied to an embedding model to generate embeddings corresponding to chunks. User queries can also be applied to this embedding model and / or another embedding model of the RAG component 1192, and the embeddings of the chunks can be compared with the embeddings of the query to identify the most similar / relevant embeddings to the query. These most similar / relevant embeddings can be provided to the generative LM 1130 to generate output.

[0350] In some embodiments, more advanced RAG techniques can be used. For example, chunks can undergo pre-retrieval processes (e.g., routing, rewriting, metadata analysis, expansion, etc.) before being passed to the embedding model. Furthermore, post-retrieval processes (e.g., re-ranking, hint compression, etc.) can be performed on the output of the embedding model before generating the final embedding, which is then used for comparison with the input query.

[0351] As a further example, modular RAG techniques can be used, such as those similar to Naive RAG and / or Advanced RAG, but also including features such as hybrid search, recursive retrieval and query engines, StepBack methods, subqueries and hypothetical document embeddings.

[0352] As another example, Graph RAG can use a knowledge graph as a source of context or factual information. Graph RAG can be implemented using a graph database as a source of contextual information sent to LLM / VLM / MMLM / etc. Instead of providing the model with data chunks extracted from larger documents (which may result in a lack of context, factual accuracy, linguistic accuracy, etc.) (or anything other than providing the model with data chunks extracted from larger documents), Graph RAG can also provide structured entity information to LLM / VLM / MMLM / etc. by combining structured entity text descriptions with their many attributes and relationships, thus giving the model deeper insights. In implementing Graph RAG, the systems and methods described herein use graphs as content stores and extract relevant document chunks, requiring LLM / VLM / MMLM / etc. to use them to answer questions. In such embodiments, the knowledge graph may contain relevant textual content and metadata about the knowledge graph, or it may be integrated with a vector database. In some embodiments, Graph RAG can use the graph as a subject matter expert, where descriptions of concepts and entities relevant to the query / hint can be extracted and passed to the model as semantic context. These descriptions may include relationships between concepts. In other examples, the graph can be used as a database where a portion of a query / hint can be mapped to a graph query, the graph query can be executed, and LLM / VLM / MMLM / etc. can aggregate the results. In such examples, the graph can store relevant factual information and can be used for queries (natural language queries) and entity links to graph query tools (NL to graph query tools). In some embodiments, the graph RAG (e.g., using a graph database) can be combined with standard (e.g., vector database) RAGs and / or other RAG types to benefit from a variety of approaches.

[0353] In any embodiment, RAG component 1192 may implement plugins, APIs, user interfaces, and / or other functions to perform RAG. For example, LLM / VLM / MMLM / etc. may use graph RAG plugins to run queries on knowledge graphs to extract relevant information to feed into the model, and may use standard or vector RAG plugins to run queries on vector databases. For example, the graph database may interact with the plugin's REST interface, thus decoupling the graph database from the vector database and / or the embedded model.

[0354] The tokenizer 1110 can segment (e.g., processed) text data into smaller units (tags) for subsequent analysis and processing. Depending on the implementation, the tags can represent individual words, sub-words, characters, audio / video / images, etc. Word-based tokenization divides the text into individual words, treating each word as a separate tag. Sub-word tokenization breaks words down into smaller meaningful units (e.g., prefixes, suffixes, stems), enabling the generative LM 1130 to understand morphological changes and process words outside the vocabulary more effectively. Character-based tokenization represents each character as a separate tag, enabling the generative LM 1130 to process text at a fine-grained level. The choice of tokenization strategy can depend on factors such as the language being processed, the task at hand, and / or the characteristics of the training dataset. Therefore, the tokenizer 1110 can transform (e.g., processed) text into a structured format according to the tokenization scheme implemented in a particular embodiment.

[0355] Embedding component 1120 can use any known embedding technique to transform discrete tokens into semantically meaningful (e.g., dense, continuous vector) representations. For example, embedding component 1120 can use pre-trained word embeddings (e.g., Word2Vec, GloVe, or FastText), one-hot encoding, Term Frequency-Inverse Document Frequency (TF-IDF) encoding, one or more embedding layers of a neural network, and / or others.

[0356] In some implementations where input 1101 includes image data / video data, etc., input processor 1101 may resize the data to a standard size compatible with the format of the corresponding input channel and / or normalize pixel values ​​to a common range (e.g., 0 to 1) to ensure consistent representation, and embedding component 1120 may encode the image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations where input 1101 includes audio data, input processor 1101 may resample the audio file to a consistent sampling rate for uniform processing, and embedding component 1120 may use any known technique to extract and encode audio features, such as in the form of a spectrogram (e.g., a Mel spectrogram). In some implementations where input 1101 includes video data, input processor 1101 may extract frames or apply resizing to extracted frames, and embedding component 1120 may extract features such as optical flow embedding or video embedding and / or encode temporal information or frame sequences. In some implementations where input 1101 includes multimodal data, the embedded component 1120 may use techniques such as early fusion (concatenation), late fusion (sequential processing), and attention-based fusion (e.g., self-attention, cross-attention) to fuse representations of different types of data (e.g., text, images, audio, USD, video, design, etc.).

[0357] Other components of the generative LM 1130 and / or the generative LM system 1100 may use different types of neural network architectures depending on the implementation scheme. For example, a transducer-based architecture (e.g., the architecture used in models such as GPT) may be implemented, and it may include a self-attention mechanism that weights the importance of different words or tokens in the input sequence and / or a feedforward network that processes the output of the self-attention layer, which applies a nonlinear transformation to the input representation and extracts higher-level features. Some non-limiting example architectures include transducers (e.g., encoder-decoder, decoder-only, multimodal), RNNs, LSTMs, fusion models, diffusion models, cross-modal embedding models that learn a joint embedding space, graph neural networks (GNNs), hybrid architectures that combine different types of adversarial networks (e.g., generative adversarial networks or GANs or adversarial autoencoders (AAEs) for joint distribution learning, linear time series modeling using a selective state-space modeling (SSM) architecture (e.g., the Mamba LLM architecture), etc. Therefore, depending on the implementation scheme and architecture, the embedded component 1120 can apply the encoded representation of the input 1101 to the generative LM 1130, and the generative LM 1130 can process the encoded representation of the input 1101 to generate an output 1190, which may include response text and / or other types of data.

[0358] As described herein, in some embodiments, the generative LM 1130 may be configured to access or use (or be able to access or use) plugins / APIs 1195 (which may include one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc.). For example, for certain tasks or operations where the generative LM 1130 is not ideally suited, the model may have instructions (e.g., as a result of training, and / or based on instructions in a given prompt, such as instructions retrieved using RAG component 1192) to access one or more plugins / APIs 1195 (e.g., third-party plugins) to help process the current input. In such an example, when at least part of the prompt is related to a restaurant or weather, the model may access one or more restaurant or weather plugins (e.g., via one or more APIs), sending at least part of the prompt related to a particular plugin / API 1195 to the plugin / API 1195, which may process the information and return an answer to the generative LM 1130, which may use the response to generate output 1190. This process can be repeated (e.g., recursively) an arbitrary number of iterations and repeated with an arbitrary number of plugins / APIs 1195 until an output 1190 that resolves each query / question / request / process / action / etc. from input 1101 is generated. Therefore, the model can rely not only on its own knowledge acquired from training on a large dataset and / or from data retrieved using the RAG component 1192, but also on the expertise or optimized properties of one or more external resources (e.g., plugins / APIs 1195).

[0359] In some embodiments, one or more converter engines (TEs) can be implemented. Converter engines can use micro-tensor scaling to optimize performance and accuracy, such as enabling 16-bit floating-point (FP16), 8-bit floating-point (FP8), and / or 4-bit floating-point (FP4) AI processing. For example, a converter engine can use 16-bit or 8-bit floating-point precision and 8-bit or 4-bit floating-point data formats, combined with software algorithms, to improve AI performance and capabilities. By reducing mathematical operations to 8 or 4 bits, TEs can train larger networks faster without compromising accuracy. For example, TEs can include libraries for accelerating converter models on processing devices such as GPUs to provide better performance in training and inference with lower memory utilization. When TEs are combined with other technologies, such as high-speed interconnects between nodes (e.g., using NVLink switches) and tensor cores (which enable mixed-precision computation, such as micro-scaling precision support), server clusters can be more capable of training massive networks (e.g., billions of parameters) at high speeds. Therefore, it can support tensor core precision for FP64, TF32, BF16, FP16, FP8, INT8, FP6 and FP4, as well as CUDA core precision for FP64, FP32, FP16 and BF16.

[0360] The LLM / VLM / MMLM / VLA and other architectures described herein are intended to be examples only, and other suitable architectures may be implemented within the scope of this disclosure.

[0361] Example computing device

[0362] Figure 12This is a block diagram of an example computing device 1200 suitable for implementing some embodiments of the present disclosure. The computing device 1200 may include an interconnect system 1202 directly or indirectly coupled to the following devices: memory 1204, one or more central processing units (CPUs) 1206, one or more graphics processing units (GPUs) 1208, a communication interface 1210, input / output (I / O) ports 1212, input / output components 1214, a power supply 1216, one or more presentation components 1218 (e.g., one or more displays, one or more speakers, etc.), and one or more logic units 1220. In at least one embodiment, one or more computing devices 1200 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 1208 may include one or more vGPUs, one or more CPUs 1206 may include one or more vCPUs, and / or one or more logic units 1220 may include one or more virtual logic units. Thus, one or more computing devices 1200 may include discrete components (e.g., a full GPU dedicated to computing device 1200), virtual components (e.g., a portion of the GPU dedicated to computing device 1200), or combinations thereof.

[0363] although Figure 12 The various blocks are shown as connected via interconnect system 1202 using lines, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 1218 (such as a display device) may be considered I / O component 1214 (e.g., if the display is a touchscreen). As another example, CPU 1206 and / or GPU 1208 may include memory (e.g., memory 1204 may represent a storage device other than the memory of GPU 1208, CPU 1206, and / or other components). Therefore, Figure 12 The computing devices described are for illustrative purposes only. No distinction is made between such categories as “workstation,” “server,” “laptop computer,” “desktop computer,” “tablet computer,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of… Figure 12 Within the scope of computing devices.

[0364] Interconnect system 1202 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 1202 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 1206 may be directly connected to memory 1204. Further, CPU 1206 may be directly connected to GPU 1208. In cases where there is a direct or point-to-point connection between components, interconnect system 1202 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required to be included in computing device 1200.

[0365] The memory 1204 may include any computer-readable medium from a variety of computer-readable media. The computer-readable medium may be any available medium accessible by the computing device 1200. The computer-readable medium may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, the computer-readable medium may include computer storage media and communication media.

[0366] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1204 may store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, Digital Universal Disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 1200. As used herein, computer storage media does not include the signal itself.

[0367] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and includes any information transmission medium. The term "modulated data signal" can refer to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the above should also be included within the scope of computer-readable media.

[0368] CPU 1206 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1200 to perform one or more of the methods and / or processes described herein. Each CPU 1206 may contain one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling numerous software threads simultaneously. CPU 1206 may contain any type of processor and may contain different types of processors depending on the type of computing device 1200 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 1200, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (such as math coprocessors), computing device 1200 may also include one or more CPUs 1206.

[0369] In addition to or in lieu of one or more CPUs 1206, one or more GPUs 1208 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1200 to perform one or more of the methods and / or processes described herein. One or more GPUs 1208 may be integrated GPUs (e.g., with one or more CPUs 1206) and / or one or more GPUs 1208 may be discrete GPUs. In embodiments, one or more GPUs 1208 may be coprocessors of one or more CPUs 1206. GPUs 1208 may be used by computing device 1200 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 1208 may be used for general-purpose computing on a GPU (GPGPU). GPUs 1208 may include hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. GPUs 1208 may generate pixel data of an output image in response to rendering commands (e.g., rendering commands received from CPUs 1206 via a host interface). GPU 1208 may include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 1204. GPU 1208 may include two or more GPUs operating in parallel (e.g., via links). The links may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 1208 may produce pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for an analog image). Each GPU may include its own memory or may share memory with other GPUs.

[0370] In addition to or in lieu of CPU 1206 and / or GPU 1208, logic unit 1220 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1200 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 1206, one or more GPUs 1208, and / or one or more logic units 1220 may execute any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 1220 may be a portion of one or more CPUs 1206 and / or GPUs 1208 and / or integrated into one or more CPUs 1206 and / or GPUs 1208, and / or one or more logic units 1220 may be discrete components or otherwise external to CPUs 1206 and / or GPUs 1208. In an embodiment, one or more of the logic units 1220 may be coprocessors of one or more of the CPU 1206 and / or one or more of the GPU 1208.

[0371] Examples of logic unit 1220 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree lateral unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), programmable vision accelerator (PVA) (which may include one or more direct memory access (DMA) systems), and one or more vision or vector processing units (VPUs). This includes one or more pixel processing engines (PPEs) (e.g., comprising a 2D array of processing elements, each of which communicates north, south, east, and west with one or more other processing elements in the array), one or more decoupled accelerators or units (e.g., decoupled lookup table (DLUT) accelerators or units), vision processing units (VPUs), optical flow accelerators (OFAs), field-programmable gate arrays (FPGAs), neuromorphic chips, quantum processing units (QPUs), associative processing units (APUs), arithmetic logic units (ALUs), application-specific integrated circuits (ASICs), floating-point units (FPUs), input / output (I / O) elements, peripheral component interconnects (PCIs) or fast peripheral component interconnects (PCIe) elements, etc.

[0372] Communication interface 1210 may include one or more receivers, transmitters, and / or transceivers enabling computing device 1200 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). Communication interface 1210 may include components and functions for implementing communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wirelessband), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, logic unit 1220 and / or communication interface 1210 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via interconnect system 1202 to one or more GPUs 1208 (e.g., memory of one or more GPUs 1208).

[0373] I / O port 1212 enables computing device 1200 to be logically coupled to other devices including I / O component 1214, one or more presentation components 1218, and / or other components, some of which may be built into (e.g., integrated into) computing device 1200. Illustrative I / O component 1214 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dish antennas, scanners, printers, wireless devices, etc. I / O component 1214 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by the user. In some cases, input may be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, pen recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of computing device 1200. Computing device 1200 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. Additionally, the computing device 1200 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables motion detection. In some examples, the computing device 1200 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0374] Power supply 1216 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 1216 may provide power to computing device 1200 to enable the components of computing device 1200 to operate.

[0375] The presentation component 1218 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 1218 may receive data from other components (e.g., GPU 1208, CPU 1206, etc.) and output the data (e.g., as images, videos, sounds, etc.).

[0376] Example network environment

[0377] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 12 It is implemented on one or more instances of one or more computing devices 1200—for example, each device may include similar components, features, and / or functions of one or more computing devices 1200. Furthermore, in the case of implementing back-end devices (e.g., servers, NAS, etc.), the back-end devices may be included as part of a data center, such as, but not limited to, those described herein.

[0378] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0379] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein for the server can be implemented on any number of client devices.

[0380] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or application may respectively include network-based service software or applications. In embodiments, one or more client devices may use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open-source software network application framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0381] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers at the state, region, country, global, etc.). The core server may assign at least a portion of the functionality to the edge server if the connection to the user (e.g., a client device) is relatively close to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0382] One or more client devices may include the information described in this article. Figure 12 At least some of the components, features, and functions of one or more example computing devices 1200 described. By way of example and not limitation, the client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming equipment or system, entertainment system, vehicle computer system, embedded system controller, chat booth, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0383] In some embodiments, the systems and methods described herein can be performed in a simulated environment (e.g., simulated environment data and simulated sensor data of simulated sensors of virtual or simulated vehicles, robots, or machines in the simulated environment) using simulated data (e.g., simulated environment data and simulated sensor data of simulated sensors of virtual or simulated vehicles, robots, or machines in the simulated environment). For example, simulated input data (e.g., map data, perception data, self-motion data, haptic data, and / or any other data described herein) can be used to identify numerous scenarios involving interactions and / or occlusions between self-machines and agent machines, and to mine these scenarios to define more scenarios for training machine learning models, etc. This information can be used to perform virtual machine-related operations in the simulated environment. These simulated operations can be used to test the performance of underlying algorithms, systems, and / or processes before deploying them to the real world. In some cases, simulations can be used to generate synthetic training data, such as scenarios involving actual or potential interactions between self-machines and agent machines and / or occlusions in the simulation. The synthetic training data (in addition to or as an alternative to real-world data) can then be used or processed to mine scenarios involving self-machines and agent machines and / or occlusions.

[0384] In any example, such as when the simulation environment is used for testing, validation, training, etc., one or more optical transport simulation algorithms (e.g., one or more ray tracing and / or path tracing algorithms) may be used to render or otherwise generate the simulation environment and / or associated training data. When using optical transport simulation, the simulation system may employ one or more dedicated ray tracing hardware accelerators and / or processors (e.g., NVIDIA's RTX, or another real-time ray tracing GPU, such as a GPU including one or more ray tracing (RT) cores), which are optimized to perform real-time or near real-time optical transport simulation operations together with one or more other processors in the system (e.g., GPUs, CPUs, accelerators, etc.). In some embodiments, the simulation environment and / or one or more of its objects, features, or components may be generated or managed within a three-dimensional (3D) content collaboration platform (e.g., NVIDIA's OMNIVERSE), which may be optimized or suitable for industrial digitization, generative physical artificial intelligence, and / or other use cases, applications, and / or services. For example, a content collaboration platform or system may include a system for using or developing generic scene descriptor (USD) data (e.g., OpenUSD) to manage objects, features, scenes, etc., within simulated environments, digital environments, etc. The platform may include realistic physics simulations (e.g., using NVIDIA's PhysX software development kit (SDK)) to simulate real physics and physical interactions with simulations hosted on the platform. The platform may integrate OpenUSD with ray tracing / path tracing / light transport simulations (e.g., NVIDIA's RTX rendering technology) into software tools and simulation workflows for building, training, deploying, and / or testing AI systems, such as systems for testing, validating, training (e.g., machine learning models, neural networks, etc.) and / or other tasks related to automobiles, robots, other machine types, and / or other systems and applications. In some examples, the simulated environment may include a digital twin of a real-world environment, such as a specific road segment, warehouse, data center, airport, geographic area, ocean area, and / or any other real-world environment in which autonomous or semi-autonomous vehicles or machines can operate.

[0385] In some embodiments, remote control or a remote operating system can be used to perform remote operation or control of vehicles, robots, and / or other machines. For example, the systems and methods described herein can be used to mine scenarios involving interactions and / or occlusions between autonomous and agent machines, where these scenarios can be included in a visualization or mapping of the environment to help a remote operator control an autonomous or semi-autonomous machine through the environment (or provide waypoints or other control or navigation instructions). Thus, a remote operator can use the visual, auditory, textual, and / or other cues or indicators generated by the systems and methods described herein to assist in navigating vehicles, robots, machines, etc., in a real-world environment using a remote operating system.

[0386] In some embodiments, the systems and methods described herein can be deployed in robotic applications. For example, a robot or robotic system may include one or more onboard processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), hardware-based programmable vision accelerator (PVA) – which may include one or more vector processing units (VPU), direct memory access (DMA) systems and / or pixel processing engines (PPE), hardware-based optical flow accelerator (OFA), SoC, etc.) and memory and / or storage (e.g., for storing control algorithms, sensor data and one or more machine learning models). Robotic systems can use these processors to execute one or more machine learning models (e.g., language models, visual language models (VLM), large language models (LLM), visual-language-action (VLA) models, multimodal language models (MMLM), etc.), enabling them to perform complex tasks autonomously or semi-autonomously, such as interacting with and / or manipulating static and / or dynamic objects, or navigating in their environment using sensors such as cameras, LiDAR, RADAR, and ultrasonic sensors. The system can use sensor fusion techniques to combine data from multiple sensors (e.g., cameras, infrared, LiDAR, RADAR, accelerometers) to create a comprehensive model of the robot's surrounding environment. This data can be processed locally on the robot or sent to a remote server to perform computationally intensive tasks such as 3D mapping or SLAM (simultaneous localization and mapping). In one or more embodiments, data from a single robot (e.g., sensor data, task status, or environmental conditions) can be uploaded to the cloud, where a centralized AI model can analyze optimized commands and distribute them to the entire robot swarm. In some embodiments, the machine learning models described herein (e.g., language models, VLM, VLA, LLM, MMLM, diffusion models, NeRF models, DNN, etc.) can be used to allow a robot to perceive and reason about its environment and / or communicate with one or more other robots and / or people in the environment. In some embodiments, the robot may communicate with one or more locally hosted servers / computing devices and / or one or more remotely located servers / computing devices (e.g., in one or more data centers) using one or more network interface cards (NICs) and / or data processing units (DPUs).

[0387] In some embodiments, the systems and methods described herein can be deployed in in-vehicle infotainment (IVI) systems or in-vehicle experience (IX) applications. For example, an infotainment system within a vehicle (e.g., a car, truck, drone, construction equipment, robot, semi-autonomous vehicle, or autonomous vehicle) may include one or more onboard processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), hardware-based programmable vision accelerator (PVA) – which may include one or more vector processing units (VPU), direct memory access (DMA) systems and / or pixel processing engines (PPE), hardware-based optical flow accelerators (OFA), SoCs, etc.) and memory and / or storage (e.g., for storing control algorithms, sensor data, and one or more machine learning models), and memory and / or storage (e.g., for storing entertainment content, navigation data, and user preferences). The system can use these processors to execute one or more machine learning models (e.g., language models) to enable functions such as voice control, personalized media recommendations, dynamic navigation, and real-time communication with other services via network connectivity. In-vehicle infotainment systems can also use natural language processing (NLP) models to enable voice-based interaction. One or more machine learning models can be stored locally or accessed via one or more APIs connected to cloud services, enabling the system to process requests in real-time or near real-time.

[0388] In some examples, the machine learning models described herein (e.g., deep neural networks, language models, LLMs, VLMs, multimodal language models, visual-language-action (VLA) models, perceptual models, tracking models, fusion models, transformer models, diffusion models, encoder-only models, decoder-only models, encoder-decoder models, neural rendering field (NERF) models, etc.) can be packaged as microservices, such as inference microservices (e.g., NVIDIA NIM). These microservices can include containers (e.g., operating system (OS) level virtualization packages) that can include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model "engine." For example, an inference microservice can include the container itself and the model (e.g., weights and biases). In some cases, such as when the machine learning model is small enough (e.g., has a sufficiently small number of parameters), the model can be contained within the container itself. In other examples—e.g., when the model is large—the model can be hosted / stored in the cloud (e.g., in a data center) and / or hosted on-premises and / or at the edge (e.g., on a local server or computing device, but outside the container). In such embodiments, the model can be accessed via one or more APIs (e.g., REST APIs). Therefore, in some embodiments, the machine learning models described herein can be deployed as inference microservices to accelerate model deployment on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, an optimized inference engine (e.g., software builds using standardized AI model deployment and execution, such as NVIDIA's Triton Inference Server, and / or one or more APIs for high-performance deep learning inference, which may include inference runtime and model optimizations to provide low latency and high throughput for production applications—e.g., NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). The machine learning models described herein can be included as part of a microservice, along with acceleration infrastructure capable of deployment with a single command and / or orchestration and autoscaling using container orchestration systems on the acceleration infrastructure (e.g., reaching data center scale on a single device). Therefore, the inference microservice may include a machine learning model (e.g., optimized for high-performance inference), inference runtime software for executing the machine learning model and providing output / response to inputs (e.g., user queries, prompts, etc.), and enterprise management software for providing health checks, identity verification, and / or other monitoring. In some embodiments, the inference microservice may include software for performing in-situ replacements and / or updates to the machine learning model. During replacement or update, the software performing the replacement / update may maintain user configurations for both the inference runtime software and the enterprise management software.

[0389] In some embodiments, one or more converter engines (TEs) may be implemented. Converter engines can use micro-tensor scaling to optimize performance and accuracy, for example, to enable 16-bit floating-point (FP16), 8-bit floating-point (FP8), and / or 4-bit floating-point (FP4) AI processing. For example, a converter engine can combine 16-bit or 8-bit floating-point precision and 8-bit or 4-bit floating-point data formats with software algorithms to improve AI performance and capabilities. By reducing mathematical operations to 8 or 4 bits, TEs allow for faster training of larger networks without compromising accuracy. For example, TEs may include libraries for accelerating converter models on processing devices (e.g., GPUs) to provide better performance in training and inference with lower memory utilization. When TEs are combined with other technologies, such as high-speed interconnects between nodes (e.g., using switches, such as NVLink switches) and tensor cores (enabling mixed-precision computations, such as micro-scaling precision support), server clusters may be better equipped to train massive networks (e.g., billions of parameters) at high speeds. Therefore, it can support tensor core precision for FP64, TF32, BF16, FP16, FP8, INT8, FP6 and FP4, as well as CUDA core precision for FP64, FP32, FP16 and BF16.

[0390] The architectures described in this paper, including those of other LLM / VLM / MMLM / VLA, are merely examples, and other suitable architectures can be implemented within the scope of this disclosure.

[0391] The disclosed systems and methods also include a data generation pipeline comprising a simulator that performs various types of simulations related to a machine (e.g., a self-contained machine) navigating within an environment (e.g., a road). Data generated by the simulator based on the simulation includes (but is not limited to) rendered images of the environment surrounding the machine (e.g., from the perspective of one or more cameras on the machine and / or a bird's-eye view visualization), semantic labels associated with the images (e.g., segmentation maps, detected objects, bounding shapes, etc.), the machine's state (e.g., position, heading, velocity, etc.), and / or a map of free and / or occupied space within the environment. The data generation pipeline also includes a scene generator that determines targets within the map, such as (but not limited to) target locations to be navigated to in the environment.

[0392] The data generation pipeline also includes a planner that generates commands to instruct the machine to take scene-related actions, such as causing the machine to follow the scene's linear and / or angular velocities. These commands are sent to the simulator, which updates the machine's state, rendered images, semantic labels, maps, and / or other data based on the action. The simulator also sends some or all of the updated data to the planner, allowing the planner to generate new commands based on the updated data and the objectives from the scene generator. This process is repeated until the machine achieves the objective, executes a certain number of time steps in the simulation, and / or meets another condition indicating the end of the simulation.

[0393] The data generation pipeline also includes a recorder that records and synchronizes data output by other components across time steps. For example, the recorder may record data from other components in the order in which corresponding events occur in the simulation. The recorder may also downsample some or all of the data (e.g., on a spatial and / or temporal basis) to reduce the size of the recorded data.

[0394] Post-processors in the data generation pipeline adapt the generated data to various machine learning models and / or use cases. For example, post-processors may resample, compress, format, and / or otherwise transform the generated data into a form that can be used to train and / or evaluate machine learning models, hardware configurations, and / or other components of the machine.

[0395] The data generation pipeline can be configured and / or customized through one or more sets of configuration parameters. For example, configuration parameters may include unique names and / or identifiers for a given scenario (e.g., a combination of specific environments, machines, objectives, policies, etc.) in which data will be generated and collected. Configuration parameters can also be used to customize the environment and / or the type of machine to be simulated, the objective, the planner type, the type of data to be recorded, the frequency of data recording, and / or the way the recorded data is converted into a format suitable for training and / or evaluating machine learning models and / or another component of the machine. Different sets of configuration parameters can be used to launch different instances of the data generation pipeline (e.g., in parallel on multiple nodes of a distributed system) to generate data capturing different scenarios related to navigation and / or other types of tasks performed by machines in the environment.

[0396] Compared to previous methods, one advantage of the disclosed system and method is its ability to rapidly generate training data encompassing a large number of diverse scenarios by leveraging data caching within the execution engine. Another advantage is the ability to define scenarios spanning diverse environments, machine types, behaviors, and / or other types of data related to navigation and / or other tasks performed by the machine. These scenarios can be used to train, test, and / or evaluate / validate machine learning models and / or other components of the machine, thereby facilitating the machine's fault tolerance and / or generalization to different scenarios and / or use cases. These scenarios can also be used to train machine learning models to generate a wider range of data (additional scenarios) for evaluating / validating the system.

[0397] In some embodiments, a method includes:

[0398] Receive first data describing one or more agent trajector trajector trajector types and one or more agent types of one or more agent machines in an environment, and second data describing the self-trajector ...

[0399] Define a scenario including at least one interaction between the one or more agent machines in the environment and the ego machine, wherein the at least one interaction includes one or more of the relative velocity, relative distance, or relative acceleration between the ego machine and a corresponding agent machine among the one or more agent machines; and

[0400] Based on the first data and the second data, a third data representing the at least one interaction included in the scenario is identified.

[0401] 2. The method according to Clause 1 further comprises: modifying one or more of the first data or the second data to define at least one additional scenario, the at least one additional scenario including at least one additional interaction between the one or more agent machines in the environment and the ego machine, the third data further representing the at least one interaction included in the at least one additional scenario.

[0402] 3. The method according to Clause 1 or Clause 2 further comprises: changing one or more of the agent type of the one or more agent machines or the self type of the self machine to define at least one additional scenario, the at least one additional scenario including at least one additional interaction between the one or more agent machines and the self machine in the environment, the third data further representing the at least one interaction included in the at least one additional scenario.

[0403] 4. The method according to any one of Clauses 1-3, further comprising: changing one or more of the relative velocity, relative acceleration, or relative distance between the ego machine and each of the one or more agent machines to define at least one additional scenario, the at least one additional scenario including at least one additional interaction between the one or more agent machines and the ego machine in the environment, the third data further representing the at least one interaction included in the at least one additional scenario.

[0404] 5. The method according to any one of Clauses 1-4, wherein one or more of the first data or the second data includes one or more of the following: offline detection and tracking data, online detection and tracking data from multiple sensors on the self-machine, or manually labeled data.

[0405] 6. The method according to any one of clauses 1-5, wherein the at least one interaction comprises one or more of the following:

[0406] The self-machine moves at a constant linear speed with respect to the one or more agent machines;

[0407] Lane change of the one or more agent machines in front of the self machine;

[0408] The one or more agent machines surpass the ego machine;

[0409] Acceleration of at least one of the one or more agent machines relative to the ego machine;

[0410] The self-machine moves at a constant speed along the curved path between the self-machine and the one or more agent machines;

[0411] The respective movements of the self-machine and the one or more agent machines in opposite directions cause the self-machine and the one or more agent machines to approach each other;

[0412] The deceleration of at least one of the one or more agent machines in front of the self machine;

[0413] The self-machine and the one or more agent machines move in the same direction, such that the self-machine and the one or more agent machines are moving side by side in the same direction.

[0414] The self-machine is approaching at least one stationary agent machine;

[0415] The one or more agent machines outperform the ego machine, wherein at least one of the one or more agent machines accelerates relative to the ego machine.

[0416] The size difference between the self machine and at least one of the one or more agent machines;

[0417] The self-machine is obscured by one of the one or more agent machines, while another of the one or more agent machines is obscured.

[0418] At least one of the one or more agent machines passes through either side of the ego machine;

[0419] The self-machine performs a turn with respect to at least one of the one or more agent machines in the vicinity;

[0420] The self-machine approaches one or more agent machines in stop-and-go traffic;

[0421] The self-machine proceeds behind one or more agent machines that occupy the same lane as the self-machine and at least one lane on either side of the self-machine.

[0422] The self-machine is approaching one or more parked agent machines; or

[0423] One or more of the proxy machines blocked the ego machine's view of much smaller objects passing in front of it.

[0424] 7. The method according to any one of clauses 1-6 further includes: receiving fourth data, the fourth data including one or more occlusions present in the environment, the definition including: at least one interaction between the one or more agent machines in the environment and the ego machine in the presence of the one or more occlusions.

[0425] 8. The method according to any one of clauses 1-7 further includes: training one or more machine learning systems on the third data.

[0426] 9. The method according to any one of Clauses 1-8, wherein the self-machine and the one or more agent machines are selected from the group consisting of autonomous machines and semi-autonomous machines.

[0427] 10. The method according to any one of Clauses 1-9, wherein the autonomous machine and the semi-autonomous machine are selected from the group consisting of autonomous or semi-autonomous vehicles, autonomous or semi-autonomous mobile robots, autonomous or semi-autonomous humanoid robots, ships, underwater vehicles, unmanned aerial vehicles or aircraft.

[0428] 11. In some embodiments, at least one processor includes:

[0429] Processing circuitry, the processing circuitry being configured to cause an operation to be performed, the operation including:

[0430] Receive first data describing one or more agent trajector trajector trajector types and one or more agent types of one or more agent machines in an environment, and second data describing the self-trajector ...

[0431] Define a scenario including at least one interaction between the one or more agent machines in the environment and the ego machine, wherein the at least one interaction includes one or more of the relative velocity, relative distance, or relative acceleration between the ego machine and a corresponding agent machine among the one or more agent machines; and

[0432] Based on the first data and the second data, a third data representing the at least one interaction included in the scenario is identified.

[0433] 12. The at least one processor according to Clause 11, wherein the operation further comprises: modifying one or more of the first data or the second data to define at least one additional scenario, the at least one additional scenario including at least one additional interaction between the one or more agent machines in the environment and the ego machine, the third data further representing the at least one interaction included in the at least one additional scenario.

[0434] 13. At least one processor according to Clause 11 or Clause 12, wherein the operation further comprises: changing one or more of the agent type of the one or more agent machines or the self type of the self machine to define at least one additional scenario, the at least one additional scenario including at least one additional interaction between the one or more agent machines and the self machine in the environment, the third data further representing the at least one interaction included in the at least one additional scenario.

[0435] 14. At least one processor according to any one of clauses 11-13, wherein the operation further comprises: changing one or more of the relative velocity, relative acceleration, or relative distance between the ego machine and each of the one or more agent machines to define at least one additional scenario, the at least one additional scenario including at least one additional interaction between the one or more agent machines and the ego machine in the environment, the third data further representing the at least one interaction included in the at least one additional scenario.

[0436] 15. At least one processor according to any one of Clauses 11-14, wherein one or more of the first data or the second data comprises one or more of the following: offline detection and tracking data, online detection and tracking data from multiple sensors on the self-machine, or manually labeled data.

[0437] 16. At least one processor according to any one of clauses 11-15, wherein the operation further comprises: receiving fourth data, the fourth data including one or more occlusions present in the environment, the definition including: at least one interaction between the one or more agent machines in the environment and the ego machine in the presence of the one or more occlusions.

[0438] 17. At least one processor according to any one of clauses 11-16, wherein the operation further comprises: training one or more machine learning systems on the third data.

[0439] 18. At least one processor according to any one of clauses 11-17, wherein said at least one processor is included in at least one of the following:

[0440] Control systems for autonomous or semi-autonomous machines;

[0441] Sensing systems for autonomous or semi-autonomous machines;

[0442] A system for performing one or more simulation operations;

[0443] A system for performing one or more digital twin operations;

[0444] A system for performing one or more optical transmission simulations;

[0445] A system for collaborative content creation of 3D assets;

[0446] A system for performing one or more wireless cellular transmissions using a wireless cellular network;

[0447] A system that provides one or more cloud gaming applications;

[0448] A system for performing one or more deep learning operations;

[0449] Systems implemented using edge devices;

[0450] Systems implemented using robots;

[0451] A system for performing one or more generative AI operations;

[0452] A system for performing one or more conversational AI operations;

[0453] A system for performing operations using one or more large language model LLMs;

[0454] A system for performing operations using one or more visual language models (VLMs);

[0455] A system for performing operations using one or more multimodal language models (MMLMs);

[0456] A system for performing operations using one or more visual-language-action (VLA) models;

[0457] A system for performing one or more conversational AI operations;

[0458] A system for performing one or more synthetic data generation operations;

[0459] A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content;

[0460] A system that uses or deploys one or more inference microservices;

[0461] A system that includes deploying one or more machine learning models and operating system-level virtualization packages (e.g., containers) within a service or microservice;

[0462] A system containing one or more virtual machines (VMs);

[0463] A system that is at least partially implemented in a data center; or

[0464] A system that utilizes cloud computing resources at least in part.

[0465] 19. In some embodiments, a system includes:

[0466] One or more processors are configured to: receive first data describing one or more agent trajectories and one or more agent types of one or more agent machines in an environment, and second data describing the self-trajector trajectory and self-type of a self-machine in the environment; define a scenario including at least one interaction between one or more agent machines in the environment and a self-machine, said at least one interaction including one or more of a relative velocity, relative distance, or relative acceleration between the self-machine and a corresponding agent machine among the one or more agent machines.

[0467] 20. The system according to Clause 19, wherein said system is included in at least one of the following:

[0468] Control systems for autonomous or semi-autonomous machines;

[0469] Sensing systems for autonomous or semi-autonomous machines;

[0470] A system for performing one or more simulation operations;

[0471] A system for performing one or more digital twin operations;

[0472] A system for performing one or more optical transmission simulations;

[0473] A system for collaborative content creation of 3D assets;

[0474] A system for performing one or more wireless cellular transmissions using a wireless cellular network;

[0475] A system that provides one or more cloud gaming applications;

[0476] A system for performing one or more deep learning operations;

[0477] Systems implemented using edge devices;

[0478] Systems implemented using robots;

[0479] A system for performing one or more generative AI operations;

[0480] A system for performing one or more conversational AI operations;

[0481] A system for performing operations using one or more large language model LLMs;

[0482] A system for performing operations using one or more visual language models (VLMs);

[0483] A system for performing operations using one or more multimodal language models (MMLMs);

[0484] A system for performing operations using one or more visual-language-action (VLA) models;

[0485] A system for performing one or more conversational AI operations;

[0486] A system for performing one or more synthetic data generation operations;

[0487] A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content;

[0488] A system that uses or deploys one or more inference microservices;

[0489] A system that includes deploying one or more machine learning models and operating system-level virtualization packages (e.g., containers) within a service or microservice;

[0490] A system containing one or more virtual machines (VMs);

[0491] A system that is at least partially implemented in a data center; or

[0492] A system that utilizes cloud computing resources at least in part.

[0493] This disclosure can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program modules, which are executed by a computer or other machine (such as a personal data assistant or other handheld device). Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure can also be practiced in distributed computing environments, where tasks are performed by remote processing devices linked via a communication network.

[0494] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0495] This document describes in detail the subject matter of this disclosure to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the discloser has envisioned that the claimed subject matter may be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of the method employed, these terms should not be construed as suggesting any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. A method comprising: Receive first data describing one or more agent trajector trajector trajector types and one or more agent types of one or more agent machines in an environment, and second data describing the self-trajector ... Define a scenario including at least one interaction between the one or more agent machines in the environment and the ego machine, wherein the at least one interaction includes one or more of the relative velocity, relative distance, or relative acceleration between the ego machine and a corresponding agent machine among the one or more agent machines; and Based on the first data and the second data, a third data representing the at least one interaction included in the scenario is identified.

2. The method according to claim 1, further comprising: The third data modifies one or more of the first data or the second data to define at least one additional scenario, the at least one additional scenario including at least one additional interaction between the one or more agent machines in the environment and the ego machine, the third data further representing the at least one interaction included in the at least one additional scenario.

3. The method according to claim 1, further comprising: The agent type of the one or more agent machines or the self type of the self machine is changed to define at least one additional scenario, the at least one additional scenario including at least one additional interaction between the one or more agent machines and the self machine in the environment, the third data further representing the at least one interaction included in the at least one additional scenario.

4. The method according to claim 1, further comprising: The third data further represents the at least one additional scenario defined by changing one or more of the relative velocity, relative acceleration, or relative distance between the ego machine and each of the one or more agent machines in the environment, wherein the at least one additional scenario includes at least one additional interaction between the one or more agent machines and the ego machine in the environment.

5. The method of claim 1, wherein one or more of the first data or the second data comprises one or more of the following: offline detection and tracking data, online detection and tracking data from multiple sensors on the self-machine, or manually labeled data.

6. The method of claim 1, wherein the at least one interaction comprises one or more of the following: The self-machine moves at a constant linear speed with respect to the one or more agent machines; Lane change of the one or more agent machines in front of the self machine; The one or more agent machines surpass the ego machine; Acceleration of at least one of the one or more agent machines relative to the ego machine; The self-machine moves at a constant speed along the curved path between the self-machine and the one or more agent machines; The respective movements of the self-machine and the one or more agent machines in opposite directions cause the self-machine and the one or more agent machines to approach each other; The deceleration of at least one of the one or more agent machines in front of the self machine; The self-machine and the one or more agent machines move in the same direction, such that the self-machine and the one or more agent machines are moving side by side in the same direction. The self-machine is approaching at least one stationary agent machine; The one or more agent machines outperform the ego machine, wherein at least one of the one or more agent machines accelerates relative to the ego machine. The size difference between the self machine and at least one of the one or more agent machines; The self-machine is obscured by one of the one or more agent machines, while another of the one or more agent machines is obscured. At least one of the one or more agent machines passes through either side of the ego machine; The self-machine performs a turn with respect to at least one of the one or more agent machines in the vicinity; The self-machine approaches one or more agent machines in stop-and-go traffic; The self-machine proceeds behind one or more agent machines that occupy the same lane as the self-machine and at least one lane on either side of the self-machine. The self-machine is approaching one or more parked agent machines; or One or more of the proxy machines blocked the ego machine's view of much smaller objects passing in front of it.

7. The method according to claim 1, further comprising: Receive fourth data, the fourth data including one or more occlusions present in the environment, the definition including: at least one interaction between the one or more agent machines and the ego machine in the environment in the presence of the one or more occlusions.

8. The method according to claim 1, further comprising: Train one or more machine learning systems on the third data.

9. The method of claim 1, wherein the self-machine and the one or more agent machines are selected from the group consisting of autonomous machines and semi-autonomous machines.

10. The method of claim 1, wherein the autonomous machine and the semi-autonomous machine are selected from the group consisting of autonomous or semi-autonomous vehicles, autonomous or semi-autonomous mobile robots, autonomous or semi-autonomous humanoid robots, ships, underwater vehicles, drones, or aircraft.

11. At least one processor, comprising: Processing circuitry, the processing circuitry being configured to cause an operation to be performed, the operation including: Receive first data describing one or more agent trajector trajector trajector types and one or more agent types of one or more agent machines in an environment, and second data describing the self-trajector ... Define a scenario including at least one interaction between the one or more agent machines in the environment and the ego machine, wherein the at least one interaction includes one or more of the relative velocity, relative distance, or relative acceleration between the ego machine and a corresponding agent machine among the one or more agent machines; and Based on the first data and the second data, a third data representing the at least one interaction included in the scenario is identified.

12. The at least one processor according to claim 11, wherein the operation further comprises: The third data modifies one or more of the first data or the second data to define at least one additional scenario, the at least one additional scenario including at least one additional interaction between the one or more agent machines in the environment and the ego machine, the third data further representing the at least one interaction included in the at least one additional scenario.

13. The at least one processor according to claim 11, wherein the operation further comprises: The agent type of the one or more agent machines or the self type of the self machine is changed to define at least one additional scenario, the at least one additional scenario including at least one additional interaction between the one or more agent machines and the self machine in the environment, the third data further representing the at least one interaction included in the at least one additional scenario.

14. The at least one processor according to claim 11, wherein the operation further comprises: The third data further represents the at least one additional scenario defined by changing one or more of the relative velocity, relative acceleration, or relative distance between the ego machine and each of the one or more agent machines in the environment, wherein the at least one additional scenario includes at least one additional interaction between the one or more agent machines and the ego machine in the environment.

15. The at least one processor of claim 11, wherein one or more of the first data or the second data comprises one or more of the following: offline detection and tracking data, online detection and tracking data from multiple sensors on the self-machine, or manually labeled data.

16. The at least one processor according to claim 11, wherein the operation further comprises: Receive fourth data, the fourth data including one or more occlusions present in the environment, the definition including: at least one interaction between the one or more agent machines and the ego machine in the environment in the presence of the one or more occlusions.

17. The at least one processor according to claim 11, wherein the operation further comprises: Train one or more machine learning systems on the third data.

18. The at least one processor according to claim 11, wherein the at least one processor is comprised of at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing one or more optical transmission simulations; A system for collaborative content creation of 3D assets; A system for performing one or more wireless cellular transmissions using a wireless cellular network; A system that provides one or more cloud gaming applications; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing one or more conversational AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models (MMLMs); A system for performing operations using one or more visual-language-action (VLA) models; A system for performing one or more conversational AI operations; A system for performing one or more synthetic data generation operations; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system that uses or deploys one or more inference microservices; A system that includes deploying one or more machine learning models and operating system-level virtualization packages (e.g., containers) within a service or microservice; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

19. A system comprising: One or more processors are configured to: receive first data describing one or more agent trajectories and one or more agent types of one or more agent machines in an environment, and second data describing the self-trajector trajectory and self-type of a self-machine in the environment; define a scenario including at least one interaction between one or more agent machines in the environment and a self-machine, said at least one interaction including one or more of a relative velocity, relative distance, or relative acceleration between the self-machine and a corresponding agent machine among the one or more agent machines.

20. The system of claim 19, wherein the system comprises at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing one or more optical transmission simulations; A system for collaborative content creation of 3D assets; A system for performing one or more wireless cellular transmissions using a wireless cellular network; A system that provides one or more cloud gaming applications; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing one or more conversational AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models (MMLMs); A system for performing operations using one or more visual-language-action (VLA) models; A system for performing one or more conversational AI operations; A system for performing one or more synthetic data generation operations; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system that uses or deploys one or more inference microservices; A system that includes deploying one or more machine learning models and operating system-level virtualization packages (e.g., containers) within a service or microservice; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

Images