Probabilistic state simulation for end-to-end drive stack learning for autonomous and semi-autonomous machines and applications

Abstract

In various examples, perception encoders use one or more neural networks implemented using a transformer architecture, sensor-perspective coding, a planned navigation route, and / or captured self-motion to extract a scene embedding that represents one or more aspects of an observed scene, such as visual information, motion information, an EGO state of an EGO machine, a planned navigation route, and / or other types of information.The perception encoder can be used in a probabilistic state simulation stack and / or can be used to extract and apply a scene embedding as an input for 3D perception or reconstruction tasks, such as object detection and classification, semantic segmentation, depth map extraction, trajectory prediction, path planning, navigation control and / or localization or mapping, to name a few example tasks.

Description

CROSS-REFERENCE TO RELATED REGISTRATIONS

[0001] This application is a continuation of U.S. Application No. 19 / 041,044, filed on January 30, 2025, which claims priority over preliminary U.S. Application No. 63 / 675,196, filed on July 24, 2024, the contents of which are hereby incorporated by reference in their entirety. BACKGROUND

[0002] Many autonomous machines, such as self-driving cars and drones, use a navigation (e.g., driving) policy to make decisions about how to behave in various (e.g., driving) scenarios in complex, real-world environments. Navigation policies are often implemented using neural motion planners (or neural planems)—advanced decision-making systems that use deep neural networks (DNNs) to determine the best actions, for example, based on the machine's current state and its environment. Some neural planners are trained using imitation learning, where they learn by observing and imitating recorded behaviors of human experts (e.g., how a human driver navigates traffic or avoids obstacles). The neural planner then generalizes from this data to make its own decisions.

[0003] During deployment, however, these models often face a challenge known as covariance shift, which occurs when the distribution of states the neural planner encounters during training differs from the one it encounters in real-world situations. This discrepancy arises because the training data—representing recorded expert behavior or other target behavior—may not encompass all possible situations the neural planner might encounter in practice, such as unpredictable human behavior (e.g., pedestrians walking, cyclists weaving between cars, vehicles suddenly performing illegal maneuvers). As a result, the neural planner may perform well in situations represented in its training data but generate random or incorrect driving actions in response to previously unseen situations during deployment.The discrepancy between training and operational environments can lead to compounding errors, where an autonomous vehicle drifts away from optimal trajectories when guided by the trained neural planners, resulting in degraded performance and potentially unsafe or inefficient behavior.

[0004] Several conventional techniques attempt to address the covariance shifting problem using simulated training data. Instead of using training data consisting solely of expert demonstrations performed under typical, safe driving conditions, these techniques utilize simulation environments or other tools to manually generate simulated training data that represents potential edge cases, errors, and / or unusual scenarios the neural planner might encounter during deployment. For example, developers or domain experts can manually create the neural planner and expose it to simulated training data representing hard-crafted scenarios, such as unusual road layouts, unexpected obstacles, sudden traffic changes, adverse weather, and other rare (long-tail) situations that fall outside the typical scenarios seen during training.In practice, this manual creation of simulated data often involves adding disturbances to real-world data by introducing noise. While training on this enriched dataset can help the neural planner handle a wider range of driving contexts, the process of manually creating simulation data is often error-prone, does not perfectly represent traffic behavior, and often results in the neural planner assuming unrealistic driving behavior.

[0005] In the context of simulation, a gap or discrepancy can arise between the simulated environment and real-world conditions because even advanced simulators may struggle to accurately replicate the complexity, variability, and unpredictability of rare real-world scenarios. Factors contributing to this gap include limitations in the fidelity of simulation models, differences in sensor inputs, and the challenge of capturing the full spectrum of environmental and behavioral nuances present in real-world driving. As a result, while a neural planner might perform well in simulated long-tail cases, its performance in the real world may be less reliable if these rare situations are not perfectly mirrored in the simulator.

[0006] More generally, the accuracy of a navigation (e.g., driving) directive and the actions it selects can depend on a variety of factors, including how accurately the surrounding environment is perceived and encoded into corresponding inputs for the directive. Conventional driving directives often rely on spatial representations of environmental features derived from raw sensor data, such as occupancy maps depicting detected road boundaries and obstacles. These encoded inputs can be applied to deep neural networks (DNNs) or rule-based systems that define the driving directive and determine which actions to take. The accuracy and richness of these encoded inputs directly affect the effectiveness and safety of the driving directive.Although perceptual techniques tend to improve over time, the current state of the art is limited in how accurately they perceive and code the real-world environment, as well as in the level of detail and comprehensiveness of that coding. These limitations can lead to an incomplete or inaccurate understanding of the environment, which in turn affects driving policy. For example, if a perceptual system misinterprets important information or the resolution of the resulting coding is insufficient, driving policy may make suboptimal or unsafe decisions, such as incorrectly judging when to brake or change lanes. More generally, other types of tasks that rely on perception may experience corresponding deteriorations in performance resulting from limitations in existing perceptual techniques.

[0007] Therefore, there is a need for improved techniques for perceiving and determining which actions autonomous or semi-autonomous machines such as self-driving cars and drones should take. SUMMARY

[0008] Embodiments of the present disclosure relate to perception, scene coding, probabilistic state simulation and / or generation of (e.g., ground truth) recovery scenarios for autonomous machines and applications.

[0009] In some embodiments, a probabilistic state simulation stack can be used to estimate and represent the state of a scene, including the state of an ego-machine (e.g., speed or position), traffic dynamics (e.g., the behavior of other road users), and / or static elements in the environment (e.g., roads, signs, or obstacles). Co-training a driving (or other navigation) policy as part of a probabilistic state simulation stack addresses the covariance shifting problem. For example, the probabilistic state simulation stack can be trained using a ground-truth representation of human driving data, and at least a portion of the trained probabilistic state simulation stack can be used as an end-to-end drive stack in an autonomous or semi-autonomous machine (or another type of control stack for other applications).This approach can be used to develop a robust driving policy that improves upon the current state of the art by sampling from predicted state distributions to generate (e.g., simulate) any number of new (e.g., driving) situations and traffic scenarios, and training the policy to handle these previously unseen scenarios, thereby improving the accuracy and performance of the policy.

[0010] In some embodiments, the probabilistic state simulation stack includes a perception encoder that uses one or more neural networks implemented using a transformer architecture, sensor perspective coding, a planned navigation route and / or detected ego motion to extract a scene embedding representing one or more aspects of an observed scene, such as visual information, motion information, ego state of an ego machine, a planned navigation route and / or other types of information.In addition to or as an alternative to using the perception encoder in a probabilistic state simulation stack, the perception and scene coding techniques described herein can be used to extract and use a scene embedding as input for 3D perception or reconstruction tasks, such as object detection and classification (e.g., identifying pedestrians, vehicles, traffic signs, obstacles, etc.), semantic segmentation (e.g., labeling one or more elements in the scene by class), depth map extraction, trajectory prediction, path planning, navigation control (e.g., by a control stack that predicts one or more actions for the ego machine to perform), and / or localization or mapping (e.g., generating a 3D representation of the environment or localizing a 3D position within the environment), to name a few example tasks.

[0011] In some embodiments, a generative DNN can be trained as part of a probabilistic state simulation stack to predict a state distribution that models a set of possible world states, and predicted state distributions (or world models) can be generatively sampled to generate recovery scenarios (e.g., ground truth recovery scenarios) for other navigation policies or other supervised DNNs (e.g., a neural planner) that were not part of the probabilistic state simulation stack and that can run in a modular control stack. For example, an initial trajectory that drifts from an optimal or target trajectory can be generated in various ways, such as…This can be achieved by using a neural planner to control the navigation of an ego machine in a simulation environment or in a latent space of the generative DNN, or by using a control stack trained as part of a probabilistic state simulation stack to generate control actions, where the control actions are perturbed, and using the perturbed control actions to control the navigation of an ego machine in a simulation environment. As such, the control stack trained as part of the probabilistic state simulation stack can be used to recover from the initial trajectory, and the resulting recovery trajectory can be recorded and used to train a navigation policy such as a neural planner (e.g., one that generated the initial trajectory).

[0012] As such, a probabilistic state simulation stack can be trained using sensor data to predict what will happen next in the world, and the trained probabilistic state simulation stack can be used for various purposes, such as learning and validating a constituent end-to-end drive stack (e.g., featuring a perception encoder, generative DNN, and drive policy chain), generating simulated ground truth data to train another DNN (e.g.,Training neural planners using recovery trajectories sampled from the state distribution predicted by a generative DNN state estimator; training other perceptual DNNs or navigational guideline DNNs using ground truth traffic scenarios in perspective and / or top-down views generated by sampling from predicted state distributions; simulating future scenarios conditioned on self-actions for planning; and / or other tasks. Co-training generative DNN state estimators with a perceptual encoder, navigational guideline, and / or latent space decoder facilitates the training of end-to-end control stacks (e.g., drives) without the use of hand-coded code, improving the accuracy of the resulting trained models and the efficiency of the training process.Thus, the techniques presented here can be used to increase the accuracy and performance of control stacks such as drives; navigation guidelines such as neural planners; perception; and / or downstream tasks that rely on perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The present systems and methods for perception, scene coding, probabilistic state simulation and / or generation of recovery scenarios for autonomous machines and applications are described in detail below with reference to the attached drawings, wherein: Fig. Figure 1 illustrates an exemplary technique in which a probabilistic state simulation stack can be used to solve covariance shifting by training its navigation policy to drive in the direction of states encountered in human expert demonstrations, according to some embodiments of the present disclosure. Fig. Figure 2 illustrates an exemplary probabilistic state simulation stack according to some embodiments of the present disclosure. Fig. Figure 3 illustrates an exemplary perception encoder according to some embodiments of the present disclosure. Fig. 4A- Fig. 4B illustrates some exemplary reconstructed representations of an estimated state of a scene according to some embodiments of the present disclosure. Fig. Figure 5 illustrates an exemplary technique for generating recovery trajectories according to some embodiments of the present disclosure. Fig. Figure 6A illustrates an exemplary technique for initializing a probabilistic state simulation stack according to some embodiments of the present disclosure. Fig. Figure 6B illustrates an exemplary technique for generating an initial trajectory using a neural planner to update a simulation environment according to some embodiments of the present disclosure. Fig. Figure 6C illustrates an exemplary technique for generating an initial trajectory using a neural planner to update a simulation in the latent space of a probabilistic state simulation stack according to some embodiments of the present disclosure. Fig. Figure 7 is a flowchart showing a method for generating a scene embedding representing an environment according to some embodiments of the present disclosure; Fig. Figure 8 is a flowchart showing a method for operating at least part of a probabilistic state simulation stack as a control stack of an ego machine according to some embodiments of the present disclosure. Fig. Figure 9 is a flowchart showing a method for generating one or more recovery trajectories for one or more simulated ego machines according to some embodiments of the present disclosure. Fig. Figure 10A is an illustration of an exemplary autonomous vehicle according to some embodiments of the present disclosure. Fig. 10B is an example of camera positions and fields of view for the exemplary autonomous vehicle of Fig. 10A according to some embodiments of the present disclosure. Fig. 10C is a block diagram of an exemplary system architecture for the exemplary autonomous vehicle of Fig. 10A according to some embodiments of the present disclosure. Fig. 10D is a system diagram for communication between one or more cloud-based servers and the exemplary autonomous vehicle of Fig. 10A according to some embodiments of the present disclosure. Fig. Figure 11 is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure; and Fig. Figure 12 is a block diagram of an exemplary data center suitable for use in implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0014] Systems and methods are disclosed relating to perception, scene coding, probabilistic state simulation, and / or the generation of recovery scenarios for autonomous machines and applications. The techniques presented can be used for perception tasks, end-to-end drive training, neural planner training, and / or other purposes, and can be used to control the navigation of autonomous vehicles, semi-autonomous vehicles, robots, and / or other types of objects or machines.

[0015] Although the present disclosure relates to an exemplary autonomous or semi-autonomous vehicle or an exemplary autonomous or semi-autonomous machine 1000 (hereinafter alternatively referred to as "Vehicle 1000" or "Ego-Machine 1000"), of which an example is given in relation to the Fig. 10A- Fig. The fact that the systems and procedures described herein may be used (as described in Section 10D) is not intended to be restrictive. For example, the systems and procedures described herein may be used by, without limitation, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more advanced driver assistance systems (ADAS)), autonomous vehicles or machines, controlled and uncontrolled robots or robotic platforms, warehouse vehicles, all-terrain vehicles, vehicles coupled to one or more trailers, flying objects, boats, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, airplanes, construction vehicles, trains, underwater vehicles, remotely controlled vehicles such as drones, and / or other types of vehicles.Additionally, although the present disclosure may be described in relation to perception or navigation control for autonomous driving, this is not intended to be limiting, and the systems and methods described herein may be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications and / or any other technology spaces where perception or navigation control may be used.

[0016] In some embodiments, a probabilistic state simulation stack can be used to estimate and represent the state of an ego-machine (e.g., speed or position), traffic dynamics (e.g., the behavior of other road users), and / or static elements in the environment (e.g., roads, signs, or obstacles). The probabilistic state simulation stack can process sensor data (e.g.,Encode camera images, LiDAR data, RADAR data) into a scene embedding, generate a posterior distribution representing an estimated current state of the scene by projecting the scene embedding and a representation of the overall course of the scene into a latent space using a first generative DNN, and generate a prior distribution representing an estimated future state of the scene by projecting the representation of the overall course of the scene into the latent space using a second generative DNN. Thus, the estimated prior and posterior state distributions (which can be referred to as world models) can be probabilistically sampled and applied to 1) a driving guideline to predict a trajectory and / or steering action (e.g.,The system consists of: 1) the following actions (acceleration, braking, steering) that the ego-machine is to take; 2) one or more decoders to reconstruct one or more representations of the estimated state of the observed scene (e.g., simulated images, detected objects, semantic segmentations); and / or 3) a history aggregation network that generates the representation of the aggregated history for the subsequent iteration. During training, the predicted control action, divergence between the estimated previous and posterior state distributions, and / or reconstructed representation(s) of the estimated state of the observed scene can be used to generate corresponding losses, which can be used to update one or more components of the probabilistic state simulation stack.Thus, the probabilistic state simulation stack can execute iteratively, recursively propagate the sampled state from a previous iteration, and use it with detected sensor data representing a subsequent time segment during a subsequent iteration.

[0017] Co-training the driving (or other navigation) policy as part of a probabilistic state simulation stack addresses the covariance shifting problem. The second generative DNN effectively predicts a distribution of future world states (or world models), which facilitates sampling new states from the learned latent space representing scenarios not present in the training data. Running the probabilistic state simulation stack, as described above, uses these sampled states to train the policy to recover from errors by selecting actions corresponding to future latent states that are closer to states observed in human demonstrations, an example of which is shown in Fig. Figure 1 illustrates this. As such, the trained probabilistic state simulation stack can be used to solve or mitigate the covariance shift problem in imitation learning. For example, the probabilistic state simulation stack can be trained using a ground truth representation of human driving data, and a chain of components from the probabilistic state simulation stack (e.g., the trained perceptual encoder that generates the scene embedding, the first generative DNN that generates a posterior distribution representing an estimated current state of the scene, the driving policy, and the history aggregation network) can be used as an end-to-end drive stack in an autonomous or semi-autonomous machine (or another type of control stack for other applications).This approach can be used to develop a robust driving policy that improves upon the current state of the art by sampling from predicted world models (state distributions) to generate (e.g., simulate) any number of new (e.g., driving) situations and traffic scenarios, and training the policy to handle these previously unseen scenarios, thereby improving the accuracy and performance of the policy.

[0018] In some embodiments, the probabilistic state simulation stack may include a perceptual encoder that uses one or more neural networks implemented using a transformer architecture, sensor perspective coding, a planned navigation route, and / or detected ego motion to learn a representation of the scene. For example, the perceptual encoder may encode a temporal sequence of frames from sensor data (e.g., two consecutive frames of image data), and the encoded sequence of frames may be processed using a cross-attention layer between queries and key values—effectively looking back into (latent state) history using attention—to generate a set of motion feature tokens representing the currently observed scene and containing the motion information.In some embodiments, the perception encoder can encode a representation of the perspective or viewing direction of the corresponding sensor(s) (e.g., intrinsic parameters of a sensor and extrinsic parameters representing its position relative to a reference coordinate system, such as a rig coordinate system or a universe) or the perspective or viewing direction represented by a subset of the sensor data generated by the sensor, such as a pixel or a patch of an image (e.g., the direction of a ray projected from the center of the sensor through a corresponding pixel or patch, and the position of the sensor relative to a reference coordinate system, such as a rig coordinate system or a universe).As such, the perceptual encoder can combine the coded representation of the sensor perspective with corresponding positional codings for the coded sensor data to map the coded sensor data (e.g., tokens encoding corresponding patches of sensor data) onto a corresponding section of the three-dimensional (3D) scene, thereby adding inductive priors for improved 3D estimation and enhancing the resulting motion feature tokens generated by the cross-attention layer. The perceptual encoder can process the motion feature tokens using self-attention and a scene query that crosses keys and values ​​to output a scene embedding representing the observed scene. In some embodiments, a representation of a navigation route (e.g.,a binary mask representing a top-down trajectory, a sequence of 2D waypoints, a sequence of navigation route commands) and / or a representation of detected ego movement (e.g., relative to a previous time segment) is encoded and combined with the scene embedding to integrate detected and / or planned ego movement.

[0019] Generating a scene embedding using a transformer architecture, sensor perspective coding, a planned navigation route, and / or detected ego motion provides a richer and more accurate representation of the scene than previous techniques and can be used to improve the accuracy of a downstream task. For example, in embodiments where the perception encoder is implemented in a probabilistic state simulation stack or control stack that uses the scene embedding to predict a trajectory and / or control action (e.g., acceleration, braking, steering) for the ego machine to take, the improved scene embedding should also improve the accuracy of the trajectories and / or control actions generated by the navigation policy.In some embodiments, the perception and scene coding techniques described herein can be used for 3D perception or reconstruction tasks, such as object detection or classification, depth map extraction, semantic segmentation, and / or other tasks. Thus, various embodiments can effectively provide multi-view structure-from-motion functionality that encodes a 3D scene with appropriate semantics, 3D information, and / or motion information (e.g., velocities).

[0020] In some embodiments, a generative DNN trained as part of a probabilistic state simulation stack can be used to predict state distributions (or world models) that can be generatively sampled to generate recovery scenarios (e.g., ground truth or equivalent recovery scenarios) for other navigation policies or other supervised DNNs (e.g., a neural planner) that were not part of the probabilistic state simulation stack and that can run in a modular control stack. For example, the generative DNN predicting a posterior distribution representing an estimated current state of the scene and the history aggregation network can be initialized with a real episode (e.g.,(Recorded driving actions, a sequence of frames of sensor data) to aggregate the course of the scene, and a neural planner pre-trained using imitation learning can be run in a simulation environment to generate predicted trajectories or steering actions based on the real episode, allowing the neural planner to effectively take over driving in a simulation.In addition to or as an alternative to running the neural planner in a simulation environment, using the predicted trajectories or control actions to update the simulation, and using simulated sensor data to generate predicted trajectories or control actions for subsequent iterations, the simulation can be effectively run in a latent space by applying predicted control actions generated using the neural planner to the generative DNN, which predicts a previous distribution representing an estimated future state of the scene, sampling a future state of the scene, and applying the sampled future scene state to one or more decoders corresponding to the neural planner's input(s) to reconstruct the inputs for the next iteration.As such, the neural planner can be exposed to a scenario for which it has not been trained, causing it to drift away from an optimal trajectory due to covariance shift. Accordingly, after a simulated segment in which the neural planner controls navigation and drift, the probabilistic state simulation stack can be executed with its trained navigation policy to roll out the remainder of the episode into latent space and generate a recovery trajectory.In addition to or as an alternative to the neural planner controlling navigation to generate the scenarios from which the navigation policy is to recover, one or more perturbations can be applied to predicted trajectories or control actions generated by the probabilistic state simulation stack's navigation policy. The probabilistic state simulation stack's navigation policy can then be used to introduce the remainder of the episode into latent space and generate a recovery trajectory. Accordingly, one or more recovery trajectories can be recorded and used to train the neural planner and counteract covariance shift.

[0021] As such, a probabilistic state simulation stack can be trained using sensor data to predict what will happen next in the world, and the trained probabilistic state simulation stack can be used for various purposes, such as learning and validating a constituent end-to-end drive stack (e.g., featuring a perception encoder, generative DNN, and drive policy chain), generating simulated ground truth data to train another DNN (e.g.,Training neural planners using recovery trajectories sampled from state distributions predicted by the probabilistic state simulation stack; training other perceptual DNNs or navigation policy DNNs using ground truth traffic scenarios in perspective and / or top-down views generated by sampling from predicted state distributions; simulating future scenarios conditioned on self-actions for planning; and / or other tasks. Co-training generative DNN state estimators with a perceptual encoder, navigation policy, and / or latent space decoder facilitates the training of end-to-end control stacks (e.g., drives) without the use of hand-coded code, improving the accuracy of the resulting trained models and the efficiency of the training process.Thus, the techniques presented here can be used to increase the accuracy and performance of control stacks such as drives; navigation guidelines such as neural planners; perception; and / or downstream tasks that rely on perception.

[0022] With reference to Fig. 2 is Fig. 2 An exemplary probabilistic state simulation stack 200 according to some embodiments of the present disclosure. It is understood that this and other arrangements described herein are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components, or in conjunction with other components, in any suitable combination and at any suitable location. Various functions described herein as being performed by entities may be executed by hardware, firmware, and / or software.For example, various functions can be performed by a processor that executes instructions stored in memory. In some embodiments, the systems, methods, and processes described here can be performed using components, features, and / or functionalities similar to those of the exemplary autonomous vehicle 1000. Fig. 10A- Fig. 10D, the exemplary calculating device 1100 of Fig. 11 and / or the exemplary data center 1200 of Fig. 12 are similar.

[0023] Fig. Figure 2 illustrates some exemplary configurations of a probabilistic state simulation stack 200, including one that can be used during training (illustrated with both solid and dotted arrows), and one that can be used as a control stack after training (illustrated with solid arrows). The probabilistic state simulation stack 200 can include multiple deep neural networks (DNNs) that can be co-trained end-to-end, and the blocks represented by a dotted pattern in Figure 2. Fig. As illustrated in Figure 2, the trained DNNs can represent information exchanged via tensors, be optimized using backward propagated gradients, and be used in an ego-machine (e.g., a vehicle) as an end-to-end control stack (e.g., drive stack).

[0024] In the Fig. In the illustrated embodiment 2, the probabilistic state simulation stack 200 includes a perception encoder 210, a current state estimation network 220, a history aggregation network 230, a navigation policy 240, and a future state estimation network 250. In an exemplary data flow, the probabilistic state simulation stack 200 (which may also be called a probabilistic or stochastic generative world model) accepts input data 205 representing an observed or simulated environment. For example, the input data 205 may be a number of sequential frames of sensor data (e.g., two sequential RGB frames of image data generated by a front-vehicle camera at 10 Hz), a representation of corresponding ego states of the ego machine during the time portion represented by each frame (e.g., ego machine position, velocity, etc.).), a planned navigation route and / or other inputs. Depending on the scenario, the input data 205 can represent a previously recorded human driving session, simulated data, and / or a current driving session for real-time processing, to name a few examples. The input data 205 can be applied to the perception encoder 210, and the perception encoder 210 can provide a scene embedding at time t (illustrated as the observation feature vector o(t) in ). Fig. 2) extract from the input data 205. Taking an exemplary training scenario using a previously recorded human (e.g., expert) driving session, the probabilistic state simulation stack 200 can apply the observation feature vector o(t), a coded representation of the human driving action A at time tl (e.g., embedded using a DNN such as a multi-layer perceptron), and the previous state history H(t-1) to the current state estimation network 220 to compute the posterior latent state ŝ at time t. The current state estimation network 220 (e.g., a generative DNN) can compute a Gaussian distribution from which the posterior state ŝ(t), representing the estimated current state of the scene in the 3D environment (the world), can be sampled. Previous latent states (before time t) can be represented by a trend aggregation network 230 (e.g.a recurrent neural network (RNN) of a gate-controlled recurrent unit (GRU)) is processed to accumulate history H(t-1), and history H(tl) can be provided to the current state estimation network 220 and the future state estimation network 250.

[0025] Navigation policy 240 can retrieve the previous history (e.g., as direct input or processed by a preceding component, as in Fig. 2 illustrated), a representation of a planned navigation route (e.g., as direct input or processed by an upstream component, as in Fig. 2 illustrated) and use the sampled latent state at time t-1 to generate a representation of ego actions a (e.g., steering, accelerating, braking) that the ego machine is to take at time t. The future state estimation network 250 (e.g., a generative DNN) can accept the sampled latent state at time t-1, the history at time t-1, and the steering actions a computed by the navigation policy 240 to generate a prior estimate of the next predicted latent state at time t by computing a Gaussian distribution from which a prior state s(t), representing the prior (estimated future) state of the scene in the 3D environment (the world), can be sampled. Latent state samples can be applied to one or more decoders 260 to generate a representation of the (e.g.,to reconstruct the observed scene represented in the input data 205 (e.g., top-down segmentation mask(s), perspective RGB view images, etc.).

[0026] During training, the probabilistic state simulation stack 200 can be updated using one or more losses, such as the Evidence Lower Bound (ELBO) loss, to minimize the divergence (e.g., Kullback-Leibler (KL)) between the sampled previous latent state s (estimated future state) and the sampled posterior latent state ŝ (estimated observed state). In this example, the observed posterior latent state ŝ, generated from sensor data and human (e.g., expert) actions, effectively represents a desired state demonstrated by a human. Conversely, the previous latent state s, derived from policy actions and world predictions, effectively represents a potential new state that could be either beneficial or detrimental.

[0027] The divergence loss (e.g., KL) can be used to train the navigation policy 240 to select actions that guide the world closer to the states observed in human demonstrations (posterior states). This approach allows the system to explore the planning space during training while being guided by human demonstrations. Consequently, the navigation policy 240 can learn to recover from errors or undesired states and navigate toward favorable states observed in human demonstrations, as in Fig. 1 illustrates.

[0028] The probabilistic state simulation stack 200 can include a selector 225 and / or a selector 235 that stochastically samples posterior and prior states from predicted Gaussian distributions, enabling further exploration. This process effectively trains the navigation policy 240 to mitigate covariance shift. As such, training can be continued through a specific intended goal (e.g., when all DNNs converge), after which the blocks containing a dotted pattern in Fig. Figure 2 illustrates how an end-to-end control (e.g., drive) stack can be used in an Ego machine. Additionally or alternatively, the future state estimation network 2350 can be used as a neural latent dream simulator, facilitating the simulation of long-tail scenarios and / or the use of reinforcement learning to further improve the navigation policy 240 or another machine learning model (e.g., neural network).

[0029] Returning to the input data 205, the input data 205 may comprise or otherwise represent a number of sequential frames of sensor data generated using any number and / or any type of sensor, such as, without limitation, one or more cameras, LiDAR sensors, radar sensors, ultrasonic sensors, and / or other sensor types, such as those described below in relation to the autonomous vehicle 1000. Fig. 10A to Fig. 10D can be described. For example, the sensor(s) can include one or more sensors of an ego-machine, and the sensor(s) can be used to generate (e.g., a temporal sequence of) frames of sensor data (e.g., two sequential RGB frames of image data generated by a front-vehicle camera at 10 Hz) representing objects in the 3D environment around the ego-machine. Additionally or alternatively, the input data can be a planned navigation route (e.g., generated using any known path-planning algorithm), a representation of an ego-state of the ego-machine for the time portion represented by each frame, such as... B. Ego machine position (e.g., detected using one or more inertial measurement unit (IMU) sensors 1066, such as one or more accelerometers, gyroscope(s), magnetic compass(s), magnetometer(s), etc.), Ego machine velocity (e.g.The input data 205 may include, for example, a previously recorded human driving session, simulated data, and / or a current driving session for real-time processing. This data may be estimated using one or more speed sensors, a global positioning system (GPS), etc. Depending on the application, the input data 205 may represent a previously recorded human driving session, simulated data, and / or a current driving session for real-time processing, to name a few examples. As such, the probabilistic state simulation stack 200 can collect, receive, and / or otherwise access a representation of the input data 205 and apply it to the perception encoder 210.

[0030] Depending on the embodiment, the perception encoder 210 (and the others in Fig. The components illustrated in the two examples may be implemented using neural network(s), such as a Convolutional Neural Network (CNN), but this is not intended to be restrictive. For example, and without restriction, the object detection and / or tracking component 110 may include any type of network or machine learning model, such as one or more machine learning models using linear regression, logistic regression, decision trees, support vector machines (SVMs), Naive Bayes, k-nearest neighbor (Knn), K-means clustering, random forest, dimensionality reduction algorithms, gradient amplification algorithms, neural networks (e.g.,Autocoder, Convolutional, Transformer, Recurrent, Perceptrons, Long / Short Term Memory (LSTM), Large Language Model (LLM), Visional Language Model (VLM), multimodal language model, Transformer, Diffusion, Encoder-Only, Decoder-Only, Encoder-Decoder, Hopfield, Boltzmann, Deep Belief, Deconvolutional, Generative Adversarial, Liquid State Machine, etc.) and / or other types of machine learning models.

[0031] In some examples, the machine learning model(s) / neural network(s) can be packaged as a microservice—such as an inference microservice (e.g., NVIDIA NIMs)—which may contain a container (e.g., an operating system (OS)-level virtualization package) that may include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model engine. In some cases, such as where the machine learning model(s) / neural network(s) is / are small enough (e.g., has / have a sufficiently small number of parameters), the model(s) can be contained within the container itself. In other examples – such as when the model(s) is / are large – the model(s) can be hosted / stored in the cloud (e.g., in a data center) and / or can be hosted on-premises and / or at the edge (e.g.,on a local server or computing device, but outside the container). In such implementations, the model can be accessible via one or more APIs—such as REST APIs. As such, and in some implementations, the machine learning models described here can be used as an inference microservice to accelerate the deployment of models on any cloud, data center, or edge computing system while ensuring data security. For example, the inference microservice might include one or more APIs, a pre-configured container for simplified deployment, an optimized inference engine (e.g.,built using a standardized AI model deployment, execution software such as NVIDIA's Triton Inference Server, and / or one or more APIs for high-performance deep learning inference, which may include inference runtime and model optimizations that deliver low latency and high throughput for production applications (such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). The machine learning model(s) described here can be deployed as part of the microservice along with accelerated infrastructure capable of single-command deployment and / or orchestrated and automatically scaled with a container orchestration system on accelerated infrastructure (e.g., from a single device to data center scale).Thus, the inference microservice can include the machine learning model(s) (optimized, for example, for high-performance inference), inference runtime software for executing the machine learning model(s) and providing outputs / responses to inputs (e.g., user queries, prompts, etc.), and enterprise management software for providing integrity checks, identity verification, and / or other monitoring. In some embodiments, the inference microservice can include software to perform on-site replacement and / or updates of the machine learning model(s). During replacement or update, the software performing the replacement / update can maintain user configurations of the inference runtime software and the enterprise management software.

[0032] Fig. Figure 3 illustrates an exemplary perception encoder 300 according to some embodiments of the present disclosure. For example, the perception encoder 300 represents a possible implementation of the perception encoder 210 of Fig. 2. However, the Perception Encoder 300 can be implemented in other applications alongside a probabilistic state simulation stack. In general, the Perception Encoder 300 can accept any of the inputs and can be implemented using any of the architectures and techniques described above in relation to the Perception Encoder 210. Fig. 2 are described. As such, depending on the embodiment, a representation of sensor data (e.g., one or more images from one or more cameras, measured 3D data, such as a detected 3D point cloud, a projected 2D representation of detected 3D data, a temporal sequence of sensor data, etc.), a representation of corresponding detected or calculated ego states, such as corresponding detected or calculated ego positions 320 (e.g., of the ego machine) and / or current vehicle speed (not illustrated), a planned navigation route 365, and / or other types of input can be applied to the perception encoder 300, which can encode the input(s) into a scene embedding (e.g., scene embedding 360, scene embedding 399) of the (e.g., observed, simulated) scene represented in the input(s) (e.g., and which the Observation feature vector o(t) of Fig. 2 can correspond). In some scenarios (e.g., during operation), one or more sensors (e.g., those listed below in relation to the autonomous vehicle 1000 of the Fig. 10A- Fig. 10D are described) are used to generate a set of input(s) (e.g., a sequence of frames) for each time segment (e.g., at a specific frame rate, such as 30 frames per second (fps)), and the set of input(s) for each time segment can be used by the Perception Encoder 300 to extract a scene embedding (which can be used, for example, to perform one or more downstream tasks), whether generated at the same frame rate as the sensor data or at a different frame rate.

[0033] In the Fig. In the illustrated example 3, the perception encoder 300 accepts two consecutive frames 305 (e.g., of RGB image data) representing time points t and (t-1) (e.g., sampled at 10 Hz). Each of the frames 305 can be applied to a feature builder 310 (or feature extractor) to extract a corresponding embedding (and, for example, store the extracted embedding for use in subsequent iteration(s)). In general, the feature builder 310 can be selected to match the type of input it is feature-building and can be implemented using any type of machine learning model or neural network, such as the one described above. In one exemplary implementation, the feature builder 310 instructs a (e.g., self-monitoring) vision transformer backbone (e.g., a 3D processor).Distillation with No Labels (DINO)v2) is used, which extracts a set of image tokens 315 from corresponding frames 305. For example, the Vision Transformer Backbone can divide each image into smaller (e.g., fixed-size) patches, flatten each patch into a vector representation, and linearly transform the vector for each patch into a corresponding token. The Vision Transformer Backbone can combine these patch embeddings with positional codings to preserve spatial information and apply the resulting tokens to one or more transformer layers (e.g., self-attention and / or forward-feedback networks) to learn rich contextual features for each patch and output a representation of the extracted features for each image (e.g., a sequence of image tokens 315, one for each patch, representing the learned features of the entire image).

[0034] In some embodiments, the extracted features (e.g., the image tokens 315) for each of the frames 305 can be applied to one or more transformer layers. Depending on the implementation, the extracted features may include different embeddings (e.g., tokens) corresponding to different regions (e.g., patches) of the input frames 305 (e.g., representative of an extracted feature map for each image). However, since the feature creator 310 may not encode a representation of the position of the patch represented by each token into the token itself, position encodings can be combined (e.g., added 330) with corresponding embeddings (image tokens 315) to integrate information about the relative position of each token (patch) within a corresponding frame (e.g., image) before the resulting embeddings are applied to subsequent transformer sequences.

[0035] In addition or alternatively to combining the embedding (image token) for each of the patches with a position encoding that represents the 2D position of each patch in a corresponding 2D frame, the embedding (image token) for each of the patches can be combined (e.g. added 330) with a perspective embedding that maps the corresponding frame (e.g. image) or patch onto a corresponding section of the 3D scene. For example, the perception encoder 300 may include a perspective encoder 325 that generates a coded representation of the perspective of the sensor(s) that generated the input frames of sensor data, and the perception encoder 300 may combine (e.g., add 330) the resulting sensor perspective coding(s) with the corresponding frame embedding(s) (e.g., image token 315) extracted by the feature builder 310.In general, the perspective or viewing direction of a given sensor can be represented in various ways, such as via the sensor's intrinsic parameters (e.g., represented as a vector, array, matrix, etc.), one or more components of the sensor's position (e.g., its 3D position and / or orientation) relative to a reference 3D coordinate system, such as a coordinate system defined relative to the Ego-Machine (e.g., a rig coordinate system with an origin corresponding to a reference point associated with the Ego-Machine, such as its center of mass or the midpoint between rear axes), or a global 3D coordinate system (worldspace). For example, the Perspective Encoder 325 can generate a representation of the sensor's Ego position relative to worldspace by encoding and / or combining the sensor's extrinsic parameters (e.g., calibrated) (e.g., as a vector, array, matrix, etc.).(as shown), defining the position of the sensor relative to a coordinate system of the ego-machine, with a representation of a detected ego position 320 of the ego-machine relative to space (e.g., detected using one or more inertial measurement unit (IMU) sensors 1066, such as one or more accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.). Thus, in some embodiments, the perspective encoder 325 can generate a coded representation of the sensor's intrinsic parameters and a coded representation of the detected ego position of the sensor (e.g., using any suitable feature builder (or feature builders) implemented using any type of machine learning model or neural network, such as the one described above), and can employ any known technique (e.g., via addition 330, a neural network such as an MLP, etc.).) to combine the resulting embeddings with the position codings (e.g., image token 315) for a corresponding frame of sensor data generated using this sensor. This is merely an example, and other variations may be implemented within the scope of this disclosure (e.g., combining intrinsic parameters, extrinsic parameters, and / or the captured ego position 320 of the ego machine prior to coding, separate coding prior to combining, coding the perspective relative to a different reference coordinate system, etc.).

[0036] In addition to or as an alternative to calculating and encoding a representation of the perspective of the center of a sensor, the perspective encoder 325 can, in some embodiments, calculate and encode the representation of the perspective represented by a subset of the sensor data generated by the sensor, such as a pixel or a patch of an image. For example, the perspective encoder 325 can calculate the perspective or viewing direction of each individual pixel or patch encoded by a corresponding embedding (e.g., image token) extracted by the feature builder 310. Continuing with the example described above, in which the feature builder 310 extracts image tokens 315 representing corresponding patches of each of the frames 305, the perspective encoder 325 can calculate the perspective or viewing direction of a representative (e.g., center) pixel from each patch in each of the frames 305.

[0037] For example, the perspective encoder 325 can generate a representation of the direction of a ray from the center of the sensor through a single pixel in one of the frames 305 by transforming the coordinates of that pixel into a 3D direction in the sensor coordinate system using the sensor's intrinsic parameters (e.g., and normalizing the resulting direction vector), and can generate a representation of one or more components of the sensor's position (e.g., 3D position and / or orientation relative to a reference coordinate system, such as a rig coordinate system or a universe, calibrated extrinsic parameters, captured ego position 320 of the ego machine, etc.). As such, the perspective encoder 325 can generate a coded representation of the ray direction and / or one or more components of the sensor's position (e.g.,using any suitable feature builder (or feature builders) implemented using any type of machine learning model or neural network, such as the one described above), and can use any known technique (e.g., via Addition 330, a neural network such as an MLP, etc.) to combine the resulting embedding with the position encoding (e.g., image token) for a corresponding section of the frame (e.g., the image token for a corresponding patch from which the pixel came).

[0038] As such, the image tokens 315 (each representing a patch) for each of the frames 305 can be combined (e.g., added 330) with corresponding position encodings that represent the 2D position of each patch in a corresponding 2D frame of image data and / or corresponding perspective embeddings that map each patch to a corresponding section of the 3D scene (e.g., along a 3D ray emanating from the center of the sensor and directed toward the 3D scene), and can be applied to a series of transformer encoder layers that use cross-attention 335 and self-attention 342 to process the tokens, capturing relationships between different parts of the frames 305.For example, the perceptual encoder 300 can generate cross-attention queries 332 (Q) from the tokens of the last frame representing time t, generate keys and values ​​(K, V) 333 from the tokens of a previous frame, and cross-attention 335 can apply the queries 332 to the keys and values ​​333 to generate a set of motion feature tokens 340 that represent the currently observed scene and include motion information. Embodiments that combine the image tokens 315 with appropriate perspective embeddings before cross-attention 335 effectively tell the perceptual encoder 300 which parts (e.g., pixels, patches) of different frames 305 represent the same part of the 3D environment, and cross-attention 335 effectively performs a type of triangulation or geometric multiview processing that helps generate better motion features (e.g.,A representation of how the content in the sequence of frame 305 has changed, represented by the motion feature tokens 340). Including perspective embeddings is particularly advantageous when processing real-world data. Taking automotive applications as an example, each vehicle typically has small but significant differences in sensor mounting positions, lens properties, and calibrations, and including perspective embeddings effectively accounts for these differences in vehicle hardware from vehicle to vehicle.

[0039] Continuing with the in Fig. In the exemplary implementation illustrated in Figure 3, the motion feature tokens 340 can be processed by one or more (e.g., four) blocks of self-attention 342 (e.g., with any number of heads, such as eight) to generate a contextualized embed 345 that incorporates contextual information from the scene and / or motion sequence. As such, the contextualized embed 345 can be used as keys and values, and a scene query token Q (represented by the scene query 350) can be used to cross-reference the keys and values ​​(contextualized embed 345) 355 to output a scene embed 360 (e.g., representing a single vector) of dimension 512. In some embodiments, this scene embed 360 can be provided to one or more downstream tasks.

[0040] In addition to or as an alternative to using the scene embedding 360 for one or more downstream tasks, in some embodiments the scene embedding 360 is combined (e.g., concatenated, added 395) with a representation of one or more other inputs, such as a planned navigation route 365, an ego state of the ego machine (e.g., a detected position, orientation, velocity, ego movement 380, etc.), and / or otherwise. For example, the planned navigation route 365 can be generated using any known technique (e.g., any path planning algorithm) and can be represented in a variety of ways (e.g., a rasterized binary mask visualizing a (e.g., top-down) view of a planned trajectory; a sequence of 2D waypoints; a list, array, queue, or other representation of a sequence of navigation route commands (e.g.,turn right in 100 feet, continue straight for 30 miles, turn left at the next stop light), which can be encoded using one-hot coding, embedding vectors, RNN(s) to handle sequential processing, positional coding and transformers, etc.) and a corresponding feature builder 370 (e.g., using any suitable feature builder (of feature builders) implemented using any type of machine learning model or neural network, such as the one described above) can be used to generate a corresponding embedding 375, which can be combined with the scene embedding 360 (e.g., added 395, concatenated with dimensionality reduction).

[0041] In some embodiments, one or more states of the ego machine can be detected, provided to the perception encoder 300, feature-formed, and combined with the scene embedding 360. For example, the speed of the ego machine can be estimated (e.g., using a velocity sensor, a global positioning system (GPS), etc.), encoded into a velocity embedding, and combined with the scene embedding 360 (not illustrated). In some embodiments, a representation of the ego motion 380 (e.g., speed, relative position) of the ego machine can be detected (e.g., estimated using a velocity sensor or GPS) or generated (e.g., by taking the difference between successive detected ego positions 320), and a corresponding feature-former 385 (e.g.,Using any suitable feature builder (or features) implemented using any type of machine learning model or neural network, such as the one described above, a corresponding embedding 390 can be generated, which can be combined with the scene embedding 360 (e.g., added to 395, chained with dimensionality reduction). These are just a few examples, and other components of a detected ego state (e.g., represented by GPU data, IMU data, and / or otherwise) can additionally or alternatively be feature-built and combined with the scene embedding 360. Thus, the resulting scene embedding 399 can be provided to one or more downstream tasks.

[0042] Thus, scene embedding 360 and / or scene embedding can be used to display one or more aspects of an observed scene, such as visual information, motion information, the ego state of an ego machine, a planned navigation route and / or other types of information.

[0043] The Perception Encoder 300 can be used to generate a scene embed (e.g., one that incorporates motion data) for various applications. For example, the scene embed can be used as an input for 3D perception or reconstruction tasks, such as object detection and classification (e.g., identifying pedestrians, vehicles, traffic signs, obstacles, etc.), semantic segmentation (e.g., labeling one or more elements in the scene by class), depth map extraction, trajectory prediction, path planning, navigation control (e.g., by a control stack that predicts one or more actions for the ego machine to perform), and / or localization or mapping (e.g., generating a 3D representation of the environment or localizing a 3D position within the environment), to name a few example tasks.

[0044] In some embodiments, implementing the feature builder 310 with a pre-trained vision transformer backbone facilitates the training of a robust perception encoder 300 on real-world data. Thus, in some embodiments, the perception encoder 300 can be used in a simulator to validate a downstream task (e.g., driving or other navigation) without the need for fine-tuning. Using an embodiment that trains the perception encoder 300 as part of an end-to-end control stack (e.g., as in Fig. 2) If validation is successful, the trained end-to-end control (e.g., drive) stack can be deployed directly on the ego machine (e.g., vehicle), in some embodiments where it has been trained exclusively on real-world data.

[0045] Returning to the one in Fig. 2 illustrated exemplary probabilistic state simulation stack 200, the perception encoder 210 can extract an observation feature vector o(t) for each time part, and the observation feature vector o(t) can be applied to the current state estimation network 220.

[0046] The current state estimation network 220 can include a generative DNN, such as one or more stochastic neural networks (e.g., Bayesian neural network(s), variational autoencoders (VAEs)), that model the posterior state of the scene. Specifically, the current state estimation network 220 can model a posterior distribution of a latent state that is conditioned on observations represented by the observation feature vector o(t). For example, the current state estimation network 220 can model a surrogate distribution q(s). (t)|o(t), H(tl), A(tl)) as a Gaussian distribution (or conditioned on a(tl) instead of A(tl) during other scenarios, such as deployment).

[0047] A(tl) and a(tl) represent navigation actions and can include coded representations of steering, acceleration, and / or braking states (e.g., concatenated and embedded in a 1D vector). A refers to previously recorded human (e.g., expert) actions that can be input into the current state estimation network 220 during training (e.g., through a selector 215). In some scenarios (e.g., during the use of the blocks illustrated by a dotted pattern in a control stack), a refers to the actions generated by the navigation policy 240 that can be input into the current state estimation network 220 (e.g., through selector 215). In some scenarios (e.g.,When switching control from a human driver to the control stack, a representation of the most recent human actions can be entered into the current state estimation network 220 as a(tl) for a number of iterations prior to the switch (via selector 215) to enter the actions generated by the navigation policy 240 into the current state estimation network 220 as a(tl). In some scenarios (e.g., when applying an action or trajectory predicted by a navigation policy, such as navigation policy 240, to a control algorithm, such as model predictive control (MPC), which generates an optimized control action), a representation of the executed action, as opposed to the action predicted by navigation policy 240, can be entered into the current state estimation network 220 as a(tl).

[0048] The state history H(t-1) can effectively aggregate previous scene states deterministically using a hidden state of the history aggregation network 230, which can employ any known technique to combine information from past states (e.g., a GRU RNN). Thus, the history aggregation network 230 can effectively serve as a memory that is updated at each time interval based on a current input (e.g., ŝ(tl) or s(t-1)) and the previous hidden state. The latent states ŝ(tl) and s(t-1) represent the evolution of the scene in a latent space from time t-1 to t. These states can be represented as 1D vectors (e.g., with dimensions 1 x 512).

[0049] Thus, the current state estimation network 220 can estimate or approximate a posterior probability distribution of a true hidden state using an approximation distribution (e.g., a Gaussian distribution) learned by maximizing the Evidence Lower Bound (ELBO) using any known technique. Accordingly, the current state estimation network 220 can form a generative DNN that accepts the observation feature vector o(t) (e.g., scene embedding), the past history H(t-1) up to time t-1, and embedded human (e.g., expert) actions A(tl) (during training) at time t-1, and computes parameters of a Gaussian distribution that can be sampled to generate a posterior latent state sample ŝ(t). This state sample value for time t can be used recursively as f(tl) for the next iteration of the probabilistic state simulation stack 200.

[0050] The future state estimation network 250 can include a generative DNN, such as one or more stochastic neural networks (e.g., Bayesian neural network(s), variational autoencoders (VAEs)), that model the previous (estimated future) state of the scene. Specifically, the future state estimation network 250 can model stochastic latent space-world transitions from s(t-1) to s(t) conditioned on ego action a(tl) and trajectory H(t-1). As such, the future state estimation network 250 can accept as input a representation of the past trajectory H(t-1) (which may include, for example, past state samples s() up to and including time t-1) and the ego action a(tl) generated by the navigation policy 240 for time step t-1. In some embodiments, the future state estimation network 250 may additionally or alternatively include a representation of a planned navigation route (e.g.Embedding using a suitable feature builder (suitable feature builder) as input is accepted. Accordingly, the future state estimation network 250 can form a generative DNN that predicts a latent state as a Gaussian distribution, which can be sampled to generate a previous (estimated future) latent state sample s(t). Once trained, the future state estimation network 250 can be used to introduce new states without corresponding observations (in what can be considered an imagination mode) up to a certain horizon. These rollouts can be used to re-simulate long-tail situations not seen in the training dataset.

[0051] Depending on the implementation, Navigation Policy 240 can generate a representation of a control action a(t) and / or a corresponding trajectory that the Ego machine is to execute based on the latent state at time t (e.g., ŝ(t) during deployment, ŝ(t) or s(t) during training, as explained in more detail below). Navigation Policy 240 can be implemented using any known technique, such as those that employ neural networks to predict or generate a planned trajectory or action (neural planners), classical control algorithms such as MPC, a combination thereof (e.g., a neural planner that predicts an action followed by a classical control algorithm that optimizes the predicted action), and / or otherwise. In some embodiments, Navigation Policy 240 can employ a neural planner (e.g.,(implemented using a transformer architecture) that predicts a trajectory (e.g., instead of directly predicting acceleration and steering), and uses MPC during deployment to improve the transfer from training to inference on the ego machine (e.g., because MPC considers the dynamics of the ego machine and uses a predicted ego trajectory as a reference to track or guide its steering actions). Thus, Navigation Policy 240 can effectively map the latent state of the scene to a desired navigation action (e.g., steering angle, acceleration or throttle, brake pressures, higher-level navigation decisions such as lane changes, etc.) and / or a trajectory that the ego machine should follow. During training, the predicted action or trajectory can be compared with a suitable ground truth to human (e.g.,The expert action is compared and used to update the probabilistic state simulation stack 200. During deployment, the predicted action or trajectory can be provided to the ego machine's control system(s) to guide its navigation.

[0052] For example, navigation policy 240 may include or be associated with a behavior planner, a behavior selector, a route planner, and / or a lane keeper. The behavior planner may be responsible for deciding the overarching actions that the ego-machine (e.g., the vehicle) should take in response to the current and predicted state of the environment. Taking an ego-vehicle as an example, this may include decisions such as whether to change lanes, adjust speed, overtake another vehicle, or yield to pedestrians. These decisions may be made based on predefined rules or learned guidelines. Thus, the behavior planner can embody the decision-making process that guides how the ego-machine should behave in different situations.In some embodiments, the behavior planner can generate possible behaviors or actions, and the behavior selector can choose the most appropriate one based on the current situation, goals (such as safety, comfort, and efficiency), and potentially learned preferences. This selection process can effectively choose an optimized action for the ego machine to take. The route planner can generate a higher-order path for the ego machine to follow, typically over a longer distance (e.g., city blocks or miles). Thus, the route planner can be responsible for navigation and ensuring that the ego machine reaches its destination. Navigation policy 240 (e.g., embodied in the behavior planner and selector) can define this route (e.g., whether as direct input or processed by an upstream component, as in [reference to example]). Fig. (Illustrated in Figure 2) to use this information to make tactical decisions about how to follow the planned path in real time. The lane planner can be responsible for more immediate tactical decisions regarding lane keeping, lane changes, and / or interactions with nearby traffic based on the path generated by the route planner. Thus, the navigation policy 240 can determine when and how the Ego machine should change lanes or merge, which the lane planner can then execute.

[0053] During training (e.g., after pretraining the navigation policy 240 using imitation learning or behavior cloning), the probabilistic state simulation stack 200 can iterate or repeat over time using a selector 235 to select either ŝ(t) or s(t) as input to the navigation policy 240 (and optionally one or more decoders 260) for the current iteration with a designated probability, and / or using the same selection (or using a selector 225 to select either ŝ(t) or s(t)) as input to the history aggregation network 230 for the next iteration with a designated probability. This probabilistic sampling of latent states during training mitigates covariance shifting because s(t) is sampled stochastically and may include unusual long-tail states from which recovery is possible.

[0054] The sampled latent state can effectively encode a representation of a current (or future) scene and can be used by the decoder(s) 260 to reconstruct a representation of the current (or future) scene. For example, the decoder(s) 260 can include one or more neural networks that decode latent state samples into perspective RGB view images (e.g., image 265), semantic top-down (bird's-eye view) views, semantic segmentations (e.g., segmentation 270), a representation of boundary shapes for detected objects, a representation of track positions, and / or other representations. By iterating the probabilistic state simulation stack 200 over time, the decoder(s) 260 can be used to reconstruct videos or other temporal sequences of reconstructed data representing a (current or future) moving scene (e.g.with moving traffic, updating lanes, object labels, etc.). The decoder(s) 260 can be implemented using one or more generative networks, such as Generative Adversarial Networks (GANs) or latent diffusion models (e.g., stable diffusion). The decoder(s) 260 can be trained using reconstruction losses against appropriate input data 205 (e.g., sensor data, such as perspective RGB view images) or ground truth targets (e.g., top-down segmentation masks). Thus, the decoder(s) 260 can be used to decode a sequence of latent states into a temporally coherent sequence of reconstructed data, such as a perspective RGB view video (which can be useful, for example, for system inspection and visualization). Fig. 4A- Fig. Figure 4B illustrates some exemplary reconstructed representations of an estimated state of a scene according to some embodiments of the present disclosure. For example, it illustrates Fig. 4A a simulated RGB input image 405, a Ground Truth top-down segmentation mask 410 and a corresponding reconstructed top-down segmentation mask 415 (including representations of an ego vehicle 420, other vehicles on the road, such as vehicle 425, lane lines and pedestrians, such as pedestrian 430) and corresponding predicted acceleration and steering commands. Fig. 4B illustrates a real RGB input image 450, a corresponding reconstructed top-down segmentation mask 455 (including representations of other vehicles on the road and lane lines) and corresponding predicted acceleration and steering commands.

[0055] The probabilistic state simulation stack 200 (e.g., all DNNs in the probabilistic state simulation stack 200) can be trained using variational inference by maximizing ELBO loss, which may include log likelihood of observations (e.g., perspective RGB view images, semantic segmentation(s), predicted ego action(s), etc.) and negative KL divergence between previous and posterior distributions estimated by the future state estimation network 250 and the current state estimation network 220, respectively. Log-likelihood of observations can be used to train the probabilistic state simulation stack 200 to learn a latent space that can be decoded (e.g., into real RGB images), and the future state estimation network 250 and the current state estimation network 220 can learn to predict and estimate latent states that minimize their KL divergence.It should be noted that the use of decoder(s) 260 during training is optional, and the probabilistic state simulation stack 200 can be trained without decoder(s) 260 using KL divergence between ŝ(t) or s(t) and (e.g., L1) loss between predicted and human-demonstrated actions to guide the training. Even without the use of reconstructed scene signals (e.g., detected lanes, pedestrians, traffic signs, etc.), the probabilistic state simulation stack 200 and its navigation policy 240 should still learn to control navigation well.

[0056] The training process can have different phases. For example, the perception encoder 210 and the decoder(s) 260 can be trained and / or converge first, followed by the future state estimation network 250 and finally the navigation policy 240. To avoid overfitting, a cyclic learning rate—a planning technique in which the learning rate oscillates between a minimum and a maximum value during training, rather than steadily decreasing or remaining constant—can be used. This approach allows the learning rate to periodically increase and decrease in cycles, which helps the probabilistic state simulation stack 200 escape local minima and saddle points, thus improving convergence and generalization.

[0057] As such, one or more components of the probabilistic state simulation stack 200 can be used in various ways. In one example, shown by the solid lines in Fig. As illustrated in Figure 2, the blocks with a dotted pattern can be used in Fig. Figure 2 illustrates (the perception encoder 210, the current state estimation network 220, the navigation policy 240, and the history aggregation network 230) being used in an ego-machine (e.g., a vehicle) as an end-to-end control stack (e.g., drive stack). For example, this chain of components can be implemented in real time on a system-on-chip (such as the SoC(s) 1004 of Fig. 10C) an ego machine (e.g., an automotive SoC like NVIDIA's DRIVE Orin™ SoC, which can serve as a central computer for in-vehicle data processing). Other types of SoCs that can be used in robots or ego machines include AI-optimized SoCs (e.g., NVIDIA Jetson Series), robotics SoCs (e.g., for robotics applications such as drones, service robots, or industrial robots), industrial SoCs (e.g., for industrial robots or automation systems for factory automation, robotics, or control systems), drone and unmanned aerial vehicle (UAV) SoCs (e.g., for handling flight control, navigation, and / or real-time video processing), and / or others. By training the control stack as part of the probabilistic state simulation stack 200 with the future state estimation network 250, the navigation policy 240 learns scene states (e.g.,to handle new traffic scenarios that were not represented in previous round(s) of imitation learning.

[0058] In addition to or as an alternative to training a probabilistic state simulation stack with an end-to-end navigation control stack to control the navigation of a real or simulated ego object, a probabilistic state simulation stack can be trained with a manipulation control stack to control the manipulation of one or more objects by a real or simulated ego object (e.g., a robot arm, a digital character, or an avatar, etc.). While in navigation control a control stack can use inputs such as sensor data, the ego state of the ego object, and / or a planned navigation route to perform path planning (e.g., predicting a trajectory) and / or predict navigation controls (e.g., acceleration, steering, braking), a manipulation control stack can use inputs such as sensor data, the ego state of the (e.g., robot) manipulator (e.g., joint angle, end effector position), object properties (e.g.,Size, weight, texture) and / or scene context (e.g., positions of other objects, obstacles) are used to perform grasping or manipulation planning (e.g., predicting grasping location, grasping type, gripper orientation, arm trajectory) and / or manipulation controls (e.g., joint angles and velocities, end effector position and orientation, force and torque to apply). As such, one or more components of a trained probabilistic state simulation stack can be deployed as an end-to-end manipulation control stack and used to move objects conditioned on the overall scene represented in the inputs.

[0059] In some embodiments, one or more components of the probabilistic state simulation stack 200 can be used to train or test another neural network, navigation policy, and / or control stack. For example, the other neural network, navigation policy, and / or control stack can participate in a training session using a training dataset, and the probabilistic state simulation stack 200 can be used to process an input representation of this training data and generate additional training data to extend the training dataset.For example, the probabilistic state simulation stack 200 can be run with the future state estimation network 250 (and / or the current state estimation network 220) to generate and sample a latent state, and the decoder(s) 260 can be used to decode the sampled latent state and reconstruct one or more representations of a future scene (or a current scene represented by the input data). Taking an example where the other neural network, the navigation policy and / or control stack, accepts one or more (e.g., perspective) images and detects objects, the decoder(s) 260 can be used to decode one or more sampled latent states into input training (perspective) image and ground truth bounding forms, classifications, and / or other properties of detected objects.Taking an example where the other neural network, the navigation policy and / or the control stack (e.g., top-down), accepts semantic segmentation(s) and predicts a control action, the decoder(s) 260 can be used to decode one or more sampled latent states into semantic segmentation(s) used for input training data, and the navigation policy 240 can be used to predict control actions (and / or corresponding trajectories) used for ground-truth training data. Generating sampled future states can effectively simulate scenarios that are difficult to observe (e.g., because they may be unusual or dangerous), such as scenarios where a driver swerves into or crosses into an oncoming lane.Thus, the probabilistic state simulation stack 200 can be used to generate unlimited data representing various traffic scenarios (e.g., displayed in perspective and / or top-down views) for any traffic scenario, weather, time of day, and / or other conditions by sampling predicted latent states. This data can be used to augment existing training datasets and / or to train or test other perceptual DNNs, navigation policy DNNs, or control stacks via supervised learning.

[0060] In some embodiments, reconstructed (e.g., simulated) scenarios generated by the probabilistic state simulation stack 200 can be used for re-simulation. Instead of (or in addition to) replaying pre-recorded data from real-world driving scenarios in a simulation environment (e.g., NVIDIA's DriveSIM), reconstructed (e.g., simulated introductory) scenarios generated by the probabilistic state simulation stack 200 can be input into and replayed in the simulation environment to test and / or train a neural network, navigation policy, and / or control stack.Re-creating and replaying scenarios generated by the probabilistic state simulation stack 200 allows the neural network, navigation policy and / or control stack to interact with a potentially richer simulated environment in a feedback loop during re-simulation, enabling more dynamic training and testing, especially for simulated long-tail situations.

[0061] In some embodiments, a probabilistic state simulation stack (e.g., the probabilistic state simulation stack 200 of Fig. 2) can be used to generate recovery scenarios (e.g., recovery trajectories) for another neural planner. Fig. Figure 5 illustrates an exemplary technique for generating recovery trajectories according to some embodiments of the present disclosure. In a higher-level overview, a neural planner (e.g., pre-trained using imitation learning) can be used to control an Ego-Machine 505 in a simulation (e.g., generated using a simulation environment such as DriveSIM, generated in a latent space of a probabilistic state simulation stack) – or one or more control commands generated by the probabilistic state simulation stack can be perturbed – so that the Ego-Machine 505 begins to drift away from the center of track 550.As such, a representation of this simulated initial trajectory 510 can be applied to the probabilistic state simulation stack, and the control of the ego machine 505 in the simulation can be switched to the navigation policy of the probabilistic state simulation stack. Accordingly, the ego machine 505 should recover from the drift trajectory by returning to the center of track 550, and a representation of this simulated recovery trajectory 520 (e.g., a reconstructed simulated introduction scenario generated by the probabilistic state simulation stack) can be used to train the neural planner to avoid drifting out of track 550, thereby mitigating covariance shift.

[0062] In particular, a probabilistic state simulation stack (e.g., the trained probabilistic state simulation stack 200 of Fig. 2) be initialized with a real episode to accumulate state. Fig. Figure 6A illustrates an exemplary technique for initializing a probabilistic state simulation stack according to some embodiments of the present disclosure. In particular, it illustrates Fig. 6A a configuration of the (e.g. trained) probabilistic state simulation stack 200 of Fig. 2, in which a representation of a real episode (e.g., a sequence of frames of sensor data, a recorded ego-machine state, a planned navigation route) is applied to the perception encoder 210, a representation of recorded driving action(s) (e.g., coded acceleration, steering, and / or braking commands) is used as a(tl) and applied to the current state estimation network 220, and this configuration is iterated in a loop for a certain duration, number of input frames, number of time segments, etc., so that the history aggregation network 230 can accumulate a representation of the latent state of the scene.

[0063] Returning to Fig. In some embodiments (e.g., after initializing at least some of the components of the probabilistic state simulation stack), a representation of an initial trajectory (e.g., trajectory 510) in which the Ego-Machine 505 deviates from an optimal or target trajectory can be generated and recorded. For example, a neural planner (e.g., pre-trained using imitation learning) can be used to control the Ego-Machine 505 in a simulation environment. Fig. Figure 6B illustrates an exemplary technique for generating an initial trajectory using a neural planner 620 to update a simulation environment 610. In general, the simulation environment 610 (e.g., DriveSIM) can generate a realistic virtual environment for testing and / or training neural networks, such as those used in perception or autonomous driving systems. In an exemplary automotive application, the simulation environment 610 can generate simulated driving scenarios, road conditions, and / or other aspects of various environments using simulated sensor models, vehicle dynamics, and / or interactions with traffic, pedestrians, or other dynamic or static objects or parts of the environment. Depending on the input(s) accepted by the neural planner 620, a corresponding representation of a simulated scene (e.g.,The simulation environment 610 recreates the last state of the real-world episode (used to initialize the probabilistic state simulation stack) and / or inputs it into the neural planner 620 to predict a control action and / or trajectory for the ego machine 505 to take. The predicted control action and / or trajectory can be used to control the ego machine 505 in the simulation environment 610 and update the simulation. A representation of the updated scene in the simulation (e.g., simulated sensor data) can be applied to the neural planner 620 to predict subsequent actions for later iterations. The process can be repeated for a certain duration, number of input frames, number of time segments, etc., to generate an initial trajectory (e.g., trajectory 510).In some embodiments, the probabilistic state simulation stack 630 (which, for example, is the configuration of the in . Fig. 6A illustrated probabilistic state simulation stack 200 can correspond to) a corresponding representation of the simulated scene generated by the simulation environment 610, as the input data 205 accept, and the control action and / or trajectory predicted by the neural planner 620 (and / or a corresponding control algorithm) can be used as a(tl) and applied to the current state estimation network 220, so that the history aggregation network 230 can update the accumulated history to include the simulated scene.

[0064] Additionally or alternatively, in some embodiments an initial trajectory for the Ego machine 505 can be generated by applying one or more perturbations (e.g., a small modification or disturbance) to a predicted control action (e.g., steering, accelerating, braking) or trajectory (e.g., a planned path) defined by a navigation policy and / or a control stack of the probabilistic state simulation stack (e.g., the probabilistic state simulation stack 200 of Fig. 2) be generated. For example, a corresponding representation of a simulated scene (e.g., re-creating, in the simulated environment, the last state of the real-world episode used to initialize the probabilistic state simulation stack) can be fed into a control stack of the probabilistic state simulation stack to predict a steering action and / or trajectory that the Ego Machine 505 is intended to take, and the predicted steering action and / or trajectory can be perturbed. For example, if the navigation policy of the control stack predicts a steering angle of 5 degrees, a perturbation can add or subtract a small amount (e.g., 0.5 degrees), producing a slightly altered action. In another example, if a predicted trajectory suggests a smooth curve with a series of waypoints, perturbations can slightly adjust these waypoints from the intended path (e.g.,Perturbations can be random (e.g., adding noise), predefined modifications (e.g., consistently increasing or decreasing the steering angle by a fixed amount), and / or otherwise. Thus, the modified steering action and / or trajectory can be used to control the Ego Machine 505 in the simulation and update the simulation. A representation of the updated scene in the simulation (e.g., simulated sensor data) can be applied to the control stack of the probabilistic state simulation stack to predict subsequent actions for later iterations (e.g., to perturb). The process can be repeated for a certain duration, number of input frames, number of time segments, etc., to generate an initial trajectory (e.g., trajectory 510).

[0065] Additionally or alternatively, in some embodiments an initial trajectory for the Ego-Machine 505 can be generated using a neural planner (e.g., pre-trained using imitation learning) to position the Ego-Machine 505 in the latent space of a probabilistic state simulation stack (e.g., the probabilistic state simulation stack 200 of Fig. 2) to control. Fig. Figure 6C illustrates an exemplary technique for generating an initial trajectory using a neural planner 640 to update a simulation in the latent space of a probabilistic state simulation stack according to some embodiments of the present disclosure. In particular, it illustrates Fig. 6C a configuration of the (e.g. trained) probabilistic state simulation stack 200 of Fig. 2, in which control actions predicted by the neural planner 640 and / or a corresponding control algorithm 650 (e.g., MPC) can be used as a(tl) and applied as input to the future state estimation network 250 to predict (e.g., simulate) and sample a future latent state of the scene, which can be used by the decoder(s) 260 to reconstruct a corresponding input for the neural planner 640 during a subsequent iteration of the loop.During a first iteration, depending on which input(s) the neural planner 620 accepts, a corresponding representation of the last state of the real episode, used to initialize the probabilistic state simulation stack, can be applied to the neural planner 640 to predict a control action and / or trajectory for the ego machine 505 to take (or a control algorithm 650 can predict the control action based on a trajectory predicted by the neural planner 640). The predicted control action can be used as a(tl) and applied as input to the future state estimation network 250, which can effectively perform a simulation in latent space by predicting and sampling a latent state representing a future scene state in which the ego machine 505 has executed the predicted control action or trajectory.Thus, the decoder(s) 260 can be used to decode the sampled latent state and reconstruct one or more representations of the future scene that the neural planner 640 accepts. The reconstructed representation(s) of the simulated updated scene sampled from the latent space can then be applied to the neural planner 640 to predict subsequent actions for later iterations. In each iteration, the history aggregation network 230 can use the sampled state from the previous iteration (or from the initialization phase for the first iteration) to update the accumulated history of the scene for the future state estimation network 250.Thus, the neural planner 640 can be exposed to a scenario for which it was not trained, causing it to drift away from an optimal trajectory due to covariance shift. This configuration can iterate in a loop for a certain duration, number of input frames, number of time segments, etc., to generate an initial trajectory (e.g., trajectory 510).

[0066] And so, back to... Fig. Regardless of the technique used to generate the initial trajectory (e.g., 510) that deviates from an optimal or target trajectory, a probabilistic state simulation stack can be used to recover back towards the optimal or target trajectory (e.g., back to the center of track 550). For example, the blocks that are connected with a dotted pattern in Fig. 2 illustrated (the perception encoder 210, the current state estimation network 220, the navigation guideline 240, and the history aggregation network 230) are executed in a simulation environment to generate and record a representation of a recovery trajectory. In some embodiments, a representation of the last state of the scene in which the initial trajectory was generated can be applied to the perception encoder 210, a representation of the last recorded driving action(s) (e.g., coded acceleration, steering, and / or braking commands) from the initial trajectory can be used as a(tl) and applied to the current state estimation network 220, and the accumulated history of the scene (e.g., initialized in the in Fig. The configuration illustrated in Figure 6A (updated to include a representation of the scene in which the initial trajectory was generated) can be used as H(t-1) and applied to the current state estimation network 220. Thus, the navigation policy 240 can predict a control action (or a trajectory, and a corresponding control algorithm can optimize the control action). The predicted control action and / or trajectory can be used to control the ego machine 505 in the simulation environment and update the simulation. A representation of the updated scene in the simulation (e.g., simulated sensor data) can be applied to the perception encoder 210 for the control stack of the probabilistic state simulation stack 200 to predict subsequent actions for subsequent iterations. This configuration can be used for a certain duration, number of input frames, number of time segments, etc.Iterate to generate the recovery trajectory 520. Thus, a representation of the recovery trajectory 520, the corresponding control actions, and / or the simulated environment in which the recovery trajectory 520 was generated can be recorded and used as a training episode to train a neural network, a navigation policy, and / or a control stack such as a neural planner that generated the initial trajectory (e.g., using imitation learning).

[0067] Now, referring to Fig. 7-9 Each block of procedures 700, 800, and 900 described herein comprises a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. The procedures can also be embodied as computer-usable instructions stored on computer storage media. The procedures can be provided by a standalone application, a standalone service, a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name just a few. Additionally, the procedures can be described by way of example with respect to the probabilistic state simulation stack 200 of Fig. 2 or the perception encoder 300 from Fig. 3. However, these procedures may additionally or alternatively be performed by any system or any combination of systems, including but not limited to those described herein.

[0068] Fig. Figure 7 is a flowchart illustrating a method 700 for generating a scene embedding representing an environment according to some embodiments of the present disclosure. The method 700, at block B702, involves generating, based at least on applying a representation of a temporal sequence of sensor data generated using one or more sensors of an ego-machine in an environment, and a coded representation of one or more corresponding perspectives of the one or more sensors to one or more first neural networks (NNs) comprising one or more encoder networks, a scene embedding representing the environment. Using an exemplary encoder network, such as the perception encoder 300 of Fig. 3. The frames 305 (e.g., from RGB image data) can be applied to the feature creator 310 (or feature extractor) to extract a corresponding embedding (e.g., a set of image tokens 315 for each of the frames 315), and the perspective encoder 325 can generate a coded representation of the perspective of the sensor(s) that generated the input frames of sensor data. Thus, the Perception Encoder 300 can combine (e.g., add 330) the resulting sensor-perspective encoding(s) with the corresponding frame embedding(s) (e.g., picture token 315) extracted by the Feature Builder 310 and apply the resulting embeddings to subsequent transformer layer(s) (e.g., cross-attention 335, self-attention 342, cross-attention 355) to generate a scene embedding (e.g., scene embedding 360 or scene embedding 399).

[0069] Procedure 700, at block B704, involves generating, based at least on applying a representation of scene embedding to one or more second NNs, one or more outputs. For example, with respect to Fig. The perception encoder 210, the current state estimation network 220, the navigation policy 240, and the history aggregation network 230 in an ego machine (e.g., a vehicle) can be used as an end-to-end control stack (e.g., drive stack). Thus, an observation feature vector o(t), extracted by the perception encoder 210, can be applied to the current state estimation network 220 to generate and sample a posterior state ŝ(t), and the navigation policy 240 can retrieve the previous history (e.g., as direct input or processed by an upstream component, as in Fig. 2 illustrated), a representation of a planned navigation route (e.g., as direct input or processed by an upstream component, as in Fig. (2 illustrated) and use the sampled latent state at time t-1 to generate a representation of ego actions a (e.g., steering, accelerating, braking) that the ego machine should take at time t. More generally, the scene embedding can be used as an input for 3D perception or reconstruction tasks, such as object detection and classification (e.g., identifying pedestrians, vehicles, traffic signs, obstacles, etc.), semantic segmentation (e.g., labeling one or more elements in the scene by class), depth map extraction, trajectory prediction, path planning, navigation control (e.g., by a control stack that predicts one or more actions for the ego machine to perform), and / or localization or mapping (e.g., generating a 3D representation of the environment or localizing a 3D position within the environment), to name a few example tasks.

[0070] Method 700, in block B706, involves controlling one or more operations of the Ego machine based on at least one or more outputs. The output(s) of the one or more second neural networks typically depend on the applicable task and / or implementation and can be provided to appropriate control component(s) of the Ego machine. In general, the applicable control component(s) can depend on the task and / or implementation. Taking automotive applications as an example, in some embodiments the control component(s) are part of an ADAS, such as the ADAS system 1038 from [reference missing]. Fig. 10C, and the control component(s) can coordinate and / or manage one or more functions within the ADAS. In general, the ADAS can use any known technique to assess the vehicle's surroundings, identify potential risks or hazards, and / or implement autonomous driving features such as adaptive cruise control, automatic emergency braking, lane keeping assist, and / or collision avoidance systems, to name a few examples. Using an exemplary implementation where the output(s) represent one or more control action(s) a (e.g., steering, accelerating, braking) that the ego machine is to take at time t, the control component(s) can cause the ego machine to take the predicted control action(s).Using some other possible detection tasks as examples, if the output(s) indicate a predicted trajectory for the ego machine to follow, the control component(s) can convert the trajectory into the corresponding control action(s) and cause the ego machine to take the control action(s). If the output(s) indicate one or more detected obstacles, the control component(s) can cause the ego machine to avoid the detected obstacles.

[0071] Using Driver Monitoring System (DMS) tasks such as driver drowsiness or distraction detection, if the output(s) indicate that the driver is not attentive or in a ready-to-drive position, the control component(s) may initiate one or more warnings (audible, visual, or haptic) to regain the driver's attention, adjust one or more driver assistance features such as adaptive cruise control (e.g., increasing the following distance or reducing the vehicle's speed) or lane keeping assist (e.g., being more proactive in correcting lane deviations), or cause the ADAS to perform one or more safety interventions (e.g., automatic braking, emergency steering, transitioning to autonomous driving mode), etc.For some exemplary occupant monitoring system (OMS) tasks, if the output(s) indicate the presence of a detected occupant, the control component(s) may activate safety features, such as airbags or seatbelt reminders, or activate systems, such as climate control and / or infotainment. If the output(s) indicate a detected gesture, the control component(s) may perform or initiate a corresponding function (e.g., adjusting the volume, changing the temperature, opening windows). The foregoing detection tasks and corresponding controls are intended only as non-limiting examples, and variations and other detection tasks may be implemented within the scope of this disclosure.

[0072] Fig. Figure 8 is a flowchart showing a method 800 for operating at least part of a probabilistic state simulation stack as a control stack of an ego machine according to some embodiments of the present disclosure. The method 800, at block B802, involves training a probabilistic state simulation stack comprising one or more neural encoder networks (NNs) based at least on probabilistic sampling of one or more estimated scene states. For example, with respect to the probabilistic state simulation stack 200 of Fig. 2. Iterate or repeat the probabilistic state simulation stack 200 over time using a selector 235 to select either ŝ(t) or s(t) as input to the navigation policy 240 (and optionally one or more decoders 260) for the current iteration with a certain probability, and / or using the same selection (or using a selector 225 to select either ŝ(t) or s(t)) as input to the history aggregation network 230 for the next iteration with a designated probability. Thus, in each iteration, the probabilistic state simulation stack 200 (e.g., all DNNs in the probabilistic state simulation stack 200) can be updated using variational inference by maximizing ELBO loss, the log likelihood of observations (e.g., perspective RGB view images, semantic segmentation(s), predicted ego action(s), etc.).) and may include negative KL divergence between previous and posterior distributions, estimated by the future state estimation network 250 and the current state estimation network 220, respectively. Probabilistic sampling of latent states during training attenuates covariance shifting because s(t) is sampled stochastically and may include unusual long-tail states from which recovery is required.

[0073] Procedure 800, in block B804, involves controlling one or more operations of an ego machine based on operating at least a portion of the probabilistic state simulation stack as a control stack of the ego machine. For example, after training, the blocks with a dotted pattern in Fig. Figure 2 illustrates how the following components (the perception encoder 210, the current state estimation network 220, the navigation policy 240, and the history aggregation network 230) are used in an ego machine (e.g., a vehicle) as an end-to-end control stack (e.g., drive stack). The ego machine operations controlled using the control stack and appropriate control component(s) typically depend on the applicable task and / or implementation. Taking automotive applications as an example, in some embodiments the control component(s) are part of an ADAS such as the ADAS system 1038 by Fig. 10C, and the control component(s) can coordinate and / or manage one or more functions within the ADAS. For example, the control stack can generate one or more predicted control actions (e.g., steering, accelerating, braking) and / or trajectories for the ego machine to take at time t, and the control component(s) can cause the ego machine to take the predicted control action(s) (and / or corresponding trajectories).

[0074] Fig. Figure 9 is a flowchart illustrating a method 900 for generating one or more ground truth recovery trajectories for one or more simulated ego-machines according to some embodiments of the present disclosure. The method 900, at block B902, involves generating one or more first segments of one or more navigation episodes representing one or more first trajectories of one or more simulated ego-machines. For example, with respect to Fig. 6B the neural planner 620 (e.g. pre-trained using imitation learning) is used to control an ego machine and generate an initial trajectory in the simulation environment 610 (e.g. the trajectory 510 generated by the neural planner, which is in Fig. 5 is illustrated). Additionally or alternatively, in some embodiments, an initial trajectory for the Ego machine 505 can be generated by applying one or more perturbations (e.g., a small modification or disturbance) to a predicted control action (e.g., steering, accelerating, braking) or trajectory (e.g., a planned path) defined by a navigation policy and / or a control stack of a probabilistic state simulation stack (e.g., the probabilistic state simulation stack 200 of Fig. 2) be generated. Additionally or alternatively, in some embodiments, an initial trajectory for the Ego-Machine 505 can be generated using a neural planner 640 to position the Ego-Machine 505 in the latent space of a probabilistic state simulation stack (e.g., the probabilistic state simulation stack 200 of Fig. 2) to control, as in Fig. 6C illustrates.

[0075] Procedure 900, at block B904, involves generating, using a navigation policy of a probabilistic state simulation stack, one or more second segments of the one or more navigation episodes that represent one or more ground truth recovery trajectories of the one or more simulated ego machines. For example, the blocks marked with a dotted pattern in Fig. 2 are illustrated (the perception encoder 210, the current state estimation network 220, the navigation policy 240, and the history aggregation network 230), are run in a simulation environment to generate a representation of a recovery trajectory (e.g., the recovery trajectory 520, which is shown in Fig. 5 is illustrated) to generate and record.

[0076] Procedure 900, at block B902, involves updating one or more neural motion planners based on at least one representation of the one or more recovery trajectories. For example, a representation of recovery trajectory 520, the corresponding control actions, and / or the simulated environment in which recovery trajectory 520 was generated can be recorded and used as a training episode to train a neural network, a navigation policy, and / or a control stack such as a neural planner that generated the initial trajectory (e.g., neural planner 620 from [reference missing]). Fig. 6B, the neural planner 640 from Fig. 6C, another neural planner), to train.

[0077] Method 900, in block B902, involves controlling one or more operations of an ego machine based on at least one or more neural motion planners. For example, the one or more neural motion planners in an ego machine (e.g., a vehicle) can be used as part (e.g., a navigation directive) of an end-to-end control stack (e.g., a drive stack). Taking automotive applications as an example, in some embodiments the control stack can be augmented by an ADAS such as the ADAS system 1038 from Fig. 10C implemented and used to control the navigation of the Ego machine (e.g., the autonomous vehicle 1000). Fig. 10A - 10D) can be used.

[0078] The systems and procedures described herein may be used by - or may be used in combination with - without restriction, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g. in one or more adaptive driver assistance systems (ADAS)), controlled and uncontrolled robots or robotic platforms, warehouse vehicles, all-terrain vehicles, vehicles coupled to one or more trailers, flying objects, boats, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, trains, underwater vehicles, remotely controlled vehicles such as drones and / or other types of vehicles.Furthermore, the systems and methods described herein can be used for a variety of purposes, including but not limited to machine control, machine locomotion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twinning, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), distributed or collaborative content generation for 3D assets (e.g., using Universal Scene Descriptor (USD) data, such as OpenUSD, and / or other data types), cloud computing, generative artificial intelligence (e.g., using one or more diffusion models, transformer models, etc.), and language modeling applications.B. Large Language Models (LLMs), Vision-Language-Models (VLMs), etc.) and / or any other suitable applications.

[0079] Disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented using a robot or robotic platform, aeronautical systems, medical systems, marine systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations (e.g., in a driving or vehicle simulation, in a robotics simulation, in a smart city or surveillance simulation, etc.), systems for performing digital twin operations (e.g.,in conjunction with a collaborative content creation platform or system, such as, without limitation, NVIDIA OMNIVERSE and / or any other platform, system, or service that uses USD or OpenUSD data types, systems implemented using an edge device, systems involving one or more virtual machines (VMs), systems for performing synthetic data generation operations (e.g., using one or more Neural Rendering Fields (NERFs), Gaussian splat techniques, diffusion models, transformer models, etc.), systems that are at least partially implemented in a data center, systems for performing conversational AI operations, systems implementing one or more language models—such as one or more Large Language Models (LLMs), one or more Vision Language Models (VLMs), one or more multimodal language models, etc., systems for performing light transport simulation, systems for performing collaborative content creation for 3D assets (e.g., using Universal Scene Descriptor (USD) data such as OpenUSD, computer-aided design (CAD) data, 2D and / or 3D graphics or design data and / or other data types), systems that are implemented at least partially using cloud computing resources, and / or other types of systems.

[0080] In some embodiments, the systems and methods described herein can be performed within a simulation environment (e.g., NVIDIA's DriveSIM) using simulated data (e.g., simulated sensor data from simulated sensors of a simulated machine). For example, simulated (or virtual) sensor data (e.g., images of a simulated environment, such as a highway or warehouse environment, generated from the perspective of one or more simulated sensors of a simulated ego-machine) can be applied to a neural network that includes a perception encoder to perform one or more tasks (e.g.,3D perception or reconstruction, semantic segmentation, depth map extraction, trajectory prediction, path planning, navigation control, localization, or mapping), and the response of the neural network model can be used to control the simulated ego machine within the simulated environment. These simulated operations can be used to test the performance of the underlying algorithms, systems, and / or processes before deployment in the real world. In some cases, simulation can be used to generate synthetic training data—for example,Images of a simulated environment, generated from the perspective of one or more simulated sensors of a simulated ego-machine, and the synthetic training data can be used (in addition to or as an alternative to real-world data) to train one or more neural networks, such as a navigation policy or a control stack implemented using one or more neural networks. In any example, such as when a simulation environment is used for testing, validation, training, etc., the simulation environment and / or associated training data can be rendered or otherwise generated using one or more light transport algorithms—such as ray tracing and / or path tracing algorithms.In some embodiments, the simulation environment and / or one or more objects, features, or components thereof can be created or managed within a three-dimensional (3D) content collaboration platform (e.g., NVIDIA's OMNIVERSE) for industrial digitization, generative physical AI, and / or other use cases, applications, or services. For example, the content collaboration platform or system may include a system for using or developing data from a Universal Scene Descriptor (USD) (e.g., OpenUSD) to manage objects, features, scenes, etc., within a simulated environment, digital environment, etc. The platform may include real-world physics simulation, such as using NVIDIA's PhysX SDK, to simulate real-world physics and physical interactions with simulations hosted by the platform.The platform can integrate OpenUSD together with ray tracing / path tracing / light transport simulation (e.g. NVIDIA's RTX rendering technologies) into software tools and simulation workflows for building, training, deploying or testing AI systems - such as systems for testing, validating, training (e.g. machine learning models, neural networks, etc.) and / or other tasks related to automotive, robotics, machinery or other applications. Exemplary autonomous vehicle

[0081] Fig. Figure 10A is an illustration of an exemplary autonomous or semi-autonomous vehicle or an exemplary autonomous or semi-autonomous machine 1000 according to some embodiments of the present disclosure. The autonomous or semi-autonomous vehicle or autonomous or semi-autonomous machine 1000 (hereinafter alternatively referred to as the “Vehicle 1000”, the “Machine 1000”, the “Ego-Vehicle 1000”, the “Ego-Machine 1000”, the “Robot 1000”, etc.) may, without limitation, include a passenger vehicle, such as a car, a truck, a bus, a first responder vehicle, a shuttle, an electric or motorized bicycle, a motorcycle, a fire engine, a police vehicle, an ambulance, a boat, a construction vehicle, an underwater vehicle, a robotic vehicle, a drone, an aircraft, a vehicle coupled to a trailer (e.g., a trailer, a car ...a semi-trailer truck used for transporting cargo), and / or another type of vehicle (e.g., unmanned and / or carrying one or more passengers). Autonomous vehicles are generally described in terms of automation levels defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806, published June 15, 2018, Standard No. J3016-201609, published September 30, 2016, and previous and future versions of this standard). The Vehicle 1000 may be capable of functionality in accordance with one or more of Levels 3–Level 5 of autonomous driving levels.The Vehicle 1000 may be capable of functionality in accordance with one or more of Levels 1 to 5 of autonomous driving levels. For example, the Vehicle 1000 may be capable of driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the embodiment. The term "autonomous," as used herein, may include any and / or all types of autonomy for the Vehicle 1000 or any other machine, such as fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, providing assistive autonomy, semi-autonomous, primarily autonomous, or any other designation.

[0082] The vehicle 1000 can include components such as a chassis, a vehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. The vehicle 1000 can include a propulsion system 1050, such as an internal combustion engine, a hybrid electric power plant, a purely electric motor, and / or another type of propulsion system. The propulsion system 1050 can be connected to a drivetrain of the vehicle 1000, which may include a transmission to enable the propulsion of the vehicle 1000. The propulsion system 1050 can be controlled in response to signals received from the throttle / accelerator pedal 1052.

[0083] A steering system 1054, which may include a steering wheel, can be used to steer the vehicle 1000 (e.g., along a desired path or route) when the drive system 1050 is in operation (e.g., when the vehicle is in motion). The steering system 1054 can receive signals from a steering actuator 1056. The steering wheel may be optional for full automation (Level 5) functionality.

[0084] The brake sensor system 1046 can be used to actuate the vehicle brakes in response to receiving signals from the brake actuators 1048 and / or brake sensors.

[0085] Controller(s) 1036, which includes one or more system-on-chips (SoCs) 1004 ( Fig. The controller(s) 1036, which may include a 10C) and / or GPU(s), can provide signals (e.g., representing commands) to one or more components and / or systems of the vehicle 1000. For example, the controller(s) can send signals to actuate the vehicle brakes via one or more brake actuators 1048, to actuate the steering system 1054 via one or more steering actuators 1056, and to actuate the propulsion system 1050 via one or more throttles / accelerator pedals 1052. The controller(s) 1036 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and issue operational commands (e.g., signals representing commands) to enable autonomous driving and / or to assist a human driver in driving the vehicle 1000.The controller(s) 1036 may include a first controller 1036 for autonomous driving functions, a second controller 1036 for functional safety functions, a third controller 1036 for artificial intelligence functionality (e.g., computer vision), a fourth controller 1036 for infotainment functionality, a fifth controller 1036 for redundancy under emergency conditions, and / or other controllers. In some examples, a single controller 1036 may handle two or more of the above functionalities; two or more controllers 1036 may handle a single functionality and / or any combination thereof.

[0086] The controller(s) 1036 can provide the signals to control one or more components and / or systems of the vehicle 1000 in response to sensor data received from one or more sensors (e.g. sensor inputs). The sensor data can be obtained, for example, without limitation, from one or more sensors 1058 of the global navigation satellite system (“GNSS”) (e.g., global positioning system sensor(s)), one or more radar sensors 1060, one or more ultrasonic sensors 1062, one or more LiDAR sensors 1064, one or more inertial measurement unit (IMU) sensors 1066 (e.g., accelerometer(s), gyroscope(s), magnetic compass(s), magnetometer, etc.), one or more microphones 1096, one or more stereo cameras 1068, one or more wide-angle cameras 1070 (e.g., fisheye cameras), one or more infrared cameras 1072, one or more surround cameras 1074 (e.g.,360-degree cameras), one or more long-range and / or medium-range cameras 1098, one or more speed sensors 1044 (e.g., to measure the speed of the vehicle 1000), one or more vibration sensors 1042, one or more steering sensors 1040, one or more brake sensors (e.g., as part of the brake sensor system 1046), one or more occupant monitoring system (OMS) sensors 1001 (e.g., one or more interior cameras), and / or other sensor types.

[0087] One or more of the controller(s) 1036 can receive inputs (e.g., represented by input data) from an instrument cluster 1032 of the vehicle 1000 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 1034, an audible announcer, a loudspeaker, and / or via other components of the vehicle 1000. The outputs can include information such as vehicle speed, speed, time, map data (e.g., the high-resolution ("HD") map 1022 from Fig. 10C), location data (e.g., the location of vehicle 1000, as on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and the status of objects as perceived by the controller(s) 1036, etc. For example, the HMI display 1034 can show information about the presence of one or more objects (e.g., a road sign, warning sign, traffic light change, etc.) and / or information about driving maneuvers that the vehicle has undertaken, is undertaking, or will undertake (e.g., changing lanes now, entering exit 34B in two miles, etc.).

[0088] The vehicle 1000 also includes a network interface 1024, which can use one or more wireless antenna(s) 1026 and / or modem(s) to communicate over one or more networks. For example, the network interface 1024 can be capable of communication via Long-Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communication (“GSM”), IMT-CDMA Multi-Carrier (“CDMA2000”), etc. The wireless antenna(s) 1026 can also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using local area networks, such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc., and / or low-power wide area networks (“LPWANs”), such as LoRaWAN, SigFox, etc.

[0089] Fig. 10B is an example of camera positions and fields of view for the exemplary autonomous vehicle 1000 from Fig. 10A according to some embodiments of the present disclosure. The cameras and respective fields of view are an exemplary embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included and / or the cameras may be located at different locations on the vehicle 1000.

[0090] The camera types may include, but are not limited to, digital cameras that can be adapted for use with the components and / or systems of the vehicle. The camera(s) may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera types may be capable of any frame rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the specific configuration. The cameras may be capable of using roller shutters, global shutters, another type of shutter, or a combination thereof.In some examples, the color filter array may include a red-clear-clear-clear color filter array (RCCC color filter array), a red-clear-clear-blue color filter array (RCCB color filter array), a red-blue-green-clear color filter array (RBGC color filter array), a Foveon X3 color filter array, a Bayer sensor color filter array (RGGB color filter array), a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras with an RCCC, RCCB, and / or RBGC color filter array, may be used in an effort to increase light sensitivity.

[0091] In some examples, one or more of the camera(s) can be used to perform advanced driver assistance systems (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign recognition, and intelligent headlight control. One or more of the camera(s) (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0092] One or more of the cameras can be mounted in an assembly, such as a custom-designed (three-dimensional ("3D") printed) assembly, to filter out stray light and reflections from inside the car (e.g., reflections from the dashboard that are reflected in the windshield mirrors) that can interfere with the camera's image acquisition capabilities. Referring to vane mirror mounting assemblies, the vane mirror assemblies can be custom 3D printed so that the camera mounting plate matches the shape of the vane mirror. In some examples, the camera(s) can be integrated into the vane mirror. For side-view cameras, the camera(s) can also be integrated into the four pillars at each corner of the cabin.

[0093] Cameras with a field of view that includes portions of the environment in front of the vehicle (e.g., forward-facing cameras) can be used for surround view to help identify forward paths and obstacles, and to help provide, with the aid of one or more controllers and / or control SoCs, information critical for generating an occupancy grid and / or determining preferred vehicle paths. Forward-facing cameras can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. Forward-facing cameras can also be used for ADAS functions and systems, including lane departure warning (LDW), adaptive cruise control (ACC), and / or other functions, such as traffic sign recognition.

[0094] A variety of cameras can be used in a forward-facing configuration, including, for example, a monocular camera platform incorporating a complementary metal-oxide semiconductor (CMOS) color imager. Another example is a 1070 wide-angle camera, which can be used to detect objects coming into view from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Although in Fig. While Figure 10B illustrates only one wide-angle camera, there can be any number (including zero) of wide-angle cameras 1070 on the vehicle 1000. Additionally, any number of long-range camera(s) 1098 (e.g., a pair of long-range stereo cameras) can be used for depth-based object detection, particularly for objects for which a neural network has not yet been trained. The long-range camera(s) 1098 can also be used for object detection and classification, as well as basic object tracking.

[0095] Any number of stereo cameras 1068 can also be included in a forward-facing configuration. In at least one embodiment, one or more of the stereo camera(s) 1068 can include an integrated control unit comprising a scalable processing unit that can provide programmable logic (“FPGA”) and a multi-core microprocessor with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle's surroundings, including a distance estimate for all points in the image. One or more alternative stereo camera(s) 1068 can include one or more compact stereo vision sensors, which can include two camera lenses (one each on the left and right) and an image processing chip that measures the distance from the vehicle to the target object and generates the information (e.g., distance, position, and distance).B. metadata) can be used to activate the autonomous emergency braking and lane departure warning functions. Other types of stereo camera(s) 1068 can be used in addition to or as an alternative to those described here.

[0096] Cameras with a field of view that includes sections of the environment to the sides of the vehicle (e.g., side-view cameras) can be used for surround view, providing information that is used to create and update the occupancy grid and to generate side-impact collision warnings. For example, one or more surround-view cameras (e.g., four surround-view cameras, as in Fig. (Illustrated in Figure 10B) is positioned on the vehicle 1000. The surround-view camera(s) 1074 may include one or more wide-angle cameras 1070, one or more fisheye cameras, one or more 360-degree cameras, and / or the like. For example, four fisheye cameras may be positioned at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle may use three surround-view camera(s) 1074 (e.g., left, right, and rear) and may use one or more other cameras (e.g., a forward-facing camera) as a fourth surround-view camera.

[0097] Cameras with a field of view that includes sections of the area behind the vehicle (e.g., rear-view cameras) can be used for parking assistance, surround view, rear collision warnings, and creating and updating the occupancy grid. A wide variety of cameras can be used, including but not limited to cameras that are also suitable as forward-facing cameras (e.g., long-range and / or medium-range camera(s), stereo camera(s), infrared camera(s), etc.), as described herein.

[0098] Cameras with a field of view that includes sections of the interior environment within the cabin of the vehicle 1000 (e.g., one or more OMS sensor(s) 1001) can be used as part of an occupant monitoring system (OMS), such as, but not limited to, a driver monitoring system (DMS). For example, OMS sensors (e.g., the OMS sensor(s) 1001) can be used (e.g., by the controller(s) 1036) to track the gaze direction, head position, and / or turn signal of an occupant and / or driver. This gaze information can be used to determine a level of attention of the occupant or driver (e.g., to detect drowsiness, fatigue, and / or distraction) and / or to take reactive measures to prevent harm to the occupant or operator.In some embodiments, data from OMS sensors can be used to enable gaze-controlled operations triggered by the driver and / or non-driver occupants, such as, but not limited to, adjusting cabin temperature and / or airflow, opening and closing windows, controlling cabin lighting, controlling entertainment systems, adjusting mirrors, adjusting seat positions, and / or other operations. In some embodiments, an OMS can be used for applications such as determining when objects and / or occupants have been left in a vehicle cabin (e.g., by detecting the presence of occupants after the driver has left the vehicle).

[0099] Fig. 10C is a block diagram of an exemplary system architecture for the exemplary autonomous vehicle 1000 by Fig. 10A according to some embodiments of the present disclosure. It is understood that these and other arrangements described herein are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components, or in conjunction with other components, in any suitable combination and at any suitable location. Various functions described herein as being performed by entities may be executed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory.

[0100] Each of the vehicle's components, features, and systems 1000 in Fig. 10C are illustrated as connected via bus 1002. Bus 1002 may include a Controller Area Network (CAN) data interface (referred to herein alternatively as "CAN bus"). A CAN can be a network within the vehicle 1000 that is used to assist in controlling various features and functionality of the vehicle 1000, such as actuating brakes, accelerating, braking, steering, windshield wipers, etc. A CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). The CAN bus can be read to obtain steering wheel angle, ground speed, engine revolutions per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0101] Although Bus 1002 is described herein as a CAN bus, this is not intended to be restrictive. For example, FlexRay and / or Ethernet may be used in addition to or as an alternative to the CAN bus. Furthermore, although a single line is used to represent Bus 1002, this is not intended to be restrictive. For example, there may be any number of Buses 1002, which may contain one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using a different protocol. In some examples, two or more Buses 1002 may be used to perform different functions and / or for redundancy. For example, a first Bus 1002 may be used for collision avoidance functionality, and a second Bus 1002 may be used for actuation control.In any given example, each bus 1002 can communicate with any of the vehicle 1000's components, and two or more buses 1002 can communicate with the same components. In some examples, each SoC 1004, each controller 1036, and / or each computer within the vehicle can have access to the same input data (e.g., inputs from vehicle 1000 sensors) and can be connected to a common bus, such as the CAN bus.

[0102] The vehicle 1000 can include one or more control units 1036, such as those mentioned herein in relation to Fig. 10A are described. The controller(s) 1036 can be used for a variety of functions. The controller(s) 1036 can be coupled with any of the various other components and systems of the vehicle 1000 and can be used to control the vehicle 1000, the vehicle 1000's artificial intelligence, the vehicle 1000's infotainment system, and / or the like.

[0103] The vehicle 1000 can include a system-on-a-chip (SoC) 1004. The SoC 1004 can include CPU(s) 1006, GPU(s) 1008, processor(s) 1010, cache(s) 1012, accelerator(s) 1014, data storage(s) 1016, and / or other components and features not illustrated. The SoC(s) 1004 can be used to control the vehicle 1000 in a variety of platforms and systems. For example, the SoC(s) 1004 can be combined in a system (e.g., the system of the vehicle 1000) with an HD card 1022, which performs map updates and / or updates via a network interface 1024 from one or more servers (e.g., server(s) 1078). Fig. 10D).

[0104] The CPU(s) 1006 may include a CPU cluster or CPU complex (hereinafter referred to as "CCPLEX"). The CPU(s) 1006 may include multiple cores and / or L2 caches. For example, in some embodiments, the CPU(s) 1006 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU(s) 1006 may include four dual-core clusters, each cluster having a dedicated L2 cache (e.g., a 2 MB L2 cache). The CPU(s) 1006 (e.g., the CCPLEX) may be configured to support concurrent cluster operation, allowing any combination of CPU(s) 1006 clusters to be active at any given time.

[0105] The CPU(s) 1006 can implement power management capabilities that include one or more of the following features: individual hardware blocks can be automatically clocked while idle to save dynamic power; each core can be clocked when the core is not actively executing instructions due to the execution of WFI / WFE instructions; each core can be clocked independently; each core cluster can be clocked independently when all cores are clocked; and / or each core cluster can be clocked independently when all cores are clocked. The CPU(s) 1006 can further implement an improved power state management algorithm, specifying allowed power states and expected wake-up times, with the hardware / microcode determining the best power state to input for the core, cluster, and CCPLEX.The processing kernels can support simplified performance state input sequences in software, offloading the work to microcode.

[0106] The GPU(s) 1008 may include an integrated GPU (referred to herein alternatively as the "iGPU"). The GPU(s) 1008 may be programmable and efficient for parallel workloads. The GPU(s) 1008 may, in some examples, use an improved tensor instruction set. The GPU(s) 1008 may include one or more streaming microprocessors, each streaming microprocessor being able to include an L1 cache (e.g., an L1 cache with a minimum memory size of 96 KB), and two or more of the streaming microprocessors being able to share an L2 cache (e.g., an L2 cache with a memory size of 512 KB). In some embodiments, the GPU(s) 1008 may include at least eight streaming microprocessors. The GPU(s) 1008 can use computer application programming interface(s) (API(s)). Additionally, the GPU(s) 1008 can utilize one or more parallel computing platforms and / or programming models (e.g., ).B. CUDA from NVIDIA).

[0107] The GPU(s) 1008 can be performance-optimized for best performance in automotive and embedded applications. For example, the GPU(s) 1008 can be manufactured using a FinFET field-effect transistor. However, this is not a limitation, and the GPU(s) 1008 can be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can include a number of mixed-precision processing cores partitioned into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be partitioned into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSOR cores for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, a dispatch unit and / or a 64 KB register file.Additionally, streaming microprocessors can include independent parallel integer and floating-point data paths to provide efficient execution of workloads with a mix of computational and addressing tasks. Streaming microprocessors can include independent thread scheduling capabilities to enable finer synchronization and cooperation between parallel threads. Streaming microprocessors can also include a combined L1 data cache and shared memory to improve performance while simplifying programming.

[0108] The GPU(s) 1008 can include High Bandwidth Memory (HBM) and / or a 16 GB HBM2 memory subsystem to provide a peak memory bandwidth of approximately 900 GB / second in some examples. In some examples, in addition to or as an alternative to HBM memory, synchronous graphics random access memory (SGRAM), such as double-data-rate type five synchronous graphics random access memory (GDDR5), can be used.

[0109] The GPU(s) 1008 can incorporate unified memory technology, including access counters to enable more accurate migration of memory pages to the processor that accesses them most frequently, thereby improving efficiency for memory areas shared between processors. In some examples, support for Address Translation Services (ATS) can be used to allow the GPU(s) 1008 to directly access the page tables of the CPU(s) 1006. In such examples, if the memory management unit (MMU) of the GPU(s) 1008 experiences an error, an address translation request can be passed to the CPU(s) 1006. In response, the CPU(s) 1006 can search its page tables for the virtual-physical mapping for the address and pass the translation back to the GPU(s) 1008.As such, unified memory technology can enable a single unified virtual address space for memory of both the CPU(s) 1006 and the GPU(s) 1008, thereby simplifying the programming and porting of applications from the GPU(s) 1008 to the GPU(s) 1008.

[0110] Additionally, the GPU(s) 1008 may include an access counter that tracks the frequency of GPU(s) accessing memory on other processors. This access counter can help ensure that memory pages are moved to the physical memory of the processor that accesses them most frequently.

[0111] The SoC(s) 1004 can include any number of Cache(s) 1012, including those described herein. For example, the Cache(s) 1012 can include an L3 cache available to both the CPU(s) 1006 and the GPU(s) 1008 (e.g., connected to both the CPU(s) 1006 and the GPU(s) 1008). The Cache(s) 1012 can include a write-back cache capable of tracking line states, such as using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache can be 4 MB or larger, depending on the implementation, although smaller cache sizes are possible.

[0112] The SoC(s) 1004 may include an arithmetic logic unit (ALU) that can be used to perform processing related to any of the many tasks or operations of the Vehicle 1000, such as processing deep neural networks (DNNs). Additionally, the SoC(s) 1004 may include a floating-point unit (FPU) or other mathematical or numeric coprocessor types for performing mathematical operations within the system. For example, the SoC(s) 1004 may include one or more FPUs integrated as execution units within one or more CPUs 1006 and / or GPUs 1008.

[0113] The SoC(s) 1004 can include one or more Accelerators 1014 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC(s) 1004 can include a hardware acceleration cluster, which may contain optimized hardware accelerators and / or large on-chip memory. The large on-chip memory (e.g., 4 MB SRAM) can enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster can be used to complement the GPU(s) 1008 and offload some of the GPU's tasks (e.g., to free up more GPU cycles for other tasks). As an example, the Accelerator 1014 can be used for targeted workloads (e.g., perception, Convolutional Neural Networks (CNNs), etc.) that are stable enough to be accelerable.The term “CNN”, as used herein, can include all types of CNNs, including region-based or regional Convolutional Neural Networks (RCNNs) and fast RCNNs (e.g., such as those used for object detection).

[0114] The Accelerator 1014 (e.g., the hardware acceleration cluster) can include one or more Deep Learning Accelerators (DLAs). The DLA(s) can include one or more Tensor Processing Units (TPUs), which can be configured to provide an additional ten trillion operations per second for deep learning applications and inference. The TPUs can be accelerators configured and optimized to perform image processing functions (e.g., for CNNs, RCNNs, etc.). The DLA(s) can further be optimized for a specific set of neural network types and floating-point operations, as well as inference. The design of the DLA(s) can deliver more performance per millimeter than a general-purpose GPU and significantly exceeds the performance of a CPU.The TPU(s) can perform several functions, including a single-instance folding function that supports, for example, INT8, INT16 and FP16 data types for both features and weights, as well as post-processor functions.

[0115] The DLA(s) can quickly and efficiently execute neural networks, especially CNNs, on processed or unprocessed data for any of a wide variety of functions, including, but not limited to: a CNN for object identification and detection using data from camera sensors; a CNN for distance estimation using data from camera sensors; a CNN for emergency vehicle detection and identification using data from microphones; a CNN for facial recognition and vehicle owner identification using data from camera sensors; and / or a CNN for safety and / or security-related events.

[0116] The DLA(s) can perform any function of the GPU(s) 1008, and by using an inference accelerator, a designer can, for example, target either the DLA(s) or the GPU(s) 1008 for any given function. For example, the designer can focus the processing of CNNs and floating-point operations on the DLA(s) and leave other functions to the GPU(s) 1008 and / or the other accelerator(s) 1014.

[0117] The Accelerator 1014 (e.g., the hardware acceleration cluster) may include one or more Programmable Vision Accelerators (PVAs), which may alternatively be referred to herein as computer vision accelerators. The PVA(s) may be designed and configured to accelerate computer vision algorithms for applications such as Advanced Driver Assistance Systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR). The PVA(s) may provide a balance between performance and flexibility. For example, each PVA may, without limitation, include any number of Reduced Instruction Set Computer (RISC) cores, Direct Memory Access (DMA), and / or any number of vector processors.

[0118] The RISC cores can interact with image sensors (e.g., the image sensors of any of the cameras described herein), image signal processor(s), and / or the like. Each RISC core can include any amount of memory. Depending on the implementation, the RISC cores can use any of a number of protocols. In some examples, the RISC cores can run a real-time operating system (RTOS). The RISC cores can be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores can include an instruction cache and / or tightly coupled RAM.

[0119] The DMA can enable components of the PVA(s) to access system memory independently of the CPU(s). The DMA can support any number of features used to provide optimization for the PVA, including, but not limited to, supporting multidimensional addressing and / or circular addressing. In some examples, the DMA can support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block step size, vertical block step size, and / or depth step size.

[0120] Vector processors can be programmable processors designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA might include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, DMA machine(s) (e.g., two DMA machines), and / or other peripherals. The vector processing subsystem can act as the primary processing machine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). A VPU core might include a digital signal processor, such as a single-instruction, multiple-data (SIMD), or very-long instruction word (VLIW) digital signal processor. Combining SIMD and VLIW can improve throughput and speed.

[0121] Each vector processor can include an instruction cache and can be coupled to dedicated memory. Consequently, in some examples, each vector processor can be configured to run independently of the others. In other examples, the vector processors included in a particular PVA can be configured to employ data parallelism. For example, in some embodiments, the multitude of vector processors included in a single PVA can execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can simultaneously execute different computer vision algorithms on the same image, or even different algorithms on sequential images or sections of an image.Among other things, any number of PVAs can be included in the hardware acceleration cluster, and any number of vector processors can be included in each of the PVAs. Additionally, the PVA(s) can include additional error correction code (ECC) memory to improve overall system security.

[0122] The Accelerator 1014 (e.g., the hardware acceleration cluster) can include an on-chip computer view network and SRAM to provide high-bandwidth, low-latency SRAM for the Accelerator 1014. In some examples, the on-chip memory can include at least 4 MB of SRAM, consisting, for example, and without limitation, of eight field-configurable memory blocks accessible to both the PVA and the DLA. Each pair of memory blocks can include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory can be used. The PVA and DLA can access the memory via a backbone that provides high-speed memory access for the PVA and DLA. The backbone can include an on-chip computer view network that connects the PVA and DLA to the memory (e.g., using the APB).

[0123] The on-chip computer network can include an interface that, prior to the transmission of any control signal, address, or data, determines that both the PVA and DLA provide ready and valid signals. Such an interface can provide separate phases and channels for transmitting control signals, addresses, and data, as well as burst-type communication for continuous data transfer. This type of interface can conform to ISO 26262 or IEC 61508 standards, although other standards and protocols can be used.

[0124] In some examples, the SoC(s) 1004 may include a real-time ray-tracing hardware accelerator, as described in U.S. Patent No. 10,885,698, issued January 5, 2021. The real-time ray-tracing hardware accelerator can be used to quickly and efficiently determine the positions and dimensions of objects (e.g., within a simulation) to generate real-time visualization simulations for radar signal interpretation, sound propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison with LiDAR data for localization purposes, and / or other functions and / or other uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray-tracing-related operations.

[0125] The Accelerator 1014 (e.g., the hardware acceleration cluster) has a wide range of uses for autonomous driving. The PVA can be a programmable vision accelerator that can be used for key processing stages in ADAS and autonomous vehicles. The PVA's capabilities are a good match for algorithmic domains that require predictable processing with low power consumption and low latency. As such, the PVA performs well on semi-dense or dense regular computation, even on small datasets, that require predictable runtimes with low latency and low power consumption. Therefore, in the context of autonomous vehicle platforms, PVAs are designed to execute classical computer vision algorithms because they are efficient at object detection and operate on integer mathematics.

[0126] For example, according to one embodiment of the technology, the PVA is used to perform computer stereo vision. A semi-global matching-based algorithm can be used in some examples, although this is not intended to be restrictive. Many Level 3-5 autonomous driving applications require on-the-fly motion estimation / stereo matching (e.g., structure from motion, pedestrian detection, lane detection, etc.). The PVA can perform a computer stereo vision function on input from two monocular cameras.

[0127] In some examples, PVA can be used to perform dense optical flow processing. According to this process, raw RADAR data is processed (e.g., using a 4D fast Fourier transform) to provide processed RADAR data. In other examples, PVA is used for runtime depth processing, for example, by processing raw runtime data to provide processed runtime data.

[0128] The DLA can be used to run any type of network to improve control and driving safety, including, for example, a neural network that outputs a confidence score for each object detection. Such a confidence score can be interpreted as a probability or as providing a relative "weighting" of each detection compared to other detections. This confidence score allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a confidence threshold and consider only those detections that exceed the threshold as true positives.In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically initiate emergency braking, which is obviously undesirable. Therefore, only the safest detections should be considered as triggers for AEB. The DLA can execute a neural network to reduce the confidence level. The neural network can take as its input at least a subset of parameters, such as boundary box dimensions, ground plane estimation obtained (e.g., from another subsystem), inertial measurement unit (IMU) sensor output 1066 correlated with the vehicle's orientation 1000, distance, and 3D position estimates of the object obtained from the neural network and / or other sensors (e.g., LiDAR sensor(s) 1064 or radar sensor(s) 1060), among others.

[0129] The SoC(s) 1004 may include data storage(s) 1016 (e.g., memory). The data storage(s) 1016 may be on-chip memory of the SoC(s) 1004 capable of storing neural networks to be executed on the GPU and / or the DLA. In some examples, the data storage(s) 1016 may be large enough to store multiple instances of neural networks for redundancy and security. The data storage(s) 1016 may include L2 or L3 cache(s) 1012. Reference to the data storage(s) 1016 may include reference to memory associated with the PVA, DLA, and / or another accelerator(s) 1014, as described herein.

[0130] The SoC(s) 1004 can include one or more Processor(s) 1010 (e.g., embedded processors). The Processor(s) 1010 can include a Boot and Power Management Processor, which may be a dedicated processor and subsystem to handle boot power and management functions and associated security enforcement. The Boot and Power Management Processor can be part of the SoC(s) 1004's boot sequence and can provide runtime power management services. The Boot and Power Management Processor can provide clock and voltage programming, support for system low-power state transitions, management of SoC(s) 1004's thermal and temperature sensors, and / or management of SoC(s) 1004's power states.Each temperature sensor can be implemented as a ring oscillator whose output frequency is proportional to the temperature, and the SoC(s) 1004 can use the ring oscillators to detect temperatures of the CPU(s) 1006, GPU(s) 1008, and / or accelerator(s) 1014. If temperatures are determined to exceed a threshold, the boot and power management processor can enter a temperature fault routine and put the SoC(s) 1004 into a lower power state and / or put the vehicle 1000 into a chauffeur-to-safe-stop mode (e.g., bring the vehicle 1000 to a safe stop).

[0131] The 1010 processor(s) may further include a set of embedded processors that can serve as an audio processing engine. The audio processing engine may be an audio subsystem that provides full hardware support for multi-channel audio across multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor and dedicated RAM.

[0132] The 1010 processor(s) may further include an always-on processor machine that can provide the necessary hardware features to support low-power sensor management and wake-up use cases. The always-on processor machine may include a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0133] The 1010 processor(s) can further include a security cluster machine, which contains a dedicated processor subsystem for handling security management for automotive applications. The security cluster machine can include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, an interrupt controller, etc.), and / or routing logic. In a security mode, the two or more cores can operate in lockstep mode and function as a single core with comparison logic to detect differences between their operations.

[0134] The 1010 processor(s) may also include a real-time camera machine, which may include a dedicated processor subsystem for handling real-time camera management.

[0135] The 1010 processor(s) may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware machine that is part of the camera processing pipeline.

[0136] The 1010 processor(s) may include a video image assembler, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to generate the final image for the player window. The video image assembler may perform lens distortion correction on the 1070 widescreen camera(s), the 1074 omnidirectional camera(s), and / or in-cabin monitoring camera sensors. The in-cabin monitoring camera sensor is preferably monitored by a neural network running on a separate instance of the Advanced SoC, configured to identify and respond to events in the cabin.A system in the cabin can lip-read to activate cellular service and make phone calls, dictate emails, change the vehicle's destination, activate or change the infotainment system and vehicle settings, or provide voice-activated web browsing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode and are otherwise disabled.

[0137] The video frame assembler can include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, if there is motion in a video, the noise reduction appropriately weights spatial information, thereby reducing the impact of information provided by adjacent frames. If an image or portion of an image does not contain motion, the temporal noise reduction performed by the video frame assembler can use information from the previous image to reduce noise in the current image.

[0138] The video image composer can also be configured to perform stereo equalization on input stereoscopic frames. Furthermore, the video image composer can be used for user interface composition when the operating system desktop is in use, thus preventing the GPU(s) 1008 from continuously rendering new surfaces. Even when the GPU(s) 1008 are powered on and actively performing 3D rendering, the video image composer can be used to offload the GPU(s) 1008 to improve performance and responsiveness.

[0139] The SoC(s) 1004 may further include a Mobile Industrial Processor Interface (MIPI) camera serial interface for receiving video and camera input, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functions. The SoC(s) 1004 may also include one or more software-controlled input / output controllers that can be used to receive I / O signals not assigned to a specific role.

[0140] The SoC(s) 1004 can also include a wide range of peripheral interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The SoC(s) 1004 can be used to process data from cameras (e.g., connected via Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LiDAR sensor(s) 1064, RADAR sensor(s) 1060, etc., which can be connected via Ethernet), data from bus 1002 (e.g., vehicle speed 1000, steering wheel position, etc.), and data from GNSS sensor(s) 1058 (e.g., connected via Ethernet or CAN bus). The SoC(s) 1004 may also include dedicated high-performance mass storage controllers, which may include their own DMA machines and which can be used to offload routine data management tasks from the CPU(s) 1006.

[0141] The SoC(s) 1004 can be an end-to-end platform with a flexible architecture spanning automation levels 3-5, providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS techniques for diversity and redundancy, and offers a platform for a flexible, reliable driving software stack along with deep learning tools. The SoC(s) 1004 can be faster, more reliable, and even more energy-efficient and space-efficient than conventional systems. For example, the 1004 accelerator(s), when combined with the 1006 CPU(s), the 1008 GPU(s), and the 1016 data storage(s), can provide a fast, efficient platform for autonomous vehicles at levels 3-5.

[0142] This technology thus provides capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be run on CPUs that can be configured using a higher-level programming language, such as C, to execute a wide variety of processing algorithms on a wide variety of visual data. However, CPUs are often unable to meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs are unable to execute complex object detection algorithms in real time, which is a requirement for in-vehicle ADAS applications and for practical Level 3-5 autonomous vehicles.

[0143] In contrast to conventional systems, the technology described herein, by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, enables multiple neural networks to run simultaneously and / or sequentially and to combine the results to enable autonomous driving functionality at levels 3-5. For example, a CNN running on the DLA or dGPU (e.g., the GPU(s) 1020) can include text and word recognition, allowing the supercomputer to read and understand traffic signs, including signs for which the neural network was not specifically trained.The DLA can further include a neural network capable of identifying and interpreting the sign, providing a semantic understanding of the sign, and forwarding this semantic understanding to the path planning modules running on the CPU complex.

[0144] As another example, multiple neural networks can run simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign stating "Warning: Flashing lights indicate icy conditions" can be interpreted independently or collectively by multiple neural networks, along with an electric warning light. The sign itself can be identified as a traffic sign by a first neural network (e.g., a trained neural network), while the text "Flashing lights indicate icy conditions" can be interpreted by a second neural network, which informs the vehicle's path planning software (preferably running on the CPU) that the detection of flashing lights indicates icy conditions.The flashing light can be identified by operating a third, deployed neural network across multiple frames, informing the vehicle's path planning software of the presence (or absence) of flashing lights. All three neural networks can run simultaneously, for example, within the DLA and / or on the GPU(s) 1008.

[0145] In some examples, a CNN for facial recognition and vehicle owner identification can use data from camera sensors to identify the presence of an authorized driver and / or owner of the vehicle 1000. The always-on sensor processing machine can be used to unlock the vehicle when the owner approaches the driver's door and turns on the lights, and in security mode, to disable the vehicle when the owner leaves. In this way, the SoC(s) 1004 provide security against theft and / or carjacking.

[0146] In another example, a CNN for emergency vehicle detection and identification can use data from microphones 1096 to detect and identify emergency vehicle sirens. Unlike conventional systems that use general classifiers to detect sirens and manually extract features, the SoC(s) 1004 uses the CNN to classify ambient and urban noise as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative closing speed of the emergency vehicle (e.g., using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the local area in which the vehicle operates, as identified by the GNSS sensor(s) 1058.Thus, for example, when operating in Europe, the CNN will attempt to detect European sirens, and when operating in the United States, it will attempt to identify only North American sirens. Once an emergency vehicle is detected, a control program can be used to execute an emergency vehicle safety routine, slowing the vehicle down, pulling it to the side of the road, parking the vehicle, and / or allowing the vehicle to idle using ultrasonic sensors 1062 until the emergency vehicle(s) pass by.

[0147] The vehicle may include a CPU(s) 1018 (e.g., discrete CPU(s) or dCPU(s)) which may be coupled to the SoC(s) 1004 via a high-speed connection (e.g., PCIe). The CPU(s) 1018 may, for example, include an x86 processor. The CPU(s) 1018 may be used to perform any one of a variety of functions, including, for example, arbitrating potentially inconsistent results between ADAS sensors and the SoC(s) 1004 and / or monitoring the status and integrity of the control unit(s) 1036 and / or the infotainment SoC 1030.

[0148] The vehicle 1000 can include a GPU(s) 1020 (e.g., discrete GPU(s) or dGPU(s)) which can be coupled to the SoC(s) 1004 via a high-speed connection (e.g., NVIDIA's NVLINK). The GPU(s) 1020 can provide additional artificial intelligence functionality, such as running redundant and / or different neural networks, and can be used to train and / or update neural networks based on inputs (e.g., sensor data) from the vehicle 1000's sensors.

[0149] The vehicle 1000 can also include the network interface 1024, which can include one or more wireless antennas 1026 (e.g., one or more wireless antennas for different communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network interface 1024 can be used to enable wireless connectivity via the internet to the cloud (e.g., to the server(s) 1078 and / or other network devices), to other vehicles, and / or to computer devices (e.g., passenger client devices). To communicate with other vehicles, a direct connection can be established between the two vehicles, and / or an indirect connection can be established (e.g., via networks and the internet). Direct connections can be provided using a vehicle-to-vehicle communication link.The vehicle-to-vehicle communication link can provide vehicle 1000 with information about vehicles in its vicinity (e.g., vehicles in front of, to the side of, and / or behind vehicle 1000). This functionality can be part of a cooperative adaptive cruise control functionality of vehicle 1000.

[0150] The network interface 1024 can include a system-on-a-chip (SoC) that provides modulation and demodulation functionality, enabling the controller(s) 1036 to communicate over wireless networks. The network interface 1024 can include a high-frequency (RF) front end for upconversion from baseband to RF and downconversion from RF to baseband. The frequency conversions can be performed using well-known processes and / or superheterodyne processes. In some examples, the RF front-end functionality can be provided by a separate chip. The network interface can include wireless functionality for communication over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0151] The vehicle 1000 may further include data storage(s) 1028, which may include off-chip memory (e.g., outside the SoC(s) 1004). The data storage(s) 1028 may include one or more memory elements, including RAM, SRAM, DRAM, VRAM, flash, hard disks, and / or other components and / or devices capable of storing at least one data bit.

[0152] The vehicle 1000 can further include one or more GNSS sensor(s) 1058. The GNSS sensor(s) 1058 (e.g., GPS, supported GPS sensors, differential GPS sensors (DGPS sensors), etc.) are used to assist with mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensor(s) 1058 can be used, including, for example, and without limitation, a single GPS using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0153] The vehicle 1000 may also include one or more RADAR sensor(s) 1060. The RADAR sensor(s) 1060 can be used by the vehicle 1000 for long-range vehicle detection, even in darkness and / or severe weather conditions. The functional safety level can be ASIL B. The RADAR sensor(s) 1060 can use the CAN and / or bus 1002 (e.g., to transmit data generated using the RADAR sensor(s) 1060) for control and to access object tracking data, with some examples using Ethernet to access raw data. A wide variety of RADAR sensor types can be used. For example, and without limitation, the RADAR sensor(s) 1060 can be suitable for use with front, rear, and side radar. In some examples, one or more pulse Doppler radar sensors are used.

[0154] The 1060 RADAR sensor(s) can incorporate various configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In some examples, long-range RADAR can be used for adaptive cruise control functionality. The long-range RADAR systems can provide a wide field of view, achieved through two or more independent scans, such as within a 250-meter range. The 1060 RADAR sensor(s) can help distinguish between stationary and moving objects and can be used by ADAS systems for emergency braking assistance and forward collision warning. Long-range RADAR sensors can incorporate monostatic multimodal RADAR with multiple (e.g., six or more) fixed RADAR antennas and a high-speed CAN and FlexRay interface.In an example with six antennas, the central four antennas can generate a focused beam pattern designed to map the area around vehicle 1000 at higher speeds with minimal interference from traffic in adjacent lanes. The other two antennas can widen the field of view, making it possible to quickly detect vehicles entering or exiting vehicle 1000's lane.

[0155] Medium-range radar systems, for example, can have a range of up to 1060 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 1050 degrees (rear). Short-range radar systems can, without restriction, include radar sensors designed to be installed at both ends of the rear bumper. When installed at both ends of the rear bumper, such radar sensor systems can generate two beams that continuously monitor the blind spot at the rear and to the sides of the vehicle.

[0156] Short-range radar systems can be used in an ADAS system for blind spot detection and / or lane change assistance.

[0157] The vehicle 1000 may also include one or more ultrasonic sensor(s) 1062. The ultrasonic sensor(s) 1062, which may be positioned at the front, rear, and / or sides of the vehicle 1000, may be used for parking assistance and / or for creating and updating an occupancy grid. A wide variety of ultrasonic sensor(s) 1062 may be used, and different ultrasonic sensor(s) 1062 may be used for different detection ranges (e.g., 2.5 m, 4 m). The ultrasonic sensor(s) 1062 may operate at functional safety levels of ASIL B.

[0158] The vehicle 1000 can include one or more LiDAR sensor(s) 1064. The LiDAR sensor(s) 1064 can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LiDAR sensor(s) 1064 can be functional safety level ASIL B. In some examples, the vehicle 1000 can include multiple LiDAR sensors 1064 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0159] In some examples, the LiDAR sensor(s) 1064 may be capable of providing a list of objects and their distances for a 360-degree field of view. The commercially available LiDAR sensor(s) 1064 may have an advertised range of approximately 1000 m with an accuracy of 2-3 cm and support for a 1000 Mbps Ethernet connection. In some examples, one or more non-protruding LiDAR sensors 1064 may be used. In such examples, the LiDAR sensor(s) 1064 may be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of the vehicle 1000. The LiDAR sensor(s) 1064 can provide up to a horizontal field of view of 120 degrees and a vertical field of view of 35 degrees with a range of 200 m in such examples, even for objects with low reflectivity.Front-mounted LiDAR sensor(s) 1064 can be configured for a horizontal field of view between 45 degrees and 135 degrees. Fig. Figure 10B illustrates exemplary horizontal long-range and short-range fields of view for a LiDAR sensor 1064 with an exemplary mounting position above the windshield, but other configurations, such as those using a grill-mounted LiDAR sensor 1064 (e.g., as in Fig. 10A illustrated) and / or include a roof-mounted LiDAR scanner (e.g. for a data collection vehicle), are possible.

[0160] In some examples, LiDAR technologies, such as 3D flash LiDAR, can also be used. 3D flash LiDAR uses a laser pulse as a transmission source to illuminate the vehicle's surroundings up to approximately 200 m. A flash LiDAR unit includes a receiver that records the laser pulse time-of-flight and the reflected light at each pixel, which corresponds to the range from the vehicle to the objects. Flash LiDAR can enable the creation of highly accurate and distortion-free images of the environment with each laser pulse. In some examples, four flash LiDAR sensors can be used, one on each side of the vehicle. Available 3D flash LiDAR systems include a solid-state 3D staring array LiDAR camera with no moving parts other than a fan (e.g., a non-scanning LiDAR device).The flash LiDAR device can use a 5-nanosecond Class I (eye-safe) laser pulse per frame and can capture the reflected laser light in the form of 3D area point clouds and jointly recorded intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor(s) 1064 may be less susceptible to motion blur, vibration, and / or shock.

[0161] The vehicle may further include one or more IMU sensor(s) 1066. The IMU sensor(s) 1066 may be located in the center of the rear axle of the vehicle 1000 in some examples. The IMU sensor(s) 1066 may include, for example, and without limitation, one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more magnetic compasses, and / or other sensor types. In some examples, such as in six-axis applications, the IMU sensor(s) 1066 may include accelerometers and gyroscopes, while in nine-axis applications, the IMU sensor(s) 1066 may include accelerometers, gyroscopes, and magnetometers.

[0162] In some embodiments, the IMU sensor(s) 1066 can be implemented as a miniature, high-performance GPS-based inertial navigation system (GPS / INS) that combines microelectromechanical system (MEMS) inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filter algorithms to provide estimates of position, velocity, and orientation. Thus, in some examples, the IMU sensor(s) 1066 can enable the vehicle 1000 to estimate its course without requiring input from a magnetic sensor by directly observing velocity changes from the GPS and correlating them with the IMU sensor(s) 1066. In some examples, the IMU sensor(s) 1066 and the GNSS sensor(s) 1058 can be combined in a single integrated unit.

[0163] The vehicle may contain one or more microphone(s) 1096, which are placed in and / or around the vehicle 1000. The microphone(s) 1096 may be used, among other things, for emergency vehicle detection and identification.

[0164] The vehicle may also include any number of camera types, including stereo camera(s) 1068, widescreen camera(s) 1070, infrared camera(s) 1072, surround-view camera(s) 1074, long-range and / or medium-range camera(s) 1098, and / or other camera types. The cameras can be used to capture image data around the entire periphery of the vehicle 1000. The camera types used depend on the embodiment and requirements for the vehicle 1000, and any combination of camera types can be used to provide the necessary coverage around the vehicle 1000. Additionally, the number of cameras can vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As an example, and without limitation, the cameras may support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet.Each of the camera(s) is described herein in relation to . Fig. 10A and Fig. 10B described in more detail.

[0165] The vehicle 1000 may also include one or more vibration sensor(s) 1042. The vibration sensor(s) 1042 can measure vibrations of vehicle components, such as the axle(s). For example, changes in vibration may indicate a change in the road surface. In another example, if two or more vibration sensors 1042 are used, the differences between the vibrations can be used to determine friction or slippage of the road surface (e.g., if the difference in vibration is between a driven axle and a freely rotating axle).

[0166] Vehicle 1000 may include an ADAS system 1038. In some examples, the ADAS system 1038 may include a system of compute (SoC). The ADAS system 1038 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic alert (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functionality.

[0167] The ACC systems can use one or more radar sensors (1060), one or more LiDAR sensors (1064), and / or one or more cameras. The ACC systems can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately in front of the vehicle (1000) and automatically adjusts the vehicle speed to maintain a safe distance from vehicles ahead. Lateral ACC maintains distance and advises the vehicle (1000) to change lanes if necessary. Lateral ACC relates to other ADAS applications such as LCA and CWS.

[0168] CACC uses information from other vehicles, which can be received via the network interface 1024 and / or the wireless antenna(s) 1026 from other vehicles either wirelessly or indirectly via a network connection (e.g., via the internet). Direct connections can be provided through a vehicle-to-vehicle (V2V) communication link, while indirect connections can be an infrastructure-to-vehicle (I2V) communication link. Generally, the V2V communication concept provides information about the vehicles immediately ahead (e.g., vehicles directly in front of and in the same lane as vehicle 1000), while the I2V communication concept provides information about traffic further ahead. CACC systems can incorporate one or both I2V and V2V information sources.Given the information from the vehicles ahead of vehicle 1000, CACC can be more reliable and has the potential to improve traffic flow and reduce congestion on the road.

[0169] FCW systems are designed to warn the driver of a hazard so that the driver can take corrective action. FCW systems use a forward-facing camera and / or one or more RADAR sensor(s) 1060 coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is / are electrically coupled with driver feedback, such as a display, speaker, and / or vibration component. FCW systems can provide a warning, such as an audible signal, a visual warning, vibration, and / or an emergency braking pulse.

[0170] AEB systems detect an impending forward collision with another vehicle or object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. AEB systems can use one or more forward-facing cameras and / or one or more radar sensors coupled with a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first warns the driver to take corrective action to avoid the collision. If the driver fails to take corrective action, the AEB system can automatically apply the brakes to prevent or at least mitigate the effects of the predicted collision. AEB systems may incorporate techniques such as dynamic brake assist and / or anticipatory braking.

[0171] Lane Departure Warning (LDW) systems provide visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle crosses lane markings. An LDW system will not activate if the driver indicates an intention to leave the lane by activating a turn signal. LDW systems may use forward-facing cameras coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is / are electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0172] Lane Keeping Assist (LKA) systems are a variation of Lane Departure Warning (LDW) systems. LKA systems provide steering input or braking to correct the vehicle if it begins to leave its lane.

[0173] Blind Spot Warning (BSW) systems detect and warn the driver of vehicles in a car's blind spot. BSW systems can provide a visual, audible, and / or tactile warning to indicate that merging or changing lanes is unsafe. The system can provide an additional warning if the driver uses a turn signal. BSW systems can use one or more rear-facing cameras and / or one or more radar sensors (1060) coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled with driver feedback, such as a display, speaker, and / or vibration component.

[0174] RCTW systems can provide visual, audible, and / or tactile alerts when an object is detected outside the rear camera's field of view while the vehicle is reversing. Some RCTW systems incorporate AEB to ensure the vehicle's brakes are applied to avoid a collision. RCTW systems may use one or more rear-facing radar sensors coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0175] Conventional ADAS systems can be prone to false positives, which can be annoying and distracting for a driver, but are typically not catastrophic because the ADAS systems warn the driver and allow the driver to decide whether a safety condition actually exists and act accordingly. However, in an autonomous vehicle 1000, the vehicle 1000 itself must decide, in the case of conflicting results, whether to heed the result from a primary computer or a secondary computer (e.g., a first controller 1036 or a second controller 1036). For example, in some embodiments, the ADAS system 1038 can be a backup and / or secondary computer for providing perceptual information to a backup computer rationality module.The backup computer rationality monitor can run redundant, diverse software on hardware components to detect errors in perception and dynamic driving tasks. Outputs from the ADAS system 1038 can be provided to a monitoring MCU. If outputs from the primary and secondary computers conflict, the monitoring MCU must determine how to resolve the conflict to ensure safe operation.

[0176] In some examples, the primary computer can be configured to provide the monitoring MCU with a confidence score indicating its level of trust in the chosen outcome. If the confidence score exceeds a certain threshold, the monitoring MCU can follow the primary computer's direction, regardless of whether the secondary computer provides a conflicting or inconsistent result. If the confidence score does not meet the threshold, and if the primary and secondary computers report different results (e.g., conflicting results), the monitoring MCU can arbitrate between the computers to determine the appropriate outcome.

[0177] The monitoring MCU can be configured to run a neural network (or networks) that is trained and configured to determine, based on outputs from the primary and secondary computers, the conditions under which the secondary computer provides false alarms. Thus, the neural network (or networks) in the monitoring MCU can learn when the output from the secondary computer can be trusted and when it cannot. For example, if the secondary computer is a radar-based FCW system, a neural network (or networks) in the monitoring MCU can learn when the FCW system identifies metallic objects that are not actually hazards, such as a drainage grate or manhole cover, triggering an alarm.Similarly, if the secondary computer is a camera-based lane departure warning (LDW) system, a neural network in the monitoring MCU can learn to override the LDW when cyclists or pedestrians are present and leaving the lane is indeed the safest maneuver. In embodiments that include a neural network (or networks) running on the monitoring MCU, the monitoring MCU may include at least one DLA or GPU suitable for running the neural network (or networks) with associated memory. In preferred embodiments, the monitoring MCU may include the SoC(s) 1004 and / or be included as a component thereof.

[0178] In other examples, the ADAS system 1038 can include a secondary computer that performs ADAS functionality using conventional computer vision rules. Thus, the secondary computer can use classic computer vision rules (if-then), and the presence of a neural network (or neural networks) in the monitoring MCU can improve reliability, safety, and performance. For example, the diverse implementation and intended non-identity make the overall system more fault-tolerant, especially to errors caused by software (or software-hardware interface) functionality.For example, if there is a software bug or error in the software running on the primary computer, and the non-identical software code running on the secondary computer provides the same overall result, the monitoring MCU can have greater confidence that the overall result is correct, and the bug in software or hardware on the primary computer does not cause a material error.

[0179] In some examples, the output of the ADAS system 1038 can be fed into the perception block and / or the dynamic driving task block of the primary computer. For example, if the ADAS system 1038 displays a forward collision warning due to an object immediately ahead, the perception block can use this information when identifying objects. In other examples, the secondary computer may have its own trained neural network, thus reducing the risk of false positives, as described herein.

[0180] The Vehicle 1000 may also include the Infotainment SoC 1030 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as a single SoC, the infotainment system may not be a SoC and may include two or more discrete components. The Infotainment SoC 1030 may include a combination of hardware and software that can be used to provide audio (e.g., music, a personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., television, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation systems, parking assistance, a radio data system, vehicle-related information such as fuel level, total distance traveled, brake fluid level, oil level, door opening / closing information, air filter information, etc.) to the Vehicle 1000.For example, the Infotainment SoC 1030 can include radios, record players, navigation systems, video game players, USB and Bluetooth connectivity, car computers, in-vehicle entertainment, Wi-Fi, steering wheel audio controls, hands-free voice control, a head-up display (HUD), an HMI display 1034, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. The Infotainment SoC 1030 can also be used to provide information (e.g., visual and / or audible) to a vehicle user(s), such as information from the ADAS system 1038, autonomous driving information such as planned vehicle maneuvers, trajectories, environmental information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0181] The infotainment SoC 1030 can include GPU functionality. The infotainment SoC 1030 can communicate with other devices, systems, and / or components of the vehicle 1000 via the bus 1002 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 1030 can be coupled with a monitoring MCU so that the infotainment system's GPU can perform some autonomous functions in the event of a failure of the primary controller(s) 1036 (e.g., the primary and / or backup computers of the vehicle 1000). In such an example, the infotainment SoC 1030 can put the vehicle 1000 into a chauffeur-to-safe-stop mode, as described herein.

[0182] The vehicle 1000 may further include an instrument cluster 1032 (e.g., a digital instrument cluster, an electronic instrument cluster, a digital dashboard, etc.). The instrument cluster 1032 may include a controller and / or a supercomputer (e.g., a discrete controller or a supercomputer). The instrument cluster 1032 may include a set of instruments, such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn signals, gear position indicator, seat belt warning light(s), parking brake warning light(s), engine malfunction light(s), airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 1030 and the instrument cluster 1032. Thus, the instrument cluster 1032 may be included as part of the infotainment SoC 1030, or vice versa.

[0183] Fig. 10D is a system diagram for communication between one or more cloud-based servers and the exemplary autonomous vehicle 1000. Fig. 10A according to some embodiments of the present disclosure. The system 1076 may include one or more servers 1078, one or more networks 1090, and vehicles, including the vehicle 1000. The server(s) 1078 may include a plurality of GPUs 1084(A) - 1084(H) (hereinafter collectively referred to as GPUs 1084), PCIe switches 1082(A) - 1082(D) (hereinafter collectively referred to as PCIe switches 1082), and / or CPUs 1080(A) - 1080(B) (hereinafter collectively referred to as CPUs 1080). The GPUs (1084), CPUs (1080), and PCIe switches can be connected via high-speed links, such as, but not limited to, NVLink interfaces (1088) developed by NVIDIA and / or PCIe connections (1086). In some examples, the GPUs (1084) are connected via NVLink and / or NVSwitch SoCs, and the GPUs (1084) and PCIe switches (1082) are connected via PCIe links.Although eight GPUs 1084, two CPUs 1080, and two PCIe switches are illustrated, this is not intended to be restrictive. Depending on the implementation, each Server 1078 can include any number of GPUs 1084, CPUs 1080, and / or PCIe switches. For example, the Server 1078 can each include eight, sixteen, thirty-two, and / or more GPUs 1084.

[0184] The server(s) 1078 can receive image data from the network(s) 1090 and the vehicles, representing images that show unexpected or changed road conditions, such as recently started roadworks. The server(s) 1078 can transmit neural networks 1092, updated neural networks 1092, and / or map information 1094, including information regarding traffic and road conditions, to the vehicles via the network(s) 1090 and the vehicles. Updates to the map information 1094 may include updates to the HD map 1022, such as information regarding construction sites, potholes, detours, flooding, and / or other obstacles.In some examples, the neural networks 1092, the updated neural networks 1092 and / or the map information 1094 may result from new training and / or experience represented in data received from any number of vehicles in the environment, and / or based on training performed in a data center (e.g. using server(s) 1078 and / or other servers).

[0185] Server(s) 1078 can be used to train machine learning models (e.g., neural networks) based on training data. The training data can be generated using vehicles and / or in a simulation (e.g., using a game machine). In some examples, the training data is tagged (e.g., where the neural network benefits from supervised learning) and / or subjected to other preprocessing, while in other examples, the training data is not tagged and / or preprocessed (e.g., where the neural network does not require supervised learning).The training can be performed according to any one or more classes of machine learning techniques, including, without limitation, classes such as: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, diverse learning, representational learning (including substitute dictionary learning), rule-based machine learning, anomaly detection, and any variants or combinations thereof. Once the machine learning models are trained, they can be used by the vehicles (e.g., transmitted to the vehicles via network(s) 1090) and / or the machine learning models can be used by server(s) 1078 to remotely monitor the vehicles.

[0186] In some examples, the Server(s) 1078 can receive data from the vehicles and apply the data to current real-time neural networks for intelligent real-time inference. The Server(s) 1078 can include deep learning supercomputers and / or dedicated AI computers powered by the GPU(s) 1084, such as NVIDIA's DGX and DGX Station machines. However, in some examples, the Server(s) 1078 can include deep learning infrastructure that uses only CPU-powered data centers.

[0187] The deep learning infrastructure of server(s) 1078 is capable of fast, real-time inference and can use this capability to assess and verify the integrity of the processors, software, and / or associated hardware in vehicle 1000. For example, the deep learning infrastructure can receive periodic updates from vehicle 1000, such as a sequence of images and / or objects indicating where vehicle 1000 was located within that sequence of images (e.g., via computer vision and / or other machine learning object classification techniques).The deep learning infrastructure can operate its own neural network to identify the objects and compare them with the objects identified by vehicle 1000, and if the results do not match and the infrastructure concludes that the AI ​​in vehicle 1000 is malfunctioning, the server 1078 can transmit a signal to vehicle 1000 that instructs a fail-safe computer in vehicle 1000 to take over control, notify the passengers and complete a safe parking maneuver.

[0188] For inference, the server(s) 1078 can include the GPU(s) 1084 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of GPU-powered servers and inference acceleration can enable real-time responsiveness. In other scenarios, such as when performance is less critical, servers powered by CPUs, FPGAs, and other processors can be used for inference. INFERENCE AND TRAINING LOGIC

[0189] One or more embodiments can be implemented using inference and / or training logic to perform inference and / or training operations. Details regarding inference and / or training logic are provided below.

[0190] In at least one embodiment, inference and / or training logic can, without limitation, include code and / or data storage to store forward and / or output weights and / or input / output data and / or other parameters to configure neurons or layers of a neural network that are trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, training logic can include or be coupled with code and / or data storage to store graphics code or other software to control the timing and / or order in which weights and / or other parameter information are loaded to configure logic, including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)).In at least one embodiment, code, such as graphics code, loads weights or other parameter information into processor ALUs based on a neural network architecture to which the code corresponds. In at least one embodiment, code and / or data storage stores weight parameters and / or input / output data of each layer of a neural network, which, in conjunction with one or more embodiments, is trained or used during forward propagation of input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments. In at least one embodiment, any part of the code and / or data storage may be contained in another on-chip or off-chip data storage, including the L1, L2, or L3 cache or system memory of a processor.

[0191] In at least one embodiment, any part of the code and / or data storage can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage can be cache memory, dynamic direct addressable memory (“DRAM”), static direct addressable memory (“SRAM”), non-volatile memory (e.g., flash memory), or other types of memory. In at least one embodiment, the choice of whether the code and / or data storage is, for example, internal or external to a processor, or whether it comprises DRAM, SRAM, flash, or another type of memory, can depend on available on-chip versus off-chip memory, the latency requirements of trained and / or inferencing functions being performed, the batch size of data used in inference and / or training of a neural network, or a combination of these factors.

[0192] In at least one embodiment, inference and / or training logic can, without limitation, include code and / or data storage to store backward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network that are trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the code and / or data storage stores weight parameters and / or input / output data of each layer of a neural network that are trained or used in conjunction with one or more embodiments during backward propagation of input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments.In at least one embodiment, training logic can include or be coupled with code and / or data storage to store graphics code or other software to control the timing and / or sequence into which weights and / or other parameter information are loaded to configure logic, including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, code, such as graphics code, loads weights or other parameter information into processor ALUs based on a neural network architecture to which the code corresponds. In at least one embodiment, any part of the code and / or data storage can be contained in another on-chip or off-chip data storage, including the L1, L2, or L3 cache or system memory of a processor.In at least one embodiment, any part of the code and / or data storage can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage can be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other types of memory. In at least one embodiment, the choice of whether the code and / or data storage is, for example, internal or external to a processor, or whether it comprises DRAM, SRAM, flash memory, or another type of memory, can depend on available on-chip versus off-chip memory, the latency requirements of trained and / or inferencing functions being performed, the batch size of data used in inference and / or training of a neural network, or a combination of these factors.

[0193] In at least one embodiment, code and / or data storage can be separate memory structures. In at least one embodiment, code and / or data storage can be the same memory structure. In at least one embodiment, code and / or data storage can be partly the same memory structure and partly separate memory structures. In at least one embodiment, any part of code and / or data storage can be contained in another on-chip or off-chip data storage, including the L1, L2, or L3 cache or system memory of a processor.

[0194] In at least one embodiment, inference and / or training logic can, without limitation, include one or more arithmetic logic units (“ALU(s)”), including integer and / or floating-point units, to perform logical and / or mathematical operations that are at least partially based on or indicated by training and / or inference code (e.g., graphics code), wherein a result thereof can generate activations (e.g., output values ​​of layers or neurons within a neural network) that are stored in an activation memory, which are functions of input / output and / or weight parameter data stored in code and / or data memory.In at least one embodiment, activations stored in activation memory are generated according to linear algebraic and / or matrix-based mathematics, performed by ALU(s) in response to the execution of instructions or other code, wherein weight values ​​stored in code and / or data memory and / or code and / or data memory are used as operands together with other values, such as bias values, gradient information, pulse values, or other parameters or hyperparameters, any or all of which may be stored in code and / or data memory or code and / or data memory or other on-chip or off-chip memory.

[0195] In at least one embodiment, ALU(s) are contained within one or more processors or other hardware logic devices or circuits, whereas in another embodiment, ALU(s) may be external to a processor or other hardware logic device or circuit that uses them (e.g., a coprocessor). In at least one embodiment, one or more ALUs may be contained within the execution units of a processor or otherwise in a bank of ALUs that can be accessed by the execution units of a processor, either within the same processor or distributed among different processors of different types (e.g., central processing units, graphics processing units, fixed functional units, etc.).In at least one embodiment, code and / or data storage, code and / or data storage and activation memory can be located on the same processor or other hardware logic device or circuit, whereas in another embodiment they can be located on different processors or other hardware logic devices or circuits, or a combination of the same and different processors or other hardware logic devices or circuits. In at least one embodiment, any part of the activation memory can be contained in another on-chip or off-chip data storage, including the L1, L2, or L3 cache or system memory of a processor.Furthermore, inference and / or training code may be stored with other code that a processor or other hardware logic or circuitry can access and that is retrieved and / or processed using the fetch, decode, schedule, execute, retract and / or other logical circuitry of a processor.

[0196] In at least one embodiment, activation memory can be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other memory. In at least one embodiment, activation memory can be located wholly or partially inside or external to one or more processors or other logic circuits. In at least one embodiment, the choice of whether activation memory is, for example, internal or external to a processor, or whether it comprises DRAM, SRAM, flash, or another type of memory, can depend on available on-chip versus off-chip memory, the latency requirements of trained and / or inferencing functions being performed, the batch size of data used in inference and / or training of a neural network, or a combination of these factors.In at least one embodiment, inference and / or training logic can be used in conjunction with an application-specific integrated circuit (“ASIC”), such as a Google TensorFlowe processing unit, a Graphcore™ inference processing unit (IPU), or an Intel Corp. Nervana® processor (e.g., “Lake Crest”). In at least one embodiment, inference and / or training logic can be used in conjunction with hardware of a central processing unit (“CPU”), graphics processing unit (“GPU”), or other hardware, such as field-programmable gate arrays (“FPGAs”).

[0197] In at least one embodiment, inference and / or training logic can, without limitation, include hardware logic in which computing resources are dedicated or otherwise used exclusively in conjunction with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, inference and / or training logic can be used in conjunction with an application-specific integrated circuit (ASIC), such as a Google TensorFlow® processing unit, a Graphcore™ inference processing unit (IPU), or an Intel Corp. Nervana® processor (e.g., "Lake Crest").In at least one embodiment, inference and / or training logic can be used in conjunction with hardware of a central processing unit (CPU), graphics processing unit (GPU), or other hardware, such as field-programmable gate arrays (FPGAs). In at least one embodiment, inference and / or training logic includes, without limitation, code and / or data storage that can be used to store code (e.g., graphics code), weight values, and / or other information, including bias values, gradient information, pulse values, and / or other parameter or hyperparameter information. In at least one embodiment, each of the code and / or data storage is dedicated to a specific computing resource, such as compute hardware.In at least one embodiment, each of the computing hardware and computing hardware comprises one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on information stored in code and / or data memory, or code and / or data memory, the result of which is stored in activation memory.

[0198] In at least one embodiment, each of the code and / or data storage and corresponding computing hardware corresponds to different layers of a neural network, such that the resulting activation from one memory / computing pair of code and / or data storage and computing hardware is provided as an input to another memory / computing pair of code and / or data storage and computing hardware to reflect a conceptual organization of a neural network. In at least one embodiment, each of the memory / computing pairs can correspond to more than one layer of a neural network. In at least one embodiment, additional memory / computing pairs (not shown) can be included downstream of or in parallel to memory / computing pairs in inference and / or training logic. EXAMPLE CALCULATION DEVICE

[0199] Fig. Figure 11 is a block diagram of an exemplary computing device(s) 1100 suitable for use in implementing some embodiments of the present disclosure. Computing device 1100 may include a connection system 1102 that directly or indirectly couples the following devices: memory 1104, one or more central processing units (CPUs) 1106, one or more graphics processing units (GPUs) 1108, a communication interface 1110, input / output (I / O) ports 1112, input / output components 1114, a power supply 1116, one or more presentation components 1118 (e.g., display(s)), and one or more logic units 1120. In at least one embodiment, the computing device(s) 1100 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components).For non-restrictive examples, one or more of the GPUs 1108 can comprise one or more vGPUs, one or more of the CPUs 1106 can comprise one or more vCPUs, and / or one or more of the logic units 1120 can comprise one or more virtual logic units. Accordingly, a computing device (or devices) 1100 can include discrete components (e.g., a complete GPU dedicated to the computing device 1100), virtual components (e.g., a portion of a GPU dedicated to the computing device 1100), or a combination thereof.

[0200] Although the various blocks made Fig. Where components 11 are shown to be connected via the connection system 1102, this is not intended to be restrictive and serves only for clarity. For example, in some embodiments, a presentation component 1118, such as a display device, may be considered an I / O component 1114 (e.g., if the display is a touchscreen). As another example, the CPUs 1106 and / or GPUs 1108 may include memory (e.g., the memory 1104 may be representative of a memory device in addition to the memory of the GPUs 1108, the CPUs 1106, and / or other components). Thus, the computing device is composed of Fig. 11 is for illustrative purposes only. No distinction is made between categories such as "workstation", "server", "laptop", "desktop", "tablet", "client device", "mobile device", "handheld device", "game console", "electronic control unit (ECU)", "virtual reality system" and / or other device or system types, as all fall within the scope of the computing device of Fig. 11 are to be considered.

[0201] The 1102 interconnection system can represent one or more connections or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The 1102 interconnection system can include one or more bus or connection types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or connection. In some embodiments, there are direct connections between components. For example, the CPU 1106 can be directly connected to the memory 1104. Furthermore, the CPU 1106 can be directly connected to the GPU 1108. When there is a direct or point-to-point connection between components, the 1102 interconnection system can include a PCIe connection to implement the connection.In these examples, a PCI bus does not need to be included in the computing device 1100.

[0202] The memory 1104 can contain any of a variety of computer-readable media. Computer-readable media can be any available media accessible to the computing device 1100. Computer-readable media can include both volatile and non-volatile media, and removable and non-removable media. By way of example, and without limitation, computer-readable media can include computer storage media and communication media.

[0203] Computer storage media can include both volatile and non-volatile media, and / or removable and non-removable media, implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1104 can store computer-readable instructions (which represent, for example, a program or programs and / or a program element or program elements), such as an operating system.Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory, or any other storage technology; CD-ROM, Digital Versatile Discs (DVD), or any other optical disk storage; magnetic cartridges, magnetic tape, magnetic disk storage, or other magnetic storage devices; or any other medium that can be used to store the desired information and that can be accessed by the Computing Device 1100. As used herein, computer storage media do not contain signals per se.

[0204] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and include any information delivery medium. The term "modulated data signal" can refer to a signal in which one or more of its properties are set or modified in such a way as to encode information in the signal. By way of example, and without limitation, computer storage media can include wired media, such as a wired network or a directly wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media. Combinations of any of the foregoing should also be included within the scope of computer-readable media.

[0205] The CPU(s) 1106 can be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1100 to perform one or more of the procedures and / or processes described herein. The CPU(s) 1106 can each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling a large number of software threads concurrently. The CPU(s) 1106 can include any type of processor and can include different types of processors depending on the type of computing device 1100 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers).For example, depending on the type of computing device 1100, the processor can be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC), or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 1100 can include one or more CPUs 1106 in addition to one or more microprocessors or supplementary coprocessors, such as mathematical coprocessors.

[0206] In addition to or alternatively to the CPU(s) 1106, the GPU(s) 1108 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1100 to perform one or more of the procedures and / or processes described herein. One or more of the GPU(s) 1108 can be an integrated GPU (e.g., one or more of the CPU(s) 1106 and / or one or more of the GPU(s) 1108 can be a discrete GPU). In embodiments, one or more of the GPU(s) 1108 can be a coprocessor of one or more of the CPU(s) 1106. The GPU(s) 1108 can be used by the computing device 1100 to render graphics (e.g., 3D graphics) or to perform general-purpose computing. For example, the GPU(s) 1108 can be used for general-purpose computing on GPUs (GPGPU).The GPU(s) 1108 can contain hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. The GPU(s) 1108 can generate pixel data for output images in response to rendering commands (e.g., rendering commands from the CPU(s) 1106 received via a host interface). The GPU(s) 1108 can include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. The display memory can be included as part of the memory 1104. The GPU(s) 1108 can contain two or more GPUs operating in parallel (e.g., via a link). The link can connect the GPUs directly (e.g., using NVLINK) or connect them via a switch (e.g., using NVSwitch).When combined, each GPU can generate 1108 pixel data or GPGPU data for different parts of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can have its own dedicated memory or share memory with other GPUs.

[0207] In addition to or as an alternative to the CPU(s) 1106 and / or the GPU(s) 1108, the logic unit(s) 1120 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1100 to perform one or more of the methods and / or processes described herein. In embodiments, the CPU(s) 1106, the GPU(s) 1108, and / or the logic unit(s) 1120 may discretely or jointly perform any combination of the methods, processes, and / or sections thereof. One or more of the logic units 1120 may be part of and / or integrated within one or more of the CPU(s) 1106 and / or the GPU(s) 1108, and / or one or more of the logic units 1120 may be discrete components or otherwise separate from the CPU(s) 1106 and / or the GPU(s) 1108.In embodiments, one or more of the logic units 1120 can be a coprocessor of one or more of the CPU(s) 1106 and / or one or more of the GPU(s) 1108.

[0208] Examples of logic unit(s) 1120 include one or more processing cores and / or components thereof, such as data processing units (DPUs), tensor cores (TCs), tensor processing units (TPUs), visual pixel cores (PVCs), vision processing units (VPUs), graphics processing clusters (GPCs), texture processing clusters (TPCs), streaming multiprocessors (SMs), tree traversal units (TTUs), artificial intelligence accelerators (AIAs), deep learning accelerators (DLAs), arithmetic logic units (ALUs), application-specific integrated circuits (ASICs), floating-point units (FPUs), input / output (I / O) elements, peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) elements, and / or the like.

[0209] The communication interface 1110 can include one or more receivers, transmitters, and / or transceivers that enable the computing device 1100 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communication. The communication interface 1110 can include components and functionality to enable communication over any number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or InfiniBand), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.In one or more embodiments, logic unit(s) 1120 and / or communication interface 1110 may include one or more data processing units (DPUs) to directly transfer data received via a network and / or through the connection system 1102 to one or more GPU(s) 1108 (e.g., a memory thereof).

[0210] The I / O ports 1112 enable the computing device 1100 to be logically coupled with other devices, including the I / O components 1114, the presentation component(s) 1118, and / or other components, some of which may be built into (e.g., integrated with) the computing device 1100. Illustrative I / O components 1114 include a microphone, mouse, keyboard, joystick, gamepad, game controller, satellite dish, scanner, printer, wireless device, etc. The I / O components 1114 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some cases, inputs can be transferred to a suitable network element for further processing.A NUI can implement any combination of speech recognition, pen recognition, facial recognition, biometric recognition, gesture recognition (both on-screen and beside the screen), air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with a display of the Computing Device 1100. The Computing Device 1100 can include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture detection and recognition. Additionally, the Computing Device 1100 can include accelerometers or gyroscopes (e.g., as part of an inertial measurement unit (IMU)) that enable motion detection. In some examples, the output from the accelerometers or gyroscopes can be used by the Computing Device 1100 to render immersive augmented reality or virtual reality.

[0211] The power supply 1116 can be a hardwired power supply, a battery power supply, or a combination of both. The power supply 1116 can provide power to the computing device 1100 to enable the components of the computing device 1100 to operate.

[0212] The presentation component(s) 1118 can include a display (e.g., a monitor, a touchscreen, a television screen, a head-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The presentation component(s) 1118 can receive data from other components (e.g., the GPU(s) 1108, the CPU(s) 1106, DPUs, etc.) and output the data (e.g., as an image, video, sound, etc.). EXEMPLARY DATA CENTER

[0213] Fig. Figure 12 illustrates an exemplary data center 1200 that can be used in at least one embodiment of the present disclosure. The data center 1200 can include a data center infrastructure layer 1210, a framework layer 1220, a software layer 1230, and / or an application layer 1240.

[0214] As in Fig. As shown in Figure 12, the data center infrastructure layer 1210 can include a resource orchestrator 1212, clustered compute resources 1214, and node compute resources (“node CRs”) 1216(1)-1216(N), where “N” is any positive integer. In at least one embodiment, the node CRs 1216(1)-1216(N) can include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), storage devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules, and / or cooling modules, etc. In some embodiments, one or more nodes CRs can be made up of the nodes CRs 1216(1)-1216(N) correspond to a server that has one or more of the computing resources mentioned above. Additionally, in some embodiments, the nodes CRs 1216(1)-1216(N) may include one or more virtual components, such as vGPUs, vCPUs and / or the like, and / or one or more of the nodes CRs 1216(1)-1216(N) may correspond to a virtual machine (VM).

[0215] In at least one embodiment, grouped compute resources 1214 can include separate groupings of node CRs 1216 located in one or more racks (not shown) or many racks located in data centers at different geographic locations (also not shown). Separate groupings of node CRs 1216 within grouped compute resources 1214 can include grouped compute, network, storage, or memory resources that can be configured or allocated to support one or more workloads. In at least one embodiment, multiple node CRs 1216, including CPUs, GPUs, DPUs, and / or other processors, can be grouped in one or more racks to provide compute resources to support one or more workloads.The one or more racks can also contain any number of power modules, cooling modules and / or network switches in any combination.

[0216] The resource orchestrator 1212 can configure or otherwise control one or more node CRs 1216(1)-1216(N) and / or grouped compute resources 1214. In at least one embodiment, the resource orchestrator 1212 can include a software design infrastructure (SDI) management entity for the data center 1200. The resource orchestrator 1212 can include hardware, software, or a combination thereof.

[0217] In at least one embodiment, as in Fig. As shown in Figure 12, the framework layer 1220 can include a job scheduler 1233, a configuration manager 1234, a resource manager 1236, and / or a distributed file system 1238. The framework layer 1220 can include a framework to support software 1232 of the software layer 1230 and / or one or more applications 1242 of the application layer 1240. The software 1232 or the application(s) 1242 can each include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1220 can be a type of free and open-source software web application framework, such as Apache Spark™ (hereinafter "Spark"), which can use the distributed file system 1238 for large-scale data processing (e.g., "Big Data"), but is not limited to this.In at least one embodiment, the job scheduler 1233 can include a Spark driver to facilitate the scheduling of workloads supported by different layers of the data center 1200. The configuration manager 1234 can be capable of configuring various layers, such as the software layer 1230 and the framework layer 1220, including Spark and the distributed file system 1238, to support big data processing. The resource manager 1236 can be capable of managing clustered or grouped compute resources allocated or assigned to support the distributed file system 1238 and the job scheduler 1233. In at least one embodiment, clustered or grouped compute resources can include grouped compute resources 1214 on the data center infrastructure layer 1210.The resource manager 1236 can coordinate with the resource orchestrator 1212 to manage these allocated or assigned computing resources.

[0218] In at least one embodiment, the software 1232, which is included in software layer 1230, may include software used by at least parts of the nodes CRs 1216(1)-1216(N), the grouped computing resources 1214, and / or the distributed file system 1238 of the framework layer 1220. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0219] In at least one embodiment, the application(s) 1242 included in the application layer 1240 may include one or more types of applications used by at least parts of the nodes CRs 1216(1)-1216(N), the grouped compute resources 1214, and / or the distributed file system 1238 of the framework layer 1220. One or more types of applications may include any number of a genomic application, a cognitive computation, and a machine learning application, including, but not limited to, training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0220] In at least one embodiment, any configuration manager 1234, resource manager 1236, and resource orchestrator 1212 can implement any number and type of self-modifying actions based on any set and type of data acquired in any technically feasible way. Self-modifying actions can relieve a data center operator of data center 1200 of potentially making poor configuration decisions and potentially avoiding underutilized and / or poorly performing sections of a data center.

[0221] The Data Center 1200 may include tools, services, software, or other resources to train one or more machine learning models or to predict or derive information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model(s) may be trained by calculating weight parameters according to a neural network architecture using software and / or computing resources described above in relation to the Data Center 1200.In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks can be used to derive or predict information using resources described above in relation to the Computing Center 1200, by using weight parameters calculated by one or more training techniques, such as, but not limited to, those described here.

[0222] In at least one embodiment, the data center can use 1200 CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or equivalent virtual computing resources) to perform training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above can be configured as a service to allow users to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services. EXEMPLARY NETWORK ENVIRONMENTS

[0223] Network environments suitable for use in implementing embodiments of the disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may run on one or more instances of the computing device(s) 1100. Fig. 11 be implemented - e.g., each device may include similar components, features, and / or functionality to the computing device(s) 1100. Additionally, if backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices may be included as part of a data center 1200, an example of which is given herein in relation to Fig. 12 is described in more detail.

[0224] Components of a network environment can communicate with each other over a network, which can be wired, wireless, or both. The network can include multiple networks or a network of networks. For example, the network can include one or more Wide Area Networks (WANs), one or more Local Area Networks (LANs), one or more public networks, such as the Internet and / or a Public Switched Telephone Network (PSTN), and / or one or more private networks. If the network includes a wireless telecommunications network, components such as a base station, a communications tower, or even access points (as well as other components) can provide wireless connectivity.

[0225] Compatible network environments can include one or more peer-to-peer network environments—in which case a server may not be included in a network environment—and one or more client-server network environments—in which case one or more servers may be included in a network environment. In peer-to-peer network environments, the functionality described herein with respect to one or more servers can be implemented on any number of client devices.

[0226] In at least one embodiment, a network environment can include one or more cloud-based network environments, a distributed computing environment, a combination thereof, etc. A cloud-based network environment can include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. A framework layer can include a framework to support software of a software layer and / or one or more application layer applications. The software or application(s) may each include web-based service software or applications. In embodiments, one or more of the client devices can use the web-based service software or applications (e.g.,by accessing the service software and / or applications via one or more application programming interfaces (APIs). The framework layer can be a type of free and open-source software web application framework, such as one that can use a distributed file system for large data processing (e.g., "Big Data"), but is not limited to this.

[0227] A cloud-based network environment can provide cloud computing and / or cloud storage, performing any combination (or one or more parts) of the computing and / or data storage functions described herein. Each of these different functions can be distributed across multiple locations of central or core servers (e.g., one or more data centers, which may be distributed across a state, region, country, the globe, etc.). If a connection to a user (e.g., a client device) is relatively close to one or more edge servers, one or more core servers may designate at least one part of the functionality for the edge server(s). A cloud-based network environment can be private (e.g., restricted to a single organization), public (e.g., available to many organizations), and / or a combination of both (e.g., a hybrid cloud environment).

[0228] The client device(s) may include at least some of the components, features, and functionality of the exemplary computing device(s) 1100 described herein in relation to Fig.11 are described. By way of example, and not as a limitation, a client device may be a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smartwatch, a portable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video game player, a video camera, a surveillance device or system, a vehicle, a boat, an aircraft, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded control system, a remote control, a device, a consumer electronics device, a workstation, an edge device,any combination of these defined devices or any other suitable device.

[0229] The disclosure can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program modules, which are executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs specific tasks or implements specific abstract data types. The disclosure can be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, specialized computing devices, etc. The disclosure can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected by a communications network.

[0230] Other variations are within the scope of this disclosure. While disclosed techniques are thus susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described in detail above. It is understood, however, that there is no intention to limit the disclosure to specific disclosed forms or shapes, but rather the intention is to cover all modifications, alternative constructions, and equivalents that fall within the spirit and scope of the disclosure, as defined in the attached claims.

[0231] The use of the terms "a," "an," and "the," and similar references in connection with the description of disclosed embodiments (particularly in connection with the following claims) is to be interpreted as covering both the singular and the plural unless otherwise specified herein or the context clearly contradicts this, and not as a definition of a term. The terms "comprising," "having," "including," and "containing" are to be interpreted as open terms (meaning "including but not limited to") unless otherwise specified. The term "connected," when unmodified and referring to physical connections, is to be interpreted as partially or completely contained in, attached to, or connected to one another, even if something is in between.The recitation of ranges of values ​​is hereinafter intended merely as a short-lived procedure for referring individually to each separate value falling within the range, unless otherwise specified herein, and each separate value is included in the specification as if it were individually recited herein. The use of the term "set" (e.g., "a set of elements") or "subset" is, unless otherwise specified or the context contradicts, to be understood as a non-empty collection comprising one or more elements. Furthermore, unless otherwise specified or the context contradicts, the term "subset" of a corresponding set does not necessarily denote a proper subset of a corresponding set, but subset and corresponding set may be the same.

[0232] Subjunctive language, such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C," unless explicitly stated otherwise or the context clearly contradicts it, is otherwise understood, in context, as it is generally used to indicate that an element, term, etc., can be either A, B, C, or any non-empty subset of a sentence containing A, B, and C. For example, in an illustrative example of a three-element sentence, subjunctive phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such subjunctive language is generally not intended to imply that certain embodiments require the presence of at least one of A, at least one of B, and at least one of C.Additionally, unless otherwise stated or the context contradicts it, the term "multitude" indicates that it is plural (e.g., "a multitude of elements" indicates multiple elements). A multitude is at least two elements, but may be more if this is either explicitly stated or indicated by context. Furthermore, unless otherwise stated or clear from the context, the phrase "based on" means "at least partly based on" and not "exclusively based on".

[0233] Operations of processes described herein may be performed in any suitable order unless otherwise specified herein or the context clearly contradicts this. In at least one embodiment, a process, such as those described herein (or variations and / or combinations thereof), is performed under the control of one or more computer systems configured with executable instructions and is implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors by hardware or combinations thereof. In at least one embodiment, code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions that are executable by one or more processors.In at least one embodiment, a computer-readable storage medium is a non-volatile computer-readable storage medium that excludes volatile signals (e.g., a propagating transient electrical or electromagnetic transmission) but includes non-volatile data storage circuitry (e.g., buffers, caches, and queues) within transceivers of volatile signals. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-volatile computer-readable storage media containing executable instructions (or other memory for storing executable instructions) which, when executed (i.e., as a result of execution) by one or more processors of a computer system, cause the computer system to perform the operations described herein.In at least one embodiment, a set of non-volatile, computer-readable storage media comprises multiple non-volatile storage media, and one or more individual non-volatile storage media within a set of multiple non-volatile, computer-readable storage media lack all code, while multiple non-volatile, computer-readable storage media collectively store all code. In at least one embodiment, executable instructions are executed such that different instructions are executed by different processors—for example, a non-volatile, computer-readable storage medium stores instructions, and a central processing unit (CPU) executes some of the instructions, while a graphics processing unit (GPU) executes other instructions.In at least one embodiment, different components of a computer system have separate processors, and different processors execute different subsets of instructions.

[0234] Accordingly, in at least one embodiment, computer systems are configured to implement one or more services that individually or jointly perform operations of the processes described herein, and such computer systems are configured with applicable hardware and / or software that enables the performance of operations. Furthermore, a computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment, it is a distributed computer system comprising several devices that operate differently, such that the distributed computer system performs the operations described herein and such that a single device does not perform all the operations.

[0235] The use of any and all examples or of the exemplary language provided herein (e.g., "like") is intended only to better illustrate embodiments of the disclosure and does not constitute a limitation of the scope of the disclosure unless otherwise claimed. No language in the specification should be interpreted as indicating that an unclaimed element is essential for the practical implementation of the disclosure.

[0236] Unless expressly stated otherwise, terms such as "process", "calculate", "calculate", "determine" or the like throughout this specification are understood to refer to actions and / or processes of a computer or computing system or similar electronic computing device that manipulate and / or convert data represented as physical, such as electronic, quantities within the registers and / or memory of the computing system into other data similarly represented as physical quantities within the memory, registers or other such information storage, transmission or display devices of the computing system.

[0237] Similarly, the term "processor" can refer to any device or part of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As non-restrictive examples, "processor" can be a CPU or a GPU. A "computing platform" can have one or more processors. As used herein, "software" processes can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Furthermore, each process can refer to multiple processes for executing instructions sequentially or in parallel, continuously or intermittently.The terms “system” and “procedure” are used interchangeably herein, insofar as the system can embody one or more procedures and procedures can be considered as a system.

[0238] This document may refer to the acquisition, capture, reception, or input of analog or digital data into a subsystem, computer system, or computer-implemented machine. Acquiring, capturing, receiving, or inputting analog and digital data can be accomplished in a variety of ways, such as receiving data as a parameter of a function call or an application programming interface (API) call. In some implementations, the process of acquiring, capturing, receiving, or inputting analog or digital data can be achieved by transmitting data over a serial or parallel interface. In another implementation, the process of acquiring, capturing, receiving, or inputting analog or digital data can be achieved by transmitting data over a computer network from a providing entity to a receiving entity.It can also refer to the provision, output, transmission, sending, or presentation of analog or digital data. In various examples, the process of providing, outputting, transmission, sending, or presenting analog or digital data can be achieved by transmitting data as an input or output parameter of a function call, a parameter of an application programming interface, or an interprocess communication mechanism.

[0239] Although the preceding discussion presents exemplary implementations of the described techniques, other architectures may be used to implement the described functionality and are intended to be within the scope of this disclosure. Furthermore, although specific distributions of responsibilities are defined above for the purpose of discussion, various functions and responsibilities may be distributed and subdivided in different ways depending on the circumstances.

[0240] Although the subject matter has been described in language specific to structural features and / or methodological actions, it is further understood that the subject matter claimed in the appended claims is not necessarily limited to certain described features or actions. Rather, specific features and actions are disclosed as exemplary forms of implementing the claims. The subject matter of this disclosure is described herein with specificity to satisfy legal requirements. However, the description itself is not intended to limit the scope of this disclosure. Instead, the inventors have considered that the claimed subject matter could also be embodied in other ways to include different steps or combinations of steps similar to those described in this document, in conjunction with other present or future technologies.Although the terms “step” and / or “block” may be used herein to denote different elements of procedures used, the terms should not be further interpreted as implying any particular order between or between different steps disclosed herein, unless and except where the order of individual steps is explicitly described. EXEMPLARY LITERARY SUPPORT

[0241] The disclosure of this application also includes the following numbered clauses:

[0242] Clause 1. One or more processors comprising processing circuitry for obtaining a scene embedding representing the environment, based at least on applying a representation of a sequence of sensor data generated using one or more sensors of an ego machine and a coded representation of one or more corresponding perspectives of the one or more sensors to one or more first neural networks (NNs) comprising one or more encoder networks.

[0243] Clause 2. The one or more processors according to Clause 1, wherein the processing circuit shall furthermore produce one or more outputs based at least on applying a representation of the scene embedding to one or more second NNs.

[0244] Clause 3. The one or more processors according to Clause 1 or 2, wherein the processing circuit shall further control one or more operations of the Ego machine based on at least the one or more outputs.

[0245] Clause 4. The one or more processors according to Clause 1, 2 or 3, wherein the processing circuit shall further provide the scene embedding based at least on processing the representation of the sequence of sensor data using cross-attention.

[0246] Clause 5. The one or more processors according to Clause 1, 2 or 3, wherein the processing circuit shall further apply the coded representation of the one or more corresponding perspectives of the one or more sensors to the one or more first NNs based at least on combining one or more coded calibration parameters associated with the one or more sensors with one or more position codings.

[0247] Clause 6. The one or more processors according to Clause 1, 2 or 3, wherein the processing circuit shall further apply the coded representation of the one or more corresponding perspectives of the one or more sensors to the one or more first NNs based at least on combining one or more coded directions of one or more light beams thrown into the environment by the one or more sensors with one or more position codes.

[0248] Clause 7. The one or more processors according to Clause 1, 2 or 3, wherein the processing circuit shall further apply the coded representation of the one or more corresponding perspectives of the one or more sensors to the one or more first NNs based at least on combining a representation of one or more corresponding positions of the one or more sensors relative to a reference point associated with the Ego machine with one or more position encodings.

[0249] Clause 8. The one or more processors according to Clause 1, 2 or 3, wherein the processing circuit shall further receive the scene embedding based at least on extracting one or more scene tokens using the one or more first NNs and combining the one or more scene tokens with a representation of one or more planned navigation routes of the Ego machine.

[0250] Clause 9. The one or more processors according to Clause 1, 2 or 3, wherein the processing circuit shall further receive the scene embedding based at least on extracting one or more scene tokens using the one or more first NNs and combining the one or more scene tokens with one or more top-down representations of one or more planned trajectories of the Ego machine.

[0251] Clause 10. The one or more processors according to Clause 1, 2 or 3, wherein the processing circuit shall further generate the scene embedding based at least on extracting one or more scene tokens using the one or more first NNs and combining the one or more scene tokens with a representation of a planned sequence of two-dimensional waypoints of the Ego machine.

[0252] Clause 11. The one or more processors according to Clause 1, 2 or 3, wherein the processing circuit shall further generate the scene embedding based at least on extracting one or more scene tokens using the one or more first NNs and combining the one or more scene tokens with a representation of a sequence of planned navigation route commands assigned to the Ego machine.

[0253] Clause 12. The one or more processors according to Clause 1, 2 or 3, wherein the processing circuit shall further generate the scene embedding based at least on extracting one or more scene tokens using the one or more first NNs and combining the one or more scene tokens with a representation of detected ego movement of the ego machine.

[0254] Clause 13. The one or more processors according to claim 1, 2 or 3, wherein the one or more first and second NNs represent at least a part of a probabilistic state simulation stack having a navigation policy.

[0255] Clause 14. The one or more processors according to Clause 1, 2 or 3, wherein the application of the scene embedding representation to the one or more second NNs causes the one or more second NNs to perform at least one of one or more 3D perception tasks or one or more 3D reconstruction tasks based at least on the scene embedding extracted using one or more transformer NNs of the one or more encoder networks.

[0256] Clause 15. One or more processors according to Clause 1, 2 or 3, wherein the one or more processors are included in at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content generation for 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or displaying one or more augmented reality content, virtual reality content or mixed reality content; a system implemented using an edge device;a system implemented using a robot; a system for performing conversational AI operations; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system implementing one or more vision language models (VLMs); a system implementing one or more multimodal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system for performing one or more generative AI operations; a system involving one or more virtual machines (VMs); a system using or employing one or more inference microservices; a system involving one or more machine learning models deployed in a service or microservice along with an OS-level virtualization package (e.g., a container);a system that is at least partially implemented in a data center; or a system that is at least partially implemented using cloud computing resources.

[0257] Clause 16. A method comprising extraction based at least on applying a representation of a temporal sequence of sensor data generated using one or more sensors of an Ego-Machine in an environment to one or more neural networks (NNs) having one or more encoders, a scene embedding representing at least part of the environment.

[0258] Clause 17. The procedure according to Clause 16, which further includes controlling one or more operations of the Ego Machine based at least on the scene embedding.

[0259] Clause 18. The method according to Clause 16 or 17, wherein the extraction of the scene embedding is further based at least on applying a coded representation of one or more corresponding perspectives of the one or more sensors to one or more NNs.

[0260] Clause 19. The method according to Clause 16 or 17, wherein the scene embedding is obtained based at least on the fact that one or more NNs process the representation of the temporal sequence of sensor data using cross-attention.

[0261] Clause 20. A method according to Clause 16 or 17, wherein the method is performed by at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or displaying one or more augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot;a system for performing conversational AI operations; a system that implements one or more language models; a system that implements one or more large language models (LLMs); a system that implements one or more vision language models (VLMs); a system that implements one or more multimodal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system for performing one or more generative AI operations; a system that involves one or more virtual machines (VMs); a system that uses or employs one or more inference microservices; a system that involves one or more machine learning models deployed in a service or microservice along with an OS-level virtualization package (e.g., a container); a system that is at least partially deployed in a data center;or a system that is implemented at least partially using cloud computing resources.

[0262] Clause 21. A system comprising one or more processors for controlling, within a simulation rendered using one or more light transport simulation algorithms, one or more operations of a simulated ego machine based on at least one scene embedding representing a simulated environment in the simulation, wherein the scene embedding is obtained based on at least applying a representation of a sequence of simulated sensor data generated using one or more simulated sensors of the simulated ego machine to one or more first neural networks (NNs) comprising one or more encoders.

[0263] Clause 22. System according to Clause 21, wherein the simulation is generated at least using a three-dimensional (3D) content collaboration platform for 3D assets.

[0264] Clause 23. System according to Clause 22, wherein the 3D content collaboration platform for 3D assets uses OpenUSD.

[0265] Clause 24. One or more processors comprising processing circuitry for updating one or more neural networks (NNs) of a probabilistic state simulation stack based at least on probabilistic sampling of one or more estimated scene states.

[0266] Clause 25. The one or more processors according to Clause 24, wherein the processing circuit is further provided for controlling one or more operations of an Ego machine based at least on operating at least a part of the probabilistic state simulation stack as a control stack of the Ego machine.

[0267] Clause 26. The one or more processors according to claim 24 or 25, wherein the one or more NNs comprise one or more neural transformer networks, and the probabilistic state simulation stack comprises a navigation policy implemented using the one or more neural transformer networks.

[0268] Clause 27. The one or more processors according to Clause 24 or 25, wherein the processing circuit is further provided for predicting one or more Ego trajectories of the Ego machine using a navigation guideline of at least part of the probabilistic state simulation stack.

[0269] Clause 28. The one or more processors according to Clause 24 or 25, wherein the processing circuitry is further provided for updating the probabilistic state simulation stack based at least on decoding the one or more estimated scene states using latent diffusion.

[0270] Clause 29. The one or more processors according to Clause 24 or 25, wherein the processing circuit is further provided for updating the probabilistic state simulation stack based at least on updating one or more navigation guidelines of the probabilistic state simulation stack at least partially concurrently with one or more neural scene state estimation networks of the probabilistic state simulation stack.

[0271] Clause 30. The one or more processors according to Clause 24 or 25, wherein operating at least part of the probabilistic state simulation stack as the control stack includes: extracting one or more visual features using a first neural transformer network (NN) of the one or more NNs and extracting one or more scene tokens based on at least a second transformer NN of the one or more NNs that processes a representation of the one or more visual features.

[0272] Clause 31. The one or more processors according to claim 24 or 25, wherein operating at least the part of the probabilistic state simulation stack as the control stack comprises applying a coded representation of one or more corresponding perspectives of one or more sensors of the Ego machine to one or more transformer NNs of the one or more NNs.

[0273] Clause 32. The one or more processors according to Clause 24 or 25, wherein the processing circuit is further provided for updating the probabilistic state simulation stack without decoding the one or more estimated scene states into one or more reconstructed representations.

[0274] Clause 33. The one or more processors according to Clause 24 or 25, wherein the operation of at least part of the probabilistic state simulation stack includes at least one of: a perception task, a future scene state estimation task, or the generation of one or more control actions of the Ego machine.

[0275] Clause 34. One or more processors according to Clause 24 or 25, wherein the one or more processors are included in at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or displaying one or more augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device;a system implemented using a robot; a system for performing conversational AI operations; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system implementing one or more vision language models (VLMs); a system implementing one or more multimodal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system for performing one or more generative AI operations; a system involving one or more virtual machines (VMs); a system using or employing one or more inference microservices; a system involving one or more machine learning models deployed in a service or microservice along with an OS-level virtualization package (e.g., a container);a system that is at least partially implemented in a data center; or a system that is at least partially implemented using cloud computing resources.

[0276] Clause 35. A method comprising controlling one or more operations of an Ego machine based on at least one or more predicted control actions generated using a control stack comprising at least a portion of a probabilistic state simulation stack comprising one or more neural networks (NNs).

[0277] Clause 36. The procedure according to Clause 35, which further includes updating the probabilistic state simulation stack based at least on decoding one or more estimated scene states using latent diffusion.

[0278] Clause 37. The procedure according to Clause 35, which further includes updating the probabilistic state simulation stack based at least on the co-training of one or more navigation guidelines of the probabilistic state simulation stack with one or more neural scene state estimation networks of the probabilistic state simulation stack.

[0279] Clause 38. The method according to Clause 35, further comprising operating at least part of the probabilistic state simulation stack as the control stack based at least on extracting one or more visual features using a first transformer NN of one or more NNs and extracting one or more scene tokens based at least on a second transformer NN of one or more NNs that processes a representation of one or more visual features.

[0280] Clause 39. The procedure according to Clause 35, which further comprises operating at least part of the probabilistic state simulation stack as the control stack based at least on applying a coded representation of one or more corresponding perspectives of one or more sensors of the Ego machine to one or more transformer NNs of the one or more NNs.

[0281] Clause 40. The procedure according to Clause 35, wherein the part of the probabilistic state simulation stack includes one or more perception networks, one or more state estimation networks, and a navigation policy.

[0282] Clause 41. Method according to Clause 35, wherein the method is performed by at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or displaying one or more augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot;a system for performing conversational AI operations; a system that implements one or more language models; a system that implements one or more large language models (LLMs); a system that implements one or more vision language models (VLMs); a system that implements one or more multimodal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system for performing one or more generative AI operations; a system that involves one or more virtual machines (VMs); a system that uses or employs one or more inference microservices; a system that involves one or more machine learning models deployed in a service or microservice along with an OS-level virtualization package (e.g., a container); a system that is at least partially deployed in a data center;or a system that is implemented at least partially using cloud computing resources.

[0283] Clause 42. A system comprising one or more processors for control, within a simulation rendered using one or more light transport simulation algorithms, of a simulated ego machine based on at least one or more control actions generated using a control stack, wherein the control stack comprises at least part of a probabilistic state simulation stack incorporating one or more neural networks.

[0284] Clause 43. System according to Clause 42, wherein the simulation is generated at least using a three-dimensional (3D) content collaboration platform for 3D assets.

[0285] Clause 44. System according to Clause 43, wherein the 3D content collaboration platform for 3D assets uses OpenUSD.

[0286] Clause 45. One or more processors comprising processing circuitry to obtain one or more first segments of one or more navigation episodes representing one or more first trajectories of one or more simulated ego machines.

[0287] Clause 46. The one or more processors according to Clause 45, wherein the processing circuit is further provided for obtaining, using a navigation policy of a probabilistic state simulation stack, one or more second segments of the one or more navigation episodes representing one or more recovery trajectories of the one or more simulated ego machines.

[0288] Clause 47. The one or more processors of Clause 45 or 46, wherein the processing circuitry further serves to update one or more neural motion planners based on at least one representation of the one or more recovery trajectories.

[0289] Clause 48. The one or more processors of Clause 45, 46 or 47, wherein the processing circuit is further provided for controlling one or more operations of an ego machine based on at least the one or more neural motion planners.

[0290] Clause 49. The one or more processors according to Clause 45, 46, 47 or 48, wherein the processing circuit is further provided for initializing at least a part of the probabilistic state simulation stack based on at least one or more initial segments of the one or more navigation episodes representing one or more real trajectories of one or more physical ego machines.

[0291] Clause 50. The one or more processors of Clause 45, 46, 47 or 48, wherein the processing circuit is further provided for generating the one or more first trajectories based at least on using the one or more neural motion planners to control the one or more simulated ego machines in one or more simulation environments.

[0292] Clause 51. The one or more processors of Clause 45, 46, 47 or 48, wherein the processing circuit is further provided for generating the one or more first trajectories based at least on generating one or more control actions using the one or more neural motion planners and simulating the one or more control actions in a latent space of the probabilistic state simulation stack.

[0293] Clause 52. The one or more processors of Clause 45, 46, 47 or 48, wherein the processing circuit is further provided for generating the one or more first trajectories based at least on executing the one or more neural motion planners with one or more decoders of the probabilistic state simulation stack, which produce a reconstructed representation of one or more estimated future scenes in a format that corresponds to one or more inputs of the one or more neural motion planners.

[0294] Clause 53. The one or more processors according to Clause 45, 46, 47 or 48, wherein the processing circuit is further provided for generating the one or more first trajectories based at least on adapting at least one of: one or more control instructions or one or more trajectories generated using the navigation policy of the probabilistic state simulation stack.

[0295] Clause 54. The one or more processors according to Clause 45, 46, 47 or 48, wherein the processing circuit is further provided for obtaining the one or more recovery trajectories based at least on applying a representation of one or more control actions of the Ego machine according to the one or more first trajectories to a neural scene state estimation network of the probabilistic state simulation stack.

[0296] Clause 55. The one or more processors of Clause 45, 46, 47 or 48, wherein the processing circuit is further provided for generating the one or more recovery trajectories based at least on generating a representation of one or more estimated scene states corresponding to the one or more first segments using a generative neural network of the probabilistic state simulation stack and applying the representation of the one or more estimated scene states to the navigation policy.

[0297] Clause 56. The one or more processors of Clause 45, 46, 47 or 48, wherein the processing circuit is further provided for generating the one or more recovery trajectories based at least on generating a representation of one or more estimated scene states corresponding to the one or more first segments using a generative neural network of the probabilistic state simulation stack and applying the representation of the one or more estimated scene states to a history aggregation network of the probabilistic state simulation stack.

[0298] Clause 57. One or more processors according to clauses 45, 46, 47 or 48, wherein the one or more processors are included in at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content generation for 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or displaying one or more augmented reality content, virtual reality content or mixed reality content; a system implemented using an edge device;a system implemented using a robot; a system for performing conversational AI operations; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system implementing one or more vision language models (VLMs); a system implementing one or more multimodal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system for performing one or more generative AI operations; a system involving one or more virtual machines (VMs); a system using or employing one or more inference microservices; a system involving one or more machine learning models deployed in a service or microservice along with an OS-level virtualization package (e.g., a container);a system that is at least partially implemented in a data center; or a system that is at least partially implemented using cloud computing resources.

[0299] Clause 58. A method comprising generating, using a navigation policy of a probabilistic state simulation stack, one or more recovery trajectories of one or more simulated ego machines.

[0300] Clause 59. The procedure according to Clause 58, which further comprises training one or more neural movement planners based on at least one or more recovery trajectories.

[0301] Clause 60. The method according to Clause 58 or 59, which further comprises controlling one or more operations of an ego machine based on at least one or more control actions generated using the one or more neural motion planners.

[0302] Clause 61. The procedure according to Clause 58, 59 or 60, which further comprises initializing at least a part of the probabilistic state simulation stack based on at least one or more real trajectories of one or more physical ego-machines.

[0303] Clause 62. The method according to Clause 58, 59 or 60, which further comprises generating one or more initial trajectories of the one or more simulated ego machines based at least on the use of the one or more neural motion planners to control the one or more simulated ego machines in one or more simulation environments.

[0304] Clause 63. The method according to Clause 58, 59 or 60, which further comprises generating one or more initial trajectories of the one or more simulated ego machines based at least on generating one or more initial control actions using the one or more neural motion planners and simulating the one or more initial control actions in a latent space of the probabilistic state simulation stack.

[0305] Clause 64. The method according to Clause 58, 59 or 60, which further comprises generating one or more initial trajectories of the one or more simulated ego machines based at least on executing the one or more neural motion planners with one or more decoders of the probabilistic state simulation stack, producing a reconstructed representation of one or more estimated future scenes in a format that corresponds to one or more inputs of the one or more neural motion planners.

[0306] Clause 65. The procedure according to Clause 58, 59 or 60, which further comprises generating one or more initial trajectories of the one or more simulated ego machines based at least on adjusting at least one of one or more control commands or one or more trajectories generated using the navigation policy of the probabilistic state simulation stack.

[0307] Clause 66. A method according to Clause 58, 59 or 60, wherein the method is performed by at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or displaying one or more augmented reality content, virtual reality content or mixed reality content; a system implemented using an edge device; a system implemented using a robot;a system for performing conversational AI operations; a system that implements one or more language models; a system that implements one or more large language models (LLMs); a system that implements one or more vision language models (VLMs); a system that implements one or more multimodal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system for performing one or more generative AI operations; a system that involves one or more virtual machines (VMs); a system that uses or employs one or more inference microservices; a system that involves one or more machine learning models deployed in a service or microservice along with an OS-level virtualization package (e.g., a container); a system that is at least partially deployed in a data center;or a system that is implemented at least partially using cloud computing resources.

[0308] Clause 67. A system comprising one or more processors for controlling, within a simulation rendered using one or more light transport simulation algorithms, a simulated ego machine based on at least one or more control actions generated using one or more neural motion planners, wherein the one or more neural motion planners are trained based on at least one or more recovery trajectories generated using a navigation policy of a probabilistic state simulation stack.

[0309] Clause 68. System according to Clause 67, wherein the simulation is generated at least using a three-dimensional (3D) content collaboration platform for 3D assets.

[0310] Clause 69. System according to Clause 68, wherein the 3D content collaboration platform for 3D assets uses OpenUSD. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 19 / 041,044

[0001] US 63 / 675,196

[0001] US 10,885,698

[0124] Cited non-patent literature

[0000] the Society of Automotive Engineers (SAE) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806, published on June 15, 2018, Standard No. J3016-201609, published on September 30, 2016

[0081]

Claims

[1] One or more processors that have processing circuitry for: Updating one or more neural networks (NNs) of a probabilistic state simulation stack based on at least the probabilistic sampling of one or more estimated scene states; and Controlling one or more operations of an ego-machine based at least on operating at least part of the probabilistic state simulation stack as a control stack of the ego-machine. [2] The one or more processors according to claim 1, wherein the one or more NNs comprise one or more neural transformer networks, and the probabilistic state simulation stack comprises a navigation policy implemented using the one or more neural transformer networks. [3] The one or more processors according to claim 1, wherein the processing circuit is further provided for predicting one or more ego trajectories of the ego machine using a navigation guideline of at least part of the probabilistic state simulation stack. [4] The one or more processors according to claim 1, wherein the processing circuit is further provided for updating the probabilistic state simulation stack based at least on decoding the one or more estimated scene states using latent diffusion. [5] The one or more processors according to claim 1, wherein the processing circuit is further provided for updating the probabilistic state simulation stack based at least on updating one or more navigation guidelines of the probabilistic state simulation stack at least partially simultaneously with one or more neural scene state estimation networks of the probabilistic state simulation stack. [6] Extracting one or more visual features using a first neural transformer network (NN) of one or more NNs and extracting one or more scene tokens based on at least a second transformer NN of one or more NNs that processes a representation of the one or more visual features. [7] The one or more processors according to claim 1, wherein operating at least the part of the probabilistic state simulation stack as the control stack comprises applying a coded representation of one or more corresponding perspectives of one or more sensors of the Ego machine to one or more transformer NNs of the one or more NNs. [8] The one or more processors according to claim 1, wherein the processing circuit is further provided for updating the probabilistic state simulation stack without decoding the one or more estimated scene states into one or more reconstructed representations. [9] The one or more processors according to claim 1, wherein the operation of at least part of the probabilistic state simulation stack comprises at least one of: a perception task, a future scene state estimation task or the generation of one or more control actions of the ego machine. [10] The one or more processors according to claim 1, wherein the one or more processors are contained in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for conducting collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or displaying one or more augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system that is implemented using a robot; a system for performing conversational AI operations; a system that implements one or more language models; a system that implements one or more Large Language Models (LLMs); a system that implements one or more Vision Language Models (VLMs); a system that implements one or more multimodal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system for performing one or more generative AI operations; a system that includes one or more virtual machines (VMs); a system that uses or employs one or more inference microservices; a system that incorporates one or more machine learning models deployed in a service or microservice along with an OS-level virtualization package (e.g., a container); a system that is at least partially implemented in a data center; or a system that is implemented at least partially using cloud computing resources. [11] A process comprising: Controlling one or more operations of an ego machine based on at least one or more predicted control actions generated using a control stack comprising at least part of a probabilistic state simulation stack comprising one or more neural networks (NNs). [12] The method according to claim 11, further comprising updating the probabilistic state simulation stack based at least on decoding one or more estimated scene states using latent diffusion. [13] The method according to claim 11, further comprising updating the probabilistic state simulation stack based at least on co-training one or more navigation guidelines of the probabilistic state simulation stack with one or more neural scene state estimation networks of the probabilistic state simulation stack. [14] The method according to claim 11, further comprising operating at least part of the probabilistic state simulation stack as the control stack based at least on extracting one or more visual features using a first transformer NN of one or more NNs and extracting one or more scene tokens based at least on a second transformer NN of one or more NNs that processes a representation of one or more visual features. [15] The method according to claim 11, further comprising operating at least the part of the probabilistic state simulation stack as the control stack based at least on applying a coded representation of one or more corresponding perspectives of one or more sensors of the ego machine to one or more transformer NNs of the one or more NNs. [16] The method according to claim 11, wherein the part of the probabilistic state simulation stack comprises one or more perception networks, one or more state estimation networks and a navigation guideline. [17] The method of claim 11, wherein the method is carried out by at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for conducting collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or displaying one or more augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system that is implemented using a robot; a system for performing conversational AI operations; a system that implements one or more language models; a system that implements one or more Large Language Models (LLMs); a system that implements one or more Vision Language Models (VLMs); a system that implements one or more multimodal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system for performing one or more generative AI operations; a system that includes one or more virtual machines (VMs); a system that uses or employs one or more inference microservices; a system that incorporates one or more machine learning models deployed in a service or microservice along with an OS-level virtualization package (e.g., a container); a system that is at least partially implemented in a data center; or a system that is implemented at least partially using cloud computing resources. [18] A system, comprising: one or more processors for controlling, within a simulation rendered using one or more light transport simulation algorithms, a simulated ego machine based on at least one or more control actions generated using a control stack, wherein the control stack comprises at least part of a probabilistic state simulation stack incorporating one or more neural networks. [19] The system according to claim 18, wherein the simulation is generated at least using a three-dimensional (3D) content collaboration platform for 3D assets. [20] The system according to claim 19, wherein the 3D content collaboration platform for 3D assets uses OpenUSD.

Images