Visible pixel optimization for surround view and other environmental visualizations

Abstract

The present disclosure relates to visible pixel optimization for surround view and other environmental visualizations. In various examples, ego machines can be equipped with any number and type of sensors that can be used to generate overlapping image frames, and image data can be sampled from respective sensors using back-projection and generate back-projection views through a designated or applicable viewport. Area magnification factors or pixel densities can be measured or computed, and different sampling strategies can be used to sample image data for different areas of the back-projection view according to the area magnification factors or pixel densities to control local sampling strategies in local areas of the back-projection view. Image quality processing can be performed on pixels that are visible in the back-projection view (e.g., rather than on raw image data). Processed visualizations can be displayed on displays visible to occupants or operators of the ego machine.

Description

Background Technology

[0001] Vehicle environmental visualization systems provide occupants with a view of the area surrounding the vehicle. For the driver, this allows them to see their surroundings, including blind spots where their view is obstructed by vehicle components or other objects in the environment, without having to change position (e.g., turn their head, leave the driver's seat, or tilt in a particular direction). This visualization can assist and facilitate various driving maneuvers, such as smoothly entering or exiting parking spaces and avoiding collisions with objects such as curbs, pedestrians, or other vehicles. An increasing number of vehicles, especially luxury brands and newer models, are equipped with environmental visualization systems.

[0002] Existing vehicle environment visualization systems typically use fisheye cameras (usually mounted in the front, left, rear, and right directions of the vehicle) to perceive the surrounding area from multiple directions. In special cases, such as for long trucks or vehicles with trailers, additional cameras may be added. Image data frames from the individual cameras can be stitched together using camera parameters to align the frames, and overlapping areas can be combined using blending techniques to provide a 360° horizontal surround view visualization, which is then displayed on a monitor visible to the operator or occupant. In some implementations, the operator or occupant can interact with the visualization through a user interface, such as navigating virtual cameras to different viewports, zooming in or out of the image to focus on a specific area of ​​interest.

[0003] However, existing environmental visualization systems and other technologies used to generate visualizations of the surrounding environment have several drawbacks. For example, raw image data generated using surround-view cameras often contains sensor noise and exhibits limited dynamic range. Image signal processors (ISPs) can be used to perform tasks such as noise reduction, automatic exposure, white balance, and initial color correction, followed by post-ISP image quality processing algorithms such as temporal noise reduction (TNR), color correction, global tone mapping, and color coordination across images from different cameras. However, existing image processing techniques are often computationally inefficient and result in a range of visual artifacts that degrade image quality and user experience. For example, TNR may not completely eliminate noise and may introduce ghosting or texture distortion, such as blurring or the disappearance of moving objects. Color correction can lead to inconsistent color reproduction in stitched images, resulting in unnatural hue or tone mismatches in regions of interest. Global tone mapping can cause incorrect exposure adjustments in regions of interest, producing areas that are too dark or too bright, with colors appearing differently than what our eyes perceive. Color coordination can introduce geometric distortions, such as size or shape misalignments between different camera viewpoints within the region of interest. These artifacts reduce the accuracy and realism of environmental visualization systems, often distract the driver, and in some scenarios, interfere with the driver's perception of the surrounding environment and the safe operation of the vehicle.

[0004] Therefore, visualization techniques need to be improved to reduce visual artifacts, better present useful visual information, and / or otherwise improve the visual quality of the final image. Summary of the Invention

[0005] Embodiments of this disclosure relate to the optimization of visible pixels in surround view and other environmental visualizations. Unlike conventional systems, the systems and methods disclosed in this invention use back projection to render visualizations through a selected or applicable two-dimensional (2D) viewport and apply image quality processing to the pixels visible in the viewport.

[0006] For example, the self-contained machine can be equipped with any number and type of sensors (e.g., one or more cameras, such as fisheye cameras), and these sensors can be used to generate overlapping sensor data frames (e.g., overlapping image data) that represent substantially the same time slices. Image data can be sampled from the respective sensors using backprojection, and a backprojected view can be generated through a specified or applicable viewport. A region magnification factor or pixel density (e.g., quantizing the density of input pixel values ​​or color measurements to the density of output pixels) can be measured by comparing corresponding regions of the point image or image patch before and after projection through the viewport. In some embodiments, image data of different regions of the backprojected view can be sampled using different sampling strategies (e.g., upsampling, downsampling, super-resolution, anti-aliasing, region subsampling, warp subsampling pixels, etc.) based on the region magnification factor or pixel density, thereby controlling the local sampling strategy for local regions of the backprojected view. Furthermore, or alternatively, image quality processing (e.g., temporal noise reduction, color adjustment, tone mapping) can be performed on the pixels visible in the backprojected view (e.g., instead of performing this image processing on the original image data). The generated visualizations can be displayed on a monitor visible to the occupants or operators of the vehicle (e.g., driver or passenger, remote operator). The techniques described herein can be used to improve the visual quality of surround-view visualizations and other environmental visualizations, and reduce the latency in generating these visualizations, thereby promoting safer vehicle operation. Attached Figure Description

[0007] The system and method for optimizing visible pixels in surround view and other environmental visualizations will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram illustrating an example data flow of an example surround-view system according to some embodiments of the present disclosure; Figure 2 This is an illustration of example dot matrix images according to some embodiments of the present disclosure; Figure 3 The figures illustrate some embodiments according to the present disclosure. Figure 2 How can a raster image in the sample projection surface be projected onto the sample projection surface? Figure 4 and Figure 5 The illustration shows how applying temporal noise reduction in a viewport according to some embodiments of the present disclosure can be used to affect image quality; Figure 6 This is a flowchart of a method for performing one or more image quality operations in a reverse-projected view according to some embodiments of the present disclosure; Figure 7AExamples of sensor locations having a corresponding field of view or sensing field, such as an autonomous or semi-autonomous machine, according to at least some embodiments of this disclosure; Figure 7B This is an illustration of examples of the locations of components and sensors on an autonomous or semi-autonomous vehicle according to at least some embodiments of this disclosure; Figure 7C This is a block diagram of an example system architecture for autonomous or semi-autonomous vehicles, robots, and / or other machine types according to at least some embodiments of this disclosure; Figure 7D This is a block diagram of an example architecture of a computing system (e.g., a system-on-a-chip (SoC)) according to at least some embodiments of the present disclosure; Figure 7E This is a system diagram of communication between a cloud-based server and example autonomous or semi-autonomous vehicles, robots and / or other machine types according to at least some embodiments of this disclosure; Figure 8 This is a system diagram illustrating three computer ecosystems according to at least some embodiments of the present disclosure, including a computing system for generating or creating artificial intelligence (AI) (e.g., AI training and validation data), a computing system for training artificial intelligence, and a computing system for deploying AI at the edge; Figure 9 This is a block diagram of an example computing system for generative artificial intelligence (AI) according to at least some embodiments of the present disclosure; and Figure 10 This is a block diagram of an example computing device according to at least some embodiments of the present disclosure. Detailed Implementation

[0008] This invention discloses systems and methods related to the optimization of visible pixels for surround view and other environmental visualization. For example, this invention discloses systems and methods for rendering visualization images using back projection through a selected or applicable two-dimensional (2D) viewport and applying image quality processing to the pixels visible in the viewport. The techniques of this invention can be used to visualize the environment around self-contained machines (e.g., vehicles, robots, and / or other types of objects), such as in systems like parking visualization systems, surround view systems, and / or other systems.

[0009] Although this disclosure may relate to exemplary autonomous or semi-autonomous vehicles, robots, and / or other machine types 700 (which may alternatively be referred to herein as "vehicle 700," "self-vehicle 700," "machine 700," "self-machine 700," "robot 700," and / or "self-robot 700"), examples of which pertain to... Figures 7A-7E(This is a description, but it is not intended to be restrictive.) For example, the systems and methods described herein can be used, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms (e.g., autonomous mobile robots (AMRs), humanoid robots, robotic arms and / or end effectors, warehouse vehicles, off-road vehicles, vehicles coupled with one or more trailers, aircraft, water vehicles, shuttles (e.g., robotaxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones and / or other types of vehicles, robots or machines. Furthermore, although this disclosure may describe surround-view visualization of vehicles, this is not intended to be limiting, and the systems and methods described herein can be used in augmented reality (AR), virtual reality (VR), mixed reality (MR), robotics, security and supervision (e.g., smart cities), autonomous or semi-autonomous machine applications, industrial manufacturing, simulation and / or any other technological space where environmental visualization can be used. In some embodiments, the systems, methods and / or processes described herein can be used with… Figures 7A-7E The example machine 700 shown Figure 8 Example computing ecosystem 800, Figure 9 Example generative language model system 900 and / or Figure 10 The example computing device 1000 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the operation.

[0010] Environment virtualization techniques typically use three-dimensional (3D) shapes (e.g., a 3D bowl) to model the environment surrounding a self-operated machine. For example, the self-operated machine may be equipped with cameras pointing in different directions (e.g., mounted in front, left, back, and right), and the 3D model of the surrounding environment (e.g., the bowl) can be used as an intermediate projection surface to map image data to a 2D view of the environment as observed from the perspective of the virtual camera. For instance, back projection can be used to trace the rays of each pixel in a selected or applicable 2D viewport (defined by a virtual camera located in and oriented within the 3D scene using a 3D model of the surrounding environment) to a corresponding intersection on the surface of the 3D model, map that intersection to the corresponding pixel in the camera image (or some intermediate representation, such as a 2D top-down stitched image), and retrieve the corresponding pixel data. Stitching and blending can be performed before or after projection, depending on whether an intermediate stitched representation is used. By mapping pixels in the 2D viewport back to the 3D model and retrieving the corresponding image data, back projection can be used to avoid projecting pixels in the original image or intermediate representation that are ultimately invisible in the applicable 2D viewport.

[0011] Unlike previous techniques that performed certain types of image processing on complete frames of image data generated using a camera (e.g., after ISP processing), some embodiments perform corresponding image processing (e.g., image quality enhancement, warping, blending, etc.) on pixels visible in the back-projected view. This invention discloses a system and method for rendering visualizations using back projection through a selected or applicable two-dimensional (2D) viewport and applying image quality processing to pixels visible in the viewport. Furthermore, performing certain image processing in the back-projected view can improve image quality compared to techniques that perform image processing on complete frames of image data, partially or entirely of the original image. For example, conventional image processing techniques typically rely on global parameters (e.g., brightness, contrast, color histogram, noise level) derived from statistical analysis of the entire input image, but these global parameters are often affected by image portions not visible in the 2D viewport. Taking color adjustment as an example, images generated using a side-mounted camera may contain areas of light and dark. Conventional color adjustment might optimize the entire scene in the image, including areas of light and dark. However, if the 2D viewport only displays dark areas, global parameters calculated based on the entire input image rarely produce optimal color adjustments for a subset of the image displayed in the 2D viewport (e.g., darker areas). Better image quality can be achieved by performing image quality operations (e.g., temporal denoising, super-resolution, anti-aliasing, color adjustment, tone mapping, etc.) on pixels visible in the back-projected view, while ignoring these image quality operations on pixels outside the viewport.

[0012] In some embodiments, the backprojection process can be used to control or influence one or more image quality operations. For example, the backprojection process can be used to identify overlapping image data regions, and different temporal denoising methods can be applied to these regions (which contain twice the amount of information as non-overlapping regions). For example, one possible design is to reduce the temporal denoising intensity in overlapping regions. Since overlapping regions are prone to stitching artifacts (e.g., ghosting), reducing the temporal denoising intensity in overlapping regions should reduce ghosting artifacts. Another possible design is to use stronger filtering to improve sensitivity to motion. Overlapping regions may contain image data from cameras with significantly different viewpoints, so increasing the sensitivity to motion in temporal denoising may be more beneficial, thus suppressing more motion. (The availability of twice the amount of reference information in this region should offset the increased sensitivity). Another possible design is to maintain the same temporal denoising intensity and apply twice the denoising in overlapping regions.

[0013] In some embodiments, multi-camera image processing (e.g., color coordination) can be disabled or simplified based on which cameras have visible pixels in the back-projected view. Conventional techniques perform color coordination on all pan-view images, while the back-projection process can be used to determine which images or regions within specific images contribute to the rendered view, and if only one image or one region within an image contributes, color coordination can be disabled. If more than one, but not all, of the images contribute, color coordination can be performed on the contributing images (e.g., their visible portions). This allows color coordination to be completely avoided or significantly simplified. The above are merely a few examples, and various variations can be implemented within the scope of this disclosure.

[0014] Typically, the projection process may result in different levels of magnification for different parts of the back-projected view, depending on the shape of the 3D (e.g., bowl) model, how the image data is projected onto the 3D model, and the position and orientation of the virtual camera. Therefore, the magnification factor or density (pixel density) from input pixels to output pixels can be quantized in individual regions (e.g., pixels, pixel tiles) of the back-projected view, and the resulting region magnification factor or pixel density can be used to control or influence image processing in the corresponding regions of the back-projected view. In some embodiments, the region magnification factor or pixel density can be calculated by projecting a raster image through a projection surface onto a 2D viewport and measuring the magnification factor or pixel density of each region (e.g., tile) in the back-projected view; or by back-projecting from the 2D viewport to the corresponding pixels of the input image and counting how many input pixels map to how many output pixels. Therefore, a magnification factor or pixel density map quantifying the magnification factor or pixel density of each region in the back-projected view can be generated, and this magnification factor or pixel density map can be used to control or influence image processing in various ways (e.g., image quality processing, warping, blending, etc.).

[0015] For example, in some embodiments, the region magnification or pixel density of a local area of ​​the backprojected view can be used to control the sampling strategy. Typically, the backprojection process can be viewed as a form of sampling because it involves mapping discrete points on a two-dimensional image plane back to three-dimensional space and sampling the corresponding pixel values. If the magnification factor of a particular region in the backprojected view is less than 1, the region will be magnified or reduced, and therefore downsampling or anti-aliasing can be used in that region (e.g., multiple input pixels can contribute to a single output pixel). If the magnification factor of a particular region in the backprojected view is greater than 1, the region will be magnified, and therefore upsampling or interpolation can be used in that region (e.g., a single input pixel can be stretched onto multiple output pixels). For regions with relatively large magnification factors, super-resolution algorithms (e.g., deep learning supersampling (DLSS)) can be applied.

[0016] In some embodiments, for regions with a magnification factor less than 1 and which have already undergone upsampling, region subsampling can be applied to improve system performance. For example, the projection indexes during backprojection can be subsampled (e.g., by projecting every other pixel coordinate and storing the indexes in shared memory, then interpolating between them) to reduce computational load. More specifically, instead of computing the mapping for every pixel in the backprojected view, a subset of pixels (e.g., every other pixel) is projected onto a 3D model (e.g., a bowl-shaped model), their indices (e.g., 3D coordinates) are stored, and index interpolation is applied to estimate the mapping for the remaining pixels based on the projection indices of nearby subsampled pixels, rather than performing a full backprojection for every pixel in that region. This approach can significantly optimize performance by reducing redundant computations while maintaining a smooth and accurate reconstruction of the backprojected view. Furthermore, since warps (e.g., fisheye distortion correction) can be used to map pixels from one coordinate system to another and sample values ​​from the original image, warps can be considered a form of sampling. In some embodiments, for regions where the magnification factor is less than 1 and upsampling is required, deformation can be performed on subsampled pixels (e.g., by deforming every other pixel) and interpolation can be performed between them. These are just a few examples, and various variations can be implemented within the scope of this disclosure.

[0017] In some embodiments, temporal denoising can be controlled by a region magnification factor or pixel density in a local area of ​​the back-projected view. In some embodiments, stronger temporal denoising can be applied to regions with higher magnification or lower pixel density because amplified noise may occur in these areas, and vice versa. In another possible design, since regions with lower magnification (e.g., below 1) should have a correspondingly higher available input pixel density, stronger temporal denoising can be applied to non-textured regions to reduce resolution and achieve better denoising. For object or textured regions, weaker temporal denoising can be applied because reduction also degrades the noise level, and maintaining a stronger level of temporal denoising may introduce ghosting artifacts. In some embodiments, spatial filtering and temporal filtering can be mixed based on magnification or pixel density to maintain sharpness or reduce resolution. Since the size of the spatial or temporal filters in temporal denoising can be considered a form of sampling rate control because they determine the effective resolution (spatial and temporal resolution) for noise removal and detail preservation, the region magnification factor or pixel density can be considered a factor driving the sampling rate.

[0018] In some embodiments, color adjustment and / or tone mapping can be controlled using a region magnification factor or pixel density in a local area of ​​the back-projected view. Since region upsampling or downsampling may produce updated color values, color adjustment and / or tone mapping can be performed in each local area, and the results blended. For example, if one of the design goals is to maintain high dynamic range, and the projection process results in magnification, the projection process may alter the image statistics of the visible area. Therefore, performing color adjustment and / or tone mapping in local areas based on a region magnification factor or pixel density, and blending or interpolating between adjacent tiles, should improve image quality.

[0019] Therefore, the techniques described herein can be used to improve the visual quality of surround-view visualizations and other environmental visualizations, reduce the latency in generating these visualizations, and thus promote safe vehicle operation.

[0020] In some embodiments, the systems and methods described herein can be performed in a simulated environment (e.g., simulated environment data and simulated sensor data of simulated sensors of virtual or simulated vehicles, robots, or machines in the simulated environment) using simulated data (e.g., NVIDIA's DriveSIM, ISAAC Sim, ISAAC Gym, ISAAC Lab, etc.). For example, a visualization of the simulated environment can be generated using simulated input data (e.g., map data, perception data, self-motion data, tactile data, and / or any other data described herein), and this information can be used to perform virtual machine-related operations in the simulated environment. These simulated operations can be used to test the performance of underlying algorithms, systems, and / or processes prior to actual deployment. In some cases, simulations can be used to generate synthetic training data, such as simulated environment visualizations generated from the simulated environment. This synthetic training data (in addition to real-world data, or as a substitute for real-world data) can then be used for training or testing.

[0021] In any example, such as when using a simulated environment for testing, validation, training, etc., the simulated environment and / or associated training data may be rendered or otherwise generated using one or more optical transport simulation algorithms (e.g., one or more ray tracing and / or path tracing algorithms). When using optical transport simulation, the simulation system may employ one or more dedicated ray tracing hardware accelerators and / or processors (e.g., NVIDIA's RTX or other real-time ray tracing GPUs, such as GPUs including one or more ray tracing (RT) cores) optimized to perform real-time or near-real-time optical transport simulation operations in conjunction with one or more other processors in the system (e.g., GPUs, CPUs, accelerators, etc.). In some embodiments, the simulated environment and / or one or more of its objects, features, or components may be generated or managed in a three-dimensional (3D) content collaboration platform (e.g., NVIDIA's OMNIVERSE) optimized for or suitable for industrial digitization, generative physical artificial intelligence, and / or other use cases, applications, and / or services. For example, a content collaboration platform or system may include a system for using or developing common scene descriptors (USD) (such as OpenUSD) data to manage objects, features, scenes, etc., in simulated environments, digital environments, etc. The platform may include realistic physical simulations (e.g., using NVIDIA's PhysX software development kit (SDK)) to simulate real physical phenomena and physical interactions with simulations hosted on the platform. The platform may integrate OpenUSD with ray tracing / path tracing / light transport simulations (such as NVIDIA's RTX rendering technology) into software tools and simulation workflows for building, training, deploying, and / or testing AI systems, such as systems for testing, validating, training (e.g., machine learning models, neural networks, etc.) and / or other tasks related to automobiles, robots, other machine types, and / or other systems and applications. In some examples, the simulated environment may include a digital twin of a real environment, such as a specific road segment, warehouse, data center, airport, geographic area, ocean area, and / or any other real environment in which autonomous or semi-autonomous vehicles or machines can operate.

[0022] In some embodiments, remote operation or control of vehicles, robots, and / or other machines can be performed using remote control or a remote operating system. For example, the systems and methods described herein can be used to generate visualizations or maps of an environment to assist remote operators in controlling autonomous or semi-autonomous machines to navigate within that environment, or to provide waypoints or other control or navigation instructions. Therefore, remote operators can use visual, auditory, textual, and / or other cues or indicators generated using the systems and methods described herein to assist vehicles, robots, machines, etc., in navigating real-world environments using a remote operating system.

[0023] In some embodiments, the systems and methods described herein can be deployed in robotic applications. For example, a robot or robotic system may include one or more onboard processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), hardware-based programmable vision accelerator (PVA) (which may include one or more vector processing units (VPU)), direct memory access (DMA) system and / or pixel processing engine (PPE), hardware-based optical flow accelerator (OFA), SoC, etc.), and memory and / or storage devices (e.g., for storing control algorithms, sensor data, and a...). One or more machine learning models. The robotic system can use these processors to execute one or more machine learning models (e.g., language models, visual language models (VLM), large language models (LLM), visual-language-action (VLA) models, multimodal language models (MMLM), etc.), which allow the robotic system to autonomously or semi-autonomously perform complex tasks, such as interacting with and / or manipulating static and / or dynamic objects, or navigating the environment using sensors (such as cameras, LiDAR, RADAR, ultrasonic sensors, etc.). The system can use sensors... Sensor fusion technology combines data from multiple sensors (e.g., cameras, infrared, LiDAR, RADAR, accelerometers) to create a comprehensive model of the robot's surrounding environment. This data can be processed locally on the robot or sent to a remote server for computationally intensive tasks such as 3D mapping or SLAM (simultaneous localization and mapping). In one or more embodiments, data from individual robots (e.g., sensor data, task status, or environmental conditions) can be uploaded to the cloud, where a centralized AI model can analyze and optimize commands and distribute them across the entire fleet. In some embodiments, the data described herein... One or more machine learning models (e.g., language models, VLM, VLA, LLM, MMLM, diffusion models, NeRF models, DNN, etc.) can be used to allow the robot to perceive and reason about its environment and / or communicate with one or more other robots and / or people in the environment. In some embodiments, the robot may (e.g., using one or more network interface cards (NICs) and / or data processing units (DPUs)) communicate with one or more locally hosted servers / computing devices and / or with one or more remotely located servers / computing devices (e.g., in one or more data centers).

[0024] In some embodiments, the systems and methods described herein can be deployed in in-vehicle infotainment (IVI) systems or in-vehicle experience (IX) applications. For example, an infotainment system within a vehicle (e.g., a car, truck, drone, construction machinery, robot, semi-autonomous vehicle, or autonomous vehicle) may include one or more in-vehicle processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), hardware-based programmable vision accelerator (PVA) – which may include one or more vector processing units (VPU), direct memory access (DMA) systems and / or pixel processing engines (PPE), hardware-based optical flow accelerators (OFA), SoC, etc.) and memory and / or storage devices (e.g., for storing control algorithms, sensor data, one or more machine learning models, entertainment content, navigation data, user preferences, etc.). The system may utilize these processors to generate or present visualizations of the surrounding environment and may utilize these processors to execute one or more machine learning models (e.g., language models) to enable functions such as voice control, personalized media recommendations, dynamic navigation, and real-time communication with other services via network connectivity. In-vehicle infotainment systems can also utilize natural language processing (NLP) models to enable voice-based interaction. One or more machine learning models can be stored locally or accessed via one or more APIs connected to cloud services, enabling the system to process requests in real-time or near real-time.

[0025] In some examples, one or more machine learning models described herein (e.g., deep neural networks, language models, LLMs, VLMs, multimodal language models, visual-language-action (VLA) models, perceptual models, tracking models, fusion models, transformer models, diffusion models, encoder-only models, decoder-only models, encoder-decoder models, neural rendering field (NERF) models, etc.) can be packaged as microservices, such as inference microservices (e.g., NVIDIA NIM), which may include containers (e.g., operating system (OS) level virtualization packages) that may include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model "engine". For example, an inference microservice may include the container itself and one or more models (e.g., weights and biases). In some instances (e.g., where one or more machine learning models are small enough (e.g., have a sufficiently small number of parameters)), one or more models may be included within the container itself. In other examples (such as when one or more models are large), one or more models may be hosted / stored in the cloud (e.g., in a data center) and / or hosted on-premises and / or at the edge (e.g., on a local server or computing device, but outside a container). In these embodiments, one or more models may be accessed via one or more APIs (such as REST APIs). Thus, and in some embodiments, one or more machine learning models described herein may be deployed as inference microservices to accelerate the deployment of one or more models on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, optimized inference engines (e.g., execution software built using standardized AI model deployments, such as NVIDIA's Triton Inference Server) and / or one or more APIs for high-performance deep learning inference, which may include inference runtime and model optimizations for delivering low-latency and high-throughput inference for production applications (such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). One or more machine learning models described herein can be included as part of a microservice along with an acceleration infrastructure capable of being deployed with a single command and / or orchestrated and automatically scaled (e.g., to data center scale on a single device) using a container orchestration system. Thus, an inference microservice can include machine learning models (e.g., optimized for high-performance inference), inference runtime software that executes one or more machine learning models and provides outputs / responses to inputs (e.g., user queries, prompts, etc.), and enterprise management software that provides health checks, identity, and / or other monitoring.In some embodiments, the inference microservice may include software that performs in-situ replacement and / or updates to one or more machine learning models. When a replacement or update is performed, the software performing the replacement / update may maintain user configurations for the inference runtime software and enterprise management software.

[0026] While examples of machine learning models such as neural networks are described in this paper, this is not intended to be limiting. For example, but not limited to, any of the various machine learning models and / or neural networks described herein can include any type of machine learning model, such as those using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoder neural networks, artificial neural networks (ANN), convolutional neural networks (CNN), recurrent neural networks (RNN), perceptrons, long short-term memory (LSTM) networks, multilayer perceptron (MLP) networks, etc. Deep Stacked Networks (DSN), Generative Pre-trained (GPT) models or networks, Feedforward networks, Radial Basis Function ANNs, Self-Organizing Maps (SOM), Kohonen Maps, Hopfield Networks, Boltzmann Machines, Deep Belief Neural Networks, Deconvolutional Neural Networks, Generative Adversarial Networks (GANs), Liquid Machines, Modular Neural Networks, Sequence-to-Sequence Models, Networks Using Transformer Architectures, State-Space Models (SSM) (e.g., Networks Using Mamba Architectures (e.g., Mamba-1, Mamba...) 2) Networks using selective state-space models, networks using structured state-space sequence models, etc., diffusion models (e.g., diffusion probability models, fraction-based generative models, etc.), Neural radiation field (NeRF) models, Gaussian sputtering models, Kolmogorov-Arnold networks (KAN), models with encoder-only architectures, models with decoder-only architectures, models with encoder-decoder architectures, generative machine learning models, language models, large language models (LLM), visual language models (VLM), multimodal language models (MMLM), large action models (LAM), visual-language-action (VLA) models, etc.) and one or more machine learning models and / or other types of machine learning models.

[0027] The systems and methods described herein may be used by, but are not limited to, the following: non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, spacecraft, ships, shuttles (e.g., robotaxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones, and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, as examples but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and supervision, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets (e.g., NVIDIA's Omniverse), cloud computing and / or any other suitable application.

[0028] The disclosed embodiments can be included in a variety of different systems, such as motor vehicle systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines, etc.), systems implemented using robots, aviation systems, medical systems, rowing systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing language models (such as large language models (LLM), visual language models (VLM), visual-language-action (VLA) models, and / or multimodal language models), systems using or deploying one or more inference microservices, systems containing one or more machine learning models deployed in services or microservices and OS-level virtualization packages (e.g., containers), systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0029] refer to Figure 1 , Figure 1Schematic diagrams of example environment visualization systems (e.g., surround view system 100) according to some embodiments of this disclosure are depicted. It should be understood that such and other arrangements described herein are illustrative only. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or as alternatives to the arrangements and elements shown, and some elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and location (e.g., at the edge, in the field (e.g., locally hosted servers, remotely located in one or more computing or server devices in one or more data centers such as the cloud), and / or on local devices, vehicles, or machines). The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., central processing units (one or more CPUs), graphics processing units (one or more GPUs), microprocessors, microcontrollers, embedded processors, digital signal processors (DSPs), image signal processors (ISPs), physical processing units (PPUs), field-programmable gate arrays (FPGAs), one or more accelerators (e.g., deep learning accelerators (DLAs), deep learning accelerator clusters (XNNs), neural network accelerators (NNAs) and / or neural processing units (NPUs), programmable vision accelerators (PVAs), optical flow accelerators (OFAs), etc.), application-specific integrated circuits (ASICs), data processing units (DPUs), quantum processors, etc.). In some embodiments, the systems, methods, and processes described herein can be used with... Figures 7A-7E Example machine 700, Figure 8 Example computing ecosystem 800, Figure 9 Example generative language model system 900 and / or Figure 10 The example computing device 1000 uses components, features, and / or functions similar to those of other computing devices to perform the task.

[0030] At a high level, the surround view system 100 can be used to generate a visualization 170 of a 3D environment (e.g., around a self-generating machine, such as a vehicle) based on image data frames representing the environment. The surround view system 100 can be integrated into the self-generating machine (e.g., Figures 7A-7DThe surround view system 100 (in a vehicle 700) may contain any number and type of sensors 101, such as one or more cameras, which generate image data as the autonomous machine navigates its three-dimensional environment. Typically, the surround view system 100 can use the image data (or other sensor data) to generate a surround view visualization representing at least a portion of the surrounding environment. For example, the surround view system 100 may include a projection component 120 that renders a back-projected view 130 of the image data through a selected or applicable viewport 110; and an image processing component 160 that applies image processing to the pixels visible in the back-projected view 130. The generated visualization 170 may be presented on a display 180 visible to the occupants or operator of the autonomous machine (e.g., a driver or passenger, a remote operator). Thus, the sensors 101 can be used to generate sensor data frames for each time slice (e.g., at a specific frame rate), and the sensor data frames for each time slice can be used to generate and / or present a visualization of the surrounding environment on the display 180 (regardless of whether the frame rate is the same as the frame rate at which the sensor data was generated, or some other frame rate). The surround view system 100 can operate at any suitable frame rate, thereby generating a video stream of visualizations. Any suitable user interface (e.g., provided by the display 180) can accept input for navigating the virtual camera defining the viewport 110 to different positions or orientations, selecting a pre-selected rendering viewport, zooming in or out to focus on a specific region of interest, etc. Thus, the surround view system 100 can update the viewport 110 accordingly and generate a corresponding visualization 170.

[0031] More specifically, self-machines (e.g., Figures 7A-7D The vehicle 700 may be equipped with any number and type of sensors 101 (e.g., one or more cameras, such as fisheye cameras), and the sensors 101 may be used to generate overlapping sensor data frames (e.g., overlapping image data) representing substantially the same time slices. Examples of sensors that may be used include one or more of the vehicle 700's stereo camera 768A, wide-angle camera 768B (e.g., fisheye camera), infrared camera 768C, surround-view camera 768D (e.g., 360° camera), and / or long-range and / or mid-range camera 768E. In one example configuration, four fisheye cameras may be mounted at the front, left, rear, and right sides of the vehicle, generating corresponding surround-view video at a specified frame rate. An image signal processor (ISP) may be used to perform tasks such as noise reduction, automatic exposure, white balance, and initial color correction.

[0032] Typically, projection component 120 can use back projection to sample image data from corresponding sensors 101 and generate a back-projected view 130 through viewport 110. For example, projection component 120 can trace pixel paths in the screen space of viewport 110 to corresponding intersections on the surface of a 3D model of the environment (e.g., a 3D bowl), map these intersections to corresponding pixels in the image data from the corresponding sensors 101 (e.g., using inverse UV mapping), and retrieve or identify image data (e.g., color values, such as RAW or RGB values; infrared data, such as intensity or temperature measurements; image or pixel metadata generated by an image signal processor (ISP) during image capture and / or processing, such as exposure levels, exposure corrections, white balance adjustments, initial color correction matrices, etc.) from the corresponding pixels. Therefore, projection component 120 can store image data, reference image data, or otherwise associate image data with corresponding pixels in the back-projected view 130 (e.g., a 2D image through viewport 110). Any known technique can be used to generate a 3D model of the environment (e.g., using a fixed 3D bowl, or using an adaptive 3D bowl, such as a bowl whose shape is adjusted according to distance and / or orientation to fit the detected object, etc.), and any known technique can be used to select or specify viewport 110 (e.g., specified by a virtual camera controlled by an operator or occupant, or selected from a set of preset viewports via a user interface (e.g., a touch interface or voice interface provided by display 180), or automatically selected according to the scene, etc.).

[0033] Depending on the specific implementation, the surround view system 100 may perform warping (e.g., fisheye distortion correction), align image data from different sensors 101 to a common coordinate system (e.g., relative to a self-defined coordinate system, such as the vehicle rig coordinate system, a global 3D coordinate system, or world space), place stitching, blend at the stitching points, and / or other types of image processing, and may be performed in any suitable order. These operations may be performed before the projection component 120 generates the reverse-projected view 130 (e.g., in this case, the projection component 120 may sample from warped and / or stitched image data (e.g., a 2D top-down stitched image)) or after the projection component 120 generates the reverse-projected view 130 (e.g., in this case, the projection component 120 or the image processing component may perform warping, alignment, stitching placement, and / or blending in the reverse-projected view 130), and may omit corresponding image processing for pixels outside the reverse-projected view 130.

[0034] In some embodiments, the projection component 120 uses different sampling strategies to sample image data from the sensor 101 for different regions of the back-projected view 130. More specifically, the projection component 120 can use a region magnification factor or pixel density (e.g., quantizing the density of input pixel values ​​or color measurements to the density of output pixels) to control the local sampling strategy for local regions of the back-projected view 130.

[0035] In some embodiments, the magnification calculation component 150 can use the bitmap image 140 to calculate a region magnification factor or pixel density. More specifically, each sensor 101 can be associated with a corresponding instance of the bitmap image 140, the projection component 120 can generate a corresponding viewport bitmap image 145 of the bitmap image 140 for each sensor 101 via the viewport 110 (e.g., using forward or reverse projection), and the magnification calculation component 150 can calculate the region magnification factor or pixel density by measuring the point magnification in different regions of the viewport bitmap image 145. The bitmap image 145 can use any suitable bitmap as a reference (e.g., points of equal size and / or equal spacing) and can represent points and blank areas in any suitable manner (e.g., using binary values, such as 1 for points and 0 for blank areas; using different colors to represent points of different sensors 101; using different colors to represent points in different fields of view of the corresponding sensor 101, etc.).

[0036] Figure 2 This is an illustration of an example bitmap image 210 according to some embodiments of the present disclosure. In this example, the bitmap image 210 has equally spaced, equally sized dots that have different colors (represented by different grayscale colors) in different regions or subdivisions (e.g., in this example, in different cells of a 3×3 grid). Figure 2 The lower half of the illustration shows a dot matrix image 210, which is positioned to represent corresponding images from four surround-view cameras (e.g., facing the front, left, rear, and right of the self-machine, respectively), which are arranged in the same coordinate system. Figure 3 The diagram illustrates how to... Figure 2 The raster image 210 in the image is projected from the positions of these arrangements onto an example projection surface, such as a 3D bowl 310. Figure 3 The image includes the bowl (for illustration purposes only and is not an actual render), as well as some example viewports 320 and 330, on which area magnification factors are overlaid for illustrative purposes. For example, points relatively close to the rear of the car in viewport 220 have been magnified by 0.5 to 0.6 times (or downsampled), while points relatively far from the car in viewport 230 have been magnified by 1.7 to 7.6 times (or upsampled).

[0037] Back Figure 1 In some embodiments, if the 3D model of the surrounding environment has a fixed shape (e.g., a fixed 3D bowl), the projection component 120 can pre-calculate a viewport bitmap image 145 for a preset viewport using the fixed 3D model. In some embodiments, the shape of the 3D model of the surrounding environment may vary, and the shape (e.g., the ground inside the bowl, the top outside the bowl) can be parameterized to a corresponding range of values ​​(e.g., the radius, circumference, and / or diameter of a circular bowl; the minor and major axes, focal point, eccentricity, and / or circumference of an ellipse; etc.), and the projection component 120 can pre-calculate a viewport bitmap image 145 for a preset viewport using a representative set of parameters for each range.

[0038] Therefore, the magnification calculation component 150 can divide the viewport 110 into multiple regions (e.g., pixels, pixel tiles in a tiled grid, such as a 16x16 grid), and for each region, the magnification calculation component 150 can calculate the region magnification or pixel density by comparing the points in the corresponding region of the viewport bitmap image 145 and the bitmap image 140. For example, the magnification calculation component 150 can use any known technique to detect the area of ​​a point in a given tile in the viewport bitmap image 145 and divide it by the known area of ​​the point in the bitmap image 140. In a simple example, if the bitmap image 140 uses single pixels and the tile in the viewport bitmap image 145 has a five-pixel point, the magnification calculation component 150 can divide the area of ​​the point by the magnification factor to calculate a magnification factor of five for the tile. In some embodiments, different colored dots are used to represent images from different cameras (or different regions of the field of view). The magnification calculation component 150 can use any known technique to identify the color of dots in a given tile of the viewport bitmap image 145 and identify which camera (or which part of its field of view) contributed to that tile. Therefore, the magnification calculation component 150 can repeat this process for each region (e.g., tile) of the viewport bitmap image 145 to generate a magnification map (or pixel density map) that quantifies the magnification factor (or its reciprocal, i.e., pixel density) of each region in the back-projected view 130.

[0039] In some embodiments (e.g., without using raster image 140 to calculate region magnification factor or pixel density), magnification calculation component 150 may coordinate with projection component 120 to back-project viewport 110 to a corresponding intersection with the surface of the 3D model and retrieve the corresponding region (e.g., patch) from the original image data (e.g., using reverse UV mapping). Therefore, magnification calculation component 150 can compare the size of the projected patch in viewport 110 with its corresponding area in the original image data to calculate the magnification factor of that region (e.g., how many viewport pixels map to how many camera pixels). Thus, magnification calculation component 150 may repeat this process for each region (e.g., tile) of viewport 110 to generate a magnified map (or pixel density map) that quantifies the magnification factor (or its reciprocal, i.e., pixel density) in each region of the back-projected view 130.

[0040] Therefore, in some embodiments, the projection component 120 can use a region magnification factor or pixel density in a magnification map or pixel density map to control the sampling of local regions of the back-projected view 130. For example, the projection component 120 can divide the viewport 110 into multiple regions (e.g., corresponding to regions in the magnification map or pixel density map), and the projection component 120 can use a sampling strategy that depends on the magnification factor or pixel density in each region to perform back-projection (or the projection component 120 can sample in smaller sub-tiles and interpolate between tiles). For example, if the magnification factor of a particular region in the magnification map is less than 1 (or the pixel density is greater than 1), the corresponding region of the image data will be reduced or minimized, so the projection component 120 may use downsampling (e.g., averaging) and / or anti-aliasing filtering in that region to prevent loss of detail or jagged artifacts. Conversely, if the magnification factor of a particular region in the magnification map is greater than 1 (or the pixel density is less than 1), that region will be magnified, so the projection component 120 may use upsampling (e.g., interpolation) in that region to estimate missing pixel values. In some embodiments, if the magnification factor in a specific region of the magnification map is higher than a specified threshold (e.g., some relatively large magnification factor, such as 2, 5, 10, etc.), the projection component 120 may apply any known super-resolution algorithm (e.g., deep learning supersampling (DLSS)) to that region (or the image processing component 160 may apply a super-resolution algorithm as a post-processing step).

[0041] In some embodiments, in regions where the magnification factor is less than 1, the projection component 120 may apply region subsampling. For example, the projection component 120 may subsample the projection index during back projection (e.g., by projecting the coordinates every other pixel, storing the index in shared memory, and interpolating therebetween) to reduce computational load. More specifically, instead of computing the mapping for every pixel in the back-projected view 130, the projection component 120 may project a subset of pixels (e.g., every other pixel) onto a 3D (e.g., bowl-shaped) model, store their indices (e.g., 3D coordinates), and apply index interpolation to estimate the mapping for the remaining pixels based on the projection indices of nearby subsampled pixels. Furthermore, or alternatively, in regions where the magnification factor is less than 1, the projection component 120 may apply deformation (e.g., fisheye distortion correction) to the subsampled pixels (e.g., by deforming every other pixel) and interpolate therebetween.

[0042] Therefore, projection component 120 can apply a region sampling strategy in each region of the back-projected view 130 using the corresponding magnification factor or pixel density calculated by magnification calculation component 150, and projection component 120 (or image processing component 160) can apply any known blending technique to the final region of back-projected view 130. In some embodiments, projection component 120 samples image data before stitching, and the projection process can be used to align image data from the respective sensors 101 and determine whether there are any overlapping image data regions (e.g., whether multiple images contribute to the same pixel). Therefore, projection component 120 or image processing component 160 can perform stitching placement and / or blending in back-projected view 130.

[0043] Typically, image processing component 160 can perform image quality processing on pixels visible in the back-projected view 130 (e.g., instead of performing image processing on the raw image data). Some example types of image quality processing that image processing component 160 can apply to pixels in the back-projected view 130 include temporal noise reduction, harmonic reconciliation, color adjustment, and tone mapping. Although the foregoing discussion considers embodiments where projection component 120 applies super-resolution and / or anti-aliasing as part of the sampling process, image processing component 160 can additionally or alternatively apply super-resolution and / or anti-aliasing as part of the image quality operations. Performing image quality operations on pixels visible in the back-projected view 130 while ignoring these image quality operations on pixels outside viewport 110 can result in better image quality.

[0044] Figure 4 and Figure 5 This describes how temporal noise reduction can be applied in a viewport to affect image quality, according to some embodiments of this disclosure. Figure 4 The illustration shows three different images of the road surface near the vehicle. Each image has a magnification factor less than 1, and its pixels are downsampled from the original fisheye camera image. In the left image 410, temporal denoising is disabled, resulting in significant noise. In the middle image 420, temporal denoising was applied to the original image before rendering. While the noise is reduced, some noise remains. In the right image 430, temporal denoising was applied to the rendered viewport, achieving the best result among the three options. Figure 5 Another example is provided, showing three images of the road surface at a distance from the vehicle. Each image has a magnification factor greater than 1 and contains pixels upsampled from the original fisheye camera image. In the left image 510, temporal denoising is not enabled, resulting in significant noise. In the middle image 520, temporal denoising is applied to the original image before rendering. In the right image 530, temporal denoising is applied to the rendered viewport, achieving the best result among the three options.

[0045] In some embodiments, projection component 120 may identify one or more overlapping image data regions, and image processing component 160 may apply temporal denoising within the overlapping image data regions using different parameters than those used outside these regions. For example, image processing component 160 may identify which regions of viewport 110 have overlapping image data, and image processing component 160 may reduce the intensity of temporal denoising in the overlapping regions to reduce ghosting artifacts. Temporal denoising typically includes spatial and temporal filters. Temporal filtering reduces noise by averaging pixel values ​​over multiple frames, while spatial filtering smooths noise within a single frame based on adjacent pixels. Therefore, image processing component 160 may reduce the intensity of temporal denoising in overlapping regions by using a smaller spatial filter window size (defining the region around a pixel within a single frame that contributes to denoising) and / or a smaller temporal filter window size (defining the number of past and / or future frames used to refine pixel values ​​over time). In some embodiments, image processing component 160 may enhance the intensity of temporal denoising in overlapping regions to suppress more motion.

[0046] In some embodiments, projection component 120 may determine which sensors 101 contribute pixels to the back-projected view 130 (e.g., which sensors 101's fields of view are visible through viewport 110) and provide this information to image processing component 160, which may disable or simplify color coordination based on which sensors 101 contributed pixels. For example, if the back-projected view 130 is from a single camera, color coordination is not required, and image processing component 160 may disable it. If more than one sensor 101, but not all sensors 101, have contributed, image processing component 160 may apply color coordination to the corresponding portions of the image data in the back-projected view 130.

[0047] By way of example, some available example viewports include: a top perspective view of a 3D bowl-shaped projection (similar to...) Figure 3 The viewport can be configured as follows: a 3D bowl (310); a top-down plan view of the vehicle from an aerial perspective, including sections with four surround-view cameras; a single-camera view providing a view from any one of the cameras; a front-to-back triptych or three-panel composite image view, where the central panel provides a perspective view from a single camera, and adjacent panels use partial correction from the side cameras to expand the field of view without fully merging them; and a segmented plan view showing a top-down view of the front and rear of the vehicle, respectively. Depending on the viewport (and camera configuration), the viewport can use image data from one, two, three, or four cameras. Disabling or simplifying color coordination to apply only to the visible frames should significantly improve the quality and speed of the resulting color coordination, especially when the color statistics of the visible portion of the image differ greatly from the whole image. Deriving coordination parameters (e.g., target coordination mean or standard deviation), target tone mapping curves, or target automatic white balance (AWB) values ​​from the visible portion of the image (rather than the entire input image) should improve the visual quality of the resulting visualization presented to the user.

[0048] In some embodiments, the image processing component 160 may use a region magnification factor or pixel density in a magnification or pixel density map to control one or more image quality operations in a local region of the back-projected view 130. For example, the image processing component 160 may divide the viewport 110 into multiple regions (e.g., corresponding to regions in a magnification or pixel density map), and the image processing component 160 may use corresponding parameters that depend on the magnification factor or pixel density in each region to perform one or more image quality operations (e.g., temporal noise reduction, color adjustment, tone mapping, etc.) (or the image processing component 160 may operate on smaller sub-tiles and interpolate between tiles).

[0049] Taking temporal noise suppression as an example, the image processing component 160 can use the corresponding region magnification factor or pixel density from the magnification map or pixel density map to perform temporal noise suppression on various regions of the back-projected view 130. In some embodiments, since noise in regions with higher magnification or lower pixel density may ultimately be amplified, the image processing component 160 can apply stronger temporal noise suppression to regions with higher magnification or lower pixel density, and vice versa. For example, in Figure 4 In the middle, the magnification factor of each image is less than 1 (closer to the vehicle), while in... Figure 5 In this example, the magnification factor for each image is greater than 1 (for images further away from the vehicle). Because upsampling amplifies noise, the image processing component 160 can... Figure 5 The right-hand image 530 application ratio Figure 4 The right-hand image 430 exhibits stronger temporal noise suppression. In some embodiments, since regions with lower magnification (e.g., below 1) should have a correspondingly higher available input pixel density, and downsampling typically reduces noise by applying an averaging filter, the image processing component 160 can apply weaker temporal noise attenuation and a smaller window size in these regions to reduce or avoid ghosting artifacts in textured / object regions. For smooth regions, the image processing component 160 can apply stronger temporal noise attenuation and a larger window size to further reduce noise levels. Additionally or alternatively, the image processing component 160 can mix spatial and temporal filtering, controlling the mixing ratio based on the magnification or pixel density.

[0050] In some embodiments, the image processing component 160 can control the intensity of temporal noise filtering based on the determination of whether a region is occupied or textured. For example, in some embodiments that generate visualizations of the environment surrounding a vehicle using a 3D bowl model, a lower magnification factor should generally correspond to ground areas closer to the vehicle, and these areas can be assumed to be unoccupied and untextured. Therefore, the image processing component 160 can determine to apply stronger temporal noise reduction to regions with lower magnification factors. Conversely, the image processing component 160 can determine to apply weaker temporal noise reduction to occupied or textured regions to reduce or avoid ghosting or blur artifacts. The image processing component 160 may assume that a region is occupied or textured based on: a magnification factor (e.g., greater than 1); determination that a region of the back-projected view 130 is occupied by an undetected object (e.g., detected using any known object detection technique); determination that a region of the back-projected view 130 depicts the ground (e.g., by mapping the ground plane of a 3D bowl-shaped model of the surrounding environment to the corresponding portion of the back-projected view 130 and removing pixels corresponding to the detected object, etc.); and / or any other known technique.

[0051] Therefore, the image processing component 160 can apply temporal noise suppression to local areas of the back-projected view 130, use different filtering intensities in different areas, and mix the results from each area.

[0052] In some embodiments, the image processing component 160 may apply color adjustments to various regions of the back-projected view 130 using a corresponding region magnification factor or pixel density from a magnification map or pixel density map, and blend the results between different regions. In some embodiments, the projection component 120 may identify which sensor(s)(s)(s)(s) contributed which pixels to the back-projected view 130 and provide this information to the image processing component 160, which may then apply color adjustments to the corresponding regions of the back-projected view 130 using the corresponding ISP metadata. Alternatively, since magnification may affect brightness, contrast, and / or color balance, the image processing component 160 may recalculate image statistics (e.g., local histogram, contrast, white balance) for each local region, use the recalculated image statistics to adjust the brightness, contrast, and / or color balance of each local region, and blend the results from each region.

[0053] In some embodiments, image processing component 160 may apply tone mapping to various regions of the back-projected view 130 using a corresponding region magnification factor or pixel density from a magnification map or pixel density map. For example, image processing component 160 may apply local tone mapping to corresponding local regions of the back-projected view 130 and blend the results of each region. In some embodiments, projection component 120 may identify which sensor(s)(s)(s)(s) contributed which pixels to the back-projected view 130 and provide this information to image processing component 160. Therefore, image processing component 160 may use the corresponding ISP metadata to process the corresponding regions of the back-projected view 130. In some implementations, image processing component 160 may adjust the applicable tone curve according to the magnification factor of each local region, or generate separate tone curves for regions with relatively high or low magnification factors. Therefore, image processing component 160 may perform tone mapping (e.g., adjust brightness, contrast, exposure, etc.) in each local region using adjusted or separated tone curves and blend the results of each region. These are just a few examples, and various variations can be implemented within the scope of this disclosure.

[0054] Therefore, the resulting visualization 170 can be displayed on a display 180 (e.g., a monitor, projector, wearable display, etc.) for access by the occupants or operators of the self-machine (e.g., driver or passenger, remote operator).

[0055] Now for reference Figure 6 Each module of the method 600 described herein contains a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. Method 600 can also be embodied in the form of computer-usable instructions stored on a computer storage medium. Method 600 can be provided as a standalone application, a standalone service, a managed service (alone or in combination with other managed services), or a plug-in to another product, etc. Furthermore, this document uses… Figure 1 Method 600 is described using the surround view system 100 shown as an example. However, these methods can also be implemented by any system or any combination of systems, including but not limited to the systems described herein.

[0056] Figure 6 This is a flowchart illustrating a method 600 for performing one or more image quality operations in a back-projected view, according to some embodiments of the present disclosure. Method 600 includes, at block B602, generating at least one frame representing a back-projected view of the environment, based at least on one or more frames of image data frames back-projected from a two-dimensional (2D) viewport to at least a partially overlapping view representing an environment surrounding a self-machine. Figure 1 The surround view system 100 shown, self-machine (e.g.) Figures 7A-7D The vehicle 700 can be equipped with any number and type of sensors 101 (e.g., one or more cameras, such as fisheye cameras), and the sensors 101 can be used to generate overlapping sensor data frames (e.g., overlapping image data) representing substantially the same time slices. In one example configuration, four fisheye cameras can be mounted on the front, left, rear, and right sides of the vehicle, and corresponding surround-view videos can be generated at a specified frame rate. An image signal processor (ISP) can be used to perform tasks such as noise reduction, automatic exposure, white balance, and initial color correction. The projection assembly 120 can sample image data from the respective sensors 101 using back projection and generate a back-projected view 130 through the viewport 110. For example, the projection assembly 120 can trace pixel rays in the screen space of the viewport 110 to corresponding intersections on the surface of a 3D model of the environment (e.g., a 3D bowl), map these intersections to corresponding pixels in the image data from the respective sensors 101 (e.g., using inverse UV mapping), and then retrieve or identify the image data from the corresponding pixels. Therefore, projection component 120 may store image data, reference image data, or otherwise associate image data with corresponding pixels in the reverse-projected view 130 (e.g., a two-dimensional image through viewport 110).

[0057] Method 600 includes, at block B604, performing one or more image quality operations on at least one frame. For example, for Figure 1 The surround view system 100 shown may allow the projection component 120 to control the sampling of local regions of the back-projected view 130 using a region magnification factor or pixel density (e.g., upsampling, downsampling, super-resolution, anti-aliasing, region subsampling, morphing subsampling pixels, etc.). Alternatively, the image processing component 160 may perform image quality processing on the pixels visible in the back-projected view 130 (e.g., instead of performing this image processing on the original image data). Some example image quality processing types that the image processing component 160 may apply to the pixels in the back-projected view 130 include temporal noise reduction, harmonization, color adjustment, and tone mapping.

[0058] The systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles connected to one or more trailers, aircraft, ships, shuttles (e.g., robotaxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones, and / or other types of vehicles. Furthermore, the systems and methods described herein can be used for a wide range of purposes, including, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets (e.g., NVIDIA's Omniverse), cloud computing, and / or any other suitable applications.

[0059] The disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines, etc.), systems implemented using robots, aviation systems, medical systems, marine systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing language models (such as large language models (LLM), visual language models (VLM), and / or multimodal language models), systems using or deploying one or more inference microservices, systems including deploying one or more machine learning models in services or microservices and OS-level virtualization packages (e.g., containers), systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in data centers, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0060] Example autonomous or semi-autonomous machines Figure 7A Examples of sensor positions with corresponding fields of view or sensing fields of view of autonomous or semi-autonomous vehicles 700A, autonomous mobile robots (AMRs) 700B, and humanoid robots 700C according to some embodiments of this disclosure. While three types of machines 700 are shown, this is not intended to be limiting, and the machines 700 described herein may include vehicles, cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police or emergency vehicles, ambulances, boats, construction vehicles, underwater vehicles, robots (e.g., AMRs, humanoid robots, robotic arms, end effectors, forklifts, etc.), drones, aircraft, vehicles coupled to trailers (e.g., semi-trailer tractors for hauling goods), and / or another type of vehicle or machine (e.g., driverless and / or vehicles or machines accommodating one or more passengers). In some cases, vehicle 700A, AMR 700B, humanoid robot 700C, and / or other machine types may be collectively referred to herein as machine 700.

[0061] Regarding Vehicle 700A, autonomous and semi-autonomous vehicles are typically described by automation levels, defined by the National Highway Traffic Safety Administration (NHTSA) (a division of the U.S. Department of Transportation) and the Society of Automotive Engineers (SAE) in their "Classification and Definition of Terms Related to Driving Automation Systems for Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 700 may have one or more functions that meet Level 3 through Level 5 of autonomous driving. For example, Vehicle 700 may be able to provide driver assistance (Level 1), partial automation (Level 2, Level 2+, Level 2++), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the specific implementation. The term “autonomy” as used herein can include any and / or all types of autonomy of Machine 700 or other machines, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, provision of auxiliary autonomy, semi-autonomy, primary autonomy or other specified autonomy.

[0062] about Figure 7A The sensors and their respective fields of view (not shown for clarity) or sensing fields (not shown for clarity) are an example embodiment and are not intended to be limiting. Although not shown, each sensor may have a corresponding field of view (e.g., a 360-degree field of view around camera 768D, a 180-degree field of view around wide-angle camera 768B, a 360-degree sensing field of view of LiDAR sensor 764, etc.). For example, only a subset of the sensors shown may be included, additional sensors may be included, alternative sensors may be included, the number of each sensor mode may be different, the sensor modes may be different (e.g., LiDAR or RADAR may not be included, SONAR, thermal sensors, etc. may be included), and the sensor locations may differ from those shown on vehicles 700A, AMR 700B, and / or humanoid robots 700C, etc. For example, for vehicle 700A, the location, number, modes, and / or other sensor information may differ depending on the type (e.g., SUV, truck, car, robot, motorcycle, etc.), size (e.g., 18-wheeler, transport vehicle, small car, etc.), and associated functions (e.g., L2 vs. L5). Similarly, for the AMR 700B and / or the humanoid robot 700C, the shape, size, purpose, implementation method, model, etc., can determine the number and type of sensors used.

[0063] like Figure 7AAs shown, autonomous or semi-autonomous vehicles 700A, AMR 700B, and humanoid robots 700C may include different sensor types, numbers, and locations. As a non-limiting example, vehicle 700A may include twelve cameras 764, such as a front wide-angle camera (e.g., 120-degree field of view (FOV)), a front telephoto camera (e.g., 30-degree FOV), a side-rear left camera (e.g., 70-degree FOV), a side-rear right camera (e.g., 70-degree FOV), a front fisheye camera (e.g., 200-degree FOV), a rear fisheye camera (e.g., 200-degree FOV), a left fisheye camera (e.g., 200-degree FOV), a right fisheye camera (e.g., 200-degree FOV), a front telephoto satellite camera (e.g., 30-degree FOV), a rear telephoto camera (e.g., 30-degree FOV), a cross-left camera (e.g., 120-degree FOV), and a cross-right camera (e.g., 120-degree FOV). In this embodiment, the camera 764 may use a gigabit multimedia serial link (GMSL) interface (e.g., GMSL2) as input / output (I / O).

[0064] In some embodiments, although Figure 7A As not shown, vehicle 700A may include an in-cabin occupant and / or driver monitoring system, which may include various sensors. For example, in-cabin sensors may include various cameras 768, such as a driver monitoring camera (e.g., located in front of the driver's seat and facing the driver's seat at a 55-degree FOV), a front occupant monitoring camera (e.g., located in front of the front occupant seat and facing the front occupant seat at a 190-degree FOV), and a rear occupant monitoring camera (e.g., located in front of the rear occupant seat and facing the rear occupant seat at a 190-degree FOV). Similar to external cameras 768, in embodiments, internal cameras 768 may use a GMSL (e.g., GMSL2) interface for I / O.

[0065] As another non-limiting example, vehicle 700A may also include nine RADAR sensors 760. For example, vehicle 700A may include a front center imaging RADAR sensor (e.g., 120-degree FOV or sensing field), a left front corner RADAR sensor (e.g., 160-degree FOV or sensing field), a right front corner RADAR sensor (e.g., 160-degree FOV or sensing field), a right rear corner RADAR sensor (e.g., 160-degree FOV or sensing field), a left RADAR sensor (e.g., 160-degree FOV or sensing field), a right RADAR sensor (e.g., 160-degree FOV or sensing field), a left rear RADAR sensor (e.g., 50-degree FOV or sensing field), and a right rear RADAR sensor (e.g., 50-degree FOV or sensing field). In embodiments, the RADAR sensors 760 may use an Ethernet interface as I / O.

[0066] As a non-limiting example, the vehicle 700A may also include twelve ultrasonic sensors 762. Figure 7A As shown, the ultrasonic sensor can be placed along the front and rear bumpers of vehicle 700A and along the sides of vehicle 700A, and can be used to detect objects (static and dynamic) close to vehicle 700A. In some embodiments, the ultrasonic sensor 762 can use a DS13 interface as I / O.

[0067] As a non-limiting example, the vehicle 700A may also include a LiDAR sensor 764, such as a front-center LiDAR sensor (e.g., a 120-degree horizontal FOV or sensing field and a 30-degree vertical FOV or sensing field). In some embodiments, such as when using additional or alternative LiDAR sensors, the LiDAR sensor may have different horizontal and vertical fields of view or sensing fields. For example, the LiDAR sensor 764 may include a 360-degree horizontal FOV or sensing field (e.g., located in a rotating LiDAR sensor) and a 90-degree vertical FOV or sensing field. In some embodiments, the LiDAR sensor 764 may use an Ethernet interface as I / O.

[0068] As a non-limiting example, the Autonomous Mobile Robot (AMR) 700B may include three LiDAR sensors 764. For example, the topmost LiDAR sensor 764 may include a beam or 3D LiDAR sensor (e.g., a 360-degree horizontal and 90-degree vertical FOV or sensing field), and the front and rear LiDAR sensors may include planar or 2D LiDAR sensors (e.g., a 180-degree horizontal FOV or sensing field).

[0069] As a non-limiting embodiment, the AMR 700B may further include eight cameras 768, such as a front stereo camera (e.g., 120-degree FOV), a rear stereo camera (e.g., 120-degree FOV), a left stereo camera (e.g., 120-degree FOV), a right stereo camera (e.g., 120-degree FOV), a front fisheye camera (e.g., 202-degree ± 3-degree FOV), a rear fisheye camera (e.g., 202-degree ± 3-degree FOV), a left fisheye camera (e.g., 202-degree ± 3-degree FOV), and a right fisheye camera (e.g., 202-degree ± 3-degree FOV).

[0070] The AMR 700B may also include a charging port, charging port contacts, status indicators, one or more (e.g., four) RGB LEDs, one or more IMU sensors 766, a magnetometer, and a barometer. The AMR 700B is capable of high-precision time synchronization between sensors using hardware timestamps and PTPs over Ethernet with sensor acquisition times of less than 10 microseconds. In embodiments, the AMR 700B provides simultaneous camera capture within 100 microseconds of a single hardware trigger on all cameras 768, and can write sensor captures to disk at a rate of 4 GB / s for writing packets (e.g., writing to ROSbags of the Robot Operating System (ROS)). Therefore, the AMR 700B is capable of running ROS (e.g., NVIDIA's IsaacROS), can be remotely operated (as described herein), can map the environment, and can navigate the environment using vision cameras 768, LiDAR 764, and / or other sensor types or modalities.

[0071] The humanoid robot 700C may include (as a non-limiting example) a LiDAR sensor 764. For example, the LiDAR sensor 764 may include a beam or a 3D LiDAR sensor (e.g., a 360-degree horizontal and 90-degree vertical FOV or sensing field), or it may include a planar or 2D LiDAR sensor (e.g., a 180-degree horizontal FOV or sensing field).

[0072] As a non-limiting embodiment, the humanoid robot 700C may also include four cameras 768, such as a front stereo camera (e.g., 120-degree FOV), a rear stereo camera (e.g., 120-degree FOV), a front fisheye camera (e.g., 202-degree ± 3-degree FOV), and a rear fisheye camera (e.g., 202-degree ± 3-degree FOV).

[0073] As a non-limiting embodiment, the humanoid robot 700C may also include four ultrasonic sensors 762, such as a left arm ultrasonic sensor, a right arm ultrasonic sensor, a left leg ultrasonic sensor, and a right leg ultrasonic sensor.

[0074] The humanoid robot 700C may also include any number of actuators, such as those allowing control and manipulation of joints. For example, the humanoid robot 700C may include actuators that allow for various degrees of freedom (DoF) depending on the design. In a non-limiting embodiment, the humanoid robot 700C may have a total of 40 degrees of freedom (DoF) (e.g., 6DoF x2 for arms, 6DoF x2 for hands, 6DoF x2 for legs, 2DoF for the torso, and 2DoF for the neck). Actuators can convert energy into physical motion, thereby allowing actions such as joint movement, locomotion, and grasping / manipulation. For example, motors and servos can be used to perform joint movements to control the rotation of joints in an arm or manipulator and allow for reaching, grasping, and manipulating objects. Locomotion can be achieved by moving through the environment using wheels, tracks, or other locomotion devices (robot legs). Grasping and manipulation can be performed using end effectors or hands / fingers, which can be equipped with actuators to grasp objects, apply forces, and perform specific tasks. In some examples, the humanoid robot 700C may include position and orientation sensors, such as encoders, gyroscopes, etc., to determine the robot 700C's position in space, thereby enabling position determination and motion tracking. In embodiments, the humanoid robot 700C may include force and pressure sensors to detect environmental interactions, enabling the robot 700C to grasp objects with appropriate force and avoid obstacles along its path. Perception sensors (e.g., cameras, LiDAR, RADAR, ultrasound, sonar (SONAR), etc.) may be used in conjunction with tactile sensors to enable the robot 700C to perceive objects, shapes, and textures, and to understand when to begin and stop touching (along with force sensors that regulate the force used during touching). As a non-limiting example, the humanoid robot 700C may have a height of approximately 1-2 meters (e.g., 1.7 meters or 5 feet 6 inches), a weight of 50-70 kilograms, be able to move at speeds of 8 km / h or higher, and be able to carry a payload of 20-100 kilograms, depending on the system design and requirements.

[0075] In embodiments, the humanoid robot 700C may include a dialogue system—such as a dialogue system driven by a language model (e.g., LLM, VLM, MMLM, VLA, etc.)—to help understand the environment, reason, and communicate with humans, animals, devices, and / or other robots, and / or make planning, control, and navigation decisions. Therefore, in addition to performing various tasks, the humanoid robot 700C can also use onboard sensors, microphones, and speakers to understand speech, audio, and visual cues, while also being able to communicate with the environment.

[0076] Referring to camera 768 of machine 700, the camera type of camera 768 may include, but is not limited to, digital cameras applicable to components and / or systems of machine 700. For vehicle 700A implementation, camera 768 may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may have any image capture rate, such as 30 frames per second (fps), 60 fps, 120 fps, 240 fps, etc., depending on the embodiment. The camera may use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red colorless colorless colorless (RCCC) color filter array, a red colorless colorless blue (RCCB) color filter array, a red blue green colorless (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a colorless pixel camera (e.g., a camera with an array of RCCC, RCCB, and / or RBGC color filters) can be used to improve light sensitivity.

[0077] The field of view includes cameras (e.g., front-facing cameras) in the area in front of the machine 700, which can be used for surround view to help identify the path and obstacles ahead, and, with the help of one or more controllers 736 and / or control SoCs, provide information crucial for generating an occupancy grid and / or determining preferred machine movement, trajectory, and / or path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including lane departure warning (“LDW”), adaptive cruise control (“ACC”), and / or other functions such as traffic sign recognition.

[0078] Various cameras can be used in front-mounted configurations, including, for example, monocular camera platforms that include complementary metal-oxide-semiconductor (“CMOS”) color imagers. Another example could be a wide-angle camera 768B, which can be used to perceive objects entering the field of view from the periphery (e.g., pedestrians, warehouse vehicles, other robots, pedestrian traffic, or bicycles). Furthermore, any number of long-range cameras 768E (e.g., a pair of long-range stereo cameras) can be used for depth-based object detection, especially for objects for which neural networks have not yet been trained. The long-range camera 768E can also be used for object detection and classification, as well as basic object tracking.

[0079] Any number of stereo cameras 768A can also be included in front-mounted and / or other (e.g., rear-mounted) configurations. In at least one embodiment, one or more stereo cameras 768A may include an integrated control unit comprising a scalable processing unit that can provide programmable logic (“FPGA”) and a multi-core microprocessor with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the machine 700 environment, including distance estimates of midpoints in the image (e.g., parallax or depth images). Alternative stereo cameras 768A may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip that can measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 768A may be used in addition to those described herein, or as an alternative to those described herein. For example, in some embodiments, cameras other than stereo cameras (e.g., two monocular cameras with at least partially overlapping fields of view) may be used to perform stereo depth estimation.

[0080] Cameras with a field of view including portions of the side environment of machine 700 (e.g., side-view cameras) can be used, for example, for surround view, providing information for creating and updating occupancy grids, and generating side collision warnings and / or indicating to AMR 700B or humanoid robot 700C, for example, the presence of objects, features, and / or people on the side. For example, surround cameras 768D can be mounted on machine 700. Surround cameras 768D can include wide-angle cameras 768B, fisheye cameras, 360-degree cameras, etc. For example, four fisheye cameras can be mounted on the front, rear, and sides of machine 700. In an alternative arrangement, machine 700 can use three surround cameras 768D (e.g., left, right, and rear), and can utilize one or more other cameras (e.g., front-facing cameras) as a fourth surround-view camera.

[0081] A camera 768 (e.g., a rear-view camera) having a field of view including a portion of the environment behind the machine 700 can be used to understand objects, features, people, and / or other information behind the machine 700, such as for parking assistance, surround view, rear collision warning, planning, control, and navigation determination, and / or to create and update occupancy grids, BEV images representing the environment, height maps, etc. A wide variety of cameras 768 can be used, including but not limited to those also suitable for use as front cameras (e.g., long-range and / or mid-range cameras 768E, stereo cameras 768A, infrared cameras 768C, etc.), rear cameras, side cameras, downward cameras, upward cameras, and / or similar cameras 768, as described herein.

[0082] Similarly, for LiDAR sensor 764, RADAR sensor 760, ultrasonic sensor 762 and / or other sensor modes or types, the location and placement of the sensors and their corresponding fields of view or sensing fields can be determined based on the use case, implementation or design of the particular machine 700.

[0083] For example, machine 700 includes a RADAR sensor 760, which can be used by machine 700 for long-range object detection, even in dark and / or inclement weather conditions. In embodiments, the RADAR functional safety level may be ASIL B. RADAR sensor 760 can be controlled and accessed for object tracking data using CAN and / or bus 702 (e.g., for transmitting data generated by RADAR sensor 760), and in some examples, raw data can be accessed using Ethernet. Various types of RADAR sensors can be used. For example, but not limited to, RADAR sensor 760 can be suitable for front, rear, and side radar applications. In some examples, a pulse Doppler RADAR sensor is used.

[0084] The RADAR sensor 760 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range lateral coverage, etc. In some examples, long-range radar can be used for adaptive cruise control (ACC) functions. A long-range RADAR system can provide a wide field of view, for example, within a 250m range, achieved through two or more independent scans. The RADAR sensor 760 can help distinguish between static and moving objects; ADAS systems can use it for emergency braking assistance and forward collision warning, and robots can use it to detect dynamic objects in various environments (e.g., low-light or no-light environments). A long-range RADAR sensor can include a single static multimodal RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record the machine 700's surroundings at higher speeds while minimizing peripheral interference (e.g., traffic from adjacent lanes). The other two antennas expand the field of view, enabling rapid detection of objects entering or leaving the machine's direct path (e.g., lanes).

[0085] Mid-range RADAR systems can include, for example, a range of up to 960m (front) or 80m (rear), and a field of view of up to 42 degrees (front) or 150 degrees (rear). Short-range RADAR systems can include, but are not limited to, RADAR sensors designed to be mounted at both ends of a side surface (e.g., rear bumper), allowing continuous monitoring of blind spots behind and beside a machine (e.g., vehicle, robot, etc.) using two beams. Therefore, short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0086] Machine 700 may also include an ultrasonic sensor 762. The ultrasonic sensor 762 may be located at the front, rear, and / or sides of machine 700 and may be used to assist near-field perception, such as for parking assistance, collision avoidance (e.g., for robot parts), and / or to create and update occupancy grids, evidence grid maps (EGMs), height maps, BEV images, and / or other representations of objects and features in the machine 700 environment. Multiple ultrasonic sensors 762 may be used, and different ultrasonic sensors 762 may be used for different detection ranges (e.g., 2.5 m, 4 m). For example, the ultrasonic sensor 762 may operate at an ASIL B functional safety level.

[0087] Machine 700 may include a LiDAR sensor 764. The LiDAR sensor 764 can be used for object and feature detection, pedestrian and other robot detection, emergency braking, collision avoidance, simultaneous localization and mapping (SLAM), free space detection, and / or other functions. In embodiments, the LiDAR sensor 764 may be of functional safety level ASIL B. In some examples, machine 700 may include multiple LiDAR sensors 764 (e.g., two, four, six, etc.), which can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0088] In some examples, the LiDAR sensor 764 may be able to provide a 360-degree field of view of objects and their distances. For example, a commercially available LiDAR sensor 764 may have a advertised range of approximately 900m, an accuracy of 2cm-3cm, and support for 900Mbps Ethernet connectivity. In some examples, one or more non-protruding LiDAR sensors 764 may be used. In such examples, the LiDAR sensor 764 can be implemented as a small device that can be embedded in the front, rear, side, top, and / or corner of the machine 700. In such examples, the LiDAR sensor 764 can provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, with a range of up to 200m even for low-reflectivity objects. A front-mounted LiDAR sensor 764 can be configured with a horizontal field of view between 45 and 135 degrees.

[0089] In some examples, LiDAR technology, such as 3D flash LiDAR, can also be used. 3D flash LiDAR uses a laser flash as a transmission source, illuminating the environment around a vehicle up to approximately 200m away. The flash LiDAR unit includes a receiver that records the laser pulse transmission time and the reflected light on each pixel, which in turn corresponds to the distance from the vehicle to the object. Flash LiDAR allows for the generation of highly accurate and distortion-free images of the surrounding environment with each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of machine 700. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras with no moving parts other than a fan (e.g., non-scanning LiDAR devices). Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of 3D distance point clouds and co-registration intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, LiDAR sensor 764 may be less susceptible to motion blur, vibration, and / or shock.

[0090] Figure 7B This is an illustration of the location of sensors and components of an example autonomous or semi-autonomous vehicle 700A (also referred to herein as "vehicle 700", "self-vehicle 700", "self-machine 700", or "machine 700") according to some embodiments of this disclosure. While vehicle 700A is shown in the figure, this is not intended to be limiting, and similar components and / or sensors may be included on any other machine type without departing from the scope of this disclosure. For example, similar sensors and / or components may be used in vehicles, cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police cars, ambulances, ships, construction vehicles, underwater vehicles, robots (e.g., AMRs, humanoid robots, robotic arms, end effectors, forklifts, etc.), drones, aircraft, vehicles coupled to trailers (e.g., semi-trailers for hauling goods), and / or another type of vehicle or machine (e.g., driverless and / or capable of accommodating one or more passengers).

[0091] Figure 7CThis is a block diagram of an example system architecture for a machine 700 (e.g., an autonomous or semi-autonomous vehicle 700A, an autonomous mobile robot (AMR) 700B, a humanoid robot 700C, and / or other types of machines) according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are presented as examples only. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of the arrangements shown, and some elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components or together with other components, and can be implemented in any suitable combination and location (e.g., on a local device, vehicle, or edge machine, in the field (e.g., a locally hosted server), in a remote location (e.g., in one or more computing or server devices in one or more data centers in the cloud) and / or other locations). The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, microcontroller, embedded processor, digital signal processor (DSP), image signal processor (ISP), physical processing unit (PPU), field-programmable gate array (FPGA), accelerators (e.g., deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural network accelerator (NNA) and / or neural processing unit (NPU), programmable vision accelerator (PVA), optical flow accelerator (OFA), etc.), application-specific integrated circuit (ASIC), data processing unit (DPU), quantum processor, etc.) executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 7A to 7E Example machine 700, Figure 8 Example computing ecosystem 800, Figure 9 Example Generative Language Model System 900 and / or Figure 10 The example computing device 1000 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the operation.

[0092] Figure 7CEach component, feature, and system of machine 700 is shown connected via bus 702 (or referred to as "machine communication network 702" or simply "communication network 702"). Bus 702 may include a Controller Area Network (CAN) data interface (or referred to herein as "CAN bus"). CAN can be a network within machine 700 used to help control various features and functions of machine 700, such as brake actuation, acceleration, braking, steering, windshield wipers, etc. CAN bus can be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CANID). CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button position, and / or other vehicle status indicators. CAN bus may conform to ASIL B standard. In some embodiments, in addition to or as an alternative to the CAN bus, bus 702 may include FlexRay, embedded buses (e.g., SPI, I2C), local interconnect links (LIN), NVIDIA's NVLink, USB (2.0, 3.0 and above), radio frequency (RF), Ethernet (e.g., 10BASE / 100BASE, 1000BASE, 10G, etc.), and / or other communication protocols or functions. Furthermore, while a single line is used to represent bus 702, this is not limiting. For example, there can be any number of buses 702, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 702 may be used to perform different functions and / or for redundancy. For example, a first bus 702 may be used for collision avoidance, while a second bus 702 may be used for actuation control. In any example, each bus 702 may communicate with any component of machine 700, and two or more buses 702 may communicate with the same component. In some examples, each computer or computing engine within each SoC 704, each controller 736, and / or machine 700 can access the same input data (e.g., input from sensors in the machine 700) and can be connected to a common bus, such as the CAN bus.

[0093] Machine 700 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, batteries, side mirrors, and / or other parts of the vehicle or machine. Machine 700 may include a propulsion system 750, such as an internal combustion engine, a hybrid power plant, an all-electric motor, a hydrogen fuel cell engine, and / or another type of propulsion system. Propulsion system 750 may be connected to the drivetrain of machine 700, which may include a transmission to enable propulsion of machine 700. Propulsion system 750 may be controlled in response to a signal received from throttle / accelerator 752.

[0094] Steering system 754 may include a steering wheel and / or other steering mechanisms (e.g., remote steering and / or local steering) for steering machine 700 (e.g., along a desired path or route) while propulsion system 750 is in operation (e.g., when the vehicle is moving). Steering system 754 may receive signals from steering actuator 756. In some embodiments, a steering wheel or other steering mechanism may be omitted, for example, for machine 700 capable of fully automated (e.g., level 5) functionality.

[0095] The brake sensor system 746 can be used to operate the vehicle brakes in response to signals received from the brake actuator 748 and / or the brake sensor.

[0096] Machine 700 may include one or more controllers 736, such as those described herein. Figure 7AThe controller 736 is described. It can be used for a variety of functions and can be coupled to any of the various other components and systems of the machine 700. For example, the controller 736 can be used to control the machine 700, artificial intelligence performed on the machine 700, infotainment on the machine 700, etc. For example, one controller 736 can be used for some or all of the functions, or different controllers 736 can be used for different functions, for example, to ensure availability and safety separation between various controllers for different tasks. For example, the controller 736 can use a system-calculated plan (e.g., the path or trajectory of vehicle 700A or AMR 700B, or the movement, component trajectory, motion position or displacement, etc. of joints or components (e.g., manipulators, end effectors, limbs, hands, fingers, legs, feet, etc.) of humanoid robot 700C to control the machine 700 in the environment. In some cases, the controller 736 may include a proportional-integral-derivative (PID) controller, a fuzzy logic controller, a neural controller (e.g., a controller embodied as one or more neural networks), a force control controller, a programmable logic controller (PLC), and / or other types of controllers. For example, in the humanoid robot 700C, the controller 736 can act as the brain, responsible for analyzing sensor data, making decisions, and sending commands to the actuators. The controller 736 may include a low-level controller that handles basic motor control, ensuring accurate and precise movement of the individual joints and actuators. The controller 736 may also include a high-level controller to coordinate multiple actuators and sensors, plan complex movements, and adapt to constantly changing environments.

[0097] In some embodiments, controller 736 may include an artificial intelligence controller that can use AI algorithms (e.g., DNN, MLM, etc.) to learn, make decisions, and autonomously perform tasks of machine 700. In some embodiments, controller 736 may use a fixed open-loop control algorithm and may not adjust its actions based on the environment. In other embodiments, closed-loop control may be used, which incorporates a feedback mechanism to monitor the robot's performance and make necessary adjustments. In examples, controller 736 may implement reactive control to directly respond to sensor inputs, enabling rapid reflexive actions and real-time changes. Furthermore, in some examples, deliberate control may be implemented, using internal models and planning algorithms to generate advanced actions, which may be suitable for complex tasks requiring reasoning, decision-making, and long-term planning.

[0098] The controller 736 may include one or more system-on-chip (SoC) 704 ( Figure 7C and Figure 7DThe controller 736 may provide signals (e.g., signals representing commands or messages) to one or more components and / or systems of the machine 700, including components such as CPUs, GPUs, and accelerators. Although the controller 736 is listed separately from the SoC 704, this is not intended to be limiting, and in some embodiments, one or more components of the SoC 704 may perform the operations of the controller 736. For example, the controller may send signals to operate machine brakes via one or more brake actuators 748, to operate steering system 754 via one or more steering actuators 756, to operate propulsion system 750 via one or more throttle / accelerators 752, and so on. The controller 736 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous or semi-autonomous navigation and movement and / or assist human operators using the machine 700. Controller 736 may include a first controller 736 for autonomous control and navigation functions, a second controller 736 for functional safety functions, a third controller 736 for artificial intelligence functions (e.g., computer vision), a fourth controller 736 for infotainment functions, a fifth controller 736 for redundancy in emergency situations, and / or other controllers. For example, hardware for safety monitoring and other safety functions (e.g., functional safety islands) may be discrete or partitioned (physically or by processing separation) relative to hardware for processing sensor data to make perception and vehicle control decisions. Similarly, hardware for controlling in-vehicle infotainment and / or in-cabin monitoring (e.g., controllers, SOCs, etc.) may be separate or independent from hardware for vehicle perception and control. In some examples, a single controller 736 may handle two or more of the above functions, two or more controllers 736 may handle a single function, and / or any combination thereof.

[0099] The controller 736 may provide signals for controlling one or more components and / or systems of the machine 700 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data can be received from, for example but not limited to, Global Navigation Satellite System (“GNSS”) sensor 758 (e.g., Global Positioning System sensor), RADAR sensor 760, ultrasonic sensor 762, LiDAR sensor 764, Inertial Measurement Unit (IMU) sensor 766 (e.g., accelerometer, gyroscope, magnetic compass, magnetometer, etc.), microphone 796, camera 768 (e.g., stereo camera 768A, wide-angle camera 768B (e.g., fisheye camera), infrared camera 768C, surround camera 768D (e.g., 360-degree camera), long-range and / or medium-range camera 768E and / or other types of cameras), speed sensor 744 (e.g., for measuring the speed of machine 700), vibration sensor 742, steering sensor 740, braking sensor (e.g., as part of braking sensor system 746), actuators and / or other types of sensors.

[0100] One or more controllers 736 may receive input (e.g., represented by input data) from the instrument panel 732 of the machine 700 and provide output (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 734 (e.g., screen, head-up display, mirror display, facial display, robot display, etc.), sound alarms, loudspeakers, speakers, and / or via other components of the machine 700. Outputs may include parameters such as machine speed, rate, time, and... Figure 7C The system may display map data corresponding to map 722 (e.g., from a navigation map, a standard definition (SD) map, a high definition (“HD”) map, etc.), location data (e.g., the location of machine 700, such as its location on map 722), direction, the location of other vehicles (e.g., occupancy map, elevation map, bird's-eye view (BEV) image, grid, etc.), information about objects perceived by the system and their status, system status information, and so on. For example, HMI display 734 may display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.), and / or information about driving maneuvers that the vehicle has performed, is performing, or will perform (e.g., changing lanes now, exiting from exit 34B in two miles, etc.).

[0101] Machine 700 may include one or more System-on-a-Chip (SoC) 704 (in Figure 7D(described in more detail below). SoC 704 may include CPU 706, GPU 708, processor 710, cache 712, accelerator 714, data storage 716, and / or other components and features. SoC 704 can be used to process and provide data for various operations, such as navigation, planning, reasoning, inference, perception, control, and / or actuation operations of machine 700 on various platforms and systems. For example, SoC 704 can process real-time perception data (e.g., from cameras, LiDAR, RADAR, ultrasound, etc.) and map data corresponding to one or more maps 722 (e.g., HD maps, SD maps, navigation maps, occupancy maps, etc.) to perform or assist in performing various operations of machine 700. When using maps and / or AI, maps and / or AI (e.g., model parameter updates, fine-tuning, etc.) are transmitted via network interface 724 from one or more servers (e.g., [unclear]). Figure 7E Refresh and / or update the server 778 (e.g., one or more servers in a cloud-based data center).

[0102] Despite Figures 7A to 7E The SoC 704 is shown; however, additional or alternative components and / or architectures, such as multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiPs), field-programmable gate arrays (FPGAs), heterogeneous integration (HI), and single-board computers (SBCs), may be used without departing from the scope of this disclosure. For example, depending on the type of machine 700, the purpose of machine 700, the model of machine 700, and the capabilities required by machine 700, one or more SoC 704s and / or alternative architectures and / or components may be used to meet a particular implementation.

[0103] Machine 700 may include CPU 718 (e.g., a discrete CPU or dCPU) which may be coupled to SoC 704 via a high-speed interconnect (e.g., PCIe). CPU 718 may include, for example, an x86 processor. CPU 718 may be used to perform any of a variety of functions, including arbitrating potentially inconsistent results between ADAS sensors and SoC 704, and / or monitoring the status and health of controller 736 and / or infotainment SoC 730.

[0104] Machine 700 may include GPU 720 (e.g., a discrete GPU or dGPU) which may be coupled to SoC 704 via a high-speed interconnect (e.g., NVIDIA's NVLink). GPU 720 may provide additional artificial intelligence capabilities, such as by executing redundant and / or different neural networks, and may be used to train and / or update neural networks based on inputs from sensors (e.g., sensor data) from Machine 700.

[0105] The machine 700 may also include a network interface 724, which may include one or more wireless antennas 726 and / or modems (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 724 can be used to establish wireless connectivity with the cloud (e.g., with server 778 and / or other network devices) via the Internet, and with other vehicles and / or computing devices (e.g., passenger client devices). For communication with other vehicles, direct links and / or indirect links (e.g., across networks and via the Internet) can be established between the two vehicles. A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide the machine 700 with information about vehicles near the machine 700 (e.g., vehicles in front, to the side, and / or behind the machine 700). This functionality can be part of the machine 700's cooperative adaptive cruise control function.

[0106] Network interface 724 may include a SoC that provides modulation and demodulation functions and enables controller 736 to communicate over a wireless network. Network interface 724 may include a radio frequency (RF) front-end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. Frequency conversion can be performed by well-known processes and / or using superheterodyne processes. In some examples, the RF front-end functionality may be provided by a separate chip. For example, network interface 724 may be able to communicate via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multicarrier (“CDMA2000”), 5G, 6G, and / or other cellular and / or wireless communication standards. The wireless antenna 726 can also enable communication between objects in the environment (such as vehicles, mobile devices, etc.) using local area networks (such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc.) and / or low power wide area networks (“LPWAN”) (such as LoRaWAN, SigFox, etc.).

[0107] Machine 700 may also include data memory 728, which may include off-chip (e.g., outside of SoC 704) storage. Data memory 728 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, Flash, hard disk and / or other components and / or devices capable of storing at least one bit of data.

[0108] The machine 700 may also include a GNSS sensor 758. The GNSS sensor 758 (e.g., a GPS, an auxiliary GPS sensor, a differential GPS (DGPS) sensor, etc.) is used to assist in mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 758 can be used, such as, but not limited to, GPS with a USB connector having an Ethernet-to-serial (RS-232) bridge.

[0109] Machine 700 may also include an IMU sensor 766. In some examples, the IMU sensor 766 may be located at the center of the rear axis of machine 700. The IMU sensor 766 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 766 may include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 766 may include an accelerometer, a gyroscope, and a magnetometer.

[0110] In some embodiments, the IMU sensor 766 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 766 can enable the machine 700 to estimate its heading by directly observing and correlating velocity changes from GPS to the IMU sensor 766, without requiring input from a magnetic sensor. In some examples, the IMU sensor 766 and the GNSS sensor 758 can be combined in a single integrated unit.

[0111] The vehicle may include one or more microphones 796 placed inside and / or around the machine 700. The microphones 796 can be used for emergency vehicle detection and identification, etc.

[0112] Machine 700 may also include vibration sensors 742. Vibration sensors 742 can measure vibrations of machine parts, such as the arm or leg of humanoid robot 700C, or the axle of vehicle 700A or AMR 700B. For example, changes in vibration may indicate changes in roads, walking, or traversable surfaces. In another example, when two or more vibration sensors 742 are used, differences between vibrations can be used to determine friction or slippage on surfaces (e.g., when the vibration difference is between an electrically driven shaft and a freely rotating axle).

[0113] Machine 700 may include ADAS system 738, for example, when machine 700 is vehicle 700A. In some examples, ADAS system 738 may include a dedicated SoC. ADAS system 738 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision or collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), blind spot monitoring (BSM), rear cross traffic warning (RCTW), pedestrian detection, driver monitoring, collision warning system (CWS), traffic sign recognition, speed limit detection, automatic parking, lane centering (LC), high beam safety system and / or other features and functions.

[0114] Machine 700 may also include an infotainment SoC 730 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include one or more discrete components, such as multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiP), heterogeneous integration (HI), single-board computers (SBCs), etc. The infotainment SoC 730 may include a combination of hardware and software that can be used to provide machine 700 with audio (e.g., music, personal digital assistant, navigation commands, news, radio, etc.), video (e.g., television, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., wireless, Wi-Fi, etc.) and / or information services (e.g., navigation systems, rear parking assist, radio data systems, vehicle-related information (e.g., fuel level, total driving distance, brake fluid level, fuel level, door opening / closing, air filter information, etc.). For example, an infotainment SoC 730 may be a radio, disk player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, Wi-Fi, steering wheel audio controls, hands-free voice controls, head-up display (HUD), HMI display 734, telematics device, control panel (e.g., for controlling various components, features and / or systems and / or interacting with various components, features and / or systems) and / or other components. The infotainment SoC 730 may also be used to provide information to vehicle users (e.g., visual and / or auditory), such as information from ADAS system 738, autonomous driving information (e.g., planned vehicle maneuvers, trajectory), surrounding environment information (e.g., intersection information, vehicle information, road information, etc.) and / or other information.

[0115] The infotainment SoC 730 may include GPU functionality. The infotainment SoC 730 can communicate with other devices, systems, and / or components of the machine 700 via bus 702 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 730 may be coupled to a monitoring MCU, allowing the GPU of the infotainment system to perform some autonomous driving functions in the event of a failure of the main controller 736 (e.g., the main computer and / or backup computer of the machine 700). In such an example, the infotainment SoC 730 may place the machine 700 into a driver-safe stop mode, as described herein.

[0116] In some embodiments, the infotainment system may provide a digital or virtual assistant, which may be voice-only or may have visual components (e.g., in the form of a digital human or digital avatar). The assistant may provide basic functions such as sending text messages, adjusting vehicle settings, controlling music or video, navigation, etc., and / or more advanced functions, such as those supported by one or more language models (e.g., Large Language Model (LLM), Visual Language Model (VLM), Multimodal Language Model (MMLM), etc.). For example, the driver and / or occupants may interact with the assistant in a manner similar to how a user interacts with a language model, such as asking general questions, specific questions, requesting restaurants, gas stations, and / or other recommendations and / or locations, inquiring about vehicle functions or troubleshooting (e.g., asking for tire pressure information, oil change information, battery swap information, etc.). Therefore, the machine 700 (whether it is a vehicle 700A, AMR 700B, humanoid robot 700C, or other type of machine) may include locally stored language models and / or communicate with remotely hosted language models (e.g., via one or more APIs) to provide users of the machine 700 with more detailed and in-depth communication capabilities.

[0117] In some examples, the infotainment SoC 730, SoC 104, and / or another SoC or computing / processing system can perform in-cabin driver and / or occupant monitoring. For example, the computing system can perform facial recognition, and vehicle owner recognition can use data from cameras and / or other sensors to identify the presence of an authorized driver and / or owner of the machine 700. An always-on sensor processing engine can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in a security mode, to disable the vehicle when the owner leaves. Thus, SoC 704 provides security against theft and / or carjacking.

[0118] In some embodiments, one or more neural networks running on another or a dedicated SoC (e.g., an in-vehicle infotainment or in-vehicle monitoring SoC) can be used to monitor in-cabin monitoring camera sensors. This other or dedicated SoC is configured to recognize in-cabin events and respond accordingly. The in-cabin system can activate cellular services and make phone calls via lip reading, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. The in-cabin system may also include one or more in-cabin AI agents or assistants that can interact with one or more LLMs, VLMs, MMLMs, etc., in the cloud using one or more APIs or plugins. For example, the in-cabin AI agents or assistants can provide directions, vehicle or machine feedback information, answer general questions, handle music / video and / or other requests, activate windows, doors, and / or other vehicle components, etc. Therefore, one or more dedicated SoCs and / or processor groups can be used to perform in-cabin infotainment and / or in-cabin monitoring (e.g., as an occupant monitoring system (OMS)) of the machine 700.

[0119] The device 700 may also include an instrument cluster 732 (e.g., a digital instrument cluster, electronic instrument cluster, digital dashboard, etc.). The instrument cluster 732 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 732 may include a set of instruments such as a speedometer, fuel level, fuel pressure, tachometer, odometer, turn signal indicator, gearshift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 730 and the instrument cluster 732. In other words, the instrument cluster 732 may be included as part of the infotainment SoC 730, and vice versa.

[0120] Figure 7D A computing system according to at least some embodiments of this disclosure (about Figure 7C A block diagram of an example architecture (a subset of the systems described). Although illustrated as SoC 704, this is not intended to be limiting, and the computing system may additionally or alternatively include multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiP), heterogeneous integration (HI), single-board computers (SBCs), and / or other components and / or architectures without departing from the scope of this disclosure.

[0121] The SoC 704 can be an end-to-end platform with a flexible architecture spanning automation levels 2-5, or it can be specifically designed for a particular automation level (e.g., a first SoC 704 for levels 2 to 2++, a second SoC 704 for level 3, a third SoC 704 for level 4, and so on), providing a comprehensive functional safety architecture that leverages and effectively utilizes computer vision, neural network inference, robot planning, control and navigation, ADAS technologies, and more, with diversity and redundancy, to provide a flexible and reliable platform for driving or robot control software stacks, as well as deep learning tools. The SoC 704 can be faster, more reliable, and even more energy-efficient and space-saving than conventional systems. For example, when the accelerator 714 is used in conjunction with the CPU 706, GPU 708, and data repository 716, it can provide a fast and efficient platform for Level 2-5 autonomous vehicles, as well as for the safety planning, navigation, and control of AMR 700B, humanoid robot 700C, and / or other robot or machine types.

[0122] In some embodiments, for example, SoC 704 includes: a GPU 708 having 2000 or more cores (e.g., 2048 cores), 60 or more tensor cores (e.g., 64 tensor cores), and a maximum GPU frequency exceeding 1 GHz (e.g., 1.3 GHz); a CPU 706 including 10 or more cores (e.g., 12 cores), having 64-bit, 3 MB L2 and 6 MB L3 caches, and a maximum frequency of 2 GHz or more (e.g., 2.2 GHz); one or more deep learning accelerators (DLAs); deep learning accelerator clusters (XNNs); neural network accelerators (NNAs) or neural processing units (NPUs) 709 (e.g., two DLAs / XNNs / NNAs / NPUs 709); and a vision accelerator (e.g., a programmable vision accelerator (PVA)) 707. A single SoC 704 may be capable of achieving AI performance of 275 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGXOrin 64GB... The SoC meets these standards and achieves this performance.

[0123] Similarly, in an embodiment, SoC 704 includes: a GPU 708 having 1700 or more cores (e.g., 1792 cores), 50 or more tensor cores (e.g., 56 tensor cores), and a maximum GPU frequency exceeding 900MHz (e.g., 930MHz); a CPU 706 including 8 or more cores (e.g., 8 cores), having 64-bit, 2MB L2 and 4MB L3 cache memory, and a maximum frequency of 2GHz or more (e.g., 2.2GHz); one or more deep learning accelerators (DLAs); a deep learning accelerator cluster (XNN); a neural network accelerator (NNA) or neural processing unit (NPU) 709 (e.g., 2 DLAs / XNNs / NNAs / NPUs 709); and a vision accelerator (e.g., a programmable vision accelerator (PVA) 707), and a single SoC 704 may be capable of achieving AI performance of 200 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGXOrin 32GB SoC meets these standards and achieves such performance.

[0124] In some embodiments, for example, SoC 704 includes: a GPU 708 having 1,000 or more cores (e.g., 1,024 cores), 28 or more tensor cores (e.g., 32 tensor cores) and a maximum GPU frequency exceeding 900 MHz (e.g., 1,173 MHz); a CPU 706 including 8 or more cores (e.g., 8 cores), having 64-bit, 2 MB L2 and 4 MB L3 cache memory, and a maximum frequency of 2 GHz or more (e.g., 2 GHz); one or more deep learning accelerators (DLAs); a deep learning accelerator cluster (XNN); a neural network accelerator (NNA) or neural processing unit (NPU) 709 (e.g., one DLA / XNN / NNA / NPU 709); and a vision accelerator (e.g., a programmable vision accelerator (PVA) 707), and a single SoC 704 may be capable of achieving AI performance of 157 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGX Orin NX 16GB SoC meets these standards and achieves such performance.

[0125] In various embodiments, for example, the SoC 704 includes: a GPU 708 having 1000 or more cores (e.g., 1024 cores), 28 or more tensor cores (e.g., 32 tensor cores), and a maximum GPU frequency exceeding 900MHz (e.g., 1020MHz); and a CPU 706 including 6 or more cores (e.g., 6 cores), having 64-bit, 1.5MB L2 and 4MB L3 cache memory, and a maximum frequency of 1.5GHz or higher (e.g., 1.7GHz). A single SoC 704 can achieve AI performance of 67 trillion operations per second (TOPS). For example, NVIDIA's Jetson Orin Nano 8GB SoC meets these standards and achieves such performance.

[0126] SoC 704 may include one or more CPUs 706. In embodiments, CPU 706 may include CPU clusters or CPU complexes (also referred to herein as “CCPLEX”). CPU 706 may include multiple cores and / or (e.g., L2, L3) caches. For example, in some embodiments, CPU 706 may include twelve cores in a coherent multiprocessor configuration. In some embodiments, CPU 706 may include four dual-core clusters, each with a dedicated L2 cache (e.g., 3MB L2 cache). CPU 706 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, thereby allowing any combination of clusters of CPU 706 to be active at any given time.

[0127] The SoC 704 may include any type and number of GPUs 708. For example, an integrated GPU (also referred to herein as an "iGPU") may be used in some embodiments. The GPU 708 may be programmable and can be used efficiently for parallel workloads. In some examples, the GPU 708 may use an enhanced tensor instruction set. The GPU 708 may include one or more streaming microprocessors, wherein each streaming microprocessor may include a cache (e.g., an L1 cache with at least 96KB of storage), and two or more streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage). In some embodiments, the GPU 708 may include at least eight streaming microprocessors. The GPU 708 may use a computing application programming interface (API). Furthermore, the GPU 708 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0128] The GPU 708 can be power-optimized for optimal performance in automotive, robotics, and / or other embedded use cases. For example, the GPU 708 can be fabricated on FinFETs. However, this is not intended to be limiting, and the GPU 708 can be fabricated using other semiconductor fabrication methods or processes. Each streaming microprocessor can include multiple mixed-precision processing cores divided into blocks. For example, but not limited to, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSOR COREs for deep learning matrix algorithms, one (e.g., L0) instruction cache, one warp scheduler, one dispatch unit, and / or one (e.g., 64KB) register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to efficiently perform workloads involving mixed-computation and addressing computations. Streaming microprocessors can include independent thread scheduling capabilities to enable finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors can also include combined L1 data caches and shared memory units to improve performance while simplifying programming.

[0129] The GPU 708 may include high-bandwidth memory (HBM) and / or (e.g., 16GB) HBM2 memory subsystems to provide peak memory bandwidth of approximately 900GB / s in some examples. In some examples, synchronous graphics random access memory (SGRAM), such as graphics double data rate type 5 synchronous random access memory (GDDR5), may be used in addition to or as an alternative to HBM memory.

[0130] The GPU 708 may include unified memory technology, including access counters, to allow more accurate migration of memory pages to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, Address Translation Service (ATS) support may be used to allow the GPU 708 to directly access the CPU 706 page tables. In such examples, when a miss occurs in the GPU 708 Memory Management Unit (MMU), an address translation request can be transferred to the CPU 706. In response, the CPU 706 can look up the virtual-to-physical mapping of the address in its page tables and transfer the translation back to the GPU 708. Therefore, unified memory technology allows the memory of the CPU 706 and GPU 708 to use a single, unified virtual address space, simplifying GPU 708 programming and porting applications to the GPU 708.

[0131] SoC 704 may include any number of caches 712, including the caches described herein. For example, cache 712 may include L0 cache, L1 cache, L2 cache, L3 cache (e.g., available for CPU 706 and GPU 708, e.g., connected to CPU 706 and GPU 708)), etc. Cache 712 may include write-back caches that can track the state of the lines, for example, by using one or more cache coherence protocols (e.g., MEI, MESI, MSI, etc.). According to embodiments, the cache (e.g., L3) may include 4MB or more, although smaller or larger cache sizes may be used.

[0132] SoC 704 may include one or more arithmetic logic units (ALUs) 765, which can be used to perform processing associated with various tasks or operations of machine 700, such as computer vision, machine learning or deep learning processing, world model management, etc. Additionally, SoC 704 may include a floating-point unit (FPU) 767 or other mathematical coprocessors or digital coprocessors for performing mathematical operations within the system. For example, SoC 104 may include one or more FPUs 767, which are integrated as execution units within CPU 706 and / or GPU 708.

[0133] SoC 704 may include one or more accelerators 714 (e.g., hardware accelerators, software accelerators, or combinations thereof). For example, SoC 704 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. Large on-chip memory 715 (e.g., 4MB SRAM, 32GB and / or 64GB 256-bit LPDDR5 (204.8GB / s), 8GB and / or 16GB 128-bit LPDDR5 (102.4GB / s), and / or other memory types and sizes) enables the hardware acceleration cluster to accelerate neural network processing, converter processing, optical flow processing, vision processing, and / or other computations or processing. The hardware acceleration cluster can be used to supplement GPU 708 and offload some tasks from GPU 708 (e.g., freeing up more cycles of GPU 708 to perform other tasks). As an example, the accelerator 714 can be used for target workloads (e.g., perceptual, convolutional neural network (CNN), deep neural network (DNN), language model (LLM, VLM, MMLM, VLA, etc.), transformer model, diffusion model, encoder-only model, encoder-decoder model, etc., which are stable enough to be suitable for acceleration.

[0134] Accelerator 714 (e.g., a hardware acceleration cluster) may include a deep learning accelerator (DLA) 709 (also referred to herein as a “deep learning accelerator cluster (XNN) 709”, a “neural network accelerator (NNA) 709”, or a “neural processing unit (NPU) 709”). DLA 709 may include one or more tensor processing units (TPUs) 741, which may be configured to provide additional, for example, trillions of operations per second for deep learning applications and inference. TPU 741 may be an accelerator configured to perform image processing functions (e.g., for CNNs, RCNNs, DNNs, etc.) and optimized for them. DLA 709 may be further optimized for a specific set of neural network types and floating-point operations and inference. The DLA is designed to provide higher performance per millimeter than general-purpose GPUs and significantly outperforms CPUs. TPU 741 can perform several functions, including single-instance convolution functions, support for INT8, INT16, and FP16 data types for features and weights, and post-processor functions. Although TPU 741 is described as being included as part of DLA 709, this is not intended to be limiting, and TPU 741 may be included in additional or alternative accelerator 714 and / or other components, and / or may be included as a discrete processing component.

[0135] The DLA 709 can quickly and efficiently execute neural networks on processed or unprocessed data to achieve a variety of functions, including but not limited to: object and feature recognition and detection using data from one or more sensor modalities (e.g., vehicles, pedestrians, other robots, lane lines, road boundary lines, debris, potholes, boxes, warehouse items, etc.); distance estimation using data from one or more sensor modalities; emergency vehicle detection and identification using data from microphones and / or vision-based sensors; facial recognition; pick-up and place operations; maneuvering operations; occupant monitoring; vehicle owner identification; and / or other in-cabin operations using data from in-cabin cameras and / or other sensor types; and / or safety and / or safety-related events, to name just a few.

[0136] The DLA 709 can perform any function of the GPU 708, and by using inference accelerators, for example, designers can anchor either the DLA 709 or the GPU 708 for any function. For instance, designers can centralize the processing of DNNs and floating-point operations on the DLA 709, leaving other functions to the GPU 708 and / or other accelerators 714. The DLA 709 can be used to run any type of network to enhance control and security, including neural networks that, for example, output a confidence metric for each object detection.

[0137] Accelerator 714 (e.g., a hardware acceleration cluster) may include a programmable vision accelerator (PVA) 707, which may alternatively be referred to herein as a computer vision accelerator or generally as a vision accelerator. The PVA 707 may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), semi-autonomous driving, autonomous driving, robotics applications, safety and supervision applications, augmented reality (AR), virtual reality (VR), and / or mixed reality (MR) applications. The PVA 707 provides a balance between performance and flexibility. For example, each PVA 707 may include (e.g., but not limited to) any number of Reduced Instruction Set Computer (RISC) cores, Direct Memory Access (DMA) systems, Pixel Processing Engines (PPEs), Vector Processors or Vector Processing Units (VPUs), and / or other components. The PVA engine may include an Advanced Very Long Instruction Word (VLIW) or Single Instruction Multiple Data (SIMD) digital signal processor. The PVA 707 may be optimized for image processing and computer vision algorithm acceleration tasks. For example, the PVA707 offers excellent performance with extremely low power consumption and can be used asynchronously and concurrently as part of a heterogeneous computing pipeline with CPU 706, GPU 708 and / or other accelerators in systems such as vehicles, robots and so on.

[0138] The PVA 707 may include one or more (e.g., two) Vector Processing Subsystems (VPSs), each of which may include one or more Vector Processing Unit (VPU) cores, one or more Decoupled Lookup Units (DLUTs), one or more shared or vector memory (VMEMs), and one or more instruction caches (I-caches). The VPU core may be the main processing unit and may include a vector SIMD VLIW DSP 743 optimized for computer vision. The VPU core can fetch instructions via the I-cache and access data via the VMEM. The DLUT may include dedicated hardware components that enhance the efficiency of parallel lookup operations. For example, the DLUT allows parallel lookups using a single copy of the lookup table by performing these lookups in a decoupled pipeline independent of the main processor pipeline. By doing so, the DLUT can minimize or reduce memory usage and increase throughput while avoiding memory library conflicts associated with data, ultimately leading to improved overall system performance. The VPU VMEM can provide local data storage for the VPU, allowing for the efficient implementation of various image processing and computer vision algorithms. The VPU VMEM can support access from external VPS hosts, such as Direct Memory Access (DMA) and the CPU 706 (e.g., an ARM Cortex-R5 processor), thereby facilitating data exchange with the CPU 706 and other system-level components. The VPU I-cache can provide instruction data to the VPU on request, request missing instruction data from system memory, and / or maintain temporary instruction storage for the VPU. For each VPU task, the CPU 706 can configure the DMA system, optionally prefetching the VPU program into the VPU I-cache, and / or initiating each VPU-DMA pair to process the task. The PVA 707 may also include L2 SRAM memory shared between one or more (e.g., two) sets of VPS and DMA. In some embodiments, one or more (e.g., two) DMA devices are used to move data between external memory, PVA L2 memory, VMEM (e.g., one per VPS), CPU tightly coupled memory (TCM), DMA descriptor memory, and / or PVA-level configuration registers. In lightly loaded systems, two parallel DMA accesses to DRAM can achieve read / write bandwidth of up to 15 GB / s, while in heavily loaded systems, this bandwidth can reach up to 10 GB / s. Regarding compute capacity, INT8 gigabyte multiply-accumulate operations per second (GMAC) can be 2048 or greater, excluding DLUT. FP32 GMACs can be 32 per PVA instance.

[0139] The RISC core can interact with image sensors (e.g., the image sensor of any camera described herein), image signal processors, etc. Each RISC core may include any amount of memory. The RISC core can use any of a variety of protocols, depending on the implementation. In some examples, the RISC core can execute a real-time operating system (RTOS). The RISC core can be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, the RISC core may include an instruction cache and / or tightly coupled RAM.

[0140] The DMA system enables PVA 707 components to access system memory independently of the CPU 706. DMA can support any number of functions optimized for the PVA 707, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which may include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0141] A vector processor, or VPU, can be a programmable processor designed to efficiently and flexibly execute computer vision algorithms and provide signal processing capabilities. In some examples, the PVA 707 may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA 707 and may include one or more vector processing units (VPUs), one or more pixel processing engines (PPEs) (which may include a 2D layout of interconnected (e.g., for north, south, east, and west communication) processing elements), one or more instruction caches, and / or one or more shared or vector memories (e.g., VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can improve throughput and speed.

[0142] In some embodiments, each vector processor may include an instruction cache and may be coupled to dedicated memory. Therefore, in some examples, each vector processor may be configured to execute independently of other vector processors. In other examples, the vector processors included in a particular PVA 707 may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA 707 may execute the same computer vision algorithm, but for different regions of an image. In other examples, the vector processors included in a particular PVA 707 may execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on consecutive images or portions of an image. Among other things, the hardware acceleration cluster may include any number of PVA 707s, and each PVA may include any number of vector processors. Furthermore, the PVA 707 may include additional error correction code (ECC) memory to enhance overall system security.

[0143] Accelerators 714 (e.g., hardware accelerator clusters) have broad applications for autonomous and semi-autonomous machine control. PVA 707 can be a programmable vision accelerator used in critical processing stages in perception, robot understanding and reasoning, ADAS, semi-autonomous and autonomous vehicles, etc. The capabilities of PVA 707 are well-suited for algorithmic domains requiring predictable processing, featuring low power consumption and low latency. In other words, PVA 707 performs well in semi-intensive or intensive rule computation, even on small datasets requiring predictable runtime, low latency, and low power consumption. Therefore, in the context of autonomous vehicles and robotic platforms, PVA 707 is designed to run classic computer vision algorithms, as they are highly efficient in object detection and integer mathematical operations.

[0144] For example, according to one embodiment of this technology, the PVA 707 is used to perform computer stereo vision. In some examples, algorithms based on semi-global matching can be used, although this is not intended to be limiting. Many Level 3-5 autonomous driving applications require on-the-fly motion estimation / stereo matching (e.g., moving structures, pedestrian recognition, lane detection, etc.). The PVA 707 can perform computer stereo vision functions on input from two monocular cameras.

[0145] In some examples, the PVA 707 can be used to perform dense optical flow, providing processed RADAR data based on the raw RADAR data (e.g., using 4D Fast Fourier Transform). In other examples, the PVA 707 is used for time-of-flight depth processing, for example, providing processed time-of-flight data by processing the raw time-of-flight data.

[0146] While the VPU, DMA, RISC Core, VMEM, and decoupled coprocessors (e.g., DLUT) are described as being included within the PVA707, this does not imply limitation. In some embodiments, these components may be included in alternative or additional processing components and / or accelerators 714, and / or may be included as discrete components of the SoC 704 and / or other computing system architectures.

[0147] In some examples, SoC 704 may include a real-time ray tracing hardware accelerator (RTA) 751, which can be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) to generate real-time or near-real-time visualization simulations, for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for simulation of SONAR, RADAR, LiDAR, camera and / or other sensor modalities in a simulation, for general wave propagation simulation, for comparison with LiDAR data for localization, for generating realistic training data for training neural networks, and / or other functions and uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations. For example, machine 700 (or another machine or device) may perform simulations in a simulation environment, and one or more light transport simulation algorithms (e.g., ray tracing, path tracing, etc.) may be used to generate the simulation environment. Therefore, the ray tracing accelerator 751 and / or a ray tracing-optimized GPU 706 (e.g., NVIDIA's RTX GPU) may be used to accelerate these ray tracing algorithms.

[0148] Accelerator 714 (e.g., in a hardware acceleration cluster) may include one or more optical flow accelerators (OFAs) 711. For example, OFA 711 can be used to calculate optical flow and stereo disparity between sensor data frames (e.g., images). Optical flow can be accelerated on OFA 711 for uses such as object detection and tracking, and / or for stereo depth estimation, where stereo disparity is calculated between stereo image frames (e.g., two or more frames captured using two or more image sensors with at least partially overlapping fields of view).

[0149] SoC 704 may include one or more Camera Serial Interfaces (CSI) 723s. ​​For example, CSI 723 may include a Mobile Industry Processor Interface (MIPI) Camera Serial Interface (CSI) for receiving video and input from cameras, a high-speed interface, and / or a video input block that can be used for camera and associated pixel input functions. SoC 704 may also include a software-controllable input / output controller and may be used to receive I / O signals not assigned to a specific role. For example, CSI 723 may include MIPI CSI-2 connectors, such as a 16-channel MIPI CSI-2 connector, D-PHY 2.1 (up to 40Gbps), and C-PHY 2.0 (up to 164Gbps) to support 16 virtual channels and 6 or more cameras; an 8-channel MIPI CSI-2 connector, D-PHY 2.1 (up to 20Gbps) to support 8 virtual channels and 4 or more cameras; and / or 2x MIPI CSI-2, 22-pin camera connectors, depending on the embodiment and implementation.

[0150] Accelerator 714 (e.g., a hardware acceleration cluster) may include an on-chip computer vision network (CVNOC) 763 and SRAM for providing high-bandwidth, low-latency SRAM to accelerator 714. In some examples, on-chip memory may include at least 4 MB of SRAM, such as, but not limited to, eight field-configurable memory blocks accessible by PVA 707, OFA 711, DLA 709, and / or other accelerator 714. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory 715 may be used. PVA 707, OFA 711, DLA 709, and / or other accelerator 714 may access memory via a backbone providing high-speed memory access to accelerator 714. The backbone may include an on-chip computer vision network that interconnects accelerator 714 to memory (e.g., using APB).

[0151] The CGNOC 763 may include an interface that determines whether the accelerator 714 provides a ready and valid signal before transmitting any control signals / addresses / data. Such an interface can provide separate stages and separate channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, although other standards and protocols can be used.

[0152] SoC 704 may include a data repository 716 and / or memory 715. The data repository 716 may be on-chip memory 715 of the SoC 704, which may store neural networks and / or other algorithms to be executed on the CPU 706, GPU 708, and / or one or more accelerators 714. In some examples, the capacity of the data repository 716 may be large enough to store multiple neural network instances for redundancy and security. The data repository 716 may include, for example, L2 and / or L3 cache 712. The memory 715 may include SRAM, LPDDR5, and / or other memory types. For example, the memory 715 may include 4MB SRAM, 32GB and / or 64GB 256-bit LPDDR5 (204.8GB / s), 8GB and / or 16GB 128-bit LPDDR5 (102.4GB / s), and / or other memory types and sizes. References to the data repository 716 may include references to the memory associated with the PVA 707, OFA 711, DLA 709 and / or other accelerators 714, as described herein.

[0153] Data repository 116 may include various storage types, such as eMMC, NVMe, etc. For example, SoC 704 may include storage in the form of an embedded multimedia card (eMMC) (e.g., 64GB eMMC 5.1) and / or an SD card slot, with external NVMExpress (NVMe) capability, for example, via M.2 Key M. For example, data repository 716 and / or other storage may be accessed via, for example, NVMe using PCI Express (PCIe), RDMA, TCP, and / or other protocols.

[0154] SoC 704 may include one or more processors 710 (e.g., embedded processors). Processor 710 may include a boot and power management processor (BPMP) 753, which may be a dedicated processor and subsystem for handling boot power and management functions, as well as associated safety implementations. BPMP 753 may be part of the SoC 704 boot sequence and may provide runtime power management services. BPMP 753 may provide clock and voltage programming, assistance with system low-power state transitions, management of SoC 704 thermal and temperature sensors, and / or management of SoC 704 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and SoC 704 may use the ring oscillator to detect the temperature of CPU 706, GPU 708, accelerator 714, and / or other components. If the temperature is determined to exceed a threshold, BPMP 753 may enter a temperature fault routine and place SoC 704 into a lower power state and / or place machine 700 into a driver safety stop mode (e.g., safely stop machine 700).

[0155] The processor 710 may also include a set of embedded processors that can be used as the Audio Processing Engine (APE) 755. The APE 755 can be an audio subsystem capable of providing full hardware support for multi-channel audio through multiple interfaces, as well as a wide and flexible audio I / O interface. In some examples, the APE 755 is a dedicated processor core with a digital signal processor and dedicated RAM.

[0156] The processor 710 may also include an Always-On Processor Engine (AOPE) 757, which provides the necessary hardware features to support low-power sensor management and wake-up use cases. AOPE 757 may include a processor core, tightly coupled RAM, peripheral support (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0157] Processor 710 may also include a security processor 713 (or "security island 713"), which may include a security cluster engine comprising a dedicated processor or processor subsystem for handling security management for automotive, robotic, and / or other applications. Security processor 713 and / or the security cluster engine may include two or more processor cores, tightly coupled RAM, support for peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In security mode, the two or more cores may operate in lockstep mode and as a single core, with comparison logic to detect any differences in their operation. In some embodiments, security processor 713 may include discrete processors such that failure of other system components may not affect the performance and availability of security processor 713.

[0158] The processor 710 may also include a real-time or near-real-time sensor engine (SE) 759, which may include a dedicated processor subsystem for handling real-time or near-real-time camera, LiDAR, RADAR and / or other sensor modal management.

[0159] The processor 710 may also include one or more image signal processors (ISPs) 727, which may include high dynamic range signal processors and / or hardware engines as part of one or more sensor processing pipelines.

[0160] The processor 710 may include a video image synthesizer (VIC) 761, which may be a processing block (e.g., implemented on a microprocessor) that implements the video post-processing functions required by the video playback application to produce the final image of the player window. The VIC 761 can perform lens distortion correction on the wide-angle camera 768B, the surround camera 768D, the cabin monitoring camera sensor, and / or other camera sensors with distorted fields of view.

[0161] VIC 761 may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, when motion occurs in the video, noise reduction appropriately weights spatial information, thereby reducing the weight of information provided by adjacent frames. When an image or part of an image does not contain motion, temporal noise reduction performed by the video image synthesizer can use information from previous images to reduce noise in the current image.

[0162] The VIC 761 can also be configured to perform stereoscopic correction on input stereoscopic camera frames. The video image compositor can also be used for user interface compositing when the operating system desktop is in use and the GPU 708 does not need to continuously render new surfaces. Even when the GPU 708 is powered on and actively performing 3D rendering, the video image compositor can be used to offload the GPU 708 to improve performance and responsiveness.

[0163] SoC 704 may also include various peripheral interfaces for input / output (I / O) 725, for example, to enable communication with peripheral devices, audio codecs, power management and / or other devices. SoC 704 can be used to process data from cameras (e.g., via a gigabit multimedia serial link and / or Ethernet connection), from sensors (e.g., LiDAR sensor 764, RADAR sensor 760, etc., connected via Ethernet), from bus 702 (e.g., speed of machine 700, steering wheel position, etc.), and from GNSS sensor 758 (e.g., connected via Ethernet or CAN bus). SoC 704 may also include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine and can be used to free CPU 706 from routine data management tasks. In some embodiments, the SoC 704 I / O 725 may include headers (e.g., 40-pin headers or 40-pin extension headers) supporting Universal Asynchronous Receiver / Transmitter (UART), Serial Peripheral Interface (SPI), Inter-Integrated Sound (I2S), Inter-Integrated Sound (I2C), Controller Area Network (CAN), Pulse Width Modulation (PWM), Digital Microphone Interface (DMIC), Digital Speaker Station (DSPK), General Purpose I / O (GPIO), etc.; automation headers (e.g., 12-pin automation headers); audio panel headers (e.g., 10-pin audio panel headers); Joint Test Action Group (JTAG) headers (e.g., 10-pin JTAG headers); fan headers (e.g., 4-pin fan headers); RTC battery spare connectors (e.g., 2-pin battery spare connectors); a microSD slot; a DC power jack; power, force, resume, and reset buttons; and one or more display connectors (e.g., DisplayPort (DP), such as DP 1.4A (+MST), eDP 1.41, HDMI). 2.1 and / or 4K30 multi-model DP1.2 (+MST) connectors) and / or other I / O 725 elements, components or features.

[0164] The SoC 704 may include machine intranetting capabilities using, for example, Ethernet (e.g., automotive Ethernet), SERDES, Controller Area Network (CAN), FlexRay, Local Interconnect Network (LIN), Low Voltage Differential Signaling (LVDS), Media-Oriented System Transport (MOST), another network type, and / or combinations thereof. For example, the SoC 704 may include RJ45 connectors with up to 10GbE, 1GbE connectors, and / or other network connector types.

[0165] SoC 104 may include one or more digital signal processors (DSPs) 743. For example, DSP 743 may include a dedicated or specialized microprocessor chip optimized for digital signal processing, such as audio signal processing, telecommunications, digital image processing, RADAR, SONAR, LiDAR and / or other sensor processing, speech recognition and / or other applications.

[0166] SoC 704 may include one or more video encoders 719 and / or one or more video decoders 721. For example, video encoder 719 may include a hardware-based (e.g., as part of GPU 708) video encoder (e.g., supporting H.264, H.265, etc., and conforming to HEVC standards, such as NVIDIA's NVENC) that can process image input (e.g., as YUV, RGB, etc.) to generate a video bitstream. Video decoder 721 may include a video decoder engine that can provide fully accelerated hardware video decoding capabilities (e.g., supporting decoding of various bitstream formats such as AV1, H.264, H.265, VP8, VP9, ​​MPEG-1, MPEG-2, MPEG-4, VC-1, etc., and conforming to HEVC standards, such as NVIDIA's NVDEC). In some examples, video decoder 721 may be hardware-based (e.g., as part of GPU 708).

[0167] SoC 704 may include one or more General Purpose Computing Acceleration Clusters (GCACs) 729. For example, GCACs 729 may include various processor types that can be used to accelerate computing, such as one or more Vector Microcode Processors (VMPs) 733, one or more Multi-Threaded Processing Clusters (MPCs) 731, one or more Programmable Macroarrays (PMAs) 735, and / or one or more other processor types. For example, GCAC 729 may include a PMA 735, two VMPs 733, and two MPCs 731.

[0168] SoC 704 may include one or more vector microcode processors (VMPs) 733. In embodiments, VMP 733 may include wide vector (Very Long Instruction Word (VLIW) and Single Instruction Multiple Data (SIMD)) machines that perform various operations, such as short integral-type operations common in computer vision and deep learning algorithms.

[0169] SoC 704 may include one or more multi-threaded processing clusters (MPCs) 731. MPCs 731 may include processing clusters that are more general-purpose than GPUs and more efficient than CPUs in some embodiments. For example, an MPC 731 may include a multi-threaded processor that allows multiple threads to share resources and execute instructions concurrently.

[0170] The SoC 704 may include one or more programmable macro arrays (PMAs) 735. The PMA 735 may include a coarse-grained reconfigurable architecture (CGRA) dataflow machine, which has a unique architecture that delivers powerful performance on intensive computer vision and deep learning algorithms that may not be achievable in classic digital signal processing (DSP) architectures.

[0171] The SoC 704 may include one or more Display Processing Units (DPUs) 745 for performing hardware-accelerated image processing. For example, the DPU 745 may retrieve pixel data from memory 715 and send it to a display peripheral device via a standard interface. Therefore, the DPU 745 can handle display processing and rendering for displays within and / or on the machine.

[0172] The SoC 704 may include one or more Application Processing Units (APUs) 739. For example, an APU 739 may include a quad-core or dual-core processor with 48KB / 32KB L1 cache (with parity and ECC) and a 1MB L2 cache with ECC. The APU 739 may support NEON instructions and single-precision and double-precision floating-point operations.

[0173] The SoC 704 may include one or more Real-Time Processing Units (RTPUs) 769. The RTPU 769 may include a dual-core processor with 32KB / 32KB L1 cache and a 256KB TCM with ECC. The RTPU 769 can support single-precision and double-precision floating-point operations.

[0174] SoC 704 may include one or more built-in self-test (BIST) components 737. For example, BIST component 737 may include a memory BIST (MBIST) for testing the system's memory and / or a logic BIST (LBIST) for testing the system's logic. BIST component 737 may include embedded logic for directly testing the system's logic and / or memory.

[0175] SoC 704 may include one or more dynamically reconfigurable processors (DRPs) 771. For example, DRP 771 can be used to accelerate various computational operations. For instance, in one embodiment, DRP 771 may be combined with a MAC unit to function as an AI accelerator. In another embodiment, DRP 771 can execute an application while dynamically switching the circuit connection configuration of the arithmetic unit (e.g., ALU) on the chip at each operating clock cycle based on what needs to be processed. By using only the necessary arithmetic circuitry, DRP 771 can consume less power than a CPU and achieve higher speeds. Furthermore, compared to a CPU, which suffers from performance degradation due to frequent accesses to external memory for cache misses and other reasons, DRP 771 can pre-build the necessary data paths in the hardware, thereby reducing performance degradation and operating speed variations (jitter) caused by memory accesses. DRP 771 may include a dynamic loading function that switches circuit connection information each time the algorithm changes, enabling processing with limited hardware resources, even in robotics / automotive applications that require processing multiple algorithms.

[0176] In some embodiments, accelerator 714 may include an OpenCV accelerator for accelerating OpenCV processing, OpenCV being an open-source industry-standard library for computer vision processing. In some embodiments, the combination of one or more DRP 771s deployed as AI accelerators with an OpenCV accelerator can enhance AI computation and image processing algorithms to enable complex and computationally intensive operations such as visual simultaneous localization and mapping (SLAM).

[0177] Compared to conventional systems, the techniques described herein, by providing CPU complexes, GPU complexes, and hardware acceleration clusters, allow multiple neural networks to be executed simultaneously (e.g., at least partially in parallel) and / or sequentially, and the results combined to achieve Level 2-5 autonomous driving capabilities and / or autonomous robot motion, control, planning, and / or navigation operations. Furthermore, since the SoC 704 can include various computing engines (e.g., processor 710, CPU 706, GPU 708, accelerator 714, etc.), tasks can be distributed among the computing engines, and in some cases, common-cause failures are avoided due to the discrete footprint of the computing engines. Additionally, since the SoC 704 can include a dedicated safety processor 713 (or safety island 713), critical safety or redundant operations can be performed without common-cause failures of the SoC 714's main processing components or computing engines. Due to these features, the underlying system of the SoC 704 and / or machine 700 may be able to meet higher safety levels—such as the Automotive Safety Integrity Level (ASIL) D of the ISO 26262 standard.

[0178] Figure 7E According to some embodiments of this disclosure, cloud-based servers (e.g., servers such as those described herein in a data center) and Figure 7A The following is a system diagram illustrating communication between an example autonomous or semi-autonomous vehicle or machine 700. System 776 may include server 778, network 790, and machine 700. Server 778 may include multiple GPUs 784(A)-784(H) (collectively referred to herein as GPU 784), switches 782(A)-782(H) (e.g., PCIe 4.0 / 5.0 switches, M.2 slots, Thunderbolt, USB4, NVIDIA's NVLink, NVIDIA's NVSwitch, GPUDirect RDMA, GPUDirect Storage, etc.), CPUs 780(A)-780(B) (collectively referred to herein as CPU 780), accelerators, and / or other processor types. GPU 784, CPU 780, and PCIe switches may interconnect with high-speed interconnects, such as, but not limited to, NVIDIA-developed NVLink interface 788 and / or PCIe connection 786. In some examples, the GPU 784 is connected via NVLink and / or NVSwitch SoC, and the GPU 784 and PCIe switch 782 are connected via PCIe interconnect. While the figure shows eight GPUs 784, two CPUs 780, and two PCIe switches, this is not limiting. According to embodiments, each server 778 may include any number of GPUs 784, CPUs 780, and / or PCIe switches. For example, each server 778 may include eight, sixteen, thirty-two, and / or more GPUs 784.

[0179] Server 778 may receive sensor data from network 790 and machines 700 that indicates information about new or previously unexplored locations, and / or sensor data indicating changes to previously seen / stored locations (e.g., unexpected or changed road conditions, such as recently started road construction). Server 778 may transmit neural network 792, updated neural network 792, map information 794, etc., including information about traffic and road conditions, to machines 700 via network 790. Updates to map information 794 may include updates to HD maps 722, SD maps, navigation maps, etc., such as information about construction sites, potholes, detours, floods, and / or other obstacles. In some examples, neural network 792, updated neural network 792, map information 794, and / or other information may come from new training and / or experience, represented in data received from any number of machines 700 in the environment, and / or based on training performed in a data center (e.g., using server 778 and / or other servers).

[0180] Server 778 can be used to train machine learning models (e.g., neural networks) based on training data. Training data can be generated by machine 700 and / or generated in simulations (e.g., using a game engine). In some examples, the training data is labeled (e.g., neural networks benefit from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is unlabeled and / or preprocessed (e.g., neural networks do not require supervised learning). Training can be performed according to any one or more machine learning techniques, including but not limited to the following categories: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component analysis and clustering analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, it can be used by machine 700 (e.g., transmitted to machine 700 via network 790), and / or the machine learning model can be used by server 778 to remotely monitor and / or control machine 700.

[0181] In some examples, server 778 can receive data from machine 700 and apply the data to state-of-the-art real-time neural networks for real-time intelligent inference. Server 778 may include a deep learning supercomputer powered by GPU 784 and / or a dedicated AI computer, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, server 778 may include a deep learning infrastructure in a data center using only CPU power.

[0182] The deep learning infrastructure of server 778 may be capable of rapid, real-time inference and can use this capability to assess and verify the health of the processor, software, and / or related hardware in machine 700. For example, the deep learning infrastructure may receive periodic updates from machine 700, such as image sequences and / or objects located by machine 700 in the image sequence (e.g., through computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them with the objects identified by machine 700. If the results do not match and the infrastructure concludes that the AI ​​in machine 700 has malfunctioned, server 778 can send a signal to machine 700 instructing the fail-safe computer of machine 700 to take over control, notify the occupants, and perform safety maneuvers or operations, such as slowing down, returning control to the driver, stopping, and / or pulling over / closing the vehicle.

[0183] For inference, server 778 may include GPU 784 and one or more programmable inference accelerators (such as NVIDIA's TensorRT). The combination of GPU-driven servers and inference acceleration enables real-time response. In other examples, such as in scenarios where performance is less critical, servers driven by CPUs, FPGAs, and other processors can be used for inference.

[0184] Computational ecosystem for generating, training, and deploying AI Figure 8 This is a system diagram illustrating a three-computer ecosystem 800 according to at least some embodiments of the present disclosure, including a first computing system 802 for generating or creating artificial intelligence (AI) (e.g., AI training and validation data), a second computing system 804 for training the AI, and a third computing system 806 for deploying AI at the edge (which may include or correspond to...). Figures 7A-7E (SoC 704). For example, to develop and deploy materialized or physical AI, a three-computer ecosystem 800 can be used, comprising three accelerated computer systems to handle physical AI training, simulation, and runtime (e.g., edge deployment). These systems can use scalable, physical-based simulations of the Machine 700 and its world to generate training data and train multimodal base models (and / or other model types). By doing so, simulations of the Machine 700 can be performed at scale, allowing skills (e.g., robotic skills) to be improved, tested, and optimized in virtual worlds that simulate the laws of physics (e.g., using NVIDIA's OMNIVERSE), thereby helping to reduce the cost of real-world data acquisition and ensuring that the Machine 700 can operate safely in a controlled environment.

[0185] Computing system 804 (e.g., NVIDIA's DGX platform) can be used to train and fine-tune powerful foundational and generative AI models. Models such as general foundational models (e.g., NVIDIA's Project GR00T) can be used to enable robots and other machines 700 to understand natural language and mimic actions by observing human movements. Computing system 804 may include a platform that integrates software, infrastructure, and expertise into a modern, unified AI development and training solution. Computing system 804 may include individual computing devices 810 (e.g., NVIDIA's DGX B200, H200, etc.) and / or any number of computing devices 810 (e.g., NVIDIA's DGX SuperPOD) within data center infrastructure 812.

[0186] For example, a standalone computing device 810 may include GPUs (e.g., 8 GPUs with a total GPU memory of 1,440 GB) and CPUs (e.g., 2 CPUs with a total of 112 cores, 2.1 GHz or 4 GHz (with enhancements)), providing up to 72 petaFLOPS of training capability and 144 petaFLOPS of inference capability. The computing device 810 may include memory (e.g., 4 TB of memory) and storage (e.g., 2 x 1.9 TB NVMe M.2 OS storage and 8 x 3.84 TB NVMe U.2 internal storage). The computing device 810 may include various networking and network management components, such as OSFP ports (e.g., 4 OSFP ports) for servicing single-port intelligent host channel adapters (e.g., 8 single-port ConnextX-7 Virtual Protocol Interconnect (VPI)), providing up to 400 GB / s of Infiniband / Ethernet. The computing device 810 may also include, for example, a dual-port quad small pluggable (QSFFP) data processing unit (DPU) (e.g., two dual-port QSFP112 DPUs – such as NVIDIA’s BlueField-3DPU) providing up to 400Gb / s of InfiniBand / Ethernet. The computing device 810 may include an onboard network interface card (NIC) (e.g., a 10Gb / s onboard NIC with RJ45), a dual-port Ethernet NIC (e.g., a 100GB / s dual-port Ethernet NIC), and / or a host board management controller (MBC) (e.g., with RJ45). In some embodiments, the NIC for the computing device 810 may include a SuperNIC (e.g., NVIDIA’s ConnectX-8 SuperNIC) to provide up to 800Gb / s of data throughput for in-network computing acceleration engines, thereby providing the performance and robust feature set required to support trillion-parameter-scale AI factories and scientific computing workloads. In other embodiments, computing device 810 may include a smart host channel adapter (HCA) (e.g., NVIDIA's ConnectX-7) to provide ultra-low latency and 400 Gb / s throughput for computing acceleration engines within the network.

[0187] Data center infrastructure 812 may include: any number of computing devices 810, and an operating system (OS) (e.g., a DGX OS extension of a Linux distribution) for maximizing system uptime, security, and reliability; network / storage acceleration libraries and management for accelerating end-to-end infrastructure performance; cluster management for scaling and managing a single node (e.g., a computing device 810) to thousands of nodes; job scheduling and orchestration to ensure the smooth execution of each developer's jobs; AI workflow management and machine learning operations (MLOps) for moving more models from prototypes to production; and enterprise software to accelerate developer success.

[0188] Computational systems 802 (e.g., NVIDIA's OVX servers) can provide development and simulation platforms for testing and optimizing physics-based AI using APIs and frameworks for simulation (e.g., NVIDIA's DriveSIM, ISAAC Sim, ISAAC Gym, ISAAC Lab, etc.). Computational systems 802 allow developers to use simulation frameworks to simulate and validate robot models, and / or generate large amounts of physics-based synthetic data to guide model training. Computational systems 802 can support learning frameworks that power robot reinforcement learning and imitation learning to accelerate robot policy training and refinement. For example, computational systems 802 can be used to generate any number of simulations 808, such as in NVIDIA's OMNIVERSE. Computational systems 802 can be used to optimize and accelerate the entire software stack, from training, fine-tuning, and deploying generative AI to supporting industrial digitization within content collaboration platforms (including APIs, software development kits (SDKs), and services). These platforms allow the integration of OpenUSD, ray tracing rendering technologies (e.g., NVIDIA's RTX), and generative physics-based AI into existing software tools and simulation workflows, such as for industrial and robotics use cases (e.g., NVIDIA's OMNIVERSE). Therefore, computing system 802 can host or support the native OpenUSD software platform, enabling enterprises to connect 3D pipelines and develop advanced real-time 3D applications for industrial digitalization. With powerful ray tracing-accelerated AI and graphics capabilities, computing system 802 delivers robust performance for workloads such as extended reality (XR), multi-user design collaboration, and digital twins. This allows for the creation of physically accurate models with high-fidelity ray tracing and path tracing material rendering, large-scale, AI-enabled simulation operations, and the generation of realistic 3D synthetic data for training. Computing system 802 can include a single computing device 814 (e.g., an NVIDIA OVX L40S server) and / or any number of computing devices 814 (e.g., NVIDIA OVX systems) within data center infrastructure 816.

[0189] Computing device 814 (which may include a server) may include CPUs (e.g., 2 CPUs, each with 32 cores) and GPUs (e.g., 4 or 8 GPUs, each including 48GB GDDR6 with ECC memory, 864GB / s memory bandwidth, a PCIe Gen4x16: 64GB / s bidirectional interconnect interface, 18,176 CUDA cores, 142 ray tracing (RT) cores, and 568 Tensor cores). Computing device 814 may include various networking and network management components, such as Intelligent Host Channel Adapters (HCAs) (e.g., 2 or 4 single-port ConnextX-7, 200Gb / s per port, providing up to 800Gb / s InfiniBand / Ethernet), one or more DPUs (e.g., dual-port QSFP112 DPUs, such as the NVIDIA BlueField-3 DPU), providing up to 400Gb / s InfiniBand / Ethernet. In some embodiments, the NIC for computing device 814 may include a SuperNIC (e.g., NVIDIA's ConnectX-8 SuperNIC) to provide up to 800Gb / s of data throughput for in-network compute acceleration engines, thereby providing the performance and robust feature set required to support trillion-parameter-scale AI factories and scientific computing workloads. In other embodiments, computing device 814 may include a Smart Host Channel Adapter (HCA) (e.g., NVIDIA's ConnectX-7) to provide ultra-low latency and 400Gb / s throughput for in-network compute acceleration engines. Computing device 814 may include host memory (e.g., 384Gb DDR5 ECC for four GPUs, or 768Gb DDR5 ECC for eight GPUs) and may include dual in-line memory module (DIMM) slots, host boot drives (e.g., 1TB NVMe), and / or host storage (e.g., 24TB NVMe).

[0190] Similar to data center infrastructure 812, data center infrastructure 816 allows any number of computing devices 814 to be combined into a cluster configuration based on a reference architecture.

[0191] Computing system 806 can be used to deploy trained AI models on runtime computers, such as the SoC 704 described herein. For example, these computing systems 806 can be designed for compact onboard computing needs, including ensembles of models such as control policies, vision, and language models deployed on an energy-efficient onboard edge computing system 806. (See also: [link to document]). Figures 7A to 7E A more detailed description of the components, features, and capabilities of the computing system 806.

[0192] Example Generative Model In at least some embodiments, language models such as Large Language Models (LLM), Visual Language Models (VLM), Multimodal Language Models (MMLM), Visual Language Action (VLA) models, and / or other types of generative artificial intelligence (AI) can be implemented. These models may be able to understand, summarize, translate, and / or otherwise generate text (e.g., natural language text, code, etc.), images, videos, computer-aided design (CAD) assets, OMNIVERSE and / or METAVERSE file information (e.g., USD formats such as OpenUSD), and / or the like, based on context provided in input prompts or queries. In embodiments, these language models may be considered "large" because they are trained on massive datasets and have architectures with a large number of learnable network parameters (weights and biases)—e.g., millions or billions of parameters. This disclosure allows for the implementation of LLM / VLM / MMLM, etc., for summarizing textual data, analyzing data (e.g., text, images, videos, etc.), extracting insights from data (e.g., text, images, videos, etc.), and generating new text / images / videos / , etc., in a user-specified style, tone, and / or format. In some embodiments, the LLM / VLM / MMLM, etc., of this disclosure may be specifically designed for text processing, while in other embodiments, multimodal LLMs may be implemented to accept, understand, and / or generate text and / or other types of content, such as images, audio (sound, synthesized speech, etc.), 2D and / or 3D data (e.g., USD format), and / or video. For example, a visual language model (VLM) or more specifically a multimodal language model (MMLM) may be implemented to accept images, videos, sensor data, audio, text, 3D designs (e.g., CAD), and / or other input data types and / or generate or output images, videos, audio, text, 3D designs, and / or other output data types.

[0193] Various types of LLM / VLM / MMLM / etc. architectures can be implemented in various embodiments. For example, different architectures can be implemented using different techniques to understand and generate outputs (e.g., text, audio, video, images, 2D and / or 3D design or asset data, etc.). In some embodiments, LLM / VLM / MMLM / etc. architectures (e.g., recurrent neural networks (RNNs) or long short-term memory networks (LSTMs)) can be used, while in other embodiments, converter architectures (e.g., architectures relying on self-attention and / or cross-attention (e.g., between contextual data and textual data) mechanisms) can be used to understand and recognize relationships between words or tokens and / or contextual data (e.g., other text, video, images, design data, USD, etc.). One or more generative processing pipelines including LLM / VLM / MMLM / etc. may also include one or more diffusion blocks (e.g., noise reduction blocks). The LLM / VLM / MMLM / etc. of this disclosure may include encoder and / or decoder blocks. For example, discriminative or encoder-only models (e.g., BERT (Bidirectional Encoder Representations from Transformers)) can be implemented for tasks involving language understanding (e.g., classification, sentiment analysis, question answering, and named entity recognition). As another example, generative or decoder-only models (e.g., GPT (Generative Pretrained Transformer)) can be implemented for tasks involving language and content generation (e.g., text completion, story generation, and dialogue generation). LLM / VLM / MMLM / etc., including encoder and decoder components (e.g., T5 (Text-to-Text Transformer)), can be implemented to understand and generate content, such as for translation and summarization. These examples are not intended to be limiting and any architecture type (including, but not limited to, those described herein) can be implemented depending on the specific implementation and the task performed using LLM / VLM / MMLM / etc.

[0194] In various embodiments, LLM / VLM / MMLM / etc. can be trained using unsupervised learning, where LLM / VLM / MMLM / etc. learns patterns from a large amount of unlabeled text / audio / video / image / design / USD / etc. data. Due to the extensive training, in these embodiments, the model may not require task-specific or domain-specific training. An LLM / VLM / MMLM / etc. extensively pre-trained on a large amount of unlabeled data can be referred to as a base model and can excel at various tasks, such as question answering, summarizing, filling in missing information, translation, and image / video / design / USD / data generation. Some LLM / VLM / MMLM / etc. can be customized for specific use cases using techniques such as cue tuning, fine-tuning, retrieval augmentation generation (RAG), adding adapters (e.g., custom neural networks and / or neural network layers to tune or adjust cues or labels to bias the language model towards a specific task or domain), and / or using optimization models for specific tasks and / or other fine-tuning or customization techniques for specific domains.

[0195] In some embodiments, the LLM / VLM / MMLM / etc. disclosed herein can be implemented using various model alignment techniques. For example, in some embodiments, guardrails can be implemented to identify incorrect or unwanted inputs (e.g., prompts) and / or outputs of the model. In this process, the system can use guardrails and / or other model alignment techniques to prevent the processing of specific unwanted inputs using LLM / VLM / MMLM / etc., and / or to prevent the output or presentation of information generated by LLM / VLM / MMLM / etc. (e.g., displays, audio outputs, etc.). In some embodiments, one or more additional models (or layers thereof) can be implemented to identify problems with the model's inputs and / or outputs. For example, these "protective" models can be trained to identify "safe" or otherwise okay or desired inputs and / or outputs and / or "unsafe" or otherwise unwanted inputs and / or outputs for a particular application / implementation. Therefore, the LLM / VLM / MMLM / etc. disclosed herein are unlikely to output language / text / audio / video / design data / USD data / etc. that may be offensive, vulgar, inappropriate, insecure, out of scope, and / or unwanted for a particular application / implementation.

[0196] In some embodiments, an LLM / VLM / etc. can be configured or able to access or use one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc. For example, for certain tasks or operations where the model is not ideally suited, the model may have instructions for accessing one or more plugins (e.g., third-party plugins) to help process the current input (e.g., as a result of training, and / or based on instructions in a given prompt). In such an example, when at least part of the prompt relates to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs) to retrieve relevant information. Another example is that if at least part of the response requires mathematical computation, the model can access one or more mathematical plugins or APIs to help solve the problem, and then the response from the plugins and / or APIs can be used in the model's output. This process can be repeated (e.g., recursively) an arbitrary number of iterations, using any number of plugins and / or APIs, until a response to each query / question / request / process / operation / etc. can be generated in response to the input prompt. Therefore, models can rely not only on their own knowledge gained from training on large datasets, but also on the expertise or optimized properties of one or more external resources (such as APIs, plugins, etc.).

[0197] In some embodiments, multiple language models (e.g., LLM / VLM / MMLM / etc., multiple instances of the same language model, and / or multiple hints provided to the same language model or instances of the same language model) can be implemented, executed, or accessed (e.g., using one or more plugins, user interfaces, APIs, databases, data stores, repositories, etc.) to provide output in response to the same query or in response to separate parts of a query. In at least one embodiment, the same input query and hints (e.g., a set of constraints, condition generators, etc.) can be provided to multiple language models (e.g., language models with different architectures, language models trained on different (e.g., updated) data corpora). In one or more embodiments, the language models can be different versions of the same base model. In one or more embodiments, at least one language model can be instantiated as multiple agents, for example, providing more than one hint to constrain, guide, or otherwise influence the style, content, or character of the provided output. In one or more exemplary non-limiting embodiments, the same language model can be required to provide output corresponding to different roles, perspectives, characters, or different knowledge bases, as defined by the provided hints.

[0198] In any such embodiment, the outputs of two or more (e.g., each) language models, two or more versions of at least one language model, two or more instantiated proxies of at least one language model, and / or provided to two or more prompts for at least one language model can be further processed, such as aggregated, compared, or filtered, or used to determine (and provide) a consensus response. In one or more embodiments, the output from one language model (or version, instance, or proxy) can be provided as input to another language model for further processing and / or validation. In one or more embodiments, the language model can be required to generate or otherwise obtain output about the input source material, wherein the output is associated with the input source material. This association can include, for example, generating captions or text portions embedded (e.g., as metadata) within the input source text or image. In one or more embodiments, the output of the language model can be used to determine the validity of the input source material for further processing or inclusion in the dataset. For example, the language model can be used to evaluate the presence (or absence) of a target word in a text portion or the presence (or absence) of an object in an image, wherein the text or image is annotated to indicate such presence (or absence). Alternatively, the determination from the language model can be used to determine whether the source material should be included in the curatorial dataset, for example, but not limited to this.

[0199] Figure 9 This is a block diagram of an example generative language model system 900 suitable for implementing at least some embodiments of the present disclosure. Figure 9 In the example shown, the generative language model system 900 includes a retrieval augmentation (RAG) component 992, an input processor 905, a tokenizer 910, an embedding component 920, a plugin / API 995, and a generative language model (LM) 930 (which may include LLM, VLM, MMLM, VLA models, etc.).

[0200] At a high level, the input processor 905 can receive input 901, which includes text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasound, etc.), 3D design data, CAD data, generic scene descriptor (USD) data (e.g., OpenUSD, etc.), depending on the architecture of the generative LM 930 (e.g., LLM / VLM / MMLM, etc.). In some embodiments, input 901 includes plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, input 901 may include numerical sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., tabular format, JSON, or XML). In the generative LM In some implementations of 930 capable of handling multimodal input, input 901 can combine text (or text that may be omitted) with image data, audio data, video data, design data, USD data, and / or other types of input data (e.g., but not limited to the data described herein). Taking raw input text as an example, input processor 905 can prepare the raw input text in various ways. For example, input processor 905 can perform various types of text filtering to remove noise (e.g., special characters, punctuation marks, HTML tags, stop words, portions of images, portions of audio, etc.) from the relevant text content. In examples involving stop words (common words that often have little semantic meaning), input processor 905 can remove stop words to reduce noise and enable generative LM. 930 focuses on more meaningful content. Input processor 905 can apply text normalization, for example, by converting all characters to lowercase, removing accents, and / or handling special cases (such as abbreviations or abbreviations) to ensure consistency (e.g., converting ¼ to ¼). Similarly, input processor 905 and / or post-processor can perform inverse text normalization (ITN) to convert plain language back to canonical or other forms (e.g., converting ¼ to ¼). These are just a few examples; other types of input and / or output processing can be applied.

[0201] In some embodiments, RAG component 992 (which may include one or more RAG models, and / or may be performed using generative LM 930 itself) may be used to retrieve additional information to be used as part of input 901 or a prompt. RAGs can be used to enhance input to LLM / VLM / MMLM / etc. with external knowledge to make the answer to a specific question or query or request more relevant, for example, where specific knowledge is required. RAG component 992 may obtain this additional information from one or more external sources (e.g., basic information such as basic text / images / videos / audio / USD / CAD / etc.), which can then be fed along with the prompt to LLM / VLM / MMLM / etc. to improve the accuracy of the model's response or output.

[0202] For example, in some embodiments, in addition to the data retrieved using RAG component 992, input 901 may also be generated using query or model inputs (e.g., questions, requests, etc.). In some embodiments, input processor 905 may analyze input 901 and communicate with RAG component 992 (or in embodiments, RAG component 992 may be part of input processor 905) to identify relevant text and / or other data to provide to generative LM 930 as additional context or information source, typically from which to identify responses, answers, or outputs 990. For example, when input indicates that a user is interested in the required tire pressure for a particular brand and model of vehicle, RAG component 992 may use a RAG model, for example, to perform a vector search in the embedding space to retrieve tire pressure information or its corresponding text from a digital (embedded) version of the owner's manual for that particular brand and model of vehicle. Similarly, when a user revisits a chatbot associated with a particular product for sale or service, RAG component 992 may retrieve previously stored conversation history (or at least its summary) and provide the previous conversation history along with the current inquiry / request as part of input 901 to generative LM 930.

[0203] RAG component 992 can use various RAG techniques. For example, naïve RAG can be used, where documents are indexed, chunked, and applied to an embedding model to generate embeddings corresponding to chunks. User queries can also be applied to this embedding model and / or another embedding model of RAG component 992, and the embeddings of chunks can be compared with the embeddings of the query to identify the embeddings most similar / relevant to the query. These most similar / relevant embeddings can be provided to generative LM 930 to generate output.

[0204] In some embodiments, more advanced RAG techniques can be used. For example, chunks can undergo pre-retrieval processes (e.g., routing, rewriting, metadata analysis, expansion, etc.) before being passed to the embedding model. Furthermore, post-retrieval processes (e.g., re-ranking, hint compression, etc.) can be performed on the output of the embedding model before generating the final embedding, which is then used for comparison with the input query.

[0205] As a further example, modular RAG techniques can be used, such as those similar to Naive RAG and / or Advanced RAG, but also including features such as hybrid search, recursive retrieval and query engines, StepBack methods, subqueries and hypothetical document embeddings.

[0206] As another example, Graph RAG can use a knowledge graph as a source of context or factual information. Graph RAG can be implemented using a graph database as a source of contextual information sent to LLM / VLM / MMLM / etc. Instead of providing the model with chunks of data extracted from larger documents (which can lead to a lack of context, factual accuracy, linguistic accuracy, etc.) (or anything other than providing the model with chunks of data extracted from larger documents), Graph RAG can also provide the model with structured entity information by combining structured entity text descriptions with their many attributes and relationships, thus giving the model deeper insights. In implementing Graph RAG, the systems and methods described herein use graphs as content stores and extract associated document chunks, requiring LLM / VLM / MMLM / etc. to use them to answer questions. In such embodiments, the knowledge graph may include relevant textual content and metadata about the knowledge graph, and may also be integrated with a vector database. In some embodiments, Graph RAG can use the graph as a subject matter expert, where descriptions of concepts and entities associated with queries / hints can be extracted and passed to the model as semantic context. These descriptions may include relationships between concepts. In other examples, the graph can be used as a database where a portion of a query / hint can be mapped to a graph query, the graph query can be executed, and LLM / VLM / MMLM / etc. can aggregate the results. In such examples, the graph can store associated factual information and can be used for queries (natural language queries) and entity links to graph query tools (NL to graph query tools). In some embodiments, the graph RAG (e.g., using a graph database) can be combined with standard (e.g., vector database) RAGs and / or other RAG types to benefit from a variety of approaches.

[0207] In any embodiment, the RAG component 992 can implement plugins, APIs, user interfaces, and / or other functions to perform RAG. For example, LLM / VLM / MMLM / etc. can use graph RAG plugins to run queries on knowledge graphs to extract relevant information to feed into the model, and can use standard or vector RAG plugins to run queries on vector databases. For example, the graph database can interact with the plugin's REST interface, thus decoupling the graph database from the vector database and / or the embedded model.

[0208] The tokenizer 910 can segment (e.g., processed) text data into smaller units (tags) for subsequent analysis and processing. Depending on the implementation, the tags can represent individual words, sub-words, characters, audio / video / images, etc. Word-based tokenization divides the text into individual words, treating each word as a separate tag. Sub-word tokenization breaks words down into smaller meaningful units (e.g., prefixes, suffixes, stems), enabling the generative LM 930 to understand morphological changes and process words outside the vocabulary more effectively. Character-based tokenization represents each character as a separate tag, enabling the generative LM 930 to process text at a fine-grained level. The choice of tokenization strategy can depend on factors such as the language being processed, the task at hand, and / or the characteristics of the training dataset. Therefore, the tokenizer 910 can transform (e.g., processed) text into a structured format according to the tokenization scheme implemented in a particular embodiment.

[0209] Embedding component 920 can use any known embedding technique to transform discrete tokens into semantically meaningful (e.g., dense, continuous vector) representations. For example, embedding component 920 can use pre-trained word embeddings (e.g., Word2Vec, GloVe, or FastText), one-hot encoding, Term Frequency-Inverse Document Frequency (TF-IDF) encoding, one or more embedding layers of a neural network, and / or others.

[0210] In some implementations where input 901 includes image data / video data, etc., input processor 905 may resize the data to a standard size compatible with the format of the corresponding input channel and / or normalize pixel values ​​to a common range (e.g., 0 to 1) to ensure consistent representation, and embedding component 920 may encode the image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations where input 901 includes audio data, input processor 905 may resample the audio file to a consistent sampling rate for uniform processing, and embedding component 920 may use any known technique to extract and encode audio features, such as in the form of a spectrogram (e.g., a Mel spectrogram). In some implementations where input 901 includes video data, input processor 905 may extract frames or apply resizing to extracted frames, and embedding component 920 may extract features such as optical flow embedding or video embedding and / or encode temporal information or frame sequences. In some implementations where input 901 includes multimodal data, the embedded component 920 can use techniques such as early fusion (stitching), late fusion (sequential processing), and attention-based fusion (e.g., self-attention, cross-attention) to fuse representations of different types of data (e.g., text, images, audio, USD, video, design, etc.).

[0211] Other components of the generative LM 930 and / or generative LM system 900 may use different types of neural network architectures depending on the implementation scheme. For example, a transformer-based architecture (such as the one used in models like GPT) may be implemented, which may include a self-attention mechanism that weights the importance of different words or tokens in the input sequence and / or a feedforward network that processes the output of the self-attention layer, applying a nonlinear transformation to the input representation and extracting higher-level features. Some non-limiting example architectures include transformers (e.g., encoder-decoder, decoder-only, multimodal), RNNs, LSTMs, fusion models, diffusion models, cross-modal embedding models that learn a joint embedding space, graph neural networks (GNNs), hybrid architectures that combine different types of adversarial networks (such as generative adversarial networks or GANs or adversarial autoencoders (AAEs) for joint distribution learning, linear time series modeling using a selective state-space modeling (SSM) architecture (e.g., the Mamba LLM architecture), etc. Therefore, depending on the implementation scheme and architecture, the embedded component 920 can apply the encoded representation of the input 901 to the generative LM 930, and the generative LM 930 can process the encoded representation of the input 901 to generate an output 990, which may include response text and / or other types of data.

[0212] As described herein, in some embodiments, the generative LM 930 may be configured to access or use (or be able to access or use) plugins / APIs 995 (which may include one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc.). For example, for certain tasks or operations where the generative LM 930 is not ideally suited, the model may have instructions (e.g., as a result of training, and / or based on instructions in a given prompt, such as instructions retrieved using RAG component 992) to access one or more plugins / APIs 995 (e.g., third-party plugins) to help process the current input. In such an example, when at least part of the prompt is related to a restaurant or weather, the model may access one or more restaurant or weather plugins (e.g., via one or more APIs), sending at least part of the prompt associated with a particular plugin / API 995 to the plugin / API 995, which can process the information and return an answer to the generative LM 930, which can then use the response to generate output 990. This process can be repeated (e.g., recursively) an arbitrary number of iterations and repeated with any number of plugins / APIs 995 until an output 990 that resolves each query / question / request / process / action / etc. from input 901 is generated. Therefore, the model can rely not only on its own knowledge acquired from training on a large dataset and / or from data retrieved using the RAG component 992, but also on the expertise or optimized properties of one or more external resources (e.g., plugins / APIs 995).

[0213] In some embodiments, one or more converter engines (TEs) can be implemented. Converter engines can use micro-tensor scaling to optimize performance and accuracy, such as enabling 16-bit floating-point (FP16), 8-bit floating-point (FP8), and / or 4-bit floating-point (FP4) AI processing. For example, a converter engine can use 16-bit or 8-bit floating-point precision and 8-bit or 4-bit floating-point data formats, combined with software algorithms, to improve AI performance and capabilities. By reducing mathematical operations to 8 or 4 bits, TEs can train larger networks faster without compromising accuracy. For example, TEs can include libraries for accelerating converter models on processing devices such as GPUs to provide better performance in training and inference with lower memory utilization. When TEs are combined with other technologies, such as high-speed interconnects between nodes (e.g., using NVLink switches) and tensor cores (which enable mixed-precision computation, such as micro-scaling precision support), server clusters can be more capable of training massive networks (e.g., billions of parameters) at high speeds. Therefore, it can support tensor core precision for FP64, TF32, BF16, FP16, FP8, INT8, FP6 and FP4, as well as CUDA core precision for FP64, FP32, FP16 and BF16.

[0214] The LLM / VLM / MMLM / VLA and other architectures described herein are intended to be examples only, and other suitable architectures may be implemented within the scope of this disclosure.

[0215] Example computing device Figure 10This is a block diagram of an example computing device 1000 suitable for implementing some embodiments of the present disclosure. The computing device 1000 may include an interconnect system 1002 directly or indirectly coupled to the following devices: a memory 1004, one or more central processing units (CPUs) 1006, one or more graphics processing units (GPUs) 1008, a communication interface 1010, input / output (I / O) ports 1012, input / output components 1014, a power supply 1016, one or more presentation components 1018 (e.g., one or more displays, one or more speakers, etc.), and one or more logic units 1020. In at least one embodiment, one or more computing devices 1000 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 1008 may include one or more vGPUs, one or more CPUs 1006 may include one or more vCPUs, and / or one or more logic units 1020 may include one or more virtual logic units. Thus, one or more computing devices 1000 may include discrete components (e.g., a full GPU dedicated to computing device 1000), virtual components (e.g., a portion of the GPU dedicated to computing device 1000), or combinations thereof.

[0216] although Figure 10 The various blocks are shown as connected via interconnect system 1002 using lines, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 1018 (such as a display device) may be considered I / O component 1014 (e.g., if the display is a touchscreen). As another example, CPU 1006 and / or GPU 1008 may include memory (e.g., memory 1004 may represent a storage device other than the memory of GPU 1008, CPU 1006, and / or other components). Therefore, Figure 10 The computing devices described are for illustrative purposes only. No distinction is made between such categories as “workstation,” “server,” “laptop computer,” “desktop computer,” “tablet computer,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of this category. Figure 10 Within the scope of computing devices.

[0217] Interconnect system 1002 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 1002 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 1006 may be directly connected to memory 1004. Further, CPU 1006 may be directly connected to GPU 1008. In cases where there is a direct or point-to-point connection between components, interconnect system 1002 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required to be included in computing device 1000.

[0218] The memory 1004 may include any computer-readable medium from a variety of computer-readable media. The computer-readable medium may be any available medium accessible by the computing device 1000. The computer-readable medium may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, the computer-readable medium may include computer storage media and communication media.

[0219] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1004 may store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 1000. As used herein, computer storage media does not include the signal itself.

[0220] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and includes any information transmission medium. The term "modulated data signal" can refer to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the above should also be included within the scope of computer-readable media.

[0221] CPU 1006 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1000 to perform one or more of the methods and / or processes described herein. Each CPU 1006 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling numerous software threads simultaneously. CPU 1006 may include any type of processor and may include different types of processors depending on the type of computing device 1000 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 1000, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (such as math coprocessors), computing device 1000 may also include one or more CPUs 1006.

[0222] In addition to or in lieu of one or more CPUs 1006, one or more GPUs 1008 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1000 to perform one or more of the methods and / or processes described herein. One or more GPUs 1008 may be integrated GPUs (e.g., with one or more CPUs 1006) and / or one or more GPUs 1008 may be discrete GPUs. In embodiments, one or more GPUs 1008 may be coprocessors of one or more CPUs 1006. GPUs 1008 may be used by computing device 1000 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 1008 may be used for general-purpose computing on a GPU (GPGPU). GPUs 1008 may include hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. GPUs 1008 may produce pixel data of an output image in response to rendering commands (e.g., rendering commands received from CPUs 1006 via a host interface). GPU 1008 may include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 1004. GPU 1008 may include two or more GPUs operating in parallel (e.g., via a link). The link may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 1008 may produce pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for an analog image). Each GPU may include its own memory or may share memory with other GPUs.

[0223] In addition to or in lieu of CPU 1006 and / or GPU 1008, logic unit 1020 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1000 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 1006, one or more GPUs 1008, and / or one or more logic units 1020 may execute any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 1020 may be a portion of one or more CPUs 1006 and / or GPUs 1008 and / or integrated into one or more CPUs 1006 and / or GPUs 1008, and / or one or more logic units 1020 may be discrete components or otherwise external to CPUs 1006 and / or GPUs 1008. In an embodiment, one or more of the logic units 1020 may be coprocessors of one or more of the CPU 1006 and / or one or more of the GPU 1008.

[0224] Examples of logic unit 1020 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree lateral unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), programmable vision accelerator (PVA) (which may include one or more direct memory access (DMA) systems), and one or more vision or vector processing units (VPUs). This includes one or more pixel processing engines (PPEs) (e.g., a 2D array of processing elements, each of which communicates north, south, east, and west with one or more other processing elements in the array), one or more decoupled accelerators or units (e.g., decoupled lookup table (DLUT) accelerators or units), vision processing units (VPUs), optical flow accelerators (OFAs), field-programmable gate arrays (FPGAs), neuromorphic chips, quantum processing units (QPUs), associative processing units (APUs), arithmetic logic units (ALUs), application-specific integrated circuits (ASICs), floating-point units (FPUs), input / output (I / O) elements, peripheral component interconnects (PCIs) or fast peripheral component interconnects (PCIe) elements, etc.

[0225] The communication interface 1010 may include one or more receivers, transmitters, and / or transceivers enabling the computing device 1000 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). The communication interface 1010 may include components and functions for enabling communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wirelessband), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 1020 and / or the communication interface 1010 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via interconnect system 1002 to one or more GPUs 1008 (e.g., memory of one or more GPUs 1008).

[0226] I / O port 1012 enables computing device 1000 to be logically coupled to other devices including I / O component 1014, one or more presentation components 1018, and / or other components, some of which may be built into (e.g., integrated into) computing device 1000. Illustrative I / O component 1014 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dish antennas, scanners, printers, wireless devices, etc. I / O component 1014 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by the user. In some cases, input may be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, pen recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of computing device 1000. Computing device 1000 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. Additionally, the computing device 1000 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables motion detection. In some examples, the computing device 1000 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0227] The power supply 1016 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 1016 may provide power to the computing device 1000 so that the components of the computing device 1000 can operate.

[0228] The presentation component 1018 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 1018 may receive data from other components (e.g., GPU 1008, CPU 1006, etc.) and output the data (e.g., as images, videos, sounds, etc.).

[0229] Example network environment A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 10 It is implemented on one or more instances of one or more computing devices 1000—for example, each device may include similar components, features, and / or functions of one or more computing devices 1000. Furthermore, in the case of implementing back-end devices (e.g., servers, NAS, etc.), the back-end devices may be included as part of a data center, such as, but not limited to, those described herein.

[0230] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. In cases where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0231] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein for the server can be implemented on any number of client devices.

[0232] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or application may respectively include network-based service software or applications. In embodiments, one or more client devices may use network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs). The framework layer may be, but is not limited to, a free and open-source software network application framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0233] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers at the state, region, country, global, etc.). If the connection to a user (e.g., a client device) is relatively close to an edge server, the core server can assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0234] One or more client devices may include the information described in this article. Figure 10 At least some of the components, features, and functions of one or more example computing devices 1000 described. By way of example and not limitation, the client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming device or system, entertainment system, vehicle computer system, embedded system controller, chat booth, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0235] Example Terms Clause 1. One or more processors, comprising a processing circuitry system configured to: generate at least one frame, based at least on back-projection from a two-dimensional 2D viewport to one or more of a plurality of image data frames, the at least one frame representing a back-projected view of an environment surrounding the self-machine, the plurality of image data frames representing at least partially overlapping views of the environment.

[0236] Clause 2. One or more processors as described in Clause 1, wherein the processing circuitry is further configured to perform one or more image quality operations on the at least one frame.

[0237] Clause 3. One or more processors as described in Clause 1 or Clause 2, wherein the one or more image quality operations include at least one of the following: temporal noise reduction, super-resolution, anti-aliasing, color adjustment, or tone mapping.

[0238] Clause 4. One or more processors according to Clause 1 or Clause 2, wherein the processing circuitry is further configured to omit one or more image quality operations for pixels in the plurality of image data frames that depict views other than the back-projected view.

[0239] Clause 5. One or more processors according to Clause 1 or Clause 2, wherein the processing circuitry is further configured to control temporal noise reduction of one or more image quality operations in the one or more regions based at least on identifying one or more regions in the at least one frame corresponding to one or more overlapping regions of the image data using the back projection.

[0240] Clause 6. One or more processors according to Clause 1 or Clause 2, wherein the processing circuitry is further configured to control a color coordination process of one or more image quality operations based at least on identifying, using the back projection, which of the plurality of image data frames is visible in the back-projected view.

[0241] Clause 7. One or more processors according to Clause 1 or Clause 2, wherein the processing circuitry is further configured to generate representations of one or more local magnification factors in one or more local regions of the back-projected view, and to perform the one or more image quality operations in the one or more local regions of at least one frame based at least on the one or more local magnification factors.

[0242] Clause 8. One or more processors according to Clause 7, wherein the processing circuitry is further configured to identify one or more local magnification factors in one or more local regions of the at least one frame, based at least on the magnification of the projection representation of the measured bitmap image.

[0243] Clause 9. One or more processors according to Clause 7, wherein the processing circuitry is further configured to identify one or more local magnification factors in one or more local regions of the at least one frame, at least based on mapping output pixels to corresponding input pixels using the back projection.

[0244] Clause 10. One or more processors according to Clause 1 or Clause 2, wherein the processing circuitry is further configured to generate the at least one frame based at least on sampling from the one or more frames of the plurality of image data frames, wherein the sampling is based at least on one or more local magnification factors of one or more local regions of the at least one frame.

[0245] Clause 11. One or more processors according to Clause 10, wherein the sampling includes at least one of the following: subsampling one or more projection indices during the back projection, or deforming one or more subsampled pixels of one or more local regions of the at least one frame based at least on the one or more local magnification factors.

[0246] Clause 12. One or more processors as described in Clause 10, wherein the sampling comprises: performing at least one of super-resolution processing or anti-aliasing processing in one or more local regions of the at least one frame, based at least on the one or more local magnification factors.

[0247] Clause 13. One or more processors according to Clause 1 or Clause 2, wherein the processing circuitry is further configured to: control at least one of the one or more image quality operations of temporal noise reduction, color adjustment, or tone mapping in one or more local regions of the at least one frame, based at least on one or more local magnification factors corresponding to the one or more local regions.

[0248] Clause 14. One or more processors as described in Clause 1 or Clause 2, wherein said one or more processors are included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more analog operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using edge devices; a system implemented using robots; a system for performing one or more generative AI operations; a system for using one or more large language models (LLMs). A system for performing operations; a system for performing operations using one or more visual language models (VLMs); a system for performing operations using one or more multimodal language models (MMLMs); a system for performing operations using one or more visual-language-action (VLA) models; a system for using or deploying one or more inference microservices; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0249] Clause 15. A system comprising one or more processors, said one or more processors being configured to: generate at least one frame, based at least on back-projection from a two-dimensional 2D viewport to one or more of a plurality of image data frames, said at least one frame representing a back-projected view of an environment of the self-machine, said plurality of image data frames representing at least partially overlapping views of said environment.

[0250] Clause 16. The system according to Clause 15, wherein the one or more processors further perform one or more image quality operations on the at least one frame.

[0251] Clause 17. A system according to Clause 15 or Clause 16, wherein the system is included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more analog operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs). A system; a system for performing operations using one or more visual language models (VLMs); a system for performing operations using one or more multimodal language models (MMLMs); a system for performing operations using one or more visual-language-action (VLA) models; a system for using or deploying one or more inference microservices; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0252] Clause 18. A system comprising one or more processors for controlling an infotainment system of a simulated vehicle in a simulated environment, rendered using one or more optical transmission simulation algorithms, to present a visualization of the simulated environment, the visualization being generated based at least on: back-projecting from a two-dimensional 2D viewport to at least one frame of a plurality of simulated image data frames, the plurality of simulated image data frames representing at least partially overlapping views of the simulated environment; and performing one or more image quality operations on pixels corresponding to the at least one frame.

[0253] Clause 19. The system pursuant to Clause 18, wherein the simulation is generated at least in part using one or more content creation applications of a 3D content collaboration platform for 3D assets.

[0254] Clause 20. The system according to Clause 19, wherein the simulation environment is represented using the OpenUSD format in at least one of the one or more content creation applications.

[0255] Clause 21. The system according to Clause 18, wherein one or more processors are further configured to ignore the one or more image quality operations for pixels located outside the 2D viewport in the at least one frame.

[0256] Clause 22. The system according to Clause 18, wherein at least one of the one or more language models is implemented in at least one of a plurality of processing nodes in a data center, and one or more remote clients are able to access the at least one language model through an application programming interface (API) or an application plugin.

[0257] This disclosure can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions, such as program modules, that are executed by a computer or other machine, such as a personal data assistant or other handheld device. Typically, a program module, which includes routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure can also be practiced in distributed computing environments, where tasks are performed by remote processing devices linked via a communication network.

[0258] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0259] This document describes in detail the subject matter of this disclosure to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the discloser has envisioned that the claimed subject matter may be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of the method employed, these terms should not be construed as suggesting any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. One or more processors, including a processing circuitry system, said processing circuitry system being used for: At least one frame is generated based on backprojection from a 2D viewport to one or more of a plurality of image data frames, said at least one frame representing a backprojected view of the environment surrounding the self-machine, said plurality of image data frames representing at least partially overlapping views of said environment; and Perform one or more image quality operations on the at least one frame.

2. The processors according to claim 1, wherein, The one or more image quality operations include at least one of the following: temporal noise reduction, super-resolution, anti-aliasing, color adjustment, or tone mapping.

3. The processors according to claim 1, wherein, The processing circuitry is also configured to omit one or more image quality operations for pixels in the plurality of image data frames that depict views other than the back-projected view.

4. The processors according to claim 1, wherein, The processing circuitry is also configured to control temporal noise reduction of one or more image quality operations in the one or more regions based at least on identifying one or more regions in the at least one frame corresponding to one or more overlapping regions in the image data using the back projection.

5. The processors according to claim 1, wherein, The processing circuitry is also configured to control the color coordination process of one or more image quality operations, at least based on identifying which of the plurality of image data frames is visible in the back-projected view using the back projection.

6. The processors according to claim 1, wherein, The processing circuitry is also configured to generate representations of one or more local magnification factors in one or more local regions of the back-projected view, and to perform one or more image quality operations in the one or more local regions of at least one frame based at least on the one or more local magnification factors.

7. The processors according to claim 6, wherein, The processing circuitry is also configured to identify one or more local magnification factors in one or more local regions of the at least one frame, based at least on the magnification of the projection representation of the measured dot matrix image.

8. The processors according to claim 6, wherein, The processing circuitry is also configured to identify one or more local magnification factors in one or more local regions of the at least one frame, based at least on mapping the output pixels to the corresponding input pixels using the back projection.

9. The processors according to claim 1, wherein, The processing circuit system is also configured to generate the at least one frame based at least on sampling from one or more frames of the plurality of image data frames, wherein the sampling is based at least on one or more local magnification factors of one or more local regions of the at least one frame.

10. One or more processors according to claim 9, wherein, The sampling includes at least one of the following: subsampling one or more projection indices during the back projection, or deforming one or more subsampled pixels in one or more local regions of the at least one frame based at least on the one or more local magnification factors.

11. One or more processors according to claim 9, wherein, The sampling includes performing at least one of super-resolution processing or anti-aliasing processing in the one or more local regions of the at least one frame, based at least on the one or more local magnification factors.

12. The processors according to claim 1, wherein, The processing circuit system is also configured to: control at least one of the one or more image quality operations, namely temporal noise reduction, color adjustment, or tone mapping, in one or more local regions of the at least one frame, based at least on one or more local magnification factors corresponding to the one or more local regions.

13. The processors according to claim 1, wherein, The one or more processors are included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing optical transmission simulation; A system for performing collaborative content creation for 3D assets; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models (MMLMs); A system for performing operations using one or more visual-language-action (VLA) models; A system for using or deploying one or more inference microservices; A system for performing one or more conversational AI operations; A system for generating synthetic data; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

14. A system comprising one or more processors, said one or more processors being used to: At least one frame is generated based on backprojection from a 2D viewport to one or more of a plurality of image data frames, said at least one frame representing a backprojected view of the machine's environment, said plurality of image data frames representing at least partially overlapping views of said environment; and Perform one or more image quality operations on the at least one frame.

15. The system according to claim 14, wherein, The system is included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing optical transmission simulation; A system for performing collaborative content creation for 3D assets; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models (MMLMs); A system for performing operations using one or more visual-language-action (VLA) models; A system for using or deploying one or more inference microservices; A system for performing one or more conversational AI operations; A system for generating synthetic data; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

16. A system comprising: One or more processors are configured to control an infotainment system of a simulated vehicle in a simulated environment in a simulation rendered using one or more optical transmission simulation algorithms, to present a visualization of the simulated environment, the generation of which is based at least on: back-projection from a two-dimensional 2D viewport to at least one of a plurality of simulated image data frames, the plurality of simulated image data frames representing at least partially overlapping views of the simulated environment. And perform one or more image quality operations on the pixels corresponding to the at least one frame.

17. The system according to claim 16, wherein, The simulation was generated at least in part using one or more content creation applications from a 3D content collaboration platform for 3D assets.

18. The system according to claim 17, wherein, The simulation environment is represented using the OpenUSD format in at least one of the one or more content creation applications.

19. The system according to claim 16, wherein, One or more processors are also configured to ignore one or more image quality operations for pixels located outside the 2D viewport in the at least one frame.

20. The system according to claim 16, wherein, At least one of the one or more language models is implemented in at least one of the multiple processing nodes in the data center, and one or more remote clients are able to access the at least one language model through an application programming interface (API) or an application plugin.

Images