Object detection with temporal information for autonomous and semi-autonomous systems and applications

Abstract

The present disclosure relates to object detection with temporal information for autonomous and semi-autonomous systems and applications. In various examples, described herein are systems and methods for autonomous and / or semi-autonomous systems and applications that utilize temporal information to detect object attributes. The disclosed systems and methods use one or more neural networks, such as a convolutional neural network (CNN), to determine attributes associated with an object based on temporal information. For example, the neural network can use one or more first layers, such as a CNN feature extractor, to determine spatial features associated with an image representing an object, where the spatial features are then stored in memory. The neural network can then use one or more second layers, such as one or more CNN classifiers, to process the spatial features and determine a classification and / or one or more attributes associated with the object. For example, if the object includes a vehicle with a light signal, the neural network can determine a state of the light signal.

Description

Background Technology

[0001] Using image data to determine various object attributes, such as temporal attributes that change over time, is crucial for many autonomous and / or semi-autonomous operations of machines. For example, for an autonomous machine moving near a vehicle, it is important to determine whether the vehicle is turning, reversing, and / or braking based on the state of the vehicle's light signals. However, determining such object attributes can be difficult in certain situations, such as when visual characteristics in a single image are unclear (e.g., light signals switching between states) and / or at least part of the object is occluded (e.g., light signals are obscured in the image). Furthermore, determining attributes for all different configurations adds to the challenge, as similar types of objects may include different configurations (e.g., different vehicle models include light signals with different shapes and / or positions).

[0002] Therefore, various techniques have been developed to determine object attributes using image data. For example, computer vision techniques can process image data to identify light locations and then apply heuristics to determine the light state. However, these techniques often only work in constrained environments, offering poor performance when weather and / or lighting conditions vary widely due to the use of manually tuned heuristics. Therefore, deep learning-based techniques utilize deep learning algorithms (such as recurrent neural network (RNN)-based algorithms) to identify light locations, extract spatial features, and then integrate temporal information through the RNN. However, in some cases, analyzing temporal information using the hidden states of the RNN can be difficult. Furthermore, problems may arise due to the difference in processing time between RNN training and deployment. For example, an RNN might be trained using a fixed time series, and after deployment, the RNN might continuously track light signals until the light signals are no longer depicted in the image. Summary of the Invention

[0003] Embodiments of this disclosure relate to using temporal information to detect object attributes for autonomous and / or semi-autonomous systems and applications. The disclosed systems and methods use one or more neural networks (such as one or more convolutional neural networks (CNNs)) to determine attributes associated with an object based on temporal information. For example, the neural network may use one or more first layers (such as CNN feature extractors) to determine spatial features associated with an image representing the object, and these spatial features are then stored in memory. The neural network may then use one or more second layers (such as one or more CNN classifiers) to process the spatial features and determine the object's classification and / or one or more attributes associated with the object. For example, if the object includes a vehicle with light signals, the neural network may determine one or more states of the light signals. As described in more detail herein, in some examples, the neural network may determine attributes based on the occurrence of one or more events, such as the memory storing a threshold amount of spatial features.

[0004] Compared to conventional systems, the systems and methods of this disclosure, in some embodiments, use a temporal CNN with optimized convolutional layers to determine object attributes instead of an RNN. This can improve the performance of determining object attributes (such as by avoiding recurrent hidden states) while also improving the training process. For example, training a temporal CNN can bypass various RNN training problems, such as vanishing gradients. Furthermore, compared to conventional systems, the systems and methods of this disclosure, in some embodiments, aggregate temporal information (e.g., spatial features) simultaneously rather than sequentially for performing the analysis to determine object attributes. This can further improve the performance of the neural network by analyzing contributions from multiple time steps (e.g., each time step) each time an object attribute is determined. Additionally, the systems and methods of this disclosure, in some embodiments, can use a pure CNN architecture, which simplifies deployment compared to the RNN architecture of conventional systems. Attached Figure Description

[0005] The following describes in detail, with reference to the accompanying drawings, the system and method for detecting object attributes using usage time information in autonomous and / or semi-autonomous systems and applications, wherein:

[0006] Figure 1 An example data flow diagram is shown illustrating a process for determining attributes associated with an object using time information according to some embodiments of this disclosure;

[0007] Figure 2 Examples of generating feature information associated with an image and storing it in memory are shown according to some embodiments of the present disclosure;

[0008] Figures 3A-3BAn example is shown of storing additional information in memory as placeholders for spatial information associated with an image, according to some embodiments of the present disclosure;

[0009] Figures 4A-4B Examples of architectures associated with neural networks capable of performing one or more processes described herein are shown according to some embodiments of this disclosure;

[0010] Figure 5 Examples of procedures for training one or more neural networks to perform at least a portion of the processes described herein are shown according to some embodiments of the present disclosure;

[0011] Figure 6 A flowchart illustrating a method for determining attributes associated with an object using time information, according to some embodiments of the present disclosure, is shown;

[0012] Figure 7 A flowchart illustrating a method for determining attributes associated with an object using stored feature information, according to some embodiments of the present disclosure, is shown.

[0013] Figure 8A Examples of sensor locations having a corresponding field of view or sensing field, such as an autonomous or semi-autonomous machine, according to at least some embodiments of this disclosure;

[0014] Figure 8B This is an illustration of examples of the locations of components and sensors on an autonomous or semi-autonomous vehicle according to at least some embodiments of this disclosure;

[0015] Figure 8C This is a block diagram of an example system architecture for autonomous or semi-autonomous vehicles, robots, and / or other machine types according to at least some embodiments of this disclosure;

[0016] Figure 8D This is a block diagram of an example architecture of a computing system (e.g., a system-on-a-chip (SoC)) according to at least some embodiments of the present disclosure;

[0017] Figure 8E This is a system diagram of communication between a cloud-based server and example autonomous or semi-autonomous vehicles, robots and / or other machine types according to at least some embodiments of this disclosure;

[0018] Figure 9 This is a system diagram illustrating three computer ecosystems according to at least some embodiments of the present disclosure, including a computing system for generating or creating artificial intelligence (AI) (e.g., AI training and validation data), a computing system for training artificial intelligence, and a computing system for deploying AI at the edge;

[0019] Figure 10This is a block diagram of an example computing system for generative artificial intelligence (AI) according to at least some embodiments of the present disclosure; and

[0020] Figure 11 This is a block diagram of an example computing device according to at least some embodiments of the present disclosure. Detailed Implementation

[0021] Systems and methods for detecting object attributes for autonomous and / or semi-autonomous systems and applications using time information are disclosed. Although this disclosure may relate to examples of autonomous or semi-autonomous vehicles, robots, and / or other machine types 800 (which may alternatively be referred to herein as "vehicle 800," "self-vehicle 800," "machine 800," "self-machine 800," "robot 800," and / or "self-robot 800"), examples are provided. Figures 8A-8E The descriptions herein are provided, but are not intended to be limiting. For example, the systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms (e.g., autonomous mobile robots (AMRs), humanoid robots, robotic arms and / or end effectors), warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, ships, shuttle buses (e.g., robotaxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones and / or other vehicle, robot or machine types. Furthermore, while this disclosure may describe object detection and / or classification, this is not intended to be limiting, and the systems and methods described herein can be used in augmented reality (AR), virtual reality (VR), mixed reality (MR), robotics, security (e.g., smart cities), autonomous or semi-autonomous machine applications, industrial manufacturing, simulation and / or any other technological space where attribute detection and / or classification can be used. In some embodiments, the systems, methods, and / or processes described herein can be used with Figures 8A-8E Example machine 800, Figure 9 Example calculation ecosystem 900, Figure 10 Example generative language model system 1000 and / or Figure 11 The example computing device 1100 uses similar components, features, and / or functions to perform this task.

[0022] For example, the system can receive, generate, acquire, and / or retrieve image data obtained using one or more image sensors of the machine. As described herein, the image data can represent one or more images depicting at least an object located in an environment. For example, the image data can represent an image of another machine (and / or other types of traffic features) (such as vehicles) within the environment, including one or more light signals. In some examples, the system may then use one or more image processing tasks to process at least a portion of the image data. For example, the system can process the image data to generate processed image data representing one or more cropped images that depict a larger portion of the object relative to its surrounding environment. In some examples, the system can use any technique to perform the processing tasks, such as using machine learning models, neural networks, classifiers, algorithms, processors, and / or any other type of processing component.

[0023] The system can then use one or more neural networks to process the processed image data (and / or image data), these neural networks being trained to determine information associated with the object, such as the object's classification, one or more attributes associated with the object, and / or any other information. For the first example, if the object is another machine in the environment that includes at least one or more light signals, the neural network can be trained to at least determine the machine's classification (such as vehicle and / or vehicle type), and one or more states of the machine and / or light signals. As described herein, the states of the light signals can include, but are not limited to, cycling (e.g., repeatedly turning on and off), on, off, and / or any other state. Furthermore, the machine's state can be associated with the states of one or more light signals, such as, but not limited to, being in an emergency (e.g., if the machine is an emergency vehicle), turning left, turning right, reversing, stopped, and / or any other state. For the second example, if the object is a dynamic object, the neural network can be trained to determine the object's classification—such as a machine (e.g., car, van, bus, truck, motorcycle, etc.), a person, animal, and / or similar—and motion information associated with the object (e.g., speed, acceleration, direction of travel, etc.).

[0024] To process image data, a neural network may include one or more feature extractors—such as CNN extractors (and / or any other type of extractor, such as, but not limited to, those described herein)—that process the image data to generate feature information associated with the image. For example, for an image, the feature information may include one or more spatial features and / or feature vectors associated with that image. The system may then store the feature information in memory, such as in the order in which the feature information was generated. For example, the system may store first feature information associated with a first image, then second feature information associated with a subsequent second image, then third feature information associated with a subsequent third image, and so on. Furthermore, in some examples, the system may perform these processes until the memory is full. For example, if the memory includes storage space for feature information associated with 15 images, the system may store 15 instances of feature information in the memory. Furthermore, in some examples, such as when the memory is full, the system may remove older feature information from the memory while continuing to store new feature information in the memory. For example, the memory may include a (rolling) buffer memory that continuously stores the latest feature information associated with the latest image.

[0025] In some examples, the system may perform additional storage procedures associated with memory, such as before memory is full of feature information. For a first example, the system may initially store placeholder information that does not represent features associated with the image. For example, placeholder information may include one or more predefined value vectors, such as one or more zero vectors (and / or any other value vectors). For a second example, the system may generate copies of one or more instances of feature information and then store these copies in memory. In either of these examples, the system may perform such procedures to keep memory full. Furthermore, when storing new feature information in memory, the system may first remove this added information (e.g., placeholder information and / or copies of feature information).

[0026] The neural network can then further process feature information (and / or placeholder information) (such as by using one or more classifiers) to determine the classification and / or one or more attributes associated with the object. In some examples, the system may process the feature information based on the occurrence of one or more events, such as memory capacity meeting a threshold (e.g., memory becoming full). Furthermore, the neural network can include various architectures for processing feature information. For a first example, the neural network may include one or more temporal classifiers (such as temporal CNN classifiers (and / or any other type of temporal classifier described herein)) configured to process feature information and determine the classification and / or attributes associated with the object. For a second example, the neural network may include one or more temporal classifiers configured to process feature information to determine one or more attributes, and one or more other classifiers configured to process at least a portion of the feature information (e.g., instances of feature information associated with frames) to determine the classification associated with the object, such as another CNN classifier (and / or any other type of classifier described herein). While these are two example architectures for neural networks to process feature information, in other examples, the neural network may include additional and / or alternative architectures.

[0027] In some examples, the system can then proceed to process additional image data to further determine the classification and / or one or more attributes associated with the object. For example, the neural network can process each image represented by the image data, every other image represented by the image data, every four images represented by the image data, and / or any other interval of the image data to determine the classification and / or one or more attributes associated with the object. This is because, in some examples, attributes can change over time, such as when the object performs a new manipulation. Furthermore, in some examples, the system can perform these processes to determine one or more classifications and / or one or more attributes associated with one or more additional objects.

[0028] In some examples, the neural network can be trained to perform one or more of the processes described herein. For example, the neural network can be trained using at least training input data representing videos depicting objects over a time period and ground truth data representing classifications and / or attributes associated with the objects. For example, the neural network can process the training input data and generate output data representing predicted classifications and / or predicted attributes associated with the objects. The system can then compare the predicted classifications and / or predicted attributes with ground truth classifications and / or ground truth attributes to determine one or more losses. Furthermore, the system can use one or more losses to update the weights and / or parameters of the neural network. For example, the system can use one or more losses to update at least the weights and / or parameters of the feature extractor and / or classifier. As described in more detail herein, the system can further perform one or more additional processes to improve the training, such as using instances of temporal random jitter and / or random discarding of spatial features.

[0029] In some embodiments, the systems and methods described herein can be performed in a simulated environment (e.g., NVIDIA DriveSIM, ISAAC Sim, ISAAC Gym, ISAAC Lab, etc.) using simulated data (e.g., simulated environment data and simulated sensor data from simulated sensors of virtual or simulated vehicles, robots, or machines within the simulated environment). For example, simulated input data (e.g., map data, perception data, self-motion data, tactile data, and / or any other data described herein) can be used to determine classifications and / or attributes associated with objects, and this information can be used to perform operations associated with virtual machines in the simulated environment. These simulated operations can be used to test the performance of underlying algorithms, systems, and / or processes before deploying them to the real world. In some cases, simulations can be used to generate synthetic training data, such as image data representing objects in the simulation. Neural networks can then be used to use or process the synthetic training data (in addition to or as a substitute for real-world data) to determine the classifications and / or attributes of objects.

[0030] In any example, such as when the simulation environment is used for testing, validation, training, etc., one or more optical transport simulation algorithms (such as one or more ray tracing and / or path tracing algorithms) may be used to render or otherwise generate the simulation environment and / or associated training data. When using optical transport simulation, the simulation system may employ one or more dedicated ray tracing hardware accelerators and / or processors (e.g., NVIDIA's RTX, or other real-time ray tracing GPUs, such as GPUs including one or more ray tracing (RT) cores) optimized for performing real-time or near-real-time optical transport simulation operations in conjunction with one or more other processors in the system (e.g., GPUs, CPUs, accelerators, etc.). In some embodiments, the simulation environment and / or one or more of its objects, features, or components may be generated or managed within a three-dimensional (3D) content collaboration platform (e.g., NVIDIA's OMNIVERSE), which may be optimized for or adapted to industrial digitization, generative physics, artificial intelligence, and / or other use cases, applications, and / or services. For example, a content collaboration platform or system may include a system for using or developing generic scene descriptors (USD) (such as OpenUSD) data to manage objects, features, scenes, etc., in simulated environments, digital environments, etc. The platform may include realistic physics simulations (e.g., using NVIDIA's PhysX software development kit (SDK)) to simulate real physics and physical interactions with simulations hosted by the platform. The platform may integrate OpenUSD with ray tracing / path tracing / light transport simulations (such as NVIDIA's RTX rendering technology) into software tools and simulation workflows for building, training, deploying, and / or testing AI systems—such as systems for testing, validating, training (e.g., machine learning models, neural networks, etc.) and / or other tasks related to automobiles, robots, other machine types, and / or other systems and applications. In some examples, the simulated environment may include a digital twin of a real-world environment, such as a specific road segment, warehouse, data center, airport, geographic area, ocean area, and / or any other real-world environment in which autonomous or semi-autonomous vehicles or machines can operate.

[0031] In some embodiments, a remote control or teleoperation system can be used to teleoperate or remotely control vehicles, robots, and / or other machines. For example, the systems and methods described herein can be used to determine the classification and / or attributes of objects that can be included in the visualization or mapping of an environment to assist a remote operator in controlling an autonomous or semi-autonomous machine through the environment (or to provide waypoints or other control or navigation instructions). Thus, a remote operator can use visual, auditory, textual, and / or other cues or indicators generated using the systems and methods described herein to assist in navigating vehicles, robots, machines, etc., in a real-world environment using a teleoperation system.

[0032] In some embodiments, the systems and methods described herein can be deployed in robotic applications. For example, a robot or robotic system may include one or more onboard processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), hardware-based programmable vision accelerator (PVA) (which may include one or more vector processing units (VPU), direct memory access (DMA) systems and / or pixel processing engines (PPE), hardware-based optical flow accelerators (OFA), SoC, etc.)) and memory and / or storage devices (e.g., for storing control algorithms, sensor data and one or more machine learning models). Robotic systems can use these processors to execute one or more machine learning models (e.g., language models, visual language models (VLM), large language models (LLM), visual-language-action (VLA) models, multimodal language models (MMLM), etc.) that enable them to perform complex tasks autonomously or semi-autonomously, such as interacting with and / or manipulating static and / or dynamic objects, or navigating in their environment using sensors such as cameras, LiDAR, RADAR, ultrasonic sensors, etc. The system can use sensor fusion techniques to combine data from multiple sensors (e.g., cameras, infrared, LiDAR, RADAR, accelerometers) to create a comprehensive model of the robot's surroundings. This data can be processed locally on the robot or sent to a remote server for more computationally intensive tasks, such as 3D mapping or SLAM (Simultaneous Localization and Mapping). In one or more embodiments, data from individual robots (e.g., sensor data, task status, or environmental conditions) can be uploaded to the cloud, where a centralized AI model can analyze optimized commands and distribute them across the entire fleet. In some embodiments, the machine learning models described herein (e.g., language models, VLM, VLA, LLM, MMLM, diffusion models, NeRF models, DNN, etc.) can be used to enable the robot to perceive and reason about its environment and / or communicate with one or more other robots and / or people in the environment. In some embodiments, the robot can communicate with one or more locally hosted servers / computing devices and / or with one or more remote servers / computing devices (e.g., located in one or more data centers) using one or more network interface cards (NICs) and / or data processing units (DPUs).

[0033] In some embodiments, the systems and methods described herein can be deployed in in-vehicle infotainment (IVI) systems or cockpit experience (IX) applications. For example, an infotainment system within a vehicle (e.g., a car, truck, drone, construction equipment, robot, semi-autonomous vehicle, or autonomous vehicle) may include one or more onboard processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), hardware-based programmable vision accelerator (PVA) (which may include one or more vector processing units (VPU), direct memory access (DMA) systems, and / or pixel processing engines (PPE)), hardware-based optical flow accelerator (OFA), SoC, etc.) and memory and / or storage devices (e.g., for storing control algorithms, sensor data, and one or more machine learning models) and memory and / or storage devices (e.g., for storing entertainment content, navigation data, and user preferences). The system can use these processors to execute one or more machine learning models (e.g., language models) to enable functions such as voice control, personalized media recommendations, dynamic navigation, and real-time communication with other services via network connectivity. In-vehicle infotainment systems can also use natural language processing (NLP) models to enable voice-based interaction. One or more machine learning models can be stored locally or accessed via one or more APIs connected to cloud services, enabling the system to process requests in real-time or near real-time.

[0034] In some examples, the machine learning models described herein (e.g., deep neural networks, language models, LLMs, VLMs, multimodal language models, visual-language-action (VLA) models, perceptual models, tracking models, fusion models, transformer models, diffusion models, encoder-only models, decoder-only models, encoder-decoder models, neural rendering field (NERF) models, etc.) can be packaged as microservices (e.g., inference microservices (e.g., NVIDIA NIM)), which may include containers (e.g., operating system (OS) level virtualization packages) that may include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model "engine". For example, an inference microservice may include the container itself and the model (e.g., weights and biases). In some cases, such as when the machine learning model is small enough (e.g., has a sufficiently small number of parameters), the model may be included within the container itself. In other examples (e.g., when the model is large), the model may be hosted / stored in the cloud (e.g., in a data center) and / or may be hosted on-premises and / or at the edge (e.g., on a local server or computing device, but outside the container). In these embodiments, the model can be accessed via one or more APIs (such as REST APIs). Therefore, in some embodiments, the machine learning models described herein can be deployed as inference microservices to accelerate model deployment on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, an optimized inference engine (e.g., execution software built using standardized AI model deployments, such as NVIDIA's Triton Inference Server, and / or one or more APIs for high-performance deep learning inference, which may include inference runtimes and model optimizations for providing low latency and high throughput for production applications, such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). The machine learning models described herein, along with the accelerated infrastructure, can be included as part of a microservice capable of deployment with a single command and / or orchestrated and automatically scaled on the accelerated infrastructure using a container orchestration system (e.g., reaching data center scale on a single device). Therefore, the inference microservice may include a machine learning model (e.g., a model optimized for high-performance inference), inference runtime software for executing the machine learning model and providing output / response to inputs (e.g., user queries, prompts, etc.), and enterprise management software for providing health checks, identity verification, and / or other monitoring. In some embodiments, the inference microservice may include software for in-situ replacement and / or updating of the machine learning model. During replacement or updating, the software performing the replacement / update may maintain the user configurations of the inference runtime software and the enterprise management software.

[0035] While examples of using machine learning models (such as neural networks) are described in this article, this is not limiting. For example, but not limited to, any of the various machine learning models and / or neural networks described herein can include any type of machine learning model, such as those using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoder neural networks, artificial neural networks (ANN), convolutional neural networks (CNN), recurrent neural networks (RNN), perceptrons, long short-term memory (LSTM) networks, multilayer perceptron (MLP) networks, deep stacked networks (DSN), generative pre-trained (GPT) models or networks, feedforward networks, radial basis functions... ANN, Self-Organizing Map (SOM), Kohonen Map, Hopfield Network, Boltzmann Machine, Deep Belief Neural Network, Deconvolutional Neural Network, Generative Adversarial Network (GAN), Liquid Machine, Modular Neural Network, Sequence-to-Sequence Model, Networks Using Transformer Architecture, State-Space Model (SSM) (e.g., networks using Mamba architectures (e.g., Mamba-1, Mamba2, etc.), networks using selective state-space models, networks using structured state-space sequence models, etc.), Diffusion Models (e.g., diffusion probability models, fraction-based generative models, etc.), Neural Radiation Field (NeRF) Model, Gaussian Sputtering One or more machine learning models and / or other types of machine learning models, including splat models, Kolmogorov-Arnold networks (KAN), models with encoder-only architectures, models with decoder-only architectures, models with encoder-decoder architectures, generative machine learning models, language models, large language models (LLM), visual language models (VLM), multimodal language models (MMLM), large action models (LAM), visual-language-action (VLA) models, visual transducers, etc.

[0036] The systems and methods described herein may be used, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, spacecraft, ships, shuttles (e.g., robotaxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones, and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, such as, but not limited to, machine control, machine localization, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing), collaborative content creation of 3D assets (NVIDIA's Omniverse), cloud computing, and / or any other suitable application.

[0037] The disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aerial systems, medical systems, rowing systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems using edge devices, systems implementing language models (such as Large Language Models (LLM), Visual Language Models (VLM), Visual-Language-Action (VLA) models, and / or multimodal language models), systems using or deploying one or more inference microservices, systems including deploying one or more machine learning models in services or microservices and OS-level virtualization packages (e.g., containers), systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0038] refer to Figure 1 , Figure 1Example data flow diagrams are shown for a process 100 that uses time information to determine attributes associated with an object according to some embodiments of this disclosure. It should be understood that the arrangements and other arrangements described herein are merely illustrative. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to or in lieu of the arrangements, components, features, and elements shown, and some elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and location (e.g., on a local device, vehicle, or machine at the edge, in the field (such as a locally hosted server), remotely (such as in one or more computing or server devices in one or more data centers in the cloud), and / or elsewhere). The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be performed using one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, microcontroller, embedded processor, digital signal processor (DSP), image signal processor (ISP), physical processing unit (PPU), field-programmable gate array (FPGA), accelerators (e.g., deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural network accelerator (NNA) and / or neural processing unit (NPU), programmable vision accelerator (PVA), optical flow accelerator (OFA), etc.), application-specific integrated circuit (ASIC), data processing unit (DPU), quantum processor, etc.) that execute instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 8A-8E Example machine 800, Figure 9 Example calculation ecosystem 900, Figure 10 Example generative language model systems 1000, and / or Figure 11 The example computing device 1100 uses similar components, features, and / or functions to perform this task.

[0039] Process 100 may include receiving, generating, acquiring, and / or retrieving image data 102 obtained using one or more image sensors of a machine (e.g., example machine 800). As described herein, image data 102 may represent one or more images depicting at least one object located in an environment. For a first example, image data 102 may represent an image depicting another machine (such as a vehicle) in the environment, which includes one or more light signals. For a second example, image data 102 may represent an image depicting a dynamic object (such as a pedestrian) moving in the environment. In some examples, such as Figure 1As shown, process 100 may then include using one or more image processors 104 to process at least a portion of image data 102 to generate processed image data 106. For example, image processor 104 may process image data 102 to generate processed image data 106 representing one or more cropped images that depict a larger portion of an object relative to its surrounding environment. In such examples, image processor 104 may include machine learning models, neural networks, classifiers, algorithms, processors, and / or any other type of processing component.

[0040] Then, process 100 may include processing the processed image data 106 (and / or, in some examples, image data 102) using one or more feature extractors 108 to generate one or more instances of feature information 110 associated with the image. As described herein, in some examples, any method for performing feature extraction may be used, such as learning-based methods (e.g., CNNs, pre-trained models, etc.), pixel-based methods, transform-based methods (e.g., Fourier transforms, etc.), filter-based methods (e.g., edge detection, corner detection, etc.), and / or descriptor-based methods (e.g., scale-invariant feature transforms, accelerated robust features, etc.). For example, feature extractor 108 may include one or more CNN extractors configured to process the processed image data 106 to generate feature information 110. Additionally, for an image, feature information 110 may include one or more spatial features, one or more feature vectors, and / or any other representation of feature information.

[0041] Then, process 100 may include storing feature information 110 in memory 112. As described herein, in some examples, instances of feature information 110 may be stored in memory 112 in the order in which feature extractor 108 generates feature information. For example, first feature information 110 associated with a first image may be stored in memory 112, followed by second feature information 110 associated with a subsequent second image, then third feature information 110 associated with a subsequent third image, and so on. Furthermore, in some examples, feature information 110 may continue to be stored in memory 112 until memory 112 is full. For example, if memory 112 includes storage space for feature information 110 associated with ten images, ten instances of feature information 110 may be stored in memory 112. Furthermore, in some examples, such as when memory 112 is full, older feature information 110 may be removed from memory 112 while new feature information 110 continues to be stored in memory 112. For example, memory 112 may include a buffer memory that continuously stores the latest feature information 110 associated with the latest image.

[0042] For more details Figure 2 Examples of generating and storing image-associated feature information in memory according to some embodiments of the present disclosure are shown. As shown, feature extractor 108 can process image data representing images 202(1)-(N) (also referred to singularly as "image 202" or plurally as "a plurality of images 202"). Figure 2 In the example, image 202 may represent object 204 (such as a vehicle), which includes at least light signals 206. Based at least on processing the first image 202 (1), feature extractor 108 may generate one or more first spatial features 208 (1) associated with the first image 202 (1), and these first spatial features 208 (1) are then stored in memory 210 (which may include and / or resemble memory 112). Next, based at least on processing the second image 202 (2), feature extractor 108 may generate one or more second spatial features 208 (2) associated with the second image 202 (2), and these second spatial features 208 (2) are then stored in memory 210. The process may then continue until processing the final image 202 (N), wherein, based at least on this processing, feature extractor 108 may generate one or more final spatial features 208 (N) associated with the final image 202 (N), and these final spatial features 208 (N) are then stored in memory 210.

[0043] Therefore, in Figure 2 In one example, memory 210 may be configured to store a specific number of spatial features 208(1)-(N) associated with image 202 (also referred to singularly as “spatial feature 208” or plurally as “multiple spatial features 208”). For example, memory 210 may be configured to store 11 instances of spatial features 208 associated with 11 images 202. In some examples, the specific number of spatial features 208 may be based on one or more factors, such as the frame rate associated with image 202, the time period, the cycle associated with the attribute being analyzed, and / or any other factor. For a first example, if memory 210 is to store images 202 for a specific time period, the frame rate associated with image 202 may be used to determine the number of images whose spatial features should be stored. For a second example, if the light signal includes a specific time period for performing a complete cycle (e.g., from an off state to an on state, and then to another off state), spatial features associated with a sufficient number of images 202 covering at least that specific time period may be stored in memory 210.

[0044] Furthermore, in some examples, feature extractor 108 may continue processing additional images to generate additional spatial features for storage in memory 210. In such examples, when storing additional spatial features in memory 210, older spatial features may first be removed from memory 210. For example, if feature extractor 108 processes a new image to generate one or more new spatial features, the first spatial feature 208(1) may be deleted from memory 210 while the new spatial features are stored in memory 112.

[0045] Return to reference Figure 1 In some examples, process 100 may include performing additional storage procedures associated with memory 112, such as before memory 112 is full of feature information 110. For a first example, process 100 may include initially storing placeholder information that does not represent features associated with the image. For example, placeholder information may include one or more predefined value vectors, such as one or more zero vectors (and / or any other value vectors). In such an example, the values ​​of the vectors may be pre-specified and / or learned, such as during training. For a second example, process 100 may include generating copies of one or more instances of feature information 110 and then storing the copies of feature information 110 in memory 112. In any of these examples, process 100 may perform such procedures to keep memory 112 full. Furthermore, when new feature information 110 is stored in memory 112, process 100 may first remove the added information (e.g., placeholder information and / or copies of feature information 110).

[0046] For more details Figures 3A-3B Examples are shown of storing additional information in memory as placeholders for spatial information associated with an image, according to some embodiments of this disclosure. For example... Figure 3A As illustrated in the examples and above diagrams, memory 302 (which may include and / or be similar to memory 112) may initially store value features 304(1)-(11) (also referred to as "value features 304"). For example, in some examples, value features 304 may include a zero vector (and / or any other value vector) stored in memory 302. Next, as illustrated below, feature extractor 108 may process the image and generate one or more spatial features 306 associated with the image. Thus, the last value feature 304(11) can be removed from memory 302, and spatial features 306 can be stored in memory 302. In some examples, the process may continue as feature extractor 108 continues to generate additional spatial features associated with additional images to be stored in memory 302.

[0047] Next, as Figure 3B As illustrated in the examples and the diagram above, feature extractor 108 can process a first image and generate one or more first spatial features 308(1) associated with the first image. Furthermore, the first spatial feature 308(1) can then be copied to generate additional instances of the first spatial feature 308(1) for storage in memory 310 (which may include and / or resemble memory 302). Next, as illustrated below, feature extractor 108 can process a second image and generate one or more second spatial features 308(2) associated with the second image. Thus, instances of the first spatial feature 308(1) can be removed from memory 310 while the second spatial feature 308(2) is stored in memory 310. In some examples, this process can continue to repeat as feature extractor 108 continues to generate additional spatial features associated with additional images for storage in memory 310.

[0048] Although Figure 3B The example shows the removal of only a single instance of the first spatial feature 308(1) from memory 310 to store only a single instance of the second spatial feature 308(2). However, in other examples, the second spatial feature 308(2) can be copied to generate additional instances of the second spatial feature 308(2). In such an example, multiple instances of the first spatial feature 308(1) can be removed from memory 310 so that multiple instances of the second spatial feature 308(2) can be stored in memory 310. In other words, different numbers of copies of different spatial features from different images can be stored in memory 310 to keep memory 310 full until the feature extractor 108 processes enough images to fill memory 310 with spatial features corresponding to the images.

[0049] Return to reference Figure 1 For example, process 100 may include: using one or more classifiers 114 to process feature information 110 (and / or placeholder information) stored in memory 112 and / or newly generated feature information 110 to determine one or more categories 116 associated with one or more objects and / or one or more attributes 118 associated with those objects. As described herein, in some examples, classifier 114 may process feature information 110 based on the occurrence of one or more events, such as the capacity of memory 112 meeting (e.g., equal to or greater than) a threshold. For example, classifier 114 may process feature information 110 based on memory 112 being full. Furthermore, classifier 114 may include some type of classifier, such as a CNN classifier, a temporal classifier, an artificial neural network classifier, a support vector machine classifier, etc. Furthermore, process 100 may use any number of classifiers 114 to perform one or more processes described herein.

[0050] For the first example, process 100 may include: using a temporal classifier 114 (such as a CNN classifier 114) configured to process feature information 110 stored in memory 112 and determine a classification 116 and / or attribute 118 associated with an object. For the second example, process 100 may include: using a temporal classifier 114 (configured to process feature information 110 stored in memory 112 to determine attribute 118) and one or more other classifiers (configured to process at least a portion of feature information 110 (e.g., instances of feature information 110 associated with frames) to determine the classification 116 of the object). While these are two example architectures for processing feature information 110, additional and / or alternative architectures may be used to process feature information 110 in other examples.

[0051] For more details Figures 4A-4B Examples of architectures associated with neural networks according to some embodiments of this disclosure are shown, which can perform one or more of the processes described herein. Figure 4A As illustrated in the example, neural network 402 may include a feature extractor 404 (such as a CNN extractor (and / or any other type of extractor)) configured to process image data 406 representing an image to generate one or more spatial features 408 associated with that image. The spatial features 408 may then be stored in memory 410. Furthermore, neural network 402 may include a first classifier 412 (such as a CNN classifier (and / or any other type of classifier)) configured to process the spatial features 408 to determine a classification 414 associated with an object.

[0052] As further shown, the neural network 402 can concatenate (416) spatial features 408 with one or more additional spatial features 418 stored in memory 410 that are associated with one or more previously processed images. The neural network 402 can then include a second classifier 420 (such as a temporal CNN classifier (and / or any other type of classifier)) configured to process the concatenated spatial features to determine one or more attributes 422 associated with the object. Therefore, the neural network 402 can generate output data 424 that at least represents the classification 414 and attributes 422 associated with the object.

[0053] Next, as Figure 4BAs illustrated in the example, neural network 426 may include a feature extractor 428 (such as a CNN extractor (and / or any other type of extractor)) configured to process image data 430 representing images to generate spatial features associated with these images, wherein the spatial features are stored in memory 432. Neural network 426 may then stitch together 434 the spatial features stored in memory 432 to generate stitched spatial features. Furthermore, neural network 426 may also include a classifier 436 (such as a temporal CNN classifier (and / or any other type of classifier)) configured to process the stitched spatial features to determine a classification 438 and one or more attributes 440 associated with an object. Thus, neural network 426 can generate output data 442 representing at least the classification 438 and attributes 440 associated with the object.

[0054] Return to reference Figure 1 In some examples, the category 116 associated with the object may include general categories such as vehicles, pedestrians, animals, traffic features, structures, etc. Additionally or alternatively, in some examples, the category 116 associated with the object may include specific categories. For the first example and for vehicles, category 116 may include cars, trucks, SUVs, vans, buses, trucks, and / or any other type of vehicle. For the second example and for traffic features, category 116 may include vehicles, traffic poles, traffic lights, traffic signals, traffic signs, road markings, lane markings, and / or any other type of traffic feature. Furthermore, the attribute 118 associated with the object may be associated with any changes that occur about the object and / or components of the object over a period of time. For example, attribute 118 may include the state of the object's light signals, the object's movement, etc.

[0055] For the first example and for light signals, attribute 118 may indicate that the state of the light signal is cyclic, on, off, and / or any other state. For the second example and for a vehicle including multiple light signals, attribute 118 may indicate that the vehicle is turning left, turning right, reversing, stopped, with emergency lighting (e.g., for emergency vehicles), hazard lighting, warning lighting, and / or performing any other action that can be indicated by the vehicle's light signals. For the third example and for traffic lights, attribute 118 may indicate that the traffic lights are green, yellow, red, and / or provide any other type of light indicator. For the fourth example and for dynamic objects, attribute 118 may indicate motion information associated with the object, such as speed, acceleration, direction of travel, and / or any other motion information. In other words, attribute 118 may include any characteristic of the object that can be detected over a period of time.

[0056] For example, return to reference Figure 2For example, classifier 114 can process spatial features 208 stored in memory 210. Based at least on this processing, classifier 114 can determine that the classification associated with the object includes a vehicle, a first attribute associated with the vehicle includes that the light signal 206 is flashing, and / or a second attribute associated with the vehicle includes a right turn (e.g., based on the flashing light signal 206). This is because, within a time period associated with image 202, image 202 can switch from depicting the light signal 206 as on to depicting the light signal 206 as off. Therefore, by stitching together and then processing the spatial features 208 stored in memory 210, classifier 114 is able to determine more accurately that the light signal 206 is flashing.

[0057] Return to reference Figure 1 For example, process 100 can continue to be repeated to continue determining the categories 116 and / or attributes 118 associated with the object. For instance, process 100 can be repeated for each image represented by image data 102, every other image represented by image data 102, every four images represented by image data 102, and / or any other interval of image data 102, to determine the categories 116 and / or attributes 118 associated with the object. This is because, in some examples, attribute 118 may change over a period of time, such as when the object performs a new manipulation. Furthermore, in some examples, process 100 can be performed to determine one or more categories 116 and / or one or more attributes 118 associated with one or more additional objects.

[0058] As described herein, one or more neural networks can be trained to perform at least a portion of the processes described herein, such as determining classifications and / or attributes associated with an object. For example, Figure 5 Examples of a process 500 for training one or more neural networks 502 to perform at least a portion of the processes described herein, according to some embodiments of the present disclosure, are shown. In some examples, the neural network 502 may be coupled with... Figure 1 The process 100 is associated with, for example, including a feature extractor 108 and a classifier 114. Additionally or alternatively, in some examples, the neural network 502 may include and / or be similar to [the process described above]. Figure 4A Example neural network 402 and / or from Figure 4B Example of a neural network 426.

[0059] As shown in the figure, neural network 502 can be trained using training input data 504. In some examples, training input data 504 can represent video of an object performing one or more actions. For the first example, training input data 504 can represent video of a vehicle (and / or other type of machine) activating a light signal. For the second example, training input data 504 can represent video of a dynamic object moving in different environments. In some examples, training input data 504 can be realistically generated (e.g., obtained using one or more machines navigating in one or more environments), synthetically generated, and / or any combination thereof.

[0060] The neural network 502 can be trained using training input data 504 and corresponding ground truth data 506. As shown, ground truth data 506 can at least represent a classification 508 and an attribute 510 associated with the object represented by the training input data 504. For a first example, if an instance of the training input data 504 represents a video depicting a vehicle using its turn signals, then ground truth data 506 can represent the object's classification 508 as "vehicle" and the object's attribute 510 as including turn signal activation and / or vehicle steering. For a second example, if an instance of the training input data 504 represents a video of a pedestrian moving in a specific direction, then ground truth data 506 can represent the classification 508 as "pedestrian" and the attribute 510 as including motion information associated with the pedestrian. Therefore, in some examples, individual instances of the training input data 504 (e.g., individual videos) can be associated with corresponding ground truth data 506. In some examples, ground truth data 506 can be real-world generated, synthetically generated, manually labeled, machine-labeled, and / or any combination thereof.

[0061] As shown in the figure, process 500 may include: neural network 502 processing training input data 504 to generate output data 512 corresponding to the training input data 504. For example, output data 512 may at least represent a predicted classification and / or predicted attribute associated with the object represented by training input data 504. Process 500 may then include: one or more training engines 514 using one or more loss functions to measure the loss (e.g., error) of output data 512 compared to ground truth data 506. For example, the loss function may measure the loss at least based on the difference between the predicted classification represented by output data 512 and classification 508 represented by ground truth data 506 and / or the difference between the predicted attribute represented by output data 512 and attribute 510 represented by ground truth data 506. Training engine 514 may then perform backpropagation computation to recursively compute the gradient of the loss function with respect to training parameters in order to update the parameters and / or weights of neural network 502, as indicated by the arrows from training engine 514 to neural network 502. For example, training engine 514 can update the parameters and / or weights of the feature extractor and / or classifier of neural network 502.

[0062] As described herein, in some examples, one or more techniques may be used to improve the training of neural network 502. For a first example, neural network 502 may be configured to process images of video at a specific frame rate when deployed within a machine. In some examples, the frame rate may be based on target hardware in the machine, such as one or more image sensors that generate the image data being processed by neural network 502. Therefore, during training, temporal random jitter may be used to feed images to neural network 502 at one or more additional frame rates (such as using a range of frame rates near the actual frame rate). For a second example, randomly discarding images may also be used, such as to simulate problems that might arise with the image sensors and / or processing when executed on a machine.

[0063] Now for reference Figure 6 and Figure 7 Each block of methods 600 and 700 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be performed using one or more processors (e.g., but not limited to, the processors described herein) that execute instructions stored in one or more memories or memory systems. In some embodiments, the computer process may also embody computer-usable instructions stored on a computer storage medium. Methods 600 and 700 may be provided by standalone applications, services or managed services (alone or in combination with other managed services), application programming interfaces (APIs), and / or plug-ins to other products. Furthermore, methods 600 and 700 are described by way of example regarding... Figure 1The methods 600 and 700 are described herein. However, these methods 600 and 700 may be additionally or alternatively performed by any system or combination of systems, including but not limited to the systems described herein.

[0064] Figure 6 A flowchart of a method 600 for determining attributes associated with an object using time information according to some embodiments of the present disclosure is shown. Method 600 may include, at block B602, generating one or more spatial features associated with one or more images using one or more feature extractors and based at least on image data representing one or more images depicting the object. For example, feature extractor 108 may process image data (and / or, in some examples, processed image data 106) to generate feature information 110 representing spatial features associated with the images. As described herein, in some examples, feature extractor 108 may include a CNN extractor (and / or any other type of extractor) configured to perform one or more of the processes described herein.

[0065] Method 600 at block B604 may include storing one or more spatial features in memory. For example, spatial features may be stored in memory 112. As described herein, in some examples, such as when memory 112 is not yet full of spatial features associated with an image, padding may be used to store predefined features (e.g., value features) in memory 112. Additionally or alternatively, in some examples, such as when memory 112 is still not full of spatial features, copies of the spatial features may be generated and stored in memory 112.

[0066] Method 600 at box B606 may include: using one or more temporal classifiers and based at least on one or more spatial features from memory to determine one or more attributes associated with an object. For example, classifier 114 may process spatial features stored in memory 112 to determine attribute 118 associated with the object. In some examples, classifier 114 may also process spatial features generated by feature extractor 108 and / or spatial features stored in memory 112 to determine classification 116 associated with the object. As described herein, in some examples, classifier 114 may process spatial features based on the occurrence of one or more events (such as the capacity of memory 112 meeting a threshold).

[0067] Method 600 at box B608 may include causing the machine to perform one or more operations based on at least one or more attributes. For example, the machine may determine one or more trajectories to travel based on attribute 118. For a first example, if attribute 118 indicates the state and / or movement (such as an object turning, reversing, and / or stopping) of a light signal associated with a vehicle, the operation may be associated with traveling along a trajectory to avoid a collision with that object. For a second example, if attribute 118 indicates the state of a light signal associated with a traffic feature (such as a traffic light or traffic sign), the operation may be associated with traveling along a trajectory that causes the machine to follow a rule associated with that state (e.g., stopping when the traffic light is red).

[0068] Figure 7 A flowchart illustrating a method 700 for determining attributes associated with an object using stored feature information, according to some embodiments of the present disclosure, is shown. Method 700 may include, at block B702, storing feature information associated with one or more images in memory. For example, feature extractor 108 may process image data 102 representing images to generate feature information 110. Feature information 110 may then be stored in memory 112. As described herein, in some examples, additional padding information, such as value vectors (e.g., zero vectors, etc.), may be stored in memory 112. Additionally or alternatively, in some examples, one or more copies of feature information 110 may be generated and stored in memory 112.

[0069] Method 700 at box B704 may include determining that the capacity associated with the memory meets a threshold. For example, it may be determined that the capacity of memory 112 is equal to or greater than the threshold. In some examples, the threshold may include that memory 112 is full.

[0070] Method 700 at box B706 may include: using a neural network and at least based on feature information, at least based on a capacity threshold, to determine one or more attributes associated with one or more objects represented by one or more images. For example, based on a capacity threshold of memory 112, the neural network may use classifier 114 to process feature information 110 stored in memory 112 to determine attributes 118 associated with the object. Furthermore, in some examples, the neural network may also use classifier 114 to determine one or more categories 116 associated with the object.

[0071] The systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles connected to one or more trailers, aircraft, ships, shuttles (e.g., robotaxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones, and / or other types of vehicles. Furthermore, the systems and methods described herein can be used for a wide range of purposes, including, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets (e.g., NVIDIA's Omniverse), cloud computing, and / or any other suitable application.

[0072] The disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines, etc.), systems implemented using robots, aviation systems, medical systems, marine systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing language models (such as large language models (LLM), visual language models (VLM), and / or multimodal language models), systems using or deploying one or more inference microservices, systems containing one or more machine learning models deployed in services or microservices and OS-level virtualization packages (e.g., containers), systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0073] Example autonomous or semi-autonomous machines

[0074] Figure 8AExamples of sensor positions with corresponding fields of view or sensing fields of view of autonomous or semi-autonomous vehicles 800a, autonomous mobile robots (AMRs) 800b, and humanoid robots 800c, according to some embodiments of this disclosure. While three types of machines 800 are shown, this is not intended to be limiting, and the machines 800 described herein may include vehicles, cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police or emergency vehicles, ambulances, boats, construction vehicles, underwater vehicles, robots (e.g., AMRs, humanoid robots, robotic arms, end effectors, forklifts, etc.), drones, aircraft, vehicles coupled to trailers (e.g., semi-trailer tractors for hauling goods), and / or another type of vehicle or machine (e.g., driverless and / or vehicles or machines accommodating one or more passengers). In some cases, vehicles 800a, AMRs 800b, humanoid robots 800c, and / or other machine types may be collectively referred to herein as machines 800.

[0075] Regarding Vehicle 800A, autonomous and semi-autonomous vehicles are typically described by automation levels, defined by the National Highway Traffic Safety Administration (NHTSA) (a division of the U.S. Department of Transportation) and the Society of Automotive Engineers (SAE) in their "Classification and Definition of Terms Related to Driving Automation Systems for Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 800 may have one or more functions that meet Level 3 through Level 5 of autonomous driving. For example, Vehicle 800 may be able to provide driver assistance (Level 1), partial automation (Level 2, Level 2+, Level 2++), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the specific implementation. The term “autonomy” as used herein can include any and / or all types of autonomy of Machine 800 or other machines, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, provision of auxiliary autonomy, semi-autonomy, primary autonomy or other specified autonomy.

[0076] about Figure 8AThe sensors and their respective fields of view (not shown for clarity) or sensing fields (not shown for clarity) are an example embodiment and are not intended to be limiting. Although not shown, each sensor may have a corresponding field of view (e.g., a 360-degree field of view around camera 868D, a 180-degree field of view around wide-angle camera 868B, a 360-degree sensing field of view of LiDAR sensor 864, etc.). For example, only a subset of the sensors shown may be included, additional sensors may be included, alternative sensors may be included, the number of each sensor mode may be different, the sensor modes may be different (e.g., LiDAR or RADAR may not be included, SONAR, thermal sensors, etc. may be included), and the sensor locations may differ from those shown on vehicles 800a, AMR 800b, and / or humanoid robots 800c, etc. For example, for vehicle 800a, the location, number, modes, and / or other sensor information may differ depending on the type (e.g., SUV, truck, car, robot, motorcycle, etc.), size (e.g., 18-wheeler, transport vehicle, small car, etc.), and associated functions (e.g., L2 vs. L5). Similarly, for the AMR 800b and / or the humanoid robot 800c, the shape, size, purpose, implementation method, model, etc., can determine the number and type of sensors used.

[0077] like Figure 8A As shown, autonomous or semi-autonomous vehicles 800A, AMR 800B, and humanoid robots 800C may include different sensor types, numbers, and locations. As a non-limiting example, vehicle 800A may include twelve cameras 868, such as a front wide-angle camera (e.g., 120-degree field of view (FOV)), a front telephoto camera (e.g., 30-degree FOV), a rear-side left camera (e.g., 70-degree FOV), a rear-side right camera (e.g., 70-degree FOV), a front fisheye camera (e.g., 200-degree FOV), a rear fisheye camera (e.g., 200-degree FOV), a left fisheye camera (e.g., 200-degree FOV), a right fisheye camera (e.g., 200-degree FOV), a front telephoto satellite camera (e.g., 30-degree FOV), a rear telephoto camera (e.g., 30-degree FOV), a cross-left camera (e.g., 120-degree FOV), and a cross-right camera (e.g., 120-degree FOV). In this embodiment, the camera 868 may use a Gigabit Multimedia Serial Link (GMSL) interface (e.g., GMSL2) as input / output (I / O).

[0078] In some embodiments, although Figure 8AAs not shown, vehicle 800A may include an in-cabin occupant and / or driver monitoring system, which may include various sensors. For example, in-cabin sensors may include various cameras 868, such as a driver monitoring camera (e.g., located in front of the driver's seat and facing the driver's seat at a 55-degree FOV), a front occupant monitoring camera (e.g., located in front of the front occupant seat and facing the front occupant seat at a 190-degree FOV), and a rear occupant monitoring camera (e.g., located in front of the rear occupant seat and facing the rear occupant seat at a 190-degree FOV). Similar to external cameras 868, in embodiments, internal cameras 868 may use a GMSL (e.g., GMSL2) interface for I / O.

[0079] As another non-limiting example, vehicle 800A may also include nine RADAR sensors 860. For example, vehicle 800A may include a front center imaging RADAR sensor (e.g., 120-degree FOV or sensing field), a left front corner RADAR sensor (e.g., 160-degree FOV or sensing field), a right front corner RADAR sensor (e.g., 160-degree FOV or sensing field), a right rear corner RADAR sensor (e.g., 160-degree FOV or sensing field), a left RADAR sensor (e.g., 160-degree FOV or sensing field), a right RADAR sensor (e.g., 160-degree FOV or sensing field), a left rear RADAR sensor (e.g., 50-degree FOV or sensing field), and a right rear RADAR sensor (e.g., 50-degree FOV or sensing field). In embodiments, the RADAR sensors 860 may use an Ethernet interface as I / O.

[0080] As a non-limiting example, the vehicle 800A may also include twelve ultrasonic sensors 862. Figure 8A As shown, the ultrasonic sensor can be placed along the front and rear bumpers of the vehicle 800A and along the sides of the vehicle 800A, and can be used to detect objects (static and dynamic) close to the vehicle 800A. In some embodiments, the ultrasonic sensor 862 can use a DS13 interface as I / O.

[0081] As a non-limiting example, the vehicle 800A may also include a LiDAR sensor 864, such as a front-center LiDAR sensor (e.g., a 120-degree horizontal FOV or sensing field and a 30-degree vertical FOV or sensing field). In some embodiments, such as when using additional or alternative LiDAR sensors, the LiDAR sensor may have different horizontal and vertical fields of view or sensing fields. For example, the LiDAR sensor 864 may include a 360-degree horizontal FOV or sensing field (e.g., located in a rotating LiDAR sensor) and a 90-degree vertical FOV or sensing field. In some embodiments, the LiDAR sensor 864 may use an Ethernet interface as I / O.

[0082] As a non-limiting example, the Autonomous Mobile Robot (AMR) 800B may include three LiDAR sensors 864. For example, the topmost LiDAR sensor 864 may include a beam or 3D LiDAR sensor (e.g., a 360-degree horizontal and 90-degree vertical FOV or sensing field), and the front and rear LiDAR sensors may include planar or 2D LiDAR sensors (e.g., a 180-degree horizontal FOV or sensing field).

[0083] As a non-limiting embodiment, the AMR 800B may further include eight cameras 868, such as a front stereo camera (e.g., 120-degree FOV), a rear stereo camera (e.g., 120-degree FOV), a left stereo camera (e.g., 120-degree FOV), a right stereo camera (e.g., 120-degree FOV), a front fisheye camera (e.g., 202-degree ± 3-degree FOV), a rear fisheye camera (e.g., 202-degree ± 3-degree FOV), a left fisheye camera (e.g., 202-degree ± 3-degree FOV), and a right fisheye camera (e.g., 202-degree ± 3-degree FOV).

[0084] The AMR 800B may also include a charging port, charging port contacts, status indicators, one or more (e.g., four) RGB LEDs, one or more IMU sensors 866, a magnetometer, and a barometer. The AMR 800B is capable of high-precision time synchronization between sensors using hardware timestamps and PTPs over Ethernet with sensor acquisition times of less than 10 microseconds. In an embodiment, the AMR 800B provides simultaneous camera capture within 100 microseconds of a single hardware trigger on all cameras 868, and can write sensor captures to disk at a rate of 4 GB / s for writing packets (e.g., writing to ROSbags of the Robot Operating System (ROS)). Therefore, the AMR 800B is capable of running ROS (e.g., NVIDIA's IsaacROS), can be remotely operated (as described herein), can map the environment, and can navigate the environment using vision cameras 868, LiDAR 864, and / or other sensor types or modalities.

[0085] The humanoid robot 800C may include (as a non-limiting example) a LiDAR sensor 864. For example, the LiDAR sensor 864 may include a beam or a 3D LiDAR sensor (e.g., a 360-degree horizontal and 90-degree vertical FOV or sensing field), or it may include a planar or 2D LiDAR sensor (e.g., a 180-degree horizontal FOV or sensing field).

[0086] As a non-limiting embodiment, the humanoid robot 800C may also include four cameras 868, such as a front stereo camera (e.g., 120-degree FOV), a rear stereo camera (e.g., 120-degree FOV), a front fisheye camera (e.g., 202-degree ± 3-degree FOV), and a rear fisheye camera (e.g., 202-degree ± 3-degree FOV).

[0087] As a non-limiting embodiment, the humanoid robot 800C may also include four ultrasonic sensors 862, such as a left arm ultrasonic sensor, a right arm ultrasonic sensor, a left leg ultrasonic sensor, and a right leg ultrasonic sensor.

[0088] The humanoid robot 800C may also include any number of actuators, such as those allowing control and manipulation of joints. For example, the humanoid robot 800C may include actuators that allow for various degrees of freedom (DoF) depending on the design. In a non-limiting embodiment, the humanoid robot 800C may have a total of 40 degrees of freedom (DoF) (e.g., 6DoF x2 for arms, 6DoF x2 for hands, 6DoF x2 for legs, 2DoF for the torso, and 2DoF for the neck). Actuators can convert energy into physical motion, thereby allowing actions such as joint movement, locomotion, and grasping / manipulation. For example, motors and servos can be used to perform joint movements to control the rotation of joints in an arm or manipulator and allow for reaching, grasping, and manipulating objects. Locomotion can be achieved by moving through the environment using wheels, tracks, or other locomotion devices (robot legs). Grasping and manipulation can be performed using end effectors or hands / fingers, which can be equipped with actuators to grasp objects, apply forces, and perform specific tasks. In some examples, the humanoid robot 800C may include position and orientation sensors, such as encoders, gyroscopes, etc., to determine the robot 800C's position in space, thereby enabling position determination and motion tracking. In embodiments, the humanoid robot 800C may include force and pressure sensors to detect environmental interactions, enabling the robot 800C to grasp objects with appropriate force and avoid obstacles along its path. Perception sensors (e.g., cameras, LiDAR, RADAR, ultrasound, sonar (SONAR), etc.) may be used in conjunction with tactile sensors to enable the robot 800C to perceive objects, shapes, and textures, and to know when to begin and stop touching (in conjunction with force sensors that regulate the force used during touching). As a non-limiting example, the humanoid robot 800C may have a height of approximately 1-2 meters (e.g., 1.7 meters or 5 feet 6 inches), a weight of 50-70 kilograms, be able to move at speeds of 8 km / h or higher, and be able to carry a payload of 20-100 kilograms, depending on the system design and requirements.

[0089] In embodiments, the humanoid robot 800C may include a dialogue system—such as a dialogue system driven by a language model (e.g., LLM, VLM, MMLM, VLA, etc.)—to help understand the environment, reason, and communicate with humans, animals, devices, and / or other robots, and / or make planning, control, and navigation decisions. Therefore, in addition to performing various tasks, the humanoid robot 800C can also use onboard sensors, microphones, and speakers to understand speech, audio, and visual cues, while also being able to communicate with the environment.

[0090] Referring to camera 868 of machine 800, the camera type of camera 868 may include, but is not limited to, digital cameras applicable to components and / or systems of machine 800. For vehicle 800a implementation, camera 868 may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may have any image capture rate, such as 30 frames per second (fps), 60 fps, 120 fps, 240 fps, etc., depending on the embodiment. The camera may use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red colorless colorless colorless (RCCC) color filter array, a red colorless colorless blue (RCCB) color filter array, a red blue green colorless (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a colorless pixel camera (e.g., a camera with an array of RCCC, RCCB, and / or RBGC color filters) can be used to improve light sensitivity.

[0091] The field of view includes cameras (e.g., front-facing cameras) in front of the machine 800, which can be used for surround view to help identify the path and obstacles ahead, and, with the help of one or more controllers 836 and / or control SoCs, provide information crucial for generating an occupancy grid and / or determining preferred machine movement, trajectory, and / or path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including lane departure warning (“LDW”), adaptive cruise control (“ACC”), and / or other functions such as traffic sign recognition.

[0092] Various cameras can be used in front-mounted configurations, including, for example, monocular camera platforms that include complementary metal-oxide-semiconductor (“CMOS”) color imagers. Another example could be a wide-angle camera 868B, which can be used to perceive objects entering the field of view from the periphery (e.g., pedestrians, warehouse vehicles, other robots, pedestrian traffic, or bicycles). Furthermore, any number of long-range cameras 868E (e.g., a pair of long-range stereo cameras) can be used for depth-based object detection, especially for objects for which neural networks have not yet been trained. The long-range camera 868E can also be used for object detection and classification, as well as basic object tracking.

[0093] Any number of stereo cameras 868A may also be included in front-mounted and / or other (e.g., rear-mounted) configurations. In at least one embodiment, one or more stereo cameras 868A may include an integrated control unit comprising a scalable processing unit that may provide programmable logic (“FPGA”) and a multi-core microprocessor with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the machine 800 environment, including distance estimates of midpoints in the image (e.g., parallax or depth images). Alternative stereo cameras 868A may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip that can measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 868A may be used in addition to those described herein, or as an alternative to those described herein. For example, in some embodiments, stereo depth estimation can be performed using a camera other than a stereo camera (e.g., two monocular cameras with at least partially overlapping fields of view).

[0094] Cameras with a field of view including portions of the side environment of machine 800 (e.g., side-view cameras) can be used, for example, for surround view, to provide information for creating and updating occupancy grids, and to generate side collision warnings and / or indicate to AMR 800B or humanoid robot 800C, for example, the presence of objects, features, and / or people on the side. For example, surround cameras 868D can be mounted on machine 800. Surround cameras 868D can include wide-angle cameras 868B, fisheye cameras, 360-degree cameras, etc. For example, four fisheye cameras can be mounted on the front, rear, and sides of machine 800. In an alternative arrangement, machine 800 can use three surround cameras 868D (e.g., left, right, and rear), and can utilize one or more other cameras (e.g., front-facing cameras) as a fourth surround camera.

[0095] A camera 868 (e.g., a rear-view camera) having a field of view including a portion of the environment behind the machine 800 can be used to understand objects, features, people, and / or other information behind the machine 800, such as for parking assistance, surround view, rear collision warning, planning, control, and navigation determination, and / or to create and update occupancy grids, BEV images representing the environment, height maps, etc. A wide variety of cameras 868 can be used, including but not limited to those also suitable for use as front cameras (e.g., long-range and / or mid-range cameras 868E, stereo cameras 868A, infrared cameras 868C, etc.), rear cameras, side cameras, downward cameras, upward cameras, and / or similar cameras 868, as described herein.

[0096] Similarly, for LiDAR sensor 864, RADAR sensor 860, ultrasonic sensor 862 and / or other sensor modes or types, the location and placement of the sensors and their corresponding fields of view or sensing fields can be determined based on the use case, implementation or design of the particular machine 800.

[0097] For example, machine 800 includes a RADAR sensor 860, which can be used by machine 800 for long-range object detection, even in dark and / or inclement weather conditions. In embodiments, the RADAR functional safety level may be ASIL B. RADAR sensor 860 can be controlled and accessed for object tracking data using CAN and / or bus 802 (e.g., for transmitting data generated by RADAR sensor 860), and in some examples, raw data can be accessed using Ethernet. Various types of RADAR sensors can be used. For example, but not limited to, RADAR sensor 860 can be suitable for front, rear, and side radar applications. In some examples, a pulse Doppler RADAR sensor is used.

[0098] The RADAR sensor 860 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range lateral coverage, etc. In some examples, long-range radar can be used for adaptive cruise control (ACC) functions. A long-range RADAR system can provide a wide field of view, for example, within a 250m range, achieved through two or more independent scans. The RADAR sensor 860 can help distinguish between static and moving objects; ADAS systems can use it for emergency braking assistance and forward collision warning, and robots can use it to detect dynamic objects in various environments (e.g., low-light or no-light environments). A long-range RADAR sensor can include a single static multimodal RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record the machine 800's surroundings at higher speeds while minimizing peripheral interference (e.g., traffic from adjacent lanes). The other two antennas expand the field of view, enabling rapid detection of objects entering or leaving the machine's direct path (e.g., lanes).

[0099] Mid-range RADAR systems can include, for example, a range of up to 960m (front) or 80m (rear), and a field of view of up to 42 degrees (front) or 150 degrees (rear). Short-range RADAR systems can include, but are not limited to, RADAR sensors designed to be mounted at both ends of a side surface (e.g., rear bumper), allowing continuous monitoring of blind spots behind and beside a machine (e.g., vehicle, robot, etc.) using two beams. Therefore, short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0100] Machine 800 may also include an ultrasonic sensor 862. The ultrasonic sensor 862 may be located at the front, rear, and / or sides of machine 800 and may be used to assist near-field sensing, such as for parking assistance, collision avoidance (e.g., for robot parts), and / or to create and update occupancy grids, evidence mesh maps (EGMs), height maps, BEV images, and / or other representations of objects and features in the machine 800 environment. Multiple ultrasonic sensors 862 may be used, and different ultrasonic sensors 862 may be used for different detection ranges (e.g., 2.5 m, 4 m). For example, the ultrasonic sensor 862 may operate at a functional safety level of ASIL B.

[0101] Machine 800 may include a LiDAR sensor 864. The LiDAR sensor 864 can be used for object and feature detection, pedestrian and other robot detection, emergency braking, collision avoidance, simultaneous localization and mapping (SLAM), free space detection, and / or other functions. In embodiments, the LiDAR sensor 864 may be of functional safety level ASIL B. In some examples, machine 800 may include multiple LiDAR sensors 864 (e.g., two, four, six, etc.), which can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0102] In some examples, the LiDAR sensor 864 may be able to provide a 360-degree field of view of objects and their distances. For example, a commercially available LiDAR sensor 864 may have an advertised range of approximately 900m, an accuracy of 2cm-3cm, and support for 900Mbps Ethernet connectivity. In some examples, one or more non-protruding LiDAR sensors 864 may be used. In such examples, the LiDAR sensor 864 can be implemented as a small device that can be embedded in the front, rear, side, top, and / or corner of a machine 800. In such examples, the LiDAR sensor 864 can provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, with a range of up to 200m even for low-reflectivity objects. A front-mounted LiDAR sensor 864 can be configured with a horizontal field of view between 45 and 135 degrees.

[0103] In some examples, LiDAR technology, such as 3D flash LiDAR, can also be used. 3D flash LiDAR uses a laser flash as a transmission source, illuminating the environment around a vehicle up to approximately 200m away. The flash LiDAR unit includes a receiver that records the laser pulse transmission time and the reflected light on each pixel, which in turn corresponds to the distance from the vehicle to the object. Flash LiDAR allows for the generation of highly accurate and distortion-free images of the surrounding environment with each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of machine 800. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras with no moving parts other than a fan (e.g., non-scanning LiDAR devices). Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of 3D distance point clouds and co-registration intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, LiDAR sensor 864 may be less susceptible to motion blur, vibration, and / or shock.

[0104] Figure 8BThis is an illustration of the location of sensors and components in an example autonomous or semi-autonomous vehicle 800A (also referred to herein as "vehicle 800", "self-vehicle 800", "self-machine 800", or "machine 800") according to some embodiments of this disclosure. While vehicle 800A is shown in the figure, this is not intended to be limiting, and similar components and / or sensors may be included on any other machine type without departing from the scope of this disclosure. For example, similar sensors and / or components may be used in vehicles, cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police cars, ambulances, ships, construction vehicles, underwater vehicles, robots (e.g., AMRs, humanoid robots, robotic arms, end effectors, forklifts, etc.), drones, aircraft, vehicles coupled to trailers (e.g., semi-trailers for hauling goods), and / or another type of vehicle or machine (e.g., driverless and / or capable of accommodating one or more passengers).

[0105] Figure 8CThis is a block diagram of an example system architecture for a machine 800 (e.g., an autonomous or semi-autonomous vehicle 800A, an autonomous mobile robot (AMR) 800B, a humanoid robot 800C, and / or other types of machines) according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are presented as examples only. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of the arrangements shown, and some elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components or together with other components, and can be implemented in any suitable combination and location (e.g., on a local device, vehicle, or edge machine, in the field (e.g., a locally hosted server), in a remote location (e.g., in one or more computing or server devices in one or more data centers in the cloud) and / or other locations). The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, microcontroller, embedded processor, digital signal processor (DSP), image signal processor (ISP), physical processing unit (PPU), field-programmable gate array (FPGA), accelerators (e.g., deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural network accelerator (NNA) and / or neural processing unit (NPU), programmable vision accelerator (PVA), optical flow accelerator (OFA), etc.), application-specific integrated circuit (ASIC), data processing unit (DPU), quantum processor, etc.) that execute instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 8A to 8E Example machine 800, Figure 9 Example calculation ecosystem 900, Figure 10 Example of a generative language model system 1000 and / or Figure 11 The example computing device 1100 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the operation.

[0106] Figure 8CEach component, feature, and system of machine 800 is shown connected via bus 802 (or referred to as "machine communication network 802" or simply "communication network 802"). Bus 802 may include a Controller Area Network (CAN) data interface (or referred to herein as "CAN bus"). CAN can be a network within machine 800 used to help control various features and functions of machine 800, such as brake actuation, acceleration, braking, steering, windshield wipers, etc. CAN bus can be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CANID). CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button position, and / or other vehicle status indicators. CAN bus may conform to ASIL B standard. In some embodiments, in addition to or as an alternative to the CAN bus, bus 802 may include FlexRay, embedded buses (e.g., SPI, I2C), local interconnect links (LIN), NVIDIA's NVLink, USB (2.0, 3.0 and above), radio frequency (RF), Ethernet (e.g., 10BASE / 100BASE, 1000BASE, 10G, etc.), and / or other communication protocols or functions. Furthermore, while a single line is used to represent bus 802, this is not limiting. For example, there can be any number of buses 802, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 802 may be used to perform different functions and / or for redundancy. For example, a first bus 802 may be used for collision avoidance, while a second bus 802 may be used for actuation control. In any example, each bus 802 may communicate with any component of machine 800, and two or more buses 802 may communicate with the same component. In some examples, each computer or computing engine within each SoC 804, each controller 836, and / or machine 800 can access the same input data (e.g., input from sensors in the machine 800) and can be connected to a common bus, such as the CAN bus.

[0107] Machine 800 may include components such as a chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, batteries, side mirrors, and / or other parts of the vehicle or machine. Machine 800 may include a propulsion system 850, such as an internal combustion engine, a hybrid power plant, an all-electric motor, a hydrogen fuel cell engine, and / or another type of propulsion system. Propulsion system 850 may be connected to the drivetrain of machine 800, which may include a transmission to enable propulsion of machine 800. Propulsion system 850 may be controlled in response to a signal received from throttle / accelerator 852.

[0108] Steering system 854 may include a steering wheel and / or other steering mechanisms (e.g., remote steering and / or local steering) for steering machine 800 (e.g., along a desired path or route) while propulsion system 850 is in operation (e.g., when the vehicle is traveling). Steering system 854 may receive signals from steering actuator 856. In some embodiments, a steering wheel or other steering mechanism may not be included, for example, for machine 800 capable of fully automated (e.g., level 5) functionality.

[0109] The brake sensor system 846 can be used to operate the vehicle brakes in response to signals received from the brake actuator 848 and / or the brake sensor.

[0110] Machine 800 may include one or more controllers 836, such as those described herein. Figure 8AThe controller 836 is described. It can be used for a variety of functions and can be coupled to any of the various other components and systems of the machine 800. For example, the controller 836 can be used to control the machine 800, artificial intelligence performed on the machine 800, infotainment on the machine 800, etc. For example, one controller 836 can be used for some or all of the functions, or different controllers 836 can be used for different functions, for example, to ensure availability and safety separation between various controllers for different tasks. For example, the controller 836 can use a system-calculated plan (e.g., the path or trajectory of vehicle 800A or AMR 800B, or the movement, component trajectory, motion position or displacement, etc. of joints or components (e.g., manipulators, end effectors, limbs, hands, fingers, legs, feet, etc.) of humanoid robot 800C to control the machine 800 in the environment. In some cases, the controller 836 may include a proportional-integral-derivative (PID) controller, a fuzzy logic controller, a neural controller (e.g., a controller embodied as one or more neural networks), a force control controller, a programmable logic controller (PLC), and / or other types of controllers. For example, in the humanoid robot 800C, the controller 836 can act as the brain, responsible for analyzing sensor data, making decisions, and sending commands to the actuators. The controller 836 may include a low-level controller that handles basic motor control, ensuring accurate and precise movement of the individual joints and actuators. The controller 836 may also include a high-level controller to coordinate multiple actuators and sensors, plan complex movements, and adapt to changing environments.

[0111] In some embodiments, controller 836 may include an artificial intelligence controller that can use AI algorithms (e.g., DNN, MLM, etc.) to learn, make decisions, and autonomously perform tasks of machine 800. In some embodiments, controller 836 may use a fixed open-loop control algorithm and not adjust its actions according to the environment. In other embodiments, closed-loop control may be used, which incorporates a feedback mechanism to monitor the robot's performance and make necessary adjustments. In examples, controller 836 may implement reactive control to directly respond to sensor inputs, enabling rapid reflexive actions and real-time changes. Furthermore, in some examples, deliberate control may be implemented, using internal models and planning algorithms to generate advanced actions, which may be suitable for complex tasks requiring reasoning, decision-making, and long-term planning.

[0112] The controller 836 may include one or more system-on-chip (SoC) 804 ( Figure 8C and Figure 8DThe controller 836 may provide signals (e.g., signals representing commands or messages) to one or more components and / or systems of the machine 800, including CPUs, GPUs, accelerators, etc. Although the controller 836 is listed separately from the SoC 804, this is not intended to be limiting, and in some embodiments, one or more components of the SoC 804 may perform the operations of the controller 836. For example, the controller may send signals to operate machine brakes via one or more brake actuators 848, to operate steering system 854 via one or more steering actuators 856, to operate propulsion system 850 via one or more throttle / accelerators 852, and so on. The controller 836 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous or semi-autonomous navigation and movement and / or assist human operators using the machine 800. Controller 836 may include a first controller 836 for autonomous control and navigation functions, a second controller 836 for functional safety functions, a third controller 836 for artificial intelligence functions (e.g., computer vision), a fourth controller 836 for infotainment functions, a fifth controller 836 for redundancy in emergency situations, and / or other controllers. For example, hardware for safety monitoring and other safety functions (e.g., functional safety islands) may be discrete or partitioned (physically or by processing separation) relative to hardware for processing sensor data to make perception and vehicle control decisions. Similarly, hardware for controlling in-vehicle infotainment and / or in-cabin monitoring (e.g., controllers, SOCs, etc.) may be separate or independent from hardware for vehicle perception and control. In some examples, a single controller 836 may handle two or more of the above functions, two or more controllers 836 may handle a single function, and / or any combination thereof.

[0113] The controller 836 may provide signals for controlling one or more components and / or systems of the machine 800 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data can be received from, for example, but not limited to, Global Navigation Satellite System (“GNSS”) sensor 858 (e.g., Global Positioning System sensor), RADAR sensor 860, ultrasonic sensor 862, LiDAR sensor 864, Inertial Measurement Unit (IMU) sensor 866 (e.g., accelerometer, gyroscope, magnetic compass, magnetometer, etc.), microphone 896, camera 868 (e.g., stereo camera 868A, wide-angle camera 868B (e.g., fisheye camera), infrared camera 868C, surround camera 868D (e.g., 360-degree camera), long-range and / or medium-range camera 868E and / or other types of cameras), speed sensor 844 (e.g., for measuring the speed of machine 800), vibration sensor 842, steering sensor 840, braking sensor (e.g., as part of braking sensor system 846), actuators and / or other types of sensors.

[0114] One or more controllers 836 may receive input (e.g., represented by input data) from the instrument panel 832 of the machine 800 and provide output (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 834 (e.g., screen, head-up display, mirror display, facial display, robot display, etc.), sound alarms, loudspeakers, speakers, and / or other components of the machine 800. Outputs may include parameters such as machine speed, rate, time, and... Figure 8C The system may display map data corresponding to map 822 (e.g., from a navigation map, a standard definition (SD) map, a high definition (“HD”) map, etc.), location data (e.g., the location of machine 800, such as its location on map 822), direction, the location of other vehicles (e.g., occupancy map, elevation map, bird's-eye view (BEV) image, grid, etc.), information about objects perceived by the system and their states, system status information, and so on. For example, HMI display 834 may display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.), and / or information about driving maneuvers that the vehicle has performed, is performing, or will perform (e.g., changing lanes now, exiting from exit 34B in two miles, etc.).

[0115] Machine 800 may include one or more System-on-a-Chip (SoC) 804 (in Figure 8D(described in more detail below). SoC 804 may include CPU 806, GPU 808, processor 810, cache 812, accelerator 814, data storage 816, and / or other components and features. SoC 804 can be used to process and provide data for various operations, such as navigation, planning, reasoning, inference, perception, control, and / or actuation operations of machine 800 on various platforms and systems. For example, SoC 804 can process real-time perception data (e.g., from cameras, LiDAR, RADAR, ultrasound, etc.) and map data corresponding to one or more maps 822 (e.g., HD maps, SD maps, navigation maps, occupancy maps, etc.) to perform or assist in performing various operations of machine 800. When using maps and / or AI, maps and / or AI (e.g., model parameter updates, fine-tuning, etc.) are transmitted via network interface 824 from one or more servers (e.g., [server name missing]). Figure 8E The server 878 (e.g., one or more servers in a cloud-based data center) is refreshed and / or updated.

[0116] Despite Figures 8A to 8E The SoC 804 is shown; however, additional or alternative components and / or architectures, such as multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiPs), field-programmable gate arrays (FPGAs), heterogeneous integration (HI), and single-board computers (SBCs), may be used without departing from the scope of this disclosure. For example, depending on the type of machine 800, the purpose of machine 800, the model of machine 800, and the capabilities required by machine 800, one or more SoCs 804s and / or alternative architectures and / or components may be used to meet a particular implementation.

[0117] Machine 800 may include CPU 818 (e.g., a discrete CPU or dCPU) which may be coupled to SoC 804 via a high-speed interconnect (e.g., PCIe). CPU 818 may include, for example, an x86 processor. CPU 818 may be used to perform any of a variety of functions, including arbitrating potentially inconsistent results between ADAS sensors and SoC 804, and / or monitoring the status and health of controller 836 and / or infotainment SoC 830.

[0118] Machine 800 may include GPU 820 (e.g., a discrete GPU or dGPU) which may be coupled to SoC 804 via a high-speed interconnect (e.g., NVIDIA's NVLink). GPU 820 may provide additional artificial intelligence capabilities, such as by executing redundant and / or different neural networks, and may be used to train and / or update neural networks based on inputs (e.g., sensor data) from sensors from Machine 800.

[0119] The machine 800 may also include a network interface 824, which may include one or more wireless antennas 826 and / or modems (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 824 can be used to establish wireless connectivity with the cloud (e.g., with server 878 and / or other network devices) via the Internet, and with other vehicles and / or computing devices (e.g., passenger client devices). For communication with other vehicles, direct links and / or indirect links (e.g., across networks and via the Internet) can be established between the two vehicles. A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide the machine 800 with information about vehicles near the machine 800 (e.g., vehicles in front, to the side, and / or behind the machine 800). This functionality can be part of the machine 800's cooperative adaptive cruise control function.

[0120] Network interface 824 may include a SoC that provides modulation and demodulation functions and enables controller 836 to communicate over a wireless network. Network interface 824 may include a radio frequency (RF) front-end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. Frequency conversion can be performed by well-known processes and / or using superheterodyne processes. In some examples, the RF front-end functionality may be provided by a separate chip. For example, network interface 824 may be able to communicate via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multicarrier (“CDMA2000”), 5G, 6G, and / or other cellular and / or wireless communication standards. The wireless antenna 826 can also enable communication between objects in the environment (such as vehicles, mobile devices, etc.) using local area networks (such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc.) and / or low power wide area networks (“LPWAN”) (such as LoRaWAN, SigFox, etc.).

[0121] Machine 800 may also include data memory 828, which may include off-chip (e.g., outside of SoC 804) storage. Data memory 828 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, Flash, hard disk and / or other components and / or devices capable of storing at least one bit of data.

[0122] The machine 800 may also include a GNSS sensor 858. The GNSS sensor 858 (e.g., a GPS, an auxiliary GPS sensor, a differential GPS (DGPS) sensor, etc.) is used to assist in mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 858 can be used, such as, but not limited to, GPS sensors with a USB connector having an Ethernet-to-serial (RS-232) bridge.

[0123] Machine 800 may also include an IMU sensor 866. In some examples, the IMU sensor 866 may be located at the center of the rear axis of machine 800. The IMU sensor 866 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 866 may include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 866 may include an accelerometer, a gyroscope, and a magnetometer.

[0124] In some embodiments, the IMU sensor 866 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 866 can enable the machine 800 to estimate heading by directly observing and correlating velocity changes from GPS to the IMU sensor 866, without requiring input from a magnetic sensor. In some examples, the IMU sensor 866 and the GNSS sensor 858 can be combined in a single integrated unit.

[0125] The vehicle may include one or more microphones 896 placed inside and / or around the machine 800. The microphones 896 can be used for emergency vehicle detection and identification, etc.

[0126] Machine 800 may also include vibration sensors 842. Vibration sensors 842 can measure vibrations of machine parts, such as the arm or leg of a humanoid robot 800C, or the axle of a vehicle 800A or AMR 800B. For example, changes in vibration may indicate changes in roads, walking, or traversable surfaces. In another example, when two or more vibration sensors 842 are used, differences between vibrations can be used to determine friction or slippage on surfaces (e.g., when the vibration difference is between an electrically driven shaft and a freely rotating axle).

[0127] Machine 800 may include ADAS system 838, for example, when machine 800 is vehicle 800A. In some examples, ADAS system 838 may include a dedicated SoC. ADAS system 838 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision or collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), blind spot monitoring (BSM), rear cross traffic warning (RCTW), pedestrian detection, driver monitoring, collision warning system (CWS), traffic sign recognition, speed limit detection, automatic parking, lane centering (LC), high beam safety system and / or other features and functions.

[0128] Machine 800 may also include an infotainment SoC 830 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include one or more discrete components, such as multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiP), heterogeneous integration (HI), single-board computers (SBCs), etc. The infotainment SoC 830 may include a combination of hardware and software that can be used to provide machine 800 with audio (e.g., music, personal digital assistant, navigation commands, news, radio, etc.), video (e.g., television, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., wireless, Wi-Fi, etc.) and / or information services (e.g., navigation systems, rear parking assist, radio data systems, vehicle-related information (e.g., fuel level, total driving distance, brake fluid level, fuel level, door opening / closing, air filter information, etc.). For example, an infotainment SoC 830 may be a radio, disk player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, Wi-Fi, steering wheel audio controls, hands-free voice controls, head-up display (HUD), HMI display 834, telematics device, control panel (e.g., for controlling various components, features and / or systems and / or interacting with various components, features and / or systems) and / or other components. The infotainment SoC 830 may also be used to provide information to vehicle users (e.g., visual and / or auditory), such as information from ADAS system 838, autonomous driving information (e.g., planned vehicle maneuvers, trajectory), surrounding environment information (e.g., intersection information, vehicle information, road information, etc.) and / or other information.

[0129] The infotainment SoC 830 may include GPU functionality. The infotainment SoC 830 can communicate with other devices, systems, and / or components of the machine 800 via bus 802 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 830 may be coupled to a monitoring MCU, allowing the GPU of the infotainment system to perform some autonomous driving functions in the event of a failure of the main controller 836 (e.g., the main computer and / or backup computer of the machine 800). In such an example, the infotainment SoC 830 may place the machine 800 into a driver-safe stop mode, as described herein.

[0130] In some embodiments, the infotainment system may provide a digital or virtual assistant, which may be voice-only or may have visual components (e.g., in the form of a digital human or digital avatar). The assistant may provide basic functions such as sending text messages, adjusting vehicle settings, music or video control, navigation, etc., and / or more advanced functions, such as those supported by one or more language models (e.g., Large Language Model (LLM), Visual Language Model (VLM), Multimodal Language Model (MMLM), etc.). For example, the driver and / or occupants may interact with the assistant in a manner similar to how a user interacts with a language model, such as asking general questions, specific questions, requesting restaurants, gas stations, and / or other recommendations and / or locations, inquiring about vehicle functions or troubleshooting (e.g., asking for tire pressure information, oil change information, battery swap information, etc.). Therefore, the machine 800 (whether it is a vehicle 800A, AMR 800B, humanoid robot 800C, or other type of machine) may include locally stored language models and / or communicate with remotely hosted language models (e.g., via one or more APIs) to provide users of the machine 800 with more detailed and in-depth communication capabilities.

[0131] In some examples, the infotainment SoC 830, SoC 804, and / or another SoC or computing / processing system can perform in-cabin driver and / or occupant monitoring. For example, the computing system can perform facial recognition, and vehicle owner recognition can use data from cameras and / or other sensors to identify the presence of an authorized driver and / or owner of the machine 800. An always-on sensor processing engine can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in a security mode, to disable the vehicle when the owner leaves. In this way, SoC 804 can provide security against theft and / or carjacking.

[0132] In some embodiments, one or more neural networks running on another or a dedicated SoC (e.g., an in-vehicle infotainment or in-vehicle monitoring SoC) can be used to monitor in-cabin monitoring camera sensors. This other or dedicated SoC is configured to recognize in-cabin events and respond accordingly. The in-cabin system can activate cellular services and make phone calls by lip-reading, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. The in-cabin system may also include one or more in-cabin AI agents or assistants that can interact with one or more LLMs, VLMs, MMLMs, etc., in the cloud using one or more APIs or plugins. For example, the in-cabin AI agents or assistants can provide directions, vehicle or machine feedback information, answer general questions, handle music / video and / or other requests, activate windows, doors, and / or other vehicle components, etc. Therefore, one or more dedicated SoCs and / or processor groups can be used to perform in-cabin infotainment and / or in-cabin monitoring (e.g., as an Occupant Monitoring System (OMS)) of Machine 800.

[0133] The device 800 may also include an instrument cluster 832 (e.g., a digital instrument cluster, electronic instrument cluster, digital dashboard, etc.). The instrument cluster 832 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 832 may include a set of instruments such as a speedometer, fuel level, fuel pressure, tachometer, odometer, turn signal indicator, gearshift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 830 and the instrument cluster 832. In other words, the instrument cluster 832 may be included as part of the infotainment SoC 830, and vice versa.

[0134] Figure 8D A computing system according to at least some embodiments of this disclosure (about Figure 8C A block diagram of an example architecture (a subset of the systems described). Although illustrated as SoC 804, this is not intended to be limiting, and the computing system may additionally or alternatively include multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiP), heterogeneous integration (HI), single-board computers (SBCs), and / or other components and / or architectures without departing from the scope of this disclosure.

[0135] The SoC 804 can be an end-to-end platform with a flexible architecture spanning automation levels 2-5, or it can be specifically designed for a particular automation level (e.g., a first SoC 804 for levels 2 to 2++, a second SoC 804 for level 3, a third SoC 804 for level 4, and so on), providing a comprehensive functional safety architecture that leverages and effectively utilizes computer vision, neural network inference, robot planning, control and navigation, ADAS technologies, and more, with diversity and redundancy, to provide a flexible and reliable platform for driving or robot control software stacks, as well as deep learning tools. The SoC 804 can be faster, more reliable, and even more energy-efficient and space-saving than traditional systems. For example, when the accelerator 814 is used in conjunction with the CPU 806, GPU 808, and data repository 816, it can provide a fast and efficient platform for Level 2-5 autonomous vehicles, as well as for the safety planning, navigation, and control of AMR 800B, humanoid robot 800C, and / or other robot or machine types.

[0136] In some embodiments, for example, SoC 804 includes: a GPU 808 having 2000 or more cores (e.g., 2048 cores), 60 or more tensor cores (e.g., 64 tensor cores), and a maximum GPU frequency exceeding 1 GHz (e.g., 1.3 GHz); a CPU 806 including 10 or more cores (e.g., 12 cores), having 64-bit, 3 MB L2 and 6 MB L3 caches, and a maximum frequency of 2 GHz or more (e.g., 2.2 GHz); one or more deep learning accelerators (DLAs); a deep learning accelerator cluster (XNN); a neural network accelerator (NNA) or neural processing unit (NPU) 809 (e.g., two DLAs / XNNs / NNAs / NPUs 809); and a vision accelerator (e.g., a programmable vision accelerator (PVA)) 807. A single SoC 804 may be capable of achieving AI performance of 275 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGXOrin. The 64GB SoC meets these standards and achieves this level of performance.

[0137] Similarly, in an embodiment, SoC 804 includes: a GPU 808 having 1700 or more cores (e.g., 1792 cores), 50 or more tensor cores (e.g., 56 tensor cores), and a maximum GPU frequency exceeding 900MHz (e.g., 930MHz); a CPU 806 including 8 or more cores (e.g., 8 cores), having 64-bit, 2MB L2 and 4MB L3 cache memory, and a maximum frequency of 2GHz or more (e.g., 2.2GHz); one or more deep learning accelerators (DLAs); a deep learning accelerator cluster (XNN); a neural network accelerator (NNA) or neural processing unit (NPU) 809 (e.g., 2 DLAs / XNNs / NNAs / NPUs 809); and a vision accelerator (e.g., a programmable vision accelerator (PVA) 807), and a single SoC 804 may be capable of achieving AI performance of 200 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGXOrin 32GB SoC meets these standards and achieves such performance.

[0138] In some embodiments, for example, SoC 804 includes: a GPU 808 having 1,000 or more cores (e.g., 1,024 cores), 28 or more tensor cores (e.g., 32 tensor cores) and a maximum GPU frequency exceeding 900 MHz (e.g., 1,173 MHz); a CPU 806 including 8 or more cores (e.g., 8 cores), having 64-bit, 2 MB L2 and 4 MB L3 cache memory, and a maximum frequency of 2 GHz or more (e.g., 2 GHz); one or more deep learning accelerators (DLAs); a deep learning accelerator cluster (XNN); a neural network accelerator (NNA) or neural processing unit (NPU) 809 (e.g., one DLA / XNN / NNA / NPU 809); and a vision accelerator (e.g., a programmable vision accelerator (PVA) 807), and a single SoC 804 may be capable of achieving AI performance of 157 trillion operations per second (TOPS). For example, NVIDIA's Jetson AGX Orin NX 16GB SoC meets these standards and achieves such performance.

[0139] In various embodiments, for example, the SoC 804 includes: a GPU 808 having 1000 or more cores (e.g., 1024 cores), 28 or more tensor cores (e.g., 32 tensor cores), and a maximum GPU frequency exceeding 900MHz (e.g., 1020MHz); and a CPU 806 including 6 or more cores (e.g., 6 cores), with 64-bit, 1.5MB L2 and 4MB L3 cache memory, and a maximum frequency of 1.5GHz or higher (e.g., 1.7GHz). A single SoC 804 can achieve AI performance of 67 trillion operations per second (TOPS). For example, NVIDIA's Jetson Orin Nano 8GB SoC meets these standards and achieves such performance.

[0140] SoC 804 may include one or more CPUs 806. In embodiments, CPU 806 may include CPU clusters or CPU complexes (also referred to herein as “CCPLEX”). CPU 806 may include multiple cores and / or (e.g., L2, L3) caches. For example, in some embodiments, CPU 806 may include twelve cores in a coherent multiprocessor configuration. In some embodiments, CPU 806 may include four dual-core clusters, each with a dedicated L2 cache (e.g., 3MB L2 cache). CPU 806 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, thereby allowing any combination of clusters of CPU 806 to be active at any given time.

[0141] The SoC 804 may include any type and number of GPUs 808. For example, an integrated GPU (also referred to herein as an "iGPU") may be used in some embodiments. The GPU 808 may be programmable and can be used efficiently for parallel workloads. In some examples, the GPU 808 may use an enhanced tensor instruction set. The GPU 808 may include one or more streaming microprocessors, wherein each streaming microprocessor may include a cache (e.g., an L1 cache with at least 96KB of storage), and two or more streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage). In some embodiments, the GPU 808 may include at least eight streaming microprocessors. The GPU 808 may use a computing application programming interface (API). Furthermore, the GPU 808 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0142] The GPU 808 can be power-optimized for optimal performance in automotive, robotics, and / or other embedded use cases. For example, the GPU 808 can be fabricated on FinFETs. However, this is not intended to limit it, and the GPU 808 can be fabricated using other semiconductor fabrication methods or processes. Each streaming microprocessor can contain multiple mixed-precision processing cores divided into multiple blocks. For example, but not limited to, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSOR COREs for deep learning matrix algorithms, one (e.g., L0) instruction cache, one warp scheduler, one dispatch unit, and / or one (e.g., 64KB) register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to efficiently perform workloads involving mixed-computation and addressing computations. Streaming microprocessors can include independent thread scheduling capabilities to enable finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors can also include combined L1 data caches and shared memory units to improve performance while simplifying programming.

[0143] The GPU 808 may include a high-bandwidth memory (HBM) and / or (e.g., 16GB) HBM2 memory subsystem to provide peak memory bandwidth of approximately 900GB / s in some examples. In some examples, synchronous graphics random access memory (SGRAM), such as graphics double data rate type 5 synchronous random access memory (GDDR5), may be used in addition to or as an alternative to HBM memory.

[0144] The GPU 808 may include unified memory technology, including access counters, to allow more accurate migration of memory pages to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, Address Translation Service (ATS) support may be used to allow the GPU 808 to directly access the CPU 806 page tables. In such examples, when a miss occurs in the GPU 808 Memory Management Unit (MMU), an address translation request can be transferred to the CPU 806. In response, the CPU 806 can look up the virtual-to-physical mapping of the address in its page tables and transfer the translation back to the GPU 808. Therefore, unified memory technology allows the memory of the CPU 806 and GPU 808 to use a single, unified virtual address space, simplifying GPU 808 programming and porting applications to the GPU 808.

[0145] The SoC 804 may include any number of caches 812, including the caches described herein. For example, cache 812 may include L0 cache, L1 cache, L2 cache, L3 cache (e.g., available for CPU 806 and GPU 808, e.g., connected to CPU 806 and GPU 808)), etc. Cache 812 may include write-back caches that can track the state of the lines, for example, by using one or more cache coherence protocols (e.g., MEI, MESI, MSI, etc.). According to embodiments, the cache (e.g., L3) may include 4MB or more, although smaller or larger cache sizes may be used.

[0146] SoC 804 may include one or more arithmetic logic units (ALUs) 865, which can be used to perform processing related to various tasks or operations of machine 800, such as computer vision, machine learning or deep learning processing, world model management, etc. Furthermore, SoC 804 may include a floating-point unit (FPU) 867 or other mathematical coprocessors or digital coprocessors for performing mathematical operations within the system. For example, SoC 104 may include one or more FPUs 867, which are integrated as execution units within CPU 806 and / or GPU 808.

[0147] SoC 804 may include one or more accelerators 814 (e.g., hardware accelerators, software accelerators, or combinations thereof). For example, SoC 804 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. Large on-chip memory 815 (e.g., 4MB SRAM, 32GB and / or 64GB 256-bit LPDDR5 (204.8GB / s), 8GB and / or 16GB 128-bit LPDDR5 (102.4GB / s), and / or other memory types and sizes) enables the hardware acceleration cluster to accelerate neural network processing, converter processing, optical flow processing, vision processing, and / or other computations or processing. The hardware acceleration cluster can be used to supplement GPU 808 and offload some tasks from GPU 808 (e.g., freeing up more cycles of GPU 808 to perform other tasks). As an example, the accelerator 814 can be used for target workloads (e.g., perceptual, convolutional neural network (CNN), deep neural network (DNN), language model (LLM, VLM, MMLM, VLA, etc.), transformer model, diffusion model, encoder-only model, encoder-decoder model, etc., which are stable enough to be suitable for acceleration.

[0148] Accelerator 814 (e.g., a hardware acceleration cluster) may include a deep learning accelerator (DLA) 809 (also referred to herein as a “deep learning accelerator cluster (XNN) 809”, a “neural network accelerator (NNA) 809”, or a “neural processing unit (NPU) 809”). DLA 809 may include one or more tensor processing units (TPUs) 841, which may be configured to provide additional, for example, trillions of operations per second for deep learning applications and inference. TPU 841 may be an accelerator configured to perform image processing functions (e.g., for CNNs, RCNNs, DNNs, etc.) and optimized for them. DLA 809 may be further optimized for a specific set of neural network types and floating-point operations and inference. The DLA is designed to provide higher performance per millimeter than general-purpose GPUs and significantly outperforms CPUs. TPU 841 can perform several functions, including single-instance convolution functions, support for INT8, INT16, and FP16 data types for features and weights, and post-processor functions. Although TPU 841 is described as being included as part of DLA 809, this is not intended to be limiting, and TPU 841 may be included in additional or alternative accelerator 814 and / or other components, and / or may be included as a discrete processing component.

[0149] The DLA 809 can quickly and efficiently execute neural networks on processed or unprocessed data to achieve a variety of functions, including but not limited to: object and feature recognition and detection using data from one or more sensor modalities (e.g., vehicles, pedestrians, other robots, lane lines, road boundary lines, debris, potholes, boxes, warehouse items, etc.); distance estimation using data from one or more sensor modalities; emergency vehicle detection and identification using data from microphones and / or vision-based sensors; facial recognition; pick-up and place operations; maneuvering operations; occupant monitoring; vehicle owner identification; and / or other in-cabin operations using data from in-cabin cameras and / or other sensor types; and / or safety and / or safety-related events, to name a few.

[0150] The DLA 809 can perform any function of the GPU 808, and by using inference accelerators, for example, designers can anchor either the DLA 809 or the GPU 808 for any function. For instance, designers can centralize the processing of DNNs and floating-point operations on the DLA 809, leaving other functions to the GPU 808 and / or other accelerators 814. The DLA 809 can be used to run any type of network to enhance control and security, including neural networks that, for example, output a confidence metric for each object detection.

[0151] Accelerator 814 (e.g., a hardware acceleration cluster) may include a programmable vision accelerator (PVA) 807, which may alternatively be referred to herein as a computer vision accelerator or generally as a vision accelerator. The PVA 807 may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), semi-autonomous driving, autonomous driving, robotics applications, safety and supervision applications, augmented reality (AR), virtual reality (VR), and / or mixed reality (MR) applications. The PVA 807 provides a balance between performance and flexibility. For example, each PVA 807 may include (e.g., but not limited to) any number of Reduced Instruction Set Computer (RISC) cores, Direct Memory Access (DMA) systems, Pixel Processing Engines (PPEs), Vector Processors or Vector Processing Units (VPUs), and / or other components. The PVA engine may include an Advanced Very Long Instruction Word (VLIW) or Single Instruction Multiple Data (SIMD) digital signal processor. The PVA 807 may be optimized for image processing and computer vision algorithm acceleration tasks. For example, the PVA807 offers excellent performance with extremely low power consumption and can be used asynchronously and concurrently as part of a heterogeneous computing pipeline with CPU 806, GPU 808 and / or other accelerators in systems (e.g., vehicles, robots, etc.).

[0152] The PVA 807 may include one or more (e.g., two) Vector Processing Subsystems (VPSs), each of which may include one or more Vector Processing Unit (VPU) cores, one or more Decoupled Lookup Units (DLUTs), one or more shared or vector memory (VMEMs), and one or more instruction caches (I-caches). The VPU core may be the main processing unit and may include a vector SIMD VLIW DSP 843 optimized for computer vision. The VPU core can fetch instructions via the I-cache and access data via the VMEM. The DLUT may include dedicated hardware components that enhance the efficiency of parallel lookup operations. For example, the DLUT allows parallel lookups using a single copy of the lookup table by performing these lookups in a decoupled pipeline independent of the main processor pipeline. By doing so, the DLUT can minimize or reduce memory usage and increase throughput while avoiding data-related memory library conflicts, ultimately leading to improved overall system performance. The VPU VMEM can provide local data storage for the VPU, allowing for the efficient implementation of various image processing and computer vision algorithms. The VPU VMEM can support access from external VPS hosts, such as Direct Memory Access (DMA) and the CPU 806 (e.g., an ARM Cortex-R5 processor), thereby facilitating data exchange with the CPU 806 and other system-level components. The VPU I-cache can provide instruction data to the VPU on request, request missing instruction data from system memory, and / or maintain temporary instruction storage for the VPU. For each VPU task, the CPU 806 can configure the DMA system, optionally prefetching the VPU program into the VPU I-cache, and / or initiating each VPU-DMA pair to process the task. The PVA 807 may also include L2 SRAM memory shared between one or more (e.g., two) sets of VPS and DMA. In some embodiments, one or more (e.g., two) DMA devices are used to move data between external memory, PVA L2 memory, VMEM (e.g., one per VPS), CPU tightly coupled memory (TCM), DMA descriptor memory, and / or PVA-level configuration registers. In lightly loaded systems, two parallel DMA accesses to DRAM can achieve read / write bandwidth of up to 15 GB / s, while in heavily loaded systems, this bandwidth can reach up to 10 GB / s. Regarding compute capacity, INT8 gigabyte multiply-accumulate operations per second (GMAC) can be 2048 or greater, excluding DLUT. FP32 GMACs can be 32 per PVA instance.

[0153] The RISC core can interact with image sensors (e.g., the image sensor of any camera described herein), image signal processors, etc. Each RISC core may include any amount of memory. The RISC core can use any of a variety of protocols, depending on the implementation. In some examples, the RISC core can execute a real-time operating system (RTOS). The RISC core can be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, the RISC core may include an instruction cache and / or tightly coupled RAM.

[0154] The DMA system enables PVA 807 components to access system memory independently of the CPU 806. DMA can support any number of functions optimized for the PVA 807, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which may include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0155] A vector processor, or VPU, can be a programmable processor designed to efficiently and flexibly execute computer vision algorithms and provide signal processing capabilities. In some examples, the PVA 807 may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA 807 and may include one or more vector processing units (VPUs), one or more pixel processing engines (PPEs) (which may include a 2D layout of interconnected (e.g., for north, south, east, and west communication) processing elements), one or more instruction caches, and / or one or more shared or vector memories (e.g., VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can improve throughput and speed.

[0156] In some embodiments, each vector processor may include an instruction cache and may be coupled to dedicated memory. Therefore, in some examples, each vector processor may be configured to execute independently of other vector processors. In other examples, the vector processors included in a particular PVA 807 may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA 807 may execute the same computer vision algorithm, but for different regions of an image. In other examples, the vector processors included in a particular PVA 807 may execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on consecutive images or portions of an image. Among other things, the hardware acceleration cluster may include any number of PVA 807s, and each PVA may include any number of vector processors. Furthermore, the PVA 807 may include additional error correction code (ECC) memory to enhance overall system security.

[0157] Accelerators 814 (e.g., hardware accelerator clusters) have broad applications in autonomous and semi-autonomous machine control. PVA 807 can be a programmable vision accelerator used in critical processing stages in perception, robot understanding and reasoning, ADAS, semi-autonomous and autonomous vehicles, etc. The capabilities of PVA 807 are well-suited for algorithmic domains requiring predictable processing, featuring low power consumption and low latency. In other words, PVA 807 performs well in semi-intensive or intensive rule computation, even on small datasets requiring predictable runtime, low latency, and low power consumption. Therefore, in the context of autonomous vehicles and robotic platforms, PVA 807 is designed to run classic computer vision algorithms, as they are highly efficient in object detection and integer mathematical operations.

[0158] For example, according to one embodiment of this technology, the PVA 807 is used to perform computer stereo vision. In some examples, a semi-global matching-based algorithm can be used, although this is not intended to limit it. Many Level 3-5 autonomous driving applications require on-the-fly motion estimation / stereo matching (e.g., moving structures, pedestrian recognition, lane detection, etc.). The PVA 807 can perform computer stereo vision functions on input from two monocular cameras.

[0159] In some examples, the PVA 807 can be used to perform dense optical flow, providing processed RADAR data based on the raw RADAR data (e.g., using 4D Fast Fourier Transform). In other examples, the PVA 807 is used for time-of-flight depth processing, for example, providing processed time-of-flight data by processing the raw time-of-flight data.

[0160] While the VPU, DMA, RISC Core, VMEM, and decoupled coprocessors (e.g., DLUT) are described as being included within the PVA807, this does not imply limitation. In some embodiments, these components may be included in alternative or additional processing components and / or accelerator 814, and / or may be included as discrete components of the SoC 804 and / or other computing system architectures.

[0161] In some examples, SoC 804 may include a real-time ray tracing hardware accelerator (RTA) 851, which can be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) to generate real-time or near-real-time visualization simulations, for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for simulation of SONAR, RADAR, LiDAR, camera and / or other sensor modalities in a simulation, for general wave propagation simulation, for comparison with LiDAR data for localization, for generating realistic training data for training neural networks, and / or other functions and uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations. For example, machine 800 (or another machine or device) may perform simulations in a simulation environment, and one or more light transport simulation algorithms (e.g., ray tracing, path tracing, etc.) may be used to generate the simulation environment. Therefore, the ray tracing accelerator 851 and / or a ray tracing-optimized GPU 806 (e.g., NVIDIA's RTX GPU) may be used to accelerate these ray tracing algorithms.

[0162] Accelerator 814 (e.g., in a hardware acceleration cluster) may include one or more optical flow accelerators (OFAs) 811. For example, OFA 811 can be used to calculate optical flow and stereo disparity between sensor data frames (e.g., images). Optical flow can be accelerated on OFA 811 for purposes such as object detection and tracking, and / or for stereo depth estimation, where stereo disparity is calculated between stereo image frames (e.g., two or more frames captured using two or more image sensors with at least partially overlapping fields of view).

[0163] SoC 804 may include one or more Camera Serial Interfaces (CSI) 823. For example, CSI 823 may include a Mobile Industry Processor Interface (MIPI) Camera Serial Interface (CSI) for receiving video and input from cameras, a high-speed interface, and / or a video input block for camera and associated pixel input functions. SoC 804 may also include a software-controllable input / output controller and may be used to receive I / O signals not assigned to a specific role. For example, CSI 823 may include MIPI CSI-2 connectors, such as a 16-channel MIPI CSI-2 connector, D-PHY 2.1 (up to 40Gbps), and C-PHY 2.0 (up to 164Gbps) to support 16 virtual channels and 6 or more cameras; an 8-channel MIPI CSI-2 connector, D-PHY 2.1 (up to 20Gbps) to support 8 virtual channels and 4 or more cameras; and / or 2x MIPI CSI-2, 22-pin camera connectors, depending on the embodiment and implementation.

[0164] Accelerator 814 (e.g., a hardware acceleration cluster) may include an on-chip computer vision network (CVNOC) 863 and SRAM for providing high-bandwidth, low-latency SRAM to accelerator 814. In some examples, on-chip memory may include at least 4 MB of SRAM, such as, but not limited to, eight field-configurable memory blocks accessible by PVA 807, OFA 811, DLA 809, and / or other accelerator 814. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory 815 may be used. PVA 807, OFA 811, DLA 809, and / or other accelerator 814 may access memory via a backbone providing high-speed memory access to accelerator 814. The backbone may include an on-chip computer vision network that interconnects accelerator 814 to memory (e.g., using APB).

[0165] The CGNOC 863 may include an interface that determines whether the accelerator 814 provides ready and valid signals before transmitting any control signals / addresses / data. Such an interface can provide separate stages and separate channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, although other standards and protocols can be used.

[0166] SoC 804 may include a data repository 816 and / or memory 815. The data repository 816 may be on-chip memory 815 of the SoC 804, which may store neural networks and / or other algorithms to be executed on the CPU 806, GPU 808, and / or one or more accelerators 814. In some examples, the capacity of the data repository 816 may be large enough to store multiple neural network instances for redundancy and security. The data repository 816 may include, for example, L2 and / or L3 cache 812. The memory 815 may include SRAM, LPDDR5, and / or other memory types. For example, the memory 815 may include 4MB SRAM, 32GB and / or 64GB 256-bit LPDDR5 (204.8GB / s), 8GB and / or 16GB 128-bit LPDDR5 (102.4GB / s), and / or other memory types and sizes. References to the data repository 816 may include references to the memory associated with the PVA 807, OFA 811, DLA 809 and / or other accelerators 814, as described herein.

[0167] Data repository 816 may include various storage types, such as eMMC, NVMe, etc. For example, SoC 804 may include storage in the form of an embedded multimedia card (eMMC) (e.g., 64GB eMMC 5.1) and / or an SD card slot, with external NVMExpress (NVMe) capability, for example, via M.2Key M. For example, data repository 816 and / or other storage may be accessed via, for example, NVMe using PCI Express (PCIe), RDMA, TCP, and / or other protocols.

[0168] SoC 804 may include one or more processors 810 (e.g., embedded processors). Processor 810 may include a boot and power management processor (BPMP) 853, which may be a dedicated processor and subsystem for handling boot power and management functions, as well as associated safety implementations. BPMP 853 may be part of the SoC 804 boot sequence and may provide runtime power management services. BPMP 853 may provide clock and voltage programming, assistance with system low-power state transitions, management of SoC 804 thermal and temperature sensors, and / or management of SoC 804 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and SoC 804 may use the ring oscillator to detect the temperature of CPU 806, GPU 808, accelerator 814, and / or other components. If the temperature is determined to exceed a threshold, BPMP 853 may enter a temperature fault routine and place SoC 804 into a lower power state and / or place machine 800 into a driver safety stop mode (e.g., safely stop machine 800).

[0169] The processor 810 may also include a set of embedded processors that can be used as the Audio Processing Engine (APE) 855. The APE 855 can be an audio subsystem capable of providing full hardware support for multi-channel audio through multiple interfaces, as well as a wide and flexible audio I / O interface. In some examples, the APE 855 is a dedicated processor core with a digital signal processor and dedicated RAM.

[0170] The processor 810 may also include an Always-On Processor Engine (AOPE) 857, which provides the necessary hardware features to support low-power sensor management and wake-up use cases. AOPE 857 may include a processor core, tightly coupled RAM, support for peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0171] Processor 810 may also include a security processor 813 (or “security island 813”), which may include a security cluster engine comprising a dedicated processor or processor subsystem for handling security management for automotive, robotic, and / or other applications. Security processor 813 and / or the security cluster engine may include two or more processor cores, tightly coupled RAM, support for peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In secure mode, the two or more cores may operate in lockstep mode and as a single core, with comparison logic to detect any differences in their operation. In some embodiments, security processor 813 may include discrete processors such that failure of other system components may not affect the performance and availability of security processor 813.

[0172] The processor 810 may also include a real-time or near-real-time sensor engine (SE) 859, which may include a dedicated processor subsystem for handling real-time or near-real-time camera, LiDAR, RADAR and / or other sensor modality management.

[0173] The processor 810 may also include one or more image signal processors (ISPs) 827, which may include high dynamic range signal processors and / or hardware engines as part of one or more sensor processing pipelines.

[0174] Processor 810 may include a video image synthesizer (VIC) 861, which may be a processing block (e.g., implemented on a microprocessor) that implements the video post-processing functions required by the video playback application to produce the final image of the player window. VIC 861 can perform lens distortion correction on wide-angle camera 868B, surround camera 868D, cabin monitoring camera sensor and / or other camera sensors with distorted field of view.

[0175] VIC 861 may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, when motion occurs in the video, noise reduction appropriately weights spatial information, thereby reducing the weight of information provided by adjacent frames. When an image or part of an image does not contain motion, temporal noise reduction performed by the video image synthesizer can use information from previous images to reduce noise in the current image.

[0176] The VIC 861 can also be configured to perform stereoscopic correction on input stereoscopic camera frames. The video image compositor can also be used for user interface compositing when the operating system desktop is in use and the GPU 808 does not need to continuously render new surfaces. Even when the GPU 808 is powered on and actively performing 3D rendering, the video image compositor can be used to offload the GPU 808 to improve performance and responsiveness.

[0177] SoC 804 may also include various peripheral interfaces for input / output (I / O) 825, for example, to enable communication with peripheral devices, audio codecs, power management and / or other devices. SoC 804 can be used to process data from cameras (e.g., via a gigabit multimedia serial link and / or Ethernet connection), from sensors (e.g., LiDAR sensor 864, RADAR sensor 860, etc., connected via Ethernet), from bus 802 (e.g., speed of machine 800, steering wheel position, etc.), and from GNSS sensor 858 (e.g., connected via Ethernet or CAN bus). SoC 804 may also include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine and can be used to free CPU 806 from routine data management tasks. In some embodiments, the SoC 804I / O 825 may include headers (e.g., 40-pin headers or 40-pin extension headers) supporting Universal Asynchronous Receiver / Transmitter (UART), Serial Peripheral Interface (SPI), Inter-Integrated Sound (I2S), Inter-Integrated Sound (I2C), Controller Area Network (CAN), Pulse Width Modulation (PWM), Digital Microphone Interface (DMIC), Digital Speaker Station (DSPK), General Purpose I / O (GPIO), etc.; automation headers (e.g., 12-pin automation headers); audio panel headers (e.g., 10-pin audio panel headers); Joint Test Action Group (JTAG) headers (e.g., 10-pin JTAG headers); fan headers (e.g., 4-pin fan headers); RTC battery spare connectors (e.g., 2-pin battery spare connectors); a microSD slot; a DC power jack; power, force, resume, and reset buttons; and one or more display connectors (e.g., DisplayPort (DP), such as DP 1.4A(+MST), eDP1.41, HDMI). 2.1 and / or 4K30 multi-model DP1.2 (+MST) connectors) and / or other I / O 825 elements, components or features.

[0178] The SoC 804 may include machine intranetting capabilities using, for example, Ethernet (e.g., automotive Ethernet), SERDES, Controller Area Network (CAN), FlexRay, Local Interconnect Network (LIN), Low Voltage Differential Signaling (LVDS), Media-Oriented System Transport (MOST), another network type, and / or combinations thereof. For example, the SoC 804 may include RJ45 connectors with up to 10GbE, 1GbE connectors, and / or other network connector types.

[0179] So104 may include one or more digital signal processors (DSPs) 843. For example, DSP 843 may include a dedicated or specialized microprocessor chip optimized for digital signal processing, such as audio signal processing, telecommunications, digital image processing, RADAR, SONAR, LiDAR and / or other sensor processing, speech recognition and / or other applications.

[0180] SoC 804 may include one or more video encoders 819 and / or one or more video decoders 821. For example, video encoder 819 may include a hardware-based (e.g., as part of GPU 808) video encoder (e.g., supporting H.264, H.265, etc., and conforming to the HEVC standard, such as NVIDIA's NVENC) that can process image input (e.g., as YUV, RGB, etc.) to generate a video bitstream. Video decoder 821 may include a video decoder engine that provides fully accelerated hardware video decoding capabilities (e.g., supporting decoding of various bitstream formats such as AV1, H.264, H.265, VP8, VP9, ​​MPEG-1, MPEG-2, MPEG-4, VC-1, etc., and conforming to the HEVC standard, such as NVIDIA's NVDEC). In some examples, video decoder 821 may be hardware-based (e.g., as part of GPU 808).

[0181] The SoC 804 may include one or more General Purpose Computing Acceleration Clusters (GCACs) 829. For example, the GCAC 829 may include various processor types that can be used for accelerated computing, such as one or more Vector Microcode Processors (VMPs) 833, one or more Multi-Threaded Processing Clusters (MPCs) 831, one or more Programmable Macroarrays (PMAs) 835, and / or one or more other processor types. For example, the GCAC 829 may include a PMA 835, two VMPs 833, and two MPCs 831.

[0182] The SoC 804 may include one or more vector microcode processors (VMPs) 833. In embodiments, the VMP 833 may include wide vector (Very Long Instruction Word (VLIW) and Single Instruction Multiple Data (SIMD)) machines that perform various operations, such as short integral-type operations common in computer vision and deep learning algorithms.

[0183] SoC 804 may include one or more multi-threaded processing clusters (MPCs) 831. MPCs 831 may include processing clusters that are more general-purpose than GPUs and more efficient than CPUs in some embodiments. For example, an MPC 831 may include a multi-threaded processor that allows multiple threads to share resources and execute instructions concurrently.

[0184] The SoC 804 may include one or more programmable macro arrays (PMAs) 835. The PMA 835 may include a coarse-grained reconfigurable architecture (CGRA) dataflow machine, which has a unique architecture that delivers powerful performance on intensive computer vision and deep learning algorithms that may not be achievable in classic digital signal processing (DSP) architectures.

[0185] The SoC 804 may include one or more Display Processing Units (DPUs) 845 for performing hardware-accelerated image processing. For example, the DPU 845 may retrieve pixel data from memory 815 and send it to a display peripheral device via a standard interface. Therefore, the DPU 845 can handle display processing and rendering for displays within and / or on the machine.

[0186] The SoC 804 may include one or more Application Processing Units (APUs) 839. For example, the APU 839 may include a quad-core or dual-core processor with 48KB / 32KB L1 cache (with parity and ECC) and a 1MB L2 cache with ECC. The APU 839 may support NEON instructions and single-precision and double-precision floating-point operations.

[0187] The SoC 804 may include one or more Real-Time Processing Units (RTPUs) 869. The RTPU 869 may include a dual-core processor with 32KB / 32KB L1 cache and a 256KB TCM with ECC. The RTPU 869 can support single-precision and double-precision floating-point operations.

[0188] SoC 804 may include one or more built-in self-test (BIST) components 837. For example, BIST component 837 may include a memory BIST (MBIST) for testing the system's memory and / or a logic BIST (LBIST) for testing the system's logic. BIST component 837 may include embedded logic for directly testing the system's logic and / or memory.

[0189] SoC 804 may include one or more dynamically reconfigurable processors (DRPs) 871. For example, DRP 871 can be used to accelerate various computational operations. For instance, in one embodiment, DRP 871 may be combined with a MAC unit to function as an AI accelerator. In another embodiment, DRP 871 can execute an application while dynamically switching the circuit connection configuration of the arithmetic unit (e.g., ALU) on the chip each operating clock according to what needs to be processed. Because it uses only the necessary arithmetic circuitry, DRP 871 can consume less power than a CPU and achieve higher speeds. Furthermore, compared to a CPU, which suffers from performance degradation due to frequent accesses to external memory for cache misses and other reasons, DRP 871 can pre-build the necessary data paths in the hardware, thereby reducing performance degradation and operating speed variations (jitter) caused by memory accesses. DRP 871 may include a dynamic loading function that switches circuit connection information each time the algorithm changes, enabling processing with limited hardware resources, even in robotics / automotive applications that require processing multiple algorithms.

[0190] In some embodiments, accelerator 814 may include an OpenCV accelerator for accelerating OpenCV processing, OpenCV being an open-source, industry-standard library for computer vision processing. In some embodiments, the combination of one or more DRP 871s deployed as AI accelerators with an OpenCV accelerator can enhance AI computation and image processing algorithms, enabling complex and computationally intensive operations such as visual simultaneous localization and mapping (SLAM).

[0191] Compared to traditional systems, the techniques described herein, by providing CPU complexes, GPU complexes, and hardware acceleration clusters, allow multiple neural networks to be executed simultaneously (e.g., at least partially in parallel) and / or sequentially, and the results combined to achieve Level 2-5 autonomous driving capabilities and / or autonomous robot motion, control, planning, and / or navigation operations. Furthermore, since the SoC 804 can include various computing engines (e.g., processor 810, CPU 806, GPU 808, accelerator 814, etc.), tasks can be distributed among the computing engines, and in some cases, common-cause failures are avoided due to the discrete footprint of the computing engines. Additionally, since the SoC 804 can include a dedicated safety processor 813 (or safety island 813), critical safety or redundant operations can be performed without common-cause failures of the SoC 814's main processing components or computing engines. Due to these features, the underlying system of the SoC 804 and / or machine 800 may be able to meet higher safety levels—such as the Automotive Safety Integrity Level (ASIL) D of the ISO 26262 standard.

[0192] Figure 8E According to some embodiments of this disclosure, cloud-based servers (e.g., servers such as those described herein in a data center) and Figure 8A The following is a system diagram illustrating communication between an example autonomous or semi-autonomous vehicle or machine 800. System 876 may include server 878, network 890, and machine 800. Server 878 may include multiple GPUs 884(A)-884(H) (collectively referred to herein as GPU 884), switches 882(A)-882(H) (e.g., PCIe 4.0 / 5.0 switches, M.2 slots, Thunderbolt, USB4, NVIDIA's NVLink, NVIDIA's NVSwitch, GPUDirect RDMA, GPUDirect Storage, etc.), CPUs 880(A)-880(B) (collectively referred to herein as CPU 880), accelerators, and / or other processor types. GPU 884, CPU 880, and PCIe switches may interconnect with high-speed interconnects, such as, but not limited to, NVIDIA-developed NVLink interface 888 and / or PCIe connection 886. In some examples, the GPU 884 is connected via NVLink and / or NVSwitch SoC, and the GPU 884 and PCIe switch 882 are connected via PCIe interconnect. While the figure shows eight GPU 884s, two CPU 880s, and two PCIe switches, this is not limiting. According to embodiments, each server 878 may include any number of GPU 884s, CPU 880s, and / or PCIe switches. For example, each server 878 may include eight, sixteen, thirty-two, and / or more GPU 884s.

[0193] Server 878 may receive sensor data from network 890 and machines 800 indicating information about new or previously unexplored locations, and / or sensor data indicating changes to previously seen / stored locations (e.g., unexpected or altered road conditions, such as recently started road construction). Server 878 may transmit neural network 892, updated neural network 892, map information 894, etc., including information about traffic and road conditions, to machines 800 via network 890. Updates to map information 894 may include updates to HD maps 822, SD maps, navigation maps, etc., such as information about construction sites, potholes, detours, floods, and / or other obstacles. In some examples, neural network 892, updated neural network 892, map information 894, and / or other information may come from new training and / or experience, represented in data received from any number of machines 800 in the environment, and / or based on training performed in a data center (e.g., using server 878 and / or other servers).

[0194] Server 878 can be used to train a machine learning model (e.g., a neural network) based on training data. Training data can be generated by machine 800 and / or generated in a simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is unlabeled and / or unprocessed (e.g., the neural network does not require supervised learning). Training can be performed according to any one or more machine learning techniques, including but not limited to the following categories: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component analysis and clustering analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, it can be used by machine 800 (e.g., transmitted to machine 800 via network 890), and / or the machine learning model can be used by server 878 to remotely monitor and / or control machine 800.

[0195] In some examples, server 878 can receive data from machine 800 and apply the data to state-of-the-art real-time neural networks for real-time intelligent inference. Server 878 may include a deep learning supercomputer powered by GPU 884 and / or a dedicated AI computer, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, server 878 may include a deep learning infrastructure in a data center using only CPU power.

[0196] The deep learning infrastructure of server 878 may be capable of rapid, real-time inference and can use this capability to assess and verify the health of the processor, software, and / or related hardware in machine 800. For example, the deep learning infrastructure may receive periodic updates from machine 800, such as image sequences and / or objects located by machine 800 in the image sequence (e.g., through computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them with the objects identified by machine 800. If the results do not match and the infrastructure concludes that the AI ​​in machine 800 has malfunctioned, server 878 can send a signal to machine 800 instructing the fail-safe computer of machine 800 to take over control, notify the occupants, and perform safety maneuvers or operations, such as slowing down, returning control to the driver, stopping, and / or pulling over / closing the vehicle.

[0197] For inference, the server 878 can include a GPU 884 and one or more programmable inference accelerators (such as NVIDIA's TensorRT). The combination of GPU-driven servers and inference acceleration enables real-time responses. In other examples, such as in scenarios where performance is less critical, servers powered by CPUs, FPGAs, and other processors can be used for inference. The computing ecosystem for generating, training, and deploying AI.

[0198] Figure 9 This is a system diagram illustrating a three-computer ecosystem 900 according to at least some embodiments of the present disclosure, including a first computing system 902 for generating or creating artificial intelligence (AI) (e.g., AI training and validation data), a second computing system 904 for training the AI, and a third computing system 906 for deploying AI at the edge (which may include or correspond to...). Figures 8A to 8E (SoC 804). For example, to develop and deploy materialized or physical AI, a three-computer ecosystem 900 can be used, comprising three accelerated computer systems to handle physical AI training, simulation, and runtime (e.g., edge deployment). These systems can use scalable, physical-based simulations of the machine 800 and its world to generate training data and train multimodal base models (and / or other model types). By doing so, simulations of the machine 800 can be performed at scale, allowing skills (e.g., robotic skills) to be improved, tested, and optimized in virtual worlds that simulate physical laws (e.g., using NVIDIA's OMNIVERSE), thereby helping to reduce the cost of real-world data acquisition and ensuring that the machine 800 can operate safely in a controlled environment.

[0199] Computing system 904 (e.g., NVIDIA's DGX platform) can be used to train and fine-tune powerful foundational and generative AI models. Models such as general foundational models (e.g., NVIDIA's Project GR00T) can be used to enable robots and other machines 800 to understand natural language and mimic actions by observing human movements. Computing system 904 may include a platform that integrates software, infrastructure, and expertise into a modern, unified AI development and training solution. Computing system 904 may include individual computing devices 910 (e.g., NVIDIA's DGX B200, H200, etc.) and / or any number of computing devices 910 (e.g., NVIDIA's DGX SuperPOD) within data center infrastructure 912.

[0200] For example, a standalone computing device 910 may include GPUs (e.g., 8 GPUs with a total GPU memory of 1,440 GB) and CPUs (e.g., 2 CPUs with a total of 112 cores, 2.1 GHz or 4 GHz (with enhancements)), providing up to 72 petaFLOPS of training capability and 144 petaFLOPS of inference capability. The computing device 910 may include memory (e.g., 4 TB of memory) and storage (e.g., 2 x 1.9 TB NVMe M.2 OS storage and 8 x 3.84 TB NVMe U.2 internal storage). The computing device 910 may include various networking and network management components, such as OSFP ports (e.g., 4 OSFP ports) for servicing single-port intelligent host channel adapters (e.g., 8 single-port ConnextX-7 Virtual Protocol Interconnect (VPI)), providing up to 400 GB / s of Infiniband / Ethernet. The computing device 910 may also include, for example, a dual-port quad small pluggable (QSFFP) data processing unit (DPU) (e.g., two dual-port QSFP112 DPUs – such as NVIDIA’s BlueField-3 DPU) providing up to 400Gb / s of InfiniBand / Ethernet. The computing device 910 may include an onboard network interface card (NIC) (e.g., a 10Gb / s onboard NIC with RJ45), a dual-port Ethernet NIC (e.g., a 100GB / s dual-port Ethernet NIC), and / or a host board management controller (MBC) (e.g., with RJ45). In some embodiments, the NIC for the computing device 910 may include a SuperNIC (e.g., NVIDIA’s ConnectX-8 SuperNIC) to provide up to 800Gb / s of data throughput for in-network computing acceleration engines, thereby providing the performance and robust feature set required to support trillion-parameter-scale AI factories and scientific computing workloads. In other embodiments, computing device 910 may include a smart host channel adapter (HCA) (e.g., NVIDIA's ConnectX-7) to provide ultra-low latency and 400Gb / s throughput for computing acceleration engines within the network.

[0201] Data center infrastructure 912 may include: any number of computing devices 910, and an operating system (OS) (e.g., a DGX OS extension of a Linux distribution) for maximizing system uptime, security, and reliability; network / storage acceleration libraries and management for accelerating end-to-end infrastructure performance; cluster management for scaling and managing a single node (e.g., a computing device 910) to thousands of nodes; job scheduling and orchestration to ensure the smooth execution of each developer's jobs; AI workflow management and machine learning operations (MLOps) for moving more models from prototypes to production; and enterprise software to accelerate developer success.

[0202] Computational System 902 (e.g., NVIDIA's OVX Server) can provide a development and simulation platform for testing and optimizing physics-based AI using APIs and frameworks for simulation (e.g., NVIDIA's DriveSIM, ISAAC Sim, ISAAC Gym, ISAAC Lab, etc.). Computational System 902 allows developers to use simulation frameworks to simulate and validate robot models, and / or generate large amounts of physically-based synthetic data to guide model training. Computational System 902 can support learning frameworks that power robot reinforcement learning and imitation learning to accelerate robot policy training and refinement. For example, Computational System 902 can be used to generate any number of Simulation 908, such as in NVIDIA's OMNIVERSE. Computational System 902 can be used to optimize and accelerate the entire software stack, from training, fine-tuning, and deploying generative AI to supporting industrial digitization within content collaboration platforms (including APIs, software development kits (SDKs), and services). These platforms allow the integration of OpenUSD, ray tracing rendering technologies (e.g., NVIDIA's RTX), and generative physics-based AI into existing software tools and simulation workflows, such as for industrial and robotics use cases (e.g., NVIDIA's OMNIVERSE). Therefore, the computing system 902 can host or support the native OpenUSD software platform, enabling enterprises to connect 3D pipelines and develop advanced real-time 3D applications for industrial digitalization. With powerful ray tracing-accelerated AI and graphics capabilities, the computing system 902 delivers robust performance for workloads such as extended reality (XR), multi-user design collaboration, and digital twins. This allows for the creation of physically accurate models with high-fidelity ray tracing and path tracing material rendering, large-scale, AI-enabled simulations, and the generation of realistic 3D synthetic data for training. The computing system 902 can include a single computing device 914 (e.g., an NVIDIA OVX L40S server) and / or any number of computing devices 914 (e.g., NVIDIA OVX systems) within a data center infrastructure 916.

[0203] Computing device 914 (which may include a server) may include CPUs (e.g., two CPUs, each with 32 cores) and GPUs (e.g., four or eight GPUs, each including 48GB GDDR6 with ECC memory, 864GB / s memory bandwidth, a PCIe Gen4x16: 64GB / s bidirectional interconnect interface, 18,176 CUDA cores, 142 ray tracing (RT) cores, and 568 Tensor cores). Computing device 914 may include various networking and network management components, such as Intelligent Host Channel Adapters (HCAs) (e.g., two or four single-port ConnextX-7s, each with 200Gb / s, providing up to 800Gb / s InfiniBand / Ethernet), one or more DPUs (e.g., dual-port QSFP112 DPUs, such as the NVIDIA BlueField-3 DPU), providing up to 400Gb / s InfiniBand / Ethernet. In some embodiments, the NIC for computing device 914 may include a SuperNIC (e.g., NVIDIA's ConnectX-8 SuperNIC) to provide up to 800Gb / s of data throughput for in-network compute acceleration engines, thereby providing the performance and robust feature set required to support trillion-parameter-scale AI factories and scientific computing workloads. In other embodiments, computing device 914 may include a Smart Host Channel Adapter (HCA) (e.g., NVIDIA's ConnectX-7) to provide ultra-low latency and 400Gb / s throughput for in-network compute acceleration engines. Computing device 914 may include host memory (e.g., 384Gb DDR5 ECC for four GPUs, or 768Gb DDR5 ECC for eight GPUs) and may include dual in-line memory module (DIMM) slots, host boot drives (e.g., 1TB NVMe), and / or host storage (e.g., 24TB NVMe).

[0204] Similar to Data Center Infrastructure 912, Data Center Infrastructure 916 allows any number of computing devices 914 to be combined into a cluster configuration based on a reference architecture.

[0205] Computing system 906 can be used to deploy trained AI models on a runtime computer (such as the SoC 804 described herein). For example, these computing systems 906 can be designed for compact onboard computing needs, including ensembles of models such as control policies, vision, and language models deployed on an energy-efficient onboard edge computing system 906. (See also: [link to document]). Figures 8A to 8E A more detailed description of the components, features, and capabilities of the 906 computing system.

[0206] Example Generative Model

[0207] In at least some embodiments, language models such as Large Language Models (LLMs), Visual Language Models (VLMs), Multimodal Language Models (MMLMs), Visual Language Action (VLA) models, and / or other types of generative artificial intelligence (AI) can be implemented. These models may be able to understand, summarize, translate, and / or otherwise generate text (e.g., natural language text, code, etc.), images, videos, computer-aided design (CAD) assets, OMNIVERSE and / or METAVERSE file information (e.g., USD formats such as OpenUSD), and / or the like based on context provided in input prompts or queries. In embodiments, these language models may be considered "large" because they are trained on massive datasets and have architectures with a large number of learnable network parameters (weights and biases)—e.g., millions or billions of parameters. This disclosure allows for the implementation of LLM / VLM / MMLM, etc., for summarizing textual data, analyzing data (e.g., text, images, videos, etc.), extracting insights from data (e.g., text, images, videos, etc.), and generating new text / images / videos / , etc., in a user-specified style, tone, and / or format. In some embodiments, the LLM / VLM / MMLM, etc., of this disclosure may be specifically designed for text processing, while in other embodiments, multimodal LLMs may be implemented to accept, understand, and / or generate text and / or other types of content, such as images, audio (sound, synthesized speech, etc.), 2D and / or 3D data (e.g., USD format), and / or video. For example, a visual language model (VLM) or more specifically a multimodal language model (MMLM) may be implemented to accept images, videos, sensor data, audio, text, 3D designs (e.g., CAD), and / or other input data types and / or generate or output images, videos, audio, text, 3D designs, and / or other output data types.

[0208] Various types of LLM / VLM / MMLM / etc. architectures can be implemented in various embodiments. For example, different architectures can be implemented using different techniques to understand and generate outputs (e.g., text, audio, video, images, 2D and / or 3D design or asset data, etc.). In some embodiments, LLM / VLM / MMLM / etc. architectures (e.g., recurrent neural networks (RNNs) or long short-term memory networks (LSTMs)) can be used, while in other embodiments, converter architectures (e.g., architectures relying on self-attention and / or cross-attention (e.g., between contextual data and textual data) mechanisms) can be used to understand and recognize relationships between words or tokens and / or contextual data (e.g., other text, video, images, design data, USD, etc.). One or more generative processing pipelines including LLM / VLM / MMLM / etc. may also include one or more diffusion blocks (e.g., noise reduction blocks). The LLM / VLM / MMLM / etc. of this disclosure may include encoder and / or decoder blocks. For example, discriminative or encoder-only models (e.g., BERT (Bidirectional Encoder Representations from Transformers)) can be implemented for tasks involving language understanding (e.g., classification, sentiment analysis, question answering, and named entity recognition). As another example, generative or decoder-only models (e.g., GPT (Generative Pretrained Transformer)) can be implemented for tasks involving language and content generation (e.g., text completion, story generation, and dialogue generation). LLM / VLM / MMLM / etc., including encoder and decoder components (e.g., T5 (Text-to-Text Transformer)), can be implemented to understand and generate content, such as for translation and summarization. These examples are not intended to be limiting and any architecture type (including, but not limited to, those described herein) can be implemented depending on the specific implementation and the task performed using LLM / VLM / MMLM / etc.

[0209] In various embodiments, LLM / VLM / MMLM / etc. can be trained using unsupervised learning, whereby LLM / VLM / MMLM / etc. learns patterns from a large amount of unlabeled text / audio / video / image / design / USD / etc. data. Due to the extensive training, in these embodiments, the model may not require task-specific or domain-specific training. An LLM / VLM / MMLM / etc. extensively pre-trained on a large amount of unlabeled data can be referred to as a base model and can excel at various tasks, such as question answering, summarizing, filling in missing information, translation, and image / video / design / USD / data generation. Some LLM / VLM / MMLM / etc. can be customized for specific use cases using techniques such as cue tuning, fine-tuning, retrieval augmentation generation (RAG), adding adapters (e.g., custom neural networks and / or neural network layers to tune or adjust cues or labels to bias the language model towards a specific task or domain), and / or using optimization models for specific tasks and / or other fine-tuning or customization techniques within a specific domain.

[0210] In some embodiments, the LLM / VLM / MMLM / etc. disclosed herein can be implemented using various model alignment techniques. For example, in some embodiments, guardrails can be implemented to identify incorrect or unwanted inputs (e.g., prompts) and / or outputs of the model. In this process, the system can use guardrails and / or other model alignment techniques to prevent the processing of specific unwanted inputs using LLM / VLM / MMLM / etc., and / or to prevent the output or presentation of information generated by LLM / VLM / MMLM / etc. (e.g., displays, audio outputs, etc.). In some embodiments, one or more additional models (or layers thereof) can be implemented to identify problems with the model's inputs and / or outputs. For example, these "protective" models can be trained to identify "safe" or otherwise okay or desired inputs and / or outputs and / or "unsafe" or otherwise unwanted inputs and / or outputs for a particular application / implementation. Therefore, the LLM / VLM / MMLM / etc. disclosed herein are unlikely to output language / text / audio / video / design data / USD data / etc. that may be offensive, vulgar, inappropriate, insecure, out of scope, and / or unwanted for a particular application / implementation.

[0211] In some embodiments, an LLM / VLM / etc. can be configured or able to access or use one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc. For example, for certain tasks or operations where the model is not ideally suited, the model may have instructions for accessing one or more plugins (e.g., third-party plugins) to help process the current input (e.g., as a result of training, and / or based on instructions in a given prompt). In such an example, when at least part of the prompt relates to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs) to retrieve relevant information. Another example is that if at least part of the response requires mathematical computation, the model can access one or more mathematical plugins or APIs to help solve the problem, and then the response from the plugins and / or APIs can be used in the model's output. This process can be repeated (e.g., recursively) an arbitrary number of iterations, using any number of plugins and / or APIs, until a response to each query / question / request / process / action / etc. can be generated in response to the input prompt. Therefore, models can rely not only on their own knowledge gained from training on large datasets, but also on the expertise or optimized properties of one or more external resources (such as APIs, plugins, etc.).

[0212] In some embodiments, multiple language models (e.g., LLM / VLM / MMLM / etc., multiple instances of the same language model, and / or multiple hints provided to the same language model or instances of the same language model) can be implemented, executed, or accessed (e.g., using one or more plugins, user interfaces, APIs, databases, data stores, repositories, etc.) to provide output in response to the same query or in response to separate parts of a query. In at least one embodiment, the same input query and hints (e.g., a set of constraints, condition generators, etc.) can be provided to multiple language models (e.g., language models with different architectures, language models trained on different (e.g., updated) data corpora). In one or more embodiments, the language models can be different versions of the same base model. In one or more embodiments, at least one language model can be instantiated as multiple agents, for example, providing more than one hint to constrain, guide, or otherwise influence the style, content, or character of the provided output. In one or more exemplary non-limiting embodiments, the same language model can be required to provide output corresponding to different roles, perspectives, characters, or different knowledge bases, as defined by the provided hints.

[0213] In any such embodiment, the outputs of two or more (e.g., each) language models, two or more versions of at least one language model, two or more instantiated proxies of at least one language model, and / or provided to two or more prompts for at least one language model can be further processed, such as aggregated, compared, or filtered, or used to determine (and provide) a consensus response. In one or more embodiments, the output from one language model (or version, instance, or proxy) can be provided as input to another language model for further processing and / or validation. In one or more embodiments, the language model can be required to generate or otherwise obtain output about the input source material, wherein the output is associated with the input source material. This association may include, for example, generating captions or text portions embedded (e.g., as metadata) within the input source text or image. In one or more embodiments, the output of the language model can be used to determine the validity of the input source material for further processing or inclusion in a dataset. For example, the language model can be used to evaluate the presence (or absence) of a target word in a text portion or the presence (or absence) of an object in an image, wherein the text or image is annotated to indicate such presence (or absence). Alternatively, the determination from the language model can be used to determine whether the source material should be included in the curatorial dataset, for example, but not limited to this.

[0214] Figure 10 This is a block diagram of an example generative language model system 1000 suitable for implementing at least some embodiments of the present disclosure. Figure 10 In the example shown, the generative language model system 1000 includes a retrieval augmentation (RAG) component 1092, an input processor 1005, a tokenizer 1010, an embedding component 1020, a plugin / API 1095, and a generative language model (LM) 1030 (which may include LLM, VLM, MMLM, VLA models, etc.).

[0215] At a high level, the input processor 1005 can receive input 1001, which includes text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasound, etc.), 3D design data, CAD data, generic scene descriptor (USD) data (e.g., OpenUSD, etc.), depending on the architecture of the generative LM 1030 (e.g., LLM / VLM / MMLM, etc.). In some embodiments, input 1001 includes plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, input 1001 may include numerical sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., tabular format, JSON, or XML). In the generative LM In some implementations of 1030 capable of handling multimodal input, input 1001 can combine text (or text that may be omitted) with image data, audio data, video data, design data, USD data, and / or other types of input data (e.g., but not limited to the data described herein). Taking raw input text as an example, input processor 1005 can prepare the raw input text in various ways. For example, input processor 1005 can perform various types of text filtering to remove noise (e.g., special characters, punctuation marks, HTML tags, stop words, portions of images, portions of audio, etc.) from the relevant text content. In examples involving stop words (common words that often have little semantic meaning), input processor 1005 can remove stop words to reduce noise and enable generative LM. 1030 focuses on more meaningful content. Input processor 1005 can apply text normalization, for example, by converting all characters to lowercase, removing accents, and / or handling special cases (such as abbreviations or abbreviations) to ensure consistency (e.g., converting 1 / 4 to 1 / 4). Similarly, input processor 1005 and / or post-processor can perform inverse text normalization (ITN) to convert plain language back to canonical or other forms (e.g., converting 1 / 4 to 1 / 4). These are just a few examples; other types of input and / or output processing can be applied.

[0216] In some embodiments, RAG component 1092 (which may include one or more RAG models, and / or may be performed using generative LM 1030 itself) may be used to retrieve additional information to be used as part of input 1001 or a prompt. RAG can be used to enhance input to LLM / VLM / MMLM / etc. with external knowledge to make the answer to a specific question or query or request more relevant, for example, where specific knowledge is required. RAG component 1092 may obtain this additional information (e.g., basic information such as basic text / images / videos / audio / USD / CAD / etc.) from one or more external sources, and then feed it along with the prompt to LLM / VLM / MMLM / etc. to improve the accuracy of the model's response or output.

[0217] For example, in some embodiments, in addition to the data retrieved using RAG component 1092, input 1001 may also be generated using query or model inputs (e.g., questions, requests, etc.). In some embodiments, input processor 1005 may analyze input 1001 and communicate with RAG component 1092 (or in embodiments, RAG component 1092 may be part of input processor 1005) to identify relevant text and / or other data to provide to generative LM 1030 as additional context or information sources, typically from which responses, answers, or outputs 1090 are identified. For example, when the input indicates that a user is interested in the required tire pressure for a particular brand and model of vehicle, RAG component 1092 may use a RAG model, for example, to perform a vector search in the embedding space to retrieve tire pressure information or its corresponding text from a digital (embedded) version of the owner's manual for that particular vehicle brand and model. Similarly, when a user revisits the chatbot related to a specific product sale or service, the RAG component 1092 can retrieve previously stored conversation history (or at least its summary) and provide the previous conversation history, along with the current inquiry / request, as part of the generative LM 1030 as input 1001.

[0218] RAG component 1092 can use various RAG techniques. For example, it can use naive RAG ( The document is indexed, chunked, and applied to an embedding model to generate embeddings corresponding to chunks. User queries can also be applied to this embedding model and / or another embedding model of the RAG component 1092, and the embeddings of the chunks can be compared with the embeddings of the query to identify the most similar / relevant embeddings to the query. These most similar / relevant embeddings can be provided to the generative LM 1030 to generate output.

[0219] In some embodiments, more advanced RAG techniques can be used. For example, chunks can undergo pre-retrieval processes (e.g., routing, rewriting, metadata analysis, expansion, etc.) before being passed to the embedding model. Furthermore, post-retrieval processes (e.g., re-ranking, hint compression, etc.) can be performed on the output of the embedding model before generating the final embedding, which is then used for comparison with the input query.

[0220] As a further example, modular RAG techniques can be used, such as those similar to Naive RAG and / or Advanced RAG, but also including features such as hybrid search, recursive retrieval and query engines, StepBack methods, subqueries and hypothetical document embeddings.

[0221] As another example, Graph RAG can use a knowledge graph as a source of context or factual information. Graph RAG can be implemented using a graph database as a source of contextual information sent to LLM / VLM / MMLM / etc. Instead of providing the model with data chunks extracted from larger documents (which may result in a lack of context, factual accuracy, linguistic accuracy, etc.) (or anything other than providing the model with data chunks extracted from larger documents), Graph RAG can also provide structured entity information to LLM / VLM / MMLM / etc. by combining structured entity text descriptions with their many attributes and relationships, thus giving the model deeper insights. In implementing Graph RAG, the systems and methods described herein use graphs as content stores and extract relevant document chunks, requiring LLM / VLM / MMLM / etc. to use them to answer questions. In such embodiments, the knowledge graph may contain relevant textual content and metadata about the knowledge graph, or it may be integrated with a vector database. In some embodiments, Graph RAG can use the graph as a subject matter expert, where descriptions of concepts and entities relevant to the query / hint can be extracted and passed to the model as semantic context. These descriptions may include relationships between concepts. In other examples, the graph can be used as a database where a portion of a query / hint can be mapped to a graph query, the graph query can be executed, and LLM / VLM / MMLM / etc. can aggregate the results. In such examples, the graph can store relevant factual information and can be used for queries (natural language queries) and entity links to graph query tools (NL to graph query tools). In some embodiments, the graph RAG (e.g., using a graph database) can be combined with standard (e.g., vector database) RAGs and / or other RAG types to benefit from a variety of approaches.

[0222] In any embodiment, the RAG component 1092 may implement plugins, APIs, user interfaces, and / or other functions to perform RAG. For example, LLM / VLM / MMLM / etc. may use graph RAG plugins to run queries on knowledge graphs to extract relevant information to feed into the model, and may use standard or vector RAG plugins to run queries on vector databases. For example, the graph database may interact with the plugin's REST interface, thus decoupling the graph database from the vector database and / or the embedded model.

[0223] The tokenizer 1010 can segment (e.g., processed) text data into smaller units (tags) for subsequent analysis and processing. Depending on the implementation, the tags can represent individual words, sub-words, characters, audio / video / images, etc. Word-based tokenization divides the text into individual words, treating each word as a separate tag. Sub-word tokenization breaks words down into smaller meaningful units (e.g., prefixes, suffixes, stems), enabling the generative LM 1030 to understand morphological changes and process words outside the vocabulary more effectively. Character-based tokenization represents each character as a separate tag, enabling the generative LM 1030 to process text at a fine-grained level. The choice of tokenization strategy can depend on factors such as the language being processed, the task at hand, and / or the characteristics of the training dataset. Therefore, the tokenizer 1010 can transform (e.g., processed) text into a structured format according to the tokenization scheme implemented in a particular embodiment.

[0224] Embedding component 1020 can use any known embedding technique to transform discrete tokens into semantically meaningful (e.g., dense, continuous vector) representations. For example, embedding component 1020 can use pre-trained word embeddings (e.g., Word2Vec, GloVe, or FastText), one-hot encoding, Term Frequency-Inverse Document Frequency (TF-IDF) encoding, one or more embedding layers of a neural network, and / or others.

[0225] In some implementations where input 1001 includes image data / video data, etc., input processor 1005 may resize the data to a standard size compatible with the format of the corresponding input channel and / or normalize pixel values ​​to a common range (e.g., 0 to 1) to ensure consistent representation, and embedding component 1020 may encode the image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations where input 1001 includes audio data, input processor 1005 may resample the audio file to a consistent sampling rate for uniform processing, and embedding component 1020 may use any known technique to extract and encode audio features, such as in the form of a spectrogram (e.g., a Mel spectrogram). In some implementations where input 1001 includes video data, input processor 1005 may extract frames or apply resizing to extracted frames, and embedding component 1020 may extract features such as optical flow embedding or video embedding and / or encode temporal information or frame sequences. In some implementations where input 1001 includes multimodal data, the embedded component 1020 can use techniques such as early fusion (stitching), late fusion (sequential processing), and attention-based fusion (e.g., self-attention, cross-attention) to fuse representations of different types of data (e.g., text, images, audio, USD, video, design, etc.).

[0226] Other components of the generative LM 1030 and / or generative LM system 1000 may use different types of neural network architectures depending on the implementation scheme. For example, a transducer-based architecture (e.g., the architecture used in models such as GPT) may be implemented, and it may include a self-attention mechanism that weights the importance of different words or tokens in the input sequence and / or a feedforward network that processes the output of the self-attention layer, applying a nonlinear transformation to the input representation and extracting higher-level features. Some non-limiting example architectures include transducers (e.g., encoder-decoder, decoder-only, multimodal), RNNs, LSTMs, fusion models, diffusion models, cross-modal embedding models that learn a joint embedding space, graph neural networks (GNNs), hybrid architectures that combine different types of adversarial networks (e.g., generative adversarial networks or GANs or adversarial autoencoders (AAEs) for joint distribution learning, linear time series modeling using a selective state-space modeling (SSM) architecture (e.g., the Mamba LLM architecture), etc. Therefore, depending on the implementation scheme and architecture, the embedded component 1020 can apply the encoded representation of the input 1001 to the generative LM 1030, and the generative LM 1030 can process the encoded representation of the input 1001 to generate an output 1090, which may include response text and / or other types of data.

[0227] As described herein, in some embodiments, the generative LM 1030 may be configured to access or use (or be able to access or use) plugins / APIs 1095 (which may include one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc.). For example, for certain tasks or operations where the generative LM 1030 is not ideally suited, the model may have instructions (e.g., as a result of training, and / or based on instructions in a given prompt, such as instructions retrieved using RAG component 1092) to access one or more plugins / APIs 1095 (e.g., third-party plugins) to help process the current input. In such an example, when at least part of the prompt is related to a restaurant or weather, the model may access one or more restaurant or weather plugins (e.g., via one or more APIs), sending at least part of the prompt related to a particular plugin / API 1095 to the plugin / API 1095, which can process the information and return an answer to the generative LM 1030, which can then use the response to generate output 1090. This process can be repeated (e.g., recursively) an arbitrary number of iterations and repeated using any number of plugins / APIs 1095 until an output 1090 that resolves each query / question / request / process / action / etc. from input 1001 is generated. Therefore, the model can rely not only on its own knowledge acquired from training on a large dataset and / or from data retrieved using the RAG component 1092, but also on the expertise or optimized properties of one or more external resources (e.g., plugins / APIs 1095).

[0228] In some embodiments, one or more converter engines (TEs) can be implemented. Converter engines can use micro-tensor scaling to optimize performance and accuracy, such as enabling 16-bit floating-point (FP16), 8-bit floating-point (FP8), and / or 4-bit floating-point (FP4) AI processing. For example, a converter engine can use 16-bit or 8-bit floating-point precision and 8-bit or 4-bit floating-point data formats, combined with software algorithms, to improve AI performance and capabilities. By reducing mathematical operations to 8 or 4 bits, TEs can train larger networks faster without compromising accuracy. For example, TEs can include libraries for accelerating converter models on processing devices such as GPUs to provide better performance in training and inference with lower memory utilization. When TEs are combined with other technologies, such as high-speed interconnects between nodes (e.g., using NVLink switches) and tensor cores (which enable mixed-precision computation, such as micro-scaling precision support), server clusters can be more capable of training massive networks (e.g., billions of parameters) at high speeds. Therefore, it can support tensor core precision for FP64, TF32, BF16, FP16, FP8, INT8, FP6 and FP4, as well as CUDA core precision for FP64, FP32, FP16 and BF16.

[0229] The LLM / VLM / MMLM / VLA and other architectures described herein are intended to be examples only, and other suitable architectures may be implemented within the scope of this disclosure.

[0230] Example computing device

[0231] Figure 11This is a block diagram of an example computing device 1100 suitable for implementing some embodiments of the present disclosure. The computing device 1100 may include an interconnect system 1102 directly or indirectly coupled to the following devices: memory 1104, one or more central processing units (CPUs) 1106, one or more graphics processing units (GPUs) 1108, a communication interface 1110, input / output (I / O) ports 1112, input / output components 1114, a power supply 1116, one or more presentation components 1118 (e.g., one or more displays, one or more speakers, etc.), and one or more logic units 1120. In at least one embodiment, one or more computing devices 1100 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 1108 may include one or more vGPUs, one or more CPUs 1106 may include one or more vCPUs, and / or one or more logic units 1120 may include one or more virtual logic units. Thus, one or more computing devices 1100 may include discrete components (e.g., a full GPU dedicated to computing device 1100), virtual components (e.g., a portion of the GPU dedicated to computing device 1100), or a combination thereof.

[0232] although Figure 11 The various blocks are shown as connected via interconnect system 1102 using lines, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 1118 (such as a display device) may be considered I / O component 1114 (e.g., if the display is a touchscreen). As another example, CPU 1106 and / or GPU 1108 may include memory (e.g., memory 1104 may represent a storage device other than the memory of GPU 1108, CPU 1106, and / or other components). Therefore, Figure 11 The computing devices described are for illustrative purposes only. No distinction is made between such categories as “workstation,” “server,” “laptop computer,” “desktop computer,” “tablet computer,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of… Figure 11 Within the scope of computing devices.

[0233] Interconnect system 1102 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 1102 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 1106 may be directly connected to memory 1104. Further, CPU 1106 may be directly connected to GPU 1108. In cases where there is a direct or point-to-point connection between components, interconnect system 1102 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required to be included in computing device 1100.

[0234] The memory 1104 may include any computer-readable medium from a variety of computer-readable media. A computer-readable medium may be any available medium accessible by the computing device 1100. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0235] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1104 may store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, Digital Universal Disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 1100. As used herein, computer storage media does not include the signal itself.

[0236] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and includes any information transmission medium. The term "modulated data signal" can refer to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the above should also be included within the scope of computer-readable media.

[0237] CPU 1106 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1100 to perform one or more of the methods and / or processes described herein. Each CPU 1106 may contain one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling numerous software threads simultaneously. CPU 1106 may contain any type of processor and may contain different types of processors depending on the type of computing device 1100 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 1100, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (such as math coprocessors), computing device 1100 may also include one or more CPUs 1106.

[0238] In addition to or in lieu of one or more CPUs 1106, one or more GPUs 1108 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1100 to perform one or more of the methods and / or processes described herein. One or more GPUs 1108 may be integrated GPUs (e.g., with one or more CPUs 1106) and / or one or more GPUs 1108 may be discrete GPUs. In embodiments, one or more GPUs 1108 may be coprocessors of one or more CPUs 1106. GPUs 1108 may be used by computing device 1100 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 1108 may be used for general-purpose computing on a GPU (GPGPU). GPUs 1108 may include hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. GPUs 1108 may produce pixel data of an output image in response to rendering commands (e.g., rendering commands received from CPUs 1106 via a host interface). GPU 1108 may include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 1104. GPU 1108 may include two or more GPUs operating in parallel (e.g., via links). The links may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 1108 may produce pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for an analog image). Each GPU may include its own memory or may share memory with other GPUs.

[0239] In addition to or in lieu of CPU 1106 and / or GPU 1108, logic unit 1120 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1100 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 1106, one or more GPUs 1108, and / or one or more logic units 1120 may execute any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 1120 may be a portion of one or more CPUs 1106 and / or GPUs 1108 and / or integrated into one or more CPUs 1106 and / or GPUs 1108, and / or one or more logic units 1120 may be discrete components or otherwise external to CPUs 1106 and / or GPUs 1108. In an embodiment, one or more of the logic units 1120 may be coprocessors of one or more of the CPU 1106 and / or one or more of the GPU 1108.

[0240] Examples of logic unit 1120 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree lateral unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), programmable vision accelerator (PVA) (which may include one or more direct memory access (DMA) systems), and one or more vision or vector processing units (VPUs). This includes one or more pixel processing engines (PPEs) (e.g., comprising a 2D array of processing elements, each of which communicates north, south, east, and west with one or more other processing elements in the array), one or more decoupled accelerators or units (e.g., decoupled lookup table (DLUT) accelerators or units), vision processing units (VPUs), optical flow accelerators (OFAs), field-programmable gate arrays (FPGAs), neuromorphic chips, quantum processing units (QPUs), associative processing units (APUs), arithmetic logic units (ALUs), application-specific integrated circuits (ASICs), floating-point units (FPUs), input / output (I / O) elements, peripheral component interconnects (PCIs) or fast peripheral component interconnects (PCIe) elements, etc.

[0241] The communication interface 1110 may include one or more receivers, transmitters, and / or transceivers enabling the computing device 1100 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). The communication interface 1110 may include components and functions for enabling communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wirelessband), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 1120 and / or the communication interface 1110 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via interconnect system 1102 to one or more GPUs 1108 (e.g., memory of one or more GPUs 1108).

[0242] I / O port 1112 enables computing device 1100 to be logically coupled to other devices including I / O component 1114, one or more presentation components 1118, and / or other components, some of which may be built into (e.g., integrated into) computing device 1100. Illustrative I / O component 1114 includes microphones, mice, keyboards, joysticks, gamepads, game controllers, satellite dish antennas, scanners, printers, wireless devices, etc. I / O component 1114 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by the user. In some cases, input may be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, pen recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of computing device 1100. Computing device 1100 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. Additionally, computing device 1100 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables motion detection. In some examples, computing device 1100 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0243] The power supply 1116 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 1116 may provide power to the computing device 1100 so that the components of the computing device 1100 can operate.

[0244] The presentation component 1118 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 1118 may receive data from other components (e.g., GPU 1108, CPU 1106, etc.) and output the data (e.g., as images, videos, sounds, etc.).

[0245] Example network environment

[0246] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 11 It is implemented on one or more instances of one or more computing devices 1100—for example, each device may include similar components, features, and / or functions of one or more computing devices 1100. Furthermore, in the case of implementing back-end devices (e.g., servers, NAS, etc.), the back-end devices may be included as part of a data center, such as, but not limited to, those described herein.

[0247] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0248] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein for the server can be implemented on any number of client devices.

[0249] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or application may respectively include network-based service software or applications. In embodiments, one or more client devices may use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open-source software network application framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0250] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers at the state, region, country, global, etc.). The core server may assign at least a portion of the functionality to the edge server if the connection to the user (e.g., a client device) is relatively close to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0251] One or more client devices may include the information described in this article. Figure 11 At least some of the components, features, and functions of one or more example computing devices 1100 described. By way of example and not limitation, the client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming equipment or system, entertainment system, vehicle computer system, embedded system controller, chat booth, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0252] Example Terms

[0253] A. One or more processors, including processing circuitry, the processing circuitry being configured to: store one or more spatial features associated with one or more first images depicting an object in a memory; determine one or more attributes associated with the object using one or more neural networks and based at least on the one or more spatial features and second image data representing one or more second images depicting the object; and cause a machine to perform one or more operations based at least on the one or more attributes associated with the object.

[0254] B. One or more processors according to paragraph A, wherein the processing circuitry is further configured to: determine a classification associated with the object based at least on at least one of the one or more spatial features or the second image data, wherein the machine is further configured to perform the one or more operations based at least on the classification.

[0255] C. One or more processors according to any one of paragraphs A or B, wherein said one or more processors are in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more analog operations; a system for performing one or more digital twin operations; a system for performing one or more optical transmission simulations; a system for performing collaborative content creation of 3D assets; a system for performing one or more wireless cellular transmissions using a wireless cellular network; a system for providing one or more cloud gaming applications; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing one or more conversational AI operations; a system for using one or more... Systems that perform operations using multiple large language model LLMs; systems that perform operations using one or more visual language model VLMs; systems that perform operations using one or more multimodal language model MMLMs; systems that perform operations using one or more visual-language-action VLA models; systems that perform one or more conversational AI operations; systems that perform one or more synthetic data generation operations; systems that render at least one of virtual reality content, augmented reality content, or mixed reality content; systems that use or deploy one or more inference microservices; systems that deploy one or more machine learning models within services or microservices and OS-level virtualization packages (e.g., containers); systems that contain one or more virtual machines VMs; systems that are at least partially implemented in a data center; or systems that are at least partially implemented using cloud computing resources.

[0256] D: A method comprising: storing in memory a plurality of spatial feature instances corresponding to a plurality of processing iterations of a convolutional neural network (CNN) feature extractor, each of the plurality of processing iterations comprising: the CNN feature extractor processing image data representing one or more images depicting an object, the image data being acquired using one or more image sensors at a time corresponding to the individual processing iteration; determining one or more states associated with one or more light signals of the object using one or more CNN classifiers and at least based on the plurality of spatial feature instances; and causing a machine to perform one or more planning, navigation, or control operations at least based on the one or more states of the one or more light signals.

[0257] E. The method according to paragraph D further includes: using the one or more CNN classifiers and based on at least one of the plurality of spatial feature instances to determine a classification associated with the object, wherein causing the machine to perform the one or more operations is also based at least on the classification.

[0258] F. The method according to either paragraph D or paragraph E, further comprising: generating a combined spatial feature by splicing together the plurality of spatial feature instances stored in the memory, wherein determining the one or more states associated with the one or more light signals is based at least on the combined spatial feature.

[0259] G. The method according to any one of paragraphs DF, further comprising: determining that the number of instances of the plurality of spatial feature instances stored in the memory satisfies a threshold number of instances, wherein determining that the one or more states associated with the one or more light signals are also satisfied at least based on the threshold number of instances.

[0260] H. The method according to any of paragraphs DG further includes: removing one or more individual spatial feature instances from the memory, based at least on the number of individual spatial feature instances reaching capacity.

[0261] I. The method according to any one of paragraphs DH, further comprising: storing one or more placeholder features associated with one or more predetermined values ​​in the memory, wherein determining the one or more states associated with the one or more light signals is also based at least on the one or more placeholder features.

[0262] J. The method according to any one of paragraphs DI, wherein the one or more states include or indicate at least one of the following: cycle state; left turn state; right turn state; braking state; reversing state; hazard state; hazard light state; warning state; on state; or off state.

[0263] K. A system comprising: one or more processors, the one or more processors being configured to: generate one or more spatial features associated with the one or more images using one or more neural networks and based at least on image data representing one or more images; store the one or more spatial features in a memory; determine one or more states associated with one or more light signals of an object using the one or more neural networks and based at least on the one or more spatial features; and cause a machine to perform one or more operations based at least on the one or more states of the one or more light signals.

[0264] L. According to the system described in paragraph K, wherein the one or more processors are further configured to: determine a classification associated with the object using the one or more neural networks and based at least on at least one of the one or more spatial features, wherein the machine is further configured to perform the one or more operations based at least on the classification.

[0265] M. The system according to any one of paragraphs K or L, wherein the one or more processors are further configured to: generate combined spatial features by concatenating the one or more spatial features with one or more second spatial features stored in the memory, wherein the one or more second spatial features are associated with one or more second images, wherein the one or more states associated with the one or more light signals are determined at least based on the combined spatial features.

[0266] N. The system according to any one of paragraphs KM, wherein the one or more processors are further configured to: determine that the capacity associated with the memory satisfies a threshold, wherein the one or more states associated with the one or more light signals are further determined at least based on the capacity satisfying the threshold.

[0267] O. The system according to any one of paragraphs KN, wherein the one or more processors are further configured to: remove one or more second spatial features from the memory, the one or more second spatial features being associated with one or more second images, based at least on the one or more spatial features being stored in the memory.

[0268] P. The system according to any one of paragraphs KO, wherein the one or more processors are further configured to: store one or more placeholder features associated with one or more predefined values ​​in the memory, wherein the one or more states associated with the one or more light signals are further determined at least based on the one or more placeholder features.

[0269] Q. The system according to any one of paragraphs KP, wherein the one or more processors are further configured to: generate one or more second spatial features by at least copying the one or more spatial features; and store the one or more second spatial features in the memory, wherein the one or more states associated with the one or more light signals are further determined at least based on the one or more second spatial features.

[0270] R. A system according to any one of paragraphs KQ, wherein the one or more states associated with the one or more light signals include at least one or more of the following: cycle state; left turn state; right turn state; braking state; reversing state; hazard state; hazard light state; warning state; on state; or off state.

[0271] S. A system according to any one of paragraphs KR, wherein: the one or more spatial features associated with the one or more images are determined using one or more convolutional neural network (CNN) extractors of the one or more neural networks; and the one or more states associated with the one or more light signals are extracted using one or more CNN extractors of the one or more neural networks.

[0272] T. A system according to any one of paragraphs KS, wherein the system includes at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more analog operations; a system for performing one or more digital twin operations; a system for performing one or more optical transmission simulations; a system for performing collaborative content creation of 3D assets; a system for performing one or more wireless cellular transmissions using a wireless cellular network; a system for providing one or more cloud gaming applications; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing one or more conversational AI operations; a system for performing one or more large language modules Systems for performing operations using LLMs; systems for performing operations using one or more Visual Language Models (VLMs); systems for performing operations using one or more Multimodal Language Models (MMLMs); systems for performing operations using one or more Visual-Language-Action (VLA) models; systems for performing one or more conversational AI operations; systems for performing one or more synthetic data generation operations; systems for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; systems that use or deploy one or more inference microservices; systems that include one or more machine learning models deployed in services or microservices and OS-level virtualization packages (e.g., containers); systems that include one or more virtual machines (VMs); systems that are at least partially implemented in a data center; or systems that are at least partially implemented using cloud computing resources.

[0273] This disclosure can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program modules, which are executed by a computer or other machine (such as a personal data assistant or other handheld device). Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure can also be practiced in distributed computing environments, where tasks are performed by remote processing devices linked via a communication network.

[0274] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0275] This document describes in detail the subject matter of this disclosure to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the discloser has envisioned that the claimed subject matter may be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of the method employed, these terms should not be construed as suggesting any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. A method comprising: Multiple spatial feature instances, each corresponding to a multiple processing iteration of a convolutional neural network (CNN) feature extractor, are stored in a memory. Each of the multiple processing iterations includes: the CNN feature extractor processing image data representing one or more images of a depicted object, the image data being obtained using one or more image sensors at a time corresponding to the individual processing iteration. Using one or more CNN classifiers and based at least on the plurality of spatial feature instances, determine one or more states associated with one or more light signals of the object; and This enables the machine to perform one or more planning, navigation, or control operations based on at least one or more states of the one or more light signals.

2. The method according to claim 1, further comprising: Using one or more CNN classifiers and based on at least one spatial feature instance from the plurality of spatial feature instances, determine the classification associated with the object. The machine is made to perform the one or more operations based at least on the classification.

3. The method according to claim 1, further comprising: Combined spatial features are generated by concatenating the multiple spatial feature instances stored in the memory. The determination of the one or more states associated with the one or more light signals is based at least on the combined spatial features.

4. The method according to claim 1, further comprising: The number of instances of the plurality of spatial feature instances stored in the memory is determined to satisfy a threshold number of instances. The determination of the one or more states associated with the one or more light signals is also based on at least the satisfaction of the threshold instance number.

5. The method according to claim 1, further comprising: At least when the number of individual spatial feature instances reaches capacity, one or more individual spatial feature instances are removed from the memory.

6. The method according to claim 1, further comprising: One or more placeholder features associated with one or more predetermined values ​​are stored in the memory. The determination of the one or more states associated with the one or more light signals is also based at least on the one or more placeholder features.

7. The method of claim 1, wherein the one or more states include or indicate at least one of the following: Cyclic state; Left turn status; Right turn status; Braking state; Reversing position; Dangerous situation; Emergency light status; Warning status; On status; or The device is in the off state.

8. A system comprising: One or more processors, said one or more processors being used for: Using one or more neural networks and based at least on image data representing one or more images, generate one or more spatial features associated with said one or more images; Store one or more spatial features in memory; Using the one or more neural networks and based at least on the one or more spatial features, determine one or more states associated with one or more light signals of the object; as well as This enables the machine to perform one or more operations based on at least one or more states of the one or more light signals.

9. The system of claim 8, wherein the one or more processors are further configured to: Using the one or more neural networks and based at least one of the one or more spatial features, determine the classification associated with the object. The machine is further configured to perform one or more operations based at least on the classification.

10. The system of claim 8, wherein the one or more processors are further configured to: A combined spatial feature is generated by concatenating the one or more spatial features with one or more second spatial features stored in the memory, wherein the one or more second spatial features are associated with one or more second images. The one or more states associated with the one or more light signals are determined at least based on the combined spatial features.

11. The system of claim 8, wherein the one or more processors are further configured to: Determine that the capacity associated with the memory meets a threshold. The one or more states associated with the one or more light signals are further determined based at least on the capacity satisfying the threshold.

12. The system of claim 8, wherein the one or more processors are further configured to: remove one or more second spatial features from the memory, the one or more second spatial features being associated with one or more second images, based at least on the fact that the one or more spatial features are stored in the memory.

13. The system of claim 8, wherein the one or more processors are further configured to: One or more placeholder features associated with one or more predefined values ​​are stored in the memory. The one or more states associated with the one or more light signals are further determined based at least on the one or more placeholder features.

14. The system of claim 8, wherein the one or more processors are further configured to: At least one or more second spatial features are generated by copying the one or more said spatial features; and The one or more second spatial features are stored in the memory. The one or more states associated with the one or more light signals are further determined based at least on the one or more second spatial features.

15. The system of claim 8, wherein the one or more states associated with the one or more light signals include at least one or more of the following: Cyclic state; Left turn status; Right turn status; Braking state; Reversing position; Dangerous situation; Emergency light status; Warning status; On status; or The device is in the off state.

16. The system according to claim 8, wherein: The one or more spatial features associated with the one or more images are determined using one or more convolutional neural network (CNN) extractors from the one or more neural networks; as well as The one or more states associated with the one or more light signals are extracted using one or more CNN extractors from the one or more neural networks.

17. The system of claim 9, wherein the system comprises at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing one or more optical transmission simulations; A system for collaborative content creation of 3D assets; A system for performing one or more wireless cellular transmissions using a wireless cellular network; A system that provides one or more cloud gaming applications; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing one or more conversational AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models (MMLMs); A system for performing operations using one or more visual-language-action (VLA) models; A system for performing one or more conversational AI operations; A system for performing one or more synthetic data generation operations; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system that uses or deploys one or more inference microservices; A system that includes one or more machine learning models deployed in services or microservices and OS-level virtualization packages (e.g., containers); A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

18. One or more processors, including processing circuitry, said processing circuitry being used to: Store one or more spatial features associated with one or more first images of the depicted object in memory; Using one or more neural networks and based at least on the one or more spatial features and second image data representing one or more second images depicting the object, determine one or more attributes associated with the object; and This enables the machine to perform one or more operations based on at least one or more attributes associated with the object.

19. One or more processors according to claim 18, wherein the processing circuitry is further configured to: A classification associated with the object is determined based on at least one or more of the spatial features or at least one of the second image data. The machine is further configured to perform one or more operations based at least on the classification.

20. One or more processors according to claim 18, wherein the one or more processors are located in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing one or more optical transmission simulations; A system for collaborative content creation of 3D assets; A system for performing one or more wireless cellular transmissions using a wireless cellular network; A system that provides one or more cloud gaming applications; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing one or more conversational AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models (MMLMs); A system for performing operations using one or more visual-language-action (VLA) models; A system for performing one or more conversational AI operations; A system for performing one or more synthetic data generation operations; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system that uses or deploys one or more inference microservices; A system that includes one or more machine learning models deployed in services or microservices and OS-level virtualization packages (e.g., containers); A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

Images