Memory binding and implicit data persistence for firmware control of hardware accelerators

Abstract

The present invention relates to firmware-controlled memory binding and implicit data persistence for hardware accelerators. Various examples, systems, and methods relate to facilitating memory usage of hardware accelerators. A system can check whether an on-chip memory region has been set up for a given use. If so, the system can reuse it. If not, the system can allocate a new region and connect it to a larger off-chip memory. The system can determine whether to load data to on-chip memory or save data back to off-chip memory based on various rules.

Description

Technical Field

[0001] Implementing low-level orchestration of heterogeneous hardware accelerators, including independent addressable components such as vector processing units (VPUs), direct memory access (DMA) controllers, programmable or pixel processing engines (PPEs), and / or other components, presents technical challenges. Some existing systems rely on fixed-function firmware interfaces that use coarse-grained command structures to configure resources, serialize tasks, and / or initialize memory transactions (e.g., DRAM-to-SRAM transfers, SRAM eviction policies). These systems often expose monolithic task interfaces tightly coupled to specific programming models, resulting in limited flexibility and incomplete utilization of concurrent execution resources. Many existing solutions are insufficient to support persistent memory mapping across task windows, overlapping execution of accelerator commands, and / or user-space construction of command buffers with fine-grained hardware control, instead relying on static abstractions, firmware-level scheduling, and / or high-latency verification mechanisms. These restrictions prevent the system from issuing hardware configuration and / or synchronization commands with low overhead, limit command reuse and memory persistence across submission boundaries, and / or increase latency in runtime accelerator dispatch for performance-critical workloads. Summary of the Invention

[0002] Implementations of this disclosure relate to systems and methods for constructing, submitting, and executing accelerator command buffers in user space with minimal and / or reduced host-side verification overhead. For example, systems and methods according to this disclosure can receive command buffers constructed by a user-mode application that references at least one resource identifier stored in a resource table inaccessible to a non-privileged execution context. That is, at least one (e.g., each) resource identifier may correspond to a pre-verified configuration (e.g., DMA settings, VPU parameters, and / or execution affinity data) populated by a kernel-mode driver (KMD) at initialization time. These systems and methods can be used to submit command buffers via a submission queue, signal execution readiness, and trigger firmware-controlled dispatch to hardware components of the hardware accelerator. In some implementations, the systems and methods can verify resource usage via firmware-managed metadata and track command execution across multiple submission interfaces. Furthermore, these systems and methods can reuse previously constructed buffers, partition command buffers across submissions, and / or dispatch command buffers with affinity restrictions to avoid resource interference. These implementations provide reduced verification latency, support for command buffer reuse, and / or fine-grained hardware control by exposing the low-level firmware execution path to user-mode applications.

[0003] Implementations of this disclosure relate to systems and methods for binding local accelerator memory regions to external memory locations and maintaining the persistence of data across task windows. For example, systems and methods according to this disclosure can allocate local memory regions (e.g., L2SRAM) and bind them to DRAM-backed external memory regions via firmware-level registration. That is, at least one (e.g., each) binding operation can generate a device pointer identifying the DRAM location and reference the external memory via a firmware-managed resource table. These systems and methods can be used to configure DMA operations, store binding metadata in resource tables, and / or manage transfer policies (such as fill on miss and flush on eviction). In some implementations, these systems and methods can reuse active L2SRAM bindings across multiple task windows, perform reference counting for concurrent bindings, and / or perform lazy data initialization and lazy data eviction based on firmware state. Furthermore, firmware can handle task-level memory persistence and / or prevent redundant transfers by associating memory bindings with execution context identifiers. These implementations provide persistent access to the allocation of supporting DRAM, facilitate memory reuse across composable operators, and / or reduce transfer overhead during non-contiguous task submissions.

[0004] Implementations of this disclosure relate to systems and methods for dispatching command buffers, which include low-level commands controlling independent hardware engines of heterogeneous hardware accelerators. For example, the systems and methods of this disclosure can construct command buffers comprising sequences of hardware configuration and / or synchronization commands, at least one (e.g., each) of which references a resource identifier stored in a firmware-accessible resource table. That is, at least one (e.g., each) command can initiate a DMA transfer, configure a vector processing unit (VPU), start a programmable engine, and / or synchronize execution via a barrier command. These systems and methods can be used to initiate execution in placement order and allow overlapping execution based on hardware availability. In some implementations, these systems and methods can validate command sequences via a firmware-managed state machine, dispatch commands out of order, and / or construct multiple DMA configurations within a single VPU execution. Furthermore, the command buffer can be constructed in user space and committed with minimal and / or reduced transformation or driver interaction. These implementations provide a flexible and improved orchestration mechanism for controlling parallel execution pipelines across hardware engines, e.g., without relying on coarse-grained task sequencing.

[0005] Some implementations involve a system comprising one or more processors configured to allocate a local memory region in the local memory of a hardware accelerator bound to an external memory region via a firmware interface, the allocation returning a device pointer corresponding to the binding, the binding being tracked using a resource identifier in a resource table corresponding to a non-privileged execution context. In some implementations, the processor is configured to provide the device pointer to a memory region accessible by the non-privileged execution context for at least one subsequent construction of at least one command buffer identifying the binding from the local memory region to the external memory region.

[0006] Some implementations relate to a system in which one or more processors are configured to: construct the at least one command buffer in the external memory region of the hardware accelerator, the at least one command buffer including at least one command identifying a device pointer corresponding to the binding from the local memory region to the external memory region. Some implementations relate to a system in which one or more processors are configured to: cause the at least one command to be executed by a direct memory access (DMA) controller via the local memory region using the at least one command buffer, wherein the at least one command corresponds to at least one of: (i) reading data from the external memory region into the local memory region, or (ii) writing data from the local memory region into the external memory region.

[0007] Some implementations relate to a system where allocation includes: providing a request via the firmware interface for allocating a local memory region in the local memory of the hardware accelerator and binding the local memory region to the external memory region. In some implementations, allocation includes: receiving via the firmware interface a device pointer corresponding to the binding from the local memory region to the external memory region. In some implementations, the request is generated by a privileged execution context using a firmware access application programming interface (API) as a firmware-level configuration command, and the firmware-level configuration command identifies the binding from the local memory region to the external memory region.

[0008] Some implementations relate to a system in which the at least one command buffer includes descriptor data identifying the device pointer, and wherein the descriptor data is updated prior to execution to identify a memory access path from the external memory region via the local memory region, and wherein the memory access path corresponds to a patched physical address in the local memory region identified for the execution of at least one command. Some implementations relate to a system in which the device pointer corresponds to an address indirection including the binding from the local memory region to the external memory region, the address indirection being identified by a DMA controller to implement at least one memory access command during execution.

[0009] Some implementations relate to a system in which one or more processors are configured to: construct or receive a second command buffer in the external memory region of the hardware accelerator, the second command buffer including at least one second command identifying a resource corresponding to the local memory region. In some implementations, the processor is configured to: provide commit information of the second command buffer to a commit queue corresponding to the non-privileged execution context. In some implementations, the processor is configured to: provide a commit indicator corresponding to the commit information to a signaling interface corresponding to the commit queue. In some implementations, the processor is configured to: cause the second command buffer to be executed by at least one hardware component of the hardware accelerator.

[0010] Some implementations relate to a system in which one or more processors are configured to: determine, before allocating the local memory region, whether a previously allocated local memory region bound to the external memory region exists for the non-privileged execution context. In some implementations, the processor is configured to: increment a reference count corresponding to the binding in response to determining that the previously allocated local memory region exists. In some implementations, the processor is configured to: send a request to the firmware interface to allocate the local memory region in the local memory of the hardware accelerator bound to the external memory region, according to at least one access policy, in response to determining that the previously allocated local memory region does not exist.

[0011] Some implementations involve a system in which one or more processors are configured to: decrement the reference count corresponding to the binding in response to a release request. In some implementations, the processor is configured to: release the local memory region in response to determining that the reference count satisfies a release condition.

[0012] Some implementations relate to a system in which one or more processors are configured to: initiate a refresh operation in response to a refresh request to transfer data from the local memory region to the external memory region, the refresh operation occurring in response to a user context switch or in response to an invalidation request used to trigger reinitialization of the local memory region during subsequent access; and initiate an invalidation operation to mark data stored in the local memory region as invalid. In some implementations, in response to the user context switch, the local memory region is initialized with data from the external memory region and the local memory region is refreshed to the external memory region, and the refresh is also capable of being performed in response to an explicit request.

[0013] Some implementations relate to a system where the bindings are established according to at least one access policy, the at least one access policy including a first policy for initializing the local memory region with data from the external memory region, or a second policy for flushing data from the local memory region to the external memory region during a release. Some implementations relate to a system where the firmware interface maintains at least two concurrent bindings between multiple local memory regions and corresponding multiple external memory regions for the non-privileged execution context.

[0014] Some implementations relate to a system in which the firmware interface includes a compatibility layer configured to translate memory allocations and at least one access request from at least one software application implemented for the hardware accelerator, the compatibility layer updating at least one request for the allocation to access the local memory region and the binding to the external memory region via the firmware interface. Some implementations relate to a system in which the at least one command buffer includes descriptor data identifying the device pointer, wherein the descriptor data is modified by firmware to identify the source address and destination address of a DMA transaction performed by a DMA controller, the source address or destination address corresponding to the local memory region and the external memory region bound by the device pointer, the descriptor data including a transfer size and at least one access policy.

[0015] Some implementations relate to a system-on-a-chip (SoC) including: processing circuitry configured to: send a request via a firmware interface to allocate a local memory region in local memory for a hardware accelerator and bind the local memory region to an external memory region. In some implementations, the processing circuitry is configured to: receive via the firmware interface a device pointer corresponding to the binding from the local memory region to the external memory region, the binding being tracked using a resource identifier in a resource table corresponding to a non-privileged execution context. In some implementations, the processing circuitry is configured to: store the device pointer in a memory region accessible by the non-privileged execution context for at least one subsequent construction of at least one command buffer identifying the binding from the local memory region to the external memory region.

[0016] Some implementations involve a method comprising: allocating a local memory region in the local memory of a hardware accelerator bound to an external memory region by one or more processors via a firmware interface, the allocation returning a device pointer corresponding to the binding, the binding being tracked using a resource identifier in a resource table corresponding to a non-privileged execution context. In some implementations, the method includes: providing the device pointer by the one or more processors to a memory region accessible by the non-privileged execution context for at least one subsequent construction of at least one command buffer identifying the binding from the local memory region to the external memory region.

[0017] Some implementations involve a method further comprising: constructing the at least one command buffer by the one or more processors in the external memory region of the hardware accelerator, the at least one command buffer including at least one command identifying a device pointer corresponding to the binding from the local memory region to the external memory region. In some implementations, the method further comprises: causing the one or more processors to execute the at least one command via the local memory region using the external memory region by a direct memory access (DMA) controller utilizing the at least one command buffer, wherein the at least one command corresponds to at least one of: (i) reading data from the external memory region into the local memory region, or (ii) writing data from the local memory region into the external memory region.

[0018] Some implementations involve a method in which allocation includes: providing a request via the firmware interface for allocating the local memory region in the local memory of the hardware accelerator and binding the local memory region to the external memory region. In some implementations, a device pointer corresponding to the binding from the local memory region to the external memory region is received via the firmware interface. In some implementations, the request is generated as a firmware-level configuration command by a privileged execution context using a firmware access application programming interface (API), and the firmware-level configuration command identifies the binding from the local memory region to the external memory region.

[0019] Some implementations relate to a system in which one or more processors perform operations including: a system for managing memory persistence using firmware-level resource tracing; a system for binding one or more local memory regions to one or more DRAM-supporting regions via the firmware interface; a system for allocating one or more memory regions using one or more access policies; a system for inserting one or more refresh or invalidation operations based on DRAM persistence; a system for storing one or more device pointers for command buffer reuse; a system for implementing one or more multimodal language models (MMLMs); a system for implementing one or more large language models (LLMs); a system for implementing one or more small language models (SLMs); a system for implementing one or more visual language models (VLMs); a system for generating synthetic data; a system for generating synthetic data using AI; a control system for autonomous or semi-autonomous machines; a perception system for autonomous or semi-autonomous machines; a system for performing digital twin operations; a system for performing optical transmission simulation; a system for performing remote operations; a system implemented using edge devices; a system implemented using robots; a system for performing conversational AI operations; a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources. Attached Figure Description

[0020] The system and method described below with reference to the accompanying drawings are for (i) the construction of user-space buffers and the execution of firmware control for hardware accelerators, (ii) memory binding and implicit data persistence of firmware control for hardware accelerators, and / or (iii) command dispatch and resource-based orchestration of firmware control for hardware accelerators, wherein: Figure 1 This is a block diagram illustrating an example of a system based on some implementations of this disclosure; Figure 2This is a flowchart illustrating an example of a method for constructing a user-space buffer and executing firmware control for a hardware accelerator in an accelerator pipeline, according to some implementations of this disclosure. Figure 3 This is a flowchart illustrating an example of a method for firmware-controlled memory binding and implicit data persistence for hardware accelerators in an accelerator pipeline, according to some implementations of this disclosure. Figure 4 This is a flowchart illustrating an example of a method for command dispatching and resource-based orchestration for firmware control of hardware accelerators in an accelerator pipeline, according to some implementations of this disclosure. Figure 5 An interface architecture for task submission based on a command buffer for implementing a programmable vision accelerator (PVA) according to some implementations of this disclosure is described; Figure 6 A system view of a resource registration and verification pipeline for PVA implemented according to some implementations of this disclosure is depicted; Figure 7 The execution path from user space to firmware via a submission queue and signaling interface is described according to some implementations of this disclosure; Figure 8 The present disclosure describes a system comprising multiple queues corresponding to at least one user context and / or privileged context, according to some implementations thereof; Figure 9 The present disclosure describes a system comprising multiple input descriptors for use with a direct memory access (DMA) configuration, wherein at least one (e.g., each) input descriptor contains at least one source (SRC) slot and one destination (DST) slot. Figure 10 An example control flow for processing command buffers by the system firmware, according to some implementations of this disclosure, is described; Figure 11A and Figure 11B A vector processing unit (VPU) state machine according to some implementations of this disclosure is described, which can be facilitated by the firmware components of the system using a command buffer for execution; Figure 12 The instantiated DMA state machine for each DMA configuration set in a system according to some implementations of this disclosure is described; Figure 13 A pixel processing engine (PPE) state machine according to some implementations of this disclosure is described, which is controlled by the sequential execution of commands issued in the command buffer of the system; Figure 14A system including L2SRAM implemented as a shared local memory accessible to a DMA controller is described according to some implementations of this disclosure; Figure 15 The present disclosure describes some implementations of facilitating the mapping of local memory to device pointers for persistent use across operator submissions; Figure 16A A command buffer according to some implementations of this disclosure is described, which uses a predefined set of resources stored in a resource table to execute a VPU application for matrix addition; Figure 16B The representation of some implementations according to this disclosure is described. Figure 16A The execution timeline of firmware control over the command buffer described in the document; Figure 16C The diagram illustrates the execution timeline of back-to-back dispatch of two VPU programs by the system's firmware components, according to some implementations of this disclosure. Figure 16D The diagram illustrates the execution timeline of a single VPU program utilizing three DMA configuration sets under the control of the system's firmware components, according to some implementations of this disclosure. Figure 17 A command buffer according to some implementations of this disclosure is described, which executes query commands to capture and extract timestamps using resource tables before and after VPU program execution; Figure 18A Examples of sensor locations having a corresponding field of view or sensing field, such as an autonomous or semi-autonomous machine, according to at least some embodiments of this disclosure. Figure 18B This is an illustration of examples of the locations of components and sensors on an autonomous or semi-autonomous vehicle according to at least some embodiments of this disclosure. Figure 18C This is a block diagram of an example system architecture for autonomous or semi-autonomous vehicles, robots, and / or other machine types according to at least some embodiments of this disclosure; Figure 18D This is a block diagram of an example architecture of a computing system (e.g., a system-on-a-chip (SoC)) according to at least some embodiments of the present disclosure; Figure 18E This is a system diagram of communication between a cloud-based server and an example autonomous or semi-autonomous vehicle, robot, and / or other machine type, according to at least some embodiments of this disclosure. Figure 19This is a system diagram illustrating three computer ecosystems according to at least some embodiments of the present disclosure, including a computing system for generating or creating artificial intelligence (AI) (e.g., AI training and validation data), a computing system for training artificial intelligence, and a computing system for deploying AI at the edge; Figure 20 This is a block diagram of an example computing system for generative artificial intelligence (AI) according to at least some embodiments of this disclosure; and Figure 21 This is a block diagram of an example computing device according to at least some embodiments of the present disclosure. Detailed Implementation

[0021] Systems and methods related to (i) userspace buffer construction and firmware control execution of a hardware accelerator, (ii) memory binding and implicit data persistence of firmware control for a hardware accelerator, and / or (iii) command dispatch and resource-based orchestration of firmware control for a hardware accelerator are disclosed. Although this disclosure may relate to examples of autonomous or semi-autonomous vehicles, robots, and / or other machine types 1800 (alternately referred to herein as "vehicle 1800," "self-vehicle 1800," "machine 1800," self-machine 1800, "robot 1800," and / or "self-robot 1800"), examples are related to... Figures 18A-18EThe description is provided below, but it is not intended to be limiting. For example, the systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles and machines, manned and unmanned robots or robotic platforms (e.g., autonomous mobile robots (AMRs), humanoid robots, robotic arms and / or end effectors), warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, boats, shuttles (e.g., autonomous taxis), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones and / or other types of vehicles, robots or machines. Furthermore, while this disclosure may describe buffer and memory management with respect to firmware control in a programmable accelerator pipeline (e.g., user-space buffer construction and firmware control execution of a hardware accelerator, memory binding and data persistence of firmware control for a hardware accelerator, command dispatch and resource-based orchestration of firmware control for a hardware accelerator), this is not intended to be limiting. The systems and methods described herein can be used in augmented reality (AR), virtual reality (VR), mixed reality (MR), robotics, security and supervision (e.g., smart cities), autonomous or semi-autonomous machine applications, industrial manufacturing, simulation, and / or any other technology space that can utilize hardware-level accelerator task scheduling and memory persistence. In some implementations, the systems, methods, and / or processes described herein can use... Figures 18A-18E Example machine 1800 Figure 19 Example calculation of ecosystem 1900, Figure 20 Example generative language model systems 2000 and / or Figure 21 The example computing device 2100 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the same task.

[0022] This disclosure relates to systems and methods for constructing and providing (e.g., submitting) command buffers using a non-privileged execution context to initiate hardware accelerator execution with reduced host-side complexity. For example, systems and methods according to this disclosure facilitate the construction of command buffers in user-mode memory that reference resource identifiers (e.g., verified during initialization) stored in a resource table. The command buffers can be submitted to firmware-controlled signaling and scheduling interfaces to initiate execution using firmware-controlled hardware orchestration. That is, constructing a command buffer may include embedding references to DMA configurations, accelerator memory maps, and / or computation programs, thereby facilitating the execution of runtime tasks without requiring re-verification.

[0023] Some techniques for accelerator task submission rely on fixed-function driver stacks or privileged execution contexts for task verification, transformation, and initiation, limiting flexibility and introducing latency. These methods typically require repetitive transformation of task data, increasing host overhead and limiting direct interaction with firmware-level resources. Furthermore, firmware implementations often rely on strict task sequences, reducing parallelism between memory setup and task execution. Technical limitations relate to how these systems configure, verify, and / or initiate computational workloads on hardware accelerators. For example, sequential verification processes can prevent the pipelined operation of memory operations and computational tasks. Fixed driver interfaces limit the construction of user-mode command buffers and enforce static task submission workflows. The improved implementation described in this paper provides firmware-assisted initiation of user-mode constructed command buffers, allowing resource identifiers to reference pre-verified configurations with minimal and / or reduced host overhead.

[0024] The system and method of this disclosure provide improvements to the accelerator control pipeline by supporting direct submission of command buffers, resource table lookups for rapid verification, and a decoupled signaling interface for triggering task execution. For example, user-mode applications can construct command buffers referencing hardware configurations via resource identifiers, submit them to a firmware-controlled signaling interface, and trigger task execution without re-verification. The system supports hardware component execution based on fine-grained resource references, including memory binding and DMA setup, thereby reducing CPU cycles consumed during accelerator task startup. Furthermore, firmware-controlled execution allows for overlap between memory setup and computation task execution, improving task throughput and utilization of heterogeneous processing units.

[0025] These systems and methods can construct command buffers in memory, the command buffers including at least one command identifying at least one resource. In some implementations, workloads can be allocated and / or populated to the interface. In some implementations, the command buffers can be constructed in chunks and can be retrieved by firmware via direct memory access (DMA) with prefetch and / or linking logic (e.g., low-level operations to be executed synchronously and / or asynchronously by the firmware, such as VPU startup, DMA setup, memory copy operations, barriers, fences, and / or status reports). Furthermore, resources can be maintained per context, and verified items can be stored in a resource table (e.g., DMA configuration, executables, DRAM buffers). These systems and methods can provide commit information for the command buffers to a commit queue corresponding to a non-privileged execution context (e.g., including fields such as buffer chunk offset, pre-fence, post-fence, and / or engine affinity mask). In some implementations, commit information can be written to the queue (e.g., user space pushes a queue entry including the queue ID and / or a new tail index of the queue to the firmware). These systems and methods can provide a commit indicator corresponding to the commit message to a signaling interface corresponding to the commit queue (e.g., a Command Completion Queue (CCQ) registration interface exposed to user space). In some implementations, the updated tail index can be written to a register (e.g., a CCQ register). For example, a commit indicator can be written to a memory-mapped CCQ register, indicating the new tail index of the commit queue. These systems and methods can enable at least one hardware component of a hardware accelerator (e.g., a VPU, DMA engine, PPE) to execute a command buffer that identifies at least one resource. In some implementations, the firmware can trigger interrupt handling to begin dequeuing and / or executing the command buffer, and then dispatch the command buffer to an available hardware unit (e.g., the firmware loads a VPU executable, programs the DMA hardware, and / or executes commands).

[0026] The systems and methods described herein can be applied to accelerator control in embedded and edge computing environments, including applications in computer vision, real-time analytics, and low-power AI workloads. For example, the architecture can execute scheduling pipelines for heterogeneous accelerators used in object detection, safety features in autonomous vehicles, smart city infrastructure, robotic workloads, industrial automation, and / or parallel processing workloads deployed in embedded AI platforms. The system offers technical improvements in offloading control of task assignment to firmware, constructing commit messages from user-mode contexts, and / or executing hardware components in a controlled manner without relying on monolithic runtime drivers. These implementations address technical limitations of traditional firmware interfaces by facilitating low-latency scheduling, reducing host-side overhead, and / or enforcing execution order using hardware-controlled queues.

[0027] This disclosure also relates to systems and methods for binding local memory regions in a hardware accelerator to external memory locations via a firmware interface and exposing device pointers for command buffer construction in a non-privileged context. For example, the systems and methods according to this disclosure facilitate memory binding via firmware-based allocation commands that track associations in a resource table and expose device pointers to user-mode applications for constructing command buffers. These bindings can be reused across user contexts using device pointers, thereby reducing latency through policy-based data processing (e.g., fill on miss, flush on eviction). That is, memory operations can be policy-driven and tracked via resource identifiers, thereby supporting deferred data movement and persistent memory mapping.

[0028] Some techniques for managing accelerator memory rely on statically mapped local memory or repetitive allocation and initialization for each task, reducing flexibility and increasing redundant data movement. These methods typically require host-side memory mapping logic and repetitive refresh and initialization operations, leading to inefficient accelerator context switching and increased DMA setup latency. Furthermore, manually tracking memory associations at the application level introduces inconsistencies across execution. Technical limitations involve how these systems bind local memory regions to external memory, expose memory mappings to non-privileged applications, and / or control memory residency policies. For example, even for persistent allocations, rigid memory APIs often require explicit refresh and reinitialization. The lack of firmware coordination in memory tracking results in overhead for binding verification. The improved implementation described in this paper provides device pointer-based binding access and policy-driven residency control through a firmware-managed resource table.

[0029] The systems and methods of this disclosure provide improvements in accelerator memory allocation by providing firmware-coordinated local-to-external memory binding, memory persistence tracking via resource identifiers, and / or buffer references based on device pointers. For example, a local memory region (e.g., L2SRAM) can be bound to an external memory location (e.g., DRAM) using firmware calls, where the returned device pointer is stored in a memory buffer accessible to the non-privileged execution context. The system can provide policy-based initialization and local memory refresh during user context switching and / or upon explicit request. Furthermore, memory binding reference counts and resident metadata can be tracked via resource identifiers to facilitate memory sharing across user tasks and improve consistency across execution cycles.

[0030] These systems and methods can allocate local memory regions (e.g., segments of the hardware accelerator's L2SRAM) in the local memory of a hardware accelerator (e.g., a programmable vector accelerator (PVA) and / or any other accelerator including a VPU, DMA controller, and / or task logic) via a firmware interface, which are bound to external memory regions (e.g., dynamic random access memory (DRAM) and / or any system-accessible memory not located within the accelerator). In some implementations, contiguous regions of L2SRAM can be reserved and / or specified for use by applications, command buffers, and / or operators. The allocation can be logical and tracked via firmware using metadata that includes at least the size of the external memory and / or associations with one or more external memories. Furthermore, the allocation can return a device pointer corresponding to a binding tracked using resource identifiers from a resource table (e.g., which maintains a mapping between resource identifiers and associated metadata for binding) corresponding to a non-privileged execution context (e.g., a specific userspace application and / or task executed under a client-specific queue). These systems and methods can provide a device pointer to at least one subsequently built, non-privileged execution context-accessible memory region, pointing to at least one command buffer used to identify the binding between local and external memory regions. For example, a firmware-generated device pointer can be exposed to user-mode software.

[0031] The systems and methods described herein can be implemented in the memory configuration pipeline of embedded accelerators in latency-sensitive environments such as edge devices performing visual inference, autonomous vehicles, sensor fusion, smart city infrastructure, and / or robotic controls. For example, the system can configure memory binding between local sticky memory (e.g., L2SRAM) and DRAM using a firmware-mediated command interface. These implementations improve the initialization and persistence of local memory across task contexts, reduce redundant memory operations, and / or facilitate runtime execution to use previously configured memory mappings. The system eliminates and / or reduces the need for manual management of the DMA configuration lifecycle and introduces firmware-controlled mapping implementations that maintain isolation across user contexts.

[0032] This disclosure also relates to systems and methods for orchestrating the execution of low-level command buffers on heterogeneous hardware accelerators using firmware-controlled scheduling based on command order and / or resource availability. For example, the systems and methods according to this disclosure facilitate the construction of command buffers in user-mode memory, where the command buffers may contain individual commands for specific hardware components, such as DMA engines, VPUs, and / or PPEs. These commands may reference configuration data via resource identifiers stored in a resource table and verified by a kernel-mode driver (KMD) during initialization. That is, the firmware can dispatch and / or verify each command using pre-computed metadata in the resource table, thereby allowing commands to be completed out of order based on hardware readiness.

[0033] Some techniques used for accelerator orchestration rely on coarse-grained command execution models where the host submits a single task, which is internally staged and starts hardware components in sequence. These approaches prevent low-latency pipelined implementations and hide the parallel execution potential of user-mode software. Furthermore, firmware verification and command dispatch typically rely on opaque transition layers and / or fixed scheduling policies. These limitations relate to how these systems control the hardware execution order, facilitate latency hiding, and / or expose hardware-level orchestration flexibility. For example, sequential execution models prevent the preloading of future DMA configurations during VPU task execution. Static firmware execution logic limits reordering and / or fine-grained hardware usage. The improved implementations described herein provide user-controlled command buffers, firmware-level dispatch logic, and / or execution ordering based on hardware availability (e.g., rather than strict command order).

[0034] The systems and methods of this disclosure provide improvements in hardware accelerator orchestration by facilitating user-mode control over command buffer construction, firmware-based command verification and dispatch, and / or overlapping command execution across heterogeneous hardware units. For example, individual commands can be used to construct command buffers to configure DMA controllers, launch VPU kernels, and / or query timestamps before and after execution. Firmware components can execute commands sequentially in the order they are placed, but execution can be overlapped based on hardware availability, allowing for latency hiding and efficient hardware utilization. Furthermore, references to pre-verified configured resource identifiers in resource tables allow firmware to verify access without invoking driver-level intervention, thereby reducing commit latency and / or facilitating the execution of large computation graphs with minimal and / or reduced host overhead.

[0035] These systems and methods can construct a command buffer comprising multiple commands within the memory of a hardware accelerator. In some implementations, the sequence of commands can be constructed within a firmware-accessible memory-mapped region. Commands can represent low-level instructions corresponding to at least one firmware-consumable primitive operation. Furthermore, at least one of the multiple commands can identify at least one resource via a resource identifier (e.g., an abstract, opaque reference indexed in a resource table and maintained by a kernel-mode driver (KMD)). In some implementations, the resource table may not be accessible to the non-privileged execution context (e.g., protected by the KMD and not directly accessible to application-layer software and / or users). These systems and methods can provide commit information for the command buffer and a corresponding commit indicator to at least one commit interface (e.g., a commit queue and / or signaling mechanism, such as registers for facilitating host-to-firmware handover) corresponding to the non-privileged execution context. These systems and methods enable firmware components to execute command buffers to (i) initiate (e.g., fetch and / or execute) the execution of multiple commands in the order they are placed in the command buffers, and (ii) issue at least one subsequent command for execution (e.g., start sequentially, complete out of order) based on the availability of at least one resource, before completing at least one previously issued command. For example, command B can be issued even if command A is still in progress, but if command B targets a non-conflicting hardware resource. In some implementations, new command buffers can be made available in a submission queue. Firmware can be triggered and / or invoked to process the command buffers.

[0036] The systems and methods described herein can be applied to hardware component orchestration pipelines across heterogeneous accelerators, including use cases involving pipelined execution of computer vision tasks, robotic workloads, autonomous vehicles, medical imaging, smart city infrastructure, edge computing, and / or machine learning inference at the edge. For example, the system can control command buffer execution involving DMA, VPU, and PPE, thereby reducing latency between tasks by initiating the next task configuration during active task execution. This architecture enhances the ability of user-mode software to assemble task pipelines using granular commands and / or resource identifiers verified by firmware-managed resource tables. These implementations eliminate and / or reduce reliance on coarse-grained task control models and provide improvements in out-of-order command completion, low-level memory configuration, and / or latency hiding across parallel accelerators.

[0037] In some implementations, the systems and methods described herein can be executed in a simulated environment (e.g., simulated environment data and simulated sensor data of simulated sensors of virtual or simulated vehicles, robots, or machines in the simulated environment) using simulated data. For example, simulated input data (e.g., map data, perception data, self-motion data, haptic data, and / or any other data described herein) can be used to perform (i) user-space buffer construction and firmware-controlled execution of the hardware accelerator, (ii) firmware-controlled memory binding and implicit data persistence of the hardware accelerator, and / or (iii) firmware-controlled command dispatch and resource-based orchestration of the hardware accelerator, etc., and this information can be used to perform operations associated with virtual machines in the simulated environment. These simulated operations can be used to test the performance of the underlying algorithms, systems, and / or processes before deploying them to the real world. In some cases, simulations can be used to generate synthetic training data. For example, user-space buffer construction and firmware-controlled execution of hardware accelerators, firmware-controlled memory binding and implicit data persistence of hardware accelerators, and / or firmware-controlled command dispatch and resource-based orchestration of hardware accelerators can generate annotated accelerator trajectories, memory binding patterns, and command scheduling sequences from simulations. Then, prior to real-world deployment, synthetic training data (in addition to or replacing real-world data) can be used or processed to train performance models, validate command buffer construction pipelines, and test resource management strategies.

[0038] In any example (such as where the simulation environment is used for testing, validation, training, etc.), the simulation environment and / or associated training data can be rendered or otherwise generated using one or more optical transport simulation algorithms (such as one or more ray tracing and / or path tracing algorithms). When using optical transport simulation, the simulation system may employ one or more dedicated ray tracing hardware accelerators and / or processors (e.g., NVIDIA's RTX or another real-time ray tracing GPU, such as a GPU including one or more ray tracing (RT) cores) optimized for performing real-time or near-real-time optical transport simulation operations in conjunction with one or more other processors of the system (e.g., GPUs, CPUs, accelerators, etc.). In some embodiments, the simulation environment and / or one or more of its objects, features, or components may be generated or managed in a three-dimensional (3D) content collaboration platform (e.g., NVIDIA's OMNIVERSE), which may be optimized or suitable for industrial digitization, generative physics, artificial intelligence, and / or other use cases, applications, and / or services. For example, a content collaboration platform or system may include a system for using or developing generic scene descriptors (USD) (e.g., OpenUSD) data to manage objects, features, scenes, etc., in simulated environments, digital environments, etc. The platform may include realistic physical simulations (e.g., using NVIDIA's PhysX software development kit (SDK)) to simulate real physical phenomena and physical interactions with simulations hosted by the platform. The platform may integrate OpenUSD with ray tracing / path tracing / light transport simulations (e.g., NVIDIA's RTX rendering technology) into software tools and simulation workflows for building, training, deploying, and / or testing AI systems, such as systems for testing, validating, training (e.g., machine learning models, neural networks, etc.) and / or other tasks related to automobiles, robots, other machine types, and / or other systems and applications. In some examples, the simulated environment may include a digital twin of a real-world environment, such as a specific road segment, warehouse, data center, airport, geographic area, ocean area, and / or any other real-world environment in which autonomous or semi-autonomous vehicles or machines can operate.

[0039] In some implementations, a remote control or teleoperation system can be used to perform teleoperation or remote control of vehicles, robots, and / or other machines. For example, the systems and methods described herein can be used to perform (i) user-space buffer construction and firmware control execution for hardware accelerators, (ii) memory binding and implicit data persistence for firmware control of hardware accelerators, and / or (iii) command dispatch and resource-based orchestration for firmware control of hardware accelerators. These can be included in the visualization or mapping of the environment to assist a remote operator in control, or to provide waypoints or other indications for controlling or navigating autonomous or semi-autonomous machines through the environment. Therefore, a remote operator can use visual, auditory, textual, and / or other cues or indicators generated using the systems and methods described herein to assist in navigating vehicles, robots, machines, etc., in a real-world environment using a teleoperation system.

[0040] In some implementations, the systems and methods described herein can be deployed in robotics applications. For example, a robot or robotic system may include one or more onboard processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), hardware-based programmable vision accelerator (PVA) (which may include one or more vector processing units (VPU)), direct memory access (DMA) system and / or pixel processing engine (PPE), hardware-based optical flow accelerator (OFA), SoC, etc.), and memory and / or storage devices (e.g., for storing control algorithms, sensor data, and one or more machine learning models). Robotic systems can use these processors to execute one or more machine learning models (e.g., language models, visual language models (VLM), large language models (LLM), visual language action (VLA) models, multimodal language models (MMLM), etc.) that enable the robotic system to perform complex tasks autonomously or semi-autonomously, such as interacting with and / or manipulating static and / or dynamic objects, or navigating in the environment using sensors (such as cameras, LiDAR, RADAR, ultrasonic sensors, etc.). The system can use sensor fusion techniques to combine data from multiple sensors (e.g., cameras, infrared, LiDAR, RADAR, accelerometers) to create a comprehensive model of the robot's surroundings. This data can be processed locally on the robot or sent to a remote server for more computationally intensive tasks, such as 3D mapping or SLAM (Simultaneous Localization and Mapping). In one or more embodiments, data from individual robots (e.g., sensor data, mission status, or environmental conditions) can be uploaded to the cloud, where a centralized AI model can analyze optimized commands and distribute them throughout the fleet. In some embodiments, one or more machine learning models described herein (e.g., language models, VLM, VLA, LLM, MMLM, diffusion models, NeRF models, DNN, etc.) can be used to enable robots to perceive and infer their environment and / or communicate with one or more other robots and / or humans in the environment. In some embodiments, robots can (e.g., using one or more network interface cards (NICs) and / or data processing units (DPUs)) communicate with one or more locally hosted servers / computing devices and / or with one or more remotely located servers / computing devices (e.g., in one or more data centers).

[0041] In some implementations, the systems and methods described herein can be deployed in in-vehicle infotainment (IVI) systems or cockpit experience (IX) applications. For example, an infotainment system within a vehicle (e.g., a car, truck, drone, engineering equipment, robot, semi-autonomous vehicle, or autonomous vehicle) may include one or more onboard processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural processing unit (NPU), neural network accelerator (NNA), hardware-based programmable vision accelerator (PVA) (which may include one or more vector processing units (VPU), direct memory access (DMA) systems, and / or pixel processing engines (PPE)), hardware-based optical flow accelerator (OFA), SoC, etc.) and memory and / or storage devices (e.g., for storing control algorithms, sensor data, and one or more machine learning models), as well as memory and / or storage devices (e.g., for storing entertainment content, navigation data, and user preferences). The system can use these processors to execute one or more machine learning models (e.g., language models) to achieve features such as voice control, personalized media recommendations, dynamic navigation, and real-time communication with other services via network connectivity. In-vehicle infotainment systems can also use natural language processing (NLP) models to enable voice-based interaction. One or more machine learning models can be stored locally or accessed via one or more APIs connected to cloud services, enabling the system to process requests in real-time or near real-time.

[0042] In some examples, one or more machine learning models described herein (e.g., deep neural networks, language models, LLMs, VLMs, multimodal language models, visual-language-action (VLA) models, perceptual models, tracking models, fusion models, transformer models, diffusion models, encoder-only models, decoder-only models, encoder-decoder models, neural radiation field (NERF) models, etc.) can be packaged as microservices, such as inference microservices (e.g., NVIDIA NIM), which may include containers (e.g., operating system (OS) level virtualization packages) that may include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model "engine". For example, an inference microservice may include the container itself and one or more models (e.g., weights and biases). In some instances, such as when one or more machine learning models are small enough (e.g., have a sufficiently small number of parameters), one or more models may be included within the container itself. In other examples, such as when one or more models are large, one or more models may be hosted / stored in the cloud (e.g., in a data center) and / or hosted on-premises and / or at the edge (e.g., on a local server or computing device, but outside a container). In these implementations, one or more models may be accessed via one or more APIs (such as REST APIs). Accordingly, and in some implementations, one or more machine learning models described herein may be deployed as inference microservices to accelerate the deployment of one or more models on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, an optimized inference engine (e.g., execution software built using standardized AI model deployments (such as NVIDIA's Triton Inference Server), and / or one or more APIs for high-performance deep learning inference, which may include inference runtime and model optimizations for low latency and high throughput for production applications (such as NVIDIA's TensorRT) and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). One or more machine learning models described in this paper may be included as part of a microservice along with an acceleration infrastructure that can be deployed with a single command and / or orchestrated and auto-scaled on the acceleration infrastructure using a container orchestration system (e.g., up to data center scale on a single device).Accordingly, an inference microservice may include one or more machine learning models (e.g., one or more machine learning models optimized for high-performance inference), inference runtime software that executes one or more machine learning models and provides output / response to inputs (e.g., user queries, prompts, etc.), and enterprise management software that provides health checks, identity verification, and / or other monitoring. In some implementations, the inference microservice may include software that performs in-situ replacement and / or updates to one or more machine learning models. When replacement or updates are performed, the software performing the replacement / update may maintain the user configurations of the inference runtime software and the enterprise management software.

[0043] While examples of using machine learning models (such as neural networks) may be described in this article, this is not intended to be limiting. For example, but not limited to, any of the various machine learning models and / or neural networks described herein can include any type of machine learning model, such as those using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoder neural networks, artificial neural networks (ANN), convolutional neural networks (CNN), recurrent neural networks (RNN), perceptrons, long / short-term memory (LSTM) networks, multilayer perceptron (MLP) networks, deep stacked networks (DSN), generative pre-trained (GPT) models or networks, feedforward networks, radial basis function ANNs, self-organizing maps (SOM), Kohonen maps, Hopfield networks, Boltzmann machines, deep belief neural networks, deconvolutional neural networks, generative adversarial networks (GAN), liquid machines, modular neural networks, liquid machines, sequence-to-sequence models, networks using transformer architectures, state-space models (SSM) (e.g., using Mamba architectures (e.g., Mamba-1, Mamba...)). The following are examples of machine learning models: networks using selective state-space models, networks using structured state-space sequence models, diffusion models (e.g., diffusion probability models, fraction-based generative models), neural radiation field (NeRF) models, Gaussian sputtering models, Kolmogorov-Arnold networks (KAN), models with encoder-only architectures, models with decoder-only architectures, models with encoder-decoder architectures, generative machine learning models, language models, large language models (LLM), visual language models (VLM), multimodal language models (MMLM), large action models (LAM), visual-language-action (VLA) models, and one or more other types of machine learning models.

[0044] The systems and methods described herein may be used by, but are not limited to, the following: non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, spacecraft, ships, shuttles (e.g., autonomous taxis (robotaxis)), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, as examples and not limitations, for machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets (e.g., NVIDIA's Omniverse), cloud computing and / or any other suitable application.

[0045] The disclosed implementations can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines, etc.), systems implemented using robots, aviation systems, medical systems, rowing systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing language models (such as large language models (LLM), visual language models (VLM), visual-language-action (VLA) models, and / or multimodal language models), systems using or deploying one or more inference microservices, systems containing one or more machine learning models and OS-level virtualization packages (e.g., containers) deployed in services or microservices, systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0046] refer to Figure 1 , Figure 1This is a block diagram illustrating an example of a system 100 according to some implementations of this disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to the arrangements, components, features, and elements shown, and certain elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and at any suitable location (e.g., on a local device, vehicle, or edge machine; locally deployed (e.g., a locally hosted server); remotely deployed (e.g., in one or more computing or server devices in one or more data centers in the cloud); and / or other locations). The various functions performed by these entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, microcontroller, embedded processor, digital signal processor (DSP), image signal processor (ISP), physical processing unit (PPU), field-programmable gate array (FPGA), accelerators (e.g., deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural network accelerator (NNA) and / or neural processing unit (NPU), programmable vision accelerator (PVA), optical flow accelerator (OFA), etc.), application-specific integrated circuit (ASIC), data processing unit (DPU), quantum processor, etc.) that execute instructions stored in memory. In some implementations, the systems, methods, and processes described herein can use... Figures 18A-18E Example machine 1800 Figure 19 Example calculation of ecosystem 1900, Figure 20 Example generative language model systems 2000 and / or Figure 21 The example computing device 2100 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the same task.

[0047] System 100 can implement at least a portion of an accelerator pipeline, such as, but not limited to, a command scheduling pipeline, a memory binding pipeline, a task orchestration pipeline, a hardware configuration pipeline, a resource verification pipeline, and / or a task execution pipeline. System 100 can be used to construct command buffers, bind local memory to external memory, and / or dispatch tasks via firmware-managed scheduling in any of the various systems described herein, including but not limited to computer vision systems, robotic systems, autonomous vehicle systems, embedded edge systems, smart city systems, medical imaging systems, low-power AI systems, and / or industrial automation systems.

[0048] Typically, an accelerator pipeline may include operations performed by system 100. For example, an accelerator pipeline may include any one or more of a Command Submission and Execution (CSE) phase, a Memory Binding and Persistence (MBP) phase, and / or a Firmware Scheduling and Dispatch (FWSD) phase. Each phase of the accelerator pipeline includes one or more components of system 100 that perform the functions described herein. In some implementations, one or more phases may be performed during the training of the AI ​​model. Furthermore, one or more phases may be performed using the AI ​​model during the inference phase.

[0049] System 100 (e.g., its implementation of an accelerator pipeline) can construct command buffers in memory, which include at least one command identifying at least one resource. That is, System 100 can allocate and populate workloads (e.g., DSP kernel fetches, DMA transfer descriptors, memory fence operations, and / or any sequence of primitive operations executable by firmware). Command buffers can serve as and / or represent the interface between user-space software layers (e.g., a C-language user-space programmable vision accelerator (CUPVA), a non-privileged software layer, the host-side runtime of a programmable accelerator, a firmware-backed accelerator programming interface, and / or any user-space DSP kernel execution framework) and, for example, the firmware (FW) of a programmable vision accelerator (PVA) and / or any hardware accelerator. User-space software layers can construct command buffers and submit them to the firmware without modification and / or updating by user-mode drivers (e.g., a lightweight CUPVA runtime library and / or any user-space runtime translation layer) and / or kernel-mode drivers (e.g., a PVA scheduler driver and / or any low-privileged commit interface). That is, by constructing a command buffer and submitting it to the firmware without modification, System 100 can reduce driver complexity, improve execution determinism, and minimize host-side processing overhead.

[0050] Furthermore, system 100 can construct commands in chunks and can be acquired by firmware (e.g., firmware component 106) via direct memory access (DMA) using prefetching and linking logic to issue low-level operations (e.g., VPU boot, DMA configuration, memory copying, barriers, fences, and status updates) for synchronous and / or asynchronous execution by the firmware. Resources can be references to memory allocations, processing of DSP executable kernels, buffer identifiers for input / output data, and / or any identifier corresponding to firmware-accessible objects used during accelerator task execution. In some implementations, resources can be verified and stored in a resource table maintained in DRAM, where the resource table can include entries such as DMA configurations, executables, and / or DRAM buffers identified by resource IDs. The resource table can be maintained per context (e.g., per accelerator client, execution flow, user context, non-privileged execution context, or process isolation domain). Furthermore, the resource table can be accessed by firmware and kernel-mode drivers, and resources can be indirectly referenced in the command buffer via their corresponding resource IDs.

[0051] In some implementations, implementing the accelerator pipeline may include system 100 providing commit information for the command buffer to a commit queue corresponding to the non-privileged execution context. For example, the commit information may be a queue identifier, a new tail index, a command buffer handle, a resource identifier referencing the command buffer, a front fence identifier, a back fence identifier, a synchronization token, an execution affinity mask, a client context identifier, an encoded pointer to command buffer metadata, and / or any commit parameters used to identify, locate, and / or trigger execution of the command buffer. That is, system 100 may write the commit information to a queue (e.g., a Controlled Command Queue (CCQ) entry that the non-privileged execution context can push (e.g., representing a work or instruction unit that the firmware can acquire, decode, and / or execute)) that includes a queue ID and a new tail index. In some implementations, the commit queue may be stored in DRAM 130 as a circular buffer and / or another buffer accessible to both the UMD (e.g., as part of a software stack running in user space on a host processor (one or more CPU hosts 128)) and the firmware. That is, once notified via a signaling register (e.g., one or more registers 114), the commit queue can be written to by the UMD and later read by the firmware (e.g., firmware interface 108). For example, the firmware can interpret the new tail index as a trigger for initiating a dequeue command, wherein at least one (e.g., each) entry can remain opaque to the firmware until a commit indicator is received. Furthermore, at least one (e.g., each) entry may include fields (e.g., buffer chunk offset, front fence, back fence, and / or engine affinity mask) used by the firmware to determine scheduling behavior, select hardware components for execution, control synchronization across tasks, and / or manage execution parameters (e.g., chunk size, command type, and / or memory usage conditions).

[0052] In some implementations, the commit queue in DRAM 130 (e.g., in one or more memory regions 132) can be written to by one or more user-mode components operating in a non-privileged execution context (e.g., UMD threads, application-layer command construction routines, and / or any user-space software issuing accelerator tasks, instructions, and / or actions). That is, UMDs and / or applications executing in a non-privileged execution context can write entries to a circular buffer and / or update the tail index, where entries can remain uninterpreted by the firmware until a corresponding commit indicator is written to and / or provided to a signaling register (e.g., one or more registers 114). For example, once the signaling interface 112 receives the commit indicator, the firmware can begin reading entries from the updated tail index, thereby facilitating firmware-controlled decoding and / or execution. Typically, a “non-privileged execution context” can be any software process, application, or thread that can execute without direct access to kernel-level system resources, memory-mapped device control regions, or firmware-maintained data structures (such as user-mode drivers, application-layer programs, or isolated computation threads running in restricted address spaces).

[0053] Furthermore, implementing the accelerator pipeline may include system 100 providing a commit indicator corresponding to the commit message to a signaling interface 112 (e.g., exposed to user space) corresponding to the commit queue. For example, the commit indicator may be an updated tail index that can be stored in a register (e.g., one or more registers 114, such as a CCQ register and / or any memory-mapped signaling register). Typically, the commit indicator may be a notification used to trigger firmware processing for a new commit queue entry. That is, the commit indicator may reference a tail index value that signals the availability of at least one command buffer entry for firmware consumption. In some implementations, system 100 may write the commit indicator to a memory-mapped CCQ register, indicating a new tail index for the commit queue. Furthermore, a tail index can be incremented, and when the tail is not incremented (e.g., from a previous tail index, such as a duplicate tail index or an invalid value), the firmware (e.g., firmware component 106) can mask register 114 (e.g., disable interrupt generation from the corresponding queue), thereby preventing further interrupts from the non-privileged execution context until reset, rearmed by the firmware, and / or reinitialized by the privileged software.

[0054] In some implementations, implementing the accelerator pipeline may include system 100 such that at least one hardware component of the hardware accelerator executes a command buffer that identifies at least one resource. The hardware component may be a vector processing unit (VPU) 118, a direct memory access (DMA) controller 116 (also referred to herein as a “DMA engine”), and / or a programmable or pixel processing engine (PPE) 220.

[0055] In some implementations, the DMA controller 116 of system 100 can manage direct memory access operations between external DRAM and local memory resources such as L2SRAM. It should be understood that the DMA controller 116 can be interchangeably referred to as a DMA engine. That is, a reference to a DMA engine corresponds to the same DMA controller circuitry used to issue and process memory transfers. In various implementations, firmware component 106 can construct a DMA configuration by referencing slots and descriptors executed by the DMA controller 116 to transfer data without host CPU intervention. For example, the firmware can configure the DMA controller 116 to transfer an input image from DRAM to local L2SRAM for processing by a vector processing unit (VPU) 118. In another example, the DMA controller 116 can transfer the processed output from L2SRAM (e.g., local SRAM 124) back to DRAM 130. In some implementations, the firmware can load configurations or descriptors into the TCM using channel 0 of the DMA controller 116, while additional user channels can be explicitly allocated via command parameters (e.g., struct pva_user_DMA_allowance). That is, direct memory transfers referenced in the command buffer can be performed by the DMA controller 116 (also known as the "DMA engine") using the assigned configuration and descriptor states of the corresponding DMA set.

[0056] At least one (e.g., each) component can be acquired and released via commands in the command buffer (e.g., pva_cmd_acquire_engine, pva_cmd_release_engine, etc.). That is, a firmware interrupt can be triggered to cause the command buffer to be dequeued and executed. System 100 can dispatch the command buffer to available hardware units and / or the system. For example, firmware component 106 can load an executable associated with VPU 118 and configure runtime parameters using values ​​referenced from resource table 110. In another example, firmware component 106 can program the register settings of DMA controller 116 based on a previously verified DMA configuration identified by a resource identifier in command buffer 134. In yet another example, firmware component 106 can schedule instructions for PPE 220 to execute fixed-function processing routines identified in command buffer 134. In yet another example, firmware component 106 can issue a synchronization command to enforce a sequenced or completion dependency between the operation of VPU 118 and DMA controller 116. In these examples, execution can be facilitated by firmware component 106, which interprets and issues commands according to their placement order and / or out-of-order placement within the command buffer (e.g., using firmware interface 108) and completes execution based on the availability of the targeted hardware component. That is, command dispatch and completion can occur in a partially overlapping manner, thereby facilitating parallel hardware utilization without completing previously dispatched commands. For example, execution can be implemented under the control of resource metadata stored in resource table 110 and triggered based on resource availability and / or execution order by firmware component 106.

[0057] Typically, the systems and methods according to this disclosure can implement firmware-controlled execution of command buffers that reference proven resources in a shared resource table. That is, system 100 can dispatch commands according to the order in which the buffers are placed and / or allow out-of-order completion based on resource availability, thereby allowing firmware component 106 to manage parallel execution across hardware components such as VPU 118, DMA controller 116, and / or PPE 220. The command buffers can be submitted via a submission interface that decouples queue writing from firmware notifications, thereby improving command throughput and reducing host-side latency. These implementations facilitate firmware dispatching of computational workloads while reducing reliance on monolithic task sequences and improving utilization of independent schedulable hardware systems and / or engines.

[0058] System 100 (e.g., its implementation of an accelerator pipeline) may allocate local memory regions in the local memory (e.g., local SRAM 124, such as L1 / L2 SRAM, and / or any other on-chip memory) of a hardware accelerator (e.g., PVA subsystem 102) bound to an external memory region (e.g., one or more memory regions 132). This allocation may return a device pointer corresponding to the binding. In some implementations, bindings may be tracked using resource identifiers in resource table 110 corresponding to non-privileged execution contexts (e.g., for a specific userspace application or task executed under a specific client queue (CCQ)). Firmware component 106 may reserve and / or provide contiguous regions of L2 SRAM (e.g., local SRAM 124) for use by applications, command buffers, and / or operators. The allocation can be logical and / or tracked via firmware component 106 using metadata stored in resource table 110, including size, association with external memory (e.g., DRAM 130), binding status, reference count, residency policy, and / or any firmware-managed memory attributes.

[0059] In some implementations, firmware interface 108 can be a commit path where a kernel-mode driver (KMD) issues initialization commands, registers resource configurations, and / or allocates accelerator memory regions for use by a non-privileged execution context. Typically, the KMD operates on the host processor of CPU host 128 and interacts with and / or otherwise interfaces with control subsystem 104 of PVA subsystem 102 to perform privileged operations such as resource registration and firmware-level memory binding. That is, the KMD can interface with firmware component 106 via firmware interface 108 and can update entries in resource table 110 during system initialization. The KMD can also cooperate with UMD by providing memory mapping, allocating resource identifiers, and / or managing access policies enforced by firmware component 106.

[0060] In some implementations, system 100 may request to reserve a segment of local SRAM 124 (e.g., L2SRAM) for use. PVA subsystem 102 may include one or more VPUs 118, one or more DMA controllers 116 and / or one or more PPEs 120, and / or other task and / or execution logic. System 100 may associate (e.g., bind) allocated L2SRAM regions (e.g., local SRAM 124) with DRAM buffers (e.g., one or more command buffers 134). Firmware component 106 may manage the mapping between L2SRAM and DRAM-based memory locations to form persistent data bindings. That is, firmware component 106 may manage the mapping by maintaining metadata in resource table 110 to associate L2SRAM regions with their corresponding DRAM regions, track resident status, and / or enforce memory access policies. For example, firmware component 106 may track reference counts, access frequency, and / or data freshness between local memory regions and external memory regions. In this example, the L2SRAM memory region can be allocated without explicit location specification (i.e., without a fixed physical address), and the firmware interface 108 can select and / or assign the memory region based on an optimal fit strategy (e.g., selecting a region based on available space and alignment requirements). In another example, the firmware interface 108 can select and / or assign the memory region based on memory type, priority level, and / or any firmware-defined resource allocation constraints.

[0061] In some implementations, after creating the binding, the firmware can generate a resource identifier (e.g., a unique identifier) ​​that references the local to the external memory binding. That is, the binding can be referenced by user-space software and / or any other firmware layer using opaque handles without needing to know low-level implementation details (e.g., without exposing internal memory addresses such as physical DRAM addresses, memory mapping offsets, residency flags, and / or any firmware-controlled access states). Resource table 110 can be used to maintain a mapping between resource identifiers and associated metadata used for the binding (e.g., local memory region addresses, external memory region backup addresses, access policies, binding states, reference counts, and / or any memory allocations enforced by firmware component 106). Furthermore, the external memory region can be DRAM 130 and / or any system-accessible memory not co-located with the hardware accelerator (e.g., PVA subsystem 102). Typically, a device pointer can be a logical identifier given a non-privileged execution context to represent the mapped DRAM location. That is, it does not represent a direct memory address, but can represent an abstract handle and / or opaque value used by firmware component 106 to construct a DMA descriptor (e.g., a memory transfer instruction that includes source and destination addresses, transfer size, and binding attributes).

[0062] In some implementations, implementing the accelerator pipeline may include system 100 providing a device pointer to a memory region accessible to the non-privileged execution context (e.g., a DRAM-based buffer mapped to user space) for identifying at least one subsequent construction of at least one command buffer (e.g., command buffer 134) that binds the local memory region to the external memory region. That is, system 100 may write the device pointer to a DRAM location mapped to a user-mode address space associated with the execution context. The memory region accessible to the non-privileged execution context may be a user-mode memory region allocated in system DRAM (e.g., DRAM 130) that is accessible by user-mode software (e.g., a user-mode driver (UMD) or application). For example, system 100 may expose a firmware-generated device pointer to user-mode software (e.g., via CUPVA) by writing the device pointer to a known offset in a DRAM buffer pre-allocated for inter-context communication.

[0063] In some implementations, a command buffer 134 can be constructed in user space to reference a device pointer provided in an accessible memory region, where the device pointer identifies the binding between the local memory region and the external memory region (L2SRAM to DRAM). The user-mode context can use the device pointer as a reference (e.g., without additional binding metadata) to initiate the construction of the command buffer 134. Furthermore, at least one (e.g., each) UMD context can be associated with a corresponding KMD context and a dedicated command completion queue (CCQ) (e.g., FIFO, LIFO), thereby representing an isolated execution context for a specific user application. That is, the UMD-KMD context pairing can define the logical interface from the application to the PVA subsystem 102, thereby maintaining isolation by associating at least one (e.g., each) command stream with its own commit path and / or firmware-accessible resource range. In some implementations, the device pointer can represent a logical reference to the binding between a local memory region (e.g., L2SRAM) and an external memory region (e.g., DRAM), thereby encapsulating metadata identifying the local-to-external memory association without exposing the underlying physical address.

[0064] Typically, the systems and methods according to this disclosure can allocate local memory regions in accelerator memory (e.g., L2SRAM) and bind them to external memory regions (e.g., DRAM) using firmware-managed mappings referenced via device pointers. That is, system 100 can track bindings using resource identifiers in a firmware-maintained resource table and expose device pointers to user-mode software for command buffer construction. These bindings can persist across execution windows and facilitate memory residency strategies such as fill on miss and flush on eviction. These implementations reduce redundant memory allocation and copying operations, improve data persistence, and / or abstract binding metadata into opaque handles usable by user-space applications without exposing privileged memory access logic.

[0065] System 100 (e.g., implementing an accelerator pipeline) can construct a command buffer 134 comprising multiple commands in the memory of the hardware accelerator. At least one of the multiple commands can identify at least one resource via a resource identifier in resource table 110 (e.g., an abstract, opaque reference indexed in resource table 110 and maintained by KMD) that is inaccessible in a non-privileged execution context. That is, system 100 can construct a sequence of commands in a firmware-accessible memory-mapped region. For example, the firmware-accessible memory-mapped region can be one or more memory regions 132 of DRAM 130. Command buffer 134 can encode and / or otherwise represent hardware control operations (e.g., setting CPU parameters, loading DMA configuration, triggering barrier synchronization). System 100 can establish command buffer 134 as a direct interface between CUPVA and FW. That is, user-mode software (e.g., CUPVA) and / or user-mode applications can be used to construct command buffer 134 and submit them to firmware (e.g., firmware component 106) without modification by UMD or KMD. Resource table 110 can store metadata in a firmware-managed memory area. For example, metadata can be a memory address bound, a DMA descriptor, a VPU code pointer, an engine affinity mask, and / or any firmware-defined configuration property.

[0066] In some implementations, a command may be a primitive and / or instruction code representing a low-level instruction corresponding to a firmware-consumable operation (e.g., a primitive operation). For example, the command could be used to initialize a VPU program, prefetch DMA configuration, trigger a DMA transfer, configure engine registers, invoke a barrier group, record a timestamp, and / or any firmware-detectable execution trigger. Furthermore, at least one (e.g., each) command may independently address a hardware block (e.g., configure the DMA controller and / or engine, trigger VPU code execution, program a PPE stage, issue a memory copy operation, initiate a query buffer write, and / or any targetable compute or memory component). Resource table 110 may be protected by KMD (e.g., inaccessible) and cannot be directly accessed by the user application layer and / or the software application layer executing on one or more CPU hosts 128 (e.g., without virtual memory space mapped to the user application).

[0067] In some implementations, implementing the accelerator pipeline may include system 100 providing a commit message for a command buffer to at least one commit interface corresponding to a non-privileged execution context, along with a commit indicator corresponding to that commit message (e.g., notifying the firmware commit queue that a new command buffer is available). Typically, a "commit interface" can be a collection of communication components (e.g., logical constructs) that include a command queue and signaling registers for a memory mapping for host-to-firmware coordination. That is, a commit interface may include a commit queue (e.g., a circular buffer with head / tail indices) and signaling mechanisms (e.g., registers) that allow host-to-firmware switching. For example, system 100 may provide a commit message for a command buffer to a commit queue corresponding to a non-privileged execution context (e.g., a circular buffer allocated in DRAM 130 as part of one or more memory regions 132 of the commit interface). In another example, system 100 may provide a commit indicator corresponding to the commit message to a signaling interface corresponding to the commit queue (e.g., a control register of the memory mapping of the commit interface, such as a CCQ register within one or more registers 114 of the control subsystem 104).

[0068] Furthermore, implementing the accelerator pipeline may include system 100 causing firmware components to execute command buffers to (i) initiate the execution of multiple commands in the order they are placed in the command buffers, and (ii) issue at least one subsequent command for execution before completing at least one previously issued command, based on the availability of at least one resource. That is, firmware component 106 may be triggered and / or invoked (e.g., via an interrupt or register write received through signaling interface 112 of control subsystem 104) to process command buffer 134 (e.g., stored in a memory region of DRAM 130 and written by a user-mode component such as CUPVA, wherein a commit indicator is provided to the firmware via signaling interface 112).

[0069] In some implementations, initiation may include: retrieving a first command from command buffer 134 and decoding the command to identify a target hardware component and / or a corresponding resource identifier (e.g., in resource table 110). That is, system 100 may retrieve and / or execute at least one (e.g., each) of commands in a sequence based on a sequential traversal of command buffer entries using head index metadata and tail index metadata. This execution may be performed in the order of placement, such that the head points to the next command to be retrieved and the tail marks the boundaries of commands that have been submitted but not executed. In some implementations, issuing at least one subsequent command may include dispatching a command from command buffer 134 to a hardware component before completing a previously dispatched command. That is, system 100 may evaluate resource dependencies between commands and selectively issue non-conflicting commands. System 100 may support and / or otherwise facilitate out-of-order completion of commands regarding dispatched commands. For example, even if command A is still in progress, command B can be issued if it targets a non-conflicting hardware resource (e.g., one of n independent DMA channels, VPU pipelines, or local memory regions). In this example, firmware component 106 can use an internal state machine referencing resource usage and completion states to track active and completed commands. Therefore, system 100 can start sequentially but complete out of order. That is, the order in which commands are issued can follow buffer placement, but the actual hardware-level completion can depend on the availability and readiness of the targeted component.

[0070] Typically, the systems and methods according to this disclosure can implement user-mode construction of the accelerator command buffer using opaque resource identifiers stored in a firmware-managed resource table. The command buffer can encode granular hardware instructions (e.g., VPU program fetching, DMA transfers, and / or PPE synchronization) and / or can be submitted to the firmware via an abstract submission interface including queues and signaling registers. These implementations reduce driver complexity by facilitating direct user-mode submission of hardware instructions and decoupling firmware resource verification from host-side resource configuration. Execution can be initiated in buffered order and completed out of order based on hardware readiness, thereby allowing high-throughput task scheduling without and / or using a kernel-mode translation layer.

[0071] For more detailed information, please refer to [link / reference]. Figure 1System 100 may include a programmable vision accelerator (PVA) subsystem 102, one or more CPU hosts 128, and dynamic random access memory (DRAM) 130. The PVA subsystem 102 may perform image perception tasks, digital signal processing tasks, execution of firmware-controlled command buffers, memory region binding and data persistence operations, low-level orchestration of heterogeneous computing engines, and / or any other task involving interaction between user-space constructed workloads and firmware-managed hardware components. For example, any task requiring the movement of large amounts of image data from a source location to a target location. In another example, any task requiring the coordination of memory transfers and computational workloads across multiple accelerator units. In yet another example, any task involving the interpretation of user-defined command sequences using pre-verified hardware resource configurations.

[0072] PVA subsystem 102 may include control subsystem 104, one or more DMA controllers 116, one or more vector processing units (VPUs) 118, one or more programmable or pixel processing engines (PPEs) 120, one or more instruction caches (e.g., stored in local memory 126), one or more decoupled lookup tables (DLUTs) 122, local SRAM 124, local memory 126 (sometimes referred to herein as "vector memory"), and / or other components. In some implementations, PVA subsystem 102 includes level 1 (L1), level 2 (L2), level 3 (L3), and / or level x (Lx) buffers and / or SRAM (referred to as local SRAM 124). Local SRAM 124 may be dedicated memory for one or more DMA controllers 116, VPUs 118, and / or PPEs 120.

[0073] The control subsystem 104 can receive commands from the CPU host 128 related to various tasks and / or operations (e.g., command buffer commits, resource registration, memory binding, signaling coordination, authentication requests, and / or interactions with any firmware control of accelerator resources). The control subsystem 104 can send commands to one or more DMA controllers 116, VPUs 118, and / or PPEs 120. The control subsystem 104 can send coordination commands to one or more DMA controllers 116, VPUs 118, and / or PPEs 120 for performing memory transfer tasks, program execution tasks, post-processing tasks, synchronization tasks, and / or any firmware-distributed accelerator operations.

[0074] In some implementations, at least one (e.g., each) DMA channel can independently (and concurrently with other DMA channels) execute a sequential list of transfers (e.g., a linked list, a circular buffer, and / or any firmware-managed command sequence), wherein at least one (e.g., each) transfer can contain up to n nested data move loops (e.g., 5, 8, and / or any number of nested loops), conditionally gated memory accesses, address strides, and / or can perform automatic data padding (e.g., to implement boundary conditions, zero padding, tile alignment, and / or any structured memory layout adjustments in 2D data moves), and can be triggered on various loop dimensions. DMA channels can operate in parallel and share input-output buffers with memory interfaces such as DRAM 130 and / or local memory 126. Examples of input and output buffers can be referred to as AXI data buffers (ADB) and VMEM data buffers (VDB). Buffer allocation (e.g., ADB and VDB allocations) can be static and / or directly converted to the memory bandwidth available to each DMA channel.

[0075] One or more DMA controllers 116 can control the DMA engine to perform data movement. One or more DMA controllers 116 can receive commands from the control subsystem 104 for controlling the DMA controllers 116. In some implementations, the control subsystem 104 can generate hardware-specific commands based on high-level instructions received from the CPU host 128. In some implementations, the CPU host 128 can execute API calls to the control subsystem 104 (e.g., via CUPVA), which the control subsystem 104 can translate the API calls into device code for execution by the DMA channel.

[0076] Local memory 126 (e.g., VMEM) may be memory dedicated to PVA subsystem 102. One or more DMA controllers 116, VPU 118, and / or PPE 120 may create one or more buffers within local memory 126. One or more DMA controllers 116, VPU 118, and / or PPE 120 may move data into and out of one or more buffers within local memory 126.

[0077] In some implementations, DLUT 122 may be a hardware component of system 100, implemented using firmware-controlled execution logic to process memory-resident data. For example, DLUT 122 may operate on data stored in local memory 126 and / or external memory (e.g., DRAM 130), where DMA transfers and / or local memory binding may be used to coordinate memory access. In some implementations, command buffer 134 may include at least one command that identifies DLUT 122 as a target hardware component, for example, for performing data transformation, lookup operations, and / or other memory access operations based on one or more resource identifiers. Firmware component 106 may initiate execution of DLUT 122 by decoding entries in command buffer 134 and issuing configuration and control signals based on command content and metadata stored in resource table 110. Furthermore, depending on availability and system scheduling, execution by DLUT 122 may occur sequentially or overlap with other hardware components. In some implementations, the DLUT 122 can be integrated into the accelerator task pipeline, including command building, memory binding, and / or firmware dispatch execution.

[0078] Local memory 126 may store and / or otherwise maintain an instruction cache. The instruction cache may cache instructions for execution by one or more VPUs 118. That is, the instruction cache may be a memory structure for storing VPU program instructions fetched by firmware for execution. For example, the instruction cache may store a stream of instructions referenced by a resource identifier (loaded by firmware component 106 during command processing) in command buffer 134. In some implementations, the instruction cache is part of one or more VPUs 118. In some implementations, at least one (e.g., each) of one or more VPUs 118 includes an instruction cache.

[0079] In some implementations, the Command Submission and Execution (CSE) phase can be a phase within the accelerator pipeline, where system 100 can provide a task execution layer to schedule work units of the hardware accelerator. System 100 may include at least one control subsystem 104. Firmware component 106 of control subsystem 104 may construct a command buffer 134 in DRAM 130, the command buffer 134 including at least one command identifying at least one resource. Firmware interface 108 may engage with DRAM 130 to write command buffer data structures using direct memory access. That is, firmware component 106 may construct the command buffer 134 to be stored in DRAM 130 by filling buffer blocks with encoded commands referencing resource identifiers. For example, construction in DRAM 130 may include: using firmware interface 108 to allocate, initialize, and / or format command buffer regions aligned with firmware-defined boundaries. In this example, the buffer format and block descriptor may be written to the memory region in execution order. In some implementations, the firmware component 106 may construct the command buffer 134 and provide it to the memory region 132 of the DRAM 130 for storing the command buffer 134. That is, the firmware component 106 may allocate a region in the DRAM 130 and write a sequence of commands using prefetched metadata and command field encoding. For example, the firmware component 106 may link buffer blocks by embedding a next block resource identifier. In this example, the firmware interface 108 may populate the command header and link fields as part of the firmware control construction.

[0080] Typically, firmware component 106 (also referred to herein as "firmware" and / or "firmware system") may include firmware interface 108 and resource table 110. Firmware component 106 may be implemented as a software-controlled task management and execution module running on microcontroller logic within control subsystem 104. That is, firmware component 106 may execute from on-chip memory and manage execution ordering across hardware units. For example, firmware component 106 may use stored metadata and execution context to build, validate, and / or dispatch commands in DRAM 130. Furthermore, firmware interface 108 may access shared memory (e.g., local SRAM 124 and / or local memory 126) and signaling registers (e.g., one or more registers 114) to coordinate submission and execution. In some implementations, firmware interface 108 may be implemented as an instruction path executed by the microcontroller, including DMA handlers and resource mapping logic. That is, firmware interface 108 may receive command buffer submission indicators and decode command sequences based on tail pointer movement. For example, firmware interface 108 can retrieve and interpret command buffer entries from DRAM memory region 132 in response to queue tail index updates. Furthermore, resource table 110 can store verification metadata, DMA bindings, and / or executable references. In some implementations, resource table 110 can be implemented as a firmware-managed data structure indexed by resource identifiers. That is, resource table 110 can be updated by firmware based on initialization input from KMD. For example, resource table 110 can store executable code locations, input / output memory handles, and / or binding policies according to user context.

[0081] Furthermore, in some implementations, DRAM 130 may receive a command buffer 134 constructed by a non-privileged execution context via firmware interface 108. That is, firmware interface 108 may write the command buffer 134 into a memory region 132 in DRAM 130, which is allocated for storing command buffers corresponding to a given user context. For example, firmware interface 108 may receive a pointer to the command buffer 134 via a commit queue entry written by CUPVA to a memory-mapped commit interface. In this example, the commit queue may be a DRAM-based memory region 132 accessible to both firmware interface 108 and the non-privileged execution context. Firmware interface 108 may detect a commit indicator written to register 114 of signaling interface 112 and use it to retrieve the corresponding queue entry. In response to this commit, firmware interface 108 may retrieve the command buffer 134 from a specified location in DRAM 130. Firmware component 106 may verify the resource identifier in command buffer 134 against entries in resource table 110. Firmware component 106 can then decode commands and issue them to the hardware sequentially or out of order based on hardware resource availability.

[0082] In some implementations, at least one resource can be identified by a resource identifier (e.g., a reference to an executable file, DMA configuration, and / or DRAM buffer). Command buffer 134 may include at least one command that identifies at least one resource by the resource identifier. Furthermore, the command buffer may include multiple command buffer blocks (e.g., divided into sized units to allow linking across multiple buffer blocks). At least one of the multiple command buffer blocks may include a bounded size. For example, a first command buffer block may include a command that identifies a second command buffer block (e.g., cross-blocking). In this example, the resource identifier identified in the command can be used to retrieve the second command buffer block. In some implementations, at least one of the multiple command buffer blocks may include a next block descriptor that indicates a resource identifier corresponding to a subsequent command buffer block among the multiple command buffer blocks. That is, the next block descriptor can be used by firmware component 106 to perform prefetching during execution.

[0083] In some implementations, firmware component 106 may receive at least one command buffer 134 from a kernel-mode driver (KMD) (e.g., host software stack) corresponding to a privileged execution context via a dedicated signaling path (e.g., a signaling interface between the KMD and firmware) different from the signaling interface corresponding to the non-privileged execution context. For example, command buffer 134 may correspond to resource table init (initialization), queue creation, and / or registration of global resources (e.g., VPU executables, memory segments, and / or DMA configurations). In some implementations, the CCQ0 of one or more registers 114 may be reserved for kernel-only signaling. Furthermore, firmware component 106 performs at least one management operation to update resource table 110 based on at least one command buffer 134 received from the KMD (e.g., init and / or deinit (deinitialization) commands, appending new entries to resource table 110 with verified IOVA ranges and / or executable symbol mappings).

[0084] In some implementations, firmware component 106 may facilitate and / or allow engagement between privileged layers. For example, firmware component 106 may register (e.g., during initialization) resource configurations of a hardware accelerator (e.g., PVA subsystem 102) with a privileged execution context. In this example, resource configurations may include DMA configurations stored in a resource table corresponding to a non-privileged execution context (e.g., verified configurations may be stored in resource table 110 logically bound to a user context). Resource configurations may be identified using resource identifiers (e.g., indirect IDs) assigned during registration.

[0085] Furthermore, the at least one resource may include a direct memory access (DMA) configuration, which includes at least one static slot placeholder and at least one dynamic slot placeholder. The at least one static slot placeholder can identify at least one memory binding at registration time. That is, the static slot may be a concrete buffer provided at registration time. For example, the KMD can verify access to statically bound buffers and embed the actual address in the descriptor. The at least one dynamic slot placeholder can identify the memory binding at execution time. That is, the dynamic slot placeholder typically does not have a concrete buffer at registration time. For example, the KMD can calculate the access range of the dynamic slot. In this example, firmware component 106 can verify these ranges when binding slots to buffers during the execution of the command buffer. In some implementations, firmware component 106 can bind at least one dynamic slot placeholder to at least one memory buffer identified by at least one resource identifier in resource table 110 corresponding to a non-privileged execution context during the execution of command buffer 134 (e.g., which is stored in one or more memory regions 132 of DRAM 130). That is, firmware component 106 can allow DMA configurations to be used across different DRAM buffers.

[0086] In some implementations, firmware component 106 may perform DMA configuration verification prior to registration by verifying consistency with at least one access limit (e.g., constraints regarding size, offsets for DMA transfers within the buffer, at least one memory address range (e.g., the I / O virtual address (IOVA) range of a DRAM buffer registered with the KMD) and / or at least one descriptor formatting parameter (e.g., the structure defining a valid DMA descriptor). That is, firmware component 106 may compare the configuration parameters with predefined access policies encoded in the firmware logic. Furthermore, firmware component 106 may store verification metadata (e.g., access range, slot type, transfer size constraints, and / or any descriptor format verification results) in a resource table 110 corresponding to the non-privileged execution context before assigning a resource identifier to the DMA configuration. In other words, firmware component 106 may retain the verified configuration constraints for reference during future command buffer execution.

[0087] The CSE phase may also include firmware component 106 providing commit information for the command buffer to a commit queue corresponding to the non-privileged execution context. The non-privileged execution context can be any software process, application, or thread (such as a user-mode driver, application-level program, or isolated computation thread running in a restricted address space) that executes without direct access to kernel-level system resources, memory-mapped device control regions, and / or firmware-maintained data structures. The commit queue may be stored and / or otherwise maintained in DRAM 130. The commit queue may be implemented as a circular buffer. The commit information may include a tail index (e.g., in the case of writing the next commit) that identifies the insertion point of the command buffer in the circular buffer. The commit queue may correspond to a non-privileged execution context because the queue is mapped to a memory region accessible by the UMD. In some implementations, the commit queue may be written by user-mode software via a memory-mapped address space. That is, the commit queue may contain opaque entries acquired by firmware component 106 upon signaling.

[0088] Furthermore, firmware component 106 and / or signaling interface 112 detect updates corresponding to the current tail value of the commit queue based on commit indicators. That is, firmware component 106 can check whether at least one (e.g., each) new CCQ write (e.g., register 114) includes an incrementing tail value. In some implementations, firmware component 106 can disable (e.g., mask CCQ interrupts) signaling interface 112's reception of subsequent commit indicators in response to detecting that the current tail value is less than or equal to a previously processed tail value. That is, if a commit does not indicate progress, firmware component 106 can suppress further interruptions from the context. For example, disabling firmware component 106 can be performed in response to detecting a non-monotonic tail value (e.g., the tail value is not monotonically increasing). In some implementations, signaling interface 112 can be re-enabled in response to the closure of a non-privileged execution context and / or the creation of a new non-privileged execution context. For example, when the KMD (e.g., operating on the main processor of CPU host 128) notifies firmware component 106 that a new context has been created and a CCQ has been assigned, signaling interface 112 can be re-enabled.

[0089] In some implementations, metadata may correspond to resource identifiers stored in resource table 110. Metadata may also correspond to non-privileged execution contexts (e.g., each user and / or UMD context may maintain its own resource table). For example, metadata may include the location, access range, DMA configuration, buffers and buffer size of DRAM 130, the next set of block descriptors, and / or any memory configuration parameters referenced by the firmware during execution of executable files (e.g., VPU 118 and / or PPE 120). Furthermore, firmware component 106 may verify command buffer 134 based on metadata stored in resource table 110 before causing execution of command buffer 134. That is, the firmware component performs verification at commit time, thereby improving runtime security without requiring kernel invocation (e.g., alleviating the KMD's runtime verification responsibilities).

[0090] In some implementations, firmware component 106 may store and / or cache at least one previously constructed command buffer in DRAM 130 for reuse. That is, firmware component 106 may maintain references to buffers in resource table 110 to support multi-use commits. Furthermore, firmware component 106 may use resource identifiers stored in resource table 110 corresponding to non-privileged execution contexts to reference and / or otherwise retrieve at least one previously constructed command buffer from commit information. That is, firmware component 106 may match command buffer handles with metadata previously stored in resource table 110. For example, DMA configurations may be registered to the KMD and reused by referencing them with resource IDs. In some implementations, firmware component 106 may reuse at least one previously constructed command buffer for execution by at least one hardware component of a hardware accelerator (e.g., PVA subsystem 102). That is, firmware component 106 may reissue execution logic for a validated buffer. For example, if the configuration remains unchanged, firmware component 106 may bypass the re-resolution of the DMA descriptor. At least one hardware component may include a vector processing unit (VPU) 118 (e.g., which executes a user-submitted kernel), a direct memory access (DMA) controller 116 (e.g., handling DRAM 130 and local memory 126 (e.g., VMEM) transfers), and / or a programmable or pixel processing engine (PPE) 120 (e.g., a coprocessor for post-processing stages).

[0091] The CSE phase may also include firmware component 106 providing a commit indicator corresponding to the commit message to signaling interface 112 corresponding to the commit queue. Specifically, firmware component 106 may write an updated tail index to register 114 (e.g., a CCQ register). For example, firmware interface 108 of firmware component 106 may write a tail value to a memory-mapped register (e.g., one or more registers 114) corresponding to a CCQ slot assigned to the UMD. Signaling interface 112 may be a Command Contention Queue (CCQ) register interface exposed to user space. In some implementations, the commit indicator may be provided to and / or stored in a memory-mapped CCQ register (e.g., one or more registers 114) to indicate a new tail index for the commit queue. For example, the tail may be incremented (e.g., if firmware component 106 detects a duplicate or invalid value, it may mask the CCQ to prevent further interruptions from that context until, for example, a reset).

[0092] Typically, signaling interface 112 may include multiple registers 114. Signaling interface 112 may be implemented as a firmware-managed memory-mapped interface for interpreting user-triggered execution requests. In some implementations, signaling interface 112 may monitor CCQ entries for state transitions. That is, signaling interface 112 may detect changes in the head / tail pointers as commit triggers. For example, when a new commit arrives, signaling interface 112 may notify firmware component 106 by raising an internal interrupt (e.g., via firmware interface 108). Furthermore, registers 114 may store control values ​​indicating queue status.

[0093] In some implementations, one or more registers 114 may be implemented per context. That is, one or more registers 114 may be statically assigned per UMD context during initialization. For example, one or more registers 114 may provide isolation across multiple command queues by partitioning interrupt signals with queue access metadata. Furthermore, one or more registers 114 may store control values ​​indicating queue status, including tail and head indices, status flags, interrupt enable or mask bits, and error indicators used by firmware component 106 to track processing progress, determine the availability of new entries, and / or control the signaling behavior of the corresponding submission queue. In some implementations, firmware component 106 of control subsystem 104 may create user contexts and / or otherwise instantiate user contexts using dedicated queues and resource tables. For example, firmware component 106 may instantiate multiple unprivileged execution contexts, at least one of which corresponds to a different submission queue and a different resource table. In this example, up to seven non-privileged UMD contexts (e.g., or any number of n contexts) can be instantiated, each context being mapped to a register 114 (e.g., CCQ) and a different resource table 110. A unique commit queue for at least one of the multiple non-privileged execution contexts can correspond to a different signaling interface 112. That is, at least one (e.g., each) context can be isolated using a dedicated queue, a CCQ register, and / or resource mappings in the firmware. In some implementations, the commit queue and CCQ can be used to prevent cross-context interference.

[0094] In some implementations, command buffer 134 may include an affinity mask indicating at least one of at least one hardware component of a hardware accelerator that is permitted to execute command buffer 134. The affinity mask may be a bitmask or an identifier indicating which hardware component is allowed to execute command buffer 134. That is, the affinity mask may be used by firmware component 106 to restrict execution of command buffer 134 to a subset of at least one hardware component (e.g., one or more DMA controllers 116, one or more VPUs 118, one or more PPEs 120) to prevent interference with at least one additional command buffer of at least one additional non-privileged execution context. For example, if the hardware accelerator includes two PVAs, the affinity mask may restrict command buffers to PVE0 and / or PVE1. In this example, execution contexts may be isolated, each having its own queue and CCQ.

[0095] The CSE phase may also include firmware component 106 causing at least one hardware component of the hardware accelerator to execute a command buffer, which identifies at least one resource. That is, firmware component 106 can issue commands from command buffer 134 in placement order and implement overlapping execution based on hardware availability. In some implementations, firmware component 106 can trigger firmware interrupt handling to initiate dequeueing and / or execution of the command buffer. Firmware component 106 can dispatch command buffer 134 to available hardware units (e.g., loading a VPU executable, programming DMA hardware, and / or executing commands). Hardware components can be one or more DMA controllers 116, VPUs 118, or one or more PPEs 120. For example, at least one (e.g., each) component can be acquired and released via commands in the buffer (e.g., pva_cmd_acquire_engine, pva_cmd_release_engine). In this example, firmware component 106 can perform command decoding and manage dispatch ordering based on resource availability and component readiness.

[0096] In some implementations, the Memory Binding and Persistence (MBP) phase can be a phase within the accelerator pipeline, where system 100 can provide a data path configuration layer for creating and / or managing resource bindings and / or access policies within the hardware accelerator. Firmware component 106 of control subsystem 104 can allocate local memory regions (e.g., local SRAM 124, such as L1 / L2 SRAM, and / or any other on-chip memory) of a hardware accelerator (e.g., PVA subsystem 102, which includes a VPU, DMA controller, PPE, and task logic) bound to external memory regions (e.g., DRAM 130 and / or any system-accessible memory not co-located with the hardware accelerator) via firmware interface 108 (e.g., a commit path via KMD to send a structured command buffer to the firmware). That is, the allocation can return a device pointer corresponding to the binding. This binding can be tracked using resource identifiers in resource table 110 corresponding to a non-privileged execution context (e.g., for user-space applications and / or tasks executed under a specific CCQ). Firmware component 106 may reserve and / or provide contiguous areas of L2SRAM (e.g., local SRAM 124) for use by applications, command buffers, and / or operators. This allocation may be logical and / or tracked via metadata (e.g., size and association with external memory) by firmware component 106.

[0097] In some implementations, firmware component 106 may request to reserve segments of L2SRAM for use. Binding may include associating an allocated L2SRAM region with a DRAM buffer (e.g., one or more memory regions 132 of DRAM 130). In some implementations, firmware component 106 may facilitate mapping between L2SRAM and DRAM-based memory locations to form persistent data bindings. A device pointer may be a logical identifier given to user space to represent a mapped DRAM location (e.g., it is not a direct memory address, but an abstract handle used by software to construct a DMA descriptor). L2SRAM memory regions may be allocated without explicit location specification, and firmware interface 108 may select memory regions based on a best-fit strategy. After binding is created, firmware component 106 may generate a unique identifier referencing a specific local-to-external memory binding (e.g., an opaque handle used by user-space software or other FW layers to reference the binding without knowing the low-level implementation details). In some implementations, resource table 110 may maintain a mapping between resource identifiers and associated metadata (e.g., local memory region address, external memory region backup address, access policy, binding status, reference count) for binding purposes.

[0098] In some implementations, firmware component 106 can utilize this binding during execution that allows shared memory (L2SRAM) to work with DRAM across task windows. For example, firmware component 106 can construct at least one command buffer 134 in an external memory region of the hardware accelerator (e.g., DRAM 130), which includes at least one command identifying a device pointer corresponding to a binding between a local memory region (e.g., local SRAM 124) and an external memory region (e.g., one or more memory regions 132). That is, firmware component 106 can use the DRAM-mapped device pointer to configure DMA transactions (such as using any other DRAM buffer pointer). In some implementations, firmware component 106 can insert the device pointer into a DMA descriptor (e.g., DMA read or write, binding L2SRAM, flushing L2SRAM, or invalidating L2SRAM, including convolution computation operations). In this example, the device pointer can be embedded, but can be used by the application and / or as a typical DRAM address during runtime.

[0099] In some implementations, firmware component 106 may invoke one or more DMA controllers 116 to use command buffer 134. For example, firmware component 106 may patch DMA descriptors with physical addresses in L2 SRAM, apply access policies, and / or initiate transfers between DRAM 130 and local SRAM 124. That is, firmware component 106 may enable the use of at least one command buffer via direct memory access (DMA) controller 116 (e.g., performing transfers between DRAM 130 and local SRAM 124) to execute (e.g., submit to firmware or a work queue, causing hardware (e.g., DMA controller 116) to perform the requested operation) at least one command via a local memory region using an external memory region. In some implementations, at least one command may correspond to at least one of the following: (i) reading data from an external memory region (e.g., load: FILL_ON_MISS command (runtime start)) to a local memory region, or (ii) writing data from a local memory region (e.g., refresh: FLUSH_ON_EVICTION command (runtime end)) to an external memory region. For example, DMA controller 116 may read CNN layer weights from DRAM 130 into local SRAM 124 (e.g., input load). In another example, DMA controller 116 may write computed image feature maps from local SRAM 124 (e.g., L2 SRAM) back to DRAM 130 (e.g., output refresh). In some implementations, multiple VPUs may share a common data buffer that can be loaded and / or refreshed.

[0100] In some implementations, allocation may include providing a request to firmware interface 108 for allocating a local memory region in the hardware accelerator's local memory (e.g., local SRAM 124) and binding that local memory region to an external memory region (e.g., memory region 132). For example, the request may initiate at least one action. For example, the request may include an action for allocating a local memory region (e.g., a block of L2 SRAM) in the hardware accelerator. In another example, the request may include an action for binding that local memory region to an external memory region (e.g., a DRAM-enabled buffer). In some implementations, allocation may include: firmware component 106 receiving a device pointer from firmware interface 108 corresponding to the binding from the local memory region to the external memory region. The device pointer may represent an indirect reference to the binding and is used during DMA configuration. In some implementations, the request can be generated by a privileged execution context (e.g., a trusted OS kernel, system firmware, and / or during CUPVA runtime, in a runtime or system-level environment with the authority to issue low-level hardware control commands) using a firmware access application programming interface (API) (e.g., exposed by CUPVA), as a firmware-level configuration command. For example, the firmware-level configuration command can identify a binding between a local memory region and an external memory region (e.g., local SRAM 124 to memory region 132).

[0101] In some implementations, at least one command buffer 134 may include descriptor data (e.g., metadata) that identifies a device pointer. For example, the descriptor data may be updated prior to execution to identify a memory access path from an external memory region via a local memory region. Furthermore, the memory access path may correspond to a patched physical address identified in the local memory region for executing at least one command (e.g., the result of firmware component 106 rewriting the descriptor data of command buffer 134 to point to a valid hardware address in L2SRAM (e.g., local SRAM 124)). The device pointer may correspond to an address indirection that includes a binding from the local memory region to the external memory region. In some implementations, the address indirection may be identified by DMA controller 116 to execute at least one memory access command during execution. For example, the L2SRAM memory region (e.g., in local SRAM 124) may be accessed by a direct memory access (DMA) engine and / or a vector processing unit (VPU). In this example, the device pointer can be used to configure the DMA engine.

[0102] In some implementations, control subsystem 104 may perform task submission. Firmware component 106 may construct command buffer 134 in an external memory region of the hardware accelerator (e.g., one or more memory regions 132 of DRAM 130), the command buffer 134 including at least one command identifying a resource corresponding to a local memory region (e.g., in logical SRAM 124). Firmware component 106 may provide submission information for command buffer 134 to a submission queue (e.g., stored in DRAM 130) corresponding to a non-privileged execution context. Firmware component 106 may provide a submission indicator corresponding to the submission information (e.g., to be stored in one or more registers 114) to a signaling interface 112 corresponding to the submission queue. For example, firmware component 106 may notify the hardware that new work (e.g., command buffer 134) is ready for execution. In some implementations, firmware component 106 may cause command buffer 134 to be executed by at least one hardware component of the hardware accelerator.

[0103] In some implementations, firmware component 106 can reuse a previously allocated L2 region for a given user context with the same DRAM backup location. For example, a reference to the L2 region can be looked up in local SRAM 124 based on a resource identifier, where a reference count stored in memory region 132 can indicate whether a DRAM binding has been validly made available for reuse. Otherwise, firmware component 106 can reserve a new region in L2, associate that region with a DRAM backup, initialize L2 (based on a policy), and increment the reference count to track concurrent bindings. For example, firmware interface 108 can allocate a new local memory region in L2 memory and initialize the contents from external memory according to a FILL_ON_MISS policy.

[0104] Before allocating a local memory region, firmware component 106 can determine whether a previously allocated local memory region (e.g., in local SRAM 124) bound to an external memory region (e.g., one or more memory regions 132) exists for a non-privileged execution context. That is, when, for example, the user context remains unchanged, firmware component 106 can reuse the L2SRAM memory region without flushing and / or reloading data between command buffer executions. In response to determining that a previously allocated local memory region exists, firmware component 106 can increment the reference count corresponding to the binding. For example, firmware component 106 can detect that subsequent and preceding task windows are associated with the same user context and maintain data in the L2SRAM memory region between task windows without redundant transfers. Furthermore, in response to determining that a previously allocated local memory region does not exist, firmware component 106 can send a request by invoking firmware interface 108 for allocating a local memory region in the local memory bound to an external memory region for the hardware accelerator according to at least one access policy.

[0105] In some implementations, the firmware component 106 of the control subsystem 104 can determine whether a previously allocated local memory region bound to an external memory region exists for a non-privileged execution context by querying metadata maintained in resource table 110. For example, resource table 110 may store bindings between L2SRAM allocations and DRAM buffers, each binding associated with a reference count and one or more access policies (e.g., FILL_ON_MISS, FLUSH_ON_EVICTION). In response to determining that a previously allocated local memory region exists for the same DRAM buffer, firmware component 106 can increment the corresponding reference count, allowing commands across execution contexts to reuse the same L2SRAM allocation. In response to determining that no such binding exists, firmware component 106 can invoke firmware interface 108 to allocate a new local memory region (e.g., in L2SRAM) and bind it to the DRAM buffer based on a specified access policy. In this example, firmware interface 108 can manage allocation requests and update resource table 110 with the new binding and an initialized reference count. Furthermore, firmware component 106 can provide a device pointer corresponding to a local memory region to a memory location accessible to the non-privileged execution context for subsequent construction of a command buffer referencing that binding. That is, firmware component 106 can internally manage the reuse and allocation of local memory resources based on the execution context state via firmware interface 108.

[0106] In some implementations, firmware component 106 of system 100 can manage the L2SRAM memory lifetime by tracking reference counts associated with bindings between local memory and external DRAM. For example, in response to receiving a release request from a non-privileged execution context via signaling interface 112, firmware component 106 can identify the corresponding L2SRAM region in local SRAM 124 based on a resource identifier and decrement the reference count in memory region 132 of DRAM 130. In this example, the reference count tracks the number of active bindings of different command buffers or execution contexts to the same external DRAM region. If the reference count meets the release condition (e.g., reaches zero), firmware interface 108 can mark the corresponding L2SRAM region as free and remove the associated DRAM backer entry from resource table 110. That is, the local memory region can be released when all reference command buffers have been completed or deallocated. In some implementations, firmware component 106 can issue a pva_cmd_release_l2sram command to complete the release of the L2SRAM region, which can be executed asynchronously after the dependent command has completed.

[0107] In some implementations, firmware component 106 can force a refresh of local SRAM 124 to a DRAM backup in DRAM 130. Firmware component 106 can initiate a refresh operation in response to a refresh request to transfer data from the local memory region to an external memory region. For example, the refresh operation can occur in response to a user context switch. Firmware component 106 can initiate an invalidation operation to mark data stored in the local memory region as invalid in response to an invalidation request used to trigger reinitialization of the local memory region during subsequent accesses. For example, firmware component 106 can invalidate data by marking L2 data (e.g., metadata bits in a per-region control structure) as dirty for forcing reinitialization. In some implementations, data initialization (at allocation) and / or data refresh (at release) can occur at the time of a user context switch (e.g., lazy and optimal). For example, in response to a user context switch, the local memory region can be initialized with data from an external memory region and refreshed to the external memory region. Alternatively, a data refresh can be explicitly requested. For example, a refresh can be performed in response to an explicit request.

[0108] In some implementations, the firmware component 106 of system 100 can facilitate data persistence between local SRAM 124 and DRAM 130 based on refresh and invalidation operations initiated via signaling interface 112. For example, in response to a user context switch detected at signaling interface 112, firmware component 106 can initiate a refresh operation to transfer data from a previously bound region in local SRAM 124 to a corresponding external memory region in DRAM 130 using DMA controller 116. In this example, the refresh operation can be triggered by an explicit refresh request issued in the command buffer or implicitly as part of a firmware context switch handler. Furthermore, firmware component 106 can process invalidation requests received from a non-privileged execution context to mark an L2SRAM region as invalid. That is, the invalidation operation can remove any valid state of the local memory region in resource table 110, thus requiring reinitialization of data from DRAM 130 during the next access. In some implementations, dedicated firmware commands (such as pva_cmd_release_l2sram) that enforce persistence or invalidation policies based on access patterns can be used to represent refresh and invalidation operations in the command buffer.

[0109] In some implementations, binding can be established according to at least one access policy. The at least one access policy may include a first policy for initializing the local memory region with data from an external memory region and / or a second policy for flushing data from the local memory region to the external memory region during release. Firmware component 106 can manage L2 SRAM persistence using access policies corresponding to initialization and eviction operations. When a non-privileged execution context requests access to a resource that has not been previously loaded, firmware interface 108 can apply the FILL_ON_MISS policy to allocate and initialize a region of local SRAM 124 from the corresponding DRAM-supporting location in DRAM 130. For example, firmware component 106 can initiate a DMA transfer using DMA controller 116 to fill the allocated L2 region with data from a DRAM region identified in resource table 110. In another example, when the L2 region is released, firmware component 106 can apply the FLUSH_ON_EVICTION policy to flush data from the L2 region back to DRAM 130. That is, in response to a release request or context switch, firmware component 106 can issue a DMA transfer from the L2 region in local SRAM 124 to the corresponding DRAM address. These access policies can be associated with resource identifiers in resource table 110 and enforced using firmware-controlled access logic during command buffer execution.

[0110] In some implementations, firmware component 106 may maintain multiple concurrent bindings between different regions of local SRAM 124 and corresponding regions of DRAM 130 for a single nonprivileged execution context. For example, firmware interface 108 may associate at least two different local memory regions with at least two different external memory regions by tracking binding information in resource table 110. At least one (e.g., each) binding may be represented by a resource identifier and associated with a reference count to manage concurrent use across command buffers. In another example, when a nonprivileged execution context submits multiple command buffers referencing different DRAM-backed memory segments, firmware component 106 may allocate a separate region in local SRAM 124 for at least one (e.g., each) segment and manage persistence and access behavior independently based on a policy. That is, firmware component 106 may maintain a mapping between at least one (e.g., each) L2 region and its DRAM counterpart and track their usage status to prevent conflicts or data corruption. These concurrent bindings may remain active until at least one (e.g., each) corresponding reference count is decremented to zero, at which point firmware component 106 may release the associated local memory region.

[0111] Firmware interface 108 may include a compatibility layer implemented to translate memory allocations and at least one access request from at least one software application for a version (e.g., a previous, different, or older) of the hardware accelerator. The compatibility layer can be used to update at least one request for retrieving allocations to local memory regions and binding to external memory regions via firmware interface 108. In some implementations, firmware interface 108 may include a compatibility layer for translating memory allocations and access requests from user-mode software for a previous version of the hardware accelerator. For example, the compatibility layer may intercept and update legacy API calls corresponding to DRAM-only memory allocations to retrieve allocations to local memory regions in SRAM 124 and bind them to external memory regions in DRAM 130. In another example, in the absence of firmware-managed SRAM binding, an application for a previous generation accelerator implementation may submit a memory access request, which may be intercepted and updated by the compatibility layer to retrieve an internal request from firmware component 106 via firmware interface 108 to perform a binding operation. That is, the compatibility layer can rewrite at least one command buffer or memory allocation descriptor to conform to the persistence policy managed by the firmware supported by the current accelerator. In some implementations, the compatibility layer can maintain a mapping between legacy memory handles and internal resource identifiers stored in resource table 110, thereby allowing firmware component 106 to manage reference counting and policy enforcement without requiring changes to the application.

[0112] In some implementations, the DMA controller 116 may use a patch descriptor to perform DMA memory transactions, including a source address, destination address, size of the data to be transferred, and / or control flags or policies (e.g., for performing fill on miss or flush on eviction). For example, firmware component 106 may set a descriptor to load CNN weights from DRAM to L2SRAM before the computation kernel runs. In another example, firmware component 106 may perform a flush of intermediate results from L2SRAM back to DRAM after composable operators have completed. That is, at least one command buffer 134 may include descriptor data that identifies a device pointer. The descriptor data may be modified by firmware component 106 to identify the source and destination addresses of the DMA transactions performed by the DMA controller, the source and / or destination addresses corresponding to local and external memory regions bound by the device pointer. Furthermore, the descriptor data may also include the transfer size and at least one access policy.

[0113] The MBP phase may also include firmware component 106 providing a device pointer to a memory region accessible to the non-privileged execution context for identifying at least one subsequent build of at least one command buffer 134 bound to the local memory region and the external memory region (e.g., using the device pointer to reference the build of at least one bound command buffer 134). That is, firmware component 106 may store the device pointer in a resource table entry and expose the device pointer separately to the non-privileged execution context for later use. For example, firmware component 106 may store the device pointer in a resource table entry for internal tracing, and it may expose the pointer separately (e.g., via a return value, command buffer output, or mapped memory structure) to the non-privileged execution context. In some implementations, firmware component 106 may expose the device pointer to the non-privileged execution context by embedding the pointer value in a response buffer returned from a batch of resource registration operations. For example, firmware component 106 may return the device pointer as part of a resource registration response submitted using pva_async_operations_submit. In this example, a user-mode software component (e.g., CUPVA) can initiate DMA configuration registration by appending an operation to the operation buffer using `pva_append_ops dma_config_register`, and firmware component 106 can assign resource identifiers corresponding to the backing DRAM region and associated L2SRAM allocation. A pointer to the bound local memory region (e.g., a device pointer) can be returned in the response buffer and extracted by the user-mode software using a parsing API (e.g., `pva_parse_register_resp`). Furthermore, the device pointer can be reused by the non-privileged execution context in subsequent constructions of the memory-bound command buffer.

[0114] In some implementations, providing may include updating the metadata structure associated with the context to reference the device pointer via a resource identifier. In some implementations, providing the device pointer may include writing the device pointer to a DRAM buffer mapped to a user-mode software context for later use during command buffer construction. Firmware component 106 can expose the firmware-generated device pointer by writing the device pointer to a DRAM buffer mapped to a user-mode software context (e.g., CUPVA), thereby allowing user-mode software to retrieve the pointer for subsequent command buffer construction. The non-privileged execution context may be a UMD-KMD context pair, representing an isolated view of the user application of the PVA. At least one (e.g., each) UMD context may be associated with a KMD context and a CCQ FIFO through which command buffer submissions can be routed. The user context can be used to construct command buffer 134 at a later point in time. That is, the subsequent construction can reference the device pointer via a resource ID and construct commands for accessing L2 SRAM using the previously bound DRAM region. For example, the user command buffer may include DMA binding and / or VPU parameter updates referencing the device pointer associated with the allocated L2 region. The association between local (e.g., L2 SRAM) and external (e.g., DRAM) regions can be maintained using internal firmware metadata that links resource identifiers to both local allocations and DRAM backups. For example, resource table entries can store IOVA limits, access policies, and / or pointers to L2 memory blocks assigned to that binding.

[0115] In some implementations, the Firmware Scheduling and Dispatch (FWSD) phase can be a phase within the accelerator pipeline, in which system 100 can dispatch firmware-controlled commands and perform resource-based orchestration for the hardware accelerator. Firmware component 106 of control subsystem 104 can construct a command buffer 134 comprising multiple commands in the hardware accelerator's memory. At least one of the multiple commands (e.g., initializing the VPU program, pre-fetching DMA configuration, triggering a DMA transfer) identifies at least one resource via a resource identifier in a resource table 110 (e.g., storing metadata such as memory address bounds, DMA descriptors, VPU code pointers) that is inaccessible to the non-privileged execution context. Firmware component 106 can construct the sequence of commands in a firmware-accessible memory-mapped region (e.g., in DRAM 130). That is, firmware component 106 can encode hardware control operations (e.g., setting CPU parameters, loading DMA configuration, triggering barrier synchronization).

[0116] One or more command buffers 134 can be used to establish a direct interface between the CPU host 128 (e.g., CUPVA) and the PVA subsystem 102 (e.g., hardware accelerator). The CUPVA can construct command buffers 134 and submit them to the firmware component 106 without modification by the UMD and / or KMD. Commands can be primitives representing low-level instructions corresponding to firmware consumable primitive operations. That is, the firmware component 106 can directly interpret and execute each command without requiring higher-level translation logic. For example, the firmware component 106 can dispatch a sequence of commands to initialize the VPU executable, bind resources from resource table 110, and / or initiate VPU or DMA operations. In some implementations, at least one (e.g., each) command can independently address hardware blocks (e.g., configure the DMA engine, trigger VPU code execution, etc.). Resource table 110 can be protected by the KMD and cannot be directly accessed by application layer software and / or users. A non-privileged execution context can be a UMD and KMD context pair, representing an isolated view of a user application's PVA. At least one (e.g., each) UMD context corresponds to a KMD context and a CCQ FIFO, submitted via its routed command buffer. For example, a context pair can be represented by a file descriptor identifying the UMD context and an associated kernel object handle identifying the KMD context, with at least one (e.g., each) mapped to a unique CCQ index (e.g., fd=5, kctx=0x32 maps to CCQ3).

[0117] Typically, resource table 110 may include verification metadata corresponding to resource identifiers. During runtime, a non-privileged execution context may not be able to access the verification metadata (e.g., pre-computed during initialization and / or updated during execution). In some implementations, firmware component 106 may use the pre-verified metadata stored in resource table 110 to check whether access to a resource is permitted. For example, firmware component 106 may enable the use of verification metadata stored in resource table 110 to verify access to at least one resource identified in command buffer 134. In this example, at least one command in the command buffer may correspond to hardware configuration parameters (e.g., low-level data or control values ​​defining specific hardware components of the accelerator, such as DMA engines, VPUs, and PPEs) stored in resource table 110 and identified using resource identifiers.

[0118] Furthermore, firmware component 106 can construct command buffer 134 within a user-mode application (e.g., command buffers can be written in user space) by recognizing at least one resource identifier stored in resource table 110. The construction of command buffer 134 can be performed independently of translations performed by the kernel-mode driver (KMD). In some implementations, firmware component 106 can register resource configurations with the KMD corresponding to the privileged execution context during system or context initialization. In some implementations, the KMD is not invoked during subsequent non-privileged command buffer commits to reduce host-side verification overhead and maintain the consistency of resource configuration data. That is, the KMD can act as a trusted agent to verify and register resources. During initialization, the KMD can verify resources and / or populate resource table 110. Furthermore, registering resource configurations may include verifying the resource configuration and population of resource table 110 (e.g., resource table 110 is inaccessible to the non-privileged execution context).

[0119] In some implementations, DRAM 130 may receive a command buffer containing multiple commands via firmware interface 108. Specifically, a non-privileged execution context (e.g., CUPVA) may construct command buffer 134 within DRAM memory region 132 accessible to firmware interface 108. For example, command buffer 134 may be stored in a memory-mapped region allocated by a user-mode application and tracked via a CCQ submission queue. In this example, firmware interface 108 may access the tail index of the CCQ FIFO to determine if a new command buffer 134 has been submitted. Firmware component 106 can then retrieve command buffer 134 from DRAM 130 and parse the commands sequentially or based on hardware availability. Each command in command buffer 134 may include a reference to a resource identifier pre-registered and verified by KMD during context initialization. Firmware interface 108 may use metadata from resource table 110 to verify access to the referenced resource. In this example, firmware interface 108 can decode command fields, track execution status, and issue commands to target components such as DMA controller 116, one or more VPUs 118, and / or one or more PPEs 120.

[0120] In some implementations, the commit interface may include (i) a commit queue storing commit information and (ii) a signaling interface 112 storing commit indicators. The commit queue may be a firmware queue (e.g., in DRAM 130) exposed to user mode as part of a per-context CCQ FIFO. The firmware queue may be shared in memory space between user space and firmware. For example, at least one (e.g., each) UMD context may have up to eight queues, at least one (e.g., each) of which is mapped to a firmware queue, and entries in the queues are opaque to the firmware until notified via CCQ. The signaling interface 112 may represent a CCQ register interface including one or more registers 114 exposed to user space.

[0121] The FWSD phase may also include firmware component 106 providing commit information for command buffer 134 and a commit indicator corresponding to the commit information to at least one commit interface (e.g., (i) a commit queue, such as a buffer with a head index and a tail index, and / or (ii) a signaling mechanism, such as a signaling interface 112 with one or more registers 114 to facilitate host-to-firmware handover) corresponding to the commit interface to the non-privileged execution context. Typically, the commit interface may include a communication channel for passing command buffer references and / or execution metadata from the user-mode context to the firmware. That is, the commit interface may include a commit queue in DRAM 130 for referencing command buffer commits and signaling registers (e.g., one or more registers 114) of a memory map for notifying the firmware of commit events. For example, the commit queue may include a circular buffer in the allocated DRAM 130, which is part of one or more memory regions 132 and used to queue command buffer descriptors from the non-privileged execution context. In another example, the signaling agency may include a memory-mapped register (e.g., one or more registers 114 in the control subsystem 104) for triggering firmware processing of the submission queue.

[0122] In some implementations, at least one commit interface may be a commit queue. That is, firmware component 106 can read from the commit queue to obtain a command buffer reference associated with a specific user context. For example, providing commit information may include writing commit metadata (including a new tail index) to a DRAM-enabled circular buffer associated with a non-privileged execution context. In some implementations, at least one commit interface may be a signaling authority. That is, firmware component 106 may monitor signaling interface 112 for updated values ​​indicating pending commits. For example, providing commit information may include writing to a CCQ register to indicate the availability of a new command buffer for firmware processing.

[0123] The FWSD phase may also include firmware component 106 causing (e.g., by calling and / or triggering firmware component 106) to execute command buffer 134 to (i) initiate the execution of multiple commands in the order they are placed in command buffer 134, and / or (ii) issue at least one subsequent command for execution before completing at least one previously issued command, based on the availability of at least one resource (e.g., one of n (e.g., 16, 32, 64, 128, etc.) independent DMA channels, VPU pipelines, and / or local memory regions, at least one (e.g., all) tracked by firmware component 106). That is, firmware component 106 may perform sequential fetching and decoding operations of command headers and dispatch commands to the appropriate hardware engine based on resource readiness. For example, firmware component 106 may determine that a DMA transfer command can be issued (even though an earlier VPU command is in progress). In some implementations, firmware component 106 may sequentially fetch and / or execute at least one (e.g., each) command (e.g., parsing command buffer 134 in linear order and invoking a handler for each command type). For example, the execution of multiple commands may occur from beginning to end in command buffer 134. In some implementations, firmware component 106 may perform out-of-order completion (e.g., issuing command B even if command A is still in progress, if command B targets a non-conflicting hardware resource). For example, commands may be issued sequentially but completed out of order. In some implementations, firmware component 106 may selectively parse a subset of commands without strict buffer order based on the priority or availability of hardware resources. For example, firmware component 106 may skip commands blocked on unavailable resources and temporarily continue executing subsequent commands referencing idle components.

[0124] Furthermore, the at least one command can identify and / or otherwise target a single hardware block as a hardware accelerator and cannot be functionally performed independently. That is, the command can be a low-level primitive that relies on sequential combinations with other commands to perform functional hardware behavior. For example, at least one of the multiple commands controls at least one hardware component of the hardware accelerator (e.g., one or more DMA controllers 116, one or more VPUs 118, one or more PPEs 120, and / or other system and / or components of system 100) and is executed sequentially with at least one additional command to perform operations using the hardware accelerator. In this example, control may include configuring DMA channels, setting VPU parameters, managing sequencing and / or dependencies, and / or any state progression defined by a state machine managed by the firmware corresponding to each component. The operation may include performing hardware tasks, implementing hardware configuration, executing a set of hardware instructions, and / or affecting multi-component accelerator behavior.

[0125] In some implementations, firmware component 106 may include multiple state machines responsible for and / or otherwise implemented for processing command streams, thereby facilitating the parallel verification and / or dispatch of multiple commands. Typically, the state machines may be implemented for at least one of one or more DMA controllers 116, one or more VPUs 118, and / or one or more PPEs 120. In some implementations, a first state machine may be implemented for DMA controller 116. That is, the first state machine may manage transitions from IDLE to FETCHING, SETTING UP, READY, and RUNNING states based on the execution of DMA commands. For example, firmware component 106 may use pva_cmd_fetch_dma_configuration to transition the DMA set from IDLE to FETCHING, retrieve a TCM scratch, and then issue pva_cmd_setup_dma and pva_cmd_run_dma to complete the transfer and return to IDLE.

[0126] In some implementations, a second state machine can be implemented for VPU 118. That is, the second state machine can track execution states in response to VPU-related commands, such as IDLE, BINDED, INITIALIZING, RUNNING, and CMD_RUN_VPU COMPLETED. For example, firmware component 106 can execute pva_cmd_set_vpu_executable, pva_cmd_init_vpu_executable, and pva_cmd_run_vpu to drive the state machine from IDLE to RUNNING, thereby marking transitions corresponding to asynchronous command execution (e.g., AD).

[0127] In some implementations, a second state machine can be implemented for PPE 120. That is, the state machine can respond to sequential PPE commands driving transitions from IDLE to BINDED, CLEARING VMEM, INITIALIZING, INITIALIZED, and RUNNING. For example, firmware component 106 can execute pva_cmd_set_ppe_executable, pva_cmd_clear_vmem, pva_cmd_init_ppe_executable, and pva_cmd_run_ppe to drive PPE state progression and manage shared VMEM access. In some implementations, a generic state machine can be implemented for all (common) one or more DMA controllers 116, one or more VPUs 118, and / or one or more PPEs 120. That is, the composite model can enforce execution order, mutual exclusion, and / or hardware sequencing constraints across different subsystems. For example, if the DMA state machine is not in an active state, firmware component 106 can prevent pva_cmd_set_vpu_executable from executing, thereby synchronizing DMA and VPU access to VMEM in system 100.

[0128] For example, at least one of the multiple state machines can use a corresponding resource identifier to verify at least one of the multiple commands. In this example, the command `cab` references a DMA configuration ID, and the firmware state machine can be used to look up and / or identify the associated metadata in resource table 110 to verify whether it defines a valid source-destination address and descriptor before proceeding. In another example, at least one of the multiple state machines can dispatch at least one verified command to be executed by a hardware component of the hardware accelerator. In this example, after verifying the DMA configuration, firmware component 106 can write the configuration to the DMA engine register and trigger the execution of the transfer, where dispatch can be performed autonomously and based on internal firmware ordering. In some implementations, the commands in the command buffer may include barrier commands (e.g., synchronization primitives) for synchronizing at least one hardware component of the hardware accelerator.

[0129] Now for reference Figure 2 Each block of the methods 200 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be performed using one or more processors (such as, but not limited to, those described herein) that execute instructions stored in one or more memories or memory systems. In some implementations, the computer process may also be implemented as computer-usable instructions stored on a computer storage medium. These methods can be provided by standalone applications, services or managed services (independently or in combination with another managed service), application programming interfaces (APIs), and / or plug-ins to another product, etc. Furthermore, as an example, regarding... Figure 1 and Figure 5-21 Method 200 is described. However, these methods may be additionally or alternatively performed by any system or any combination of systems (including, but not limited to, those described herein).

[0130] In block 202, method 200 may include: processing circuitry constructing a command buffer in memory, the command buffer including at least one command identifying at least one resource. In some implementations, in block 202, the processing circuitry may receive the command buffer in memory via a firmware interface, the command buffer including at least one command identifying at least one resource.

[0131] In block 204, method 200 may include: processing circuitry providing commit information for a command buffer to a commit queue corresponding to a non-privileged execution context. In block 206, method 200 may include: processing circuitry providing a commit indicator corresponding to the commit information to a signaling interface corresponding to the commit queue. In block 208, method 200 may include: processing circuitry causing at least one hardware component of a hardware accelerator to execute a command buffer that identifies at least one resource.

[0132] In some implementations, the processing circuitry can construct a command buffer such that at least one resource can be identified by a resource identifier, and that at least one command can identify at least one resource by the resource identifier. In some implementations, the processing circuitry can validate the command buffer based on metadata stored in a resource table corresponding to the non-privileged execution context, metadata corresponding to the resource identifier, and validation that occurs before the command buffer is executed. In some implementations, the commit queue can be a circular buffer, and the commit information can include the tail index identifying the insertion point of the command buffer within the circular buffer.

[0133] In some implementations, the processing circuitry can detect updates corresponding to the current tail value of the commit queue based on the commit indicator. In some implementations, the processing circuitry can disable the signaling interface from receiving subsequent commit indicators in response to detecting that the current tail value is less than or equal to the previously processed tail value. The signaling interface can be re-enabled in response to the closure of a non-privileged execution context and the creation of a new non-privileged execution context.

[0134] In some implementations, at least one hardware component may include a vector processing unit (VPU), a direct memory access (DMA) controller, or a programmable or pixel processing engine (PPE). In some implementations, the command buffer may include multiple command buffer blocks, at least one of which includes a bounded size, and a first command buffer block may include a command that identifies a second command buffer block, the second command buffer block being capable of retrieval using a resource identifier identified in the command.

[0135] In some implementations, at least one of the multiple command buffer blocks may include a next block descriptor indicating a resource identifier corresponding to a subsequent command buffer block among the multiple command buffer blocks. In some implementations, the processing circuitry may store at least one previously constructed command buffer in memory for reuse. In some implementations, the processing circuitry may use a resource identifier stored in a resource table corresponding to a non-privileged execution context to reference at least one previously constructed command buffer in the commit message. In some implementations, the processing circuitry may reuse at least one previously constructed command buffer for execution by at least one hardware component of a hardware accelerator.

[0136] In some implementations, at least one resource may include a direct memory access (DMA) configuration comprising at least one static slot placeholder and at least one dynamic slot placeholder. The at least one static slot placeholder identifies at least one memory binding at registration time, and the at least one dynamic slot placeholder identifies the memory binding at execution time. The processing circuitry may bind at least one dynamic slot placeholder to at least one memory buffer identified by at least one resource identifier in a resource table corresponding to a non-privileged execution context during the execution of the command buffer. In some implementations, the processing circuitry may instantiate multiple non-privileged execution contexts, at least one of which corresponds to a different commit queue and a different resource table, and the unique commit queue of at least one of the multiple non-privileged execution contexts corresponds to a different signaling interface.

[0137] In some implementations, the processing circuitry may receive at least one command buffer from the kernel-mode driver (KMD) corresponding to the privileged execution context via a dedicated signaling path different from the signaling interface corresponding to the non-privileged execution context. In some implementations, the processing circuitry may perform at least one management operation (updating a resource table) based on the at least one command buffer received from the KMD. In some implementations, the processing circuitry may register the resource configuration of a hardware accelerator with the privileged execution context. The resource configuration may include a direct memory access (DMA) configuration stored in the resource table corresponding to the non-privileged execution context, and this resource configuration may be identified using a resource identifier assigned during registration.

[0138] In some implementations, the processing circuitry may perform verification of the DMA configuration prior to registration by verifying conformity with at least one access limit, at least one memory address range, and at least one descriptor formatting parameter. In some implementations, the processing circuitry may store verification metadata in a resource table corresponding to the non-privileged execution context before assigning a resource identifier for the DMA configuration. In some implementations, the command buffer may include an affinity mask indicating at least one of at least one hardware component of a hardware accelerator that is permitted to execute the command buffer, and the affinity mask restricts the execution of the command buffer to a subset of at least one hardware component to prevent interference with at least one additional command buffer of at least one additional non-privileged execution context.

[0139] Now for reference Figure 3 Each block of the methods 300 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be performed using one or more processors (such as, but not limited to, those described herein) that execute instructions stored in one or more memories or memory systems. In some implementations, the computer process may also be implemented as computer-usable instructions stored on a computer storage medium. These methods can be provided by standalone applications, services or managed services (independently or in combination with another managed service), application programming interfaces (APIs), and / or plug-ins to another product, etc. Furthermore, as an example, references... Figure 1 and Figure 5-21 Method 300 is described. However, these methods may be additionally or alternatively performed by any system or any combination of systems (including, but not limited to, those described herein).

[0140] In block 302, method 300 may include: processing circuitry allocating a local memory region in the local memory of a hardware accelerator bound to an external memory region via a firmware interface, the allocation returning a device pointer corresponding to the binding, the binding being tracked using a resource identifier in a resource table corresponding to a non-privileged execution context. In block 304, method 300 may include: processing circuitry providing a device pointer to a memory region accessible by the non-privileged execution context for identifying at least one subsequent construction of at least one command buffer of the binding from the local memory region to the external memory region.

[0141] In some implementations, the processing circuitry may construct at least one command buffer in the external memory region of the hardware accelerator. This command buffer includes at least one command that identifies a device pointer corresponding to a binding between the local memory region and the external memory region. In some implementations, the processing circuitry may cause at least one instruction to be executed by a direct memory access (DMA) controller using the external memory region via the local memory region, utilizing the at least one command buffer. In some implementations, the at least one command corresponds to at least one of: (i) reading data from the external memory region into the local memory region, or (ii) writing data from the local memory region to the external memory region.

[0142] In some implementations, allocation may include: providing a request via a firmware interface for allocating a local memory region in the hardware accelerator's local memory and binding that local memory region to an external memory region. Allocation may also include: receiving a device pointer via the firmware interface corresponding to the binding from the local memory region to the external memory region. In some implementations, the request is generated as a firmware-level configuration command by a privileged execution context using a firmware access application programming interface (API), and this firmware-level configuration command identifies the binding from the local memory region to the external memory region.

[0143] In some implementations, at least one command buffer may include descriptor data identifying a device pointer, and this descriptor data is updated prior to execution to identify a memory access path from an external memory region via a local memory region, and this memory access path corresponds to a patched physical address identified in the local memory region for executing at least one command. In some implementations, the device pointer corresponds to an address indirection (which includes a binding from the local memory region to the external memory region), and this address indirection, identified by the DMA controller, is used to implement at least one memory access command during execution.

[0144] In some implementations, the processing circuitry may construct a second command buffer in an external memory region of the hardware accelerator, the second command buffer including at least one second command identifying a resource corresponding to a local memory region. In some implementations, the processing circuitry may provide commit information for the second command buffer to a commit queue corresponding to a non-privileged execution context. In some implementations, the processing circuitry may provide a commit indicator corresponding to the commit information to a signaling interface corresponding to the commit queue. In some implementations, the processing circuitry may cause at least one hardware component of the hardware accelerator to execute the second command buffer.

[0145] In some implementations, the processing circuitry may determine, before allocating a local memory region, whether a previously allocated local memory region bound to an external memory region exists for a non-privileged execution context. In some implementations, the processing circuitry may increment the reference count corresponding to the binding in response to determining that a previously allocated local memory region exists. In some implementations, the processing circuitry may send a request to the firmware interface in response to determining that a previously allocated local memory region does not exist, for allocating a local memory region in the local memory of a hardware accelerator bound to an external memory region according to at least one access policy.

[0146] In some implementations, the processing circuitry may decrement the reference count corresponding to the binding in response to a release request. In some implementations, the processing circuitry may release the local memory region in response to determining that the reference count satisfies a release condition. In some implementations, the processing circuitry may initiate a refresh operation in response to a refresh request to transfer data from the local memory region to an external memory region. In some implementations, the refresh operation occurs in response to a user context switch, and / or an invalidation operation is initiated in response to an invalidation request to mark data stored in the local memory region as invalid, triggering a reinitialization of the local memory region during subsequent accesses. In some implementations, the local memory region is initialized with data from the external memory region and refreshed to the external memory region in response to a user context switch. In some implementations, the refresh may also be performed in response to an explicit request.

[0147] In some implementations, bindings are established according to at least one access policy, which includes a first policy of initializing the local memory region with data from an external memory region, or a second policy of flushing data from the local memory region to the external memory region during release. In some implementations, the firmware interface maintains at least two concurrent bindings between multiple local memory regions and corresponding multiple external memory regions for a non-privileged execution context. In some implementations, the firmware interface may include a compatibility layer implemented to translate memory allocations and at least one access request from at least one software application of a version implementation for a hardware accelerator, the compatibility layer updating at least one request for retrieving local memory region allocations and bindings to external memory regions via the firmware interface.

[0148] In some implementations, at least one command buffer may include descriptor data identifying a device pointer, and this descriptor data is modified by firmware to allow the DMA controller to identify the source and destination addresses of a DMA transaction, which correspond to a local memory region and an external memory region bound by the device pointer. The descriptor data includes a transfer size and at least one access policy. In some implementations, the processing circuitry may send a request via a firmware interface to allocate a local memory region in the hardware accelerator's local memory and bind that local memory region to an external memory region. In some implementations, the processing circuitry may receive, via a firmware interface, a device pointer corresponding to a local memory region-to-external memory region binding, which is tracked using a resource identifier in a resource table corresponding to a non-privileged execution context. In some implementations, the processing circuitry may store the device pointer in a memory region accessible to the non-privileged execution context for at least one subsequent construction of at least one command buffer identifying the local memory region-to-external memory region binding.

[0149] Now for reference Figure 4 Each block of the method 400 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be performed using one or more processors (such as, but not limited to, those described herein) that execute instructions stored in one or more memories or memory systems. In some implementations, the computer process may also be implemented as computer-usable instructions stored on a computer storage medium. The method can be provided by a standalone application, service, or managed service (independently or in combination with another managed service), application programming interface (API), and / or plug-in to another product, etc. Furthermore, as an example, regarding... Figure 1 and Figure 5-21 Method 400 is described. However, these methods may be additionally or alternatively performed by any system or any combination of systems (including, but not limited to, those described herein).

[0150] In block 402, method 400 may include: processing circuitry constructing a command buffer comprising a plurality of commands in the memory of a hardware accelerator. At least one of the plurality of commands identifies at least one resource via a resource identifier in a resource table inaccessible to a non-privileged execution context. In some implementations, in block 402, the processing circuitry may receive the command buffer comprising a plurality of commands in the memory of the hardware accelerator via a firmware interface. In block 404, method 400 may include: processing circuitry providing commit information of the command buffer and a commit indicator corresponding to the commit information to at least one commit interface corresponding to a non-privileged execution context. In block 406, method 400 may include: processing circuitry causing a firmware component to execute the command buffer to (i) initiate execution of a plurality of commands in the order of placement in the command buffer, and (ii) issue at least one subsequent command for execution before completing at least one previously issued command, based on the availability of at least one resource.

[0151] In some implementations, at least one of the plurality of commands controls at least one hardware component of the hardware accelerator and is executed sequentially along with at least one additional command to perform an operation using the hardware accelerator. In some implementations, the firmware component may include a plurality of state machines, at least one of which is configured to (i) verify at least one of the plurality of commands using a corresponding resource identifier, and (ii) dispatch the verified at least one command to be executed by a hardware component of the hardware accelerator.

[0152] In some implementations, the resource table may include verification metadata corresponding to resource identifiers, which is inaccessible to the non-privileged execution context during runtime. In some implementations, processing circuitry may enable the use of the verification metadata stored in the resource table to verify access to at least one resource identified in the command buffer. At least one command in the command buffer corresponds to a hardware configuration parameter stored in the resource table and identified using a resource identifier. In some implementations, processing circuitry may construct the command buffer within a user-mode application by identifying at least one resource identifier stored in the resource table, and the construction of the command buffer is performed independently of any translation performed by the kernel-mode driver (KMD).

[0153] In some implementations, the processing circuitry may register a resource configuration with the KMD corresponding to the privileged execution context during initialization. Registering the resource configuration may include verifying the configuration and populating a resource table that is inaccessible to non-privileged execution contexts. In some implementations, at least one of a plurality of commands in the command buffer may include a barrier command for synchronizing at least one hardware component of the hardware accelerator. In some implementations, at least one commit interface may include (i) a commit queue storing commit information and (ii) a signaling interface storing commit indicators.

[0154] In some implementations, the at least one command may include a configuration command for configuring the direct memory access (DMA) controller of the hardware accelerator using DMA configuration. In some implementations, the at least one command may include at least one configuration or synchronization command for configuring, activating, or synchronizing the execution of the vector processing unit (VPU) of the hardware accelerator. In some implementations, the at least one command may include at least one configuration or synchronization command for configuring, activating, or synchronizing the execution of the programmable or pixel processing engine (PPE) of the hardware accelerator. In some implementations, the at least one command may include a query command for capturing execution metadata of at least one hardware component of the hardware accelerator and storing the execution metadata in a memory buffer referenced by a resource identifier in a resource table.

[0155] The systems and methods described herein (e.g., method 200, method 300, method 400) may be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, spacecraft, ships, shuttles (e.g., autonomous taxis (robotaxis)), emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles (e.g., manned or unmanned submarines), drones and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, such as, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets (e.g., NVIDIA's Omniverse), cloud computing and / or any other suitable application.

[0156] The disclosed implementations can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines, etc.), systems implemented using robots, aviation systems, medical systems, marine systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing analog operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing language models (e.g., large language models (LLM), visual language models (VLM), visual-language-action (VLA) models, and / or multimodal language models (MMLM)), and systems using or deploying a Systems comprising one or more inference microservices, systems containing one or more machine learning models and OS-level virtualization packages (e.g., containers) deployed within services or microservices, systems for performing one or more wireless cellular transmissions using wireless cellular networks, systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0157] refer to Figure 5 It describes an interface architecture for PVA based on a command buffer for task submission, according to some implementations of this disclosure. Figure 1 System 100 can achieve Figure 5 The architecture may include multiple layers, including user space 500, kernel space 510, and PVA execution space 520. System 100 may include a host CUPVA application executing in user space. The CUPVA application may engage with a UMA (such as a PVA UMD) to build command buffers and manage command metadata. The PVA UMD may communicate with a KMD (such as a PVA KMD) to register resources, assign resource identifiers, and / or initiate memory allocations. The PVA KMD may engage with PVA firmware (FW) (e.g., firmware component 106) (such as a PVA FW) to perform resource table updates, manage command queue associations, and / or trigger execution. The PVA FW may execute in a PVA core (e.g., an R5 microcontroller) and may receive commands via command queues mapped to user space 500. System 100 may facilitate communication from the PVA UMD to the PVA FW to submit command buffers (e.g., without KMD-based translation).

[0158] In some implementations, Figure 1System 100 can implement a command buffer model that decouples command construction from firmware execution semantics. That is, CUPVA applications can construct discrete commands for memory operations, engine control, synchronization, and / or signaling without conforming to a fixed task state machine. For example, a CUPVA application can construct a command buffer that includes memory copying and barrier operations (e.g., excluding VPU executable loading and / or PPE unloading). Figure 5 The architecture supports the flexibility and scalability of firmware component 106, where firmware logic can process individual commands based on the encoding and sequential placement of individual commands. The PVA UMD can submit buffers to appropriate memory-mapped queues (e.g., in DRAM 130), and the PVA FW can interpret and dispatch commands on supported hardware blocks. The control subsystem 104 of system 100 can expose memory-mapped signaling registers (e.g., one or more registers 114) to user space 500, allowing CUPVA to notify the firmware of new tail indices in the submission queue (e.g., stored in memory region 132). The PVA KMD can perform privileged operations such as global resource initialization and / or buffer registration, but can reduce execution path latency through direct user space submission. Command buffer interfaces (e.g., firmware interface 108) can reduce and / or eliminate context switching latency and / or redundant transition logic, thereby improving task submission efficiency for latency-sensitive applications.

[0159] For example, command buffer 134 submitted by CUPVA can bypass PVA KMD for runtime execution, thereby supporting direct execution via PVA FW. Firmware interface 108 can consume command buffer entries based on head and tail index metadata in a circular queue (e.g., in DRAM 130). Firmware component 106 can be invoked using signaling interface 112 mapped to control subsystem 104 (such as command completion queue (CCQ) registers (e.g., one or more registers 114)). At least one (e.g., each) CCQ can be assigned per execution context and can be used to manage independent command streams submitted from different CUPVA threads. PVA KMD can register resource identifiers used in the command buffer, thereby facilitating isolation and authentication across CUPVA contexts. Resource table 110 can be populated with metadata of DRAM buffers, DMA configurations, and / or executable references that can be used by PVA FW (e.g., firmware component 106) during execution. CUPVA can build and commit command buffers without embedding privileged memory addresses or executable handles, thereby referencing runtime objects via verified resource IDs.

[0160] refer to Figure 6 It depicts a system view of the resource registration and verification pipeline for PVA according to some implementations of this disclosure. Figure 1 System 100 can achieve Figure 6 The architecture includes various features and functions described herein. It comprises user space 600, privileged (kernel) space 602, and / or PVA device space 604. A user 614 operating in user space 600 (e.g., operating CPU host 128) can construct a DMA configuration registration request provided to KMD 616. DMA configuration 606 can be pre-verified by KMD 616, generating a verified DMA configuration IOVA 608. The verified DMA configuration IOVA 608 can be forwarded to firmware 620 (e.g., firmware component 106) for verification. Firmware 620 can update resource table 622 with the verified entry (such as verified DMA configuration 618). User 614 can receive a DMA configuration resource ID 612 representing the verified entry, which can later be referenced in a command buffer constructed in user space. Resource ID 612 can be an opaque handle corresponding to a resource managed by internal firmware and is not mapped to the user virtual address space.

[0161] In some implementations, firmware 620 and KMD 616 can facilitate access context-based authentication. KMD 616 can authenticate fixed parameters and bounds of DMA configuration 606, such as transfer size, burst length, and / or descriptor format. For slot-based DMA configurations with placeholders, KMD 616 can calculate access ranges and transfer limits (e.g., without resolving actual buffer addresses). Firmware 620 can perform authentication by verifying the runtime association between buffer bindings and pre-calculated bounds. The authenticated DMA configuration 618 is not exposed to the user 614. That is, the user can interact via DMA configuration resource ID 612. This indirection enforces isolation and prevents user-constructed command buffers from including IOVA addresses and / or memory references that cannot be authenticated at runtime. Resource table 622 can be maintained in firmware-managed memory and updated by trusted components such as firmware 620 and KMD 616. In some implementations, resource table 622 entries may include metadata defining access limits, memory associations, and / or policy constraints for a given DMA configuration. KMD 616 may register DRAM buffers and / or kernel-generated descriptors. Firmware 620 may associate memory policies with at least one (e.g., each) binding (such as fill on miss or flush on eviction) and track a reference count per context. User 614 may register multiple DMA configurations with reusable parameters. Firmware 620 may reference resource table 622 during runtime command execution to resolve resource IDs to usable configurations.

[0162] refer to Figure 7 It describes the execution path from user space to firmware via the submission queue 710 and the signaling interface according to some implementations of this disclosure. Figure 1 System 100 can achieve Figure 7 The various features and functions described herein. User 614 can use write operation 700 to write commit information to commit queue 710. Commit queue 710 can be implemented in memory interface 714, which is accessible to both user space and firmware. User 614 can then trigger Command Completion Queue (CCQ) notification 702 operation, thereby writing to signaling register to notify firmware 708. Firmware 708 (e.g., firmware component 106) can read the tail index from the queue and begin dequeuing commands for execution. Each command can reference a resource via a resource identifier indexed in resource table 712 (e.g., resource table 110). For example, firmware 708 can use a resource ID to perform lookup 704 to retrieve metadata. Firmware 708 can then configure hardware components such as memory interface 714, verified DMA configuration 718, and / or VPUbin 716.

[0163] In some implementations, the commit queue 710 may store fixed-size entries that remain opaque to the firmware until a commit indicator is received (e.g., from signaling interface 112). At least one (e.g., each) commit entry may include command buffer metadata, such as a buffer address, execution mask, front fence index, and / or back fence signal. Firmware 708 may verify the tail index and begin command buffer traversal. The command buffer (e.g., command buffer 134) may include a sequence of primitive commands referencing resource table 712. For example, commands may include DMA start, VPU kernel fetch, memory synchronization, and / or register write. Firmware 708 may prefetch command blocks into tightly coupled memory (TCM) and issue hardware instructions while command fetching and execution overlap. The execution sequence can be determined by command order and / or hardware availability, thereby allowing overlapping operations across memory interface 714, VPU bin 716, and PPE. The memory interface 714 can move data between DRAM (e.g., DRAM 130) and local memory (e.g., local SRAM 124 and / or local memory 126) according to the configured DMA descriptor.

[0164] Furthermore, command buffer execution can begin with user 614 constructing a commit message and writing it to memory shared with the firmware. Signaling paths can bypass the KMD, thus avoiding context switching latency and / or inefficiency. Firmware 708 decodes command buffer entries based on resource references and command metadata. Resource table 712 provides verified information to prevent the execution of unauthorized or malformed commands. Firmware 708 can use scheduling mechanisms to dispatch instructions across available engines, taking into account affinity masks defined in the command buffer. Verified DMA configuration 718 and VPU bin 716 can represent firmware-registered resources used during execution. Commands can use coded instructions to acquire and release hardware engines, thereby controlling access to the VPU and DMA controllers.

[0165] refer to Figure 8 The present disclosure describes a system 800 according to some implementations of the present disclosure, the system 800 including multiple queues (e.g., queues 802, 804, 806) corresponding to at least one user context and / or privileged context. Figure 1 System 100 can achieve Figure 8The various features and functions described herein. Queue 802 may represent a queue associated with a privileged execution context (e.g., a kernel-mode driver (KMD)). Queues 804 and 806 may represent queues associated with two corresponding non-privileged execution contexts (e.g., user context 1 and user context 2). Queue 802 may submit command buffers to a first command completion queue (CCQ0) 810. Queues 804 and 806 may submit command buffers to command completion queues CCQ1 814 and CCQ2 818, respectively. Resource table 808 (e.g., resource table 110) may correspond to CCQ0, and similarly, resource tables 812 and 816 (e.g., one or more resource tables 110) may correspond to CCQ1 and CCQ2. At least one (e.g., each) resource table may be isolated per user context and store metadata of resource identifiers referenced in the command buffers submitted via the associated queues. System 800 may implement a CCQ per user context, such that at least one (e.g., each) submission queue may share a signaling interface (e.g., signaling interface 112) with the firmware for submitting command buffers (e.g., one or more command buffers 134).

[0166] In some implementations, queues 802, 804, and 806 can be implemented as ring buffers in firmware-accessible memory (e.g., in DRAM 130) and populated by user-mode or kernel-mode software. At least one (e.g., each) queue in the user context (e.g., queues 804 and 806) can be associated with a corresponding queue ID included in the commit message to facilitate firmware-level demultiplexing. Commits from command buffers of queues 804 and 806 can be routed via CCQ1 814 and CCQ2 818, which can be used as signaling paths to trigger firmware acquisition and execution. CCQ0 810 can be reserved for privileged command buffers (e.g., registration, resource table initialization, or system-wide operations) committed by the KMD. In this example, system 800 can support up to seven non-privileged user contexts (and / or any number of contexts based on implementation and / or application), each non-privileged user context having up to eight queues (and / or any number of queues based on implementation and / or application), thus generating up to 56 (more or less) user queues across system 800. At least one (e.g., each) CCQ (e.g., 810, 814, 818) can be associated with a unique signaling register (e.g., one or more registers 114) exposed to the firmware via signaling interface 112. That is, the firmware can use the signaling register to detect queue activity and process the corresponding command buffer submissions.

[0167] In some implementations, at least one (e.g., each) user context can be instantiated using a dedicated resource table (e.g., resource table 812 or 816), a dedicated set of queues (e.g., queue 804 or 806), and a dedicated signaling register corresponding to a CCQ (e.g., CCQ1 or CCQ2). Queues in at least one (e.g., each) user context can operate independently, allowing command buffers to be submitted to the firmware without the involvement of the KMD. That is, submissions from any queue (e.g., queues 0 through 7 in user context 2) can be triggered for execution via CCQ2 818, using resource identifiers mapped in resource table 816. Isolation between contexts prevents interference between concurrent user submissions, and / or context-specific resource tables allow verification and execution to remain within the scope of at least one (e.g., each) context. The firmware can use the CCQ tail index written by user mode to identify available submissions and retrieve the corresponding entries. KMD can update and manage resource table 808 (e.g., a resource table for global resources), while user-specific tables (e.g., 812 and 816) can be built by the firmware during context creation. That is, at least one (e.g., each) CCQ is used as a per-context interface for command scheduling and execution.

[0168] In some implementations, system 100 may perform error handling operations during command buffer execution using firmware component 106. That is, error handling can be implemented by classifying errors associated with at least one context, command, and / or global state into recoverable, contextual, or global categories. For example, a recoverable error may terminate the command buffer associated with an invalid DMA configuration without affecting other commands (e.g., FW: command buffer execution timeout, FW: VPU returns a non-zero exit code, KMD: insufficient memory during ioctl / devctl, KMD: registration failure due to invalid user input). In another example, a contextual error might place the entire user context in a permanent error state, such as a corrupted queue or an invalid resource descriptor. In yet another example, a global error might invalidate all open contexts in the event of a critical firmware failure (e.g., FW: R5 SMMU failure due to a FW vulnerability, FW: fast reset failure). That is, firmware component 106 can receive error signals from hardware accelerator subsystems (e.g., DMA controller 116, VPU 118, PPE 120) and propagate state transitions based on the severity and scope of the error. Furthermore, given that firmware rebooting is generally not permitted in a secure environment, system 100 can omit recovery logic for firmware reboot scenarios. Errors that cannot be detected by system 100 (e.g., FW:DMA SMMU failure) can be handled by an external management program or monitoring layer.

[0169] In some implementations, the system stack of System 100 can merge the UMD and PVA interface layers into a unified software layer (e.g., UMD). The UMD layer can expose a user interface that removes multi-PVA management logic and eliminates and / or reduces the software abstraction previously involved in translating between file descriptor and hardware queue mappings. That is, command buffer construction and commit can be performed without queue-to-file descriptor multiplexing logic. For example, this stack avoids mapping multiple queues to a single UMD context (e.g., no multiplexing across struct pva_queue). In some implementations, CUPVA can assume responsibility for multi-context or multi-device disposal. That is, high-level logic outside the PVA runtime can perform resource management across multiple accelerators. In another example, this stack can avoid dynamically resolving affinity per queue, instead treating affinity as a per-commit attribute (e.g., an affinity mask defined in command buffer commits). In this example, scheduling behavior can be explicitly controlled by firmware in response to the affinity value provided at commit time.

[0170] In some implementations, creating a context in system 100 can instantiate the logic execution boundary associated with a single PVA engine. For example, firmware component 106 can create one KMD context and one UMD context per context instantiation request (e.g., via pva_context_create(uint32_t pva_index, uint32_t max_resource_count, ...). In this example, the context can be identified by a unique context handle returned by the API. That is, at least one (e.g., each) context can be mapped to a single command queue (e.g., CCQ FIFO) interface (e.g., register 114 of signaling interface 112). In some implementations, a PVA engine index (e.g., the PVA_index argument) can be provided during context instantiation to specify the target accelerator. Once created, the context can maintain resource and queue state isolation until explicitly destroyed (e.g., pva_context_destroy). For example, firmware can enforce resource table and submission queue isolation across contexts (e.g., using queue entries in per-context resource table 110 and DRAM 130). In this example, the context is destroyed only after all queues assigned to the context are destroyed. Furthermore, the context lifetime can be limited by hardware constraints on the number of CCQ and queue resources (e.g., 7 user contexts with up to 8 queues per context; 14 or 15 user contexts with up to 16 queues per context).

[0171] In some implementations, queues in system 100 can be created within an existing context and can represent a logical flow of command buffers. At least one (e.g., each) queue can correspond to a unique firmware-managed queue identifier. For example, this can be achieved by calling a queue creation API with a maximum commit count (e.g., pva_queue_create). , uint32_t max_submission_count, Queues can be created using a queue handle. In this example, the queue handle can be returned and used to reference the queue during command buffer commits. That is, queue affinity can be decoupled from the queue instance and specified per commit. In another example, a queue can be destroyed using a queue destruction API (e.g., pva_queue_destroy). In some implementations, firmware (e.g., firmware component 106) can differentiate queue ownership and execution behavior based on the queue ID. Furthermore, queues in different contexts can reference different CCQ commit interfaces (e.g., commit queues in DRAM 130, where signaling is emitted via register 114). For example, at least one (e.g., each) context can support up to eight queues mapped to a single CCQ register. The firmware can demultiplex entries in a shared CCQ FIFO using the queue ID.

[0172] In some implementations, the memory allocation API (e.g., pva_memory_alloc(uint64_tsize, ...)) can be used. The system allocates memory objects in the PVA device access allocation system 100. The allocated memory may reside in DRAM 130 and may be mapped into the PVA address space. In some implementations, memory can be freed using appropriate APIs (e.g., pva_mory_free). For example, memory can be imported from an external handle using pva_memory_import, allowing integration with platform-specific memory allocation schemes. In another example, memory can be mapped into the CPU virtual address space using pva_memory_cpu_map and unmapped using pva_memory_cpu_unmap. That is, memory attributes (e.g., pva_memory_attrs struct) can be queried using a dedicated API (pva_mory_get_attrs) to retrieve offset, size, access mode, and / or handle information. Firmware and KMD can enforce access permissions and reference tracking during import and mapping operations. In some implementations, memory allocation and / or import operations can be registered as resources for command buffer submission (e.g., using resource table 110).

[0173] In some implementations, the system stack of System 100 can use batch asynchronous operations to facilitate resource registration. For example, the buffer for registration operations can use a buffer preparation API (e.g., pva_async_ops_buffer_init). , The buffer is initialized with a `uint32_t size`. Then, at least one (e.g., each) registration operation can be appended to the buffer using APIs such as `pva_append_ops_memory_register`, `pva_appent_ops_executable_register`, and / or `pva_append_ops_dma_config_register`. That is, the buffer can accumulate sequences of operations corresponding to multiple resource types (e.g., DRAM memory, VPU executable, DMA configuration). The bulk buffer can then be submitted to the KMD via `pva_async_operations_submit`. In this example, the response buffer can contain the resource ID assigned to the registered resource. This response can be parsed using APIs to extract the assigned resource ID (e.g., `pva_parse_register_resp`, `pva_parse_exec_register_resp`). This allows the command buffer (e.g., stored in memory region 132) to indirectly reference resources using identifiers, thereby facilitating verification by firmware component 106. In some implementations, the registration API can be used to associate DRAM buffers, VPU executables, PPE binaries, and DMA configurations with resource tables corresponding to specific execution contexts (e.g., maintaining per-UMD contexts in resource table 110).

[0174] refer to Figure 9 The present disclosure describes a system 900 according to some implementations of the present disclosure, the system 900 including a plurality of input descriptors 902 for use with DMA configuration, at least one (e.g., each) input descriptor containing at least a source (SRC) slot and a destination (DST) slot. Figure 1 System 100 can achieve Figure 9The various features and functions described herein. The source and destination slots of the descriptor can reference slot indices in a block of dynamic slot 904 or static slot 906. Dynamic slot 904 can correspond to a DRAM buffer, and / or static slot 906 can correspond to a VMEM buffer (e.g., local memory 126). At least one (e.g., each) slot reference in the descriptor can be used to locate the memory buffer to which the DMA transfer is applied. For example, descriptor 0 can reference dynamic slot 0 as the source and static slot 0 as the destination. In another example, descriptor 9 can reference static slot 1 as the source and dynamic slot 1 as the destination. Output descriptor 908 can use the resolved buffer address from the bound slot to store the resulting transfer parameters post-execution. System 900 can implement slot binding verification at execution time, where the firmware uses metadata stored in resource table 110 to verify the memory address range.

[0175] In some implementations, static slot 906 can be verified by KMD at registration time. That is, the static slot references a fixed memory buffer that is verified and stored during DMA configuration registration. The actual physical address of the static slot can be embedded into the descriptor by KMD (e.g., no longer requiring firmware to perform runtime patching). Dynamic slot 904 can be unbound at registration time and instead assigned an access range for subsequent binding. For example, slot 0 can be assigned for input DRAM and slot 1 can be assigned for output DRAM, and the firmware can perform range checks during command buffer execution. In this example, the firmware can verify at least one (e.g., each) dynamic slot binding based on access metadata pre-computed by KMD and stored in a resource table. The separation between dynamic slot verification and static slot verification allows DMA configurations to be reused with different buffer bindings. That is, the same DMA configuration structure can be reused across multiple image and / or memory regions with different dynamic slot bindings.

[0176] In some implementations, input descriptor 902 can implement sequential and / or nested DMA transfer loops for performing memory transfers using slot indirection. That is, at least one (e.g., each) descriptor can identify the source and destination slots using memory offsets and transfer lengths. During command buffer execution, the firmware can acquire descriptors, resolve slot references, verify bindings using resource table metadata, and / or program the DMA engine accordingly. For example, for a convolution kernel, descriptors 0 through 8 can load image tiles from DRAM into VMEM using dynamic slot 0 and static slot 0. The final output descriptor 908 can store the result of writing the convolved output tiles to dynamic slot 1. The firmware can execute the same descriptors across different input and output buffers by rebinding dynamic slots.

[0177] In some implementations, system 100 can use slot-based descriptors (abstracting direct buffer references) to configure DMA operations. That is, the DMA configuration in system 100 can reference source and destination slots (e.g., not explicit DRAM or VMEM addresses). These slots can be specified as static or dynamic. For example, during registration, static slots can be bound to specific memory buffers, where the KMD can verify access permissions and embed the resolved physical address into the DMA descriptor (e.g., using pva_append_ops_dma_config_register). Dynamic slots can defer memory binding to execution time, where firmware component 106 can verify access based on pre-computed ranges stored in resource table 110. For example, a convolutional application can define a DMA configuration with nine input descriptors and one output descriptor. Input file tiles in VMEM can be assigned to static slot 0, and output tiles to static slot 1. Input images in DRAM can be assigned to dynamic slot 0, and output images in DRAM to dynamic slot 1. During execution, command buffer 134 may include commands for binding dynamic slots to buffers referenced via resource IDs. Firmware component 106 can then perform associated DMA transfers by evaluating slot assignments and corresponding offset metadata.

[0178] refer to Figure 10 It depicts an example control flow 1000 for firmware processing command buffers of system 100 according to some implementations of this disclosure. Figure 1 System 100 can achieve Figure 10The various features and functions described herein. Control flow 1000 can begin from a command buffer (e.g., command buffer 134) initialized at the first command and continues sequentially across commands unless blocked. If the firmware detects an incomplete asynchronous command, it can initiate a check to determine whether the execution of the current command is valid given the current system state. For example, while an ongoing DMA operation is still in progress, a DMA configuration transfer may be unsafe, and the firmware can postpone command execution until the operation is complete. In this example, the firmware can enter a waiting state until the system transitions to a permitted state. If the firmware determines that a valid transition is impossible, it can terminate execution and abort the buffer. If the current command has been completed and / or marked as non-blocking, the firmware can proceed to the next command. In some implementations, the firmware can track pending asynchronous commands and maintain state information.

[0179] When sufficient resources are available and execution conditions are met, the firmware of system 100 (e.g., firmware component 106) can process commands. The firmware can distinguish between synchronous and asynchronous commands. For example, if the current command is asynchronous and has started but not yet completed, the firmware can skip subsequent commands until execution becomes valid. In this example, if the command is synchronous and incomplete, the firmware will halt until completion. If a command has been processed but not yet completed and is marked as asynchronous, the firmware can record the incomplete portion for later restart. In some implementations, if there are no executable commands due to pending asynchronous operations, the firmware can yield. The firmware can retry execution after the previously blocked command completes and the required hardware resources become available. In control flow 1000, commands marked with `struct pva_cmd_barrier` can be used to enforce explicit synchronization, thereby blocking further commands until the relevant operation completes.

[0180] The execution of firmware management in System 100 may include disposing of user DMA hardware allocations while processing commands in the command buffer. For example, a command requiring user DMA access may include a `struct pva_user_dma_allowance` field specifying the allocated resource. In this example, a 32-bit value describes the allocated DMA channel (4 bits), ADB range (5 bits each for start and count), and descriptor range (7 bits each for start and count). The firmware can use this allocation to avoid conflicts with ongoing asynchronous commands. If no allocation is specified, the firmware can be prevented from using user DMA hardware for the command. Commands that do not require DMA can continue independently, while commands requiring unavailable resources can be deferred until availability is restored.

[0181] In some implementations, the firmware of System 100 can execute generic commands that do not require privileged access and / or engine fetching. Generic commands may include no-operations, chunk linking, barriers, DRAM writes, resource deregistration, and / or command buffer status reporting. For example, `struct pva_cmd_noop` ​​can be used as a placeholder in the command buffer. In another example, `struct pva_cmd_link_chunk` can instruct the firmware to fetch the next chunk from a given resource ID and offset. Chunk linking commands can reference resource table entries by ID and define the offset using the `next_chunk_offset_hi` and `next_chunk_offset_lo` fields. In yet another example, `struct pva_cmd_barrier` can force the execution of asynchronous commands that complete a specified group before continuing execution, where the `barrier_group` field matches those assigned in the previous asynchronous command header. `struct pva_cmd_write_dram` can specify the DRAM resource ID and the offset for writing data. Additional general commands include `struct pva_cmd_unregister_resource`, which can remove a resource ID from the resource table, and / or `struct pva_cmd_request_status`, which can set the DRAM buffer address for firmware to report the status in the event of a command buffer abort.

[0182] In some implementations, privileged commands (corresponding to kernel-mode driver (KMD) interaction) can be issued to system 100 via CCQ0. These commands can include initialization and deinitialization operations for queues and resource tables. For example, `struct pva_cmd_init_resource_table` can specify `resource_table_id` and base address fields (e.g., `resource_table_addr_hi` and `resource_table_addr_lo`), where resource table ID 0 corresponds to the device table and IDs 1-7 correspond to the user context. In another example, `struct pva_cmd_deinit_resource_table` can deallocate a resource table previously initialized with the same ID. Queue control commands include `struct pva_cmd_init_queue`, which can specify the target CCQ ID (e.g., `ccq_id`), queue ID (e.g., `queue_id`), base address, and / or commit capacity (e.g., `max_n_submits`). Queue deinitialization commands (e.g., `struct pva_cmd_deinit_queue`) can reference the queue by ID and associated CCQ.

[0183] In some implementations, engine commands can be used within System 100 to explicitly control hardware acquisition and engine affinity for executing computational workloads. For example, `struct pva_cmd_acquire_engine` can request one or more processing engines (PVEs), where at least one (e.g., each) engine includes a VPU, a DMA controller, and / or a PPE. The acquired engines can be tracked in the firmware, and / or the execution of subsequent commands can be restricted to these engines. In another example, `struct pva_cmd_release_engine` can release all previously acquired engines (e.g., after execution completion or after an intermediate buffer). The firmware can maintain engine state to facilitate safe release behavior, even while the engines are still active. `struct pva_cmd_set_current_engine` can specify which of the acquired engines are active for the following commands.

[0184] refer to Figure 11A and Figure 11B It describes a VPU state machine 1100 according to some implementations of the present disclosure, which can be facilitated by the firmware component 106 of the system 100 using a command buffer. Figure 1 System 100 can achieve Figure 11A-11B The various features and functions described herein. In some implementations, the VPU state machine 1100 can begin in an idle state after acquiring the PVE engine. For example, a command such as pva_cmd_set_vpu_executable can transition state machine 1100 from an idle state to a bound state without initiating actual execution. In this example, the vpu_exec_resource_id field can identify the VPU executable file in resource table 110. In some implementations, the firmware can transition from a bound state to a clear state using the pva_cmd_clear_vmem command. This command clears the VMEM in preparation for VPU initialization. The firmware can enforce that pva_cmd_clear_vmem is executable when both the VPU state machine and the PPE state machine are in states such as IDLE or BINDED. Additionally, the pva_cmd_prefetch_vpu_code command can be issued to trigger the prefetching of VPU instructions from entry point 0 while remaining in a synchronized state transition path.

[0185] continue Figure 11A Once the VMEM is cleared and prefetching is complete, the firmware can transition to the INITIALIZING state using `pva_cmd_init_vpu_executable`. This command configures the VPU hardware, including setting code segment registers and copying data portions. In some implementations, this command may also carry the `pva_user_dma_allowance` field, which describes how user DMA channels and / or descriptors are allocated for the initialization task. The initialization command completes and causes a transition to the INITIALIZED state. The firmware can then begin setting parameters for the VPU program using commands such as `pva_cmd_set_vpu_parameter`, `pva_cmd_set_vpu_parameter_with_buffer`, and / or `pva_cmd_set_vpu_parameter_with_address`, at least one of which updates a symbol or buffer mapping. At least one of these commands can be asynchronous and cause the state machine to transition to the intermediate PARAMETER PENDING state based on the completion status and count of the parameter setting command.

[0186] Refer again Figure 11AWhen the parameter setting command completes, the firmware can use internal counters (such as COUNT=1 and COUNT=…) to track completion and determine when the VPU program is ready to run. Once initialization and parameter setting are complete, the firmware can issue the pva_cmd_run_vpu command. This command transitions the system from the INITIALIZED state to… Figure 11B The RUNNING state is shown (RUNNING state 1104). Since pva_cmd_run_vpu can be asynchronous, command buffer 134 can continue to issue other commands during VPU execution. At least one (e.g., each) transition to the RUNNING state can be marked with identifiers A, B, C, or D to distinguish specific execution paths in the command buffer and parameter update stream.

[0187] In some implementations, when the system is in RUNNING state 1104 ( Figure 11B When VPU execution is in progress, additional commands such as `pva_cmd_set_vpu_parameter` can be issued to update parameters midway through execution. The firmware can track parameter updates using a combination of command start / complete events and state transitions. For example, a second or third parameter setting command can transition the system to the `PARAMETER PENDING & VPU RUNNING` state, where the count field is updated accordingly. In this example, the firmware can detect that parameter injection continues asynchronously while VPU execution is in progress. The firmware can verify the symbol ID and memory offset specified in the command field (e.g., using `symbol_id` and `vmem_offset`). In another example, the command buffer can complete execution by observing that all parameter setting commands have completed and the VPU program has finished. In some implementations, the firmware can transition the VPU state machine 1100, which is in the `RUNNING` state 1104, to the `CMD_RUN_VPU COMPLETED` state. Depending on the number and order of in-flight parameter setting commands, the transition can be achieved via any of the marked paths B, C, and / or D. The firmware can also release the previously acquired PVE engine using `pva_cmd_release_engine` after the command buffer completes. In some implementations, the completion of the VPU program can be used as a synchronization point for subsequent DMA operations and / or command buffer commits. That is, system 100 can enforce ordering based on transitions to maintain correct runtime.

[0188] refer to Figure 12 It describes a DMA state machine 1200 instantiated per DMA configuration set in a system 100 according to some implementations of this disclosure. Figure 1System 100 can achieve Figure 12 The various features and functions described herein. In some implementations, at least one (e.g., each) DMA set (e.g., DMA set X, where X = 0…14) can enter the IDLE state before the firmware initiates configuration fetch. For example, the pva_cmd_fetch_dma_configuration command can initiate the transition from the IDLE state to the FETCHING state. This command can use dma_set_id to identify the set, use resource_id to specify the DMA configuration stored in DRAM, and / or use the pva_user_dma_allowance struct to optionally provide user ADB to accelerate configuration fetch. In some implementations, the firmware can fetch a sticky note memory area in the TCM to save the configuration. If no sticky note is available, the command will block. If no other command can complete and the sticky note is still unavailable, the firmware will abort the command buffer to prevent deadlock.

[0189] Furthermore, once the DMA configuration has been acquired in the TCM, state machine 1200 can transition to the FETCHED state. Based on this state, the firmware can issue `pva_cmd_setup_dma`, thus initiating the SETTING UP state. This command can use channel 0 to transfer the acquired configuration to the DMA hardware registers (e.g., user-provided ADB can expedite this step). For example, `pva_cmd_setp_dma` accepts the same `dma_set_id` and `pva_user_dma_allowance` fields to indicate the target configuration and available user buffers. During setup, the firmware can verify that all and / or some descriptor fields reference valid slots. Once setup is complete, the state machine can transition to the READY state, thus marking the DMA hardware as configured.

[0190] In the READY state, the firmware can execute DMA transfers using the pva_cmd_run_dma command. This command transitions the state machine to the RUNNING state and can be implemented to drive the configured DMA channel to completion. For example, pva_cmd_run_dma specifies dma_set_id for execution and blocking until all necessary ADB, VDB, and GPIO triggers are acquired. This command verifies that the DMA configuration being executed is associated with the VPU executable previously bound using pva_cmd_set_vpu_executable. If a mismatch is detected, the firmware aborts the command buffer. Once the DMA execution is complete, the state machine returns to the IDLE state and / or the TCM tag area can be released for use by other DMA sets.

[0191] In some implementations, the configuration of the DMA descriptor can depend on slot-based addressing managed via a binding command. For example, `pva_cmd_bind_dram_slot` can bind a DRAM surface to a dynamic slot in a specified DMA set. In this example, `dma_set_id`, `slot_id`, and `resource_id` identify the set, slot, and associated DRAM resource, respectively. The command can also include offset fields (`slot_offset_hi`, `slot_offset_lo`, `surface_base_offset_hi`, `surface_base_offset_lo`) that define how the descriptor address is calculated. The surface format flags (PVA_CMD_FLAGS_SURFACE_FORMAT_MSB, PVA_CMD_FLAGS_SURFACE_FORMAT_LSB) and / or block height flags (PVA_CMD_FLAGS_LOG2_BLOCK_HEIGHT_MSB, PVA_CMD_FLAGS_LOG2_BLOCK_HEIGHT_LSB) further affect how the descriptor can be resolved at runtime.

[0192] Additionally, virtual memory slots can be bound using `pva_cmd_bind_vmem_slot`. This command can specify `dma_set_id`, `slot_id`, `symbol_id`, and / or offset values ​​to link VMEM-based targets to the command buffer. During command buffer execution in system 100, slots can be resolved using a resource table corresponding to the active context. That is, firmware can enforce memory access permissions and / or descriptor consistency by cross-referencing bindings stored in the context's resource table. For example, firmware can verify that all dynamic slots have been bound and verified before the hardware transaction was issued when `pva_cmd_run_dma` was issued.

[0193] refer to Figure 13 It depicts a PPE state machine 1300 according to some embodiments of the present disclosure, which is controlled by the sequential execution of commands issued in a command buffer in system 100. Figure 1 System 100 can achieve Figure 13The state machine can begin with the IDLE state, where the firmware component can issue the pva_cmd_set_ppe_executable command to set the executable context for subsequent PPE operations. In this example, no additional setup is performed, and the state transitions to BINDED. From the BINDED state, the firmware component can issue the pva_cmd_clear_vmem command to drive the transition to the CLEARING VMEM state. This command clears the contents of the VMEM (e.g., local memory 126) and synchronizes across the VPU and PPE domains, and the firmware can enforce state validity by preventing VMEM updates while execution activities are in progress. After the clearing operation is complete, the state machine returns to the BINDED state.

[0194] Starting from the BINDED state, the firmware component can issue the `pva_cmd_prefetch_ppe_code` command. This command can prefetch start instructions from the entry point (e.g., without waiting for the operation to complete), and the PPE state can remain in BINDED. The firmware component can issue the `pva_cmd_init_ppe_executable` command to configure PPE hardware resources. In this example, the state transitions to INITIALIZING, and the firmware uses the `pva_user_dma_allowance` parameter to allocate user-supplied DMA resources for loading code and data segments. After initialization, the state machine transitions to INITIALIZED. The PPE hardware can then be programmed, and the firmware can continue runtime execution using command buffer instructions referencing the initialized state.

[0195] Starting from the INITIALIZED state, the firmware component can issue the pva_cmd_run_ppe command. This command initiates execution of the configured PPE procedure from the specified entry point and transitions state machine 1300 to the RUNNING state. In the RUNNING state, the firmware can track asynchronous execution and maintain execution ordering based on barrier groups and / or dependent commands. When the PPE procedure completes, the state machine returns to the INITIALIZED state. In this example, the firmware can issue a subsequent pva_cmd_set_ppe_executable or pva_cmt_run_ppe command to restart execution. The state machine can restrict transitions based on firmware-managed conditions to maintain VMEM consistency and enforce effective PPE execution ordering.

[0196] In some implementations of System 100, the firmware component can issue a query command to store a timestamp during the execution of the PPE program. For example, the `pva_cmd_capture_timestamp` command can store an 8-byte value at a specified `store_index` in the TCM resident buffer. This command can be implemented in `IMMEDIATE_MODE`, `VPU START MODE`, `VPU DONEMODE`, and / or `DEFER MODE`. In `DEFER MODE`, the command can capture a timestamp when all commands in a specified barrier group (e.g., `defer_barrier_group`) have completed. The buffer can contain up to 32 timestamp entries and / or firmware management access to avoid overwriting slots in use.

[0197] In addition, the firmware component can issue the `pva_cmd_dump_queries` command to write the captured timestamp from the TCM to DRAM. This command can specify `resource_id`, `start_index`, and / or `query_count` to describe the destination buffer in DRAM. The `defer_barrier_group_mask` parameter can be used to defer the execution of the dump command until the barrier group conditions are met. This command does not block subsequent commands and can be included in the command buffer containing PPE commands. That is, query commands such as `pva_cmd_capture_timestamp` and `pva_cmd_dump_queries` can be executed asynchronously without interfering with PPE state machine 1300 and / or the control logic affecting transitions between control states.

[0198] In some implementations, the firmware component of system 100 can issue a query command to capture a timestamp and retrieve it into a local TCM buffer for debugging or profiling purposes. For example, the firmware component can issue a `pva_cmd_capture_timestamp` command where `capture_mode` is set to `IMMEDIATE_MODE` to record the timestamp immediately upon command execution. In another example, the firmware component can issue the same command, where `capture_mode` is set to `VPU_START_MODE` or `VPU_DONE_MODE` to record the timestamp when the VPU starts or finishes execution. In yet another example, the firmware component can use `DEFER_MODE` with the `defer_barrier_group` identifier to delay capture until commands assigned to the same barrier group are completed. At least one (e.g., each) of the captured timestamps can be stored at a specified `store_index` in the TCM buffer, and the firmware component can track slot availability across up to 32 buffer locations. Subsequently, the firmware component can issue the `pva_cmd_dump_queries` command to write the stored timestamp to a DRAM buffer identified by `resource_id` using `dram_offset_lo` and `dram_offset_hi` used to calculate the write position. In some implementations, the dump command can be configured with `defer_barrier_group_mask` to defer the write until the dependent command group has completed. The query command can be included in a command buffer submitted by the user context and can be executed asynchronously.

[0199] In some implementations, the firmware component of system 100 can manage L2SRAM allocation using dedicated L2SRAM commands. For example, the firmware component can issue the `pva_cmd_bind_l2sram` command to acquire an L2SRAM region for command buffer execution. This command specifies the `dram_resource_id`, `dram_offset_lo`, and `dram_offset_hi` values ​​to define the range of backup DRAM for persistence. The firmware can determine the transfer behavior between DRAM and L2SRAM using appropriate access_policy values ​​(e.g., `FILL_ON_MISS` or `FLUSH_ON_EVICTION`). The size of the L2SRAM allocation can be indicated in the `l2sram_size` field, and user DMA resources can be assigned via the `pva_user_dma_allowance` structure included in the command. That is, the firmware component can delay the execution of subsequent commands until the L2SRAM is acquired and enforce restrictions preventing multiple L2SRAM allocations within the same command buffer. Upon completion of execution, the firmware component can issue the `pva_cmd_release_l2sram` command to release the previously allocated L2SRAM. In this example, the release command can be asynchronous and take effect only after all previously issued commands have completed. The firmware component can collaborate with resource table entries corresponding to the execution context to facilitate L2SRAM lifecycle tracking.

[0200] refer to Figure 14 It describes, according to some implementations of this disclosure, an L2SRAM 1402 that includes shared local memory accessible by DMA controllers 1414 and 1416 (also referred to herein as a “DMA engine”). Figure 1 The system 1400 has 124 local SRAM. Figure 1 System 100 can achieve Figure 14The L2SRAM 1402 may include two super-banks, labeled SUPER-BANK 0 (1404) and SUPER-BANK 1 (1406), each with a capacity of 512KB. The L2SRAM 1402 can be engaged with two DMA controllers via a 512-bit SRAM interface. Access to the L2SRAM 1402 can be arbitrated by an XBAR arbitration component 1408, which can forward requests to the decoding-firewall logic 1410. In this example, the decoding-firewall logic 1410 can enforce access permissions using a configurable set of hardware registers. In response to the detection of invalid access, the decoding-firewall logic 1410 can assert an interrupt signal to the control processor. Configuration of access policies and interrupt settings can be performed via a configuration interface 1412 that exposes a 32-bit APB bus. At least one of DMA controllers 1414 and 1416 (e.g., each) may include 16 channels, 64 descriptors, and / or a 4KB ADB pool for issuing read and write operations against L2SRAM 1402. Upon completion of a transfer, DMA0 INTR and DMA1 INTR can be interrupted by assertion from the respective DMA controller.

[0201] In some implementations, L2SRAM 1402 can be used as a persistent note space for workloads that reuse the same data across multiple operators. For example, data fetched from DRAM and used by convolution operators can be stored in one of the super-memory banks (1404 or 1406) to reduce redundant DRAM reads. In another example, intermediate results generated by one VPU can be stored in L2SRAM and consumed by another VPU unit without requiring data to travel to and from DRAM. In this configuration, the two super-memory banks can be accessed concurrently by independent DMA controllers, facilitating a pipelined access pattern across multiple execution stages. Hardware firewall configuration (via decode-firewall logic 1410) can isolate memory regions to ensure that at least one (e.g., each) user context only accesses its allocated region. Configuration writes from the host processor can update firewall and / or arbitration behavior via configuration interface 1412.

[0202] refer to Figure 15 It describes how some implementations of this disclosure facilitate mapping local memory (e.g., L2SRAM) to device pointers for persistent use across operator submissions. Figure 1 System 100 can achieve Figure 15The various features and functions described herein. At least one (e.g., each) memory layout may include an L2 region mapped by DRAM backing (e.g., MAP L2) and at least one operator commit region (e.g., OP(1), OP(2), ..., OP(N)) associated with a corresponding local memory region (e.g., VPU-L2). In composable operator commit examples 1500 and 1504, individual operator commits are assigned their own persistent VPU-L2 regions. In bulk commit examples 1502 and 1506, multiple operator blocks are committed together, and each operator may be paired sequentially with its corresponding VPU-L2 allocation. Firmware component 106 may reuse previously bound L2 regions across these operator commits without redundant DRAM-to-L2 copy operations. In some implementations, L2 allocations may be managed as device pointers exposed to user software and tracked in a resource table 110 corresponding to the commit context. This allocation may remain valid across multiple operator commits without explicit initialization or refresh instructions from the user.

[0203] In some implementations, firmware component 106 can associate L2SRAM allocations with DRAM backing based on access policy parameters (e.g., FILL_ON_MISS, FLUSH_ON_EVICTION) and track them in composable commit or bulk execution windows. For example, in bulk commit example 1502, each of OP(1) through OP(N) uses a persistent VPU-L2 allocation that remains valid during bulk commit, thus avoiding intermediate memory transfers. In another example, as illustrated in composable operator commit example 1500, OP(1) and VPU-L2 can be configured with DRAM-backed L2 isolation, and / or subsequent operator commits can reuse the same allocation without flush or invalidation steps. Both composable and bulk memory layouts can use DRAM-backed L2 mappings to reduce DRAM bandwidth usage and / or eliminate or reduce redundant copying overhead during execution.

[0204] refer to Figure 16A It describes a command buffer 1600 according to some implementations of the present disclosure, which uses a predefined set of resources stored in a resource table 1602 to execute a VPU application for matrix addition. Figure 1 System 100 can achieve Figure 16AThe various features and functions described herein. The command buffer may begin with BEGIN_CHUNK and includes engine fetch (ACQUIRE_ENGINE, where COUNT = 1), engine selection (SET_CURRENT_ENGINE, where INDEX = 0), and / or executable association (SET_VPU_EXECUTABLE, where EXEC_REOURCE_ID = 1). Subsequent commands initialize and implement VPU execution (INIT_VPU_EXECUTABLE), prefetch the VPU instruction stream (PREFETCH_VPU_CODE, where ENTRY = 0), and / or fetch DMA configuration (FETCH_DMA_CONFIGURATION, where SET = 0 and DMA_CONFIG_RESOURCE_ID = 2). Parameter setting commands (SET_VPU_PARAMETER, where VMEM_SYMBOL_ID = 1 and 2) provide input to the VPU program. Input / output surfaces are bound to dynamic slots using the BIND_DRAM_SLOT command (e.g., SLOT_ID=0, where DRAM_RESOURCE_ID=3). The command buffer configures the DMA controller (SETUP_DMA), initiates DMA execution (RUN_DMA), triggers VPU execution (RUN_VPU), and / or completes execution with engine release (RELEASE_ENGINE) and END_CHUNK. At least one (e.g., each) command references an entry in resource table 1602, including the VPU executable (MAT_ADD_VPU_EXEC), DMA configuration (MAT_ADD_DMA_CONFIG), two input images (INPUT_IMAGE0 and INPUT_IMAGE1), and one output image (OUTPUT_IMAGE).

[0205] refer to Figure 16B It describes a representation of some implementations according to this disclosure. Figure 16A The execution timeline of the firmware-controlled command buffer described in section 1604. Figure 1 System 100 can achieve Figure 16BThe various features and functions described herein. At least one (e.g., each) line represents a command, and / or time flows from left to right across the numbered time units. Synchronous commands (e.g., SET_CURRENT_ENGINE) can be executed sequentially and block subsequent execution until completion. Asynchronous commands (e.g., PREFETCH_VPU_CODE, RUN_DMA, RUN_VPU) allow the firmware to continue while execution continues in parallel. The INIT_VPU_EXECUTABLE command is executed synchronously and completes before the parameter setting command. A firmware stall occurs (e.g., after SET_VPU_PARAMETER) when prerequisites such as DMA slot binding or engine status verification have not been met. For example, the firmware stall between commands 8-10 and 13-15, as well as other stalls, describes the dependency on previous parameter setting and slot binding operations. The SETUP_DMA and RUN_DMA commands span time units 23-32 and are issued after the required DRAM slots have been bound and configured. RUN_VPU is executed in parallel with the tail of the DMA operation from time units 24-33.

[0206] In some implementations, Figures 16A-16B This collectively represents an example of a firmware-controlled execution pipeline implemented by System 100. Command buffer 1600 defines a set of low-level commands for dispatching the mat_add application, where resource table 1602 (e.g., resource table 110) serves as an indirect layer providing identifiers for executables, configurations, and buffers. Execution timeline 1604 depicts the sequential issuance of commands, driven by command dependencies and firmware-managed pauses, employing overlapping execution. Once slot bindings and memory states are verified, firmware verification and scheduling can dynamically issue commands and can still enforce completion ordering via synchronization points. For example, the BIND_DRAM_SLOT command must complete before issuing SETUP_DMA, and the firmware explicitly verifies the slot-to-resource association using resource table 1602 before triggering DMA or VPU hardware operations.

[0207] refer to Figure 16C It depicts an execution timeline 1606 according to some implementations of this disclosure, which illustrates the back-to-back dispatch of two VPU programs by the firmware components of system 100. Figure 1 System 100 can achieve Figure 16CThe various features and functions described herein. The command buffer may include (at least) two sequential sets of VPU-related commands, both executed on a single VPU pipeline. The first boot may begin with ACQUIRE_ENGINE and SET_CURRENT_ENGINE, followed by association of the first executable (SET_VPU_EXECUTABLE), initialization (INIT_VPU_EXECUTABLE), and / or instruction prefetching (PREFETCH_VPU_CODE). The firmware initiates DMA execution (RUN_DMA).<SET=0> Before VPU execution (RUN_VPU), DMA configuration acquisition (FETCH_DMA_CONFIGURATION) is triggered.<SET=0> ), issue parameter setting commands, and / or perform settings (SETUP_DMA)<SET=0> At least one of these operations (e.g., each) can correspond to a command controlled synchronously or asynchronously by the firmware. The timeline indicates that the firmware pauses after parameter and DMA settings, where it awaits prerequisite state transitions. The command sequence illustrates the first VPU execution from time 21 to 32.

[0208] The second VPU startup begins during the execution of the first VPU program. Specifically, FETCH_DMA_CONFIGURATION<SET=1> Issued between 22-25, thus overlapping with the first startup's RUN_VPU command. This overlap hides the configuration fetch delay for the second DMA set. After the first VPU program completes, the firmware initiates the second executable association (SET_VPU_EXECUTABLE), clears the VMEM (CLEAR_VMEM), and performs instruction prefetching (PREFETCH_VPU_CODE). The second setup (SETUP_DMA)...<SET=1> The initialization (INIT_VPU_EXECUTABLE) and execution are performed sequentially. The timeline shows the VPU idle period (bubble) between times 33-48, representing the gaps in active VPU execution due to delays from SETUP_DMA and parameter commands. Once the prerequisites are met, the firmware dispatches a second VPU for execution (RUN_VPU), thus ending with RELEASE_ENGINE.

[0209] In this example, execution timeline 1606 depicts the ordering and resource overlap of firmware management. The command buffer submitted by the CPU host can contain consecutive startups for the same VPU pipeline. Firmware components insert the configuration phase of a second startup alongside the execution phase of the first startup, thereby reducing overall idle time and maintaining accelerator throughput. For example, the firmware acquires a second DMA configuration while the VPU is running the first program. In this example, although DMA setup latency cannot be completely hidden, the execution plan avoids full serialization. At least one (e.g., each) DMA and VPU commands use identifiers (e.g., SET=0, SET=1) to limit the scope of resource usage to different sets of configurations, and the firmware only tracks state transitions and pauses execution when hardware dependencies require it.

[0210] Now for reference Figure 16D It depicts an execution timeline 1608 according to some implementations of the present disclosure, which illustrates a single VPU program utilizing three DMA configuration sets under the control of firmware component 106 of system 100. Figure 1 System 100 can achieve Figure 16D The various features and functions described herein. The command buffer can be started with ACQUIRE_ENGINE, SET_CURRENT_ENGINE, and / or SET_VPU_EXECUTABLE, followed by initialization (INIT_VPU_EXECUTABLE) and instruction prefetching (PREFETCH_VPU_CODE). DMA configuration set 0 can be configured via FETCH_DMA_CONFIGURATION.<SET=0> and SETUP_DMA<SET=0> This involves acquiring and / or programming the data. In some implementations, once the firmware verifies configuration integrity and transitions the associated DMA controller state machine, the first DMA execution (RUN_DMA) can begin.<SET=0> Simultaneously, the firmware can schedule the first VPU execution (RUN_VPU) to process input mapped via resource table entries. This command describes a firmware pause following parameter settings and setup commands, where dependencies on DMA ready states and resource bindings delay further progress.

[0211] The firmware can acquire the configuration of DMA set 1 during VPU execution in set 0. However, due to resource conflicts between DMA set 0 and set 1 (e.g., overlapping DMA channels and descriptors), the firmware cannot proceed with SETUP_DMA.<SET=1> Until RUN_DMA<SET=0> Completed. The pause window spans 24 to 30 seconds, as reflected in the execution timeline. This causes a VPU bubble, during which the VPU remains idle while the firmware sets the next DMA configuration. After successful setup, the second DMA execution (RUN_DMA) can proceed.<SET=1> Concurrently with execution set 1, the firmware uses FETCH_DMA_CONFIGURATION.<SET=2> The configuration of DMA set 2 is acquired, thereby preemptively initiating overlapping configuration delays. The firmware component issues the SET_VPU_PARAMETER command between setup phases to propagate the updated sequence values ​​to the VMEM.

[0212] The transition from DMA set 1 to set 2 does not involve hardware conflicts, thus allowing SETUP_DMA.<SET=2> With RUN_DMA<SET=1> Parallel execution. To enforce the correct sequence, the firmware component issues a BARRIER command at time 44, thus preventing DMA set 2 from running until set 1 has completed. Therefore, the setup time for set 2 is completely hidden after the execution of set 1, minimizing idle time in the VPU pipeline. The final DMA execution (RUN_DMA) is completed before issuing RELEASE_ENGINE.<SET=2> This example demonstrates the use of a sequential DMA configuration set within a single VPU program, where firmware components facilitate execution continuity using explicit coordination, barrier enforcement, and / or sequential updates. System 100 can track command issuance and completion across DMA and VPU hardware components and maintain runtime consistency through context-specific resource table verification.

[0213] Now for reference Figure 17 It describes a command buffer 1700 according to some implementations of this disclosure, which uses resource table 1702 to execute query commands to capture and extract timestamps before and after the execution of the VPU program. Figure 1 System 100 can achieve Figure 17The various features and functions described herein. The command buffer can construct VPU jobs for matrix addition (MAT_ADD_VPU_EXEC) programs that can consume two input and output images. Commands such as ACQUIRE_ENGINE, SET_CURRENT_ENGINE, SET_VPU_EXECUTABLE, and INIT_VPU_EXECUTABLE prepare the VPU context and initialize the executable. DMA configuration is achieved using FETCH_DMA_CONFIGURATION with resource ID 2 (MAT_ADD_DMA_CONFIG).<SET=0> The acquisition, and the referencing of resource IDs 3, 4, and 5 for the input and output images via memory slot binding, occurs. System 100 can configure and trigger DMA controller 116 using SETUP_DMA and RUN_DMA before starting the VPU program.

[0214] Before VPU execution, the firmware component can issue a CAPTURE_TIMESTAMP command, where DEFER=NONE and INDEX=0. A timestamp can be captured in query buffer slot 0 where the TCM resides. This action can establish a baseline for profiling and / or debugging and occurs before the VPU program is executed via RUN_VPU. The RUN_VPU command includes a BARRIER_GROUP=1 field, which allows the firmware to reference this group during deferred query processing. After the VPU program completes, the firmware component can issue a second CAPTURE_TIMESTAMP command, this time with DEFER=1 and INDEX=1. In some implementations, the firmware defers timestamp capture until all commands in barrier group 1 have completed.

[0215] Once both timestamps are stored in the TCM, the firmware component can issue a DUMP_QUERIES command to transfer the captured timestamps to the DRAM buffer referenced by resource ID 6 (TIMESTAMPS_BUFFER) in resource table 1702. This command specifies START_INDEX=0 and COUNT=2, indicating that both captured timestamps (before and after execution) are copied to DRAM. This allows user-mode software and / or CUPVA to retrieve and analyze timing data after the job is completed. These query commands (e.g., CAPTURE_TIMESTAMP and DUMP_QUERIES) do not interfere with the VPU state machine and can be executed asynchronously.

[0216] Example autonomous or semi-autonomous machines Figure 18AExamples of sensor positions with corresponding fields of view or sensing fields for autonomous or semi-autonomous vehicles 1800a, autonomous mobile robots (AMRs) 1800b, and humanoid robots 1800c, according to some embodiments of this disclosure. While three types of machines 1800 are shown, this is not intended to be limiting, and the machines 1800 described herein may include vehicles, cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police or emergency vehicles, ambulances, boats, construction vehicles, underwater vehicles, robots (e.g., AMRs, humanoid robots, robotic arms, end effectors, forklifts, etc.), drones, aircraft, vehicles coupled to trailers (e.g., semi-trailer tractors for hauling goods), and / or another type of vehicle or machine (e.g., driverless and / or vehicles or machines accommodating one or more passengers). In some cases, vehicles 1800a, AMRs 1800b, humanoid robots 1800c, and / or other machine types may be collectively referred to herein as machines 1800.

[0217] Regarding Vehicle 1800A, autonomous and semi-autonomous vehicles are typically described by automation levels, defined by the National Highway Traffic Safety Administration (NHTSA) (a division of the U.S. Department of Transportation) and the Society of Automotive Engineers (SAE) in their "Classification and Definition of Terms Related to Driving Automation Systems for Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 1800 may have one or more functions that meet Level 3 through Level 5 of autonomous driving. For example, Vehicle 1800 may be able to provide driver assistance (Level 1), partial automation (Level 2, Level 2+, Level 2++), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the implementation. The term “autonomy” as used herein can include any and / or all types of autonomy of Machine 1800 or other machines, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, provision of auxiliary autonomy, semi-autonomy, primary autonomy or other specified autonomy.

[0218] about Figure 18AThe sensors and their respective fields of view (not shown for clarity) or sensing fields (not shown for clarity) are an example implementation and are not intended to be limiting. Although not shown, each sensor may have a corresponding field of view (e.g., a 360-degree field of view around camera 1868D, a 180-degree field of view around wide-angle camera 1868B, a 360-degree sensing field of view of LiDAR sensor 1864, etc.). For example, only a subset of the sensors shown may be included, additional sensors may be included, alternative sensors may be included, the number of each sensor mode may be different, the sensor modes may be different (e.g., LiDAR or RADAR may not be included, SONAR, thermal sensors, etc. may be included), and the sensor positions may differ from those shown on vehicles 1800a, AMR 1800b, and / or humanoid robots 1800c, etc. For example, for vehicle 1800a, the location, number, modality, and / or other sensor information may differ depending on type (e.g., SUV, truck, car, robot, motorcycle, etc.), size (e.g., 18-wheeler, transport vehicle, small car, etc.), and associated functions (e.g., L2 vs. L5). Similarly, for AMR 1800b and / or humanoid robot 1800c, shape, size, purpose, implementation, model, etc., can determine the number and type of sensors used.

[0219] like Figure 18A As shown, autonomous or semi-autonomous vehicles 1800A, AMR 1800B, and humanoid robots 1800C may include different sensor types, numbers, and locations. As a non-limiting example, vehicle 1800A may include twelve cameras 1868, such as a front wide-angle camera (e.g., 120-degree field of view (FOV)), a front telephoto camera (e.g., 30-degree FOV), a side-rear left camera (e.g., 70-degree FOV), a side-rear right camera (e.g., 70-degree FOV), a front fisheye camera (e.g., 200-degree FOV), a rear fisheye camera (e.g., 200-degree FOV), a left fisheye camera (e.g., 200-degree FOV), a right fisheye camera (e.g., 200-degree FOV), a front telephoto satellite camera (e.g., 30-degree FOV), a rear telephoto camera (e.g., 30-degree FOV), a cross-left camera (e.g., 120-degree FOV), and a cross-right camera (e.g., 120-degree FOV). In one implementation, the camera 1868 may use a gigabit multimedia serial link (GMSL) interface (e.g., GMSL2) as input / output (I / O).

[0220] In some implementations, although Figure 18AAs not shown, vehicle 1800A may include an in-cabin occupant and / or driver monitoring system, which may include various sensors. For example, in-cabin sensors may include various cameras 1868, such as a driver monitoring camera (e.g., located in front of the driver's seat and facing the driver's seat at a 55-degree FOV), a front occupant monitoring camera (e.g., located in front of the front occupant seat and facing the front occupant seat at a 190-degree FOV), and a rear occupant monitoring camera (e.g., located in front of the rear occupant seat and facing the rear occupant seat at a 190-degree FOV). Similar to external cameras 1868, in this embodiment, internal cameras 1868 may use a GMSL (e.g., GMSL2) interface for I / O.

[0221] As another non-limiting example, vehicle 1800A may also include nine RADAR sensors 1860. For example, vehicle 1800A may include a front center imaging RADAR sensor (e.g., 120-degree FOV or sensing field), a left front corner RADAR sensor (e.g., 160-degree FOV or sensing field), a right front corner RADAR sensor (e.g., 160-degree FOV or sensing field), a right rear corner RADAR sensor (e.g., 160-degree FOV or sensing field), a left RADAR sensor (e.g., 160-degree FOV or sensing field), a right RADAR sensor (e.g., 160-degree FOV or sensing field), a left rear RADAR sensor (e.g., 50-degree FOV or sensing field), and a right rear RADAR sensor (e.g., 50-degree FOV or sensing field). In an embodiment, the RADAR sensors 1860 may use an Ethernet interface as I / O.

[0222] As a non-limiting example, the vehicle 1800A may also include twelve ultrasonic sensors 1862. Figure 18A As shown, the ultrasonic sensor can be placed along the front and rear bumpers of vehicle 1800A and along the sides of vehicle 1800A, and can be used to detect objects (static and dynamic) close to vehicle 1800A. In some embodiments, the ultrasonic sensor 1862 can use a DS13 interface as I / O.

[0223] As a non-limiting example, vehicle 1800A may also include a LiDAR sensor 1864, such as a front-center LiDAR sensor (e.g., a 120-degree horizontal FOV or sensing field and a 30-degree vertical FOV or sensing field). In some embodiments, such as when using additional or alternative LiDAR sensors, the LiDAR sensor may have different horizontal and vertical fields of view or sensing fields. For example, LiDAR sensor 1864 may include a 360-degree horizontal FOV or sensing field (e.g., located in a rotating LiDAR sensor) and a 90-degree vertical FOV or sensing field. In some embodiments, LiDAR sensor 1864 may use an Ethernet interface as I / O.

[0224] As a non-limiting example, the Autonomous Mobile Robot (AMR) 1800B may include three LiDAR sensors 1864. For example, the topmost LiDAR sensor 1864 may include a beam or 3D LiDAR sensor (e.g., a 360-degree horizontal and 90-degree vertical FOV or sensing field), and the front and rear LiDAR sensors may include planar or 2D LiDAR sensors (e.g., a 180-degree horizontal FOV or sensing field).

[0225] As a non-limiting implementation, the AMR 1800B may further include eight cameras 1868, such as a front stereo camera (e.g., 120-degree FOV), a rear stereo camera (e.g., 120-degree FOV), a left stereo camera (e.g., 120-degree FOV), a right stereo camera (e.g., 120-degree FOV), a front fisheye camera (e.g., 202-degree ± 3-degree FOV), a rear fisheye camera (e.g., 202-degree ± 3-degree FOV), a left fisheye camera (e.g., 202-degree ± 3-degree FOV), and a right fisheye camera (e.g., 202-degree ± 3-degree FOV).

[0226] The AMR 800B may also include a charging port, charging port contacts, status indicators, one or more (e.g., four) RGB LEDs, one or more IMU sensors 1866, a magnetometer, and a barometer. The AMR 800B is capable of high-precision time synchronization between sensors using hardware timestamps and PTPs over Ethernet with sensor acquisition times of less than 10 microseconds. In one implementation, the AMR 1800B provides simultaneous camera capture within 100 microseconds of a single hardware trigger on all cameras 1868, and can write sensor captures to disk at a rate of 4 GB / s for writing packets (e.g., writing to ROSbags of the Robot Operating System (ROS)). Therefore, the AMR 1800B is capable of running ROS (e.g., NVIDIA's Isaac ROS), can be remotely operated (as described herein), can map the environment, and can navigate the environment using vision cameras 1868, LiDAR 1864, and / or other sensor types or modalities.

[0227] The humanoid robot 1800C may include (as a non-limiting example) a LiDAR sensor 1864. For example, the LiDAR sensor 1864 may include a beam or a 3D LiDAR sensor (e.g., a 360-degree horizontal and 90-degree vertical FOV or sensing field), or it may include a planar or 2D LiDAR sensor (e.g., a 180-degree horizontal FOV or sensing field).

[0228] As a non-limiting implementation, the humanoid robot 1800C may also include four cameras 1868, such as a front stereo camera (e.g., 120-degree FOV), a rear stereo camera (e.g., 120-degree FOV), a front fisheye camera (e.g., 202-degree ± 3-degree FOV), and a rear fisheye camera (e.g., 202-degree ± 3-degree FOV).

[0229] As a non-limiting implementation, the humanoid robot 1800C may also include four ultrasonic sensors 1862, such as a left arm ultrasonic sensor, a right arm ultrasonic sensor, a left leg ultrasonic sensor, and a right leg ultrasonic sensor.

[0230] The humanoid robot 1800C may also include any number of actuators, such as those allowing control and manipulation of joints. For example, the humanoid robot 1800C may include actuators that allow for various degrees of freedom (DoF) depending on the design. In a non-limiting embodiment, the humanoid robot 1800C may have a total of 40 degrees of freedom (DoF) (e.g., 6DoF x2 for arms, 6DoF x2 for hands, 6DoF x2 for legs, 2DoF for the torso, and 2DoF for the neck). Actuators can convert energy into physical motion, thereby allowing actions such as joint movement, locomotion, and grasping / manipulation. For example, motors and servos can be used to perform joint movements to control the rotation of joints in an arm or manipulator, allowing for reaching, grasping, and manipulating objects. Locomotion can be achieved by moving through the environment using wheels, tracks, or other mobility devices (robot legs). Grasping and manipulation can be performed using end effectors or hands / fingers, which can be equipped with actuators to grasp objects, apply forces, and perform specific tasks. In some examples, the humanoid robot 1800C may include position and orientation sensors, such as encoders, gyroscopes, etc., to determine the robot 1800C's position in space, thereby enabling position determination and motion tracking. In implementations, the humanoid robot 1800C may include force and pressure sensors to detect environmental interactions, enabling the robot 1800C to grasp objects with appropriate force and avoid obstacles along its path. Perception sensors (e.g., cameras, LiDAR, RADAR, ultrasound, sonar (SONAR), etc.) may be used in conjunction with tactile sensors to enable the robot 1800C to perceive objects, shapes, and textures, and to know when to begin and stop touching (in conjunction with force sensors that regulate the force used during touching). As a non-limiting example, the humanoid robot 1800C may have a height of approximately 1-2 meters (e.g., 1.7 meters or 5 feet 6 inches), a weight of 50-70 kilograms, be able to move at speeds of 8 km / h or higher, and be able to carry a payload of 20-100 kilograms, depending on the system design and requirements.

[0231] In implementations, the humanoid robot 1800C may include a dialogue system—such as a dialogue system driven by a language model (e.g., LLM, VLM, MMLM, VLA, etc.)—to help understand the environment, reason, and communicate with humans, animals, devices, and / or other robots, and / or make planning, control, and navigation decisions. Therefore, in addition to performing various tasks, the humanoid robot 1800C can also use onboard sensors, microphones, and speakers to understand speech, audio, and visual cues, while also being able to communicate with the environment.

[0232] Referring to camera 1868 of machine 1800, the camera type of camera 1868 may include, but is not limited to, digital cameras suitable for components and / or systems of machine 1800. For vehicle 1800a implementation, camera 1868 may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may have any image capture rate, such as 30 frames per second (fps), 60 fps, 120 fps, 240 fps, etc., depending on the implementation. The camera may use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-neutral-neutral-colorless (RCCC) color filter array, a red-neutral-neutral-blue (RCCB) color filter array, a red-blue-green-neutral (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some implementations, colorless pixel cameras (e.g., cameras with RCCC, RCCB, and / or RBGC color filter arrays) can be used to improve light sensitivity.

[0233] The field of view includes cameras (e.g., front-facing cameras) in front of the machine 1800, which can be used for surround view to help identify the path and obstacles ahead, and, with the help of one or more controllers 1836 and / or control SoCs, provide information crucial for generating an occupancy grid and / or determining preferred machine movement, trajectory, and / or path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including lane departure warning (“LDW”), adaptive cruise control (“ACC”), and / or other functions such as traffic sign recognition.

[0234] Various cameras can be used in front-end configurations, including, for example, monocular camera platforms that include complementary metal-oxide-semiconductor (“CMOS”) color imagers. Another example could be the wide-angle camera 1868B, which can be used to perceive objects entering the field of view from the periphery (e.g., pedestrians, warehouse vehicles, other robots, pedestrian traffic, or bicycles). Furthermore, any number of long-range cameras 1868E (e.g., a pair of long-range stereo cameras) can be used for depth-based object detection, especially for objects for which neural networks have not yet been trained. The long-range camera 1868E can also be used for object detection and classification, as well as basic object tracking.

[0235] Any number of stereo cameras 1868A can also be included in front-mounted and / or other (e.g., rear-mounted) configurations. In at least one embodiment, one or more stereo cameras 1868A may include an integrated control unit that includes a scalable processing unit that can provide programmable logic (“FPGA”) and a multi-core microprocessor with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the machine 1800 environment, including distance estimates of midpoints in the image (e.g., parallax or depth images). Alternative stereo cameras 1868A may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip that can measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1868A may be used in addition to those described herein, or as an alternative to those described herein. For example, in some implementations, stereo depth estimation can be performed using cameras other than stereo cameras (e.g., two monocular cameras with at least partially overlapping fields of view).

[0236] Cameras with a field of view including portions of the side environment of machine 1800 (e.g., side-view cameras) can be used, for example, for surround view, providing information for creating and updating occupancy grids, and generating side collision warnings and / or indicating to AMR1800B or humanoid robot 1800C, for example, the presence of objects, features, and / or people on the side. For example, a surround camera 1868D can be mounted on machine 1800. The surround camera 1868D can include a wide-angle camera 1868B, a fisheye camera, a 360-degree camera, etc. For example, four fisheye cameras can be mounted on the front, rear, and sides of machine 1800. In an alternative arrangement, machine 1800 can use three surround cameras 1868D (e.g., left, right, and rear), and can utilize one or more other cameras (e.g., a front-facing camera) as a fourth surround-view camera.

[0237] A camera 1868 (e.g., a rear-view camera) having a field of view including a portion of the environment behind the machine 1800 can be used to understand objects, features, people, and / or other information behind the machine 1800, such as for parking assistance, surround view, rear collision warning, planning, control, and navigation determination, and / or to create and update occupancy grids, BEV images representing the environment, height maps, etc. A wide variety of cameras 1865 can be used, including but not limited to those also suitable for use as front cameras (e.g., long-range and / or mid-range cameras 1868E, stereo cameras 1868A, infrared cameras 1868C, etc.), rear cameras, side cameras, downward cameras, upward cameras, and / or similar cameras 1868 as described herein.

[0238] Similarly, for LiDAR sensor 1864, RADAR sensor 1860, ultrasonic sensor 1862 and / or other sensor modes or types, the location and placement of the sensors and their corresponding fields of view or sensing fields can be determined based on the use case, implementation or design of the particular machine 1800.

[0239] For example, machine 1800 includes a RADAR sensor 1860, which can be used by machine 1800 for long-range object detection, even in dark and / or inclement weather conditions. In implementations, the RADAR functional safety level can be ASIL B. RADAR sensor 1860 can be controlled and accessed for object tracking data using CAN and / or bus 1802 (e.g., for transmitting data generated by RADAR sensor 1860), and in some examples, raw data can be accessed using Ethernet. Various types of RADAR sensors can be used. For example, but not limited to, RADAR sensor 1860 can be suitable for front, rear, and side radar applications. In some examples, a pulse Doppler RADAR sensor is used.

[0240] The RADAR sensor 1860 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range lateral coverage, etc. In some examples, long-range radar can be used for adaptive cruise control (ACC) functions. Long-range RADAR systems can provide a wide field of view, such as within 250m, achieved through two or more independent scans. The RADAR sensor 1860 can help distinguish between static and moving objects; ADAS systems can use it for emergency braking assistance and forward collision warning, and robots can use it to detect dynamic objects in various environments (e.g., low-light or no-light environments). Long-range RADAR sensors can include a single static multimodal RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record the machine 1800's surroundings at higher speeds while minimizing peripheral interference (e.g., traffic from adjacent lanes). The other two antennas expand the field of view, enabling rapid detection of objects entering or leaving the machine's direct path (e.g., lanes).

[0241] Mid-range RADAR systems can include, for example, a range of up to 960m (front) or 50m (rear), and a field of view of up to 42 degrees (front) or 150 degrees (rear). Short-range RADAR systems can include, but are not limited to, RADAR sensors designed to be mounted at both ends of a side surface (e.g., a rear bumper), allowing continuous monitoring of blind spots behind and beside a machine (e.g., a vehicle, robot, etc.) using two beams. Therefore, short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0242] Machine 1800 may also include an ultrasonic sensor 1862. The ultrasonic sensor 1862 may be located at the front, rear, and / or sides of machine 1800 and can be used to assist near-field sensing, such as for parking assistance, collision avoidance (e.g., for robot parts), and / or to create and update occupancy grids, evidence mesh maps (EGMs), height maps, BEV images, and / or other representations of objects and features in the machine 1800 environment. Multiple ultrasonic sensors 1862 may be used, and different ultrasonic sensors 1862 may be used for different detection ranges (e.g., 2.5 m, 4 m). For example, the ultrasonic sensor 1862 may operate at the ASIL B functional safety level.

[0243] Machine 1800 may include a LiDAR sensor 1864. The LiDAR sensor 1864 can be used for object and feature detection, pedestrian and other robot detection, emergency braking, collision avoidance, simultaneous localization and mapping (SLAM), free space detection, and / or other functions. In some implementations, the LiDAR sensor 1864 may be of functional safety level ASIL B. In some examples, machine 1800 may include multiple LiDAR sensors 1864 (e.g., two, four, six, etc.), which can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0244] In some examples, the LiDAR sensor 1864 may be able to provide a 360-degree field of view of objects and their distances. For example, a commercially available LiDAR sensor 1864 may have a advertised range of approximately 900m, an accuracy of 2cm-3cm, and support for 900Mbps Ethernet connectivity. In some examples, one or more non-protruding LiDAR sensors 1864 may be used. In such examples, the LiDAR sensor 1864 can be implemented as a small device that can be embedded in the front, rear, side, top, and / or corner of the machine 1800. In such examples, the LiDAR sensor 1864 can provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, with a range of up to 200m even for low-reflectivity objects. The front-mounted LiDAR sensor 1864 can be configured with a horizontal field of view between 45 and 135 degrees.

[0245] In some examples, LiDAR technology, such as 3D flash LiDAR, can also be used. 3D flash LiDAR uses a laser flash as a transmission source, illuminating the environment around a vehicle up to approximately 200m away. The flash LiDAR unit includes a receiver that records the laser pulse transmission time and the reflected light on each pixel, which in turn corresponds to the distance from the vehicle to the object. Flash LiDAR allows for the generation of highly accurate and distortion-free images of the surrounding environment with each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of the machine 1800. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras with no moving parts other than a fan (e.g., non-scanning LiDAR devices). Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of 3D distance point clouds and co-registration intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 1864 may be less susceptible to motion blur, vibration, and / or shock.

[0246] Figure 18BThis is an illustration of the location of sensors and components of an example autonomous or semi-autonomous vehicle 1800A (also referred to herein as "vehicle 1800", "self-vehicle 1800", "self-machine 1800", or "machine 1800") according to some embodiments of this disclosure. While vehicle 1800A is shown in the figure, this is not intended to be limiting, and similar components and / or sensors can be included on any other machine type without departing from the scope of this disclosure. For example, similar sensors and / or components can be used in vehicles, cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police cars, ambulances, ships, construction vehicles, underwater vehicles, robots (e.g., AMRs, humanoid robots, robotic arms, end effectors, forklifts, etc.), drones, aircraft, vehicles coupled to trailers (e.g., semi-trailers for hauling goods), and / or another type of vehicle or machine (e.g., driverless and / or capable of accommodating one or more passengers).

[0247] Figure 18CThis is a block diagram of an example system architecture for a machine 1800 (e.g., an autonomous or semi-autonomous vehicle 1800A, an autonomous mobile robot (AMR) 1800B, a humanoid robot 1800C, and / or other types of machines) according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are presented as examples only. Other arrangements, components, features, and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of the arrangements shown, and some elements may be omitted entirely. Furthermore, many of the arrangements, components, features, elements, etc., described herein are functional entities that can be implemented as discrete or distributed components or together with other components, and can be implemented in any suitable combination and location (e.g., on a local device, vehicle, or edge machine, in the field (e.g., a locally hosted server), in a remote location (e.g., in one or more computing or server devices in one or more data centers in the cloud) and / or other locations). The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be implemented using one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, microcontroller, embedded processor, digital signal processor (DSP), image signal processor (ISP), physical processing unit (PPU), field-programmable gate array (FPGA), accelerators (e.g., deep learning accelerator (DLA), deep learning accelerator cluster (XNN), neural network accelerator (NNA) and / or neural processing unit (NPU), programmable vision accelerator (PVA), optical flow accelerator (OFA), etc.), application-specific integrated circuit (ASIC), data processing unit (DPU), quantum processor, etc.) executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with... Figures 18A to 18E Example machine 1800 Figure 19 Example calculation of ecosystem 1900, Figure 20 Example of generating language model system 2000 and / or Figure 21 The example computing device 2100 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the same task.

[0248] Figure 18CEach component, feature, and system of machine 1800 is shown connected via bus 1802 (or referred to as "machine communication network 1802" or simply "communication network 1802"). Bus 1802 may include a Controller Area Network (CAN) data interface (or referred to herein as "CAN bus"). CAN can be a network within machine 1800 used to help control various features and functions of machine 1800, such as brake actuation, acceleration, braking, steering, windshield wipers, etc. CAN bus can be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button position, and / or other vehicle status indicators. CAN bus may conform to ASIL B standard. In some implementations, in addition to or as an alternative to the CAN bus, bus 1802 may include FlexRay, embedded buses (e.g., SPI, I2C), Local Interconnect Link (LIN), NVIDIA's NVLink, Ultra Accelerated Link (UALink), USB (2.0, 3.0 and above), radio frequency (RF), Ethernet (e.g., 10BASE / 100BASE, 1000BASE, 10G, etc.), and / or other communication protocols or functions. Furthermore, while a single line is used to represent bus 1802, this is not limiting. For example, there can be any number of buses 1802, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 1802 may be used to perform different functions and / or for redundancy. For example, a first bus 1802 may be used for collision avoidance, while a second bus 1802 may be used for actuation control. In any example, each bus 1802 can communicate with any component of machine 1800, and two or more buses 1802 can communicate with the same component. In some examples, each SoC 1804, each controller 1836, and / or each computer or computing engine within machine 1800 can access the same input data (e.g., input from sensors of machine 1800) and can be connected to a common bus, such as a CAN bus.

[0249] Machine 1800 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, batteries, side mirrors, and / or other parts of the vehicle or machine. Machine 1800 may include a propulsion system 1850, such as an internal combustion engine, a hybrid power plant, an all-electric motor, a hydrogen fuel cell engine, and / or another type of propulsion system. Propulsion system 1850 may be connected to the drivetrain of machine 1800, which may include a transmission to enable propulsion of machine 1800. Propulsion system 1850 may be controlled in response to a signal received from throttle / accelerator 1852.

[0250] Steering system 1854 may include a steering wheel and / or other steering mechanisms (e.g., remote steering and / or local steering) for steering machine 1800 (e.g., along a desired path or route) while propulsion system 1850 is in operation (e.g., when the vehicle is traveling). Steering system 1854 may receive signals from steering actuator 1856. In some embodiments, a steering wheel or other steering mechanism may not be included, for example, for machine 1800 capable of fully automated (e.g., level 5) functionality.

[0251] The brake sensor system 1846 can be used to operate the vehicle brakes in response to signals received from the brake actuator 1848 and / or the brake sensor.

[0252] Machine 1800 may include one or more controllers 1836, such as those described herein. Figure 18AThe controller 1836 is described. It can be used for a variety of functions and can be coupled to any of the various other components and systems of the machine 1800. For example, the controller 1836 can be used to control the machine 1800, artificial intelligence performed on the machine 1800, infotainment on the machine 1800, etc. For example, one controller 1836 can be used for some or all of the functions, or different controllers 1836 can be used for different functions, for example, to ensure availability and safety separation between various controllers for different tasks. For example, the controller 1836 can use a system-calculated plan (e.g., the path or trajectory of vehicle 1800A or AMR 1800B, or the movement, component trajectory, motion position or displacement, etc. of joints or components (e.g., manipulators, end effectors, limbs, hands, fingers, legs, feet, etc.) of humanoid robot 1800C to control the machine 1800 in the environment. In some cases, the controller 1836 may include a proportional-integral-derivative (PID) controller, a fuzzy logic controller, a neural controller (e.g., a controller embodied as one or more neural networks), a force control controller, a programmable logic controller (PLC), and / or other types of controllers. For example, in the humanoid robot 1800C, the controller 1836 can act as the brain, responsible for analyzing sensor data, making decisions, and sending commands to the actuators. The controller 1836 may include a low-level controller that handles basic motor control, ensuring accurate and precise movement of the individual joints and actuators. The controller 1836 may also include a high-level controller to coordinate multiple actuators and sensors, plan complex movements, and adapt to constantly changing environments.

[0253] In some implementations, controller 1836 may include an artificial intelligence controller that can use AI algorithms (e.g., DNN, MLM, etc.) to learn, make decisions, and autonomously execute tasks of machine 1800. In some implementations, controller 1836 may use a fixed open-loop control algorithm and not adjust its actions based on the environment. In other implementations, closed-loop control may be used, incorporating feedback mechanisms to monitor the robot's performance and make necessary adjustments. In examples, controller 1836 may implement reactive control to directly respond to sensor inputs, enabling rapid reflexive actions and real-time changes. Furthermore, in some examples, deliberate control may be implemented, using internal models and planning algorithms to generate advanced actions, which may be suitable for complex tasks requiring reasoning, decision-making, and long-term planning.

[0254] The controller 1836 may include one or more system-on-chip (SoC) 1804 ( Figure 18C and Figure 18DThe controller 1836 may provide signals (e.g., signals representing commands or messages) to one or more components and / or systems of the machine 1800, including components such as CPUs, GPUs, and accelerators. Although the controller 1836 is listed separately from the SoC 1804, this is not intended to be limiting, and in some implementations, one or more components of the SoC 1804 may perform the operations of the controller 1836. For example, the controller may send signals to operate machine brakes via one or more brake actuators 1845, to operate steering system 1854 via one or more steering actuators 1856, to operate propulsion system 1850 via one or more throttle / accelerators 1852, and so on. The controller 1836 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous or semi-autonomous navigation and movement and / or assist human operators using the machine 1800. Controller 1836 may include a first controller 1836 for autonomous control and navigation functions, a second controller 1836 for functional safety functions, a third controller 1836 for artificial intelligence functions (e.g., computer vision), a fourth controller 1836 for infotainment functions, a fifth controller 1836 for redundancy in emergency situations, and / or other controllers. For example, hardware for safety monitoring and other safety functions (e.g., functional safety islands) may be discrete or partitioned (physically or by processing separation) relative to hardware for processing sensor data to make perception and vehicle control decisions. Similarly, hardware for controlling in-vehicle infotainment and / or in-cabin monitoring (e.g., controllers, SOCs, etc.) may be separate or independent from hardware for vehicle perception and control. In some examples, a single controller 1836 may handle two or more of the above functions, two or more controllers 1836 may handle a single function, and / or any combination thereof.

[0255] The controller 1836 may provide signals for controlling one or more components and / or systems of the machine 1800 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data can be received from, for example, but not limited to, Global Navigation Satellite System (“GNSS”) sensor 1858 (e.g., Global Positioning System sensor), RADAR sensor 1860, ultrasonic sensor 1862, LiDAR sensor 1864, Inertial Measurement Unit (IMU) sensor 1866 (e.g., accelerometer, gyroscope, magnetic compass, magnetometer, etc.), microphone 1896, camera 1868 (e.g., stereo camera 1868A, wide-angle camera 1865B (e.g., fisheye camera), infrared camera 1868C, surround camera 1868D (e.g., 360-degree camera), long-range and / or medium-range camera 1868E and / or other types of cameras), speed sensor 1844 (e.g., for measuring the speed of machine 1800), vibration sensor 1842, steering sensor 1840, braking sensor (e.g., as part of braking sensor system 1846), actuators and / or other types of sensors.

[0256] One or more controllers 1836 may receive input (e.g., represented by input data) from the instrument panel 1832 of the machine 1800 and provide output (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 1834 (e.g., screen, head-up display, mirror display, face display, robot display, etc.), sound alarms, loudspeakers, speakers, and / or via other components of the machine 1800. Outputs may include parameters such as machine speed, rate, time, and... Figure 18C The system can display map data corresponding to map 1822 (e.g., from a navigation map, a standard definition (SD) map, a high definition (“HD”) map, etc.), location data (e.g., the location of machine 1800, such as its location on map 1822), direction, the location of other vehicles (e.g., occupancy map, elevation map, bird's-eye view (BEV) image, grid, etc.), information about objects perceived by the system and their status, system status information, and so on. For example, HMI display 1834 can display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.), and / or information about driving maneuvers that the vehicle has performed, is performing, or will perform (e.g., changing lanes now, exiting from exit 34B in two miles, etc.).

[0257] Machine 1800 may include one or more System-on-a-Chip (SoC) 1804 (in Figure 18D(described in more detail below). SoC 1804 may include CPU 1806, GPU 1805, processor 1810, cache 1812, accelerator 1814, data storage 1816, and / or other components and features. SoC 1804 can be used to process and provide data for various operations, such as navigation, planning, reasoning, inference, perception, control, and / or actuation operations of machine 1800 on various platforms and systems. For example, SoC 1804 can process real-time perception data (e.g., from cameras, LiDAR, RADAR, ultrasound, etc.) and map data corresponding to one or more maps 1822 (e.g., HD maps, SD maps, navigation maps, occupancy maps, etc.) to perform or assist in performing various operations of machine 1800. When using maps and / or AI, maps and / or AI (e.g., model parameter updates, fine-tuning, etc.) are transmitted via network interface 1824 from one or more servers (e.g., [server name missing]). Figure 18E Refresh and / or update the server 1875 (e.g., one or more servers in a cloud-based data center).

[0258] Despite Figures 18A to 18E The SoC 1804 is illustrated; however, additional or alternative components and / or architectures, such as multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiPs), field-programmable gate arrays (FPGAs), heterogeneous integration (HI), and single-board computers (SBCs), may be used without departing from the scope of this disclosure. For example, depending on the type of machine 1800, the purpose of machine 1800, the model of machine 1800, and the capabilities required by machine 1800, one or more SoC 1804s and / or alternative architectures and / or components may be used to meet a particular implementation.

[0259] Machine 1800 may include CPU 1818 (e.g., a discrete CPU or dCPU) which may be coupled to SoC 1804 via a high-speed interconnect (e.g., PCIe). CPU 1818 may include, for example, an x86 processor. CPU 1818 may be used to perform any of a variety of functions, including arbitrating potentially inconsistent results between ADAS sensors and SoC 1804, and / or monitoring the status and health of controller 1836 and / or infotainment SoC 1830.

[0260] Machine 1800 may include GPU 1820 (e.g., a discrete GPU or dGPU) which may be coupled to SoC 1804 via a high-speed interconnect (e.g., NVIDIA's NVLink, Ultra Accelerated Link (UALink), etc.). GPU 1820 can provide additional artificial intelligence capabilities, such as by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based on inputs from sensors (e.g., sensor data) from Machine 1800.

[0261] The machine 1800 may also include a network interface 1824, which may include one or more wireless antennas 1826 and / or modems (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 1824 can be used to establish wireless connectivity with the cloud (e.g., with server 1878 and / or other network devices) via the Internet, and with other vehicles and / or computing devices (e.g., passenger client devices). For communication with other vehicles, direct links and / or indirect links (e.g., across networks and via the Internet) can be established between the two vehicles. A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide the machine 1800 with information about vehicles near the machine 1800 (e.g., vehicles in front, to the side, and / or behind the machine 1800). This functionality can be part of the machine 1800's cooperative adaptive cruise control function.

[0262] Network interface 1824 may include a System-on-Chip (SoC) that provides modulation and demodulation functions and enables controller 1836 to communicate over a wireless network. Network interface 1824 may include a radio frequency (RF) front-end for up-conversion from baseband to radio frequency (RF) and down-conversion from RF to baseband. Frequency conversion can be performed through well-known processes and / or using superheterodyne processes. In some examples, the RF front-end functionality may be provided by a separate chip. For example, network interface 1824 may be able to communicate via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multicarrier (“CDMA2000”), 5G, 6G, and / or other cellular and / or wireless communication standards. The wireless antenna 1826 can also enable communication between objects in the environment (such as vehicles, mobile devices, etc.) using local area networks (such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc.) and / or low power wide area networks (“LPWAN”) (such as LoRaWAN, SigFox, etc.).

[0263] Machine 1800 may also include data memory 1828, which may include off-chip (e.g., outside of SoC 1804) storage. Data memory 1828 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, Flash, hard disk and / or other components and / or devices capable of storing at least one bit of data.

[0264] Machine 1800 may also include a GNSS sensor 1858. The GNSS sensor 1858 (e.g., a GPS, an auxiliary GPS sensor, a differential GPS (DGPS) sensor, etc.) is used to assist in mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 1855 can be used, such as, but not limited to, GPS sensors using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0265] Machine 1800 may also include an IMU sensor 1866. In some examples, the IMU sensor 1866 may be located at the center of the rear axis of machine 1800. The IMU sensor 1866 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 1866 may include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 1866 may include an accelerometer, a gyroscope, and a magnetometer.

[0266] In some implementations, the IMU sensor 1866 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. Therefore, in some examples, the IMU sensor 1866 can enable the machine 1800 to estimate its heading by directly observing and correlating velocity changes from GPS to the IMU sensor 1866, without requiring input from a magnetic sensor. In some examples, the IMU sensor 1866 and the GNSS sensor 1855 can be combined in a single integrated unit.

[0267] The vehicle may include one or more microphones 1896 placed inside and / or around the machine 1800. The microphones 1896 may be used for emergency vehicle detection and identification, etc.

[0268] Machine 1800 may also include vibration sensor 1842. Vibration sensor 1842 can measure vibrations of machine parts, such as the arm or leg of humanoid robot 1800C, or the axle of vehicle 1800A or AMR 1800B. For example, changes in vibration may indicate changes in roads, walking, or traversable surfaces. In another example, when two or more vibration sensors 1842 are used, differences between vibrations can be used to determine friction or slippage on surfaces (e.g., when the vibration difference is between an electrically driven shaft and a freely rotating axle).

[0269] Machine 1800 may include ADAS system 1838, for example, when machine 1800 is vehicle 1800A. In some examples, ADAS system 1838 may include a dedicated SoC. ADAS system 1838 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision or collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), blind spot monitoring (BSM), rear cross traffic warning (RCTW), pedestrian detection, driver monitoring, collision warning system (CWS), traffic sign recognition, speed limit detection, automatic parking, lane centering (LC), high beam safety system and / or other features and functions.

[0270] Machine 1800 may also include an infotainment SoC 1830 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include one or more discrete components, such as multi-chip modules (MCMs), application-specific integrated circuits (ASICs), system-in-package (SiP), heterogeneous integration (HI), single-board computers (SBCs), etc. The infotainment SoC 1830 may include a combination of hardware and software that can be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., television, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., wireless, Wi-Fi, e...

Claims

1. A system comprising: One or more processors, said one or more processors being used for: The local memory region in the local memory of the hardware accelerator is allocated via the firmware interface and bound to an external memory region. The allocation returns a device pointer corresponding to the binding, which is tracked using a resource identifier in a resource table corresponding to a non-privileged execution context. as well as The device pointer is provided to a memory region accessible by the non-privileged execution context for at least one subsequent construction of at least one command buffer that identifies the binding from the local memory region to the external memory region.

2. The system of claim 1, wherein the one or more processors are configured to: The at least one command buffer is constructed in the external memory region of the hardware accelerator, the at least one command buffer including at least one command that identifies the device pointer corresponding to the binding from the local memory region to the external memory region.

3. The system of claim 2, wherein the one or more processors are used for: This enables the direct memory access DMA controller to execute the at least one command via the local memory region using the external memory region using the at least one command buffer, wherein the at least one command corresponds to at least one of the following: (i) reading data from the external memory region into the local memory region, or (ii) writing data from the local memory region into the external memory region.

4. The system of claim 1, wherein the allocation comprises: The firmware interface provides a request to allocate the local memory region in the local memory of the hardware accelerator and bind the local memory region to the external memory region. as well as Receive the device pointer corresponding to the binding from the local memory region to the external memory region via the firmware interface; The request is generated as a firmware-level configuration command by a privileged execution context using the Firmware Access Application Programming Interface (API), and the firmware-level configuration command identifies the binding between the local memory region and the external memory region.

5. The system of claim 1, wherein the at least one command buffer includes descriptor data identifying the device pointer, and wherein the descriptor data is updated prior to execution to identify a memory access path from the external memory region via the local memory region, and wherein the memory access path corresponds to a patched physical address in the local memory region identified for the execution of the at least one command.

6. The system of claim 1, wherein the device pointer corresponds to an address indirection, the address indirection including the binding from the local memory region to the external memory region, the address indirection being identified by a DMA controller to execute at least one memory access command during execution.

7. The system of claim 1, wherein the one or more processors are configured to: A second command buffer is constructed or received in the external memory region of the hardware accelerator, the second command buffer including at least one second command identifying a resource corresponding to the local memory region; The commit information of the second command buffer is provided to the commit queue corresponding to the non-privileged execution context; Provide a submission indicator corresponding to the submission information to the signaling interface corresponding to the submission queue; as well as This causes at least one hardware component of the hardware accelerator to execute the second command buffer.

8. The system of claim 1, wherein the one or more processors are used for: Before allocating the local memory region, determine whether there is a previously allocated local memory region bound to the external memory region for the non-privileged execution context; In response to determining the existence of the previously allocated local memory region, increment the reference count corresponding to the binding; as well as In response to determining that the previously allocated local memory region does not exist, a request is sent to the firmware interface to allocate the local memory region in the local memory of the hardware accelerator bound to the external memory region according to at least one access policy.

9. The system of claim 8, wherein the one or more processors are configured to: In response to a release request, decrement the reference count corresponding to the binding; and The local memory region is released in response to determining that the reference count satisfies the release condition.

10. The system of claim 1, wherein the one or more processors are configured to: A refresh operation is initiated in response to a refresh request to transfer data from the local memory region to the external memory region, wherein the refresh operation occurs in response to a user context switch; or In response to an invalidation request used to trigger the reinitialization of the local memory region during subsequent access, an invalidation operation is initiated to mark the data stored in the local memory region as invalid. In response to the user context switch, the local memory region is initialized with data from the external memory region and the local memory region is refreshed to the external memory region, and the refresh can also be performed in response to an explicit request.

11. The system of claim 1, wherein the binding is established according to at least one access policy, the at least one access policy including a first policy for initializing the local memory region with data from the external memory region, or a second policy for flushing data from the local memory region to the external memory region during release.

12. The system of claim 1, wherein the firmware interface maintains at least two concurrent bindings between multiple local memory regions and corresponding multiple external memory regions for the non-privileged execution context.

13. The system of claim 1, wherein the firmware interface includes a compatibility layer configured to translate memory allocations and at least one access request from at least one software application of a version implementation for the hardware accelerator, the compatibility layer updating at least one request for the allocation to access the local memory region and the binding to the external memory region via the firmware interface.

14. The system of claim 1, wherein the at least one command buffer includes descriptor data identifying the device pointer, and wherein the descriptor data is modified by firmware to identify the source address and destination address of a DMA transaction performed by the DMA controller, the source address or destination address corresponding to the local memory region and the external memory region bound by the device pointer, the descriptor data including transfer size and at least one access policy.

15. The system of claim 1, wherein the one or more processors are configured to perform operations including: Systems for managing memory persistence using firmware-level resource tracing; A system for binding one or more local memory regions to one or more DRAM-enabled regions via the firmware interface; A system for using one or more access policies to allocate one or more memory regions; A system for inserting one or more refresh or invalidation operations based on DRAM persistence; A system used to store one or more device pointers for reuse in a command buffer; A system that implements one or more multimodal language models (MMLMs); A system that implements one or more large language model LLMs; A system that implements one or more small language models (SLMs); A system that implements one or more Visual Language Models (VLMs); A system for generating synthetic data; Systems for generating synthetic data using AI; Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system used to perform remote operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing conversational AI operations; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

16. A system-on-a-chip (SoC), comprising: Processing circuit, the processing circuit being used for: Send a request via firmware interface to allocate a local memory region in the local memory of the hardware accelerator and bind the local memory region to an external memory region; The firmware interface receives a device pointer corresponding to the binding from the local memory region to the external memory region, the binding being tracked using a resource identifier in a resource table corresponding to a non-privileged execution context; as well as The device pointer is stored in a memory region accessible by the non-privileged execution context for at least one subsequent construction of at least one command buffer that identifies the binding from the local memory region to the external memory region.

17. A method comprising: One or more processors allocate a local memory region in the local memory of a hardware accelerator to an external memory region via a firmware interface. The allocation returns a device pointer corresponding to the binding, which is tracked using a resource identifier in a resource table corresponding to a non-privileged execution context. as well as The device pointer is provided by one or more processors to a memory region accessible by the non-privileged execution context for at least one subsequent construction of at least one command buffer that identifies the binding from the local memory region to the external memory region.

18. The method of claim 17, further comprising: The at least one command buffer is constructed by one or more processors in the external memory region of the hardware accelerator, the at least one command buffer including at least one command that identifies the device pointer corresponding to the binding from the local memory region to the external memory region.

19. The method of claim 18, further comprising: The one or more processors cause the at least one command to be executed by the direct memory access DMA controller via the local memory region using the at least one command buffer, wherein the at least one command corresponds to at least one of: (i) reading data from the external memory region into the local memory region, or (ii) writing data from the local memory region into the external memory region.

20. The method of claim 17, wherein the allocation comprises: The firmware interface provides a request to allocate the local memory region in the local memory of the hardware accelerator and bind the local memory region to the external memory region. as well as Receive the device pointer corresponding to the binding from the local memory region to the external memory region via the firmware interface; The request is generated as a firmware-level configuration command by a privileged execution context using the Firmware Access Application Programming Interface (API), and the firmware-level configuration command identifies the binding between the local memory region and the external memory region.

Images