Memory Allocation Using Graphs

Abstract

To provide apparatuses, systems, and techniques for generating one or more graph code nodes to allocate memory.SOLUTION: In at least one embodiment, one or more graph code nodes for allocating memory are generated on the basis of, for example, CUDA or other parallel computing platform code.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 214,205, entitled "MEMORY ALLOCATION USING GRAPHS," filed June 23, 2021, the entire disclosure of which is incorporated herein by reference. This application also relates to the complete disclosure of co-pending U.S. Patent Application No. _______________ (Attorney Docket No. 0112912-380US0), entitled "MEMORY DEALLOCATION USING GRAPHS," filed concurrently herewith, and incorporates by reference for all purposes.

[0002] At least one embodiment relates to a processing resource used to allocate memory using a data structure representing operations and dependencies between the operations. For example, at least one embodiment relates to a processor or computing system used to allocate memory using a data structure representing operations and dependencies between the operations that implements various novel techniques described herein. [Background technology]

[0003] Performing operations using data structures that represent dependencies between operations can often require allocated memory. However, in various cases, memory must be allocated outside of the data structures that represent dependencies between operations, which can require additional computing resources. Therefore, techniques for allocating memory using data structures that represent dependencies between operations can be improved using CUDA or other parallel computing platform code. [Brief explanation of the drawings]

[0004] [Figure 1] FIG. 1 illustrates an example of memory allocation in a graph, according to at least one embodiment. [Figure 2] FIG. 10 illustrates an example of activating a graph, according to at least one embodiment. [Figure 3] FIG. 1 illustrates an example of a graph and memory allocation, according to at least one embodiment. [Figure 4] FIG. 1 illustrates an example of a fork in a graph, according to at least one embodiment. [Figure 5] FIG. 10 illustrates an example of a block refcount array, according to at least one embodiment. [Figure 6] FIG. 1 illustrates an example of virtual address reservation, according to at least one embodiment. [Figure 7] FIG. 1 illustrates an example of address reuse, according to at least one embodiment. [Figure 8] FIG. 1 illustrates an example of physical memory sharing between graphs, according to at least one embodiment. [Figure 9] FIG. 1 illustrates an example process for allocating memory using a graph, according to at least one embodiment. [Figure 10] FIG. 1 illustrates an example process for deallocating memory using a graph, according to at least one embodiment. [Figure 11] FIG. 1 illustrates an exemplary data center, according to at least one embodiment. [Figure 12] FIG. 1 illustrates a processing system according to at least one embodiment. [Figure 13] FIG. 1 illustrates a computer system according to at least one embodiment. [Figure 14] FIG. 1 illustrates a system according to at least one embodiment. [Figure 15] FIG. 1 illustrates an exemplary integrated circuit according to at least one embodiment. [Figure 16] FIG. 1 illustrates a computing system according to at least one embodiment. [Figure 17] FIG. 1 illustrates an APU, according to at least one embodiment. [Figure 18] FIG. 1 illustrates a CPU according to at least one embodiment. [Figure 19] FIG. 2 illustrates an exemplary accelerator integration slice, according to at least one embodiment. [Figure 20A] FIG. 1 illustrates an exemplary graphics processor according to at least one embodiment. [Figure 20B] FIG. 1 illustrates an exemplary graphics processor according to at least one embodiment. [Figure 21A] FIG. 1 illustrates a graphics core according to at least one embodiment. [Figure 21B] FIG. 1 illustrates a GPGPU, according to at least one embodiment. [Figure 22A] FIG. 1 illustrates a parallel processor, according to at least one embodiment. [Figure 22B] FIG. 1 illustrates a processing cluster, according to at least one embodiment. [Figure 22C] FIG. 1 illustrates a graphics multiprocessor according to at least one embodiment. [Figure 23] FIG. 1 illustrates a graphics processor according to at least one embodiment. [Figure 24] FIG. 1 illustrates a processor according to at least one embodiment. [Figure 25] FIG. 1 illustrates a processor according to at least one embodiment. [Figure 26] FIG. 1 illustrates a graphics processor core according to at least one embodiment. [Figure 27] FIG. 1 illustrates a PPU, according to at least one embodiment. [Figure 28] FIG. 1 illustrates a GPC, according to at least one embodiment. [Figure 29] FIG. 1 illustrates a streaming multiprocessor according to at least one embodiment. [Figure 30]FIG. 1 illustrates a software stack for a programming platform according to at least one embodiment. [Figure 31] FIG. 31 illustrates a CUDA implementation of the software stack of FIG. 30, according to at least one embodiment. [Figure 32] FIG. 31 illustrates an ROCm implementation of the software stack of FIG. 30 according to at least one embodiment. [Figure 33] FIG. 31 illustrates an OpenCL implementation of the software stack of FIG. 30 according to at least one embodiment. [Figure 34] FIG. 1 illustrates software supported by a programming platform according to at least one embodiment. [Figure 35] FIG. 34 illustrates compiling code for execution on the programming platform of FIGS. 30-33 according to at least one embodiment. [Figure 36] FIG. 34 illustrates in more detail compiling code for execution on the programming platform of FIGS. 30-33, according to at least one embodiment. [Figure 37] FIG. 1 illustrates translating source code prior to compiling the source code, according to at least one embodiment. [Figure 38A] FIG. 1 illustrates a system configured to compile and execute CUDA source code using different types of processing units, according to at least one embodiment. [Figure 38B] FIG. 38B illustrates a system configured to compile and execute the CUDA source code of FIG. 38A using a CPU and a CUDA-enabled GPU, according to at least one embodiment. [Figure 38C] FIG. 38B illustrates a system configured to compile and execute the CUDA source code of FIG. 38A using a CPU and a non-CUDA-enabled GPU, according to at least one embodiment. [Figure 39]FIG. 38D illustrates an example kernel translated by the CUDA-to-HIP translation tool of FIG. 38C, in accordance with at least one embodiment. [Figure 40] FIG. 38D illustrates the non-CUDA GPU of FIG. 38C in more detail, according to at least one embodiment. [Figure 41] FIG. 41 illustrates how threads of an exemplary CUDA grid are mapped to different compute units of FIG. 40, according to at least one embodiment. [Figure 42] FIG. 1 illustrates how to migrate existing CUDA code to Data Parallel C++ code, according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0005] In at least one embodiment, one or more programming models utilize one or more data structures representing operations and dependencies between the operations to implement the operations. In at least one embodiment, a graph is a data structure representing operations and dependencies between the operations, and comprises at least nodes, also referred to as graph code nodes, which are data sets or data structures that encode information about the operations. In at least one embodiment, the techniques described herein may relate to graphs, but the techniques described herein are applicable to any suitable data structure of any suitable programming model that represents, encodes, or otherwise stores operations and / or dependencies between the operations. In at least one embodiment, the one or more programming models include models such as the Compute Unified Device Architecture (CUDA) model, the Heterogeneous Compute Interface for Portability (HIP) model, the oneAPI model, various hardware accelerator programming models, and / or variations thereof.

[0006] In at least one embodiment, the graph represents a sequence of operations performed by one or more devices, such as a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose GPU (GPGPU), a parallel processing unit (PPU), and / or variations thereof. In at least one embodiment, the graph encodes a sequence of operations, such as kernel launches, connected by dependencies. In at least one embodiment, the dependencies of the graph are defined separately from the execution of the graph. In at least one embodiment, the graph is defined once and can be launched one or more times on one or more devices.

[0007] In at least one embodiment, the graph depicts operations through the nodes of the graph, with each node of the graph corresponding to an operation and dependencies between operations forming the edges of the graph. In at least one embodiment, dependencies constrain the execution sequence of the operations. In at least one embodiment, an operation can be scheduled at any time once the node on which it depends has completed (e.g., the operation represented by the node has been executed / performed). In at least one embodiment, the operations represented by the nodes can include operations such as kernel, CPU function calls, memory management / manipulation operations, waiting for an event, recording an event, signaling an external semaphore, waiting for an external semaphore, and operations of other graphs (e.g., child graphs). In at least one embodiment, graphs are created and modified by one or more systems through various programming model application programming interface (API) functions. In at least one embodiment, operations of the graph are executed by one or more systems through various programming model API functions.

[0008] In at least one embodiment, one or more systems implement various operations and / or techniques described herein and include systems such as drivers, programming model libraries, and / or variations thereof, which may be associated with one or more programming models, such as CUDA, HIP, oneAPI, and / or variations thereof. In at least one embodiment, a driver, also referred to as a device driver, is a computer program that provides a software interface to one or more devices (e.g., GPUs). In at least one embodiment, the one or more systems provide functionality for allocating memory, freeing allocated memory, managing / utilizing allocated memory, and / or various other memory management / utilization operations using graphs. In at least one embodiment, one or more systems provide functionality for using the graph to perform various memory management / utilization operations with respect to one or more GPUs (e.g., allocating memory on one or more GPUs, freeing allocated memory on one or more GPUs, managing / utilizing allocated memory on one or more GPUs, and / or other suitable operations), which may be utilized to perform various operations of the graph. In at least one embodiment, one or more systems provide an API, where an API refers to a set of definitions, functions, and / or protocols for accessing various functionality.

[0009] In at least one embodiment, one or more systems associate memory with the graph through an explicit node creation interface (e.g., one or more APIs such as those described herein). In at least one embodiment, a graph code node for allocating memory is called a MemAlloc node or in any suitable notation, and a graph code node for deallocating memory is called a MemFree node or in any suitable notation. In at least one embodiment, a MemAlloc node performs one or more memory allocation operations that create an allocation and returns the address of the allocation for use. In at least one embodiment, a MemFree node performs one or more memory free operations that free an allocation. In at least one embodiment, to properly access an allocation in the graph, a task must be ordered after the MemAlloc node that created the allocation but before the MemFree node that frees it. In at least one embodiment, one or more systems provide one or more APIs for adding MemAlloc and / or MemFree nodes to the graph. In at least one embodiment, one or more systems associate memory with the graph through a stream capture interface, in which various stream-based API calls for allocating memory and freeing allocated memory are converted into MemAlloc and MemFree nodes, respectively.

[0010] In at least one embodiment, one or more systems provide access to allocations that are not freed in an allocating graph after the graph is executed until one or more users free the allocations by at least invoking the graph containing a MemFree node for the allocation and / or passing the allocation to one or more API calls to free the allocated memory (e.g., outside of capture). In at least one embodiment, one or more systems track the allocations so that various operations that reference allocated memory are properly validated and allowed to access the allocations, the allocations maintain ownership of the underlying physical memory they use, and / or the physical memory can be reused when freed.

[0011] In at least one embodiment, when a MemAlloc node is created, one or more systems attempt to reuse memory freed by a previous MemFree node. In at least one embodiment, one or more systems track a set of paths through the graph, each path being associated with a particular data structure. In at least one embodiment, a MemAlloc node can reuse memory from its own path without restriction, but when attempting to reuse memory from another path, the MemAlloc node can allocate only a subset of the memory based on when the two paths last diverged.

[0012] In at least one embodiment, one or more systems provide functionality for graphs to exclusively own the virtual memory they use for allocation, but share the physical memory used to back the virtual memory. In at least one embodiment, one or more systems provide functionality such that the total amount of memory allocated by all graphs can exceed the amount of memory present on the GPU.

[0013] In at least one embodiment, when the graph is instantiated (e.g., enabled), one or more systems determine a total memory footprint of the graph and represent the total memory footprint as a set of fixed-size virtual memory blocks. In at least one embodiment, at startup, the one or more systems map physical memory to each of these virtual memory blocks. In at least one embodiment, if a subsequent graph is launched in the same stream, the subsequent graph can reuse these physical blocks because the graphs will execute consecutively. In at least one embodiment, the one or more systems can remap the same physical memory block to several virtual memory blocks, which may be referred to as virtual aliasing.

[0014] In at least one embodiment, allocations allocated in one graph but freed in another graph utilize additional tracking so that the physical memory they use cannot be reused until the allocation is explicitly freed, and the contained allocations can be reused when the graph terminates. In at least one embodiment, available physical memory is associated by one or more systems with a set of events (e.g., graph completion and / or one or more API calls to free allocated memory). In at least one embodiment, when the graph starts, the graph must also wait for an event such that the graph has exclusive access to physical memory. In at least one embodiment, to reduce the amount of synchronization required, one or more systems manage physical memory on a per-stream basis. In at least one embodiment, two graphs running in the same stream can use the same memory because the graphs can be serialized. In at least one embodiment, two graphs running in different streams will use different memory because each stream maintains a separate cache of physical blocks, allowing the graphs to continue running concurrently.

[0015] In at least one embodiment, one or more systems provide functionality for graph-ordered memory allocation, also known as graph allocation. In at least one embodiment, one or more systems provide nodes for various graph memory allocation operations. In at least one embodiment, MemAlloc nodes, which allocate memory, and / or MemFree nodes, which free memory allocated by MemAlloc nodes, are collectively referred to as memory nodes. In at least one embodiment, one or more systems provide functionality for creating memory nodes using explicit API functions, stream capture, and / or variations thereof.

[0016] 1 illustrates an example 100 of memory allocation in a graph, according to at least one embodiment. In at least one embodiment, example 100 comprises a visual representation of the graph comprising nodes. In at least one embodiment, the graph represents various operations to be performed on one or more devices. In at least one embodiment, the graph, which may be referenced in relation to a programming model (e.g., CUDA, HIP, oneAPI, and / or variations thereof), is a directed acyclic graph or any suitable graph, whose nodes represent work and whose edges represent dependencies between each pair of work represented by the nodes connected by the edge.

[0017] In at least one embodiment, the graph represents operations utilizing various user-specified data and / or user-managed resources, also referred to as user objects or, more generally, objects. In at least one embodiment, objects include kernel arguments, host function arguments, workspace buffers, and / or other data utilized throughout the execution of one or more operations of the graph. In at least one embodiment, the graph is created through one or more operations referred to as stream capture. In at least one embodiment, stream capture encodes a workload represented by a stream into the graph. In at least one embodiment, a stream refers to a sequence of operations executing on a processing unit, such as a GPU, PPU, CPU, and / or variations thereof. In at least one embodiment, a stream capture sequence refers to a sequence of operations utilized to generate the graph using stream capture. In at least one embodiment, a capturing stream refers to a stream of one or more operations associated with stream capture (e.g., one or more operations to be captured in the graph). In at least one embodiment, one or more systems publish or otherwise provide the streams to processing units, which perform one or more operations on the streams. In at least one embodiment, a program can have multiple streams. In at least one embodiment, a stream can also wait for an event that can represent the completion of work in another stream.

[0018] In at least one embodiment, graphs are created and modified through various API functions, also referred to as APIs. In at least one embodiment, as an illustrative example, graphs are created through one or more API functions that create the graph, add nodes to the graph (e.g., child graph nodes, empty nodes, event record nodes, event wait nodes, external semaphore signal nodes, external semaphore wait nodes, host execute nodes, kernel execute nodes, memory copy nodes, memory set nodes, and / or variations thereof), add dependencies to the graph, and / or variations thereof. In at least one embodiment, various operations are performed by one or more systems on the graph, such as adding a node, deleting a node, modifying a node, copying a graph, clearing a graph, and / or variations thereof, through any suitable API function. In at least one embodiment, graphs are modified or otherwise managed by one or more systems in any suitable manner using any suitable API function, software library, and / or variations thereof.

[0019] In at least one embodiment, alloc102, alloc108, and alloc118 represent memory allocation operations. In at least one embodiment, alloc102, alloc108, and alloc118 are MemAlloc nodes. In at least one embodiment, one or more systems provide an API for generating one or more graph code nodes (e.g., alloc102, alloc108, and alloc118) for allocating memory, as described in further detail herein. In at least one embodiment, kernel 104, kernel 110, kernel 114, kernel 116, and kernel 120 represent kernel operations. In at least one embodiment, kernels are functions that run on one or more devices, such as GPUs. In at least one embodiment, kernel 104, kernel 110, kernel 114, kernel 116, and kernel 120 are nodes that represent execution of kernels. In at least one embodiment, Free 106, Free 112, and Free 122 represent memory deallocation operations, also referred to as memory free operations. In at least one embodiment, Free 106, Free 112, and Free 122 are MemFree nodes. In at least one embodiment, one or more systems provide an API to generate one or more graph code nodes (e.g., Free 106, Free 112, and Free 122) for deallocating memory, as described in further detail herein.

[0020] In at least one embodiment, as an example, the graph depicted in Figure 1 is created using stream capture, where one or more systems generate the graph by capturing one or more operations of the stream, and one or more nodes of the graph correspond to the one or more operations. In at least one embodiment, as an example, the graph depicted in Figure 1 is created using various API functions, where one or more systems generate the graph by utilizing one or more APIs for creating the graph and for adding one or more nodes to the graph, such as those depicted in Figure 1.

[0021] In at least one embodiment, one or more systems provide functionality for reusing memory within the graph. In at least one embodiment, when a MemAlloc node is created, the MemAlloc node attempts to reuse memory freed by a MemFree node on which it depends (e.g., potentially indirectly). In at least one embodiment, graph allocation may or may not be performed using runtime ordering information. In at least one embodiment, one or more systems select addresses for allocations at node creation time based at least on the topology of the graph. In at least one embodiment, the allocations and how they can be reused determine the memory footprint of a graph, which one or more systems allocate to the graph prior to execution.

[0022] In at least one embodiment, one or more systems back the memory footprint of the graph with physical memory before executing the graph. In at least one embodiment, the physical memory utilized for backing is owned by the invoking stream. In at least one embodiment, the one or more systems provide functionality for several graphs launched in the same stream (e.g., having only internally accessible memory allocations) to use the same physical memory because execution of items in the same stream is serialized. In at least one embodiment, the one or more systems support graph-ordered allocations, whose lifetimes extend beyond the graph for which the allocations were allocated.

[0023] In at least one embodiment, one or more systems utilize memory nodes in a graph and provide functionality for capturing stream-ordered API calls. In at least one embodiment, there are two types of graph allocations, which may depend on how one or more users allocate / free memory. In at least one embodiment, intra-graph allocations refer to allocations where the MemAlloc node and the MemFree node exist in the same graph. In at least one embodiment, inter-graph allocations refer to allocations where the MemAlloc node does not have a corresponding MemFree node in the same graph. In at least one embodiment, a graph, such as those described herein, that comprises a MemAlloc node is referred to as the graph that owns the allocation. In at least one embodiment, multiple graphs can own the same allocation through instantiation. In at least one embodiment, referring to FIG. 1, example 100 illustrates intra-graph allocations.

[0024] In at least one embodiment, a graph allocation has at least three lifetimes, or any suitable number of lifetimes. In at least one embodiment, the first two lifetimes are associated with constructing the graph, and the last lifetime is associated with executing the graph. In at least one embodiment, the API lifetime, also referred to as the first lifetime, refers to the period of time on the host during which passing an allocation to a graph node is valid. In at least one embodiment, the host refers to the CPU and its memory, and the device refers to the GPU and its memory. In at least one embodiment, the API lifetime begins when a MemAlloc node is created and ends when a MemFree node is created in the allocation graph and / or, if that does not occur, when the owning graph is destroyed.

[0025] In at least one embodiment, a topological lifetime, also referred to as a second lifetime, refers to a period of time during which a set of nodes in the graph have access to an allocation. In at least one embodiment, if the graph includes a MemAlloc node, the topological lifetime includes only nodes that are descendants of the MemAlloc node. In at least one embodiment, if the graph includes a MemFree node, the topological lifetime includes only nodes that are ancestors of the MemFree node. In at least one embodiment, if the graph includes both a MemAlloc node and a MemFree node, the MemFree node must be a descendant of the MemAlloc node.

[0026] In at least one embodiment, an execution lifetime, also referred to as a final lifetime, is the period of time during which an allocation is accessible to operations (e.g., operations of a kernel, memory copy, and / or variations thereof) of one or more systems of one or more programming models. In at least one embodiment, for an intra-graph allocation, the execution lifetime is contained entirely within the graph, begins when graph execution reaches a MemAlloc node, and ends when graph execution reaches a MemFree node. In at least one embodiment, for an inter-graph allocation, the execution lifetime begins when graph execution reaches a MemAlloc node, but extends beyond the execution of the graph until the allocation is freed (e.g., by invoking the graph with a corresponding MemFree node). In at least one embodiment, one or more systems provide functionality for inter-graph allocations to be accessed by various programming model APIs during their execution lifetime. In at least one embodiment, allocations have an externally visible execution lifetime.

[0027] In at least one embodiment, one or more systems launch the graph on one or more devices, such as GPUs, and when the graph is launched, the one or more devices perform the operations indicated by the nodes of the graph in any suitable order, such as serially, in parallel, and / or variations thereof. In at least one embodiment, with reference to FIG. 1 , alloc 102 causes a device, such as those described herein, to allocate memory, the allocated memory having an address of “0x1000,” kernel 104 causes the device to execute one or more processes indicated by kernel 104, release 106 causes the device to free the memory allocated through alloc 102 (e.g., at address “0x1000”), alloc 108 causes the device to allocate memory, the allocated memory having an address of “0x1000” since alloc 108 can reuse freed memory, kernel 110 causes the device to execute one or more processes indicated by kernel 110, and release 112 causes the device to free the memory (e.g., at address “0x1000”). kernel 114 causes the device to execute one or more processes indicated by kernel 114, kernel 116 causes the device to execute one or more processes indicated by kernel 116, alloc 118 causes the device to allocate memory, the allocated memory having an address of 0x2000 because alloc 118 cannot reuse memory at address 0x1000 as the memory may not have been freed, kernel 120 causes the device to execute one or more processes indicated by kernel 120, and release 122 causes the device to free the memory allocated through alloc 118 (for example, at address 0x2000).

[0028] In at least one embodiment, one or more systems define MemAlloc node parameters through the following code, although any variation thereof may be utilized:

number

number

[0029] In at least one embodiment, one or more systems utilize the parameters set forth in the table below, although any variation thereof may be utilized. [Table 1] In at least one embodiment, the API lifetime of an allocation begins upon creation of a MemAlloc node. In at least one embodiment, during the API lifetime, the allocation may be used by other nodes in the same or other graphs. In at least one embodiment, one or more users enforce the topological lifetime of an allocation.

[0030] In at least one embodiment, a graph allocation may be peer-accessible, which refers to allowing graphs with kernels from multiple devices to access the same graph-ordered memory allocation. In at least one embodiment, when the graph allocation is created, params->accessDescs specifies the peers to which the allocation should also be mapped. In at least one embodiment, the allocation may be mapped on more GPUs than specified to accommodate sharing physical pages with other graph-owned allocations. In at least one embodiment, accessDescs represents the minimum access required.

[0031] In at least one embodiment, one or more systems define an API function to generate one or more graph code nodes (e.g., a MemFree node) for deallocating memory in the graph through the following code, although any variation thereof may be utilized:

number

[0032] In at least one embodiment, one or more systems utilize the parameters set forth in the table below, although any variation thereof may be utilized. [Table 2] .

[0033] In at least one embodiment, every allocation begins inter-graph, since each allocation does not have a corresponding deallocation. In at least one embodiment, if an allocation is deallocated in its owning graph, it becomes an intra-graph allocation, and the allocation cannot be deallocated in a subsequent attempt. In at least one embodiment, an allocation without a deallocation can become persistently inter-graph if the allocation is deallocated in the graph other than its owner and / or if the owning graph is instantiated. In at least one embodiment, once an allocation becomes persistently inter-graph, one or more systems prevent subsequent attempts to deallocate the allocation in the owning graph, but other graphs may deallocate the allocation. In at least one embodiment, when a graph is started, one or more systems perform checks to ensure that allocations are not double-allocated or double-deallocated.

[0034] In at least one embodiment, when the graph is launched and execution of the graph reaches the point of a MemAlloc node of an inter-graph allocation, the execution life of that allocation begins, and during its execution life, the allocation may be passed to other operations (e.g., other graphs that reference the allocation or stream work) that are ordered after the execution of the graph.

[0035] FIG. 2 illustrates an example 200 of activating a graph, according to at least one embodiment. In at least one embodiment, the example 200 includes activating one or more graphs at a first time 202 (e.g., t=t0), a subsequent second time 204 (e.g., t=t1), and a subsequent third time 206 (e.g., t=t2). In at least one embodiment, the one or more graphs illustrated in FIG. 2 are graphs such as those described with respect to FIG. 1 and elsewhere herein. In at least one embodiment, one or more systems utilize one or more APIs to obtain code, compile the code into executable code, and execute the executable code on one or more devices to perform at least the execution of the graph, including memory allocation operations, memory copy operations, which may or may not be associated with the graph, and the execution of another graph, including memory deallocation operations.

[0036] In at least one embodiment, at a first time 202, one or more systems invoke the graph on a device, such as a GPU, that includes a memory allocation operation. In at least one embodiment, at the first time 202, the graph causes memory to be allocated on the device. In at least one embodiment, at a second time 204, a memory copy operation can access the memory allocated at the first time 202. In at least one embodiment, the memory copy operation refers to an operation of copying data between a device and another device. In at least one embodiment, the memory copy operation is performed with respect to the device through the graph, through one or more APIs, or any suitable manner, which may or may not involve the use of a graph. In at least one embodiment, at a third time 206, one or more systems invoke another graph on the device that includes a memory deallocation operation. In at least one embodiment, at the third time 206, the other graph causes the allocated memory to be deallocated, and the execution life of the allocation ends. In at least one embodiment, at a third time 206, although FIG. 2 illustrates the memory release operation associated with the other graph, the memory release operation may be performed in any suitable manner, such as through the other graph, through one or more APIs, or in any suitable manner, which may or may not involve the use of a graph.

[0037] In at least one embodiment, the execution life of an allocation may be terminated by one or more API calls to free the allocated memory outside of the graph and / or by invoking the graph containing a MemFree node for the allocation. In at least one embodiment, the allocation is freed in several graphs. In at least one embodiment, one or more graphs may be invoked to free the allocation after each invocation of an allocation graph. In at least one embodiment, one or more systems prevent graph invocations that attempt to allocate a still-allocated allocation, although at least one embodiment allows such operation. In at least one embodiment, one or more systems prevent invocations that attempt to free an already-freed allocation, although at least one embodiment allows such operation. In at least one embodiment, the allocation may or may not be accessed by various operations outside of its execution life.

[0038] In at least one embodiment, each allocation is matched with a free operation. In at least one embodiment, for intra-graph allocation, the owning graph contains both a MemAlloc node and a MemFree node. In at least one embodiment, for inter-graph allocation, one or more free operations (e.g., MemFree) are executed after each execution of the allocation graph. In at least one embodiment, one or more systems define the behavior of allocated memory free operations, also referred to as freeing, with respect to the following table, although any variation thereof may be utilized. [Table 3] "cudaMallocAsync" indicates an API function for allocating memory, and "cudaFreeAsync" indicates an API function for freeing allocated memory.

[0039] In at least one embodiment, one or more systems define API functions for using stream memory through the following code, although any variation thereof may be utilized:

number

[0040] In at least one embodiment, one or more systems define API functions for trimming device memory through the following code, although any variation thereof may be utilized:

number

[0041] In at least one embodiment, one or more systems define attributes for querying device memory status through the following codes, although any variation thereof may be utilized:

number

number

number

[0042] In at least one embodiment, one or more systems prevent edge or node deletion once the graph comprises either a MemAlloc node or a MemFree node, although in at least one embodiment, such actions are allowed. In at least one embodiment, one or more systems provide functionality for adding new edges to the graph. In at least one embodiment, one or more systems prevent a graph with a MemAlloc node or a MemFree node from being cloned or used as a child graph, although in at least one embodiment, such actions are allowed. In at least one embodiment, graph allocations may be accessed by nodes in child graphs and clone-able graphs and may or may not be allocated or freed. In at least one embodiment, one or more systems prevent actions such as edge deletion, node elimination, cloning, use as a child graph, multiple simultaneous instantiations of the graph, and various other actions, although in at least one embodiment, the one or more systems allow one or more of the actions.

[0043] In at least one embodiment, one or more systems provide functionality for converting various API functions related to streams into one or more API functions related to graphs, which are defined through the following table, although any variation thereof may be utilized. [Table 8]

[0044] In at least one embodiment, the explicit graph API does not allow one or more callers (e.g., users) to pass memory pools to a graph, although in at least one embodiment, such behavior is permitted. In at least one embodiment, each graph maintains its own resources internally. In at least one embodiment, the captured stream API supports explicit pools. In at least one embodiment, to support capturing, one or more systems utilize the captured pool's properties for the poolProps field of the node parameters. In at least one embodiment, one or more systems utilize the pool's identity. In at least one embodiment, only the pool's location is utilized for capturing. In at least one embodiment, one or more systems utilize the stream API pool's peer mapping to set the accessDescs field of the node creation parameters. In at least one embodiment, future changes to the captured pool may or may not be reflected in the node's accessibility. In at least one embodiment, extra mappings may be utilized.

[0045] In at least one embodiment, one or more systems prevent individual node updates, while in at least one embodiment, individual node updates are allowed. In at least one embodiment, one or more APIs for setting parameters for memory nodes are utilized by one or more systems for either instantiated or non-instantiated graphs. In at least one embodiment, changing the amount of memory used by an allocation may interfere with the placement of other allocations made after the node is changed. In at least one embodiment, the allocation may be utilized by other graph nodes, and the allocation may be updated.

[0046] In at least one embodiment, one or more systems prevent multiple simultaneous instantiations, although in at least one embodiment, multiple simultaneous instantiations are allowed. In at least one embodiment, once the graph is instantiated, the instance must be destroyed before the graph can be instantiated again. In at least one embodiment, passing the graph to one or more API functions that update the graph can count as instantiating it. In at least one embodiment, an instantiated graph can be destroyed while it is still running.

[0047] In at least one embodiment, the one or more systems provide functionality for updating an instantiated entire graph. In at least one embodiment, the one or more systems provide functionality for whole graph updates for graphs that include memory nodes. In at least one embodiment, the one or more systems use memory addresses from a new graph to replace existing addresses in their entirety. In at least one embodiment, the one or more systems generate the replacement graph in-order, eliminating dependency node placement issues.

[0048] In at least one embodiment, one or more systems provide functionality for destroying graphs. In at least one embodiment, instantiated graphs may be destroyed while they are running. In at least one embodiment, memory used by the graph remains accessible throughout the execution of the graph. In at least one embodiment, inter-graph allocations remain accessible until their execution life is normally terminated. In at least one embodiment, destroying the last graph that owns an allocation immediately terminates the API life of that allocation.

[0049] In at least one embodiment, one or more systems provide functionality to utilize a flag that may be passed to the graph at instantiation time, where the flag will change how the one or more systems handle inter-graph allocations owned by that graph. In at least one embodiment, after the graph is started, inter-graph memory allocated by the graph may be freed using various API functions for freeing allocated memory or by another graph. In at least one embodiment, one or more systems utilize a flag defined by the following code, although any variation thereof may be used:

number

[0050] In at least one embodiment, a flag can be specified at instantiation that causes the graph to exclusively own its physical memory. In at least one embodiment, graphs running on the same stream can reuse each other's memory, and a trim operation can release the memory to the OS. In at least one embodiment, when a flag is used, the graph is allocated memory immediately by one or more systems upon instantiation, and the memory cannot be reused by other graphs or returned to the OS by a trim call until the graph is destroyed; the flag is illustrated by the following code, although any variations thereof may be used:

number

[0051] In at least one embodiment, one or more systems provide functionality for tracking graph allocations. In at least one embodiment, when returned to one or more users, graph ordered allocations may be passed to nodes in the graph, which may or may not be passed down streams until their externally visible execution lifetime.

[0052] In at least one embodiment, one or more systems track graph allocations globally in a heap separate from a unified virtual addressing (UVA) heap. In at least one embodiment, every inter-graph allocation has an entry in the heap. In at least one embodiment, if an allocation is deallocated in the same graph, the corresponding entry is deleted; otherwise, it remains so that it can be found by inter-graph deallocations originating from separate graphs.

[0053] In at least one embodiment, each graph owns a pool for each device for which it owns allocations. In at least one embodiment, a driver internal pool manages virtual memory for all allocations owned by the graph. In at least one embodiment, during allocation, if the graph does not have a pool for the device specified by the allocation, a new per-graph pool on that device is created by one or more systems for the graph. In at least one embodiment, one or more systems map allocation pools onto peer devices on demand, as needed. In at least one embodiment, additional internal pools are not created, although in at least one embodiment, they may be created. In at least one embodiment, the internal pool managed by the graph owns all resources associated with memory, although there are other resources tracked by the graph system. In at least one embodiment, one or more systems utilize various structures described in the table below, although any variation thereof may be utilized. [Table 9]

[0054] In at least one embodiment, a graph allocation may be passed as an operand to a node in the graph, such as a node that performs a memory copy operation, a memory set operation, a memory deallocation operation, and / or variations thereof. In at least one embodiment, when validating a parameter, one or more systems (e.g., a driver) may check a global heap to see if it falls within the graph allocation, and if so, the one or more systems may check the allocation against the graph's inter-graph deallocation list to ensure the graph has not already deallocated the allocation.

[0055] In at least one embodiment, when validating operands, one or more systems utilize various processes described herein to identify graph memory locations. In at least one embodiment, graph allocations, if they are inter-graph, involve one or more memory objects during their execution lifetime. In at least one embodiment, one or more systems obtain various memory blocking operations from various memory objects. In at least one embodiment, one or more systems back up memory based at least in part on virtual addresses.

[0056] In at least one embodiment, one or more systems perform instantiation using one or more processes for cloning, as described herein. In at least one embodiment, the one or more systems maintain a memory pool. In at least one embodiment, the memory pool tracks whether each block is mapped, which can be a common state for all graphs created from the original. In at least one embodiment, the one or more systems copy the block refcount array. In at least one embodiment, future allocations in the original graph do not increase the physical memory footprint of the instantiated graph. In at least one embodiment, the one or more systems convert the list of owned inter-graph allocations into a state that can be used to quickly create various memory objects on the startup path. In at least one embodiment, the one or more systems copy deallocations into a format that can be used to delete various memory objects.

[0057] In at least one embodiment, one or more systems release or otherwise discard existing memory-related data in the graph and clone the data from the new graph into the instantiated graph, which in at least one embodiment shares ownership of its allocations with the new graph in addition to or instead of the original.

[0058] In at least one embodiment, a child graph has per-graph data that is completely separate from its parent graph. In at least one embodiment, a child graph's VA reservations are separate from its parent, and one or more systems prevent the child graph from having inter-graph allocations, although in at least one embodiment, the one or more systems allow such behavior, although the child graph may access its parent's intra-graph allocations. In at least one embodiment, when a parent graph is launched, the parent graph must also perform memory-related launch steps for all of its child graphs, which may appear as several graph launches from the memory allocator's perspective.

[0059] FIG. 3 illustrates an example graph and memory allocation 300, according to at least one embodiment. In at least one embodiment, the example 300 includes the state of the graph at a first time 302 (e.g., t=t0), a subsequent second time 304 (e.g., t=t1), and a subsequent third time 306 (e.g., t=t2). In at least one embodiment, the one or more graphs illustrated in FIG. 3 are graphs such as those described with respect to FIGS. 1-2 and elsewhere herein. In at least one embodiment, one or more systems utilize one or more APIs to obtain code to perform various graph operations, compile the code into executable code, and execute the executable code on one or more devices.

[0060] In at least one embodiment, a pool refers to a collection or region of memory. In at least one embodiment, each graph pool has its own virtual address reservation, and each graph pool uses a heap to manage it, but when an allocation is freed, the allocation may not be returned to the heap. In at least one embodiment, the allocation is freed to a local heap called a subpool, which allows the allocation to be reused by descendants of the MemFree node in the graph.

[0061] In at least one embodiment, a subpool is associated with a main pool (e.g., the graph pool). In at least one embodiment, memory released to the subpool must first be allocated from the main pool. In at least one embodiment, the subpool supports allocations with a minimum expiration requirement, represented by an integer called a sequence ID. In at least one embodiment, upon every release to the subpool, the subpool's sequence ID is incremented and a new value is associated with the released memory. In at least one embodiment, one or more systems provide sequence IDs to allow one fork of the graph to continue allocating from the subpool even after another fork has released memory to the pool by specifying an older (e.g., lower) sequence ID from before the fork.

[0062] In at least one embodiment, each node in the graph includes a list of zero or more subpool snapshots. In at least one embodiment, each snapshot includes at least a reference to the subpool and / or a sequence ID for the subpool when the snapshot was taken. In at least one embodiment, when a node is created, the node copies all snapshots of its dependencies into a new snapshot list and resolves duplicate entries by taking the highest (e.g., least specific) sequence ID, which may be referred to as inheriting snapshots. In at least one embodiment, a snapshot is considered current when it includes the current sequence ID of the snapshot's subpool.

[0063] In at least one embodiment, as well as inheriting snapshot lists, MemFree nodes also modify snapshot lists when they look for a snapshot that is current; if none exists, the MemFree node creates a new subpool and inserts it into the snapshot list; when the MemFree node increments the sequence ID for the selected snapshot and frees memory to a subpool using that sequence ID, this causes the MemFree node to have only the current snapshot for that subpool. In at least one embodiment, MemAlloc nodes do not modify the snapshot lists they inherit. In at least one embodiment, the MemAlloc node attempts to allocate from each snapshot list with the snapshot's sequence ID as the minimum expiration time; if the snapshot is current, the allocation is an unlimited allocation.

[0064] In at least one embodiment, at startup, a list of graph-owned inter-graph allocations, as the external lifetime of these allocations is about to begin, is turned into one or more memory objects by one or more systems so that the allocations can be used in dependent operations. In at least one embodiment, a list of the graph's unowned freed allocations is used to delete one or more memory objects. In at least one embodiment, as part of the execution lifetime, the startup path verifies the existence or non-existence of expected memory objects. In at least one embodiment, if the expectation is not met, the startup fails.

[0065] In at least one embodiment, at a first time 302, a first MemAlloc node (e.g., alloc 308) will always allocate from the subpool because there is no previous MemFree node to place memory in the subpool. In at least one embodiment, at first time 302, alloc 308 allocates directly from the main pool. In at least one embodiment, at a second time 304, when the allocated memory is freed, a MemFree node (e.g., free 310) will create a new subpool, free the allocation therein, and track the subpool in a new single-element snapshot list (e.g., sequence ID 312). In at least one embodiment, at second time 304, free 310 will free memory to the new subpool and track it as current in the new snapshot. In at least one embodiment, at this point, the sequence ID in the snapshot and the subpool match, and the snapshot is current. In at least one embodiment, at a third time 306, dependent MemAlloc nodes (e.g., alloc314 or alloc316) can attempt to allocate from the subpools, whether they are contiguous or not. In at least one embodiment, at a third time 306, dependent MemAlloc nodes (e.g., alloc314 or alloc316) can both attempt to allocate from the same subpool. In at least one embodiment, if the subpool cannot satisfy the request, the main pool will be used. In at least one embodiment, if allocation is only occurring, allocating nodes may or may not be unordered relative to each other.

[0066] Figure 4 shows an example 400 of a fork in a graph, according to at least one embodiment. In at least one embodiment, example 400 is a continuation of example 300 of Figure 3. In at least one embodiment, one or more graphs illustrated in Figure 4 are graphs such as those described with respect to Figures 1-3 and elsewhere herein. In at least one embodiment, example 400 includes states of the graph at a fourth time 402 (e.g., t=t3), a subsequent fifth time 404 (e.g., t=t4), and a subsequent sixth time 406 (e.g., t=t5).

[0067] In at least one embodiment, at fourth time 402, initially after the fork, the inherited snapshot is current on both sides of the fork, but as soon as a free occurs on one side, it increments the sequence ID of the subpool, and the other side of the fork is no longer current. In at least one embodiment, at fourth time 402, another MemFree node, free 318, increments the sequence ID on the left fork. In at least one embodiment, MemAlloc nodes on the non-current side of the fork do not allocate the freed memory on the other side because they use an older sequence ID, but if they did allocate memory, corruption could occur because they may have allocated memory before the memory was freed. In at least one embodiment, at fifth time 404, right fork MemAlloc nodes, alloc 316 and alloc 320, are limited to sequence IDs less than or equal to 1.

[0068] In at least one embodiment, additional releases 322 on the non-current side require the creation of another subpool, since releases can only occur in subpools from the current snapshot. In at least one embodiment, at a sixth time 406, right fork release 322 requires the current subpool and creates a new subpool (e.g., corresponding to sequence ID 324). In at least one embodiment, if the release were allowed, the MemAlloc node on the current side would reallocate memory before it was freed, potentially resulting in corruption. In at least one embodiment, forks in the graph can share a pre-fork subpool without restriction until a release in one of the forks utilizes that subpool for a side.

[0069] In at least one embodiment, one or more systems unify various aspects of intra-graph and inter-graph allocation. In at least one embodiment, one or more systems calculate and track how many times the graph allocates or frees a given block of VA space. In at least one embodiment, when the graph runs, each block needs to be mapped by one or more systems to free up physical memory (e.g., that has no outstanding allocations). In at least one embodiment, one or more systems perform the remapping at startup time.

[0070] 5 illustrates an example block refcount array 500, according to at least one embodiment. In at least one embodiment, the example 500 includes the state of the block refcount array at a first time 502 (e.g., t=t0), a subsequent second time 504 (e.g., t=t1), and a subsequent third time 506 (e.g., t=t2).

[0071] In at least one embodiment, each graph is associated with a virtual address (VA) reservation, which is divided into fixed-size blocks. In at least one embodiment, a VA reservation refers to a set of data that indicates a virtual address for allocating memory. In at least one embodiment, a block is the unit of physical allocation for both graph allocations. In at least one embodiment, each graph has its own VA reservation, so that each VA block is specific to a live graph and / or a non-destroyed graph. In at least one embodiment, each graph also has an array of graph-local reference counts, also called refcounts, with an element for each block of space it owns, referred to as the block refcount array.

[0072] In at least one embodiment, at a first time 502, the graph is initialized. In at least one embodiment, at the first time 502, blocks start with a graph-local refcount of 0. In at least one embodiment, at a second time 504, when an allocation is made, each VA block that contains part of the allocation increases its graph-local refcount. In at least one embodiment, at the second time 504, one or more systems iterate through the block refcount array and increment the count. In at least one embodiment, at a third time 506, creating the freed node involves one or more systems decrementing the graph-local refcount. In at least one embodiment, in the case of intra-graph freeing, the refcount is contained in the block refcount array, which may be decremented directly by one or more systems. In at least one embodiment, in the case of inter-graph freeing, the graph-local refcount of the allocation graph is not modified, and one or more systems instead create a surrogate block refcount array. In at least one embodiment, the surrogate may be reused by a release graph if another inter-graph release affects the same (eg, external) VA block.

[0073] In at least one embodiment, one or more systems, as part of instantiation, perform various mapping and physical memory allocation operations associated with pre-launch by utilizing the stream last used to launch the graph, which can reduce overhead associated with the first launch operation.

[0074] In at least one embodiment, one or more systems allocate memory for graphs from a stream-owned pool to support one or more graphs in the same stream that reuse each other's physical memory. In at least one embodiment, the pool may own memory from several devices and is entirely internal. In at least one embodiment, the memory contained in the pool is added to a total that can be queried through the pool through one or more API functions, such as the "cuDeviceGetGraphMemPool()" function.

[0075] In at least one embodiment, prior to launch, physical memory from the launch stream is used to back all allocations made by the graph in the pre-launch phase. In at least one embodiment, one or more systems use cached physical memory per stream to ensure that serialization is not introduced between graphs launched on different streams. In at least one embodiment, one or more systems provide functionality to restart graphs in the same stream to reuse the same physical memory. In at least one embodiment, after launch, when the completion marker for the graph is known, one or more systems update tracking data so that the lifetime of allocations can be accurately tracked in the post-launch phase.

[0076] In at least one embodiment, during pre-startup, all VA blocks that have ever backed allocations owned by the graph are mapped by one or more systems into physical memory from the startup stream's physical page cache, including VA blocks with a graph-local refcount of 0 (e.g., for VA blocks containing only intra-graph allocations). In at least one embodiment, a graph reuses allocations from inter-graph deallocations that are backed by allocations within the graph's VA reservation.

[0077] 6 illustrates an example virtual address reservation 600, according to at least one embodiment. In at least one embodiment, the example 600 includes a virtual address reservation at a first time 602 (e.g., t=t0) and at a subsequent second time 604 (e.g., t=t1).

[0078] In at least one embodiment, each physical block comprises at least three main fields: a reference to the owning stream (e.g., "streamID" in FIG. 6), the stream's sequence ID since the last time the block reached a refcount of 0 (e.g., "sequence ID" in FIG. 6), and / or a refcount of how many allocations are using the block (e.g., "refCount" in FIG. 6). In at least one embodiment, if the refcount is 0, the block can be used by another graph by ensuring that it has acquired the owning stream's sequence ID. In at least one embodiment, at a first time 602, VA blocks with a value of 0 contain only intra-graph allocations, and VA blocks with a value of 1 contain at least one allocation in the graph that has not been released.

[0079] In at least one embodiment, when activating the graph that was previously mapped to physical blocks, one or more blocks may not be available during activation, such as when the blocks were used by an inter-graph allocation that was not released. In at least one embodiment, the activation process checks whether any of the graph-owned VA blocks are mapped to physical memory with a non-zero refcount, and if so, those blocks need to be remapped before activation. In at least one embodiment, as an illustrative example, if existing physical blocks have a refcount of 1, remapping must be performed by one or more systems. In at least one embodiment, if physical memory has a refcount of 0, the activation process captures their sequence IDs to ensure that the release of memory is properly acquired (e.g., is a no-op for memory released in the same stream).

[0080] In at least one embodiment, blocks in use by the graph during pre-activation are preserved so that allocations made for remapping do not reuse free blocks already mapped to the graph, and so that other activations do not reuse these blocks during the graph activation (e.g., when all memory-related locks are dropped). In at least one embodiment, pageable memory copy operations may block the activation until their completion.

[0081] In at least one embodiment, one or more systems add graph local counts to blocks in a post-startup phase, which includes decrementing from a surrogate refcount object. In at least one embodiment, surrogates allow physical blocks to progress from an allocated state to a freed state when they drop refcounts. In at least one embodiment, at a second time 604, graph local counts (e.g., illustrated in FIG. 6 as “VA blocks”) are applied to unallocated blocks (e.g., illustrated in FIG. 6 as “physical blocks”). In at least one embodiment, graph local counts are applied to refCount fields. In at least one embodiment, when the graph frees any blocks (e.g., resulting in a refcount of 0), the stream's sequence ID is advanced and new values ​​are associated with those blocks. In at least one embodiment, the sequence IDs are determined pre-startup, but the graph's completion marker may not, and checking the completion of the current sequence ID may require reading the graph's marker. In at least one embodiment, when the marker is updated, one or more systems release pre-boot artificial refcounts and assign the sequence ID.

[0082] In at least one embodiment, the graph may be running when it is destroyed. In at least one embodiment, if the graph is running when it is destroyed, the graph's physical memory may not be released to the OS, and the graph's mapping may not be deleted until the graph is complete. In at least one embodiment, the graph's physical memory may be reused by the invocation that acquired it (e.g., it may be a no-op since the memory is reused in the same stream), but deleting the mapping must be done on the host.

[0083] In at least one embodiment, one or more systems provide functionality for allocating memory, using allocated memory, and freeing allocated memory using graphs through graph memory nodes. In at least one embodiment, the graph memory nodes enable graphs to make and own memory allocations. In at least one embodiment, the graph memory nodes have GPU-ordered liveness semantics, enabling stream capture of various stream-ordered allocation APIs, such as those for allocating memory and freeing allocated memory, and also enabling driver-managed memory reuse.

[0084] In at least one embodiment, graph allocations (e.g., memory allocations) have fixed addresses for the lifetime of the graph and its instantiations, which allows memory to be directly referenced by other operations within the graph without the need for updating the graph when new memory is allocated. In at least one embodiment, allocations within the graph whose graph-ordered lifetimes do not overlap can use the same fixed addresses and underlying physical memory resources.

[0085] In at least one embodiment, GPU-ordered liveness semantics allow one or more drivers to virtually alias the same physical memory to allocations from multiple graphs. In at least one embodiment, drivers can virtually alias the same physical memory to satisfy the needs of those graphs, as long as the graphs are all invoked in the same stream and deallocate their own allocations. In at least one embodiment, liveness is referred to as "GPU-ordered" because allocations that are not deallocated in an allocation graph obey various graph-ordered semantics within the graph and stream-ordered semantics between the invocation of the allocation graph and the deallocation operation (which may occur either at a node in the graph or at a deallocation call, such as an API call, to deallocate the allocated memory).

[0086] FIG. 7 illustrates an example 700 of address reuse, according to at least one embodiment. In at least one embodiment, the example 700 includes the graph at a first time 702 (e.g., t=t0) and at a subsequent second time 704 (e.g., t=t1). In at least one embodiment, one or more graphs illustrated in FIG. 7 are graphs such as those described with respect to FIGS. 1-4 and elsewhere herein. In at least one embodiment, alloc 706, new alloc 710, and new alloc 714 are nodes for memory allocation, such as those described herein. In at least one embodiment, release 708 and release 712 are nodes for freeing memory, such as those described herein.

[0087] In at least one embodiment, the driver reuses memory within the graph based on at least virtual address assignment, reuse between graphs with virtual aliasing, where different graphs can map the same physical memory mapped to their virtual addresses, and / or variations thereof. In at least one embodiment, the driver assigns virtual addresses during allocation node creation and enables them to be used in the graph. In at least one embodiment, addresses are fixed and remain unchanged across graph instantiation and activation operations. In at least one embodiment, when the graph allocation is deallocated in an allocation graph, subsequent graph allocation nodes in the same graph may reuse the virtual address range as long as there is a graph dependency edge ordering the new allocation node after the deallocated allocation node.

[0088] In at least one embodiment, at a first time 702, new allocate node alloc 710 can reuse an address released by dependent node release 708. In at least one embodiment, at a second time 704, new allocate node alloc 714 has no dependency on releasing node release 712, and therefore the new allocate node cannot use an address from associated allocate node alloc 710. In at least one embodiment, if at second time 704, allocating node alloc 710 used an address released by releasing node release 708, new allocate node alloc 714 would need a new address.

[0089] 8 shows an example 800 of physical memory sharing between graphs, according to at least one embodiment. In at least one embodiment, graph1 802, graph2 806, graph3 812, and graph4 814 are graphs such as those described with respect to FIGS. 1-4, 7, and elsewhere herein. In at least one embodiment, graph1 802 utilizes physical memory1 804, graph2 806 utilizes physical memory2 808, and release(graph1 memory) 810 is an operation that releases the memory utilized by graph1.

[0090] In at least one embodiment, graphs in the same stream can share physical memory because they do not run simultaneously. In at least one embodiment, referring to FIG. 8 , unreleased allocations prevent Graph 2 806 from sharing physical memory from Graph 1 802. In at least one embodiment, Graph 3 812 can use physical memory from either Graph 1 802 (e.g., Physical Memory 1 804) or Graph 2 806 (e.g., Physical Memory 808) because the memory is freed when Graph 3 812 starts. In at least one embodiment, Graph 3 812 utilizes Physical Memory 1 804. In at least one embodiment, referring to FIG. 8 , Graph 4 814 is launched in a separate stream and cannot use the same memory unless the other streams have completed their work before Graph 4 814 starts.

[0091] In at least one embodiment, the driver maps physical memory to virtual addresses before reaching allocation nodes in GPU order. In at least one embodiment, if multiple graphs use the same physical memory, they cannot run simultaneously. In at least one embodiment, while the graph allocation remains unreleased, the corresponding physical page cannot be used by other graphs. In at least one embodiment, at graph startup time, the driver uses stream ordering of already started graphs and queued memory operations to determine the physical memory that will be available for use by the starting graph. In at least one embodiment, the driver balances minimizing the need to remap operations with minimizing the total physical memory footprint of various graph memory nodes. In at least one embodiment, the driver utilizes ordering information to map the same physical memory to multiple allocations.

[0092] In at least one embodiment, one or more systems (e.g., the drivers of one or more programming models) associate physical memory with streams when creating new mappings during graph invocation and prioritize using physical memory associated with the invoking stream. In at least one embodiment, the one or more systems use virtual aliasing to map the same physical memory to multiple graphs launched on the same stream, since the stream orders the execution of the graphs.

[0093] In at least one embodiment, launching the same graph into a different stream may require remapping for either that graph or subsequent graphs launched in the original stream. In at least one embodiment, when the same graph is launched into a different stream, the driver replaces physical memory (e.g., so that physical memory continues to be reused without penalty by other graphs running in the original stream) and / or associates physical memory with the new stream (e.g., avoiding remapping for the current graph and allowing future graphs launched in the new stream to prioritize sharing physical memory).

[0094] In at least one embodiment, to prevent inactive streams from holding cached memory, one or more systems reallocate physical memory from other streams to the active stream instead of allocating more memory. In at least one embodiment, the driver reallocates memory when it can safely do so without introducing false dependencies.

[0095] FIG. 9 illustrates an example process 900 for allocating memory using a graph, according to at least one embodiment. In at least one embodiment, some or all of process 900 (or any other process described herein, or variations and / or combinations thereof) is performed under the control of one or more computer systems configured with computer-executable instructions and implemented as code (e.g., computer-executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors by hardware, software, or a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium in the form of a computer program comprising a plurality of computer-readable instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable medium. In at least one embodiment, at least some computer-readable instructions usable to implement process 900 are not stored using only transitory signals (e.g., propagating transitory electrical or electromagnetic transmissions). In at least one embodiment, the non-transitory computer-readable medium does not necessarily include non-transitory data storage circuitry (e.g., buffers, caches, and queues) within a transceiver of a transitory signal.

[0096] In at least one embodiment, process 900 is performed by one or more systems, such as those described in this disclosure. In at least one embodiment, the one or more systems include any suitable system having one or more collections of hardware and / or software resources along with instructions that, when executed, perform memory allocation and / or deallocation processes, such as those described herein. In at least one embodiment, process 900 is performed by a system of one or more programming models. In at least one embodiment, one or more processes of process 900 are performed in any suitable order, including serially, in parallel, and / or variations thereof, and using any suitable processing unit, such as a CPU, a GPU, a PPU, and / or variations thereof.

[0097] In at least one embodiment, the system that performs at least a portion of process 900 includes executable code for obtaining 902 at least code indicating at least the generation of one or more graph code nodes. In at least one embodiment, the one or more graph code nodes include a node corresponding to a memory allocation operation, such as a MemAlloc node. In at least one embodiment, the graph code node corresponding to a memory allocation operation, or also referred to as a graph code node for allocating memory, encodes information regarding the memory allocation, such as properties of the memory to be allocated, a size of the memory to be allocated, constraints on the memory to be allocated, an address of the memory to be allocated, and / or any suitable information.

[0098] In at least one embodiment, the code indicates at least the creation of one or more graph code nodes, such as a MemAlloc node, and / or the invocation of a graph comprising the one or more graph code nodes. In at least one embodiment, the code utilizes one or more APIs to indicate at least the creation of one or more graph code nodes. In at least one embodiment, the code comprises one or more API calls for the creation of one or more graph code nodes. In at least one embodiment, the system compiles and executes the code. In at least one embodiment, the system executes the code by converting the code into executable code and executing the executable code. More information regarding compiling and executing code can be found in the descriptions of Figures 30-39.

[0099] In at least one embodiment, the system performing at least a portion of process 900 includes executable code for implementing an API to generate one or more graph code nodes for allocating memory, at 904. In at least one embodiment, as part of the execution of obtained code that utilizes one or more APIs, the system implements one or more APIs that correspond to the one or more APIs utilized in the code. In at least one embodiment, the system implements the API to generate one or more graph code nodes for allocating memory by creating or otherwise instantiating one or more MemAlloc nodes.

[0100] In at least one embodiment, the system implements an API, such as those described herein, based on parameter values ​​of the API, which may be indicated in code. In at least one embodiment, a parameter value refers to a value of a parameter of an API, such as those described herein, and includes any suitable data, such as a numerical value, a data structure, a data object, and / or variations thereof. In at least one embodiment, as an illustrative example, code utilizes an API to generate one or more graph code nodes for allocating memory and includes a parameter value for a size of memory to be allocated, and the system implements the API to generate one or more graph code nodes for allocating memory of the size of memory. In at least one embodiment, the system implements an API, such as those described herein, and the implementation of the API results in data output to one or more data structures, data objects, locations, and / or variations thereof, as indicated by the parameter values ​​of the API. In at least one embodiment, as an illustrative example, the system implements an API to generate one or more graph code nodes for allocating memory, and data such as addresses of the allocated memory are output to one or more data structures, data objects, locations, and / or variations thereof (e.g., as indicated by parameter values).

[0101] In at least one embodiment, the system generates or otherwise obtains the graph, also referred to as a graph data structure. In at least one embodiment, the system generates one or more graph code nodes (e.g., MemAlloc nodes) for allocating memory as part of the graph data structure. In at least one embodiment, an API for generating one or more graph code nodes for allocating memory is shown using the following notation, although any variation thereof, such as those described herein, may be utilized:

number

[0102] In at least one embodiment, the system performing at least a portion of process 900 includes, at 906, executable code for at least invoking a graph to cause at least memory to be allocated. In at least one embodiment, the system provides the executable for the graph data structure to one or more devices. In at least one embodiment, the one or more devices include any suitable device, such as a GPU, a PPU, a CPU, a GPGPU, and / or variations thereof. In at least one embodiment, the system invokes the graph on one or more devices, which refers to the process of causing the one or more devices to perform one or more operations of the graph (e.g., through the executable for the graph). In at least one embodiment, the system launches the graph on one or more devices by providing the executable file for the graph to the one or more devices, and the one or more devices execute the executable file for the graph and, as part of the execution, perform one or more operations indicated by one or more nodes of the graph in any suitable manner, such as serially, in parallel, and / or variations thereof. In at least one embodiment, as part of executing the obtained code, the system launches the graph on one or more devices.

[0103] In at least one embodiment, the system causes one or more devices to allocate memory and perform one or more operations using the allocated memory by invoking one or more graphs on the one or more devices, the graphs comprising at least one or more graph code nodes for allocating memory and one or more graph code nodes corresponding to the one or more operations. In at least one embodiment, the system causes one or more devices to perform a set of operations indicated by the graph data structure using the allocated memory by invoking the graph data structure on the one or more devices, the graph data structure comprising at least one or more graph code nodes indicating the set of operations and use of the allocated memory.

[0104] In at least one embodiment, the system utilizes one or more graph code nodes for allocating memory. In at least one embodiment, the system causes one or more devices to allocate memory based on the one or more graph code nodes for allocating memory by invoking the graph comprising one or more graph code nodes on the one or more devices. In at least one embodiment, the system causes one or more devices to allocate memory through one or more operating system functions. In at least one embodiment, the system allocates memory on one or more devices. In at least one embodiment, one or more devices allocate memory using the one or more graph code nodes for allocating memory by utilizing information encoded in the one or more graph code nodes for allocating memory. In at least one embodiment, as an illustrative example, the graph code node for allocating memory encodes information indicating a size of the allocation, and the device allocates memory of said size. In at least one embodiment, one or more devices allocate memory by identifying a suitable region of memory (e.g., based on information encoded in the graph code nodes for allocating memory or from information provided by one or more systems, such as a CPU) and indicating that the suitable region of memory, also referred to as allocated memory, is reserved and / or in use for one or more operations. In at least one embodiment, one or more systems, such as a CPU, identify a suitable memory region (e.g., based on one or more graph code nodes for allocating memory) and provide the identified memory region to one or more devices, which allocate memory using the identified memory region.

[0105] In at least one embodiment, one or more devices utilize allocated memory to perform one or more operations. In at least one embodiment, the system performing at least a portion of process 900 includes executable code for at least obtaining a second graph data structure indicating one or more operations and for invoking the second graph data structure on one or more devices to cause the one or more devices to perform the one or more operations using allocated memory. In at least one embodiment, memory allocated for a first graph data structure may be used to perform the operations indicated by the first graph data structure and / or the second graph data structure. In at least one embodiment, one or more devices deallocate the allocated memory upon completion of one or more operations.

[0106] FIG. 10 illustrates an example process 1000 for deallocating memory using a graph, according to at least one embodiment. In at least one embodiment, some or all of process 1000 (or any other process described herein, or variations and / or combinations thereof) is performed under the control of one or more computer systems configured with computer-executable instructions and implemented as code (e.g., computer-executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors by hardware, software, or a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium in the form of a computer program comprising a plurality of computer-readable instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable medium. In at least one embodiment, at least some computer-readable instructions usable to implement process 1000 are not stored using only transitory signals (e.g., propagating transitory electrical or electromagnetic transmissions). In at least one embodiment, the non-transitory computer-readable medium does not necessarily include non-transitory data storage circuitry (e.g., buffers, caches, and queues) within a transceiver of a transitory signal.

[0107] In at least one embodiment, process 1000 is performed by one or more systems, such as those described in this disclosure. In at least one embodiment, the one or more systems include any suitable system having one or more collections of hardware and / or software resources along with instructions that, when executed, perform memory allocation and / or deallocation processes, such as those described herein. In at least one embodiment, process 1000 is performed by a system of one or more programming models. In at least one embodiment, one or more processes of process 1000 are performed in any suitable order, including serially, in parallel, and / or variations thereof, and using any suitable processing unit, such as a CPU, a GPU, a PPU, and / or variations thereof.

[0108] In at least one embodiment, the system performing at least a portion of process 1000 includes, at 1002, executable code for obtaining code indicating at least the generation of one or more graph code nodes. In at least one embodiment, the one or more graph code nodes include a node corresponding to a memory deallocation operation, such as a MemFree node. In at least one embodiment, a MemFree node, also referred to as a graph code node corresponding to a memory deallocation operation or a graph code node for deallocating or freeing memory, encodes information regarding memory deallocation, such as properties of the memory to be deallocated, a size of the memory to be deallocated, constraints on the memory to be deallocated, an address of the memory to be deallocated, and / or any suitable information. In at least one embodiment, the code indicates at least the generation of one or more graph code nodes, such as a MemFree node, and / or the activation of a graph comprising the one or more graph code nodes. In at least one embodiment, the code utilizes one or more APIs to indicate at least the creation of one or more graph code nodes. In at least one embodiment, the code comprises one or more API calls for the creation of one or more graph nodes. In at least one embodiment, the system compiles and executes the code. More information regarding compiling and executing the code can be found in the descriptions of Figures 30-39.

[0109] In at least one embodiment, the system performing at least a portion of process 1000 includes, at 1004, executable code for implementing an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, the system implements the API to generate one or more graph code nodes for deallocating memory by generating or otherwise instantiating one or more MemFree nodes. In at least one embodiment, the system generates or otherwise obtains the graph data structure. In at least one embodiment, the system generates one or more graph code nodes (e.g., MemFree nodes) for deallocating memory as part of the graph data structure. In at least one embodiment, the API for generating one or more graph code nodes for deallocating memory is shown using the following notation, although any variation thereof, such as those described herein, may be utilized:

number

[0110] In at least one embodiment, the system implementing at least a portion of process 1000 includes, at 1006, executable code for at least invoking a graph to cause at least memory to be deallocated. In at least one embodiment, the system invokes the graph on one or more devices by providing the executable file for the graph to the one or more devices, and the one or more devices execute the executable file for the graph and, as part of the execution, perform one or more operations indicated by one or more nodes of the graph in any suitable manner, such as serially, in parallel, and / or variations thereof. In at least one embodiment, one or more devices deallocate memory using one or more graph code nodes for deallocating memory by utilizing information encoded in the one or more graph code nodes for deallocating memory. In at least one embodiment, as an illustrative example, the graph code node for deallocating memory encodes information indicating an address of the allocation, and the device deallocates the memory located at the address.

[0111] In at least one embodiment, the system causes one or more devices to allocate memory and deallocate allocated memory by invoking, on the one or more devices, the graph comprising at least one or more graph code nodes for allocating memory and one or more graph code nodes for deallocating memory. In at least one embodiment, the system causes one or more devices to allocate memory and deallocate allocated memory by invoking, on the one or more devices, a first graph comprising at least one or more graph code nodes for allocating memory and a second graph comprising one or more graph code nodes for deallocating memory. In at least one embodiment, the system causes one or more devices to perform the one or more operations using allocated memory and to deallocate the allocated memory by invoking one or more graphs on the one or more devices, the graphs comprising at least one or more graph code nodes corresponding to the one or more operations and one or more graph code nodes for deallocating memory.

[0112] In at least one embodiment, one or more devices deallocate memory based on information encoded in one or more graph code nodes for deallocating memory, the memory allocated by the one or more devices based on one or more graph code nodes for allocating memory parts of the graph comprising the one or more graph code nodes for deallocating memory. In at least one embodiment, one or more devices deallocate memory based on information encoded in one or more graph code nodes for deallocating memory, the memory allocated by the one or more devices based on one or more graph code nodes for allocating memory parts of a graph different from the graph comprising the one or more graph code nodes for deallocating memory. In at least one embodiment, one or more devices deallocate memory based on information encoded in one or more graph code nodes for deallocating memory, the memory being allocated by the one or more devices based on one or more memory allocation processes, including those that may or may not involve use of a graph.

[0113] In at least one embodiment, the system utilizes one or more graph code nodes for deallocating memory. In at least one embodiment, the system causes one or more devices to deallocate memory through one or more operating system functions. In at least one embodiment, the system deallocates memory on one or more devices. In at least one embodiment, as an illustrative example, one or more devices deallocate memory by identifying a suitable region of allocated memory (e.g., based on information encoded in the graph code node for deallocating memory or from information provided by one or more systems, such as a CPU) and indicating that the suitable region of allocated memory is not reserved for one or more operations and / or is not in use. In at least one embodiment, one or more systems, such as a CPU, identify suitable memory regions (e.g., based on one or more graph code nodes for deallocating memory) and provide the identified memory regions to one or more devices, which utilize the identified memory regions to deallocate memory.

[0114] In at least one embodiment, one or more systems implement an API for generating one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, the API for generating one or more graph code nodes for allocating and deallocating memory may be represented using any suitable notation, which may or may not relate to a programming model, and may include any suitable parameters, such as those described herein, which may or may not relate to a programming model, including indications of one or more operations, one or more nodes, one or more graphs, properties of the one or more nodes, dependency properties, properties of memory to be allocated and / or deallocated, constraints on memory to be allocated and / or deallocated, and / or any suitable parameters.

[0115] In at least one embodiment, one or more systems implement the API to generate one or more graph code nodes for allocating and deallocating memory by generating a first graph code node for allocating memory and a second graph code node for deallocating memory, which may be part of one or more graphs (e.g., the first graph code node and the second graph code node may be part of the same graph or different graphs). In at least one embodiment, one or more systems implement the API to generate one or more graph code nodes for allocating and deallocating memory by generating a first graph code node for allocating memory and a second graph code node for deallocating memory for one or more operations (e.g., indicated through parameters of the API). In at least one embodiment, one or more systems launch one or more graphs comprising at least a MemAlloc node and a MemFree node on one or more devices to cause the one or more devices to allocate and deallocate memory.

[0116] In at least one embodiment, an API, such as those described herein, is a driver API or a runtime API. In at least one embodiment, a driver API is a low-level API that may be referred to in terms of a programming model (e.g., a CUDA driver API). In at least one embodiment, a driver API interacts directly with one or more devices. In at least one embodiment, a runtime API is a high-level API that may be referred to in terms of a programming model (e.g., a CUDA runtime API). In at least one embodiment, a runtime API operates utilizing a driver API. More information regarding driver APIs and runtime APIs may be found in the description of FIG. 31.

[0117] In at least one embodiment, a graph memory node is a graph node that represents either a memory allocation action or a memory deallocation action. In at least one embodiment, a node that allocates memory is called an allocate node. In at least one embodiment, a node that deallocates memory is called a deallocation node. In at least one embodiment, an allocation made through a graph memory node is called a graph allocation. In at least one embodiment, an allocation is considered fresh each time the graph is run. In at least one embodiment, the previous contents of the buffer are not guaranteed to be there (e.g., by reuse) by one or more systems.

[0118] In at least one embodiment, graph memory nodes are ordered by one or more systems within the graph by dependency edges. In at least one embodiment, one or more users must ensure that operations accessing graph memory must be ordered after allocation nodes and / or before operations that free memory when utilizing the graph. In at least one embodiment, GPU ordering refers to stream and / or graph ordering that determines when work executes on a GPU. In at least one embodiment, the driver assigns virtual addresses for the graph allocations at node creation time. In at least one embodiment, one or more systems fix addresses for the lifetime of allocation nodes, and allocation contents are not persistent past free operations.

[0119] In at least one embodiment, graph memory nodes are explicitly created with various API functions, such as those described herein, such as cudaGraphAddMemAllocNode, cudaGraphAddMemFreeNode, and / or variants thereof, which may be indicated in any suitable manner. In at least one embodiment, cudaGraphAddMemAllocNode fills the dptr field of the passed CUDA_MEM_ALLOC_NODE_PARAMS structure with the virtual address of the allocation. In at least one embodiment, all operations using a graph allocation in an allocation graph must be ordered after the allocation node. In at least one embodiment, a free node must be ordered after all uses of an allocation in said graph. In at least one embodiment, cudaGraphAddMemFreeNode creates a free node.

[0120] In at least one embodiment, one or more systems create the graph through the following code, although any variation thereof may be utilized:

number

[0121] In at least one embodiment, graph memory nodes may be created by capturing corresponding stream-ordered allocation and deallocation calls. In at least one embodiment, virtual addresses returned by the captured allocation APIs may be used by other operations in the graph. In at least one embodiment, one or more systems capture stream-ordered dependencies in the graph, and the ordering requirements of the stream-ordered allocation APIs ensure that the various graph memory nodes are properly ordered with respect to the captured stream operations.

[0122] In at least one embodiment, one or more systems generate the graph using stream capture via the following code, although any variation thereof may be utilized:

number

[0123] In at least one embodiment, graph allocations do not need to be deallocated by the allocation graph. In at least one embodiment, when the graph does not deallocate an allocation it makes, the allocation persists beyond execution of the graph. In at least one embodiment, an allocation can be deallocated by a normal call using various API functions to deallocate allocated memory, by the invocation of another graph with a corresponding Free node, and / or by a subsequent invocation of the graph (e.g., if the graph was instantiated with one or more flags, such as those described herein). In at least one embodiment, a deallocation operation (e.g., a MemFree node or other memory deallocation operation) must be ordered after all operations that access memory through graph dependencies, various events, and / or other mechanisms (e.g., stream ordering mechanisms). In at least one embodiment, an allocation can be accessed in another graph or directly in a stream operation, as long as the access operation is ordered after the allocation through events and stream ordering mechanisms.

[0124] In at least one embodiment, graph allocations share underlying physical memory with each other. In at least one embodiment, free operations must be ordered after all device operations (e.g., compute kernels, memory copy operations, and / or variants thereof) have completed. In at least one embodiment, out-of-band operations, such as writes to system memory as part of a compute kernel that writes to graph memory, may not be sufficient to provide ordering guarantees between memory writes to graph memory and free operations for that graph memory.

[0125] In at least one embodiment, one or more systems access and free allocated memory for graphs in the same stream through the following code, although any variation thereof may be used:

number

[0126] In at least one embodiment, one or more systems access and free a graph's allocated memory from other streams and other graphs through the following code, although any variation thereof may be used:

number

[0127] In at least one embodiment, one or more systems establish a dependency for accessing memory from other streams using a graph event node through the following code, although any variation thereof may be utilized:

number

[0128] In at least one embodiment, one or more systems provide functionality for sharing physical allocations between graphs. In at least one embodiment, an application can utilize multiple streams. In at least one embodiment, using the same graph with different graph memory nodes in multiple streams can cause the driver to thrash mappings. In at least one embodiment, an application that does not free memory in an allocation graph can impose serialization on other allocation graphs.

[0129] In at least one embodiment, physical memory is not allocated or mapped during graph instantiation. In at least one embodiment, the first graph upload or invocation incurs allocation and mapping costs. In at least one embodiment, one or more systems upload the graphs to the streams where they will be used, so that when allocated graphs are invoked on different streams, one or more systems may perform remapping.

[0130] In at least one embodiment, remapping costs may be incurred during the graph invocation waiting for stream-ordered deallocation of the graph's allocations, during streams waiting for previous uses of physical memory to complete, and / or during the execution time of OS calls to allocate, map, and unmap physical memory. In at least one embodiment, the memory remapping costs paid when the graph switches streams are represented by one or more systems through the following code, although any variation thereof may be utilized:

number

[0131] In at least one embodiment, memory reallocated to an alternate stream that causes other graph invocations in the original stream to pay the remapping cost is represented by one or more systems through the following code, although any variation thereof may be utilized:

number

[0132] In at least one embodiment, memory remapping with non-deallocated allocations is represented by one or more systems through the following code, although any variation thereof may be utilized:

number

[0133] In at least one embodiment, memory serialization with allocations freed in separate streams is represented by one or more systems through the following code, although any variation thereof may be utilized:

number

[0134] In at least one embodiment, destroying an allocation graph prevents one or more systems from returning the allocated memory to the OS for use by other processes. In at least one embodiment, to release memory to the OS, an application must use one or more API functions, such as the cudaDeviceGraphMemTrim API function. In at least one embodiment, cudaDeviceGraphMemTrim unmaps and releases reserved physical memory for graph memory nodes that are safe to unmap. In at least one embodiment, memory that is not actively in use (e.g., allocations that have not been released and scheduled or running graphs are considered to be actively using physical memory) may be referred to as safe to unmap. In at least one embodiment, one or more API functions may make the physical memory available to other allocation API functions and other applications / processes, but may cause the driver to allocate and map memory when launching a graph that has a mapping to the released memory.

[0135] In at least one embodiment, one or more systems provide functionality for applications to query their graph memory footprint through an API function denoted as cudaDeviceGetGraphMemAttribute. In at least one embodiment, querying an attribute denoted as cudaGraphMemAttrReservedMemCurrent returns the amount of physical memory reserved by one or more drivers for graph allocations in the current process. In at least one embodiment, querying an attribute denoted as cudaGraphMemAttrUsedMemCurrent returns the amount of physical memory currently mapped by at least one graph. In at least one embodiment, various attributes may be utilized to track when new physical memory is acquired by the driver for an allocated graph. In at least one embodiment, various attributes may be utilized to determine how much memory is saved by a sharing mechanism.

[0136] In at least one embodiment, one or more systems provide functionality for configuring graph allocations for access from multiple GPUs. In at least one embodiment, the driver maps allocations to one or more GPUs as needed. In at least one embodiment, the driver provides functionality for graph allocations requiring different mappings to reuse the same virtual address, and when this occurs, VAs are mapped on the set of GPUs required by the different allocations. In at least one embodiment, the set of GPUs to which an allocation is mapped can respond to changes in mappings of other allocations or changes in the driver's sub-allocation heuristics. In at least one embodiment, when an application requests the correct mapping on all multi-GPU allocations, all necessary mappings will be performed by one or more systems.

[0137] In at least one embodiment, the cudaGraphAddMemAllocNode API function, or any suitable function, accepts the mapping request in the accessDescs array field of the node parameters structure. In at least one embodiment, the poolProps.location built-in structure specifies the resident device for allocation. In at least one embodiment, it is assumed that access from the allocating GPU is required, and therefore the application does not need to specify an entry for the resident device in the accessDescs array. In at least one embodiment, one or more systems implement peer access using the graph node API through the following code, although any variation thereof may be utilized:

number

[0138] In at least one embodiment, for stream capture, the allocating node records the peer accessibility of the allocation pool at the time of capture. In at least one embodiment, changing the peer accessibility of a stream-ordered allocation pool after an API call, such as a cudaMallocFromPoolAsync call, is captured does not affect the mapping that the graph will make for allocation. In at least one embodiment, one or more systems implement peer access with stream capture through the following code, although any variation thereof may be utilized:

number

[0139] In at least one embodiment, graph capture (e.g., using stream capture to generate the graph) processes a region of execution and encodes various operations and utilized virtual addresses of one or more systems into the graph. In at least one embodiment, the graph may be utilized one or more times. In at least one embodiment, capture encodes memory addresses, and memory utilized during capture must be available for the graph to utilize during replay. In at least one embodiment, one or more systems dynamically allocate and free memory. In at least one embodiment, the graph's memory may be utilized by various other operations. In at least one embodiment, to ensure the graph's encoded addresses are safe for reuse in replay, one or more systems fulfill allocations from a graph private memory pool during capture and do not begin freeing those addresses until the graph is destroyed. In at least one embodiment, allocations within the private pool are freed and reallocated by one or more systems during capture. In at least one embodiment, memory regions are used in a consistent order by one or more systems during replay. In at least one embodiment, the private pool reserves a high-water mark of used memory off the default pool for as long as the capture(s) it serviced remain, regardless of whether those captures are idle or being replayed. In at least one embodiment, the graph's requests for private pools are mediated by one or more systems, which may be represented as DeviceAllocator (e.g., DeviceAllocator::notifyCaptureBegin, notifyCaptureEnd, and / or notifyCaptureDestroy).

[0140] In at least one embodiment, a graph can allocate and free memory through nodes that indicate the allocate and free operations. In at least one embodiment, when one or more systems allocate memory using the graph, a pointer is returned at node creation time that can be passed as an argument to subsequent nodes, and dereferencing the pointer is only allowed downstream of the allocating node (e.g., a MemAlloc node) and upstream of the freeing node (e.g., a MemFree node). In at least one embodiment, one or more systems provide each graph with a unique VA range. In at least one embodiment, virtual address ranges returned from in-graph allocations come exclusively from that graph's address pool and persist for the lifetime of the graph. In at least one embodiment, graphs can share physical allocations; contents are not preserved even between invocations of the same graph. In at least one embodiment, allocation lifetimes can extend outside of the graph. In at least one embodiment, one or more systems allow allocation in one graph and deallocation in another graph, however, the allocation graph must not be started again until a deallocation operation is performed.

[0141] In at least one embodiment, edges in a graph with memory nodes may not be modified after creation. In at least one embodiment, altering an edge may cause an upstream release node to no longer be upstream. In at least one embodiment, an allocation node causes inter-graph serialization, and one or more systems provide functionality for managing private / shared backing memory and when allocations occur. In at least one embodiment, inter-process communication (IPC) shareability must be defined by one or more systems at allocation time, and IPC shareable allocations must persist beyond the lifetime of the allocation graph.

[0142] In at least one embodiment, the allocation and deallocation may occur in the same graph. In at least one embodiment, the allocation may occur in one graph and the deallocation may occur in another graph. In at least one embodiment, the allocation may occur in one graph and the deallocation may occur via one or more API functions.

[0143] In at least one embodiment, virtual address lifetimes and physical address lifetimes are different for graphs. In at least one embodiment, each graph has a private virtual address range. In at least one embodiment, physical pages may be mapped by one or more systems at graph node creation, and virtual addresses may be returned. In at least one embodiment, virtual addresses remain valid for the lifetime (e.g., execution lifetime) of the graph. In at least one embodiment, per-graph virtual address ranges ensure that pointer lifetimes have the graph lifetime. In at least one embodiment, allocation and mapping may occur at graph instantiation, where memory is retained by one or more systems for the lifetime of the graph, at graph activation, where activation latency may increase while memory is mapped, and / or variations thereof.

[0144] In at least one embodiment, one or more systems implement shared physical page mapping to reduce computing resources for creating one or more graphs. In at least one embodiment, each graph has a private virtual address range, and a pointer lifetime has the graph lifetime. In at least one embodiment, the one or more systems reserve a set of physical pages equal to the maximum memory requirement of the graph. In at least one embodiment, the one or more systems map all graphs to the same set of pages unless they are executing simultaneously. In at least one embodiment, the one or more systems implement pre-mapping of physical pages.

[0145] In at least one embodiment, a long-lived allocation refers to an allocation that is not released within the same graph. In at least one embodiment, the virtual address returned by the allocation remains fixed for the graph. In at least one embodiment, one or more systems configure the graph for pre-release allocations so that multiple invocations may be allowed. In at least one embodiment, one or more systems track page lifetimes on a per-graph basis.

[0146] In at least one embodiment, one or more systems perform allocation at instantiation time of the graphs, which may result in minimal latency between graphs (e.g., non-dependent pages). In at least one embodiment, one or more systems perform allocation and / or mapping at startup time of the graphs, which may require allocation between graphs for allocation and / or remapping. In at least one embodiment, one or more systems perform shared allocation, which may result in minimal latency between pre-mapped graphs at instantiation time.

[0147] In at least one embodiment, one or more systems concurrently invoke graphs, which may require unique physical pages per concurrent graph. In at least one embodiment, one or more systems create a physical page pool per stream, which allows sharing between graphs in the same stream, except for concurrency between streams, which may result in increased latency for the first invocation in a new stream, although pre-allocation may be implemented. In at least one embodiment, one or more systems control when to free up stream allocations, allowing repeated invocations to keep the physical allocation and allowing a single invocation to free up the allocation.

[0148] In at least one embodiment, one or more systems provide functionality for updating a graph through one or more stream capture operations. In at least one embodiment, stream capture creates a new graph with new VA ranges. In at least one embodiment, graph update replaces the VA ranges of the original graph with the VA ranges of the updated graph. In at least one embodiment, one or more systems prevent single memory node parameter updates, while in at least one embodiment, such operations are allowed. In at least one embodiment, one or more systems return the original graph VA to the graph system for reuse.

[0149] In at least one embodiment, one or more systems provide stream capture and explicit APIs for creating memory nodes in a graph, where the memory nodes follow the semantics of various API functions for allocating memory. In at least one embodiment, one or more systems provide a per-graph VA for asynchronous allocation established at graph creation time. In at least one embodiment, the one or more systems maintain physical pages equal to the size of the largest instantiated graph. In at least one embodiment, the one or more systems implement a shared mapping for the graph VA to a physical page pool at instantiation time. In at least one embodiment, the one or more systems create one or more per-stream physical page pools upon first starting a new stream. In at least one embodiment, the one or more systems provide functionality for updating the graph and resizing page pools, such as when the memory footprint grows. In at least one embodiment, memory footprint refers to the amount of memory a program uses, references, or otherwise utilizes in various states, such as while running or otherwise executing.

[0150] In at least one embodiment, one or more systems provide functionality for performing various memory allocation operations in the graph through allocation and / or freeing nodes (e.g., MemAlloc and MemFree nodes, respectively). In at least one embodiment, one or more systems exploit dependencies between nodes to track memory reuse within the graph. In at least one embodiment, each graph has a private virtual address range, allowing allocations to have fixed addresses for the lifetime of the graph. In at least one embodiment, all graphs invoked on a given stream have their virtual footprints aliased by one or more systems onto a shared, per-stream physical memory, enabling physical memory reuse between graphs. In at least one embodiment, when the invoking stream of the graph changes and / or when memory is freed outside of the allocation graph, one or more systems (e.g., drivers) track the mapping to physical memory and physical memory usage to enable reuse. In at least one embodiment, the allocator is implemented in one or more drivers, which enables the use of low-level memory operations to limit fragmentation and the use of various information about memory consumption, stream dependencies, and / or work completion to detect opportunities for memory reclamation.

[0151] It should be noted that API functions and other related terminology, such as parameters, variable names, and / or variations thereof, may be referred to in any suitable manner using any suitable terminology, which may or may not relate to one or more functionality of the API functions. Additionally, it should be noted that while the exemplary embodiments described herein may relate to the CUDA programming model, the techniques described herein may be utilized with any suitable programming model and / or any suitable API of any suitable programming model, such as CUDA, HIP, oneAPI, and / or variations thereof.

[0152] In the preceding and following descriptions, numerous specific details are set forth in order to provide a more thorough understanding of at least one embodiment. However, it will be apparent to one skilled in the art that the inventive concept may be practiced without one or more of these specific details.

[0153] Data Center 11 illustrates an exemplary data center 1100, according to at least one embodiment. In at least one embodiment, the data center 1100 includes, but is not limited to, a data center infrastructure layer 1110, a framework layer 1120, a software layer 1130, and an application layer 1140.

[0154] In at least one embodiment, as shown in FIG. 11 , data center infrastructure layer 1110 may include a resource orchestrator 1112, grouped computing resources 1114, and node computing resources (“node CR”) 1116(1) through 1116(N), where “N” represents any positive integer. In at least one embodiment, nodes CR 1116(1)-1116(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including accelerators, field programmable gate arrays (“FPGAs”), data processing units (“DPUs”) in network devices, graphics processors, etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules, cooling modules, etc. In at least one embodiment, one or more nodes CR from among nodes CR 1116(1)-1116(N) may be a server having one or more of the computing resources described above.

[0155] In at least one embodiment, the grouped computing resources 1114 may include a distinct grouping of node CRs housed within one or more racks (not shown), or many racks housed in a data center at various geographic locations (also not shown). The distinct groupings of node CRs within the grouped computing resources 1114 may include grouped compute resources, network resources, memory resources, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several node CRs, including CPUs or processors, may be grouped within one or more racks to provide compute resources to support one or more workloads. In at least one embodiment, the one or more racks may also include any number of power modules, cooling modules, and network switches, in any combination.

[0156] In at least one embodiment, resource orchestrator 1112 may configure or otherwise control one or more nodes CR 1116(1)-1116(N) and / or grouped computing resources 1114. In at least one embodiment, resource orchestrator 1112 may include a software design infrastructure (“SDI”) management entity for data center 1100. In at least one embodiment, resource orchestrator 1112 may include hardware, software, or some combination thereof.

[0157] 11 , framework layer 1120 may include, but is not limited to, a job scheduler 1132, a configuration manager 1134, a resource manager 1136, and a distributed file system 1138. In at least one embodiment, framework layer 1120 may include a framework for supporting software 1152 in software layer 1130 and / or one or more applications 1142 in application layer 1140. In at least one embodiment, software 1152 or application(s) 1142 may include web-based service software or applications, such as those offered by Amazon Web Services, Google Cloud, and Microsoft Azure, respectively. In at least one embodiment, framework layer 1120 may be a type of free and open-source software web application framework, such as, but not limited to, Apache Spark™ (hereinafter “Spark”), which may utilize distributed file system 1138 for large-scale data processing (e.g., “big data”). In at least one embodiment, job scheduler 1132 may include a Spark driver to facilitate scheduling of workloads supported by various tiers of data center 1100. In at least one embodiment, configuration manager 1134 may be capable of configuring different tiers, such as software tier 1130 and framework tier 1120, which includes Spark and distributed file system 1138 to support large-scale data processing. In at least one embodiment, resource manager 1136 may be capable of managing clustered or grouped computing resources that are mapped or allocated to support distributed file system 1138 and job scheduler 1132.In at least one embodiment, the clustered or grouped computing resources may include grouped computing resources 1114 in data center infrastructure layer 1110. In at least one embodiment, resource manager 1136 may manage these mapped or allocated computing resources in coordination with resource orchestrator 1112.

[0158] In at least one embodiment, software 1152 included in software layer 1130 may include software used by nodes CR 1116(1)-1116(N), grouped computing resources 1114, and / or at least a portion of distributed file system 1138 of framework layer 1120. The one or more types of software may include, but are not limited to, internet web page searching software, email virus scanning software, database software, and streaming video content software.

[0159] In at least one embodiment, the application(s) 1142 included in the application layer 1140 may include one or more types of applications used by the nodes CR 1116(1)-1116(N), the grouped computing resources 1114, and / or at least a portion of the distributed file system 1138 of the framework layer 1120. At least one or more types of applications may include, but are not limited to, CUDA applications.

[0160] In at least one embodiment, any of configuration manager 1134, resource manager 1136, and resource orchestrator 1112 may implement any number and types of self-correcting actions based on any amount and type of data obtained in any technically feasible manner. In at least one embodiment, the self-correcting actions may relieve data center operators of data center 1100 from determining potentially faulty configurations and potentially avoiding underutilized and / or underperforming portions of the data center.

[0161] In at least one embodiment, one or more systems illustrated in Figure 11 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 11 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 11 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 11 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0162] Computer-Based Systems The following diagram describes an exemplary computer-based system that may be used to implement at least one embodiment, but is not limited to the embodiment.

[0163] 12 illustrates a processing system 1200, according to at least one embodiment. In at least one embodiment, processing system 1200 includes one or more processors 1202 and one or more graphics processors 1208 and may be a single-processor desktop system, a multiprocessor workstation system, or a server system having multiple processors 1202 or processor cores 1207. In at least one embodiment, processing system 1200 is a processing platform integrated into a system-on-a-chip ("SoC") integrated circuit for use in a mobile, handheld, or embedded device.

[0164] In at least one embodiment, processing system 1200 may include or be incorporated within a server-based gaming platform, a game console, a media console, a mobile gaming console, a handheld game console, or an online game console. In at least one embodiment, processing system 1200 is a mobile phone, a smartphone, a tablet computing device, or a mobile internet device. In at least one embodiment, processing system 1200 may also include, be coupled to, or be incorporated within a wearable device, such as a smart watch wearable device, a smart eyewear device, an augmented reality device, or a virtual reality device. In at least one embodiment, processing system 1200 is a television or set-top box device having one or more processors 1202 and a graphical interface generated by one or more graphics processors 1208.

[0165] In at least one embodiment, the one or more processors 1202 each include one or more processor cores 1207 for processing instructions that, when executed, perform operations for system and user software. In at least one embodiment, each of the one or more processor cores 1207 is configured to process a particular instruction set 1209. In at least one embodiment, the instruction set 1209 may facilitate Complex Instruction Set Computing ("CISC"), Reduced Instruction Set Computing ("RISC"), or computing via Very Long Instruction Words ("VLIW"). In at least one embodiment, the processor cores 1207 may each process a different instruction set 1209, and the instruction set 1209 may include instructions to facilitate emulation of other instruction sets. In at least one embodiment, the processor cores 1207 may also include other processing devices, such as a digital signal processor ("DSP").

[0166] In at least one embodiment, processor 1202 includes cache memory (“cache”) 1204. In at least one embodiment, processor 1202 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, cache memory is shared among various components of processor 1202. In at least one embodiment, processor 1202 also uses an external cache (e.g., a level 3 (“L3”) cache or a last level cache (“LLC”)) (not shown), which may be shared among processor cores 1207 using known cache coherency techniques. In at least one embodiment, processor 1202 additionally includes register file 1206, which may include different types of registers (e.g., integer registers, floating-point registers, status registers, and instruction pointer registers) for storing different types of data. In at least one embodiment, register file 1206 may include general-purpose registers or other registers.

[0167] In at least one embodiment, one or more processors 1202 are coupled to one or more interface buses 1210 for transmitting communication signals, such as address, data, or control signals, between processor(s) 1202 and other components in processing system 1200. In at least one embodiment, interface bus 1210 in one embodiment may be a processor bus, such as a version of a Direct Media Interface ("DMI") bus. In at least one embodiment, interface bus 1210 is not limited to a DMI bus, but may include one or more peripheral component interconnect buses (e.g., Peripheral Component Interconnect, PCI Express ("PCIe")), memory buses, or other types of interface buses. In at least one embodiment, processor(s) 1202 includes an integrated memory controller 1216 and platform controller hub 1230. In at least one embodiment, memory controller 1216 facilitates communication between memory devices and other components of processing system 1200, and platform controller hub (“PCH”) 1230 provides connection to I / O devices via a local input / output (“I / O”) bus.

[0168] In at least one embodiment, memory device 1220 may be a dynamic random access memory ("DRAM") device, a static random access memory ("SRAM") device, a flash memory device, a phase-change memory device, or any other memory device with performance suitable for serving as processor memory. In at least one embodiment, memory device 1220 may operate as system memory for processing system 1200 to store data 1222 and instructions 1221 for use when one or more processors 1202 execute applications or processes. In at least one embodiment, memory controller 1216 also couples to an optional external graphics processor 1212, which may communicate with one or more graphics processors 1208 in processor 1202 to perform graphics and media operations. In at least one embodiment, display device 1211 may be connected to processor(s) 1202. In at least one embodiment, display device 1211 may include one or more of an internal display device, such as in a mobile electronic device or laptop device, or an external display device attached via a display interface (e.g., DisplayPort, etc.). In at least one embodiment, display device 1211 may include a head mounted display ("HMD"), such as a stereoscopic display device for use in virtual reality ("VR") or augmented reality ("AR") applications.

[0169] In at least one embodiment, platform controller hub 1230 allows peripherals to connect to memory device 1220 and processor 1202 via a high-speed I / O bus. In at least one embodiment, the I / O peripherals include, but are not limited to, an audio controller 1246, a network controller 1234, a firmware interface 1228, a wireless transceiver 1226, a touch sensor 1225, and a data storage device 1224 (e.g., a hard disk drive, flash memory, etc.). In at least one embodiment, data storage device 1224 can be connected via a storage interface (e.g., SATA) or via a peripheral bus such as PCI or PCIe. In at least one embodiment, touch sensor 1225 can include a touch screen sensor, a pressure sensor, or a fingerprint sensor. In at least one embodiment, wireless transceiver 1226 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, or Long Term Evolution ("LTE") transceiver. In at least one embodiment, firmware interface 1228 enables communication with system firmware and may be, for example, a unified extensible firmware interface ("UEFI"). In at least one embodiment, network controller 1234 may enable network connectivity to a wired network. In at least one embodiment, a high-performance network controller (not shown) couples to interface bus 1210. In at least one embodiment, audio controller 1246 is a multi-channel high-definition audio controller.In at least one embodiment, processing system 1200 includes an optional legacy I / O controller 1240 for coupling legacy (e.g., Personal System 2 (“PS / 2”)) devices to processing system 1200. In at least one embodiment, platform controller hub 1230 can also connect to one or more Universal Serial Bus (“USB”) controller 1242-connected input devices, such as a keyboard and mouse 1243 combination, a camera 1244, or other USB input devices.

[0170] In at least one embodiment, instances of memory controller 1216 and platform controller hub 1230 may be embedded in a discreet external graphics processor, such as external graphics processor 1212. In at least one embodiment, platform controller hub 1230 and / or memory controller 1216 may be external to one or more processors 1202. For example, in at least one embodiment, processing system 1200 may include external memory controller 1216 and platform controller hub 1230, which may be configured as a memory controller hub and a peripheral controller hub within a system chipset in communication with processor(s) 1202.

[0171] In at least one embodiment, one or more systems illustrated in Figure 12 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 12 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 12 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 12 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0172] FIG. 13 illustrates a computer system 1300 according to at least one embodiment. In at least one embodiment, computer system 1300 may be a system, a SOC, or some combination with interconnected devices and components. In at least one embodiment, computer system 1300 is formed with a processor 1302, which may include an execution unit for executing instructions. In at least one embodiment, computer system 1300 may include components, such as, but not limited to, processor 1302, to employ an execution unit that includes logic for implementing algorithms for processing data. In at least one embodiment, computer system 1300 may include a processor such as the PENTIUM® processor family, Xeon™, Itanium®, XScale™, and / or StrongARM™, Intel® Core™, or Intel® Nervana™ microprocessors available from Intel Corporation of Santa Clara, California, although other systems (including PCs with other microprocessors, engineering workstations, set-top boxes, etc.) may also be used. In at least one embodiment, computer system 1300 may run a version of the WINDOWS® operating system available from Microsoft Corporation of Redmond, Washington, although other operating systems (e.g., UNIX® and Linux®), embedded software, and / or graphical user interfaces may also be used.

[0173] In at least one embodiment, computer system 1300 may be used in other devices, such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol devices, digital cameras, personal digital assistants ("PDAs"), and handheld PCs. In at least one embodiment, embedded applications may include a microcontroller, a digital signal processor (DSP), an SoC, a network computer ("NetPC"), a set-top box, a network hub, a wide area network ("WAN") switch, or any other system capable of performing one or more instructions.

[0174] In at least one embodiment, computer system 1300 may include, but is not limited to, a processor 1302, which may include one or more execution units 1308 that may be configured to execute, but is not limited to, a Compute Unified Device Architecture ("CUDA") (CUDA® is developed by NVIDIA Corporation of Santa Clara, California) programs. In at least one embodiment, the CUDA program is at least a portion of a software application written in the CUDA programming language. In at least one embodiment, computer system 1300 is a single-processor desktop or server system. In at least one embodiment, computer system 1300 may be a multiprocessor system. In at least one embodiment, processor 1302 may include, but is not limited to, a CISC microprocessor, a RISC microprocessor, a VLIW microprocessor, a processor implementing a combination of instruction sets, or any other processor device, such as, for example, a digital signal processor. In at least one embodiment, the processor 1302 may be coupled to a processor bus 1310 that may transmit data signals between the processor 1302 and other components in the computer system 1300.

[0175] In at least one embodiment, processor 1302 may include, but is not limited to, level 1 ("L1") internal cache memory ("cache") 1304. In at least one embodiment, processor 1302 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, cache memory may reside external to processor 1302. In at least one embodiment, processor 1302 may also include a combination of both internal and external cache. In at least one embodiment, register file 1306 may store different types of data in various registers, including, but not limited to, integer registers, floating-point registers, status registers, and instruction pointer registers.

[0176] In at least one embodiment, an execution unit 1308, including logic for performing integer and floating-point operations, may also be present in the processor 1302. The processor 1302 may also include a microcode (“u-code”) read-only memory (“ROM”) that stores microcode for some macroinstructions. In at least one embodiment, the execution unit 1308 may include logic for dealing with a packed instruction set 1309. In at least one embodiment, by including the packed instruction set 1309, along with associated circuitry for executing the instructions, in the instruction set of the general-purpose processor 1302, operations used by many multimedia applications may be performed using packed data in the general-purpose processor 1302. In at least one embodiment, many multimedia applications may be accelerated and run more efficiently by using the full width of the processor's data bus to perform operations on packed data, which may eliminate the need to transfer smaller units of data across the processor's data bus to perform one or more operations one data element at a time.

[0177] In at least one embodiment, the execution unit 1308 may also be used in a microcontroller, an embedded processor, a graphics device, a DSP, and other types of logic circuitry. In at least one embodiment, the computer system 1300 may include, but is not limited to, a memory 1320. In at least one embodiment, the memory 1320 may be implemented as a DRAM device, an SRAM device, a flash memory device, or other memory device. The memory 1320 may store instruction(s) 1319 and / or data 1321 represented by data signals that may be executed by the processor 1302.

[0178] In at least one embodiment, a system logic chip may be coupled to the processor bus 1310 and the memory 1320. In at least one embodiment, the system logic chip may include, but is not limited to, a memory controller hub (“MCH”) 1316, and the processor 1302 may communicate with the MCH 1316 via the processor bus 1310. In at least one embodiment, the MCH 1316 may provide a high-bandwidth memory path 1318 to the memory 1320 for instruction and data storage, and for storage of graphics commands, data, and textures. In at least one embodiment, the MCH 1316 may direct data signals between the processor 1302, the memory 1320, and other components in the computer system 1300, and may bridge data signals between the processor bus 1310, the memory 1320, and the system I / O 1322. In at least one embodiment, the system logic chip may provide a graphics port for coupling to a graphics controller. In at least one embodiment, the MCH 1316 may be coupled to memory 1320 through a high-bandwidth memory path 1318, and the graphics / video card 1312 may be coupled to the MCH 1316 via an Accelerated Graphics Port (“AGP”) interconnect 1314.

[0179] In at least one embodiment, computer system 1300 may use system I / O 1322, a proprietary hub interface bus, to couple MCH 1316 to I / O controller hub (“ICH”) 1330. In at least one embodiment, ICH 1330 may provide direct connectivity to several I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus may include, but is not limited to, a high-speed I / O bus for connecting peripherals to memory 1320, a chipset, and processor 1302. Examples may include, but are not limited to, an audio controller 1329, a firmware hub ("flash BIOS") 1328, a wireless transceiver 1326, data storage 1324, a legacy I / O controller 1323 including a user input interface 1325 and a keyboard interface, a serial expansion port 1327 such as USB, and a network controller 1334. Data storage 1324 may comprise a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0180] In at least one embodiment, FIG. 13 illustrates a system including interconnected hardware devices or "chips." In at least one embodiment, FIG. 13 may illustrate an exemplary SoC. In at least one embodiment, the devices illustrated in FIG. 13 may be interconnected with a proprietary interconnect, a standard interconnect (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of system 1300 are interconnected using a Compute Express Link ("CXL") interconnect.

[0181] In at least one embodiment, one or more systems illustrated in Figure 13 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 13 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 13 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 13 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0182] 14 illustrates a system 1400, according to at least one embodiment. In at least one embodiment, system 1400 is an electronic device utilizing a processor 1410. In at least one embodiment, system 1400 may be, for example, but not limited to, a notebook, a tower server, a rack server, a blade server, an edge device communicatively coupled to one or more on-premise or cloud service providers, a laptop, a desktop, a tablet, a mobile device, a phone, an embedded computer, or any other suitable electronic device.

[0183] In at least one embodiment, system 1400 may include, without limitation, a processor 1410 communicatively coupled to any suitable number or type of components, peripherals, modules, or devices. 2The devices may be coupled using a bus or interface, such as a C bus, a System Management Bus (“SMBus”), a Low Pin Count (“LPC”) bus, a Serial Peripheral Interface (“SPI”), a High Definition Audio (“HDA”) bus, a Serial Advance Technology Attachment (“SATA”) bus, USB (versions 1, 2, and 3), or a Universal Asynchronous Receiver / Transmitter (“UART”) bus. In at least one embodiment, FIG. 14 illustrates a system including interconnected hardware devices or “chips.” In at least one embodiment, FIG. 14 may illustrate an exemplary SoC. In at least one embodiment, the devices illustrated in FIG. 14 may be interconnected with a proprietary interconnect, a standard interconnect (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of FIG. 14 are interconnected using a CXL interconnect.

[0184] In at least one embodiment, FIG. 14 includes a display 1424, a touch screen 1425, a touch pad 1430, a Near Field Communication ("NFC") unit 1445, a sensor hub 1440, a thermal sensor 1446, an Express Chipset ("EC") 1435, a Trusted Platform Module ("TPM") 1438, a BIOS / firmware / flash memory ("BIOS,FW flash") 1422, a DSP 1460, a Solid State Disk ("SSD") or Hard Disk Drive ("HDD") 1420, a Wireless Local Area Network ("WLAN") unit 1450, a Bluetooth unit 1452, a Wireless Wide Area Network ("WWAN") unit 1454, a Bluetooth® unit 1456, a Bluetooth® unit 1458, a Bluetooth® unit 1459, a Bluetooth® unit 1460, a Bluetooth® unit 1461, a Bluetooth® unit 1462, a Bluetooth® unit 1463, a Bluetooth® unit 1464, a Bluetooth® unit 1465, a Bluetooth® unit 1466, a Bluetooth® unit 1468, a Bluetooth® unit 1469, a Bluetooth® unit 1470, a Bluetooth® unit 1472, a Bluetooth® unit 1474, a Bluetooth® unit 1476, a Bluetooth® unit 1478, a Bluetooth® unit 1479, a Bluetooth® unit 1480, a Bluetooth® unit 1482, a Bluetooth® unit 1484, a Bluetooth® unit 1486, a Bluetooth® unit 1488, a Bluetooth® unit 1488, a Bluetooth® unit 1488, a Bluetooth® unit 1489, a Bluetooth® unit 1490, a Bluetooth® unit 1492, a Bluetooth® unit 1494, a Bluetooth® unit 1496, a Bluetooth® unit 1 Network) 1456, Global Positioning System (“GPS”) 1455, a camera such as a USB 3.0 camera (“USB 3.0 Camera”) 1454, or a Low Power Double Data Rate (“LPDDR”) memory unit (“LPDDR3”) implemented, for example, in the LPDDR3 standard. Each of these components may be implemented in any suitable manner.

[0185] In at least one embodiment, other components may be communicatively coupled to the processor 1410 through the components described above. In at least one embodiment, an accelerometer 1441, an ambient light sensor (“ALS”) 1442, a compass 1443, and a gyroscope 1444 may be communicatively coupled to the sensor hub 1440. In at least one embodiment, a thermal sensor 1439, a fan 1437, a keyboard 1436, and a touchpad 1430 may be communicatively coupled to the EC 1435. In at least one embodiment, a speaker 1463, headphones 1464, and a microphone (“mic”) 1465 may be communicatively coupled to an audio unit (“audio codec and class D amplifier”) 1462, which may be communicatively coupled to the DSP 1460. In at least one embodiment, the audio unit 1462 may include, for example, without limitation, an audio coder / decoder ("codec") and a Class D amplifier. In at least one embodiment, a SIM card ("SIM") 1457 may be communicatively coupled to the WWAN unit 1456. In at least one embodiment, components such as the WLAN unit 1450 and Bluetooth unit 1452, and the WWAN unit 1456 may be implemented in a Next Generation Form Factor ("NGFF").

[0186] In at least one embodiment, one or more systems illustrated in Figure 14 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 14 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 14 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 14 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0187] FIG. 15 illustrates an exemplary integrated circuit 1500, according to at least one embodiment. In at least one embodiment, the exemplary integrated circuit 1500 is an SoC that may be fabricated using one or more IP cores. In at least one embodiment, the integrated circuit 1500 includes one or more application processors 1505 (e.g., CPU, DPU), at least one graphics processor 1510, and may additionally include an image processor 1515 and / or a video processor 1520, any of which may be modular IP cores. In at least one embodiment, the integrated circuit 1500 includes a USB controller 1525, a UART controller 1530, an SPI / SDIO controller 1535, and an I / O controller 1540. 2 S / I 2The integrated circuit 1500 may include peripheral or bus logic including a C controller 1540. In at least one embodiment, the integrated circuit 1500 may include a display device 1545 coupled to one or more of a high-definition multimedia interface ("HDMI") controller 1550 and a mobile industry processor interface ("MIPI") display interface 1555. In at least one embodiment, storage may be provided by a flash memory subsystem 1560 including a flash memory and a flash memory controller. In at least one embodiment, a memory interface may be provided via a memory controller 1565 for access to an SDRAM or SRAM memory device. In at least one embodiment, some integrated circuits additionally include an embedded security engine 1570.

[0188] In at least one embodiment, one or more systems illustrated in Figure 15 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 15 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 15 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 15 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0189] FIG. 16 illustrates a computing system 1600, according to at least one embodiment. In at least one embodiment, the computing system 1600 includes a processing subsystem 1601 having one or more processors 1602 and system memory 1604 that communicate via an interconnection path that may include a memory hub 1605. In at least one embodiment, the memory hub 1605 may be a separate component within a chipset component or may be incorporated within the one or more processors 1602. In at least one embodiment, the memory hub 1605 couples to an I / O subsystem 1611 via a communication link 1606. In at least one embodiment, the I / O subsystem 1611 includes an I / O hub 1607 that may enable the computing system 1600 to receive input from one or more input devices 1608. In at least one embodiment, I / O hub 1607 can enable a display controller, which may be included in one or more processors 1602, to provide output to one or more display devices 1610A. In at least one embodiment, the one or more display devices 1610A coupled with I / O hub 1607 can include local, internal, or embedded display devices.

[0190] In at least one embodiment, processing subsystem 1601 includes one or more parallel processors 1612 coupled to memory hub 1605 via a bus or other communication link 1613. In at least one embodiment, communication link 1613 may be one of any number of standards-based communication link technologies or protocols, such as, but not limited to, PCIe, or may be a vendor-specific communication interface or fabric. In at least one embodiment, one or more parallel processors 1612 form a computationally intensive parallel or vector processing system that may include multiple processing cores and / or processing clusters, such as a many-integrated-core processor. In at least one embodiment, one or more parallel processors 1612 form a graphics processing subsystem that can output pixels to one of one or more display devices 1610A coupled via I / O hub 1607. In at least one embodiment, the one or more parallel processors 1612 may also include a display controller and a display interface (not shown) to enable direct connection to one or more display devices 1610B.

[0191] In at least one embodiment, system storage unit 1614 may connect to I / O hub 1607 to provide storage for computing system 1600. In at least one embodiment, I / O switch 1616 may be used to provide an interface mechanism to enable connections between I / O hub 1607 and other components, such as network adapter 1618 and / or wireless network adapter 1619, which may be embedded in the platform, as well as various other devices, which may be added via one or more add-in devices 1620. In at least one embodiment, network adapter 1618 may be an Ethernet adapter or another wired network adapter. In at least one embodiment, wireless network adapter 1619 may include one or more of Wi-Fi, Bluetooth, NFC, or other network devices including one or more wireless radios.

[0192] In at least one embodiment, computing system 1600 may include other components not expressly shown that may also be connected to I / O hub 1607, including USB or other port connections, optical storage drives, video capture devices, etc. In at least one embodiment, the communication paths interconnecting the various components in FIG. 16 may be implemented using any suitable protocol, such as a PCI-based protocol (e.g., PCIe), or other bus or point-to-point communication interface and / or protocol(s), such as an NVLink high-speed interconnect, or interconnect protocol.

[0193] In at least one embodiment, one or more parallel processors 1612 incorporate circuitry optimized for graphics and video processing, including, for example, video output circuitry, to form a graphics processing unit ("GPU"). In at least one embodiment, one or more parallel processors 1612 incorporate circuitry optimized for general-purpose processing. In at least one embodiment, components of computing system 1600 may be integrated with one or more other system elements on a single integrated circuit. For example, in at least one embodiment, one or more parallel processors 1612, memory hub 1605, processor(s) 1602, and I / O hub 1607 may be integrated into an SoC integrated circuit. In at least one embodiment, components of computing system 1600 may be integrated into a single package to form a system-in-package ("SIP") configuration. In at least one embodiment, at least a portion of the components of computing system 1600 may be incorporated into a multi-chip module ("MCM"), which may be interconnected with other multi-chip modules to form a modular computing system. In at least one embodiment, I / O subsystem 1611 and display device 1610B are omitted from computing system 1600.

[0194] In at least one embodiment, one or more systems illustrated in Figure 16 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 16 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 16 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 16 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0195] Processing System The following figures describe exemplary processing systems that may be used to implement, but are not limited to, at least one embodiment.

[0196] FIG. 17 illustrates an accelerated processing unit (“APU”) 1700 according to at least one embodiment. In at least one embodiment, the APU 1700 is developed by AMD Corporation of Santa Clara, California. In at least one embodiment, the APU 1700 may be configured to execute application programs, such as CUDA programs. In at least one embodiment, the APU 1700 includes, but is not limited to, a core complex 1710, a graphics complex 1740, a fabric 1760, an I / O interface 1770, a memory controller 1780, a display controller 1792, and a multimedia engine 1794. In at least one embodiment, the APU 1700 may include, but is not limited to, any number of core complexes 1710, any number of graphics complexes 1750, any number of display controllers 1792, and any number of multimedia engines 1794 in any combination. For purposes of explanation, multiple instances of similar objects are referred to herein with a reference number that identifies the object and, where necessary, a parenthetical number that identifies the instance.

[0197] In at least one embodiment, core complex 1710 is a CPU, graphics complex 1740 is a GPU, and APU 1700 is a processing unit incorporating 1710 and 1740 on a single chip, without limitation. In at least one embodiment, some tasks may be assigned to core complex 1710 and other tasks may be assigned to graphics complex 1740. In at least one embodiment, core complex 1710 is configured to execute main control software associated with APU 1700, such as an operating system. In at least one embodiment, core complex 1710 is the master processor of APU 1700 and controls and coordinates the operation of the other processors. In at least one embodiment, core complex 1710 issues commands that control the operation of graphics complex 1740. In at least one embodiment, core complex 1710 may be configured to execute host executable code derived from CUDA source code, and graphics complex 1740 may be configured to execute device executable code derived from CUDA source code.

[0198] In at least one embodiment, core complex 1710 includes, but is not limited to, cores 1720(1)-1720(4) and an L3 cache 1730. In at least one embodiment, core complex 1710 may include, but is not limited to, any number of cores 1720 and any number and type of caches in any combination. In at least one embodiment, cores 1720 are configured to execute instructions of a particular instruction set architecture ("ISA"). In at least one embodiment, each core 1720 is a CPU core.

[0199] In at least one embodiment, each core 1720 includes, but is not limited to, a fetch / decode unit 1722, an integer execution engine 1724, a floating-point execution engine 1726, and an L2 cache 1728. In at least one embodiment, the fetch / decode unit 1722 fetches instructions, decodes such instructions, generates micro-ops, and dispatches separate micro-instructions to the integer execution engine 1724 and the floating-point execution engine 1726. In at least one embodiment, the fetch / decode unit 1722 can simultaneously dispatch one micro-instruction to the integer execution engine 1724 and another micro-instruction to the floating-point execution engine 1726. In at least one embodiment, the integer execution engine 1724 performs, but is not limited to, integer and memory operations. In at least one embodiment, the floating-point engine 1726 performs, but is not limited to, floating-point and vector operations. In at least one embodiment, the fetch decode unit 1722 dispatches microinstructions to a single execution engine that replaces both the integer execution engine 1724 and the floating-point execution engine 1726 .

[0200] In at least one embodiment, each core 1720(i), where i is an integer representing a particular instance of core 1720, may access an L2 cache 1728(i) included in core 1720(i). In at least one embodiment, each core 1720 included in a core complex 1710(j), where j is an integer representing a particular instance of core complex 1710, is connected to the other cores 1720 included in core complex 1710(j) via an L3 cache 1730(j) included in core complex 1710(j). In at least one embodiment, a core 1720 included in a core complex 1710(j), where j is an integer representing a particular instance of core complex 1710, may access all of the L3 caches 1730(j) included in core complex 1710(j). In at least one embodiment, the L3 cache 1730 may include, without limitation, any number of slices.

[0201] In at least one embodiment, the graphics complex 1740 may be configured to perform compute operations in a highly parallel manner. In at least one embodiment, the graphics complex 1740 is configured to execute graphics pipeline operations, such as drawing commands, pixel operations, geometric calculations, and other operations related to rendering an image to a display. In at least one embodiment, the graphics complex 1740 is configured to execute non-graphics related operations. In at least one embodiment, the graphics complex 1740 is configured to execute both graphics related and non-graphics related operations.

[0202] In at least one embodiment, the graphics complex 1740 includes, but is not limited to, any number of compute units 1750 and an L2 cache 1742. In at least one embodiment, the compute units 1750 share the L2 cache 1742. In at least one embodiment, the L2 cache 1742 is partitioned. In at least one embodiment, the graphics complex 1740 includes, but is not limited to, any number of compute units 1750 and any number and type of cache (including zero). In at least one embodiment, the graphics complex 1740 includes, but is not limited to, any amount of dedicated graphics hardware.

[0203] In at least one embodiment, each compute unit 1750 includes, but is not limited to, any number of SIMD units 1752 and shared memory 1754. In at least one embodiment, each SIMD unit 1752 may be configured to implement a SIMD architecture and perform operations in parallel. In at least one embodiment, each compute unit 1750 may execute any number of thread blocks, with each thread block executing on a single compute unit 1750. In at least one embodiment, a thread block includes, but is not limited to, any number of threads of execution. In at least one embodiment, a workgroup is a thread block. In at least one embodiment, each SIMD unit 1752 executes a different warp. In at least one embodiment, a warp is a group of threads (e.g., 16 threads), where each thread in a warp belongs to a single thread block and is configured to process a different set of data based on a single set of instructions. In at least one embodiment, predication may be used to disable one or more threads in a warp. In at least one embodiment, a lane is a thread. In at least one embodiment, a work item is a thread. In at least one embodiment, a wavefront is a warp. In at least one embodiment, different wavefronts in a thread block may synchronize with each other and communicate via shared memory 1754.

[0204] In at least one embodiment, fabric 1760 is a system interconnect that facilitates data and control transmission across core complex 1710, graphics complex 1740, I / O interface 1770, memory controller 1780, display controller 1792, and multimedia engine 1794. In at least one embodiment, APU 1700 may include any quantity and type of system interconnect in addition to or instead of fabric 1760, which facilitates data and control transmission across any number and type of directly or indirectly linked components, which may be internal or external to APU 1700. In at least one embodiment, I / O interface 1770 represents any number and type of I / O interface (e.g., PCI, PCI-Extended (“PCI-X”), PCIe, Gigabit Ethernet (“GBE”), USB, etc. In at least one embodiment, various types of peripheral devices are coupled to I / O interface 1770. In at least one embodiment, peripheral devices coupled to I / O interface 1770 may include, but are not limited to, a keyboard, a mouse, a printer, a scanner, a joystick or other type of game controller, a media recording device, an external storage device, a network interface card, etc.

[0205] In at least one embodiment, display controller 1792 displays images on one or more display devices, such as liquid crystal display ("LCD") devices. In at least one embodiment, multimedia engine 1794 includes any quantity and type of circuitry related to multimedia, such as, but not limited to, a video decoder, a video encoder, and an image signal processor. In at least one embodiment, memory controller 1780 facilitates data transfer between APU 1700 and unified system memory 1790. In at least one embodiment, core complex 1710 and graphics complex 1740 share unified system memory 1790.

[0206] In at least one embodiment, the APU 1700 implements a memory subsystem including, but not limited to, any quantity and type of memory controller 1780 and memory devices (e.g., shared memory 1754) that may be dedicated to one component or shared among multiple components. In at least one embodiment, the APU 1700 implements a cache subsystem including, but not limited to, one or more cache memories (e.g., L2 cache 1728, L3 cache 1730, and L2 cache 1742), each of which may be private to any number of components (e.g., cores 1720, core complexes 1710, SIMD units 1752, compute units 1750, and graphics complexes 1740) or shared among any number of components.

[0207] In at least one embodiment, one or more systems illustrated in Figure 17 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 17 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 17 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 17 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0208] FIG. 18 illustrates a CPU 1800 according to at least one embodiment. In at least one embodiment, the CPU 1800 is developed by AMD Corporation of Santa Clara, California. In at least one embodiment, the CPU 1800 may be configured to execute application programs. In at least one embodiment, the CPU 1800 may be configured to execute main control software, such as an operating system. In at least one embodiment, the CPU 1800 issues commands to control the operation of an external GPU (not shown). In at least one embodiment, the CPU 1800 may be configured to execute host executable code derived from CUDA source code, and the external GPU may be configured to execute device executable code derived from such CUDA source code. In at least one embodiment, the CPU 1800 includes, but is not limited to, any number of core complexes 1810, a fabric 1860, an I / O interface 1870, and a memory controller 1880.

[0209] In at least one embodiment, core complex 1810 includes, but is not limited to, cores 1820(1)-1820(4) and an L3 cache 1830. In at least one embodiment, core complex 1810 may include, but is not limited to, any number of cores 1820 and any number and type of caches in any combination. In at least one embodiment, cores 1820 are configured to execute instructions of a particular ISA. In at least one embodiment, each core 1820 is a CPU core.

[0210] In at least one embodiment, each core 1820 includes, but is not limited to, a fetch / decode unit 1822, an integer execution engine 1824, a floating point execution engine 1826, and an L2 cache 1828. In at least one embodiment, the fetch / decode unit 1822 fetches instructions, decodes such instructions, generates micro-ops, and dispatches separate micro-instructions to the integer execution engine 1824 and the floating point execution engine 1826. In at least one embodiment, the fetch / decode unit 1822 can simultaneously dispatch one micro-instruction to the integer execution engine 1824 and another micro-instruction to the floating point execution engine 1826. In at least one embodiment, the integer execution engine 1824 performs, but is not limited to, integer and memory operations. In at least one embodiment, the floating point engine 1826 performs, but is not limited to, floating point and vector operations. In at least one embodiment, the fetch decode unit 1822 dispatches microinstructions to a single execution engine that replaces both the integer execution engine 1824 and the floating-point execution engine 1826 .

[0211] In at least one embodiment, each core 1820(i), where i is an integer representing a particular instance of core 1820, may access an L2 cache 1828(i) included in core 1820(i). In at least one embodiment, each core 1820 included in a core complex 1810(j), where j is an integer representing a particular instance of core complex 1810, is connected to the other cores 1820 in core complex 1810(j) via an L3 cache 1830(j) included in core complex 1810(j). In at least one embodiment, a core 1820 included in a core complex 1810(j), where j is an integer representing a particular instance of core complex 1810, may access all of the L3 caches 1830(j) included in core complex 1810(j). In at least one embodiment, L3 cache 1830 may include, without limitation, any number of slices.

[0212] In at least one embodiment, fabric 1860 is a system interconnect that facilitates data and control transmission across core complexes 1810(1) through 1810(N), where N is an integer greater than 0, I / O interface 1870, and memory controller 1880. In at least one embodiment, CPU 1800 may include any quantity and type of system interconnect in addition to or instead of fabric 1860, which facilitates data and control transmission across any number and type of directly or indirectly linked components, which may be internal or external to CPU 1800. In at least one embodiment, I / O interface 1870 represents any number and type of I / O interface (e.g., PCI, PCI-X, PCIe, GBE, USB, etc.). In at least one embodiment, various types of peripheral devices are coupled to I / O interface 1870. In at least one embodiment, peripheral devices coupled to I / O interface 1870 may include, but are not limited to, a display, a keyboard, a mouse, a printer, a scanner, a joystick or other type of game controller, a media recording device, an external storage device, a network interface card, etc.

[0213] In at least one embodiment, memory controller 1880 facilitates data transfer between CPU 1800 and system memory 1890. In at least one embodiment, core complex 1810 and graphics complex 1840 share system memory 1890. In at least one embodiment, CPU 1800 implements a memory subsystem including, but not limited to, any quantity and type of memory controller 1880 and memory devices, which may be dedicated to one component or shared among multiple components. In at least one embodiment, CPU 1800 implements a cache subsystem including, but not limited to, one or more cache memories (e.g., L2 cache 1828 and L3 cache 1830), each of which may be private to any number of components (e.g., core 1820 and core complex 1810) or shared among any number of components.

[0214] In at least one embodiment, one or more systems illustrated in Figure 18 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 18 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 18 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 18 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0215] FIG. 19 illustrates an exemplary accelerator integrated slice 1990 according to at least one embodiment. As used herein, a "slice" comprises a designated portion of the processing resources of an accelerator integrated circuit. In at least one embodiment, the accelerator integrated circuit provides cache management, memory access, context management, and interrupt management services on behalf of multiple graphics processing engines included in the graphics acceleration module. The graphics processing engines may each comprise a separate GPU. Alternatively, the graphics processing engine may comprise different types of graphics processing engines within the GPU, such as a graphics execution unit, a media processing engine (e.g., a video encoder / decoder), a sampler, and a blit engine. In at least one embodiment, the graphics acceleration module may be a GPU with multiple graphics processing engines. In at least one embodiment, the graphics processing engines may be individual GPUs integrated on a common package, line card, or chip.

[0216] Application effective address space 1982 in system memory 1914 stores process elements 1983. In one embodiment, process elements 1983 are stored in response to GPU calls 1981 from applications 1980 executing on processor 1907. Process elements 1983 contain the process state of the corresponding applications 1980. Work descriptors (“WD”) 1984 contained in process elements 1983 may be single jobs requested by the application or may contain a pointer to a queue of jobs. In at least one embodiment, WD 1984 is a pointer to a job request queue in application effective address space 1982.

[0217] The graphics acceleration module 1946 and / or individual graphics processing engines may be shared by all or a subset of the processes in the system. In at least one embodiment, infrastructure may be included for setting process state and submitting WD 1984 to the graphics acceleration module 1946 to start a job in a virtualized environment.

[0218] In at least one embodiment, the dedicated process programming model is implementation-specific. In this model, a single process owns the graphics acceleration module 1946 or an individual graphics processing engine. Because the graphics acceleration module 1946 is owned by a single process, the hypervisor initializes the accelerator integration circuitry for the owning partition, and the operating system initializes the accelerator integration circuitry for the owning process when the graphics acceleration module 1946 is allocated.

[0219] In operation, a WD fetch unit 1991 in the accelerator integrated slice 1990 fetches the next WD 1984, which contains instructions for work to be performed by one or more graphics processing engines of the graphics acceleration module 1946. As shown, data from the WD 1984 is stored in registers 1945 and may be used by the memory management unit (“MMU”) 1939, the interrupt management circuitry 1947, and / or the context management circuitry 1948. For example, one embodiment of the MMU 1939 includes segment / page walk circuitry for accessing a segment / page table 1986 within the OS virtual address space 1985. The interrupt management circuitry 1947 may process interrupt events (“INT”) 1992 received from the graphics acceleration module 1946. When performing graphics operations, effective addresses 1993 generated by the graphics processing engines are translated into real addresses by the MMU 1939.

[0220] In one embodiment, the same set of registers 1945 may be replicated for each graphics processing engine and / or graphics acceleration module 1946 and initialized by the hypervisor or operating system. Each of these replicated registers may be included in the accelerator integration slice 1990. Exemplary registers that may be initialized by the hypervisor are shown in Table 1. [Table 10]

[0221] Exemplary registers that may be initialized by the operating system are shown in Table 2. [Table 11]

[0222] In one embodiment, each WD 1984 is specific to a particular graphics acceleration module 1946 and / or a particular graphics processing engine. The WD 1984 may contain all the information needed by the graphics processing engine to perform work, or the WD 1984 may be a pointer to a memory location where the application has set up a command queue for work to be completed.

[0223] In at least one embodiment, one or more systems illustrated in Figure 19 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 19 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 19 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 19 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0224] 20A-20B illustrate exemplary graphics processors, according to at least one embodiment. In at least one embodiment, any of the exemplary graphics processors may be fabricated using one or more IP cores. In addition to what is shown, in at least one embodiment, other logic and circuitry may be included, including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores. In at least one embodiment, the exemplary graphics processor is for use within an SoC.

[0225] FIG. 20A illustrates an exemplary graphics processor 2010 of an SoC integrated circuit that may be fabricated using one or more IP cores, according to at least one embodiment. FIG. 20B illustrates an additional exemplary graphics processor 2040 of an SoC integrated circuit that may be fabricated using one or more IP cores, according to at least one embodiment. In at least one embodiment, the graphics processor 2010 of FIG. 20A is a low-power graphics processor core. In at least one embodiment, the graphics processor 2040 of FIG. 20B is a higher performance graphics processor core. In at least one embodiment, each of the graphics processors 2010, 2040 may be a variation of the graphics processor 1510 of FIG. 15.

[0226] In at least one embodiment, the graphics processor 2010 includes a vertex processor 2005 and one or more fragment processors 2015A-2015N (e.g., 2015A, 2015B, 2015C, 2015D-2015N-1, and 2015N). In at least one embodiment, the graphics processor 2010 can execute different shader programs through separate logic, whereby the vertex processor 2005 is optimized to perform operations for vertex shader programs, and one or more fragment processors 2015A-2015N perform fragment (e.g., pixel) shading operations for fragment or pixel shader programs. In at least one embodiment, the vertex processor 2005 performs the vertex processing stage of a 3D graphics pipeline, generating primitive and vertex data. In at least one embodiment, the fragment processor(s) 2015A-2015N use the primitive and vertex data generated by the vertex processor 2005 to create a frame buffer that is displayed on a display device. In at least one embodiment, the fragment processor(s) 2015A-2015N are optimized to execute fragment shader programs such as those provided in the OpenGL API, which can be used to perform operations similar to pixel shader programs such as those provided in the Direct 3D API.

[0227] In at least one embodiment, the graphics processor 2010 additionally includes one or more MMUs 2020A-2020B, cache(s) 2025A-2025B, and circuit interconnect(s) 2030A-2030B. In at least one embodiment, the one or more MMUs 2020A-2020B provide virtual-to-physical address mapping for the graphics processor 2010, including the vertex processor 2005 and / or fragment processor(s) 2015A-2015N, which may reference vertex or image / texture data stored in memory in addition to vertex or image / texture data stored in the one or more caches 2025A-2025B. In at least one embodiment, one or more MMUs 2020A-2020B may be synchronized with other MMUs in the system, including one or more MMUs associated with one or more application processors 1505, image processor 1515, and / or video processor 1520 of Figure 15, thereby allowing each processor 1505-1520 to participate in a shared or unified virtual memory system. In at least one embodiment, one or more circuit interconnects 2030A-2030B enable the graphics processor 2010 to interface with other IP cores in the SoC, either via the SoC's internal bus or via a direct connection.

[0228] In at least one embodiment, graphics processor 2040 includes one or more MMUs 2020A-2020B, caches 2025A-2025B, and circuit interconnects 2030A-2030B of graphics processor 2010 of FIG. 20A. In at least one embodiment, the graphics processor 2040 includes one or more shader cores 2055A-2055N (e.g., 2055A, 2055B, 2055C, 2055D, 2055E, 2055F-2055N-1, and 2055N), where the one or more shader cores 2055A-2055N provide a unified shader core architecture in which a single core, or type, or cores can execute all types of programmable shader code, including shader program code for implementing vertex shaders, fragment shaders, and / or compute shaders. In at least one embodiment, the number of shader cores can vary. In at least one embodiment, the graphics processor 2040 includes an inter-core task manager 2045 that acts as a thread dispatcher for dispatching execution threads to one or more shader cores 2055A-2055N, and a tiling unit 2058 for accelerating tiling operations for tile-based rendering, in which rendering operations for a scene are sub-partitioned in image space, e.g., to exploit local spatial coherence within a scene or to optimize internal cache usage.

[0229] In at least one embodiment, one or more systems illustrated in Figures 20A-20B are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figures 20A-20B are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figures 20A-20B are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figures 20A-20B are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0230] FIG. 21A illustrates a graphics core 2100 according to at least one embodiment. In at least one embodiment, graphics core 2100 may be included within graphics processor 1510 of FIG. 15. In at least one embodiment, graphics core 2100 may be unified shader cores 2055A-2055N as in FIG. 20B. In at least one embodiment, graphics core 2100 includes a shared instruction cache 2102, a texture unit 2118, and a cache / shared memory 2120, which are common to execution resources within graphics core 2100. In at least one embodiment, graphics core 2100 may include multiple slices 2101A-2101N, or partitions for each core, and a graphics processor may include multiple instances of graphics core 2100. The slices 2101A-2101N may include support logic including a local instruction cache 2104A-2104N, a thread scheduler 2106A-2106N, a thread dispatcher 2108A-2108N, and a set of registers 2110A-2110N. In at least one embodiment, slices 2101A-2101N may include a set of additional function units ("AFUs") 2112A-2112N, floating-point units ("FPUs") 2114A-2114N, integer arithmetic logic units ("ALUs") 2116-2116N, address computational units ("ACUs") 2113A-2113N, double-precision floating-point units ("DPFPUs") 2115A-2115N, and matrix processing units ("MPUs") 2117A-2117N.

[0231] In at least one embodiment, the FPUs 2114A-2114N can perform single-precision (32-bit) and half-precision (16-bit) floating-point operations, and the DPFPUs 2115A-2115N perform double-precision (64-bit) floating-point operations. In at least one embodiment, the ALUs 2116A-2116N can perform variable-precision integer operations with 8-bit, 16-bit, and 32-bit precision and may be configured for mixed-precision operations. In at least one embodiment, the MPUs 2117A-2117N can also be configured for mixed-precision matrix operations, including half-precision floating-point operations and 8-bit integer operations. In at least one embodiment, the MPUs 2117A-2117N can perform various matrix operations to accelerate CUDA programs, including enabling support for accelerated general matrix-to-matrix multiplication ("GEMM"). In at least one embodiment, the AFUs 2112A-2112N can perform additional logical operations not supported by the floating-point unit or integer unit, including trigonometric operations (e.g., sine, cosine, etc.).

[0232] FIG. 21B illustrates a general-purpose graphics processing unit (“GPGPU”) 2130 according to at least one embodiment. In at least one embodiment, the GPGPU 2130 is highly parallel and suitable for deployment on a multi-chip module. In at least one embodiment, the GPGPU 2130 may be configured to allow highly parallel compute operations to be performed by an array of GPUs. In at least one embodiment, the GPGPU 2130 may be directly linked to other instances of the GPGPU 2130 to create a multi-GPU cluster to improve execution time for CUDA programs. In at least one embodiment, the GPGPU 2130 includes a host interface 2132 to enable connection with a host processor. In at least one embodiment, the host interface 2132 is a PCIe interface. In at least one embodiment, the host interface 2132 may be a vendor-specific communication interface or fabric. In at least one embodiment, the GPGPU 2130 receives commands from the host processor and distributes execution threads associated with those commands across the set of compute clusters 2136A-2136H using a global scheduler 2134. In at least one embodiment, the compute clusters 2136A-2136H share a cache memory 2138. In at least one embodiment, the cache memory 2138 can act as a higher-level cache for the cache memories within the compute clusters 2136A-2136H.

[0233] In at least one embodiment, the GPGPU 2130 includes memory 2144A-2144B coupled to the compute clusters 2136A-2136H via a set of memory controllers 2142A-2142B. In at least one embodiment, the memory 2144A-2144B can include various types of memory devices, including DRAM or graphics random access memory such as synchronous graphics random access memory ("SGRAM"), including graphics double data rate ("GDDR") memory.

[0234] In at least one embodiment, compute clusters 2136A-2136H each include a set of graphics cores, such as graphics core 2100 of FIG. 21A, which may include multiple types of integer and floating-point logic units capable of performing computational operations with various precisions, including those suitable for computations associated with CUDA programs. For example, in at least one embodiment, at least a subset of the floating-point units in each of compute clusters 2136A-2136H may be configured to perform 16-bit or 32-bit floating-point operations, and a different subset of the floating-point units may be configured to perform 64-bit floating-point operations.

[0235] In at least one embodiment, multiple instances of GPGPU 2130 may be configured to operate as a compute cluster. The compute clusters 2136A-2136H may implement any technically feasible communication technique for synchronization and data exchange. In at least one embodiment, multiple instances of GPGPU 2130 communicate via host interface 2132. In at least one embodiment, the GPGPU 2130 includes an I / O hub 2139 that couples the GPGPU 2130 to a GPU link 2140 that enables direct connection to other instances of GPGPU 2130. In at least one embodiment, the GPU link 2140 is coupled to a dedicated GPU-GPU bridge that enables communication and synchronization between multiple instances of GPGPU 2130. In at least one embodiment, the GPU link 2140 is coupled to a high-speed interconnect for sending and receiving data to other GPGPUs 2130 or parallel processors. In at least one embodiment, multiple instances of GPGPU 2130 are located in separate data processing systems and communicate via a network device accessible via host interface 2132. In at least one embodiment, GPU link 2140 may be configured to allow connection to a host processor in addition to, or as an alternative to, host interface 2132. In at least one embodiment, GPGPU 2130 may be configured to execute CUDA programs.

[0236] In at least one embodiment, one or more systems illustrated in Figures 21A-21B are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figures 21A-21B are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figures 21A-21B are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figures 21A-21B are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0237] 22A illustrates a parallel processor 2200, according to at least one embodiment. In at least one embodiment, various components of parallel processor 2200 may be implemented using one or more integrated circuit devices, such as a programmable processor, an application specific integrated circuit (“ASIC”), or an FPGA.

[0238] In at least one embodiment, parallel processor 2200 includes parallel processing units 2202. In at least one embodiment, parallel processing units 2202 include I / O units 2204 that enable communication with other devices, including other instances of parallel processing units 2202. In at least one embodiment, I / O units 2204 may be directly connected to other devices. In at least one embodiment, I / O units 2204 connect to other devices through the use of a hub or switch interface, such as memory hub 2205. In at least one embodiment, the connection between memory hub 2205 and I / O units 2204 forms a communication link. In at least one embodiment, I / O units 2204 connect to host interface 2206 and memory crossbar 2216, where host interface 2206 receives commands intended to perform processing operations and memory crossbar 2216 receives commands intended to perform memory operations.

[0239] In at least one embodiment, when host interface 2206 receives command buffers via I / O unit 2204, host interface 2206 can direct work operations to implement those commands to front end 2208. In at least one embodiment, front end 2208 is coupled to scheduler 2210, which is configured to distribute commands or other work items to processing array 2212. In at least one embodiment, scheduler 2210 ensures that processing array 2212 is properly configured and in a valid state before tasks are distributed to processing array 2212. In at least one embodiment, scheduler 2210 is implemented via firmware logic running on a microcontroller. In at least one embodiment, the microcontroller-implemented scheduler 2210 is configurable to perform complex scheduling and work distribution operations at coarse and fine granularity, allowing rapid preemption and context switching of threads executing on the processing array 2212. In at least one embodiment, host software can present workloads for scheduling on the processing array 2212 via one of multiple graphics processing bells. In at least one embodiment, the workloads can then be automatically distributed across the processing array 2212 by scheduler 2210 logic within the microcontroller that includes the scheduler 2210.

[0240] In at least one embodiment, processing array 2212 may include up to “N” clusters (e.g., cluster 2214A, cluster 2214B through cluster 2214N). In at least one embodiment, each cluster 2214A through 2214N of processing array 2212 may execute multiple concurrent threads. In at least one embodiment, scheduler 2210 may allocate work to clusters 2214A through 2214N of processing array 2212 using various scheduling and / or work distribution algorithms, which may vary depending on the workload occurring for each type of program or computation. In at least one embodiment, scheduling may be handled dynamically by scheduler 2210 or may be partially assisted by compiler logic during compilation of program logic configured for execution by processing array 2212. In at least one embodiment, different clusters 2214A through 2214N of processing array 2212 may be allocated to process different types of programs or perform different types of computations.

[0241] In at least one embodiment, processing array 2212 may be configured to perform various types of parallel processing operations. In at least one embodiment, processing array 2212 is configured to perform general-purpose parallel compute operations. For example, in at least one embodiment, processing array 2212 may include logic for performing processing tasks including filtering video and / or audio data, performing modeling operations including physics operations, and performing data transformations.

[0242] In at least one embodiment, processing array 2212 is configured to perform parallel graphics processing operations. In at least one embodiment, processing array 2212 may include additional logic to support the execution of such graphics processing operations, including, but not limited to, texture sampling logic for performing texture operations, as well as tessellation logic and other vertex processing logic. In at least one embodiment, processing array 2212 may be configured to execute graphics processing related shader programs, such as, but not limited to, vertex shaders, tessellation shaders, geometry shaders, and pixel shaders. In at least one embodiment, parallel processing unit 2202 may transfer data from system memory via I / O unit 2204 for processing. In at least one embodiment, the transferred data may be stored in on-chip memory (e.g., parallel processor memory 2222) during processing and then written back to system memory.

[0243] In at least one embodiment, when parallel processing unit 2202 is used to perform graphics processing, scheduler 2210 may be configured to divide the processing workload into tasks of approximately equal size to better enable distribution of graphics processing operations to multiple clusters 2214A-2214N of processing array 2212. In at least one embodiment, portions of processing array 2212 may be configured to perform different types of processing. For example, in at least one embodiment, to produce a rendered image for display, a first portion may be configured to perform vertex shading and topology generation, a second portion may be configured to perform tessellation and geometry shading, and a third portion may be configured to perform pixel shading or other screen space operations. In at least one embodiment, intermediate data produced by one or more of clusters 2214A-2214N may be stored in a buffer to allow the intermediate data to be transmitted between clusters 2214A-2214N for further processing.

[0244] In at least one embodiment, processing array 2212 may receive processing tasks to be performed via scheduler 2210, which receives commands defining the processing tasks from front end 2208. In at least one embodiment, a processing task may include an index of data to be processed, e.g., surface (patch) data, primitive data, vertex data, and / or pixel data, as well as state parameters and commands that define how the data should be processed (e.g., which program should be executed). In at least one embodiment, scheduler 2210 may be configured to fetch the index corresponding to the task or may receive the index from front end 2208. In at least one embodiment, front end 2208 may be configured to ensure that processing array 2212 is configured to a valid state before a workload specified by an incoming command buffer (e.g., batch buffer, push buffer, etc.) is initiated.

[0245] In at least one embodiment, each of one or more instances of parallel processing unit 2202 may be coupled to parallel processor memory 2222. In at least one embodiment, parallel processor memory 2222 may be accessed via memory crossbar 2216, which may receive memory requests from processing array 2212 as well as I / O unit 2204. In at least one embodiment, memory crossbar 2216 may access parallel processor memory 2222 via memory interface 2218. In at least one embodiment, memory interface 2218 may include multiple partition units (e.g., partition unit 2220A, partition unit 2220B through partition unit 2220N), each of which may be coupled to a portion (e.g., a memory unit) of parallel processor memory 2222. In at least one embodiment, the number of partition units 2220A-2220N is configured to be equal to the number of memory units, such that a first partition unit 2220A has a corresponding first memory unit 2224A, a second partition unit 2220B has a corresponding memory unit 2224B, and an Nth partition unit 2220N has a corresponding Nth memory unit 2224N. In at least one embodiment, the number of partition units 2220A-2220N may not be equal to the number of memory devices.

[0246] In at least one embodiment, the memory units 2224A-2224N may include various types of memory devices, including DRAM or graphics random access memory, such as SGRAM, including GDDR memory. In at least one embodiment, the memory units 2224A-2224N may also include 3D stacked memory, including but not limited to high bandwidth memory (“HBM”). In at least one embodiment, to efficiently use the available bandwidth of the parallel processor memory 2222, render targets, such as frame buffers or texture maps, may be stored across the memory units 2224A-2224N, allowing the partition units 2220A-2220N to write portions of each render target in parallel. In at least one embodiment, local instances of the parallel processor memory 2222 may be eliminated in favor of a unified memory design that utilizes system memory in conjunction with local cache memory.

[0247] In at least one embodiment, any one of the clusters 2214A-2214N of the processing array 2212 can process data that is to be written to any one of the memory units 2224A-2224N in the parallel processor memory 2222. In at least one embodiment, the memory crossbar 2216 can be configured to forward the output of each cluster 2214A-2214N to any partition unit 2220A-2220N that can perform additional processing operations on the output, or to another cluster 2214A-2214N. In at least one embodiment, each cluster 2214A-2214N can communicate with a memory interface 2218 through the memory crossbar 2216 to read from or write to various external memory devices. In at least one embodiment, memory crossbar 2216 has a connection to memory interface 2218 for communicating with I / O unit 2204, as well as a connection to local instances of parallel processor memory 2222, which allows processing units in different clusters 2214A-2214N to communicate with system memory or other memory not local to the parallel processing unit 2202. In at least one embodiment, memory crossbar 2216 can use virtual channels to separate traffic streams between clusters 2214A-2214N and partition units 2220A-2220N.

[0248] In at least one embodiment, multiple instances of parallel processing unit 2202 may be provided on a single add-in card, or multiple add-in cards may be interconnected. In at least one embodiment, different instances of parallel processing unit 2202 may be configured to interoperate even if the different instances have different numbers of processing cores, different amounts of local parallel processor memory, and / or other configuration differences. For example, in at least one embodiment, some instances of parallel processing unit 2202 may include higher precision floating-point units relative to other instances. In at least one embodiment, systems incorporating one or more instances of parallel processing unit 2202 or parallel processor 2200 may be implemented in a variety of configurations and form factors, including, but not limited to, desktop, laptop, or handheld personal computers, servers, workstations, game consoles, and / or embedded systems.

[0249] FIG. 22B illustrates a processing cluster 2294, according to at least one embodiment. In at least one embodiment, the processing cluster 2294 is included within a parallel processing unit. In at least one embodiment, the processing cluster 2294 is one of the processing clusters 2214A-2214N of FIG. 22. In at least one embodiment, the processing cluster 2294 may be configured to execute many threads in parallel, where the term "thread" refers to an instance of a particular program executing on a particular set of input data. In at least one embodiment, a single instruction, multiple data ("SIMD") instruction issue technique is used to support parallel execution of multiple threads without providing multiple independent instruction units. In at least one embodiment, a single instruction, multiple thread ("SIMT") technique is used to support parallel execution of multiple, generally synchronized threads using a common instruction unit configured to issue instructions to a set of processing engines in each processing cluster 2294.

[0250] In at least one embodiment, operation of processing cluster 2294 may be controlled via a pipeline manager 2232 that distributes processing tasks to the SIMT parallel processors. In at least one embodiment, pipeline manager 2232 receives instructions from scheduler 2210 of FIG. 22 and manages execution of those instructions via graphics multiprocessor 2234 and / or texture unit 2236. In at least one embodiment, graphics multiprocessor 2234 is an exemplary instance of a SIMT parallel processor. However, in at least one embodiment, various types of SIMT parallel processors of different architectures may be included within processing cluster 2294. In at least one embodiment, one or more instances of graphics multiprocessor 2234 may be included within processing cluster 2294. In at least one embodiment, graphics multiprocessor 2234 may process data, and data crossbar 2240 may be used to distribute the processed data to one of several possible destinations, including other shader units. In at least one embodiment, pipeline manager 2232 can facilitate the distribution of processed data by specifying a destination for the processed data to be distributed through data crossbar 2240.

[0251] In at least one embodiment, each graphics multiprocessor 2234 in a processing cluster 2294 may include an identical set of function execution logic (e.g., arithmetic logic units, load / store units (“LSUs”), etc.). In at least one embodiment, the function execution logic may be configured in a pipelined manner, such that new instructions may be issued before previous instructions complete. In at least one embodiment, the function execution logic supports a variety of operations, including integer and floating-point arithmetic, comparison operations, Boolean operations, bit shifts, and computation of various algebraic functions. In at least one embodiment, the same function unit hardware may be utilized to perform different operations, and any combination of function units may be present.

[0252] In at least one embodiment, instructions sent to processing cluster 2294 constitute threads. In at least one embodiment, a set of threads executing across a set of parallel processing engines is a thread group. In at least one embodiment, a thread group executes a program on different input data. In at least one embodiment, each thread in a thread group may be assigned to a different processing engine in graphics multiprocessor 2234. In at least one embodiment, a thread group may include fewer threads than the number of processing engines in graphics multiprocessor 2234. In at least one embodiment, when a thread group includes fewer threads than the number of processing engines, one or more of the processing engines may be idle during a cycle in which the thread group is processed. In at least one embodiment, a thread group may also include more threads than the number of processing engines in graphics multiprocessor 2234. In at least one embodiment, when a thread group includes more threads than the number of processing engines in graphics multiprocessor 2234, processing may be performed over consecutive clock cycles. In at least one embodiment, multiple thread groups may execute simultaneously on the graphics multiprocessor 2234.

[0253] In at least one embodiment, the graphics multiprocessor 2234 includes internal cache memory for performing load and store operations. In at least one embodiment, the graphics multiprocessor 2234 can forgo internal cache and use cache memory (e.g., L1 cache 2248) within the processing cluster 2294. In at least one embodiment, each graphics multiprocessor 2234 also has access to a level 2 (“L2”) cache within a partition unit (e.g., partition units 2220A-2220N in FIG. 22A ), which is shared among all processing clusters 2294 and can be used to transfer data between threads. In at least one embodiment, the graphics multiprocessor 2234 can also access off-chip global memory, which can include one or more of the local parallel processor memories and / or system memories. In at least one embodiment, any memory external to the parallel processing unit 2202 can be used as global memory. In at least one embodiment, processing cluster 2294 includes multiple instances of graphics multiprocessor 2234, and the graphics multiprocessors 2234 can share common instructions and data, which can be stored in L1 cache 2248.

[0254] In at least one embodiment, each processing cluster 2294 may include an MMU 2245 configured to map virtual addresses to physical addresses. In at least one embodiment, one or more instances of MMU 2245 may reside in memory interface 2218 of FIG. 22. In at least one embodiment, MMU 2245 includes a set of page table entries (“PTEs”) used to map virtual addresses to physical addresses of tiles and optionally cache line indices. In at least one embodiment, MMU 2245 may include an address translation lookaside buffer (“TLB”) or cache, which may reside in graphics multiprocessor 2234, L1 cache 2248, or processing cluster 2294. In at least one embodiment, physical addresses are processed to distribute surface data access locality and enable efficient request interleaving among partition units. In at least one embodiment, the cache line index may be used to determine whether a request for a cache line is a hit or a miss.

[0255] In at least one embodiment, processing cluster 2294 may be configured such that each graphics multiprocessor 2234 is coupled to a texture unit 2236 for performing texture mapping operations, such as determining texture sample locations, reading texture data, and filtering the texture data. In at least one embodiment, texture data is read from an internal texture L1 cache (not shown) or from an L1 cache within graphics multiprocessor 2234 and fetched as needed from an L2 cache, local parallel processor memory, or system memory. In at least one embodiment, each graphics multiprocessor 2234 outputs processed tasks to data crossbar 2240 to provide the processed tasks to another processing cluster 2294 for further processing, or stores the processed tasks in an L2 cache, local parallel processor memory, or system memory via memory crossbar 2216. In at least one embodiment, a pre-raster operation unit (“pre-ROP”) 2242 is configured to receive data from the graphics multiprocessor 2234 and direct the data to a ROP unit, which may be co-located with a partition unit as described herein (e.g., partition units 2220A-2220N of FIG. 22). In at least one embodiment, the pre-ROP 2242 may perform optimizations for color blending, organize pixel color data, and perform address translation.

[0256] FIG. 22C illustrates a graphics multiprocessor 2296, according to at least one embodiment. In at least one embodiment, the graphics multiprocessor 2296 is the graphics multiprocessor 2234 of FIG. 22B. In at least one embodiment, the graphics multiprocessor 2296 couples to the pipeline manager 2232 of the processing cluster 2294. In at least one embodiment, the graphics multiprocessor 2296 has an execution pipeline including, but not limited to, an instruction cache 2252, an instruction unit 2254, an address mapping unit 2256, a register file 2258, one or more GPGPU cores 2262, and one or more LSUs 2266. The GPGPU cores 2262 and the LSUs 2266 are coupled to the cache memory 2272 and the shared memory 2270 via the memory and cache interconnect 2268.

[0257] In at least one embodiment, instruction cache 2252 receives a stream of instructions to execute from pipeline manager 2232. In at least one embodiment, instructions are cached in instruction cache 2252 and dispatched for execution by instruction unit 2254. In at least one embodiment, instruction unit 2254 can dispatch instructions as thread groups (e.g., warps), with each thread of a thread group being assigned to a different execution unit within GPGPU core 2262. In at least one embodiment, instructions can access either local, shared, or global address spaces by specifying an address in the unified address space. In at least one embodiment, address mapping unit 2256 can be used to translate addresses in the unified address space into individual memory addresses that can be accessed by LSU 2266.

[0258] In at least one embodiment, register file 2258 provides a set of registers to the functional units of graphics multiprocessor 2296. In at least one embodiment, register file 2258 provides temporary storage for operands connected to the data paths of the functional units (e.g., GPGPU core 2262, LSU 2266) of graphics multiprocessor 2296. In at least one embodiment, register file 2258 is partitioned among each of the functional units such that each functional unit is allocated a dedicated portion of register file 2258. In at least one embodiment, register file 2258 is partitioned among different thread groups being executed by graphics multiprocessor 2296.

[0259] In at least one embodiment, the GPGPU cores 2262 each include an FPU and / or integer ALU used to execute instructions for the graphics multiprocessor 2296. The GPGPU cores 2262 may be of similar or different architectures. In at least one embodiment, a first portion of the GPGPU core 2262 includes a single-precision FPU and integer ALU, and a second portion of the GPGPU core 2262 includes a double-precision FPU. In at least one embodiment, the FPU may implement the IEEE 754-2008 standard for floating-point arithmetic or may enable variable-precision floating-point arithmetic. In at least one embodiment, the graphics multiprocessor 2296 may additionally include one or more fixed-function or special-function units for performing specific functions, such as rectangular copy operations or pixel blending operations. In at least one embodiment, one or more of the GPGPU cores 2262 may also include fixed or special-function logic.

[0260] In at least one embodiment, GPGPU core 2262 includes SIMD logic capable of performing a single instruction on multiple sets of data. In at least one embodiment, GPGPU core 2262 can physically execute SIMD4, SIMD8, and SIMD16 instructions and logically execute SIMD1, SIMD2, and SIMD32 instructions. In at least one embodiment, SIMD instructions for GPGPU core 2262 may be generated at compile time by a shader compiler or automatically generated when executing a program written and compiled for a single program multiple data (“SPMD”) or SIMT architecture. In at least one embodiment, multiple threads of a program configured for the SIMT execution model may be executed via a single SIMD instruction. For example, in at least one embodiment, eight SIMT threads performing the same or similar operations may be executed in parallel via a single SIMD8 logical unit.

[0261] In at least one embodiment, memory and cache interconnect 2268 is an interconnect network connecting each functional unit of graphics multiprocessor 2296 to register file 2258 and shared memory 2270. In at least one embodiment, memory and cache interconnect 2268 is a crossbar interconnect that allows LSU 2266 to implement load and store operations between shared memory 2270 and register file 2258. In at least one embodiment, register file 2258 can operate at the same frequency as GPGPU cores 2262, and therefore data transfers between GPGPU cores 2262 and register file 2258 are very low latency. In at least one embodiment, shared memory 2270 can be used to enable communication between threads executing on functional units within graphics multiprocessor 2296. In at least one embodiment, cache memory 2272 can be used as a data cache, for example, to cache texture data communicated between functional units and texture unit 2236. In at least one embodiment, shared memory 2270 can also be used as a managed program cache. In at least one embodiment, threads executing on GPGPU cores 2262 can programmatically store data in the shared memory in addition to the automatically cached data stored in cache memory 2272.

[0262] In at least one embodiment, a parallel processor or GPGPU as described herein is communicatively coupled to a host / processor core to accelerate graphics operations, machine learning operations, pattern analysis operations, and various general-purpose GPU (GPGPU) functions. In at least one embodiment, the GPU may be communicatively coupled to the host processor / core via a bus or other interconnect (e.g., a high-speed interconnect such as PCIe or NVLink). In at least one embodiment, the GPU may be integrated in the same package or chip as the cores and communicatively coupled to the cores via a processor bus / interconnect internal to the package or chip. In at least one embodiment, regardless of the manner in which the GPU is connected, the processor core may allocate work to the GPU in the form of a sequence of commands / instructions contained in a WD. In at least one embodiment, the GPU then uses dedicated circuitry / logic to efficiently process these commands / instructions.

[0263] In at least one embodiment, one or more systems illustrated in Figures 22A-22C are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figures 22A-22C are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figures 22A-22C are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figures 22A-22C are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0264] 23 illustrates a graphics processor 2300 according to at least one embodiment. In at least one embodiment, the graphics processor 2300 includes a ring interconnect 2302, a pipeline front end 2304, a media engine 2337, and graphics cores 2380A-2380N. In at least one embodiment, the ring interconnect 2302 couples the graphics processor 2300 to other processing units, including other graphics processors or one or more general-purpose processor cores. In at least one embodiment, the graphics processor 2300 is one of many processors incorporated within a multi-core processing system.

[0265] In at least one embodiment, graphics processor 2300 receives batches of commands via ring interconnect 2302. In at least one embodiment, the incoming commands are interpreted by command streamer 2303 in pipeline front end 2304. In at least one embodiment, graphics processor 2300 includes scalable execution logic for performing 3D geometry processing and media processing via one or more graphics cores 2380A-2380N. In at least one embodiment, for 3D geometry processing commands, command streamer 2303 supplies the commands to geometry pipeline 2336. In at least one embodiment, for at least some media processing commands, command streamer 2303 supplies the commands to video front end 2334, which couples to media engine 2337. In at least one embodiment, the media engine 2337 includes a video quality engine (“VQE”) 2330 for video and image post-processing and a multi-format encode / decode (“MFX”) engine 2333 for providing hardware-accelerated media data encoding and decoding. In at least one embodiment, the geometry pipeline 2336 and the media engine 2337 each spawn execution threads for thread execution resources provided by at least one graphics core 2380A.

[0266] In at least one embodiment, graphics processor 2300 includes scalable thread execution resources characterized by modular graphics cores 2380A-2380N (sometimes referred to as core slices), each having multiple sub-cores 2350A-2350N, 2360A-2360N (sometimes referred to as core sub-slices). In at least one embodiment, graphics processor 2300 can have any number of graphics cores 2380A-2380N. In at least one embodiment, graphics processor 2300 includes graphics core 2380A having at least a first sub-core 2350A and a second sub-core 2360A. In at least one embodiment, graphics processor 2300 is a low-power processor with a single sub-core (e.g., sub-core 2350A). In at least one embodiment, graphics processor 2300 includes multiple graphics cores 2380A-2380N, each including a first set of sub-cores 2350A-2350N and a second set of sub-cores 2360A-2360N. In at least one embodiment, each sub-core in the first sub-cores 2350A-2350N includes at least a first set of execution units ("EUs") 2352A-2352N and media / texture samplers 2354A-2354N. In at least one embodiment, each sub-core in the second sub-cores 2360A-2360N includes at least a second set of execution units 2362A-2362N and samplers 2364A-2364N. In at least one embodiment, each sub-core 2350A-2350N, 2360A-2360N shares a set of shared resources 2370A-2370N. In at least one embodiment, the shared resources 2370 include shared cache memory and pixel operating logic.

[0267] In at least one embodiment, one or more systems illustrated in Figure 23 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 23 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 23 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 23 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0268] FIG. 24 illustrates a processor 2400 according to at least one embodiment. In at least one embodiment, the processor 2400 may include logic circuitry for implementing instructions, including, but not limited to, x86 instructions, AMR instructions, special instructions for ASICs, etc. In at least one embodiment, the processor 2400 may include registers for storing packed data, such as 64-bit wide MMX™ registers in an MMX technology-enabled microprocessor from Intel Corporation of Santa Clara, California. In at least one embodiment, MMX registers, available in both integer and floating-point formats, may operate on packed data elements with SIMD and streaming SIMD extension (“SSE”) instructions. In at least one embodiment, 128-bit wide XMM registers associated with SSE2, SSE3, SSE4, AVX, or higher (collectively referred to as “SSEx”) technology may hold such packed data operands. In at least one embodiment, the processor 2410 may execute instructions to accelerate CUDA programs.

[0269] In at least one embodiment, processor 2400 includes an in-order front end (“front end”) 2401 for fetching instructions to be executed and preparing instructions to be used later in the processor pipeline. In at least one embodiment, front end 2401 may include several units. In at least one embodiment, an instruction prefetcher 2426 fetches instructions from memory and feeds the instructions to an instruction decoder 2428, which decodes or interprets the instructions. For example, in at least one embodiment, instruction decoder 2428 decodes received instructions into one or more operations called “microinstructions” or “micro-operations” (also called “micro-ops” or “uops”) for execution. In at least one embodiment, instruction decoder 2428 parses instructions into opcodes and corresponding data and control fields that can be used by the microarchitecture to perform the operations. In at least one embodiment, trace cache 2430 may assemble decoded uops into program-order sequences, or traces, for execution in uop queue 2434. In at least one embodiment, when trace cache 2430 encounters a complex instruction, microcode ROM 2432 provides the uops necessary to complete the operation.

[0270] In at least one embodiment, some instructions may be converted into a single micro-op, while other instructions require several micro-ops to complete the entire operation. In at least one embodiment, if more than four micro-ops are required to complete an instruction, the instruction decoder 2428 may access the microcode ROM 2432 to implement the instruction. In at least one embodiment, an instruction may be decoded into a fewer number of micro-ops for processing in the instruction decoder 2428. In at least one embodiment, an instruction may be stored in the microcode ROM 2432 if several micro-ops are required to accomplish the operation. In at least one embodiment, the trace cache 2430 references an entry point programmable logic array (“PLA”) to determine the correct microinstruction pointer to read the microcode sequence from to complete one or more instructions from the microcode ROM 2432. In at least one embodiment, after the microcode ROM 2432 finishes sequencing micro-ops for an instruction, the machine front end 2401 may resume fetching micro-ops from the trace cache 2430.

[0271] In at least one embodiment, an out-of-order execution engine ("out-of-order engine") 2403 may prepare instructions for execution. In at least one embodiment, the out-of-order execution logic has several buffers to smooth the flow of instructions and reorder them to optimize performance as they move down the pipeline and are scheduled for execution. The out-of-order execution engine 2403 includes, but is not limited to, an allocator / register renamer 2440, a memory uop queue 2442, an integer / floating point uop queue 2444, a memory scheduler 2446, a fast scheduler 2402, a slow / general purpose floating point scheduler ("slow / general purpose FP scheduler") 2404, and a simple floating point scheduler ("simple FP scheduler") 2406. In at least one embodiment, the fast scheduler 2402, the slow / general purpose floating point scheduler 2404, and the simple floating point scheduler 2406 are also collectively referred to herein as "uop schedulers 2402, 2404, 2406." The allocator / register renamer 2440 allocates machine buffers and resources required by each uop to execute. In at least one embodiment, the allocator / register renamer 2440 renames logical registers upon entry into the register file. In at least one embodiment, allocator / register renamer 2440 also allocates an entry for each uop in one of two uop queues, memory uop queue 2442 for memory operations and integer / floating point uop queue 2444 for non-memory operations, before memory scheduler 2446 and uop schedulers 2402, 2404, 2406. In at least one embodiment, uop schedulers 2402, 2404, 2406 determine when uops are ready to execute based on the readiness of their dependent input register operand sources and the availability of execution resources required by the uops to complete their operations.In at least one embodiment, the fast scheduler 2402 of at least one embodiment may schedule every half of a main clock cycle, and the slow / general purpose floating point scheduler 2404 and simple floating point scheduler 2406 may schedule once per main processor clock cycle. In at least one embodiment, the uop schedulers 2402, 2404, 2406 arbitrate for dispatch ports to schedule uops for execution.

[0272] In at least one embodiment, execution block 2411 includes, but is not limited to, integer register file / bypass network 2408, floating point register file / bypass network (“FP register file / bypass network”) 2410, address generation units (“AGUs”) 2412 and 2414, fast ALUs 2416 and 2418, slow ALU 2420, floating point ALU (“FP”) 2422, and floating point move unit (“FP move”) 2424. In at least one embodiment, integer register file / bypass network 2408 and floating point register file / bypass network 2410 are also referred to herein as “register files 2408, 2410.” In at least one embodiment, AGUs 2412 and 2414, fast ALUs 2416 and 2418, slow ALU 2420, floating-point ALU 2422, and floating-point move unit 2424 are also referred to herein as "execution units 2412, 2414, 2416, 2418, 2420, 2422, and 2424." In at least one embodiment, an execution block may include any number and type of register files (including, but not limited to, zero), bypass networks, address generation units, and execution units, in any combination.

[0273] In at least one embodiment, register files 2408, 2410 may be located between uop schedulers 2402, 2404, 2406 and execution units 2412, 2414, 2416, 2418, 2420, 2422, and 2424. In at least one embodiment, integer register file / bypass network 2408 performs integer operations. In at least one embodiment, floating point register file / bypass network 2410 performs floating point operations. In at least one embodiment, each of register files 2408, 2410 may include, but is not limited to, a bypass network that may bypass or forward recently completed results that have not yet been written to the register file to new dependent uops. In at least one embodiment, register files 2408, 2410 may communicate data with each other. In at least one embodiment, integer register file / bypass network 2408 may include, but is not limited to, two separate register files: one register file for lower 32-bit data and a second register file for higher 32-bit data. In at least one embodiment, floating-point instructions typically have operands that are 64 to 128 bits wide, so floating-point register file / bypass network 2410 may include, but is not limited to, 128-bit wide entries.

[0274] In at least one embodiment, execution units 2412, 2414, 2416, 2418, 2420, 2422, and 2424 may execute instructions. In at least one embodiment, register files 2408 and 2410 store integer and floating-point data operand values ​​that microinstructions need to execute. In at least one embodiment, processor 2400 may include, but is not limited to, any number and combination of execution units 2412, 2414, 2416, 2418, 2420, 2422, and 2424. In at least one embodiment, floating-point ALU 2422 and floating-point move unit 2424 may perform floating-point, MMX, SIMD, AVX, and SSE, or other operations. In at least one embodiment, the floating-point ALU 2422 may include a 64-bit floating-point divider for performing, but not limited to, division, square root, and remainder micro-ops. In at least one embodiment, instructions involving floating-point values ​​may be handled by floating-point hardware. In at least one embodiment, ALU operations may be passed to the high-speed ALUs 2416, 2418. In at least one embodiment, the high-speed ALUs 2416, 2418 may perform high-speed operations with an effective latency of half a clock cycle. In at least one embodiment, the low-speed ALU 2420 may include integer execution hardware for long-latency type operations such as, but not limited to, multipliers, shifts, flag logic, and branching, so that most complex integer operations proceed to the low-speed ALU 2420. In at least one embodiment, memory load / store operations may be performed by the AGUs 2412, 2414. In at least one embodiment, fast ALU 2416, fast ALU 2418, and slow ALU 2420 may perform integer operations on 64-bit data operands. In at least one embodiment, fast ALU 2416, fast ALU 2418, and slow ALU 2420 may be implemented to support various data bit sizes, including 16, 32, 128, 256, etc. In at least one embodiment, floating-point ALU 2422 and floating-point move unit 2424 may be implemented to support various operands having various bit widths.In at least one embodiment, the floating-point ALU 2422 and floating-point move unit 2424 can operate on 128-bit wide packed data operands in conjunction with SIMD and multimedia instructions.

[0275] In at least one embodiment, the uop schedulers 2402, 2404, 2406 dispatch dependent operations before the parent load finishes executing. In at least one embodiment, because uops may be speculatively scheduled and executed in the processor 2400, the processor 2400 may also include logic to handle memory misses. In at least one embodiment, if a data load misses in the data cache, there may be dependent operations in progress in the pipeline past the scheduler with temporarily incorrect data. In at least one embodiment, a replay mechanism tracks and re-executes instructions that use the incorrect data. In at least one embodiment, the dependent operations may need to be replayed, and the independent operations may be allowed to complete. In at least one embodiment, the scheduler and replay mechanism of at least one embodiment of the processor may also be designed to capture instruction sequences for text string comparison operations.

[0276] In at least one embodiment, the term “register” may refer to an on-board processor storage location that may be used as part of an instruction to identify an operand. In at least one embodiment, a register may be available externally to the processor (from a programmer's perspective). In at least one embodiment, a register may not be limited to a particular type of circuit. Rather, in at least one embodiment, a register may store data, provide data, and perform the functions described herein. In at least one embodiment, the registers described herein may be implemented by circuit elements within the processor using any number of different techniques, such as dedicated physical registers, dynamically allocated physical registers using register renaming, or a combination of dedicated and dynamically allocated physical registers. In at least one embodiment, an integer register stores 32-bit integer data. The register file of at least one embodiment also includes eight multimedia SIMD registers for packed data.

[0277] In at least one embodiment, one or more systems illustrated in Figure 24 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 24 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 24 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 24 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0278] FIG. 25 illustrates a processor 2500 according to at least one embodiment. In at least one embodiment, the processor 2500 includes, without limitation, one or more processor cores (“cores”) 2502A-2502N, an integrated memory controller 2514, and an integrated graphics processor 2508. In at least one embodiment, the processor 2500 may include additional cores, up to an additional processor core 2502N, represented by a dashed box. In at least one embodiment, each of the processor cores 2502A-2502N includes one or more internal cache units 2504A-2504N. In at least one embodiment, each processor core also has access to one or more shared cache units 2506.

[0279] In at least one embodiment, the internal cache units 2504A-2504N and the shared cache unit 2506 represent a cache memory hierarchy within the processor 2500. In at least one embodiment, the cache memory units 2504A-2504N may include at least one level of instruction and data cache within each processor core, and one or more levels of shared mid-level cache, such as L2, L3, level 4 (“L4”), or other levels of cache, where the highest level of cache before external memory is classified as LLC. In at least one embodiment, cache coherency logic maintains coherency between the various cache units 2506 and 2504A-2504N.

[0280] In at least one embodiment, processor 2500 may also include a set of one or more bus controller units 2516 and a system agent core 2510. In at least one embodiment, one or more bus controller units 2516 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, system agent core 2510 provides management functionality for various processor components. In at least one embodiment, system agent core 2510 includes one or more integrated memory controllers 2514 for managing access to various external memory devices (not shown).

[0281] In at least one embodiment, one or more of the processor cores 2502A-2502N include support for simultaneous multithreading. In at least one embodiment, the system agent core 2510 includes components for coordinating and operating the processor cores 2502A-2502N during multithreaded processing. In at least one embodiment, the system agent core 2510 may additionally include a power control unit ("PCU"), which includes logic and components for adjusting the power state of one or more of the processor cores 2502A-2502N and the graphics processor 2508.

[0282] In at least one embodiment, processor 2500 additionally includes a graphics processor 2508 for performing graphics processing operations. In at least one embodiment, graphics processor 2508 couples to a system agent core 2510 that includes a shared cache unit 2506 and one or more integrated memory controllers 2514. In at least one embodiment, system agent core 2510 also includes a display controller 2511 for driving graphics processor output to one or more coupled displays. In at least one embodiment, display controller 2511 may also be a separate module coupled to graphics processor 2508 via at least one interconnect or may be incorporated within graphics processor 2508.

[0283] In at least one embodiment, a ring-based interconnect unit 2512 is used to couple the internal components of processor 2500. In at least one embodiment, alternative interconnect units such as point-to-point interconnects, switched interconnects, or other techniques may be used. In at least one embodiment, graphics processor 2508 couples to ring interconnect 2512 via I / O link 2513.

[0284] In at least one embodiment, I / O link 2513 represents at least one of several types of I / O interconnect, including an on-package I / O interconnect that facilitates communication between various processor components and a high-performance embedded memory module 2518, such as an eDRAM module. In at least one embodiment, each of processor cores 2502A-2502N and graphics processor 2508 use embedded memory module 2518 as a shared LLC.

[0285] In at least one embodiment, processor cores 2502A-2502N are homogeneous cores that execute a common instruction set architecture. In at least one embodiment, processor cores 2502A-2502N are heterogeneous in terms of ISA, where one or more of processor cores 2502A-2502N execute a common instruction set and one or more other of processor cores 2502A-2502N execute a subset of the common instruction set or a different instruction set. In at least one embodiment, processor cores 2502A-2502N are heterogeneous in terms of microarchitecture, where one or more cores with relatively high power consumption are combined with one or more cores with lower power consumption. In at least one embodiment, processor 2500 may be implemented on one or more chips or as an SoC integrated circuit.

[0286] In at least one embodiment, one or more systems illustrated in Figure 25 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 25 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 25 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 25 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0287] FIG. 26 illustrates a graphics processor core 2600 according to at least one described embodiment. In at least one embodiment, graphics processor core 2600 is included within a graphics core array. In at least one embodiment, graphics processor core 2600, sometimes referred to as a core slice, may be one or more graphics cores within a modular graphics processor. In at least one embodiment, graphics processor core 2600 is an example of a graphics core slice, and the graphics processors described herein may include multiple graphics core slices based on target power and performance envelopes. In at least one embodiment, each graphics core 2600 may include a fixed function block 2630 coupled with multiple sub-cores 2601A-2601F, also referred to as sub-slices, that include modular blocks of general-purpose and fixed-function logic.

[0288] In at least one embodiment, fixed function block 2630 includes a geometry / fixed function pipeline 2636 that may be shared by all sub-cores in graphics processor 2600, e.g., in lower performance and / or lower power graphics processor implementations. In at least one embodiment, geometry / fixed function pipeline 2636 includes a 3D fixed function pipeline, a video front end unit, a thread spawner and thread dispatcher, and a unified return buffer manager that manages a unified return buffer.

[0289] In at least one embodiment, fixed function block 2630 also includes a graphics SoC interface 2637, a graphics microcontroller 2638, and a media pipeline 2639. Graphics SoC interface 2637 provides an interface between graphics core 2600 and other processor cores within the SoC integrated circuit. In at least one embodiment, graphics microcontroller 2638 is a programmable sub-processor that is configurable to manage various functions of graphics processor 2600, including thread dispatch, scheduling, and preemption. In at least one embodiment, media pipeline 2639 includes logic for facilitating decoding, encoding, pre-processing, and / or post-processing of multimedia data, including image and video data. In at least one embodiment, media pipeline 2639 implements media operations via requests to compute logic or sampling logic within sub-cores 2601-2601F.

[0290] In at least one embodiment, SoC interface 2637 enables graphics core 2600 to communicate with a general-purpose application processor core (e.g., a CPU) and / or other components within the SoC, including memory hierarchy elements such as shared LLC memory, system RAM, and / or embedded on-chip or on-package DRAM. In at least one embodiment, SoC interface 2637 can also enable communication with fixed-function devices within the SoC, such as a camera imaging pipeline, and enable and / or implement global memory atomics that may be shared between graphics core 2600 and a CPU within the SoC. In at least one embodiment, SoC interface 2637 can also implement power management controls for graphics core 2600 and enable an interface between the graphics core 2600 clock domain and other clock domains within the SoC. In at least one embodiment, SoC interface 2637 enables receipt of command buffers from a command streamer and global thread dispatcher configured to provide commands and instructions to each of one or more graphics cores in the graphics processor. In at least one embodiment, the commands and instructions may be dispatched to a media pipeline 2639 when a media operation is to be performed, or to a geometry and fixed function pipeline (e.g., geometry and fixed function pipeline 2636, geometry and fixed function pipeline 2614) when a graphics processing operation is to be performed.

[0291] In at least one embodiment, graphics microcontroller 2638 can be configured to perform various scheduling and management tasks for graphics core 2600. In at least one embodiment, graphics microcontroller 2638 can perform graphics and / or compute workload scheduling for various graphics parallel engines in execution unit (EU) arrays 2602A-2602F, 2604A-2604F within sub-cores 2601A-2601F. In at least one embodiment, host software running on a CPU core of an SoC including graphics core 2600 can submit a workload to one of multiple graphics processor doorbells, which invokes scheduling operations on the appropriate graphics engine. In at least one embodiment, the scheduling operations include determining which workload to run next, submitting the workload to a command streamer, preempting existing workloads running on the engines, monitoring the progress of the workload, and notifying host software when the workload is complete. In at least one embodiment, graphics microcontroller 2638 can also facilitate low power or idle states for graphics core 2600, providing graphics core 2600 with the ability to save and restore registers within graphics core 2600 across low power state transitions independent of the operating system and / or graphics driver software on the system.

[0292] In at least one embodiment, graphics core 2600 may have up to N modular sub-cores, more or fewer than the illustrated sub-cores 2601A-2601F. For each set of N sub-cores, in at least one embodiment, graphics core 2600 may also include shared function logic 2610, shared and / or cache memory 2612, geometry / fixed function pipeline 2614, and additional fixed function logic 2616 for accelerating various graphics and compute processing operations. In at least one embodiment, shared function logic 2610 may include logic units (e.g., sampler, math, and / or inter-thread communication logic) that may be shared by each of the N sub-cores in graphics core 2600. Shared and / or cache memory 2612 may be an LLC for the N sub-cores 2601A-2601F in graphics core 2600 and may also serve as shared memory accessible by multiple sub-cores. In at least one embodiment, geometry / fixed function pipeline 2614 may be included in place of geometry / fixed function pipeline 2636 in fixed function block 2630 and may include the same or similar logical units.

[0293] In at least one embodiment, graphics core 2600 includes additional fixed function logic 2616, which can include various fixed-function acceleration logic for use by graphics core 2600. In at least one embodiment, additional fixed function logic 2616 includes an additional geometry pipeline for use in position-only shading. In position-only shading, there are at least two geometry pipelines: a full geometry pipeline in geometry / fixed function pipeline 2616, 2636, and a cull pipeline, which is an additional geometry pipeline that can be included in additional fixed function logic 2616. In at least one embodiment, the cull pipeline is a scaled-down version of the full geometry pipeline. In at least one embodiment, the full pipeline and the cull pipeline can run different instances of an application, each instance having a separate context. In at least one embodiment, position-only shading can hide long cull runs of truncated triangles, which allows shading to complete faster in some instances. For example, in at least one embodiment, the cull pipeline fetches and shades vertex position attributes without rasterizing and rendering pixels to the frame buffer, allowing the cull pipeline logic in the additional fixed-function logic 2616 to execute position shaders in parallel with the main application and produce critical results faster than the full pipeline. In at least one embodiment, the cull pipeline can use the generated critical results to calculate visibility information for all triangles, regardless of whether they are culled. In at least one embodiment, the full pipeline (which may be referred to in this instance as the replay pipeline) can consume the visibility information and shade only visible triangles, skipping over culled triangles, which are ultimately passed to the rasterization phase.

[0294] In at least one embodiment, the additional fixed function logic 2616 may also include general purpose processing acceleration logic, such as fixed function matrix multiplication logic, to accelerate CUDA programs.

[0295] In at least one embodiment, each graphics sub-core 2601A-2601F includes a set of execution resources that can be used to perform graphics operations, media operations, and compute operations in response to requests by a graphics pipeline, a media pipeline, or a shader program. In at least one embodiment, the graphics sub-cores 2601A-2601F include a plurality of EU arrays 2602A-2602F, 2604A-2604F, thread dispatch and inter-thread communication (“TD / IC”) logic 2603A-2603F, 3D (e.g., texture) samplers 2605A-2605F, media samplers 2606A-2606F, shader processors 2607A-2607F, and shared local memory (“SLM”) 2608A-2608F. EU arrays 2602A-2602F, 2604A-2604F each include multiple execution units, which are GPGPUs capable of performing floating-point and integer / fixed-point logic operations in service of graphics, media, or compute operations, including graphics, media, or compute shader programs. In at least one embodiment, TD / IC logic 2603A-2603F performs local thread dispatch and thread control operations for the execution units within a sub-core and facilitates communication between threads executing on the execution units of a sub-core. In at least one embodiment, 3D samplers 2605A-2605F can read textures or other 3D graphics-related data into memory. In at least one embodiment, the 3D samplers can read texture data differently based on the configured sample state and texture format associated with a given texture. In at least one embodiment, media samplers 2606A-2606F can perform similar read operations based on the type and format associated with the media data.In at least one embodiment, each graphics sub-core 2601A-2601F can alternatively include a unified 3D and media sampler. In at least one embodiment, threads executing on execution units within each of the sub-cores 2601A-2601F can utilize shared local memory 2608A-2608F within each sub-core to allow threads executing within a thread group to execute using a common pool of on-chip memory.

[0296] In at least one embodiment, one or more systems illustrated in Figure 26 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 26 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 26 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 26 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0297] FIG. 27 illustrates a parallel processing unit (“PPU”) 2700 according to at least one embodiment. In at least one embodiment, the PPU 2700 comprises machine-readable code that, when executed by the PPU 2700, causes the PPU 2700 to perform some or all of the processes and techniques described herein. In at least one embodiment, the PPU 2700 is a multi-threaded processor implemented on one or more integrated circuit devices and utilizing multi-threading as a latency-hiding technique designed to process computer-readable instructions (also referred to as machine-readable instructions or simply instructions) in parallel on multiple threads. In at least one embodiment, a thread refers to a thread of execution and is an instantiation of a set of instructions configured to be executed by the PPU 2700. In at least one embodiment, the PPU 2700 is a GPU configured to implement a graphics rendering pipeline for processing three-dimensional (“3D”) graphics data to generate two-dimensional (“2D”) image data for display on a display device, such as an LCD device. In at least one embodiment, PPU 2700 is utilized to perform computations such as linear algebra operations and machine learning operations. Figure 27 depicts an exemplary parallel processor for illustrative purposes only and should be construed as a non-limiting example of a processor architecture that may be implemented in at least one embodiment.

[0298] In at least one embodiment, one or more PPUs 2700 are configured to accelerate high performance computing ("HPC"), data center, and machine learning applications. In at least one embodiment, one or more PPUs 2700 are configured to accelerate CUDA programs. In at least one embodiment, the PPUs 2700 include, but are not limited to, an I / O unit 2706, a front end unit 2710, a scheduler unit 2712, a work distribution unit 2714, a hub 2716, a crossbar ("Xbar") 2720, one or more general processing clusters ("GPC") 2718, and one or more partition units ("memory partition units") 2722. In at least one embodiment, the PPU 2700 is connected to a host processor or other PPUs 2700 via one or more high-speed GPU interconnects (“GPU interconnects”) 2708. In at least one embodiment, the PPU 2700 is connected to a host processor or other peripheral devices via a system bus or interconnect 2702. In at least one embodiment, the PPU 2700 is connected to local memory comprising one or more memory devices (“memory”) 2704. In at least one embodiment, the memory devices 2704 include, but are not limited to, one or more dynamic random access memory (DRAM) devices. In at least one embodiment, the one or more DRAM devices are configured and / or configurable as a high-bandwidth memory (“HBM”) subsystem with multiple DRAM dies stacked within each device.

[0299] In at least one embodiment, the high-speed GPU interconnect 2708 may refer to a wire-based, multi-lane communication link used by the system to scale and include one or more PPUs 2700 in combination with one or more CPUs, supporting cache coherence between the PPUs 2700 and the CPUs and CPU mastering. In at least one embodiment, data and / or commands are transmitted by the high-speed GPU interconnect 2708 through a hub 2716 to and from other units of the PPU 2700, such as one or more copy engines, a video encoder, a video decoder, a power management unit, and other components that may not be explicitly shown in FIG. 27 .

[0300] In at least one embodiment, I / O unit 2706 is configured to receive and send communications (e.g., commands, data) from a host processor (not shown in FIG. 27 ) via system bus 2702. In at least one embodiment, I / O unit 2706 communicates with the host processor directly via system bus 2702 or through one or more intermediate devices, such as a memory bridge. In at least one embodiment, I / O unit 2706 may communicate with one or more other processors, such as one or more of PPUs 2700, via system bus 2702. In at least one embodiment, I / O unit 2706 implements a PCIe interface for communication over the PCIe bus. In at least one embodiment, I / O unit 2706 implements an interface for communicating with external devices.

[0301] In at least one embodiment, I / O unit 2706 decodes packets received via system bus 2702. In at least one embodiment, at least some of the packets represent commands configured to cause PPU 2700 to perform various operations. In at least one embodiment, I / O unit 2706 transmits the decoded commands to various other units of PPU 2700 specified by the commands. In at least one embodiment, the commands are transmitted to front end unit 2710 and / or to hub 2716 or other units of PPU 2700, such as one or more copy engines, video encoders, video decoders, or power management units (not explicitly shown in FIG. 27 ). In at least one embodiment, I / O unit 2706 is configured to route communications to and between various logical units of PPU 2700.

[0302] In at least one embodiment, a program executed by the host processor encodes a command stream in a buffer that provides a workload to the PPU 2700 for processing. In at least one embodiment, the workload includes instructions and data to be processed by those instructions. In at least one embodiment, the buffer is an area in memory that is accessible (e.g., readable / writable) by both the host processor and the PPU 2700, and the host interface unit may be configured to access the buffer in system memory connected to the system bus 2702 via memory requests sent by the I / O unit 2706 over the system bus 2702. In at least one embodiment, the host processor writes the command stream to the buffer and then sends a pointer to the start of the command stream to the PPU 2700, whereupon the front end unit 2710 receives the pointer to one or more command streams and manages the one or more command streams, reading commands from the command streams and forwarding the commands to various units of the PPU 2700.

[0303] In at least one embodiment, front end unit 2710 is coupled to a scheduler unit 2712 that configures various GPCs 2718 to process tasks defined by one or more command streams. In at least one embodiment, scheduler unit 2712 is configured to track state information related to the various tasks managed by scheduler unit 2712, where the state information may indicate which of GPCs 2718 a task is assigned to, whether the task is active or inactive, the priority level associated with the task, etc. In at least one embodiment, scheduler unit 2712 manages the execution of multiple tasks on one or more of GPCs 2718.

[0304] In at least one embodiment, scheduler unit 2712 is coupled to a work distribution unit 2714 configured to dispatch tasks for execution on GPCs 2718. In at least one embodiment, work distribution unit 2714 tracks the number of scheduled tasks received from scheduler unit 2712, and work distribution unit 2714 manages a pending task pool and an active task pool for each of GPCs 2718. In at least one embodiment, the pending task pool may comprise a number of slots (e.g., 32 slots) containing tasks assigned to be processed by a particular GPC 2718, and the active task pool may comprise a number of slots (e.g., 4 slots) for tasks being actively processed by the GPC 2718, such that when one of the GPCs 2718 completes execution of a task, the task is removed from the active task pool for the GPC 2718 and one of the other tasks from the pending task pool is selected and scheduled for execution on the GPC 2718. In at least one embodiment, when an active task is idle on the GPC2718, such as while waiting for a data dependency to be resolved, the active task is evicted from the GPC2718 and returned to the pending task pool, while another task in the pending task pool is selected and scheduled for execution on the GPC2718.

[0305] In at least one embodiment, work distribution unit 2714 communicates with one or more GPCs 2718 via X-bar 2720. In at least one embodiment, X-bar 2720 is an interconnection network coupling many units of PPU 2700 to other units of PPU 2700 and may be configured to couple work distribution unit 2714 to a particular GPC 2718. In at least one embodiment, one or more other units of PPU 2700 may also be connected to X-bar 2720 via hub 2716.

[0306] In at least one embodiment, tasks are managed by scheduler unit 2712 and dispatched by work distribution unit 2714 to one of GPCs 2718. GPC 2718 is configured to process the task and produce a result. In at least one embodiment, the result may be consumed by other tasks within GPC 2718, routed to a different GPC 2718 via Xbar 2720, or stored in memory 2704. In at least one embodiment, the result may be written to memory 2704 via partition unit 2722, which implements a memory interface for reading and writing data to / from memory 2704. In at least one embodiment, the result may be sent to another PPU 2704 or a CPU via high-speed GPU interconnect 2708. In at least one embodiment, the PPU 2700 includes U partition units 2722 equal to, but not limited to, the number of distinct individual memory devices 2704 coupled to the PPU 2700 .

[0307] In at least one embodiment, the host processor executes a driver kernel, which implements an application programming interface ("API") that allows one or more applications executing on the host processor to schedule operations for execution on the PPU 2700. In at least one embodiment, multiple compute applications are executed simultaneously by the PPU 2700, which provides isolation, quality of service ("QoS"), and independent address spaces for the multiple compute applications. In at least one embodiment, an application generates instructions (e.g., in the form of API calls) that cause the driver kernel to generate one or more tasks for execution by the PPU 2700, and the driver kernel outputs the tasks to one or more streams that are processed by the PPU 2700. In at least one embodiment, each task comprises one or more groups of participating threads, which may be referred to as a warp. In at least one embodiment, a warp comprises multiple participating threads (e.g., 32 threads) that may be executed in parallel. In at least one embodiment, cooperating threads can refer to multiple threads that contain instructions to perform a task and exchange data through a shared memory.

[0308] In at least one embodiment, one or more systems illustrated in Figure 27 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 27 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 27 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 27 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0309] 28 illustrates a GPC 2800, according to at least one embodiment. In at least one embodiment, the GPC 2800 is the GPC 2718 of FIG. 27. In at least one embodiment, each GPC 2800 includes several hardware units for processing tasks, including, but not limited to, a pipeline manager 2802, a pre-raster operation unit ("PROP") 2804, a raster engine 2808, a work distribution crossbar ("WDX") 2816, an MMU 2818, one or more data processing clusters ("DPC") 2806, and any suitable combination of parts.

[0310] In at least one embodiment, operation of the GPC 2800 is controlled by a pipeline manager 2802. In at least one embodiment, the pipeline manager 2802 manages the configuration of one or more DPCs 2806 to process tasks allocated to the GPC 2800. In at least one embodiment, the pipeline manager 2802 configures at least one of the one or more DPCs 2806 to implement at least a portion of a graphics rendering pipeline. In at least one embodiment, the DPC 2806 is configured to execute vertex shader programs on a programmable streaming multiprocessor (“SM”) 2814. In at least one embodiment, pipeline manager 2802 is configured to route packets received from the work distribution unit to the appropriate logical unit within GPC 2800; in at least one embodiment, some packets may be routed to a fixed function hardware unit in PROP 2804 and / or to Raster Engine 2808, while other packets may be routed to DPC 2806 for processing by Primitive Engine 2812 or SM 2814. In at least one embodiment, pipeline manager 2802 configures at least one of DPC 2806 to implement a computing pipeline. In at least one embodiment, pipeline manager 2802 configures at least one of DPC 2806 to execute at least a portion of a CUDA program.

[0311] In at least one embodiment, the PROP unit 2804 is configured to route data generated by the raster engine 2808 and the DPC 2806 to a raster operation ("ROP") unit in a partition unit, such as the memory partition unit 2722 described in more detail above in conjunction with FIG. 27. In at least one embodiment, the PROP unit 2804 is configured to perform optimizations for color blending, organize pixel data, perform address translation, and the like. In at least one embodiment, the raster engine 2808 includes several fixed-function hardware units configured to perform various raster operations, including, but not limited to, a setup engine, a coarse raster engine, a culling engine, a clipping engine, a fine raster engine, a tile coalescing engine, and any suitable combination thereof. In at least one embodiment, the setup engine receives the transformed vertices and generates plane equations associated with the geometric primitives defined by the vertices. The plane equations are sent to a coarse raster engine to generate coverage information for the primitives (e.g., x, y coverage masks for tiles). The output of the coarse raster engine is sent to a culling engine to cull fragments associated with primitives that fail a z-test and to a clipping engine to clip fragments that are outside the view frustum. In at least one embodiment, fragments that pass clipping and culling are passed to a fine raster engine to generate attributes for the pixel fragments based on the plane equations generated by the setup engine. In at least one embodiment, the output of the raster engine 2808 includes fragments to be processed by any suitable entity, such as by a fragment shader implemented within the DPC 2806.

[0312] In at least one embodiment, each DPC 2806 included in GPC 2800 includes, but is not limited to, an M-Pipe Controller (“MPC”) 2810, a Primitive Engine 2812, one or more SMs 2814, and any suitable combination thereof. In at least one embodiment, MPC 2810 controls the operation of DPC 2806, routing packets received from pipeline manager 2802 to the appropriate unit in DPC 2806. In at least one embodiment, packets related to vertices are routed to primitive engine 2812, which is configured to fetch vertex attributes associated with the vertices from memory; in contrast, packets related to shader programs may be sent to SM 2814.

[0313] In at least one embodiment, SM2814 includes a programmable streaming processor configured to process tasks represented by, but not limited to, several threads. In at least one embodiment, SM2814 is multithreaded and configured to simultaneously execute multiple threads (e.g., 32 threads) from a particular group of threads and implements a SIMD architecture, where each thread in a group of threads (e.g., a warp) is configured to process a different set of data based on the same set of instructions. In at least one embodiment, all threads in a group of threads execute the same instructions. In at least one embodiment, SM2814 implements a SIMT architecture, where each thread in a group of threads is configured to process a different set of data based on the same set of instructions, but individual threads in a group of threads are allowed to diverge during execution. In at least one embodiment, a program counter, call stack, and execution state are maintained for each warp to enable concurrent processing between warps and serial execution within a warp when threads within a warp diverge. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread to allow equal concurrency among all threads, within and between warps. In at least one embodiment, execution state is maintained for each individual thread, so that threads executing the same instructions can converge and run in parallel for better efficiency. At least one embodiment of SM2814 is described in further detail in conjunction with FIG. 29.

[0314] In at least one embodiment, MMU 2818 provides an interface between GPC 2800 and a memory partition unit (e.g., partition unit 2722 of FIG. 27), and MMU 2818 provides virtual to physical address translation, memory protection, and arbitration of memory requests. In at least one embodiment, MMU 2818 provides one or more translation lookaside buffers (TLBs) for performing translations from virtual addresses to physical addresses in memory.

[0315] In at least one embodiment, one or more systems illustrated in Figure 28 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 28 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 28 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 28 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0316] Figure 29 illustrates a streaming multiprocessor ("SM") 2900, according to at least one embodiment. In at least one embodiment, the SM 2900 is the SM 2814 of Figure 28. In at least one embodiment, the SM 2900 includes, but is not limited to, an instruction cache 2902, one or more scheduler units 2904, a register file 2908, one or more processing cores ("cores") 2910, one or more special function units ("SFUs") 2912, one or more LSUs 2914, an interconnect network 2916, a shared memory / L1 cache 2918, and any suitable combination thereof. In at least one embodiment, the work distribution unit dispatches tasks for execution on the GPCs of the parallel processing unit (PPU), with each task being assigned to a particular data processing cluster (DPC) within the GPC, and if the task is associated with a shader program, the task is assigned to one of the SMs 2900. In at least one embodiment, the scheduler unit 2904 receives tasks from the work distribution unit and manages instruction scheduling for one or more thread blocks assigned to the SMs 2900. In at least one embodiment, the scheduler unit 2904 schedules the thread blocks for execution as warps of parallel threads, with each thread block being assigned at least one warp. In at least one embodiment, each warp executes a thread. In at least one embodiment, scheduler unit 2904 manages multiple different thread blocks, allocates warps to the different thread blocks, and then dispatches instructions from multiple different interlocking groups to various functional units (e.g., processing cores 2910, SFUs 2912, and LSUs 2914) during each clock cycle.

[0317] In at least one embodiment, a "coordination group" may refer to a programming model for organizing groups of communicating threads, where the programming model allows developers to express the granularity at which threads are communicating, enabling richer and more efficient expression of parallel decompositions. In at least one embodiment, a coordination invocation API supports synchronization between thread blocks for the execution of parallel algorithms. In at least one embodiment, traditional programming model APIs provide a single simple construct for synchronizing coordinated threads, i.e., a barrier across all threads in a thread block (e.g., the syncthreads() function). However, in at least one embodiment, a programmer may define groups of threads at a granularity finer than a thread block and synchronize within the defined group, enabling higher performance, design flexibility, and software reuse in the form of a collective group-wide functional interface. In at least one embodiment, coordination groups allow programmers to explicitly define groups of threads at sub-block and multi-block granularity and perform collective operations, such as synchronization, on threads in the coordination group. In at least one embodiment, the sub-block granularity is as small as a single thread. In at least one embodiment, the programming model supports clean composition across software boundaries, allowing libraries and utility functions to safely synchronize within their local context without having to make assumptions about convergence. In at least one embodiment, the interlocking group primitive enables new patterns of interlocking parallelism, including, but not limited to, producer-consumer parallelism, opportunistic parallelism, and global synchronization across a grid of thread blocks.

[0318] In at least one embodiment, the dispatch unit 2906 is configured to send instructions to one or more of the functional units, and the scheduler unit 2904 includes, without limitation, two dispatch units 2906, allowing two different instructions from the same warp to be dispatched during each clock cycle. In at least one embodiment, each scheduler unit 2904 includes a single dispatch unit 2906 or additional dispatch units 2906.

[0319] In at least one embodiment, each SM2900 includes, in at least one embodiment, a register file 2908 that provides a set of registers to the functional units of the SM2900. In at least one embodiment, the register file 2908 is divided among each of the functional units such that each functional unit is allocated a dedicated portion of the register file 2908. In at least one embodiment, the register file 2908 is divided among different warps being executed by the SM2900, and the register file 2908 provides temporary storage for operands connected to the data paths of the functional units. In at least one embodiment, each SM2900 includes, in at least one embodiment, a plurality of, inclusive of L, processing cores 2910. In at least one embodiment, the SM2900 includes, inclusive of a large number (e.g., 128 or more) of individual processing cores 2910. In at least one embodiment, each processing core 2910 includes, but is not limited to, fully pipelined single-precision, double-precision, and / or mixed-precision processing units, including, but not limited to, a floating-point arithmetic logic unit and an integer arithmetic logic unit. In at least one embodiment, the floating-point arithmetic logic unit implements the IEEE 754-2008 standard for floating-point arithmetic. In at least one embodiment, processing core 2910 includes, but is not limited to, 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0320] In at least one embodiment, the tensor cores are configured to perform matrix operations. In at least one embodiment, one or more tensor cores are included in processing core 2910. In at least one embodiment, the tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inference. In at least one embodiment, each tensor core operates on a 4×4 matrix and performs a matrix multiply and accumulate operation D=A×B+C, where A, B, C, and D are 4×4 matrices.

[0321] In at least one embodiment, matrix multiplication inputs A and B are 16-bit floating-point matrices, and sum matrices C and D are 16-bit floating-point or 32-bit floating-point matrices. In at least one embodiment, the tensor cores operate on 16-bit floating-point input data with a 32-bit floating-point sum. In at least one embodiment, the 16-bit floating-point multiplication uses 64 operations, resulting in a full-precision product, which is then added using 32-bit floating-point addition with other intermediate products for a 4x4x4 matrix multiplication. In at least one embodiment, the tensor cores are used to perform much larger two-dimensional or even higher-dimensional matrix operations that build up from these smaller elements. In at least one embodiment, an API such as the CUDA-C++ API exposes specialized matrix load, matrix multiply-and-add, and matrix store operations to efficiently use the tensor cores from CUDA-C++ programs. In at least one embodiment, at the CUDA level, the warp-level interface assumes a matrix of size 16x16 that spans all 32 threads of a warp.

[0322] In at least one embodiment, each SM2900 includes M SFUs 2912 that perform special functions (e.g., but not limited to, attribute evaluation, reciprocal square root, etc.). In at least one embodiment, the SFUs 2912 include, but are not limited to, a tree traversal unit configured to traverse a hierarchical tree data structure. In at least one embodiment, the SFUs 2912 include, but are not limited to, a texture unit configured to perform texture map filtering operations. In at least one embodiment, the texture unit is configured to load texture maps (e.g., 2D arrays of texels) from memory and sample texture maps to produce sampled texture values ​​for use in shader programs executed by the SM2900. In at least one embodiment, the texture maps are stored in shared memory / L1 cache 2918. In at least one embodiment, the texture unit implements texture operations such as filtering operations using mip maps (e.g., texture maps with different levels of detail). In at least one embodiment, each SM2900 includes, but is not limited to, two texture units.

[0323] In at least one embodiment, each SM 2900 includes, but is not limited to, N LSUs 2914 that implement load and store operations between shared memory / L1 cache 2918 and register file 2908. In at least one embodiment, each SM 2900 includes, but is not limited to, an interconnection network 2916 that connects each of the functional units to register file 2908 and connects LSUs 2914 to register file 2908 and shared memory / L1 cache 2918. In at least one embodiment, interconnection network 2916 is a crossbar that can be configured to connect any of the functional units to any of the registers in register file 2908 and connect LSUs 2914 to memory locations in register file 2908 and shared memory / L1 cache 2918.

[0324] In at least one embodiment, the shared memory / L1 cache 2918 is an array of on-chip memory that enables data storage and communication between the SM2900 and the primitive engines and between threads in the SM2900. In at least one embodiment, the shared memory / L1 cache 2918 has, but is not limited to, 128 KB of storage capacity and is in the path from the SM2900 to the partition unit. In at least one embodiment, the shared memory / L1 cache 2918 is used to cache reads and writes. In at least one embodiment, one or more of the shared memory / L1 cache 2918, the L2 cache, and the memory are backing stores.

[0325] In at least one embodiment, combining data cache and shared memory functionality into a single memory block provides improved performance for both types of memory access. In at least one embodiment, the capacity is used as a cache or available for use by programs that do not use shared memory, such as when the shared memory is configured to use half the capacity and texture and load / store operations can use the remaining capacity. In at least one embodiment, the integration within the shared memory / L1 cache 2918 allows the shared memory / L1 cache 2918 to function as a high-throughput conduit for streaming data while simultaneously providing high-bandwidth, low-latency access to frequently reused data. In at least one embodiment, when configured for general-purpose parallel computation, a simpler configuration may be used compared to graphics processing. In at least one embodiment, the fixed-function GPU is bypassed, creating a much simpler programming model. In at least one embodiment, and in the general-purpose parallel computation configuration, the work distribution unit directly assigns and distributes blocks of threads to the DPCs. In at least one embodiment, blocked threads execute the same program using unique thread IDs in computations to ensure each thread produces unique results, use SM2900 to execute the program and perform computations, use shared memory / L1 cache 2918 to communicate between threads, and use LSU2914 to read and write global memory through shared memory / L1 cache 2918 and memory partition unit 2914. In at least one embodiment, when configured for general-purpose parallel computation, SM2900 writes commands that scheduler unit 2904 can use to launch new work on DPCs.

[0326] In at least one embodiment, the PPU is included in or coupled to a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smart phone (e.g., a wireless handheld device), a PDA, a digital camera, a vehicle, a head-mounted display, a handheld electronic device, etc. In at least one embodiment, the PPU is embodied on a single semiconductor substrate. In at least one embodiment, the PPU is included in an SoC with one or more other devices, such as additional PPUs, memory, a RISC CPU, an MMU, a digital-to-analog converter ("DAC"), etc.

[0327] In at least one embodiment, the PPU may be included on a graphics card that includes one or more memory devices. In at least one embodiment, the graphics card may be configured to interface with a PCIe slot on a motherboard of a desktop computer. In at least one embodiment, the PPU may be an integrated GPU ("iGPU") included in the chipset of the motherboard.

[0328] In at least one embodiment, one or more systems illustrated in Figure 29 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 29 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 29 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 29 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0329] Software Constructs for General-Purpose Computing The following diagram describes an exemplary software construct for implementing at least one embodiment, but is not limited to this embodiment.

[0330] FIG. 30 illustrates a software stack for a programming platform, according to at least one embodiment. In at least one embodiment, the programming platform is a platform for leveraging hardware on a computing system to accelerate computational tasks. In at least one embodiment, the programming platform may be accessible to software developers through libraries, compiler directives, and / or extensions to a programming language. In at least one embodiment, the programming platform may be, but is not limited to, CUDA, Radeon Open Compute Platform ("ROCm"), OpenCL (OpenCL™ is developed by the Khronos group), SYCL, or the Intel One API.

[0331] In at least one embodiment, the programming platform's software stack 3000 provides an execution environment for an application 3001. In at least one embodiment, the application 3001 may include any computer software capable of being run on the software stack 3000. In at least one embodiment, the application 3001 may include, but is not limited to, an artificial intelligence (“AI”) / machine learning (“ML”) application, a high performance computing (“HPC”) application, a virtual desktop infrastructure (“VDI”), or a data center workload.

[0332] In at least one embodiment, application 3001 and software stack 3000 run on hardware 3007. In at least one embodiment, hardware 3007 may include one or more GPUs, CPUs, FPGAs, AI engines, and / or other types of compute devices supporting a programming platform. In at least one embodiment, such as in the case of CUDA, software stack 3000 may be vendor-specific and compatible only with devices from a particular vendor(s). In at least one embodiment, such as in the case of OpenCL, software stack 3000 may be used with devices from different vendors. In at least one embodiment, hardware 3007 includes a host connected to another device that may be accessed to perform computational tasks via application programming interface (“API”) calls. In at least one embodiment, in contrast to a host in hardware 3007, which may include, but is not limited to, a CPU (but may also include a compute device) and its memory, a device in hardware 3007 may include, but is not limited to, a GPU, FPGA, AI engine, or other compute device (but may also include a CPU) and its memory.

[0333] In at least one embodiment, the programming platform's software stack 3000 includes, but is not limited to, several libraries 3003, a runtime 3005, and a device kernel driver 3006. In at least one embodiment, each of the libraries 3003 may include data and programming code used by a computer program and utilized during software development. In at least one embodiment, the libraries 3003 may include, but are not limited to, pre-written code and subroutines, classes, values, type specifications, configuration data, documentation, help data, and / or message templates. In at least one embodiment, the libraries 3003 include functionality optimized for execution on one or more types of devices. In at least one embodiment, the libraries 3003 may include, but are not limited to, functionality for performing mathematics, deep learning, and / or other types of operations on the device. In at least one embodiment, the libraries 3003 are associated with corresponding APIs 3002, which may include one or more APIs that expose the functionality implemented in the libraries 3003.

[0334] In at least one embodiment, application 3001 is written as source code that is compiled into executable code, as described in more detail below in conjunction with FIGS. 35-37. In at least one embodiment, the executable code of application 3001 may run, at least in part, on an execution environment provided by software stack 3000. In at least one embodiment, during execution of application 3001, code that needs to run on the device, as opposed to the host, may be reached. In at least one embodiment, in such cases, runtime 3005 may be called to load and launch the requisite code on the device. In at least one embodiment, runtime 3005 may include any technically feasible runtime system capable of supporting execution of application S01.

[0335] In at least one embodiment, runtime 3005 is implemented as one or more runtime libraries associated with corresponding API(s), shown as API(s) 3004. In at least one embodiment, one or more such runtime libraries may include, but are not limited to, functionality for memory management, execution control, device management, error handling, and / or synchronization, among others. In at least one embodiment, memory management functionality may include, but is not limited to, functionality for allocating, deallocating, and copying device memory, as well as transferring data between host and device memory. In at least one embodiment, execution control functionality may include, but is not limited to, functionality for launching functions on the device (sometimes referred to as "kernels" when the functions are global functions callable from the host) and setting attribute values ​​in buffers maintained by the runtime library for a given function to be executed on the device.

[0336] In at least one embodiment, the runtime library and corresponding API(s) 3004 may be implemented in any technically feasible manner. In at least one embodiment, one (or any number of) APIs may expose a low-level set of functionality for fine-grained control of the device, while another (or any number of) APIs may expose a higher-level set of such functionality. In at least one embodiment, a high-level runtime API may be built on top of the low-level API. In at least one embodiment, one or more of the runtime APIs may be language-specific APIs layered on top of a language-agnostic runtime API.

[0337] In at least one embodiment, device kernel driver 3006 is configured to facilitate communication with the underlying device. In at least one embodiment, device kernel driver 3006 may provide low-level functionality upon which APIs, such as API(s) 3004 and / or other software rely. In at least one embodiment, device kernel driver 3006 may be configured to compile intermediate representation (“IR”) code into binary code at runtime. In at least one embodiment, in the case of CUDA, device kernel driver 3006 may compile non-hardware-specific Parallel Thread Execution (“PTX”) IR code into binary code for a particular target device at runtime (with caching of the compiled binary code), which may be referred to as “finalizing” the code. In at least one embodiment, doing so may allow the finalized code to run on the target device, which may not have been present when the source code was initially compiled into PTX code. Alternatively, in at least one embodiment, the device source code may be compiled into binary code offline without requiring the device kernel driver 3006 to compile the IR code at run time.

[0338] In at least one embodiment, one or more systems illustrated in Figure 30 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 30 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 30 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 30 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0339] Figure 31 illustrates a CUDA implementation of the software stack 3000 of Figure 30, according to at least one embodiment. In at least one embodiment, the CUDA software stack 3100, upon which an application 3101 may be launched, includes a CUDA library 3103, a CUDA runtime 3105, a CUDA driver 3107, and a device kernel driver 3108. In at least one embodiment, the CUDA software stack 3100 executes on hardware 3109, which may include a GPU that supports CUDA and is developed by NVIDIA Corporation of Santa Clara, California.

[0340] In at least one embodiment, application 3101, CUDA runtime 3105, and device kernel driver 3108 may perform functionality similar to application 3001, runtime 3005, and device kernel driver 3006, respectively, described above in conjunction with FIG. 30. In at least one embodiment, CUDA driver 3107 includes a library (libcuda.so) that implements CUDA driver API 3106. In at least one embodiment, similar to CUDA runtime API 3104 implemented by the CUDA runtime library (cudart), CUDA driver API 3106 may expose functionality for, among other things, but not limited to, memory management, execution control, device management, error handling, synchronization, and / or graphics interoperability. In at least one embodiment, the CUDA driver API 3106 differs from the CUDA runtime API 3104 in that the CUDA runtime API 3104 simplifies device code management by providing implicit initialization, context management (similar to processes), and module management (similar to dynamically loaded libraries). In at least one embodiment, in contrast to the high-level CUDA runtime API 3104, the CUDA driver API 3106 is a low-level API that provides more fine-grained control of the device, particularly with respect to context and module loading. In at least one embodiment, the CUDA driver API 3106 may expose functionality for context management not exposed by the CUDA runtime API 3104. In at least one embodiment, the CUDA driver API 3106 is also language-independent, e.g., supports OpenCL in addition to the CUDA runtime API 3104. Additionally, in at least one embodiment, the development libraries including the CUDA runtime 3105 may be considered separate from the driver components including the user-mode CUDA driver 3107 and the kernel-mode device driver 3108 (sometimes referred to as the "display" driver).

[0341] In at least one embodiment, CUDA libraries 3103 may include, but are not limited to, math libraries, deep learning libraries, parallel algorithm libraries, and / or signal / image / video processing libraries that may be utilized by parallel computing applications such as application 3101. In at least one embodiment, CUDA libraries 3103 may include, among other things, math libraries such as the cuBLAS library, which is an implementation of the Basic Linear Algebra Subprograms (“BLAS”) for performing linear algebra operations, the cuFFT library for computing fast Fourier transforms (“FFTs”), and the cuRAND library for generating random numbers. In at least one embodiment, CUDA libraries 3103 may include, among other deep learning libraries, such as the cuDNN library of primitives for deep neural networks and the TensorRT platform for high-performance deep learning inference.

[0342] In at least one embodiment, one or more systems illustrated in Figure 31 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 31 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 31 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 31 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0343] Figure 32 illustrates a ROCm implementation of software stack 3000 of Figure 30, according to at least one embodiment. In at least one embodiment, ROCm software stack 3200, upon which application 3201 may be launched, includes language runtime 3203, system runtime 3205, thunk 3207, and ROCm kernel driver 3208. In at least one embodiment, ROCm software stack 3200 runs on hardware 3209, which may include a GPU, which supports ROCm and is developed by AMD Corporation of Santa Clara, California.

[0344] In at least one embodiment, application 3201 may perform functionality similar to application 3001 described above in conjunction with FIG. 30. Additionally, in at least one embodiment, language runtime 3203 and system runtime 3205 may perform functionality similar to runtime 3005 described above in conjunction with FIG. 30. In at least one embodiment, language runtime 3203 and system runtime 3205 differ in that system runtime 3205 is a language-agnostic runtime that implements ROCr system runtime API 3204 and utilizes the Heterogeneous System Architecture ("HSA") runtime API. In at least one embodiment, the HSA runtime API is a thin user-mode API that exposes an interface for accessing and interacting with an AMD GPU, including functions for, among other things, memory management, execution control via kernel architected dispatch, error handling, system and agent information, and runtime initialization and shutdown. In at least one embodiment, the language runtime 3203, as opposed to the system runtime 3205, is an implementation of a language-specific runtime API 3202 layered on top of the ROCr system runtime API 3204. In at least one embodiment, the language runtime API may include, but is not limited to, the Heterogeneous Compute Interface for Portability (“HIP”) language runtime API, the Heterogeneous Compute Compiler (“HCC”) language runtime API, or the OpenCL API, among others. In particular, the HIP language is an extension of the C++ programming language with a functionally similar version of the CUDA facilities, and in at least one embodiment, the HIP language runtime API includes functionality similar to the functionality of the CUDA runtime API 3104 described above in conjunction with FIG. 31, such as functionality for memory management, execution control, device management, error handling, and synchronization, among others.

[0345] In at least one embodiment, thunk (ROCt) 3207 is an interface 3206 that can be used to interact with an underlying ROCm driver 3208. In at least one embodiment, ROCm driver 3208 is a ROCk driver that is a combination of an AMDGPU driver and an HSA kernel driver (amdkfd). In at least one embodiment, the AMDGPU driver is a device kernel driver for GPUs developed by AMD that implements functionality similar to device kernel driver 3006 described above in conjunction with FIG. 30. In at least one embodiment, the HSA kernel driver is a driver that allows different types of processors to more effectively share system resources through hardware features.

[0346] In at least one embodiment, various libraries (not shown) may be included in the ROCm software stack 3200 above the language runtime 3203 and may provide functionality similar to the CUDA library 3103 described above in conjunction with Figure 31. In at least one embodiment, the various libraries may include, but are not limited to, mathematics, deep learning, and / or other libraries, such as, among others, a hipBLAS library that implements functionality similar to that of CUDA cuBLAS, a rocFFT library for computing FFTs that are similar to CUDA cuFFT, and the like.

[0347] In at least one embodiment, one or more systems illustrated in Figure 32 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 32 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 32 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 32 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0348] Figure 33 illustrates an OpenCL implementation of software stack 3000 of Figure 30, according to at least one embodiment. In at least one embodiment, OpenCL software stack 3300, upon which application 3301 may be launched, includes OpenCL framework 3310, OpenCL runtime 3306, and driver 3307. In at least one embodiment, OpenCL software stack 3300 executes on non-vendor-specific hardware 3109. In at least one embodiment, OpenCL is supported by devices developed by different vendors, and therefore, specific OpenCL drivers may be required to interoperate with hardware from such vendors.

[0349] In at least one embodiment, application 3301, OpenCL runtime 3306, device kernel driver 3307, and hardware 3308 may perform functionality similar to application 3001, runtime 3005, device kernel driver 3006, and hardware 3007, respectively, described above in conjunction with Figure 30. In at least one embodiment, application 3301 further includes an OpenCL kernel 3302 with code to be executed on the device.

[0350] In at least one embodiment, OpenCL defines a "platform" that allows a host to control devices connected to the host. In at least one embodiment, the OpenCL framework provides a platform layer API and a runtime API, shown as platform API 3303 and runtime API 3305. In at least one embodiment, the runtime API 3305 uses contexts to manage the execution of kernels on devices. In at least one embodiment, each identified device may be associated with a respective context, and the runtime API 3305 may use each context to manage command queues, program objects, and kernel objects for that device, among other things, and share memory objects. In at least one embodiment, the platform API 3303 exposes functions that allow device contexts to be used to, among other things, select and initialize devices, submit work to devices via command queues, and enable data transfer to and from devices. In at least one embodiment, the OpenCL framework also provides various built-in functions (not shown), including, among other things, mathematical functions, relational functions, and image processing functions.

[0351] In at least one embodiment, a compiler 3304 is also included in the OpenCL framework 3310. In at least one embodiment, source code may be compiled offline prior to executing the application or online while the application is running. In contrast to CUDA and ROCm, OpenCL applications in at least one embodiment may be compiled online by compiler 3304, which is included to represent any number of compilers that may be used to compile source code and / or IR code into binary code, such as Standard Portable Intermediate Representation ("SPIR-V") code. Alternatively, in at least one embodiment, OpenCL applications may be compiled offline prior to executing such applications.

[0352] In at least one embodiment, one or more systems illustrated in Figure 33 are utilized to implement an API to generate one or more graph code nodes for allocating memory. In at least one embodiment, one or more systems illustrated in Figure 33 are utilized to implement an API to generate one or more graph code nodes for deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 33 are utilized to implement an API to generate one or more graph code nodes for allocating and deallocating memory. In at least one embodiment, one or more systems illustrated in Figure 33 are utilized to implement one or more systems and / or processes, such as those described with respect to Figures 1-10.

[0353] 34 illustrates software supported by a programming platform, according to at least one embodiment. In at least one embodiment, programming platform 3404 is configured to support various programming models 3403, middleware and / or libraries 3402, and frameworks 3401 upon which applications 3400 may rely. In at least one embodiment, applications 3400 may be AI / ML applications implemented using deep learning frameworks such as MXNet, PyTorch, or TensorFlow, which may rely on libraries such as cuDNN, the NVIDIA Collective Communications Library (“NCCL”), and / or the NVIDIA Developer ...

Claims

1. A circuit that implements an application programming interface (API) for adding one or more first graph code nodes to one or more graphs for allocating memory. one or more processors comprising: one or more processors, wherein adding the one or more first graph code nodes causes one or more second graph code nodes in the one or more graphs to have one or more memory allocation dependencies on the one or more first graph code nodes.

2. The circuit further comprises: Obtaining code that indicates at least the API; Implementing the API by executing at least the code; One or more processors according to claim 1 .

3. The circuit further comprises: Generate a graph data structure; adding the one or more first graph code nodes as part of the graph data structure; One or more processors according to claim 1 .

4. The one or more processors of claim 1 , wherein the circuitry is further configured to implement the API based at least in part on one or more parameter values ​​indicating at least properties of the memory to be allocated.

5. 2. The one or more processors of claim 1, wherein the one or more first graph code nodes for allocating the memory correspond to a set of graph code nodes for deallocating the memory.

6. 10. The one or more processors of claim 1, wherein the circuitry further causes a graphics processing unit (GPU) to allocate the memory based at least in part on the one or more first graph code nodes.

7. The one or more processors of claim 1 , wherein the circuitry further causes one or more devices to perform one or more operations using the memory.

8. One or more computers having one or more processors for implementing an application programming interface (API) for adding one or more first graph code nodes to one or more graphs for allocating memory. A system comprising: adding the one or more first graph code nodes causes one or more second graph code nodes in the one or more graphs to have one or more memory allocation dependencies on the one or more first graph code nodes.

9. The system of claim 8 , wherein the one or more processors further implement the API based at least in part on a set of parameter values ​​indicating at least a size of the memory to be allocated.

10. 9. The system of claim 8, wherein the one or more processors further cause a parallel processing unit (PPU) to allocate the memory using the one or more first graph code nodes.

11. The system of claim 8 , wherein the one or more first graph code nodes encode properties of the allocated memory.

12. the one or more processors further comprising: Obtaining a graph data structure representing one or more operations; causing one or more devices to use the graph data structure to perform the one or more operations using the allocated memory; The system of claim 8.

13. The system of claim 8 , wherein the API is a runtime API.

14. 1. A method implemented by one or more processors, comprising: receiving an application programming interface (API) call to add one or more first graph code nodes in the graph for allocating memory to one or more graphs; implementing the API to add the one or more first graph code nodes to the one or more graphs for allocating memory; Including, wherein adding the one or more first graph code nodes causes one or more second graph code nodes in the one or more graphs to have one or more memory allocation dependencies on the one or more first graph code nodes.

15. 15. The method of claim 14, further comprising adding the one or more first graph code nodes as part of a graph data structure.

16. The method of claim 14 , further comprising obtaining code including parameter values ​​for the API.

17. 15. The method of claim 14, wherein the one or more first graph code nodes are data objects that encode information regarding memory allocation, and further wherein the information is calculated based at least in part on one or more parameter values.

18. The method of claim 14 , wherein the API is a driver API.

19. adding the one or more first graph code nodes as part of a first graph data structure; allocating the memory based at least in part on the one or more first graph code nodes; obtaining a second graph data structure representing one or more operations; causing one or more devices to utilize the allocated memory to perform the one or more operations; 15. The method of claim 14, further comprising:

20. 15. The method of claim 14, further comprising causing a general purpose graphics processing unit (GPGPU) to allocate the memory using the one or more first graph code nodes.

21. A circuit that implements an application programming interface (API) for adding one or more first graph code nodes for allocating memory and one or more second graph code nodes for deallocating memory to one or more graphs. one or more processors comprising: one or more processors, wherein adding the one or more first graph code nodes and the one or more second graph code nodes causes the one or more second graph code nodes to have one or more memory allocation dependencies with the one or more first graph code nodes.

22. 22. The one or more processors of claim 21 , wherein the circuitry is further to generate a first graph data structure and add the one or more first graph code nodes as part of the first graph data structure.

23. 22. The one or more processors of claim 21, wherein the circuitry further causes a device to allocate the memory based at least in part on the identified memory region.

24. 22. The one or more processors of claim 21, wherein the circuitry further implements the API based at least in part on parameter values ​​indicating constraints on allocating and deallocating the memory.

25. 22. The one or more processors of claim 21, wherein the circuitry further causes one or more devices to use the memory to perform a set of operations.

26. 1. A method implemented by one or more processors, comprising: receiving an application programming interface (API) call to add one or more first graph code nodes in the graph for allocating memory and one or more second graph code nodes in the graph for deallocating memory to one or more graphs; implementing the API to add the one or more first graph code nodes for allocating memory and the one or more second graph code nodes for deallocating memory to the one or more graphs; Including, wherein adding the one or more first graph code nodes and the one or more second graph code nodes causes the one or more second graph code nodes to have one or more memory allocation dependencies with the one or more first graph code nodes.

27. generating a first graph data structure and a second graph data structure; adding the one or more first graph code nodes as part of the first graph data structure and adding the one or more second graph code nodes as part of the second graph data structure; 27. The method of claim 26, further comprising:

28. 27. The method of claim 26, further comprising causing a central processing unit (CPU) to use the one or more first graph code nodes to allocate the memory and the one or more second graph code nodes to deallocate the memory.

29. computing a first set of actions to be performed by one or more devices; causing the one or more devices to use the memory to perform the first set of operations; 27. The method of claim 26, further comprising:

30. 27. The method of claim 26, further comprising causing one or more devices to perform one or more operations indicated by a graph data structure including at least one of the one or more first graph code nodes and the one or more second graph code nodes.

Images