Memory management using registers

Abstract

The invention discloses memory management using registers. A first request to allocate one or more memory blocks of a first plurality of memory blocks associated with a first memory is received by a processing device. A contiguous set of first bit portions of a first register having a first logic state is identified. The first logic state indicates that a corresponding memory block of the one or more memory blocks is free. A first operation is performed that adjusts a contiguous set of first bit portions of the first register to a second logical state. An allocated address including an index of a contiguous set of first bit portions of the first register is sent to the first request, where the allocated address is available to access a corresponding memory block.

Description

Technical Field

[0001] The present disclosure relates to memory management in computer systems, and more particularly, to memory management using registers. Background Art

[0002] Modern computer systems often include memory management features that enable programs to request memory during runtime. Programs can use memory to store data, and the amount of memory used by a program can vary over time. A program can submit requests for memory and requests to release memory, and memory allocation and deallocation can be performed separately. Memory allocation can involve identifying available memory and enabling a program to use that memory. Memory deallocation can involve freeing previously used memory so that it can be reused later. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 An illustrative architecture for a computing device in accordance with at least one embodiment of the present disclosure is depicted.

[0004] Figure 2 An illustrative architecture of a parallel processing unit (PPU) in accordance with at least one embodiment of the present disclosure is depicted.

[0005] Figure 3 Depicted are examples of tensor memory and software status registers for accelerating bitmap allocation in accordance with at least one embodiment of the present disclosure.

[0006] Figure 4 A flowchart of a method for ensuring memory allocation fairness according to at least one embodiment of the present disclosure is shown.

[0007] Figure 5 A flowchart of a method for memory management using registers according to at least one embodiment of the present disclosure is shown.

[0008] Figure 6 A parallel processing unit according to an embodiment is shown.

[0009] Figure 7A It shows the embodiment Figure 6 A general-purpose processing cluster within a parallel processing unit.

[0010] Figure 7B It shows the embodiment Figure 6 A memory partitioning unit of a parallel processing unit.

[0011] Figure 8A It shows the embodiment Figure 7A Streaming multiprocessor.

[0012] Figure 8B is used according to the embodiment Figure 6 Conceptual diagram of the processing system implemented by the PPU.

[0013] Figure 8C An exemplary system is shown that can implement the various architecture and / or functionality of various previous embodiments.

[0014] Figure 9 According to the embodiment Figure 6 Conceptual diagram of the graphics processing pipeline implemented by the PPU. DETAILED DESCRIPTION

[0015] Modern computing systems include memory management components that manage the allocation and deallocation of memory and include techniques to optimize memory reuse. Memory reuse can involve fulfilling a memory allocation request with previously deallocated memory. Memory management involves dividing available memory into memory blocks, tracking the status of the memory blocks, and providing memory blocks to programs / processes when requested.

[0016] A memory management strategy includes using a bitmap structure for memory allocation / deallocation. Bitmap allocation is a memory management technique that uses a bit sequence (e.g., a bitmap) to represent the status of memory blocks in a memory. Memory can be logically divided into memory blocks (e.g., fixed-size memory blocks, variable-size memory blocks), and when a program requests a memory block, software (e.g., an operating system, a memory allocator, etc.) is responsible for allocating and deallocating memory blocks. Each bit in the bitmap can correspond to a specific memory block, and its value can indicate whether the block is free or allocated. For example, a bit in the bitmap with a logical value of '0' can indicate that the corresponding memory block is free. A given bit in the bitmap with a logical value of '1' can indicate that the corresponding memory block is allocated. Initially, the memory can be logically considered to be a set of contiguous free memory blocks, and the corresponding bitmap is initialized with all bits set to '0'. When a program requests memory, a software algorithm can search the bitmap for a sequence of free blocks that provide the requested size. For example, a software algorithm can create a bit mask representing the allocation size and find a set of contiguous bits with a logical value of "0" that matches the mask. Once found, the software algorithm can set the corresponding bit to "1" and return the address to the program. When managing a memory pool that involves applications allocating / deallocating contiguous memory blocks of fixed sizes, bitmap allocation can be an efficient technique for tracking the allocation status of each memory block.

[0017] Conventional systems may maintain a bitmap data structure in software, with the software layer responsible for initializing, updating, and querying the bitmap data structure. However, the runtime of such software can have long execution times. For example, conventional software implementations may use atomic instructions that require 300 or more clock cycles to execute. Furthermore, conventional implementations of memory allocation using bitmaps can lead to memory fragmentation. Memory fragmentation occurs when free memory blocks are scattered throughout the address space in discontinuous blocks. Memory fragmentation can lead to a situation where there may be enough memory overall to satisfy a memory allocation request, but the memory is not contiguous, making it difficult or even impossible to satisfy a memory allocation request for many memory blocks.

[0018] Aspects and implementations of the present disclosure address the aforementioned shortcomings and other shortcomings of conventional memory management systems by providing a register (referred to herein as a "software status register") to store a bitmap and instructions for operating on the software status register. It is strongly noted that the term software status register is used herein for convenience and not limitation; a software status register is a hardware register that can store values ​​that can be used by one or more of the techniques described herein. The software status register can be designed so that software (e.g., an operating system (OS), a driver, etc.) can use defined capabilities and operations to implement / accelerate various software algorithms. In at least one embodiment, one or more operations can be defined as operations on the software status register to perform memory allocation / deallocation. In one example, a computing device may receive a request to allocate memory. In at least one embodiment, the request may include the size of the requested allocation. For example, the request may include an allocation size of '7', indicating a request to allocate seven memory blocks, each of which has a predetermined size. The software status register may identify seven consecutive bits having a predetermined logical state indicating that the corresponding memory blocks are free. For example, the computing device may identify two consecutive bits having a logical state of '0' indicating that the corresponding memory blocks are free. The computing device may update the identified two consecutive bits of the software status register to a logic state of '1', indicating that the corresponding memory block is now allocated. The computing device may return an index of the two consecutive bits, which may be used as an address to access the corresponding memory block. Thus, utilizing the software status register allows allocation to be performed using a single instruction, rather than executing 100 or more instructions to query and update a software-implemented bitmap.

[0019] In at least one embodiment, aspects and implementations of the present disclosure can address memory fragmentation by implementing an allocation policy that enforces alignment of memory allocations with their sizes. For example, an allocation policy can only allow a requestor to request a subset of allocation sizes, such as 32KB, 64KB, and 128KB. Allocation sizes can be limited accordingly, thereby mitigating memory fragmentation.

[0020] In at least one embodiment, aspects and implementations of the present disclosure may be implemented in a dedicated hardware architecture such as a parallel processing unit (PPU). A PPU is designed to extract high performance using a large number of small parallel execution threads on a dedicated programmable multiprocessor. In a PPU, a group of threads (e.g., a warp) can concurrently execute the same instruction on a multiprocessor (e.g., a streaming multiprocessor (SM), a compute unit (CU), etc.) with different input data. This execution mode is called single instruction multiple thread (SIMT), which is commonly used for parallel computing. The PPU is designed to execute programs (e.g., kernels, shader programs, etc.) in parallel by many thread groups on the PPU, where each thread in the thread group typically operates on a different part of the data.

[0021] In at least one embodiment, a software status register may be included within a multiprocessor of a PPU and may be used to allocate memory provided on the corresponding multiprocessor. For example, a PPU may include multiple texture processor clusters (TPCs), which are higher-level groupings of the architecture and capabilities of a portion of the PPU. Each TPC may include two or more multiprocessors, each of which includes a processing core, functional units, registers, memory, and other components. In an illustrative example, a multiprocessor may include tensor memory. The memory management techniques described herein may allow multiple thread groups to collaboratively use the tensor memory of a given multiprocessor. For example, the allocation of tensor memory may be managed by a programming model associated with the PPU that provides a set of operations for allocating and deallocating tensor memory using software status registers. The PPU may implement allocation requests by using software status registers to identify free blocks of tensor memory.

[0022] In some instances, two different processes may request a uniform allocation of corresponding memory, meaning that the two different processors may request the same allocation address returned. For example, two multiprocessors of a given TPC of a PPU may request a uniform allocation of corresponding tensor memory. Such requests are typically handled using complex software algorithms to coordinate between the two independent requestors and implement uniform memory allocation. In at least one embodiment, uniform memory allocation can be simplified by defining an operation in the PPU programming model that enables the same tensor memory allocation to be executed across two or more multiprocessors of the TPC and return the same allocation address.

[0023] In some embodiments, software stage registers may be designed to allow software to implement one or more algorithms in addition to implementing memory allocation / deallocation in the context of a PPU architecture. For example, aspects and implementations of the present disclosure may provide an instruction set that operates on the upper N bits of a software status register of a multiprocessor to perform memory allocation / reallocation. Additionally, an instruction set may be provided that operates on the lower M bits of a software status register of an SM. For example, the lower M bits of the software status register may be used for messaging or synchronization between thread groups (e.g., warps, cooperative thread arrays (CTAs), thread blocks, etc.) and / or to implement anti-starvation techniques associated with memory allocation, as described in detail below.

[0024] Advantages of the techniques disclosed herein include, but are not limited to, reducing latency and memory fragmentation associated with bitmap-based memory allocation. This can be achieved by providing a software status register for maintaining a bitmap and an instruction set that uses the bitmap maintained in the software status register to perform memory allocation / deallocation. In addition, the PPU architecture can utilize a portion of the bits in the software status register to implement various additional algorithms, such as message passing, synchronization, and the anti-starvation algorithm described herein.

[0025] Figure 1 An illustrative architecture for a computing device in accordance with at least one embodiment of the present disclosure is described. It should be noted that other architectures for the computing device 100 are possible, and implementations of computing devices utilizing embodiments of the present disclosure are not necessarily limited to the particular architecture depicted. The computing device 100 can be a single host machine or multiple host machines arranged in a heterogeneous or homogeneous group (e.g., a cluster or grid), and can include one or more rack-mounted servers, workstations, desktop computers, laptops, tablet computers, mobile phones, palm-sized computing devices, personal digital assistants (PDAs), and the like. In one example, the computing device 100 can be a server computer running on x86 hardware (e.g., ) is implemented as a computing device. In another example, the computing device 100 may be a or other hardware-implemented computing devices. Figure 1 In the example shown, computing device 100 may include memory 130 , a memory manager 140 including software state registers 120 , and one or more processes 110A-C.

[0026] Memory 130 may correspond to one or more data storage devices capable of storing data and may include volatile or non-volatile data storage. Volatile data storage (e.g., non-persistent storage) may store data for any duration but may lose that data after a power cycle or power outage. Non-volatile data storage (e.g., persistent storage) may store data for any duration and may retain that data after a power cycle or power outage. Memory 110 may include one or more hardware memory devices and may include volatile memory devices (e.g., random access memory (RAM)), non-volatile memory devices (e.g., flash memory, NVRAM), and / or other types of memory devices. In one example, memory 130 may be the primary storage of a computing device and may be referred to as the main memory of computing device 100. Memory 130 may be part of a cache hierarchy having multiple different physical storage devices organized in multiple levels that include a combination of one or more other storage devices, such as processor registers, cache devices, hard drives, solid-state drives (SSDs), other data storage devices, or a combination thereof. Memory 130 may be organized or managed as one or more blocks 132A-C.

[0027] Blocks 132A-C may be units of memory 130 that may be allocated for use by one or more processes 110A-C. Blocks 132A-C may include portions of virtual memory, logical memory, physical memory, other memory, or a combination thereof. Each block 132A-C may include a sequence of bytes or bits and may be a contiguous chunk (e.g., blocks 132A-C), a non-contiguous chunk, or a combination thereof. Each of blocks 132A-C may correspond to one or more memory pages, memory frames, memory segments, memory blocks, other portions of memory, or a combination thereof. Figure 1 In the example shown, the memory 110 may include three blocks 132A-C, and each block may be designated for use by one or more processes 110A-C. For example, block 132A may be assigned to process 110A, block 132B may be assigned to process 110B, and block 132C may be assigned to process 110C.

[0028] The processes 110A-C may be any computing process comprising program instructions executed by the computing device 100. The processes 110A-C may include user space processes (e.g., application processes), kernel processes (e.g., system processes), hypervisor processes, virtual machine processes, container processes, other processes, or combinations thereof. Each process 110A-C may include one or more threads or instruction streams that may request access to memory resources and may be able to increase or decrease the amount of memory assigned to it by submitting one or more allocation requests 112 and deallocation requests 114, respectively.

[0029] An allocation request 112 may be a message transmitted from a process requesting that additional memory resources be allocated to the requesting process. An allocation request 122 may or may not indicate the amount of memory, the usage of the memory (e.g., read-only, read / write, copy-on-write (CoW), compare-and-swap (CaS), find-and-set (FaS), etc.), the type of memory, the duration, other attributes, or a combination thereof. The memory manager 140 may respond to the allocation request 122 by allocating a memory block using the software status register 120 and providing an index to the allocated block. A deallocation request 122 may be a message transmitted from a process to the memory manager 140 requesting the release of memory resources. The deallocation request 122 may indicate the block to be released by providing an index of the software status register 120 associated with the block.

[0030] The process can initiate allocation requests 122 and deallocation requests 122 by executing shared code of the memory management component 130. Each process can access the shared code, and the shared code can be statically linked or dynamically linked to the process. The shared code can be derived from a shared library (e.g., a shared object (SO), a dynamic link library (DLL)), an executable file (EXE), other locations, or a combination thereof. The shared code can include one or more function calls, operands, instructions, opcodes, commands, or a combination thereof, and can be based on proprietary code, open source code, standardized code, or a combination thereof. In one example, the shared code can be derived from one or more standard libraries that are the same or similar to the C standard library (libc, stdlibc), the C++ standard library (libc++, stdlibc++, libcu++, etc.), the GNU C library (glibc), the Microsoft standard library (stdlib), other libraries, or a combination thereof. The standard library may include functions to allocate and deallocate memory, and allocation request 122 may be the result of a process executing a call to malloc, alloc, realloc, calloc, new, or other memory allocation function, and deallocation request 122 may be the result of a process executing a call to free, delete, release, or other memory deallocation function.

[0031] Memory manager 140 can manage memory 110 and provide memory allocation for processes 110A-C. Memory management component 140 can include features provided by code executing in user space, kernel space, or firmware, or features provided by hardware circuitry of a memory device or processor (e.g., a memory management unit (MMU), a memory controller). Memory allocation can involve processing allocation requests 112 and deallocation requests 114 to perform allocation and deallocation of memory. Memory manager 140 can provide access to memory to processes 110A-C by providing them with an index into software status register 120.

[0032] The software status register 120 indexes may include addresses / pointers to memory blocks that have been allocated by the memory management component 140. The indexes may be generated by the memory management component 140 during memory allocation and may be provided to the processes 110A-C in response to an allocation request 112. These indexes may also be provided by the processes to the memory manager 140 and subsequent deallocation requests 114. Each index may include one or more memory addresses that correspond to (e.g., are mapped to) the beginning, end, or middle of a block of memory 130. The one or more memory addresses may correspond to a virtual memory address (e.g., a virtual address), a logical memory address (e.g., a logical address), a physical memory address (e.g., a physical address), other addresses, or a combination thereof.

[0033] Figure 2 An example system 200 is shown that includes a host system 210 and a parallel processing unit (PPU) 220, in accordance with at least one embodiment of the present disclosure. Computing system 200 may be a computing device, such as a desktop computer, a laptop computer, a network server, a mobile device, a vehicle (e.g., an airplane, drone, train, car, or other transportation), an Internet of Things (IoT)-enabled device, an embedded computer (e.g., a computer included in a vehicle, industrial equipment, or networked business equipment), or such a computing device that includes a host system (e.g., a central processing unit) and a PPU 220 (such as a graphics processing unit (GPU)).

[0034] System 200 may include a host system 210 coupled to one or more PPUs 220 . Figure 1 An example of a host system 210 coupled to a PPU 220 is shown. As used herein, "coupled to" or "coupled with..." generally refers to a connection between components, which can be an indirect communication connection or a direct communication connection (e.g., without intervening components), including electrical, optical, magnetic, etc. connections, and can be wired or wireless.

[0035] The host system 210 may include a processor chipset and a software stack executed by the processor chipset. The processor chipset may include one or more cores, one or more caches, and a device driver 212. The host system 210 may be coupled to the PPU 220 via an interface. In at least one embodiment, the devices of the system 200 may be interconnected using a proprietary interconnect, a standardized interconnect (e.g., PCI or PCIe), or a combination thereof. In at least one embodiment, the devices of the system 200 may be interconnected using a chip-to-chip (C2C) or chip-to-package interconnect. Typically, the host system 210 may access multiple input / output (I / O) devices via the same interface, multiple independent interfaces, and / or a combination of interfaces.

[0036] The PPU 220 may be composed of multiple processing clusters (eg, GPU processing clusters (GPCs), texture processing clusters (TPCs), multiprocessors, memory controllers, and caches). Figure 1 In the illustrated example, the PPU 220 includes multiple TPCs 230 comprised of multiple multiprocessors 240A-N (generally referred to herein as "multiprocessors 240"). For example, each multiprocessor 240 may be a streaming multiprocessor (SM), a compute unit (CU), a multi-integrated core (MIC), or the like. Each multiprocessor 240 includes, but is not limited to, multiple processing cores 242, software state registers 244, and tensor memory 246. Processing cores 242 (e.g., Compute Unified Device Architecture (CUDA) cores) can execute separate threads, enabling parallel processing. In at least one embodiment, the multiprocessors 240 can operate according to a single instruction, multiple thread (SIMT) architecture. For example, all threads in a thread group (e.g., a warp, a CUDA thread block array (CTA), etc.) can execute the same instruction simultaneously, but may operate on different data. Tensor memory may refer to memory specifically used to store tensor data (tensors). Tensors are multidimensional arrays, which are fundamental data structures in deep learning applications. Tensors can represent scalars, vectors, matrices, or high-dimensional arrays. In at least one embodiment, software status register 244 may store a bitmap for maintaining the allocation of tensor memory 246. For example, software status register 244A may maintain a bitmap for the allocation of tensor memory 246A for multiprocessor 240A, while software status register 244N may maintain a bitmap for the allocation of tensor memory 246N for multiprocessor 240N.

[0037] In at least one embodiment, the host system 210 may include a device driver 212. The device driver 212 may implement an application programming interface (API) that defines various functions that can be utilized by applications to generate graphics data for display. A device driver is a software program that includes instructions that control the operation of the PPU 220. In at least one embodiment, the device driver 212 includes instructions to manage the allocation / deallocation of tensor memory 146 using corresponding software status registers 144.

[0038] In at least one embodiment, the device driver 212 may include operations for allocating tensor memory. For example, the device driver 212 may include a find and set (FaS) operation that can be executed by a process to initiate a tensor memory allocation request. It should be understood that the FaS operation is used herein by way of example and not by way of limitation, noting that the allocation techniques described herein can be performed in response to a call to malloc, alloc, realloc, calloc, new, or other memory allocation functions. FaS is designed to accelerate the allocation of tensor memory 146 using a bitmap maintained in the software state register 144. In at least one embodiment, the request may include the size of the requested allocation.

[0039] For example, a request may include an allocation size of "7," indicating a request to allocate seven memory blocks, where each memory block has a predetermined size (e.g., 8KB, 16KB, 32KB, etc.). The software status register 244 may be read to identify seven consecutive bits of a predetermined logical state indicating that the corresponding block of tensor memory is free, as described below with respect to Figure 3 shown.

[0040] Figure 3 An example of a tensor memory 310 and a software status register 320 for accelerating bitmap allocation according to at least one embodiment of the present disclosure is depicted. The tensor memory includes allocated memory blocks indicated by an allocated bit (e.g., '1') of the software status register 320, and unallocated memory blocks indicated by a free bit (e.g., '0') of the software status register 320.

[0041] A contiguous block of free bits (e.g., '0') can be determined by combining the software status register value 322 with a bit mask 324 to generate a combined value 326. The bit mask 324 can be a numerical value based on one or more binary values ​​(e.g., 00000000 1111 1111), a hexadecimal value (e.g., 0xFF), a decimal value (e.g., 255), an integer value, or a combination thereof. The bit mask 324 can be a value determined based on the size of the FaS operation. For example, the FaS operation associated with allocating tensor memory 310 can indicate a block size of 8. The bit mask 324 can use one or more contiguous binary values ​​(e.g., 0000 0000 1111 1111) to represent the block size.

[0042] Combining software status register value 322 with bit mask 324 may involve one or more mathematical operations to generate combined value 326. The mathematical value may include bitwise operations (e.g., or, xor, and, not, shifts), arithmetic operations (e.g., addition, subtraction, multiplication, division), other mathematical operations, or combinations thereof. Figure 3 In the example shown, software status register value 322 may be based on available tensor memory and represented as a binary value (e.g., 1111 1111 0000 0000), and bit mask 324 may be represented as a binary value (e.g., 0000 0000 1111 111). The bit mask may correspond to the size indicated in the FaS operation. Combining software status register value 322 with bit mask 324 may involve performing a bitwise AND operation that is configured to produce combined value 326 (e.g., 0000 0000 00000000). Combined value 326 may indicate the bit values ​​of bit mask 324 corresponding to the free blocks of tensor memory 310. In at least one embodiment, a shift operation may be used to manipulate bit mask 324 until a contiguous free memory block is determined or a contiguous free memory block of the indicated size is determined to be absent.

[0043] return Figure 2 In at least one embodiment, a FaS operation may include an input parameter for enforcing an allocation policy that aligns allocations according to their size. To ensure that allocations are aligned to their size, the allocation policy may cause the starting memory address of the allocated block to be a multiple of the allocation size. For example, if a process is calling a FaS operation that requests a 64KB allocation of tensor memory 246 with the alignment parameter enabled, the starting address of the allocated tensor memory 246 must be divisible by 64,000 (assuming the tensor memory 246 is byte-addressable).

[0044] In at least one embodiment, a FaS operation may include an input parameter for implementing uniform allocation of multiple tensor memories of a given multiprocessor. Such an input parameter may be referred to herein as a "cluster size," where the cluster size determines the number of multiprocessors 240 that uniformly perform tensor memory allocations. For example, TPC 230 may include multiprocessor 240A and multiprocessor 240N, each of which includes software status registers (software status registers 244A and 244N, respectively) and tensor memories (tensor memories 246A and 246N, respectively). A process may call a FaS operation with a cluster size of 2. Allocations may be uniformly performed on the same address of tensor memories 246A and 246N using software status registers 244A and 244N, respectively, such that the FaS operation returns a single address corresponding to both allocations.

[0045] In at least one embodiment, the device driver 212 may include an operation for deallocating tensor memory. For example, the device driver 212 may include a compare and swap (CaS) operation that can be executed by a process to initiate tensor memory deallocation. It should be understood that the CaS operation is used herein by way of example and not by way of limitation, and it should be noted that the allocation techniques described herein can be performed in response to executing a call to a free, delete, release, or other memory deallocation function. CaS is designed to accelerate the deallocation of tensor memory 146 using a bitmap maintained in the software status register 144. The CaS operation request 122 can indicate the memory block to be released by providing an index of the memory block to be released. The CaS operation can release / deallocate the memory block by causing the software status register 244 to be updated with the indicated index of the memory block within the software status register. For example, the corresponding bit in the software status register 244 can be set to "0" to indicate that the block is free.

[0046] In at least one embodiment of the present disclosure, a class of instructions supported by PPU 220 is referred to herein as "atomic" instructions. Each of these instructions is atomic in the sense that, as a single instruction, it can perform a series of steps to update the software status register without being interrupted by another memory access. The series of steps performed by an atomic instruction may include retrieving data from software status register 244, performing an operation (e.g., an addition operation) on the data, and storing the resulting value back to a memory location. The entire series of steps is completed in conjunction with a single atomic instruction. For example, a read-modify-write instruction (e.g., FaS, CaS, etc.) of PPU 220 can be an atomic instruction. Many threads or thread groups can have outstanding requests to perform read-modify-write operations on software status register 244. Although PPU 220 can receive atomic operations from many threads, PPU 220 (e.g., hardware within PPU 220) can ensure that read-modify-write instructions are serialized so that they can be executed atomically with respect to any other thread attempting to perform any other operation on software status register 244. Therefore, no spinning or locking is required.

[0047] In a multithreaded processing unit such as PPU 220, these atomic instructions can be used to prevent memory access conflicts between different threads. PPU 220 can support SIMD instructions issued across multiple processing cores 242 of multiprocessor 240, thereby allowing multiple threads to run concurrently (in flight). Some concurrently executing threads may cause instructions (e.g., CaS, FaS, etc.) to access the same software status register. This can cause memory conflict issues because one instruction may be in the middle of operating on data at the software status register 244, and before the instruction is completed, another instruction may change the data at the memory location. The use of atomic instructions can effectively prevent such potential memory conflicts.

[0048] For example, a first thread may be executing an atomic instruction while a second thread may begin another instruction (which may be a different atomic instruction, or a different instance of the same atomic instruction). The atomic instructions are designed so that while the first thread is executing its atomic instruction to update a bit of the software status register 244, the second thread is not allowed to overwrite the bit at that memory location.

[0049] In at least one embodiment, allocation operations (e.g., FaS) and deallocation operations (e.g., CaS) can be atomic operations. In at least one embodiment, PPU 220 can support one or more additional operations (which can also be atomic operations) to interact with software status register 244, including but not limited to load operations, store operations, bitwise operations (e.g., and, or, xor, not, shift, etc.), and arithmetic operations (e.g., addition, subtraction, multiplication, division, etc.).

[0050] In at least one embodiment, software status registers 244A-N may be used by one or more additional software algorithms for purposes other than memory allocation and deallocation. In such an embodiment, a portion of software status register 244 may be used for tensor memory allocation / deallocation operations, and another portion of software status register 244 may be used by other software algorithms. For example, the lower N bits of software status register 244 may be used to maintain a bitmap for allocating / deallocating tensor memory, while the upper N bits of software status register 244 may be used by one or more other software algorithms. In at least one embodiment, a portion of the bits of software status register 244 may be used for message passing and / or synchronization between thread groups (e.g., warps, thread blocks, CTAs, etc.) running on a corresponding multiprocessor 240 using one or more load, store, bitwise, and / or arithmetic operations that interact with software status register 244.

[0051] In at least one embodiment, a portion of the bits of the software status register 244 can be utilized to mitigate starvation issues associated with memory allocation and ensure fairness in memory allocation. In the context of memory allocation, starvation can refer to a situation where a process of a system is unable to obtain a memory resource despite memory allocation attempts being repeated multiple times. For example, a first process and a second process can attempt to allocate the same tensor memory 246A using the FaS operation described above. The first process can successfully allocate and immediately begin a subsequent allocation, thereby preventing the second process from obtaining service. In some cases, starvation issues can cause the system to run slower due to unfair memory allocation. In at least one embodiment, a portion of the bits of the software status register 244 can be used to enable the following regarding Figure 4 The memory allocation fairness technology described.

[0052] Figure 4A flowchart of a method 400 for ensuring memory allocation fairness according to at least one embodiment of the present disclosure is shown. Method 400 can be executed by processing logic including hardware, firmware, or any combination thereof. In at least one embodiment, method 400 can be executed by a processing device such as PPU 220 or computing device 100. In at least one embodiment, a non-transitory computer-readable storage medium can store instructions that, in response to being executed by a processing device, cause the processing device to perform method 400.

[0053] More illustrative information will now be provided regarding various optional architectures and features that may be used to implement the above-described framework, depending on the user's desires. It should be noted that the following information is provided for illustrative purposes and should not be construed as limiting in any way. Any of the following features may be optionally combined with or without excluding the other features described.

[0054] Although shown in a particular sequence or order, unless otherwise indicated, the order of operations may be modified. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated operations may be performed in a different order, and some operations may be performed in parallel. In addition, one or more operations may be omitted in various embodiments. Therefore, not all operations are required in every embodiment.

[0055] At operation 402 of method 400, a process may send an allocation request to processing logic. For example, the process may send the allocation request using the FaS operation described above. In at least one embodiment, the allocation request may include the size of the allocation (e.g., the number of memory blocks to be allocated) and an allocation token identifier (ID). For example, a process may set its allocation token ID to "clear," indicating that the next allocation token ID has not yet been set / assigned by processing logic.

[0056] At operation 404 of method 400, processing logic may receive an allocation request that includes the size of the requested allocation and an allocation token ID. If the allocation token ID is not set (e.g., the allocation token ID is "clear"), the processing logic may set the allocation token ID to a value equal to the next allocation token ID. In at least one embodiment, the value of the next allocation token ID in a software status register may be tracked, such as Figure 2 244. In response to updating the allocation token ID of the process to the next allocation ID, the process may update (eg, increment) the value of the next allocation ID in the software status register.

[0057] At operation 406 of method 400, processing logic may compare the allocation token ID of the requesting process with the current allocation token ID. In at least one embodiment, the value of the current allocation token ID may be stored in a software status register (e.g., Figure 2 In response to determining that the allocation token ID is equal to the current allocation token ID, the method continues with operation 408. In response to determining that the allocation token ID is not equal to the current allocation token ID, the method continues with operation 410.

[0058] At operation 408 of method 400, processing logic may attempt to service the memory allocation request. Processing logic may perform the allocation according to one or more of the techniques described above. In at least one embodiment, if the allocation fails (e.g., due to insufficient free memory blocks), method 400 continues with operation 412. If the allocation succeeds, method 400 continues with operation 414.

[0059] At operation 412 of method 400, processing logic may indicate that the allocation token ID has been set and enable anti-starvation mode. By indicating that the allocation token ID has been set, when an allocation is retried, the allocation token ID will not be reset at operation 404. By enabling anti-starvation mode, processing logic (at operation 410) may prevent allocation requests from attempting to allocate memory out of sequence (e.g., when the allocation token ID is not equal to the current allocation token ID).

[0060] At operation 414 of memory method 400, processing logic may enable normal operating mode (e.g., disable anti-starvation mode). Thus, in some embodiments, anti-starvation mode may be enabled when an allocation fails. In other embodiments, operation 414 may maintain anti-starvation mode so that anti-starvation mode is enabled throughout the entire process of multiple memory allocations. Processing logic may update (e.g., increment) the value of the current allocation ID in the software status register so that subsequent allocation requests may be serviced. Processing logic may return an offset of the memory allocation to the requesting process to enable access to the allocation of the corresponding memory.

[0061] At operation 410 of method 400, in response to determining that anti-starvation mode is enabled, method 400 may proceed to operation 404. In response to determining that anti-starvation mode is not enabled (eg, mode is set to "normal"), method 400 may proceed to operation 408.

[0062] Figure 5A flowchart of a method 500 for memory management using registers according to at least one embodiment of the present disclosure is shown. Although method 500 is described in the context of a processing unit, method 500 may also be performed by a program, custom circuitry, or a combination of custom circuitry and a program. For example, method 400 may be performed by a parallel processing unit (PPU), a CPU (central processing unit), or any processing device capable of memory management. Furthermore, one of ordinary skill in the art will appreciate that any system that performs method 500 is within the scope and spirit of the embodiments of the present invention.

[0063] More illustrative information about various optional architectures and features that can be used to implement the above-described framework will now be described, depending on the user's expectations. It should be noted that the following information is set forth for illustrative purposes and should not be construed as limiting in any way. Any of the following features may be selectively combined with or without excluding other features described. Although shown in a particular sequence or order, the order of operations may be modified unless otherwise stated. Therefore, the embodiments shown should be understood as examples only, and the operations shown may be performed in a different order, and some operations may be performed in parallel. In addition, one or more operations may be omitted in various embodiments. Therefore, not all operations are required in every embodiment.

[0064] At operation 502 of method 500 , processing logic may receive a first request to allocate one or more memory blocks of a first plurality of memory blocks associated with a first memory.

[0065] At operation 504 of method 500, processing logic may identify a consecutive set of first portions of bits of a first register having a first logical state. The first logical state (e.g., a logical '0') indicates that a corresponding memory block in one or more memory blocks is free. In at least one embodiment, the consecutive set of first portions of bits of the first register is aligned with the size of the corresponding memory block. Wherein the processing logic includes a graphics processing unit (GPU), the first register is associated with a first multiprocessor within the GPU. For example, the processing logic may be included in PPU 220, and the first register may include software status register 244A. In at least one embodiment, the first memory may be a tensor memory, such as tensor memory 246A.

[0066] At operation 506 of method 500 , processing logic may perform a first operation to adjust a contiguous set of first bit portions of the first register to a second logic state (eg, logic '1'). The second logic state may indicate that the corresponding memory block has been allocated.

[0067] At operation 508 of method 500, processing logic may send an allocation address to the first request, the allocation address including an index of a contiguous set of first bit portions of the first register. The allocation address may be used to access a corresponding memory block.

[0068] In at least one embodiment, processing logic may receive a request to deallocate one or more memory blocks.Processing logic may further identify a contiguous set of first bit portions and perform a second operation to adjust the contiguous set of first bit portions to a first logic state.

[0069] In at least one embodiment, processing logic receives an instruction along with a first request to allocate one or more memory blocks of a second plurality of memory blocks associated with a second memory together with one or more memory blocks of the first plurality of memory blocks. The processing logic may use the contiguous set of first bit portions of the first register to identify the contiguous set of first bit portions of the second register. The processing logic may perform a second operation to modify the contiguous set of first bit portions of the second register to a second logical state. The contiguous set of first bit portions of the second register is indexed the same as the contiguous set of first bit portions of the first register.

[0070] In at least one embodiment, the first request may include a first instruction issued by a first thread executing on the PPU. In at least one embodiment, the processing logic may identify one or more instructions issued by one or more additional threads executing concurrently with the first thread. The processing logic may serialize the one or more operations of the one or more additional threads to prevent the one or more operations from changing a contiguous set of first bit portions of the first register until the first operation is completed.

[0071] In at least one embodiment, the processing logic may use the second portions of bits of the first register for at least one of message passing, synchronization, and memory allocation fairness. In an illustrative example, the upper N bits of the first register may be used by the processing logic for memory allocation techniques, such as those described above with respect to Figure 1-Figure 3 The lower N bits of the software status register may be used by processing logic to implement one or more memory allocation fairness techniques, such as those described above with respect to Figure 4 Describes memory allocation fairness.

[0072] Parallel processing architecture

[0073] Figure 6A parallel processing unit (PPU) 600 is shown according to one embodiment. In one embodiment, the PPU 600 is a multi-threaded processor implemented on one or more integrated circuit devices. The PPU 600 is a latency-hiding architecture designed for processing multiple threads in parallel. A thread (e.g., an execution thread) is an instance of a group of instructions configured to be executed by the PPU 600. In one embodiment, the PPU 600 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device (such as a liquid crystal display (LCD) device). In other embodiments, the PPU 600 can be used to perform general-purpose computations. Although an exemplary parallel processor is provided herein for illustrative purposes, it should be strongly noted that such a processor is for illustrative purposes only and that any processor may be used in addition to and / or in place of the same processor.

[0074] One or more PPUs 600 can be configured to accelerate thousands of high-performance computing (HPC), data center, and machine learning applications. PPUs 600 can be used to accelerate numerous deep learning systems and applications, including autonomous vehicle platforms, deep learning, high-precision speech, image, and text recognition systems, intelligent video analysis, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations.

[0075] like Figure 6 As shown, the PPU 600 includes an input / output (I / O) unit 605, a front-end unit 615, a scheduler unit 620, a work distribution unit 625, a hub 630, a crossbar switch (Xbar) 670, one or more processing clusters 650 (e.g., general processing clusters (GPCs)), and one or more memory partitioning units 680. The PPU 600 can be connected to a host processor or other PPUs 600 via one or more high-speed NVLink 610 interconnects. The PPU 600 can be connected to a host processor or other peripheral devices via an interconnect 602. The PPU 600 can also be connected to a local storage device 604 that includes multiple memories. In one embodiment, the local memory can include multiple dynamic random access memory (DRAM) devices. The DRAM devices can be configured as a high-bandwidth memory (HBM) subsystem, with multiple DRAM dies stacked in each device.

[0076] The NVLink 610 interconnect enables the system to scale and include one or more PPUs 600 in conjunction with one or more CPUs, supporting cache coherency between the PPU 600 and the CPU, and CPU mastering. Data and / or commands can be transmitted by the NVLink 610 through the hub 630 to other units of the PPU 600, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). Figure 8B NVLink 610 is described in more detail.

[0077] I / O unit 605 is configured to send and receive communications (e.g., commands, data, etc.) from a host processor (not shown) via interconnect 602. I / O unit 605 can communicate with the host processor directly via interconnect 602, or through one or more intermediary devices, such as a memory bridge. In one embodiment, I / O unit 605 can communicate with one or more other processors (such as one or more PPUs 600) via interconnect 602. In one embodiment, I / O unit 605 implements a Peripheral Component Interconnect Express (PCIe) interface for communicating over a PCIe bus, and interconnect 602 is a PCIe bus. In alternative embodiments, I / O unit 605 can implement other types of known interfaces for communicating with external devices.

[0078] I / O unit 605 decodes data packets received via interconnect 602. In one embodiment, the data packets represent commands configured to cause PPU 600 to perform various operations. I / O unit 605 sends the decoded commands to various other units of PPU 600, as these commands may specify. For example, some commands may be transmitted to front-end unit 615. Other commands may be transmitted to hub 630 or other units of PPU 600, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In other words, I / O unit 605 is configured to route communications within and between the various logical units of PPU 600.

[0079] In one embodiment, a program executed by a host processor encodes a command stream in a buffer that provides tasks to the PPU 600 for processing. A task may include several instructions and the data processed by these instructions. A buffer is an area of ​​memory that is accessible (e.g., read / write) to both the host processor and the PPU 600. For example, the I / O unit 605 may be configured to access a buffer in system memory connected to the interconnect 602 via a memory request transmitted over the interconnect 602. In one embodiment, the host processor writes a command stream into the buffer and then transmits a pointer to the starting point of the command stream to the PPU 600. The front end unit 615 receives pointers to one or more command streams. The front end unit 615 manages the one or more streams, reads commands from the streams, and forwards the commands to the various units of the PPU 600.

[0080] The front-end unit 615 is coupled to a scheduler unit 620, which configures various processing clusters 650 to process tasks defined by one or more flows. The scheduler unit 620 is configured to track status information related to the various tasks managed by the scheduler unit 620. The status may indicate which processing cluster 650 the task is assigned to, whether the task is active or inactive, the priority associated with the task, etc. The scheduler unit 620 manages the execution of multiple tasks on one or more processing clusters 650.

[0081] Scheduler unit 620 is coupled to work allocation unit 625, which is configured to schedule tasks for execution on processing cluster 650. Work allocation unit 625 can track multiple scheduled tasks received from scheduler unit 620. In one embodiment, work allocation unit 625 manages a pending task pool and an active task pool for each processing cluster 650. The pending task pool can include multiple time slots (e.g., 32 time slots) containing tasks assigned to be processed by a particular processing cluster 650. The active task pool can include multiple time slots (e.g., 4 time slots) for tasks actively being processed by processing cluster 650. When a processing cluster 650 completes execution of a task, the task is removed from the active task pool of processing cluster 650 and another task is selected from the pending task pool and scheduled for execution on processing cluster 650. If an active task is idle on processing cluster 650 , for example, while waiting for data dependencies to be resolved, the active task is ejected from processing cluster 650 and returned to the pending task pool while another task in the pending task pool is selected and scheduled for execution on processing cluster 650 .

[0082] Work distribution unit 625 communicates with one or more processing clusters 650 via XBar 670. XBar 670 is an interconnect network that couples many units of PPU 600 to other units of PPU 600. For example, XBar 670 can be configured to couple work distribution unit 625 to a specific processing cluster 650. Although not explicitly shown, one or more other units of PPU 600 can also be connected to XBar 670 via hub 630.

[0083] Tasks are managed by the scheduler unit 620 and dispatched to the processing cluster 650 by the work distribution unit 625. The processing cluster 650 is configured to process the task and generate a result. The result can be used by other tasks in the processing cluster 650, or can be routed to a different processing cluster 650 via the XBar 670, or stored in the memory 604. The result can be written to the memory 604 via the partition unit 680, which implements a memory interface for reading and writing data to the memory 604. The result can be transferred to another PPU 600 or CPU via the NVLink 610. In one embodiment, the PPU 600 includes a number U of partition units 680 that is equal to the number of independent and different storage devices 604 coupled to the PPU 600. Figure 7B The partition unit 680 is described in more detail.

[0084] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API) that enables one or more applications executing on the host processor to schedule operations for execution on the PPU 600. In one embodiment, multiple computing applications are executed simultaneously by the PPU 600, which provides isolation, quality of service (QoS), and independent address spaces for the multiple computing applications. The application can generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks for execution by the PPU 600. The driver kernel outputs the tasks to one or more streams being processed by the PPU 600. Each task can include one or more groups of related threads, referred to herein as warps. In one embodiment, a warp includes 32 related threads that can be executed in parallel. Collaborating threads can refer to multiple threads that include instructions to perform tasks and can exchange data through shared memory. In combination Figure 8A Describes threads and cooperative threads in more detail.

[0085] Figure 7A According to one embodiment, Figure 6 The processing cluster 650 of the PPU 600 is Figure 7AAs shown, each processing cluster 650 includes several hardware units for processing tasks. In one embodiment, each processing cluster 650 includes a pipeline manager 710, a pre-raster operation unit (PROP) 715, a raster engine 725, a work distribution crossbar (WDX) 780, a memory management unit (MMU) 790 and one or more data processing clusters (DPCs) 720. It should be understood that Figure 7A The processing cluster 650 may include instead Figure 7A Other hardware units or in addition to the units shown Figure 7A Other hardware units for the units shown.

[0086] In one embodiment, the operation of processing cluster 650 is controlled by pipeline manager 710. Pipeline manager 710 manages the configuration of one or more DPCs 720 for processing tasks assigned to processing cluster 650. In one embodiment, pipeline manager 710 may configure at least one of one or more DPCs 720 to implement at least a portion of a graphics rendering pipeline. For example, DPC 720 may be configured to execute vertex shading programs on programmable multiprocessor 740. Multiprocessor 740 may typically include a streaming multiprocessor, a compute unit, a number of integrated cores, etc. Pipeline manager 710 may also be configured to route packets received from work distribution unit 625 to appropriate logic within processing cluster 650. For example, some packets may be routed to fixed-function hardware units in PROP 715 and / or raster engine 725, while other packets may be routed to DPC 720 for processing by primitive engine 735 or multiprocessor 740. In one embodiment, pipeline manager 710 may configure at least one of one or more DPCs 720 to implement a neural network model and / or a compute pipeline.

[0087] PROP unit 715 is configured to route data generated by raster engine 725 and DPC 720 to the raster operations (ROP) unit, in conjunction with Figure 7B PROP unit 715 may also be configured to perform optimizations for color blending, organize pixel data, perform address translation, and the like.

[0088] The raster engine 725 includes several fixed-function hardware units configured to perform various raster operations. In one embodiment, the raster engine 725 includes a setup engine, a coarse raster engine, a culling engine, a clipping engine, a fine raster engine, and a tile aggregation engine. The setup engine receives the transformed vertices and generates plane equations associated with the geometric primitives defined by the vertices. The plane equations are passed to the coarse raster engine to generate coverage information for the primitives (e.g., the x, y coverage mask of the tile). The output of the coarse raster engine is passed to the culling engine, where fragments associated with primitives that fail the z test are culled, and the unculled fragments are passed to the clipping engine, where fragments outside the viewing frustum are clipped. Fragments that survive clipping and culling can be passed to the fine raster engine to generate attributes for the pixel fragments based on the plane equations generated by the setup engine. The output of the raster engine 725 includes fragments to be processed, for example, fragments processed by the fragment shader implemented in the DPC 720.

[0089] Each DPC 720 included in the processing cluster 650 includes an M-pipeline controller (MPC) 730, a primitive engine 735, and one or more multiprocessors 740. The MPC 730 controls the operation of the DPC 720 and routes data packets received from the pipeline manager 710 to the appropriate units in the DPC 720. For example, data packets associated with vertices may be routed to the primitive engine 735, which is configured to retrieve vertex attributes associated with the vertices from the memory 604. Conversely, data packets associated with shading programs may be transmitted to the multiprocessor 740.

[0090] In some embodiments, the multiprocessor 740 includes a programmable multiprocessor, such as a programmable streaming multiprocessor, which is configured to process tasks represented by multiple threads. Each multiprocessor 740 is multithreaded and is configured to concurrently execute multiple threads (e.g., 32 threads) from a particular thread group. In one embodiment, the multiprocessor 740 implements a SIMD (Single Instruction, Multiple Data) architecture, in which each thread in a thread group (e.g., a warp) is configured to process a different data set based on the same instruction set. All threads in a thread group execute the same instruction. In another embodiment, the multiprocessor 740 implements a SIMT (Single-Instruction, Multiple Thread) architecture, in which each thread in a thread group is configured to process a different data set based on the same instruction set, but individual threads in a thread group are allowed to diverge during execution. In one embodiment, a program counter, call stack, and execution state are maintained for each warp, enabling concurrency between warps and serial execution within a warp when threads within the warp diverge. In another embodiment, the program counter, call stack, and execution state are maintained for each individual thread, thereby enabling equal concurrency between all threads, within warps, and between warps. When the execution state of each individual thread is maintained, threads executing the same instruction can be aggregated and executed in parallel for maximum efficiency. Figure 8A Multiprocessor 740 is described in more detail.

[0091] The MMU 790 provides an interface between the processing cluster 650 and the partition unit 680. The MMU 790 can provide virtual address to physical address translation, memory protection, and memory request arbitration. In one embodiment, the MMU 790 provides one or more translation lookaside buffers (TLBs) for performing translation of virtual addresses into physical addresses in the memory 604.

[0092] Figure 7B According to one embodiment, Figure 6 The memory partition unit 680 of the PPU 600. Figure 7BAs shown, the memory partition unit 680 includes a raster operations (ROP) unit 750, a level 2 (L2) cache 760, and a memory interface 770. The memory interface 770 is coupled to the memory device 604. The memory interface 770 can implement a 32-bit, 64-bit, 128-bit, 1024-bit data bus, etc. for high-speed data transfer. In one embodiment, the PPU 600 includes a U memory interface 770, one for each pair of partition units 680, where each pair of partition units 680 is connected to a corresponding memory device 604. For example, the PPU 600 can be connected to up to Y memory devices 604, such as a high-bandwidth memory stack or graphics double data rate version 5 synchronous dynamic random access memory or other types of persistent memory.

[0093] In one embodiment, memory interface 770 implements an HBM2 memory interface, with Y equal to half of U. In one embodiment, the HBM2 memory stack resides on the same physical package as the PPU 600, providing high power and area savings compared to traditional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory dies, with Y equal to four, where the HBM2 stack includes two 128-bit channels per chip, for a total of eight channels and a data bus width of 1024 bits.

[0094] In one embodiment, memory 604 supports single-error correction and double-error detection (SECDED) error correction code (ECC) to protect data. ECC provides increased reliability for computing applications sensitive to data corruption. Reliability is particularly important in large-scale cluster computing environments where PPU 600 processes very large data sets and / or long-running applications.

[0095] In one embodiment, the PPU 600 implements a multi-level memory hierarchy. In one embodiment, the memory partitioning unit 680 supports unified memory to provide a single, unified virtual address space for the CPU and PPU 600 memory, enabling data sharing between virtual memory systems. In one embodiment, the frequency of PPU 600 accesses to memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 600 that accesses the page more frequently. In one embodiment, NVLink 610 supports address translation services, allowing the PPU 600 to directly access the CPU's page tables and providing the PPU 600 with full access to the CPU's memory.

[0096] In one embodiment, the copy engine transfers data between multiple PPUs 600 or between a PPU 600 and a CPU. The copy engine can generate a page fault for an address that is not mapped to a page table. The memory partition unit 680 can then handle the page fault, map the address into a page table, and then the copy engine can perform the transfer. In conventional systems, fixed memory (e.g., non-pageable) is operated for multiple copy engines between multiple processors, which greatly reduces the available memory. For hardware page faults, the address can be passed to the copy engine without having to worry about whether the memory page is resident, and the copy process is transparent.

[0097] Data from memory 604 or other system memory can be retrieved by memory partition unit 680 and stored in an L2 cache 760 located on-chip and shared between the various processing clusters 650. As shown, each memory partition unit 680 includes a portion of the L2 cache 760 associated with the corresponding memory device 604. Lower-level caches can then be implemented in various units within the processing cluster 650. For example, each multiprocessor 740 can implement a level 1 (L1) cache. The L1 cache is a dedicated memory dedicated to a particular multiprocessor 740. Data can be retrieved from the L2 cache 760 and stored in each L1 cache for processing within the functional units of the multiprocessor 740. The L2 cache 760 is coupled to the memory interface 770 and the XBar 670.

[0098] The ROP unit 750 performs graphics raster operations related to pixel color, such as color compression, pixel blending, etc. The ROP unit 750 also implements depth testing in conjunction with the raster engine 725, receiving the depth of the sample position associated with the pixel fragment from the culling engine of the raster engine 725. The depth is tested against the corresponding depth in the depth buffer of the sample position associated with the fragment. If the fragment passes the depth test for the sample position, the ROP unit 750 updates the depth buffer and transmits the result of the depth test to the raster engine 725. It should be understood that the number of partition units 680 can be different from the number of processing clusters 650, and therefore, each ROP unit 750 can be coupled to each processing cluster 650. The ROP unit 750 tracks the packets received from the different processing clusters 650 and determines to which processing cluster 650 the results generated by the ROP unit 750 are routed via the Xbar 670. Although in Figure 7B The ROP unit 750 is included within the memory partition unit 680, but in other embodiments, the ROP unit 750 may be located external to the memory partition unit 680. For example, the ROP unit 750 may reside in the processing cluster 650 or other units.

[0099] Figure 8AAccording to one embodiment, Figure 7A The multiprocessor 740. Figure 8A As shown, the multiprocessor 740 includes an instruction cache 805, one or more scheduler units 810, a register file 820, one or more processing cores 850, one or more special function units (SFUs) 852, one or more load / store units (LSUs) 854, an interconnection network 880, and a shared memory / L1 cache 870.

[0100] As described above, the work distribution unit 625 dispatches tasks to be executed on the processing clusters 650 of the PPU 600. Tasks are assigned to specific DPCs 720 within the processing cluster 650 and, if the task is associated with a shader program, may be assigned to the multiprocessor 740. The scheduler unit 810 receives tasks from the work distribution unit 625 and manages the scheduling of instructions for one or more thread blocks assigned to the multiprocessor 740. The scheduler unit 810 schedules the thread blocks for execution as warps of parallel threads, with each thread block assigned at least one warp. In one embodiment, each warp executes 32 threads. The scheduler unit 810 can manage multiple different thread blocks, assign warps to different thread blocks, and then dispatch instructions from multiple different cooperating groups to various functional units (e.g., cores 850, SFUs 852, and LSUs 854) during each clock cycle.

[0101] Cooperative Groups is a programming model for organizing groups of communicating threads, allowing developers to express the granularity of thread communication, enabling richer and more efficient decompositions of parallelism. The cooperative launch API supports synchronization between thread blocks to execute parallel algorithms. Traditional programming models provide a single, simple construct for synchronizing cooperating threads: a barrier that spans all threads of a thread block (e.g., the syncthreads() function). However, programmers often want to define thread groups at a granularity smaller than a thread block and synchronize within the defined groups, enabling higher performance, design flexibility, and software reuse in the form of a collective group-wide function interface.

[0102] Cooperative Groups enables programmers to explicitly define thread groups at sub-block (e.g., down to a single thread) and multi-block granularity, and perform collective operations such as synchronization on threads in a cooperative group. The programming model supports clean composition across software boundaries so that libraries and utility functions can safely synchronize in their local context without assuming convergence. The Cooperative Groups primitive supports new patterns of cooperative parallelism, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire grid of thread blocks.

[0103] The dispatch unit 815 is configured to transmit instructions to one or more functional units. In this embodiment, the scheduler unit 810 includes two dispatch units 815, which enable scheduling two different instructions from the same thread warp in each clock cycle. In alternative embodiments, each scheduling unit 810 may include a single dispatch unit 815 or additional dispatch units 815.

[0104] Each multiprocessor 740 includes a register file 820 that provides a set of registers for the functional units of the multiprocessor 740. In one embodiment, the register file 820 is divided between each functional unit so that each functional unit is allocated a dedicated portion of the register file 820. In another embodiment, the register file 820 is divided between the different warps executed by the multiprocessor 740. The register file 820 provides temporary storage space for operands connected to the data paths of the functional units.

[0105] Each multiprocessor 740 includes L processing cores 850. In one embodiment, the multiprocessor 740 includes a large number (e.g., 128, etc.) of different processing cores 850. Each processing core 850 may include a fully pipelined, single-precision, double-precision, and / or mixed-precision processing unit, including a floating-point arithmetic logic unit (ALU) and an integer ALU. In one embodiment, the floating-point ALU implements the IEEE 754-2008 standard for floating-point arithmetic. In one embodiment, the processing cores 850 include 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0106] Tensor Cores are configured to perform matrix operations, and in one embodiment, one or more Tensor Cores are included in Core 850. Specifically, Tensor Cores are configured to perform deep learning matrix algorithms, such as convolution operations for neural network training and inference. In one embodiment, each Tensor Core operates on a 4x4 matrix and performs a matrix multiplication and accumulation operation A×B+C, where A, B, C, and D are 4×4 matrices.

[0107] In one embodiment, the matrix multiplication inputs A and B are 16-bit floating point matrices, while the accumulation matrices C and D can be either 16-bit floating point matrices or 32-bit floating point matrices. The Tensor Cores operate on the 16-bit floating point input data with 32-bit floating point accumulations. The 16-bit floating point multiplication requires 64 operations to obtain a fully precise product, which is then followed by a 4x4x4 matrix multiplication using 32-bit floating point additions and other intermediate products. In practice, Tensor Cores are used to perform larger two-dimensional or high-dimensional matrix operations that are built up from these smaller elements. APIs, such as the CUDA 9 C++ API, expose specialized matrix load, matrix multiplication and accumulation, and matrix store operations to efficiently use Tensor Cores in CUDA-C++ programs. At the CUDA level, the warp-level interface assumes that the 16×16 size matrix spans all 32 threads of the warp.

[0108] Each multiprocessor 640 also includes M SFUs 852 that perform special functions (e.g., attribute evaluation, reciprocal square root, etc.). In one embodiment, the SFUs 852 may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, the SFUs 852 may include a texture unit configured to perform texture map filtering operations. In one embodiment, the texture unit is configured to load a texture map (e.g., a two-dimensional texture array) from memory 604 and sample the texture map to generate sampled texture values ​​for use in shading programs executed by the multiprocessor 740. In one embodiment, the texture map is stored in shared memory / L1 cache 770. The texture unit implements texture operations, such as filtering operations using mip maps (e.g., texture maps of different levels of detail). In one embodiment, each multiprocessor 740 includes two texture units.

[0109] Each multiprocessor 740 also includes N LSUs 854 that implement load and store operations between the shared memory / L1 cache 870 and the register file 820. Each multiprocessor 740 includes an interconnection network 880 that connects each functional unit to the register file 820 and connects the LSUs 854 to the register file 820 and the shared memory / L1 cache 870. In one embodiment, the interconnection network 880 is a crossbar switch that is configurable to connect any functional unit to any register in the register file 820 and to connect the LSUs 854 to memory locations in the shared memory / L1 cache 870 and the register file.

[0110] Shared memory / L1 cache 870 is an on-chip memory array that allows data storage and communication between the multiprocessor 740 and the primitive engine 735, as well as between threads in the multiprocessor 740. In one embodiment, shared memory / L1 cache 870 includes 128KB of storage capacity and is located in the path from the multiprocessor 740 to the partition unit 680. Shared memory / L1 cache 870 can be used to cache read and write operations. One or more of the shared memory / L1 cache 870, L2 cache 760, and memory 604 are backing stores.

[0111] Combining data cache and shared memory functionality into a single memory block provides the best overall performance for both types of memory access. This capacity is available for use as a cache by programs that do not use the shared memory. For example, if the shared memory is configured to use half of its capacity, texture and load / store operations can use the remaining capacity. This integration in shared memory / L1 cache 870 enables shared memory / L1 cache 870 to act as a high-throughput pipeline for streaming data, while providing high-bandwidth and low-latency access to frequently reused data.

[0112] When configured for general parallel computing, a simpler configuration can be used compared to graphics processing. Specifically, bypassing Figure 6 The fixed-function graphics processing unit (GPU) shown creates a simpler programming model. In the general-purpose parallel computing configuration, the work distribution unit 625 assigns thread blocks directly to the DPC 720. The threads in the block execute the same program, using unique thread IDs in computations to ensure each thread produces unique results. The multiprocessor 740 executes the program and performs computations, using the shared memory / L1 cache 870 for communication between threads, and using the LSU 854 to read and write to global memory via the shared memory / L1 cache 870 and the memory partition unit 680. When configured for general-purpose parallel computing, the multiprocessor 740 can also write commands that the scheduler unit 620 can use to start new work on the DPC 720.

[0113] The PPU 600 may be included in a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., a wireless handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, etc. In one embodiment, the PPU 600 is included on a single semiconductor substrate. In another embodiment, the PPU 600 is included in a system-on-chip (SoC) along with one or more other devices, such as an additional PPU 600, a memory 604, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), etc.

[0114] In one embodiment, the PPU 600 may be included on a graphics card that includes one or more storage devices 604. The graphics card may be configured to interface with a PCIe slot on a desktop computer motherboard. In another embodiment, the PPU 600 may be an integrated graphics processing unit (iGPU) or a parallel processor included in a motherboard chipset.

[0115] Exemplary Computing System

[0116] As developers expose and exploit greater parallelism in applications such as artificial intelligence computing, systems with multiple GPUs and CPUs are being used across a wide range of industries. High-performance GPU-accelerated systems with tens to thousands of compute nodes are being deployed in data centers, research institutions, and supercomputers to solve ever-larger problems. As the number of processing devices in high-performance systems increases, communication and data transfer mechanisms need to scale to support the increased bandwidth.

[0117] Figure 8B According to one embodiment, the Figure 6 Conceptual diagram of a processing system 800 implemented with a PPU 600. The exemplary system 865 may be configured to implement Figure 5 The method 500 shown. The processing system 800 includes a CPU 830, a switch 812, and multiple PPUs 600 and respective memories 604. NVLink 610 provides a high-speed communication link between the various PPUs 600. Figure 8B A specific number of NVLink 610 and interconnect 602 connections are shown, but the number of connections for each PPU 600 and CPU 830 may vary. Switch 812 is the interface between interconnect 602 and CPU 830. PPU 600, memory 604, and NVLinks 610 may be located on a single semiconductor platform to form a parallel processing module 825. In one embodiment, switch 812 supports two or more protocols for interfacing between various different connections and / or links.

[0118] In another embodiment (not shown), NVLink 610 provides one or more high-speed communication links between each PPU 600 and CPU 830, and a switch 812 interface is provided between the interconnect 602 and each PPU 600. The PPUs 600, memory 604, and interconnect 602 may be located on a single semiconductor platform to form a parallel processing module 825. In another embodiment (not shown), the interconnect 602 provides one or more communication links between each PPU 600 and CPU 830, and a switch 812 interface is provided using NVLink 610 to provide one or more high-speed communication links between each PPU 600. In another embodiment (not shown), NVLink 610 provides one or more high-speed communication links between the PPUs 600 and CPU 830 via the switch 812. In another embodiment (not shown), the interconnect 602 provides one or more communication links directly between each PPU 600. One or more high-speed communication links of NVLink 610 may be implemented as a physical NVLink interconnect, or an on-chip or on-die interconnect that uses the same protocol as NVLink 610 .

[0119] In the context of this specification, a single semiconductor platform may refer to a unique, single semiconductor-based integrated circuit fabricated on a wafer or chip. It should be noted that the term single semiconductor platform may also refer to a multi-chip module with enhanced connectivity that emulates on-chip operations and provides substantial improvements over conventional bus implementations. Of course, various circuits or devices may also be placed individually or in a combination of various semiconductor platforms, depending on the needs of the user. Alternatively, the parallel processing module 825 may be implemented as a circuit board substrate, and each PPU 600 and / or memory 604 may be a packaged device. In one embodiment, the CPU 830, switch 812, and parallel processing module 825 are located on a single semiconductor platform.

[0120] In one embodiment, the signaling rate of each NVLink 610 is 20 to 26 Gbit / s, and each PPU 600 includes six NVLink 610 interfaces (e.g., Figure 8B As shown, each PPU 600 includes five NVLink 610 interfaces. Each NVLink 610 provides a data transfer rate of 25 Gbit / s in each direction, and six links provide a data transfer rate of 600 Gbit / s. When the CPU 830 includes one or more NVLink 610 interfaces, the NVLink 610 can be used alone, such as Figure 8B Communication shown is between PPUs, or some combination of PPU to PPU and PPU to CPU.

[0121] In one embodiment, NVLink 610 allows direct load / store / atomic access from the CPU 830 to each PPU's 600 memory 604. In one embodiment, NVLink 610 supports coherency operations, allowing data read from memory 604 to be stored in the CPU 830's cache hierarchy, thereby reducing cache access latency for the CPU 830. In one embodiment, NVLink 610 includes support for Address Translation Services (ATS), allowing the PPU 600 to directly access page tables within the CPU 830. One or more NVLinks 610 can also be configured for a low-power operating mode.

[0122] Figure 8C An exemplary system 865 is shown in which various architectures and / or functionalities of various previous embodiments may be implemented. The exemplary system 865 may be configured to implement Figure 4 The method 400 shown in FIG.

[0123] As shown, a system 865 is provided that includes at least one central processing unit 830 connected to a communication bus 875. The communication bus 875 may be implemented using any suitable protocol, such as Peripheral Component Interconnect (PCI), PCI-express, Accelerated Graphics Port (AGP), HyperTransport, or any other bus or point-to-point communication protocol. The system 865 also includes a main memory 840. Control logic (software) and data are stored in the main memory 840, which may take the form of random access memory (RAM).

[0124] System 865 also includes an input device 860, a parallel processing system 825, and a display device 845, such as a conventional cathode ray tube (CRT), a liquid crystal display (LCD), a light emitting diode (LED), a plasma display, etc. User input can be received from input device 860, such as a keyboard, a mouse, a touchpad, a microphone, etc. Each of the above modules and / or devices can even be located on a single semiconductor platform to form system 865. Alternatively, various modules can be placed individually or in various combinations of semiconductor platforms according to user needs.

[0125] Furthermore, system 865 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a point-to-point network, a wired network, etc.) through network interface 835 for communication purposes.

[0126] System 865 may also include secondary storage (not shown). Secondary storage includes, for example, a hard drive and / or a removable storage drive, representing a floppy disk drive, a tape drive, an optical disk drive, a digital versatile disk (DVD) drive, a recording device, or a universal serial bus (USB) flash drive. The removable storage drive reads and / or writes to a removable storage unit in a well-known manner.

[0127] Computer programs or computer control logic algorithms may be stored in the main memory 840 and / or the secondary memory. Such computer programs, when executed, enable the system 865 to perform various functions. The memory 840, memory, and / or any other memory are possible examples of computer-readable media.

[0128] The architecture and / or functionality of the various preceding figures can be implemented in the context of a general-purpose computer system, a circuit board system, a game console system dedicated for entertainment purposes, a dedicated system, and / or any other desired system. For example, system 865 can take the form of a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, a mobile telephone device, a television, a workstation, a game console, an embedded system, and / or any other type of logic.

[0129] Although various embodiments have been described above, it should be understood that they have been presented by way of example only and not limitation. Therefore, the breadth and scope of a preferred embodiment should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

[0130] Graphics processing pipeline

[0131] In one embodiment, the PPU 600 includes a graphics processing unit (GPU). The PPU 600 is configured to receive commands specifying shaders for processing graphics data. Graphics data can be defined as a set of primitives, such as points, lines, triangles, quadrilaterals, triangle strips, etc. Typically, a primitive includes data specifying a plurality of vertices of the primitive (e.g., in a model space coordinate system) and attributes associated with each vertex of the primitive. The PPU 600 can be configured to process the primitives to generate a frame buffer (e.g., pixel data for each pixel of a display).

[0132] The application writes the model data of the scene (e.g., a collection of vertices and attributes) to a memory, such as system memory or memory 604. The model data defines each object that may be visible on the display. The application then issues an API call to the driver kernel, which requests that the model data be rendered and displayed. The driver kernel reads the model data and writes commands to one or more streams to perform operations to process the model data. These commands can reference different shader programs to be implemented on the multiprocessors 740 of the PPU 600, including one or more of a vertex shader, a hull shader, a domain shader, a geometry shader, and a pixel shader. For example, one or more of the multiprocessors 740 can be configured to execute a vertex shader program that processes multiple vertices defined by the model data. In one embodiment, different multiprocessors 740 can be configured to execute different shader programs concurrently. For example, a first subset of the multiprocessors 740 can be configured to execute a vertex shader program, while a second subset of the multiprocessors 740 can be configured to execute a pixel shader program. A first subset of multiprocessors 740 processes vertex data to generate processed vertex data and writes the processed vertex data to L2 cache 460 and / or memory 604. After rasterizing the processed vertex data (e.g., converting three-dimensional data into two-dimensional data in screen space) to generate fragment data, a second subset of multiprocessors 740 executes pixel shaders to generate processed fragment data, which is then blended with other processed fragment data and written to a frame buffer in memory 604. Vertex shader programs and pixel shader programs can execute concurrently, processing different data from the same scene in a pipelined manner until all model data for the scene is rendered to the frame buffer. The contents of the frame buffer are then transmitted to a display controller for display on a display device.

[0133] Figure 9 According to one embodiment, Figure 6 600. The graphics processing pipeline 900 is an abstract flow chart of the processing steps used to generate a 2D computer-generated image from 3D geometric data. As is well known, pipeline architectures can more efficiently perform long-latency operations by splitting the operations into multiple stages, where the output of each stage is coupled to the input of the next consecutive stage. Thus, the graphics processing pipeline 900 receives input data 901 that is transmitted from one stage of the graphics processing pipeline 900 to the next stage to generate output data 902. In one embodiment, the graphics processing pipeline 900 may represent a graphics processing pipeline composed of API-defined graphics processing pipeline. As an option, graphics processing pipeline 900 can be implemented within the functional and architectural context of the preceding figures and / or any one or more subsequent figures.

[0134] like Figure 9 As shown, graphics processing pipeline 900 includes a pipeline architecture comprising multiple stages. These stages include, but are not limited to, a data assembly stage 910, a vertex shading stage 920, a primitive assembly stage 930, a geometry shading stage 940, a viewport scale, cull, and clip (VSCC) stage 950, a rasterization stage 960, a fragment shading stage 970, and a raster operation stage 980. In one embodiment, input data 901 includes commands that configure a processing unit to implement each stage of graphics processing pipeline 900 and the geometric primitives (e.g., points, lines, triangles, quads, triangle strips, or fans) processed by each stage. Output data 902 may include pixel data (e.g., color data), which is copied to a frame buffer or other type of surface data structure in memory.

[0135] The data assembly stage 910 receives input data 901 specifying vertex data for a high-order surface, primitive, or the like. The data assembly stage 910 collects the vertex data from temporary storage or a queue, for example, by receiving a command from the host processor that includes a pointer to a buffer in memory and reading the vertex data from the buffer. The vertex data is then transferred to the vertex shading stage 920 for processing.

[0136] The vertex shading stage 920 processes vertex data by executing a set of operations (e.g., a vertex shader or program) on each vertex. For example, a vertex may be specified as a 4-coordinate vector (e.g.,<x,y,z,w> ). The vertex shading stage 920 can manipulate individual vertex attributes, such as position, color, texture coordinates, etc. In other words, the vertex shading stage 920 performs operations on vertex coordinates or other vertex attributes associated with the vertex. These operations typically include lighting operations (e.g., modifying the color attribute of a vertex) and transformation operations (e.g., modifying the coordinate space of a vertex). For example, a vertex can be specified using coordinates in object coordinate space, which are transformed by multiplying the coordinates by a matrix that converts the coordinates from object coordinate space to world space or normalized device coordinate (NCD) space. The vertex shading stage 920 generates transformed vertex data that is transmitted to the primitive assembly stage 930.

[0137] The primitive assembly stage 930 collects the vertices output by the vertex shading stage 920 and groups the vertices into geometric primitives for processing by the geometry shading stage 940. For example, the primitive assembly stage 930 can be configured to group every three consecutive vertices into geometric primitives (e.g., triangles) for transmission to the geometry shading stage 940. In some embodiments, certain vertices can be reused for consecutive geometric primitives (e.g., two consecutive triangles in a triangle strip can share two vertices). The primitive assembly stage 930 transmits the geometric primitives (e.g., a collection of related vertices) to the geometry shading stage 940.

[0138] The geometry shading stage 940 processes geometric primitives by executing a set of operations (e.g., geometry shaders or programs) on the geometric primitives. Tessellation operations can generate one or more geometric primitives from each geometric primitive. In other words, the geometry shading stage 940 can subdivide each geometric primitive into a finer grid of two or more geometric primitives for processing by the rest of the graphics processing pipeline 900. The geometry shading stage 940 transmits the geometric primitives to the viewport SCC stage 950.

[0139] In one embodiment, the graphics processing pipeline 900 may be executed within a multiprocessor, and the vertex shading stage 920, primitive assembly stage 930, geometry shading stage 940, fragment shading stage 970, and / or hardware / software associated therewith may sequentially perform processing operations. Once the sequential processing operations are complete, in one embodiment, the viewport SCC stage 950 may utilize the data. In one embodiment, the raw data processed by one or more stages in the graphics processing pipeline 900 may be written to a cache (e.g., an L1 cache, a vertex cache, etc.). In this case, in one embodiment, the viewport SCC stage 950 may access the data in the cache. In one embodiment, the viewport SCC stage 950 and the rasterization stage 960 are implemented as fixed-function circuits.

[0140] The viewport SCC stage 950 performs viewport scaling, culling, and clipping of geometric primitives. Each surface rendered to is associated with an abstract camera position. The camera position represents the position from which the viewer views the scene and defines a viewing cone that contains the scene objects. The viewing cone may include a viewing plane, a back plane, and four clipping planes. Any geometric primitive that is completely outside the viewing cone may be culled (e.g., discarded) because the geometric primitive does not contribute to the final rendered scene. Any geometric primitive that is partially inside the viewing cone and partially outside the viewing cone may be clipped (e.g., converted to a new geometric primitive that is enclosed within the viewing cone). In addition, each geometric primitive may be scaled based on the depth of the viewing cone. All potentially visible geometric primitives are then transferred to the rasterization stage 960.

[0141] The rasterization stage 960 converts 3D geometric primitives into 2D fragments (e.g., capable of being used for display, etc.). The rasterization stage 960 can be configured to use the vertices of the geometric primitives to set a set of plane equations from which various attributes can be interpolated. The rasterization stage 960 can also calculate a coverage mask for multiple pixels, which indicates whether one or more sample positions of the pixel intercept the geometric primitive. In one embodiment, a z test can also be performed to determine whether the geometric primitive has been occluded by other geometric primitives being rasterized. The rasterization stage 960 generates fragment data (e.g., interpolated vertex attributes associated with a particular sample position for each covered pixel), which is transmitted to the fragment shading stage 970.

[0142] The fragment shading stage 970 processes the fragment data by executing a set of operations (e.g., a fragment shader or program) on each fragment. The fragment shading stage 970 can generate pixel data (e.g., color values) for the fragment, for example by performing lighting operations or sampling texture mapping using the fragment's interpolated texture coordinates. The fragment shading stage 970 produces pixel data that is passed to the raster operations stage 980.

[0143] The raster operations stage 980 may perform various operations on the pixel data, such as performing alpha testing, stencil testing, and blending the pixel data with other pixel data corresponding to other fragments associated with the pixel. When the raster operations stage 980 completes processing the pixel data (e.g., output data 902), the pixel data may be written to a render target such as a frame buffer, a color buffer, or the like.

[0144] It should be understood that one or more additional stages may be included in the graphics processing pipeline 900 in addition to or in place of one or more of the stages described above. Various implementations of the abstract graphics processing pipeline may implement different stages. Furthermore, in some embodiments (e.g., the geometry shading stage 940), one or more of the stages described above may be excluded from the graphics processing pipeline. Other types of graphics processing pipelines are considered within the scope of this disclosure. Furthermore, any stage of the graphics processing pipeline 900 may be implemented by one or more dedicated hardware units within a graphics processor, such as the PPU 600. Other stages of the graphics processing pipeline 900 may be implemented by programmable hardware units, such as the multiprocessor 740 of the PPU 600.

[0145] The graphics processing pipeline 900 can be implemented by an application executed by a host processor (e.g., a CPU). In one embodiment, a device driver can implement an application programming interface (API) that defines various functions that the application can use to generate graphics data for display. A device driver is a software program that includes multiple instructions that control the operation of the PPU 600. The API provides programmers with a concept that allows them to utilize specialized graphics hardware (such as the PPU 600) to generate graphics data without requiring them to utilize the PPU 600's specific instruction set. An application may include an API call, which is routed to the PPU 600's device driver. The device driver interprets the API call and performs various operations in response to the API call. In some cases, the device driver may perform operations by executing instructions on the CPU. In other instances, the device driver may perform operations at least in part by initiating operations on the PPU 600 using an input / output interface between the CPU and the PPU 600. In one embodiment, the device driver is configured to implement the graphics processing pipeline 900 using the PPU 600's hardware.

[0146] Various programs can be executed within the PPU 600 to implement various stages of the graphics processing pipeline 900. For example, a device driver can launch a kernel on the PPU 600 to execute the vertex shading stage 920 on one multiprocessor 740 (or multiple multiprocessors 740). The device driver (or the initial kernel executed by the PPU 600) can also launch other kernels on the PPU 600 to execute other stages of the graphics processing pipeline 900, such as the geometry shading stage 940 and the fragment shading stage 970. In addition, certain stages of the graphics processing pipeline 900 can be implemented in fixed unit hardware, such as the rasterizer or data assembler implemented within the PPU 600. It should be understood that the results from one kernel may be processed by one or more intermediate fixed-function hardware units before being processed by subsequent kernels on the multiprocessor 740.

[0147] Machine Learning

[0148] Deep Neural Networks (DNNs) developed on processors like the PPU 600 have been put to use in a variety of applications, from self-driving cars to faster drug development, from automatic image captioning in online image databases to intelligent real-time language translation in video chat applications. Deep learning is a technology that mimics the neural learning process of the human brain, continuously learning over time, getting smarter, and providing more accurate results faster. A child is initially taught by an adult to correctly identify and classify various shapes, and eventually can recognize shapes without any guidance. Similarly, deep learning or neural learning systems need to be trained in object recognition and classification in order to become smarter and more effective at recognizing basic objects, occluded objects, and so on, while also assigning context to objects.

[0149] At the simplest level, neurons in the human brain observe the various inputs they receive, assign importance to each input, and then pass the output to other neurons for processing. An artificial neuron, or perceptron, is the most basic model of a neural network. In one example, a perceptron can receive one or more inputs representing various features of the object it is being trained to recognize and classify, and assign a weight to each of these features based on its importance in defining the object's shape.

[0150] DNN models consist of multiple layers of connected nodes (e.g., perceptrons, Boltzmann machines, radial basis functions, convolutional layers, etc.) and can be trained with large amounts of input data to solve complex problems quickly and accurately. In one example, the first layer of a DNN model breaks down an input image of a car into its parts and looks for basic patterns such as lines and angles. The second layer assembles the lines and looks for higher-level patterns such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the final layers generate labels for the input image, identifying models of specific car brands.

[0151] Once trained, a deep neural network can be deployed and used to recognize and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include recognizing handwritten digits on checks deposited at an ATM, identifying images of friends in photos, providing movie recommendations to over 50 million users, identifying and classifying different types of cars, pedestrians, and road hazards in self-driving cars, or translating human speech in real time.

[0152] During training, data flows through the DNN in a forward propagation phase until it produces a prediction for a label corresponding to the input. If the neural network does not correctly label the input, the error between the correct and predicted labels is analyzed, and the weights of each feature are adjusted in a backpropagation phase until the DNN correctly labels the input and other inputs in the training dataset. Training complex neural networks requires a large amount of parallel computing performance, including floating-point multiplications and additions supported by the PPU 600. Inference, which is less computationally intensive than training, is a latency-sensitive process in which a trained neural network is applied to new, previously unseen inputs to classify images, translate speech, and generally infer new information.

[0153] Neural networks rely heavily on matrix math operations, and complex, multi-layer networks require significant floating-point performance and bandwidth for efficiency and speed. The PPU 600, with thousands of processing cores optimized for matrix math operations and delivering tens to hundreds of TFLOPS of performance, is a computing platform capable of delivering the performance required for deep neural network-based artificial intelligence and machine learning applications.

Claims

1. A system comprising: a first memory comprising a first plurality of memory blocks; a first register configured to maintain a bitmap using a first bit portion of the first register, wherein each bit in the first bit portion of the first register represents a corresponding memory block of the first plurality of memory blocks; a processing device operatively coupled to the first memory and the first register, wherein the processing device is configured to: receiving a first request to allocate one or more memory blocks of the first plurality of memory blocks; identifying a contiguous set of the first bit portions of the first register having a first logic state, wherein the first logic state indicates that a corresponding memory block of the one or more memory blocks is free; performing a first operation to adjust the contiguous set of the first bit portions of the first register to a second logic state; and An allocated address including an index of the contiguous set of first bit portions of the first register is sent to the first request, wherein the allocated address can be used to access the corresponding memory block. 2 . The system of claim 1 , wherein the contiguous set of the first bit portions of the first register are aligned with a size of the corresponding memory block.

3. The system of claim 1 , wherein the processing device is further configured to: receiving a request to deallocate the one or more memory blocks; identifying the contiguous set of the first bit portions; and A second operation is performed to adjust the contiguous set of first bit portions to the first logic state.

4. The system according to claim 1, further comprising: a second memory operatively coupled to the processing device, the second memory comprising a second plurality of memory blocks; a second register operatively coupled to the processing device, the second register configured to maintain a bitmap using a first bit portion of the second register, wherein each bit in the first bit portion of the second register represents a corresponding memory block of the second plurality of memory blocks, and wherein the processing device is further configured to: receiving, along with the first request, an indication to allocate one or more of the second plurality of memory blocks collectively with the one or more of the first plurality of memory blocks; identifying a contiguous set of first bit portions of the second register using the contiguous set of first bit portions of the first register; as well as A second operation is performed to modify the contiguous set of the first bit portions of the second register to a second logic state, wherein an index of the contiguous set of the first bit portions of the second register is the same index as an index of the contiguous set of the first bit portions of the first register. 5 . The system of claim 1 , wherein the processing device comprises a parallel processing unit (PPU), and wherein the first request comprises a first instruction issued by a first thread executing on the PPU.

6. The system of claim 5, wherein the PPU is configured to: identifying one or more instructions issued by one or more additional threads executing concurrently with the first thread; One or more operations of the one or more additional threads are serialized to prevent the one or more operations from changing the contiguous set of the first bit portion of the first register until the first operation is completed. The system of claim 1 , wherein the first memory is a tensor memory.

8. The system of claim 1, wherein the processing device is configured to use the second bit portion of the first register for at least one of message passing, synchronization, or memory allocation fairness.

9. A method comprising: receiving, by a processing device, a first request to allocate one or more memory blocks of a first plurality of memory blocks associated with a first memory; identifying a contiguous set of first bit portions of a first register of the processing device having a first logic state, wherein the first logic state indicates that corresponding memory blocks of the one or more memory blocks are free; performing a first operation to adjust the contiguous set of the first bit portions of the first register to a second logic state; and An allocated address including an index of the contiguous set of first bit portions of the first register is sent to the first request, wherein the allocated address can be used to access the corresponding memory block.

10. The method of claim 9, wherein the contiguous set of the first bit portions of the first register are aligned to a size of the corresponding memory block.

11. The method according to claim 9, further comprising: receiving a request to deallocate the one or more memory blocks; identifying said contiguous set of said first bit portions; as well as A second operation is performed to adjust the contiguous set of first bit portions to the first logic state.

12. The method according to claim 9, further comprising: receiving, along with the first request, an indication to allocate one or more memory blocks of a second plurality of memory blocks associated with a second memory together with the one or more memory blocks of the first plurality of memory blocks; identifying a contiguous set of first bit portions of a second register using the contiguous set of the first bit portions of the first register; as well as A second operation is performed to modify the contiguous set of the first bit portions of the second register to a second logic state, wherein an index of the contiguous set of the first bit portions of the second register is the same index as an index of the contiguous set of the first bit portions of the first register.

13. The method of claim 9, wherein the processing device comprises a parallel processing unit (PPU), and wherein the first request comprises a first instruction issued by a first thread executing on the PPU.

14. The method according to claim 13, further comprising: identifying one or more instructions issued by one or more additional threads executing concurrently with the first thread; as well as One or more operations of the one or more additional threads are serialized to prevent the one or more operations from changing the contiguous set of the first bit portion of the first register until the first operation is completed.

15. The method of claim 9, wherein the processing device comprises a graphics processing unit (GPU), and wherein the first register is associated with a first multiprocessor within the GPU. The method of claim 9 , wherein the first memory is a tensor memory.

17. The method according to claim 9, further comprising: The second bit portion of the first register is used for at least one of message passing, synchronization, or memory allocation fairness.

18. A parallel processing unit (PPU), comprising: a first memory comprising a first plurality of memory blocks; a first register configured to maintain a bitmap using a first bit portion of the first register, wherein each bit in the first bit portion of the first register represents a corresponding memory block of the first plurality of memory blocks, and wherein the PPU is to: receiving a first request to allocate one or more memory blocks of the first plurality of memory blocks; identifying a contiguous set of the first bit portions of the first register having a first logic state, wherein the first logic state indicates that a corresponding memory block of the one or more memory blocks is free; performing a first operation to adjust the contiguous set of the first bit portions of the first register to a second logic state; and An allocated address including an index of the contiguous set of first bit portions of the first register is sent to the first request, wherein the allocated address can be used to access the corresponding memory block.

19. The PPU of claim 18, wherein the contiguous set of the first bit portions of the first register are aligned to a size of the corresponding memory block.

20. The PPU of claim 18, wherein the PPU is further configured to: receiving a request to deallocate the one or more memory blocks; identifying the contiguous set of the first bit portions; and A second operation is performed to adjust the contiguous set of first bit portions to the first logic state.

Images