Dynamic partitioning of execution resources

Abstract

A computer-implemented method for allocating execution resources to groups of threads within a graphics processing unit, wherein the method comprises: Receiving a notification from a process that a first group of threads, which is assigned to a first subcontext, is to be started; Determine, based on a count of a credit counter (434), whether at least one processor credit remains from an initial number of processor credits that was initially allocated to the first subcontext, wherein the initial number of processor credits specifies a maximum number of processors (340) on which the first subcontext can be processed concurrently, and wherein the credit counter is initially set to the initial number of processor credits and is decremented each time a group of threads allocated to the first subcontext is started for execution on a processor not yet allocated to the first subcontext; if at least one processor credit remains, then identify a first processor that is contained in a plurality of processors (340) and that has a processing load less than or equal to the processor loads allocated to all the other processors contained in the plurality of processors; Starting the first group of threads for execution on the first processor, or If no processor credits remain, then identify a second processor in a subset of processors already assigned to the first subcontext, where the second processor has a processor load less than or equal to the processor load of all other processors in the subset of processors, and Starting the first group of threads for execution on the second processor.

Description

Background of the invention; Field of the invention

[0001] Embodiments of the present invention relate generally to computer processing and in particular to dynamic partitioning of execution resources. Description of the associated technology

[0002] Modern graphics processing units (GPUs) are typically configured to execute a large number of threads in parallel. A GPU is usually restricted to working in one context at a time, with all threads running in the same context. This restriction means, among other things, that every thread running at any given time shares the same GPU state and virtual address space. Such an operating model works well for a process exhibiting high levels of parallelism, allowing the process to utilize a significant percentage of the GPU's available processing resources.Many applications, however, run multiple processes, with each process exhibiting only enough parallelism to utilize a small percentage of the GPU's available processing resources, and each process requiring a different GPU state and virtual address space. Consequently, applications running multiple processes often run inefficiently on GPUs.

[0003] To solve this problem, some GPU execution models operate within a single context, which contains multiple subcontexts, each assigned to a different process. With this approach, all subcontexts operate within the same context, but each has a different GPU state and virtual address space. In one particular implementation, each subcontext is statically assigned to one or more specific Texture Processing Clusters (TPCs), with each TPC containing two or more Streaming Multiprocessors (SMs), allowing each TPC to execute a certain number of threads concurrently. Furthermore, the subset of virtual address space reserved for use as local thread memory for each subcontext is statically allocated to each TPC that the subcontext is permitted to use.

[0004] One disadvantage of the above implementation is that the allocation of TPCs and memory spaces to the various subcontexts associated with the different processes is static, limiting the system's ability to load balance between these TPCs and memory resources. For example, the different processes associated with the various subcontexts may require different amounts of processing and memory resources. Furthermore, the processing and memory requirements for a given process may change over time. However, because the TPCs and memory allocated to the various subcontexts are static, these resources cannot be increased or decreased as conditions or requirements change.For example, a process running in a subcontext allocated to a set of heavily loaded TPCs might start a new Cooperative Thread Array (CTA). However, the new CTA would only be able to start on the set of heavily loaded TPCs, since these are the ones allocated to that process, even if other TPCs allocated to other processes are less loaded or idle. Therefore, static partitioning of resources across different subcontexts can also lead to inefficient allocation and execution of GPU resources. Reference is also made in this technical field to US 2011 / 0 191 627 A1, US 2017 / 0 031 719 A1, US 2011 / 0 072 244 A1, US 2008 / 0 250 422 A1, and US 2017 / 0286 168 A1. US 2012 / 0 110 586 A1 discloses the execution of thread groups belonging to multiple CTAs.

[0005] As the foregoing illustrates, more effective techniques for allocating execution resources within a processor are needed in engineering. Brief description of the invention

[0006] Embodiments of the present invention constitute a computer-implemented method for allocating execution resources to groups of threads within a graphics processing unit according to claim 1. The method includes, among other things, receiving a signal from a process to start a first group of threads. The method further includes identifying a first processor, which is included in a plurality of processors and which has a processing load that is less than or equal to the processor loads allocated to all other processors included in the plurality of processors. The method further includes starting the first group of threads for execution on the first processor.

[0007] Other embodiments of the present invention include, without limitation, a parallel processing subsystem according to claim 12 for carrying out one or more aspects of the disclosed techniques, and a system for carrying out one or more aspects of the disclosed techniques.

[0008] At least one advantage of the disclosed techniques is that execution and local memory resources can be flexibly and efficiently allocated to subcontexts corresponding to multiple processes within a parallel processing system. This increases the utilization of execution and local memory resources compared to previous methods. Another advantage of the disclosed techniques is that the maximum amount of execution and local memory resources that can be allocated or assigned to a subcontext is selectable and can be limited within the constraints of the number of available TPCs and local memory blocks, thus enabling more subcontexts to execute concurrently. A further advantage of the disclosed techniques is that all subcontexts execute within a single context but maintain separate virtual address spaces and separate state data.This allows TPCs to quickly switch from executing a CTA for one subcontext to executing a CTA for another subcontext without requiring a full context switch. Brief description of the drawings

[0009] For a more detailed understanding of the features of the present invention specified above, a more specific description of the invention summarized above is available with reference to embodiments, some of which are illustrated in the accompanying drawings. It should be noted, however, that the accompanying drawings only illustrate typical embodiments of this invention and are therefore not to be considered as limiting its scope of protection, since the invention may permit other equally effective embodiments. Fig. Figure 1 is a block diagram illustrating a computer system configured to implement one or more aspects of the present invention; Fig. Figure 2 is a block diagram of a parallel processing unit (PPU) located in the parallel processing subsystem of Fig. 1 is included, according to various embodiments of the present invention; Fig. Figure 3 is a block diagram of a general processing cluster (GPC) located in the parallel processing unit (PPU) of Fig. 2 is included, according to various embodiments of the present invention; Fig. 4 is a more detailed view of the task / work unit of Fig. 2 according to different embodiments of the present invention; Fig. Figures 5A-5B illustrate a TPC release table and an LMEM block index table for static TPC partitioning according to various embodiments of the present invention; Fig. Figures 6A-6B illustrate a TPC release table and an LMEM block index table for static TPC partitioning according to further various embodiments of the present invention; Fig. Figure 7 illustrates a TPC release table 800 for dynamic TPC partitioning according to various embodiments of the present invention; Fig. Figure 8 illustrates a TPC release table 800 for dynamic TPC partitioning according to other different embodiments of the present invention; Fig. Figures 9A-9C show a flowchart of process steps for allocating execution resources within a processor according to various embodiments of the present invention; and Fig. Figures 10A-10B show a flowchart of process steps for allocating local memory resources within a processor according to different embodiments of the present invention. Detailed description

[0010] The following description presents numerous specific details to facilitate a better understanding of the present invention. However, a person skilled in the art will recognize that the present invention can be realized without one or more of these specific details. System overview

[0011] Fig. Figure 1 is a block diagram illustrating a computer system 100 configured to implement one or more aspects of the present invention. In some embodiments, the computer system 100 is a server machine operating in a data center or cloud computing environment, providing scalable computing resources as a service over a network. As shown, the computer system 100 includes, without limitation, a central processing unit (CPU) 102 and system memory 104, which is coupled to a parallel processing subsystem 112 via a memory bridge 105 and a communication path 113. The memory bridge 105 is further coupled to an I / O (input / output) bridge 107 via a communication path 106, and the I / O bridge 107 is in turn coupled to a switch 116.

[0012] In operation, the I / O bridge 107 is configured to receive user input information from optional input devices 108, such as a keyboard or mouse, and to forward the input information to the CPU 102 for processing via the communication path 106 and the memory bridge 105. In some embodiments, the computer system 100 may be a server machine in a cloud computing environment. In such embodiments, the computer system 100 may not have input devices 108. Instead, the computer system 100 can receive equivalent input information by receiving commands in the form of messages sent over a network and received via the network adapter 118. The switch 116 is configured to provide connections between the I / O bridge 107 and other components of the computer system 100, such as a network adapter 118 and various add-in cards 120 and 121.

[0013] As also shown, the I / O bridge 107 is coupled to a system disk 114, which can be configured to store content, applications, and data for use by the CPU 102 and the parallel processing subsystem 112. Generally, the system disk 114 provides non-volatile memory for applications and data and can contain hard or removable disk drives, flash memory devices, and CD-ROM (Compact Disc Read-only Memory), DVD-ROM (Digital Versatile Disc-ROM), Blu-ray, HD-DVD (High Definition DVD), or other magnetic, optical, or semiconductor storage devices. Finally, although not explicitly shown, other components such as USB (Universal Serial Bus) or other port connections, CD drives, DVD drives, movie recording devices, and the like can also be connected to the I / O bridge 107.

[0014] In various embodiments, the memory bridge 105 can be a northbridge chip and the I / O bridge 107 can be a southbridge chip. Furthermore, the communication paths 106 and 113, as well as other communication paths within the computer system 100, can be implemented using any technically suitable protocols, including, but not limited to, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol known in the art.

[0015] In some embodiments, the parallel processing subsystem 112 includes a graphics subsystem that supplies pixels to an optional display device 110, which may be a conventional cathode ray tube, a liquid crystal display, a light-emitting diode display, or the like. In such embodiments, the parallel processing subsystem 112 includes a circuit optimized for graphics and video processing, which, for example, includes a video output circuit. As described below in conjunction with Fig. 2 and Fig. As described in more detail in Section 3, such a circuit can be incorporated via one or more parallel processing units (PPUs), also referred to herein as parallel processors, which are included in the parallel processing subsystem 112. In further embodiments, the parallel processing subsystem 112 includes a circuit optimized for general-purpose and / or computational processing. Here, too, such a circuit can be incorporated via one or more PPUs included in the parallel processing subsystem 112 and configured to perform such general-purpose and / or computational operations. In still further embodiments, the one or more PPUs included in the parallel processing subsystem 112 can be configured to perform graphics processing, general-purpose processing, and computational processing operations.The system memory 104 contains at least one device driver 103 that is configured to manage the processing operations of one or more PPUs within the parallel processing subsystem 112.

[0016] In various embodiments, the parallel processing subsystem 112 can be combined with one or more of the other elements of Fig. 1. be integrated into a single system. For example, the parallel processing subsystem 112 can be integrated with the CPU 102 and other interconnect circuitry on a single chip to form a system-on-a-chip (SoC).

[0017] During operation, CPU 102 is the master processor of computer system 100, controlling and coordinating the operations of other system components. Specifically, CPU 102 issues instructions that control the operation of PPUs. In some embodiments, the communication path 113 is a PCI Express connection, with dedicated lanes allocated to each PPU, as is known in the art. Other communication paths may also be used. The PPU advantageously implements a highly parallel processing architecture. A PPU may be equipped with any amount of local parallel processing memory (PP memory).

[0018] Note that the system shown here is for illustrative purposes only and that deviations and modifications are possible. The connection topology, including the number and arrangement of the bridges, the number of CPUs 102, and the number of parallel processing subsystems 112, can be changed as needed. For example, in some embodiments, the system memory 104 could be connected directly to the CPU 102, rather than via the memory bridge 105, and other devices would communicate with the system memory 104 via the memory bridge 105 and the CPU 102. In other alternative topologies, the parallel processing subsystem 112 could be connected to the I / O bridge 107 or directly to the CPU 102, rather than via the memory bridge 105. In still other embodiments, the I / O bridge 107 and the memory bridge 105 could be integrated into a single chip, instead of existing as one or more discrete devices.Finally, in certain embodiments, one or more of the elements in . Fig. Some of the components shown may not be present. For example, switch 116 could be removed, and the network adapter 118 and add-in cards 120 and 121 would be directly connected to the I / O bridge 107.

[0019] Fig. Figure 2 is a block diagram of a parallel processing unit (PPU) 202, which is part of the parallel processing subsystem 112 of Fig. 1 is included, according to various embodiments of the present invention. Fig. Figure 2 shows a (numeric word) PPU, as indicated above, but the parallel processing subsystem 112 can contain any number of PPUs 202. As shown, the PPU 202 is coupled to a local parallel processing memory (PP memory) 204. The PPU 202 and the PP memory 204 can be implemented using one or more integrated circuits, such as programmable processors, application-specific integrated circuits (ASICs), or memory devices, or in any other technically feasible way.

[0020] In some embodiments, the PPU 202 includes a graphics processing unit (GPU) that can be configured to implement a graphics rendering pipeline to perform various operations related to generating pixel data based on graphics data supplied by the CPU 102 and / or the system memory 104. When processing graphics data, the PP memory 204 can be used as graphics memory, storing one or more conventional frame buffers and, if required, one or more other render targets. Among other things, the PP memory 204 can be used to store and update pixel data and to deliver final pixel data or display images to an optional display device 110 for display. In some embodiments, the PPU 202 can also be configured for general-purpose processing and computational operations. In some embodiments, the computer system 100 can be a server machine in a cloud computing environment.In such embodiments, the computer system 100 may not have a display device 110. Instead, the computer system 100 can generate equivalent output information by sending commands in the form of messages over a network via the network adapter 118.

[0021] In some embodiments, the CPU 102 is the master processor of the computer system 100, controlling and coordinating the operations of other system components. Specifically, the CPU 102 issues instructions that control the operation of the PPU 202. In some embodiments, the CPU 102 writes an instruction stream for the PPU 202 to a data structure (neither in Fig. 1 still in Fig. (2 explicitly shown), which may be located in system memory 104, PP memory 204, or another memory location accessible to both the CPU 102 and the PPU 202. A pointer to the data structure is written to an instruction queue, also referred to herein as a push buffer, to initiate the processing of the instruction stream in the data structure. The PPU 202 reads instruction streams from the instruction queue and then executes instructions asynchronously to the operation of the CPU 102. In embodiments where multiple push buffers are created, execution priorities for each push buffer can be set by an application program via the device driver 103 to control the scheduling of the different push buffers.

[0022] As also shown, the PPU 202 contains an I / O (input-output) unit 205, which communicates with the rest of the computer system 100 via the communication path 113 and the memory bridge 105. The I / O unit 205 generates packets (or other signals) for transmission on the communication path 113 and also receives all incoming packets (or other signals) from the communication path 113, routing the incoming packets to appropriate components of the PPU 202. For example, commands relating to processing tasks can be routed to a host interface 206, while commands relating to memory operations (e.g., reading from or writing to the PP memory 204) can be routed to a crossbar unit 210. The host interface 206 reads each command queue and sends the command stream stored in the command queue to a preprocessor 212.

[0023] As above in connection with Fig. As mentioned in Figure 1, the connection of the PPU 202 to the rest of the computer system 100 can be modified. In some embodiments, the parallel processing subsystem 112, which contains at least one PPU 202, is implemented as an add-in card that can be inserted into an expansion slot of the computer system 100. In other embodiments, the PPU 202 can be integrated on a single chip with a bus bridge, such as a memory bridge 105 or an I / O bridge 107. In still other embodiments, some or all elements of the PPU 202 can be included together with the CPU 102 in a single integrated circuit or system-on-a-chip (SoC).

[0024] During operation, the preprocessor 212 sends processing tasks received from the host interface 206 to a workload distribution unit (not shown) within the task / work unit 207. The workload distribution unit receives pointers to processing tasks, which are encoded as task metadata (TMD) and stored in memory. The pointers to TMDs are contained in an instruction stream, which is stored as an instruction queue and received by the preprocessor 212 from the host interface 206. Processing tasks, which can be encoded as TMDs, contain indices associated with the data to be processed, as well as state parameters and instructions that specify how the data is to be processed. For example, the state parameters and instructions could define the program to be executed on the data. The TMDs could also specify, for example, the number and configuration of the set of command-line actions (CTAs).In general, each TMD corresponds to a task. The task / work unit 207 receives tasks from the preprocessor 212 and ensures that GPCs 208 are configured in a valid state before initiating the processing task specified by each TMD. A priority can be specified for each TMD used to schedule the execution of the processing task. Processing tasks can also be received from the processing cluster array 230. Optionally, the TMD can include a parameter that controls whether the TMD is added to the head or the end of a list of processing tasks (or a list of pointers to the processing tasks), thus providing another layer of control over execution priority.

[0025] The PPU 202 advantageously implements a highly parallel processing architecture based on a processing cluster array 230, which contains a set of C general processing clusters (GPCs) 208, where C ≥ 1. Each GPC 208 is capable of executing a large number (e.g., hundreds or thousands) of threads concurrently, with each thread being an instance of a program. In different applications, different GPCs 208 can be allocated to process different types of programs or to perform different types of computations. The allocation of GPCs 208 can vary depending on the workload for each type of program or computation.

[0026] The memory interface 214 contains a set D of partition units 215, where D ≥ 1. Each partition unit 215 is coupled to one or more dynamic random-access memories (DRAMs) 220 located in the PPM memory 204. In some embodiments, the number of partition units 215 corresponds to the number of DRAMs 220, and each partition unit 215 is coupled to a different DRAM 220. In other embodiments, the number of partition units 215 may differ from the number of DRAMs 220. It is understood that a DRAM 220 can be replaced by any other technically suitable storage medium. In operation, various rendering targets, such as... B. how texture images and image buffers are stored via DRAMs 220, so that the partition units 215 can write parts of each render target in parallel to efficiently utilize the available bandwidth of the PP memory 204.

[0027] A given GPC 208 can process data to be written to one of the DRAMs 220 in the PP memory 204. The coupling unit 210 is configured to route the output of each GPC 208 to the input of any partition unit 215 or to another GPC 208 for further processing. The GPCs 208 communicate with the memory interface 214 via the coupling unit 210 to read from or write to different DRAMs 220. In some embodiments, the coupling unit 210 has a connection to the I / O unit 205 as well as a connection to the PP memory 204 via the memory interface 214, allowing the processing cores within the different GPCs 208 to communicate with the system memory 104 or other memory that is not local to the PPU 202. In the embodiment of Fig. 2 The coupling field unit 210 is directly connected to the I / O unit 205. In various embodiments, the coupling field unit 210 can use virtual channels to separate traffic flows between the GPCs 208 and the partition units 215.

[0028] Here too, GPCs 208 can be programmed to perform processing tasks related to a wide variety of applications, including but not limited to linear and nonlinear data transformations, filtering of video and / or audio data, modeling operations (e.g., applying physical laws to determine the position, velocity, and other attributes of objects), image rendering operations (e.g., tessellation shader, vertex shader, geometry shader, and / or pixel / fragment shader programs), general arithmetic operations, etc. In operation, the PPU 202 is configured to send data from system memory 104 and / or PP memory 204 to one or more on-chip memory units, process the data, and write the resulting data back to system memory 104 and / or PP memory 204.Other system components can then access the result data, including CPU 102, another PPU 202 in parallel processing subsystem 112, or another parallel processing subsystem 112 in computer system 100.

[0029] As mentioned above, any number of PPUs 202 can be included in a parallel processing subsystem 112. For example, multiple PPUs 202 can be deployed on a single add-in card, or multiple add-in cards can be connected via the communication path 113, or one or more PPUs 202 can be integrated into a bridge chip. PPUs 202 in a multi-PPU system can be identical or different. For example, different PPUs 202 can have different numbers of processing cores and / or different amounts of PP memory 204. In implementations where multiple PPUs 202 are present, these PPUs can be operated in parallel to process data at a higher throughput than is possible with a single PPU 202.Systems with one or more PPUs 202 can be implemented in a variety of configurations and form factors, including but not limited to desktops, laptops, handheld PCs or other handheld devices, servers, workstations, game consoles, embedded systems and the like.

[0030] Fig. Figure 3 is a block diagram of a general processing cluster (GPC) 208, which is located in the parallel processing unit (PPU) 202 of Fig. 2 is included, according to various embodiments of the present invention. As shown, the GPC 208 includes, without limitation, a pipeline manager 305, one or more texture units 315, a PreROP unit 325, a work distribution coupling field 330, an L1.5 cache 335, and one or more texture processing clusters (TPCs) 340. The TPCs 340, the texture units 315, and the L1.5 cache 335 are coupled to an MMU 320.

[0031] Each TPC 340 contains several SMs 310 along with other associated circuitry (not shown). In one example, each TPC 340 contains two SMs 310. The task / work unit 207 initiates CTAs directed to the TPCs 340 contained within the various GPCs 408. The pipeline manager 305 receives an initiated CTA from the task / work unit 207 and passes the CTA to the appropriate TPC 340. The TPC 340 then executes the CTA on one or more SMs 310 contained within it.

[0032] In operation, the GPC 208 can be configured to execute a large number of threads in parallel to perform graphics, general-purpose processing, and / or computational operations. As used herein, a "thread" refers to an instance of a particular program running on a specific set of input data. In some embodiments, Single Instruction, Multiple Data (SIMD) instruction-issuing techniques are used to support the parallel execution of a large number of threads without providing multiple independent units of instruction.In other embodiments, single-instruction, multiple-thread (SIMT) techniques are used to support the parallel execution of a large number of generally synchronized threads, using a common instruction unit configured to issue instructions to a set of processing machines within GPC 208. Unlike a SIMD execution regime, where all processing machines typically execute identical instructions, SIMT execution allows different threads to more easily follow diverging execution paths through a given program. A person skilled in the art recognizes that a SIMD processing regime is a functional subset of a SIMT processing regime.

[0033] The operation of the GPC 208 is controlled by a pipeline manager 305, which distributes processing tasks received by a work distribution unit (not shown) within the task / work unit 207 to one or more streaming multiprocessors (SMs) 310. The pipeline manager 305 can also be configured to control a work distribution coupling field 330 by specifying destinations for processed data output by SMs 310.

[0034] In various embodiments, the GPC 208 contains a set M of SMs 310, where M ≥ 1. Each SM 310 also contains a set of functional execution units (not shown), such as execution units and load-store units. Processing operations specific to any of the functional execution units can be handled in a pipeline, allowing a new instruction to be issued for execution before a previous instruction has been fully executed. Any combination of functional execution units can be provided within a given SM 310. In various embodiments, the functional execution units can be configured to support manifold different operations, including integer and floating-point arithmetic (e.g.,Addition and multiplication), comparison operations, Boolean operations (AND, OR, XOR), bit shifting, and calculation of various algebraic functions (e.g., planar interpolation and trigonometric, exponential, and logarithmic functions, etc.). An advantage is that the same functional execution unit can be configured to perform different operations.

[0035] In operation, each SM 310 is configured to process one or more thread groups. As used herein, a "thread group" or "warp" refers to a group of threads that simultaneously execute the same program on different input data, with one thread of the group being assigned to a different execution unit within an SM 310. A thread group may contain fewer threads than the number of execution units within the SM 310, in which case part of the execution may be idle during cycles in which that thread group is being processed. A thread group may also contain more threads than the number of execution units within the SM 310, in which case processing may occur over successive clock cycles.Since each SM 310 can support up to G thread groups simultaneously, it follows that at any given time up to G*M thread groups can be executed in the GPC 208.

[0036] Additionally, a large number of related thread groups (in different stages of execution) can be active simultaneously within an SM 310. This collection of thread groups is referred to herein as the "Cooperative Thread Array" (CTA) or "Thread Array." The size of a given CTA is equal to m*k, where k is the number of concurrently executing threads in a thread group, which is typically an integer multiple of the number of execution units within the SM 310, and m is the number of thread groups simultaneously active within the SM 310. In some embodiments, a single SM 310 can support multiple CTAs concurrently, with such CTAs representing the granularity at which work is distributed among the SMs 310.

[0037] Although in Fig. Not shown in Figure 3, each SM 310 contains a Level One (L1) cache or uses space in a corresponding L1 cache outside the SM 310 to support, among other things, load and store operations performed by the execution units. Each SM 310 also has access to Level Two (L2) caches (not shown) shared by all GPCs 208 in the PPU 202. The L2 caches can be used to transfer data between threads. Finally, SMs 310 also have access to off-chip “global” memory, which may include PP memory 204 and / or system memory 104. It can be seen that any memory outside the PPU 202 can be used as global memory. Additionally, as shown in Figure 3, the SMs 310 can also be used as global memory. Fig. Figure 3 shows a Level 1.5 (L1.5) cache 335 contained within the GPC 208 and configured to receive and hold data requested from memory by the SM 310 via the memory interface 214. This data can include instructions, uniform data, and constant data without limitation. In embodiments with multiple SMs 310 within the GPC 208, the SMs 310 can advantageously share common instructions and data cached in the L1.5 cache 335.

[0038] Each GPC 208 can have an associated memory management unit (MMU) 320 configured to map virtual addresses to physical addresses. In various embodiments, the MMU 320 can reside either within the GPC 208 or within the memory interface 214. The MMU 320 contains a set of page table entries (PTEs) used to map a virtual address to a physical address of a tile or memory page, and optionally a cache row index. The MMU 320 can include address translation lookaside buffers (TLBs) or caches, which may reside in SMs 310, in one or more L1 caches, or within the GPC 208.

[0039] In graphics and computing applications, the GPC 208 can be configured so that each SM 310 is coupled with a texture unit 315 to perform texture mapping operations, such as determining texture sampling positions, reading texture data, and filtering texture data.

[0040] In operation, each SM 310 sends a processed task to the work distribution switching unit 330 to make the processed task available to another GPC 208 for further processing, or to store the processed task via the switching unit 210 in an L2 cache (not shown), parallel processing memory 204, or system memory 104. Additionally, a pre-raster operations (preROP) unit 325 is configured to receive data from the SM 310, route data to one or more raster operations (ROP) units within the partition units 215, perform color mixing optimizations, organize pixel color data, and perform address translations.

[0041] Note that the core architecture described herein is for illustrative purposes only and that changes and modifications are possible. Among other things, any number of processing units, such as SMs 310, texture units 315, or preROP units 325, can be incorporated into the GPC 208. Furthermore, the PPU 202, as described above in conjunction with Fig. As described in Figure 2, the PPU 202 contains any number of GPCs 208 configured to be functionally similar to one another, such that the execution behavior does not depend on which GPC 208 receives a particular processing task. Furthermore, each GPC 208 operates independently of the other GPCs 208 in the PPU 202 to perform tasks for one or more application programs. In light of the foregoing, the person skilled in the art recognizes that the GPCs 208 in the PPU 202 are configured to be functionally similar to one another, such that the execution behavior does not depend on which GPC 208 receives a particular processing task. Fig. The architecture described in 1-3 does not in any way limit the scope of protection of the present invention. Dynamic partitioning of execution resources

[0042] According to various embodiments of the present invention, the compute work distributor (CWD) within the GPU is designed to include a credit-based system for dynamic partitioning and allocation of TPCs 340 to different subcontexts, which are assigned to different processes running on the GPU. Such a method leads, among other things, to more effective resource allocation among the different processes running on the GPU and to more efficient GPU execution. A CPU process can be assigned to one or more subcontexts.

[0043] According to various embodiments of the present invention, during GPU initialization, a GPU device driver allocates a number of TPC credits to each subcontext, wherein each subcontext corresponds to a different process and the number of TPC credits determines the maximum number of TPCs 340 that the subcontext can use concurrently. In some embodiments, the subcontexts may not correspond to separate processes. For example, a particular process could be associated with multiple subcontexts. When a new CTA is started for a particular subcontext at runtime, the CWD allocates one of the TPC credits to the corresponding subcontext by decrementing a credit counter associated with the subcontext. The CWD acquires the least burdened TPC 340 for non-exclusive use by that subcontext.Upon acquiring the TPC 340, the CWD assigns it a virtual TPC identifier (ID) and makes this virtual TPC ID available to the physical TPC 340 to which the new CTA is launched. The physical TPC 340 then uses this virtual TPC ID when performing local memory address calculations for the new CTA. Subsequent CTAs belonging to different subcontexts will have different virtual TPC IDs. Generally, the virtual TPC ID is used for CTAs associated with a specific subcontext. For example, a physical TPC 340 could have up to 64 virtual TPC IDs simultaneously—one for each of the 64 subcontexts. Threads within CTAs from a particular subcontext on a TPC 340 use that subcontext's virtual TPC ID for that TPC 340 to perform local memory address calculations.

[0044] In certain cases, the CWD may determine that the lowest-value TPC 340 has already been acquired, or that the subcontext initiating the new CTA no longer has any TPC balances. In such cases, the CWD simply initiates the CTA associated with the subcontext within the current set of acquired TPC 340s.

[0045] In general, the CWD is able to make the most efficient load balancing decisions when a subcontext initiating a new CTA has available TPC credits. Therefore, various embodiments of the present invention provide techniques for releasing TPCs 340 and returning TPC credits whenever possible. In various embodiments, as the TPCs 340 execute their respective CTAs, the CWD maintains a count of the number of TPCs 340 executing CTAs for each subcontext. When a TPC 340 has completed executing the CTAs for a given subcontext, the CWD releases the virtual TPC ID associated with that particular TPC 340 back into the pool of available IDs for the subcontext and cancels the allocation of the corresponding TPC credit to the subcontext by incrementing the credit counter associated with the subcontext.

[0046] The TPC acquisition and release mechanism described above dynamically adjusts the resources allocated to each subcontext during operation, improving overall load balancing and GPU execution compared to state-of-the-art methods. This TPC acquisition and release mechanism will now be described in more detail.

[0047] Fig. 4 is a more detailed view of the task / work unit of Fig. 2 according to the various embodiments of the present invention. As shown, the task / work unit 207 includes, without limitation, a scheduler 410 and a compute work distributor (CWD) 420. As also shown, the CWD 420 includes a load balancer 422, N TPC resource trackers (TRTs) 425(0), 425(1), ... 425(N-1), a TPC release table 430, a local memory (LMEM) block index table 432, credit counters 434, a task table 436, and a priority-sorted task table 438. Each of the TRTs 425(0), 425(1), ... 425(N-1) communicates with a corresponding TPC 340(0), 340(1), ... 340(N-1).

[0048] Scheduler 410 receives tasks from preprocessor 212 for various processes running on CPU 102. Each process running on the CPU issues such tasks, with the tasks issued for a given process being directed to one or more subcontexts. Each task corresponds to a group of CTAs to be started for the corresponding subcontext. In general, each task corresponds to a TMD, and each TMD corresponds to a task. Scheduler 410 sends tasks to CWD 420. CWD 420, in turn, maintains a task table 436, which contains a separate task list for each subcontext. Task table 436 contains all tasks that have at least one CTA yet to be started or at least one CTA currently being executed (referred to herein as an in-flight CTA).Scheduler 410 maintains a priority-sorted task table 438, which contains all tasks that have at least one CTA yet to be started. Tasks in priority-sorted task table 438 are sorted by arrival time at CWD 420, by a specified priority value, or by a combination of arrival time and priority value. For example, tasks in priority-sorted task table 438 could first be sorted by their specified priority values. Then, each group of tasks with the same specified priority value could be sorted by arrival time. When a task is received from the scheduler, it is placed in task table 436. The task is also placed in the appropriate position in priority-sorted task table 438 according to its arrival time and / or specified priority value.Once all CTAs for a specific task have been started, the task is removed from the priority-sorted task table 438. However, the specific task remains in task table 436 as long as one or more CTAs are in-flight CTAs. After all CTAs for the specific task have been completed, no in-flight CTAs remain for that specific task. The specific task is then removed from task table 436.

[0049] The processes running on CPU 102 continue to send new tasks for various subcontexts to Scheduler 410 via Preprocessor 212. Scheduler 410 then sends these tasks to CWD 420. CWD 420, in turn, adds these new tasks to Task Table 436 and Priority-Sorted Task Table 438. Each task issued by a process contains one or more Cooperative Thread Arrays (CTAs) to be started for execution on TPCs 340. Generally, a given task can contain as few as one CTA or up to several thousand CTAs. During operation, CWD 420 selects a task from Priority-Sorted Task Table 438 and assigns the task to one or more TRTs 425. Each of the one or more TRTs 425 then communicates with the corresponding TPC 340 to determine the number of free slots on each of the corresponding TPCs 340.Then, in each cycle, the load controller 422 selects the TRT 425 with the largest number of free slots, and the selected TRT 425 initiates a CTA from the task to the TPC 340 corresponding to that TRT 425. The TRTs 425 track free slots as CTAs are initiated and completed, allowing the load controller 422 to select a TRT 425 for the next CTA initiation.

[0050] In various embodiments, the PPU 202 can be configured to execute CTAs for any technically possible number of subcontexts on any technically possible number of TPCs 340. For example, the PPU 202 could be configured to execute CTAs for up to 64 subcontexts on up to 42 TPCs 340. The 42 TPCs 340 could be distributed across 7 GPCs 408, with each GPC 408 containing 6 TPCs 340. Accordingly, the CWD 420 would contain 42 TRTs 425. In another example, the PPU 202 could be configured to execute CTAs for up to 16 subcontexts on up to 14 TPCs 340. The 14 TPCs 340 could be distributed across 7 GPCs 408, with each GPC 408 containing 2 TPCs 340. Accordingly, the CWD 420 would contain 14 TRTs 425.

[0051] Here too, the CWD 420 contains, without restriction, a load balancer 422 and several TPC resource trackers (TRTs) 425(0), 425(1), ... 425(N-1). During each clock cycle of the PPU 202, the load balancer 422 in the CWD 420 selects a task from the priority-sorted task table 438 that contains at least one CTA to be started on one of the TPCs 340(0), 340(1), ... 340(N-1). Based on the TPC release table 430, the load balancer 422 determines the TPCs 340 on which the CTAs for the selected task are released for execution. In general, the load balancer 422 selects the task with the highest priority that is suitable for execution on one or more TPCs 340. The load controller 422 then assigns the task to all available TRTs 425. Each available TRT 425 sends a request to the corresponding TPC 340, requesting the number of available slots.Each available TRT 425 receives a message from the corresponding TPC 340, indicating the number of free slots available for executing CTAs on that TPC 340. Each available TRT 425 then sends a message to the load balancer 422, referred to herein as the "resource offer," containing the number of free execution slots. For example, if each TPC 340 has four execution slots, a TRT 425 for a TPC 340 that is not currently executing any CTAs would send a value of four. A TRT 425 for a TPC 340 that is currently executing a CTA would send a value of three, and so on. Each available TRT 425, which is equivalent to a TPC 340 with available slots, sends a resource offer to the load balancer 422, which includes the number of available CTA execution slots.

[0052] Upon receiving the resource offers, the load balancer 422 selects a TPC 340 to execute the current CTA based on the resource offers from the TRTs 425. Specifically, the load balancer 422 selects the TPC 340 with the highest number of available CTA execution slots to execute the current CTA. Generally, the TPC 340 with the highest number of available CTA execution slots has a processing load that is lower than the processor load assigned to the other TPCs 340s. If more than one TPC 340 has the same number of available slots, the load balancer 422 can select any of the TPCs 340s with the highest number of available slots. The load balancer 422 decrements the credit counter 434 for the subcontext according to the task. The load distributor 422 then cancels the assignment of the task to the available TRTs 425 that the load distributor 422 has not selected.The selected TRT 425 then sends CTAs for the task to the corresponding TPC 340.

[0053] In general, a task can be assigned to a particular TRT 425 if the given TRT 425 is not currently assigned to any other task and the subcontext for the task has at least one credit, as reflected in the corresponding credit counter 434. As described in more detail herein, the TPC release table 430 identifies, for each subcontext, which TPCs 340 are available to execute CTAs for that particular subcontext. As also described in more detail herein, the CWD 420 also maintains the LMEM block index table 432, also referred to herein as the Local Memory (LMEM) block allocation table, which identifies the memory blocks to be used as local memory for each subcontext and each TPC 340.

[0054] Each of the TRTs 425 performs various operations to manage the execution of CTAs on the corresponding TPCs 340. Each TRT 425 maintains a count of the total number of execution slots on the corresponding TPC 340. Likewise, each TRT 425 maintains a count of the number of execution slots on the corresponding TPC 340 that are currently executing CTAs. The difference between these two counts is the number of execution slots available for executing incoming CTAs. In some embodiments, tasks whose CTAs consume different amounts of TPC resources may have different numbers of execution slots.

[0055] When a TRT 425 prepares a CTA to start on the corresponding TPC 340, the TRT 425 prepares a startup package containing the CTA. The startup package also includes the subcontext count for the subcontext corresponding to the CTA. The PPU 202 maintains a separate page directory base address for each subcontext, allowing each subcontext to have a separate virtual memory address space. The TRT 425 then sends the prepared startup package to the TPC 340 for execution. The TRT 425 further determines whether a local memory block has been allocated and is valid for the subcontext on that particular TPC 340. If not, a local memory block is allocated to the TRT 425, and the LMEM block index table 432 is updated accordingly. Thus, the startup package contains local memory allocation information for the CTA, enabling the TPC 340 to locate the allocated local memory blocks.In some embodiments, the virtual TPC ID can be the same as a local memory block index, where the local memory block index selects an area within memory that is allocated for use by a subcontext as local memory.

[0056] When a TPC 340 receives a boot packet from a TRT 425, the TPC 340 loads the CTA into the boot packet and prepares the CTA for execution. The TPC 340 retrieves the subcontext count from the boot packet and instructs the TPC 340 to access the state data corresponding to the subcontext when executing the CTA. Each TPC 340 maintains separate state data for each subcontext. For example, if the PPU 202 supports 64 subcontexts, each TPC 340 maintains 64 instances of state data, one instance for each of the 64 subcontexts. In another example, if the PPU 202 supports 16 subcontexts, each TPC 340 maintains 16 instances of state data, one for each of the 16 subcontexts. The TPC 340 uses the subcontext number to retrieve the page directory base address and access the page table corresponding to the virtual address space that corresponds to the subcontext.Finally, the TPC 340 retrieves the local memory allocation information from the startup package and instructs the TPC 340 to access the corresponding local memory blocks.

[0057] In some embodiments, the mapping between a subcontext number and a page directory base can be maintained in a memory management unit such as the MMU 320, with the memory management unit being responsible for mapping virtual address to physical address.

[0058] The process for initiating CTAs for execution on TPCs 340 on behalf of various processes running on CPU 102 will now be described in more detail.

[0059] In operation, multiple processes running on CPU 102 have various tasks that PPU 202 is tasked with performing. Assuming that the tasks assigned to a particular CPU process do not consume all of PPU 202's resources, PPU 202's resources are not fully utilized by a single process running on it. Furthermore, PPU 202 executes in one context at a time. As a result, all tasks for all processes running on PPU 202 at any given time share certain functions. These shared functions include context scheduling and fault isolation. However, as described in more detail herein, PPU 202 accommodates multiple subcontexts within a given context, with each subcontext having a unique virtual address space and unique state data. Consequently, subcontexts within a context have separate virtual address spaces and separate state data.However, the subcontexts within a context are planned jointly and undergo context changes together, as described in more detail below.

[0060] To accommodate separate virtual address spaces for each subcontext, the CWD 420 maintains subcontext numbers, each corresponding to a page directory base address, with each page directory base address pointing to a specific page table. To accommodate unique state data for each subcontext, the entire PPU 202 maintains a separate instance of state data for each subcontext. Specifically, each of the TPCs 340 maintains a separate instance of state data for each subcontext. When a CTA is initiated for execution on a particular TPC 340, a TRT 425 includes the corresponding subcontext number in the startup packet sent to the TPC 340. In response, when executing the CTA, the TPC 340 accesses the correct instance of state data according to the subcontext number.

[0061] In some embodiments, even if each subcontext has a different virtual address space, two or more subcontexts could be set to the same virtual address space by setting the page directory base address for the two or more subcontexts to the same address. In this way, the two or more subcontexts share the same virtual address space but have separate state data. For example, certain subcontexts relating to graphics functions and certain other subcontexts relating to computational functions could run with the same virtual address space but with separate state data.

[0062] The load balancer 422 in the CWD 420 maintains a TPC share table 430 and an LMEM block index table 432. During operation, the load balancer 422, or an operating system or hypervisor running on the CPU 102, initializes or updates the TPC share table 430. The load balancer 422, the operating system, or the hypervisor can initialize or update the TPC share table 430 when the PPU 202 is initialized, a context is initialized, or the PPU 202 is otherwise in an idle state. Additionally, the load balancer 422, the operating system, or the hypervisor can update cells within the TPC share table 430 for specific subcontext rows when those subcontexts are currently idle. In some embodiments, subcontext lines for subcontexts that are not currently idle may not be updated to simplify the design by preventing read / write conflicts.Furthermore, the load balancer 422, the operating system, or the hypervisor can update cells within the TPC release table 430 for certain subcontext columns when these TPCs 340 are currently idle. Again, in some embodiments, subcontext columns for TPCs 340 that are not currently idle may not be updated to simplify the design by preventing read / write conflicts.

[0063] The TPC share table 430 contains one row per subcontext and one column per TPC 340. A value of '1' in the cell of the TPC share table 430 indicates that the subcontext corresponding to the row in which the cell is located may be executed on the TPC 340 corresponding to the column in which the cell is located. A value of '0' in the cell of the TPC share table 430 indicates that the subcontext corresponding to the row in which the cell is located may not be executed on the TPC 340 corresponding to the column in which the cell is located.

[0064] The LMEM block index table 432 also contains one row per subcontext and one column per TPC 340. Each cell in the LMEM block index table 432 contains a virtual TPC identifier that identifies the virtual TPC assigned to the specific subcontext, corresponding to the row of the LMEM block index table 432. Each cell in the LMEM block index table 432 further identifies the physical TPC 340 that corresponds to the column of the LMEM block index table 432 that executes the virtual TPC for the subcontext. Additionally, the virtual TPC identifier identifies a corresponding local memory block in the local memory that is bound to the physical TPC 340, corresponding to the column of the LMEM block index table 432 for the subcontext, corresponding to the row of the LMEM block index table 432. In this way, the LMEM block index table 432 identifies all currently active TPCs 340 for all subcontexts and the positions for the corresponding local memory blocks.Note that a local memory block is generally not used concurrently by two different TPCs 340 to prevent two TPCs from writing to the same memory location, which would result in memory corruption. However, a local memory block allocated to a particular TPC 340 can be used by any SM 310 within that TPC 340. Furthermore, LMEM blocks are allocated contiguously, with the virtual TPC identifiers in the LMEM block index table 432 ranging from 0 to the maximum credit count (as initially stored in the credit counters 434) minus 1. Finally, when a TPC 340 has completed all CTAs for a given subcontext and becomes idle, the corresponding local memory blocks are released for reassignment. Additionally, the credit counter 434 for the subcontext is incremented by one.If the subcontext initiates a CTA start to the same TPC 340 before the local memory block is freed, the CTA will execute on the TPC 340 without the need to reallocate the local memory block.

[0065] The PPU 202 can operate in three different modes: static TPC partitioning, dynamic TPC partitioning, and hybrid static / dynamic TPC partitioning. Static TPC partitioning, dynamic TPC partitioning, and hybrid static / dynamic TPC partitioning are also referred to herein as static resource allocation, dynamic resource allocation, and hybrid static / dynamic allocation, respectively. With static TPC partitioning, the operating system or hypervisor initializes and updates both the TPC share table 430 and the LMEM block index table 432. In some TPC partitioning configurations, the CWD 420 can initialize and update the LMEM block index table 432. In some configurations, the number of LMEM block indexes can be equal to the number of TPCs 340, so that the mapping is not dynamically changed. Each subcontext is statically assigned to run on specific TPCs 340.Furthermore, each TPC 340 is assigned specific virtual TPC identifiers for each subcontext, so that each subcontext has statically allocated local memory blocks for each TPC 340. In some implementations of static TPC partitioning, the balance count is not used because virtual TPC identifiers are not modified.

[0066] With dynamic TPC partitioning, the operating system or hypervisor initializes and updates the TPC share table 430, while the CWD 420 dynamically updates the LMEM block index table 432, based on incoming tasks and the current load on the TPCs 340. Each subcontext receives an initial credit count indicating the maximum number of TPCs 340 that the subcontext can access at any given time. The load balancer 422 in the CWD 420 loads this initial credit count for each subcontext into the credit counter 434. For example, if the initial credit count is three, each subcontext can execute CTAs on up to three TPCs 340 concurrently. A particular subcontext could execute on any three available TPCs 340, which are identified by the TPC share table 430.After the start of a CTA for a specific TPC 340, the load distributor 422 decrements the credit counter 434 for the corresponding subcontext.

[0067] In hybrid static-dynamic TPC partitioning, each subcontext is forced to execute on a specific subset of the total number of TPCs 340. Within the specified subset of TPCs 340, the subcontext can execute CTAs on any of the TPCs 340 contained in that subset and allocate any local memory blocks corresponding to the same subset of TPCs 340, subject to a maximum credit count for the given subcontext.

[0068] With static TPC partitioning, dynamic TPC partitioning via a balance counter, and hybrid static-dynamic TPC partitioning via a balance counter, the number of TPCs (340) that each subcontext can use simultaneously can be limited. Accordingly, the number of local memory blocks that can be allocated to each subcontext at one time can also be limited.

[0069] In some embodiments, the PPU 202 can operate in a partial dynamic mode. In such embodiments, each subcontext can be assigned to a portion of the total number of TPCs 340. Within the assigned portion of the TPCs 340, however, each subcontext can execute on any specific TPCs 340 up to the initial credit count stored in the credit counter 434. For example, if a PPU 202 supports 64 subcontexts and 42 TPCs 340, the TPC release table 430 could specify that each of the subcontexts 0 to 31 could execute on any TPCs from 0 to 20, and each of the subcontexts 32 to 63 could execute on all TPCs from 21 to 42. If each subcontext receives an initial credit count of 3, each of the subcontexts 0 to 31 could execute on any 3 TPCs from 0 to 20. Likewise, each of the subcontexts 32 to 63 could execute on any 3 TPCs from 21 to 42.

[0070] Generally, CTAs are initiated on the available TPC 340, which currently has the lowest workload, within the limits of the number of credits the subcontext has and the static TPC release table 430, which identifies the available set of TPCs 340 for each context. This method can offer improved fairness in scheduling and allocation compared to previous methods.

[0071] As a specific example, consider a PPU 202 configured to receive tasks for two subcontexts running on four TPCs 340. The priority-sorted task list 438 contains three tasks: a first task for subcontext 0, a first task for subcontext 1, and a second task for subcontext 0, in descending order of priority. Each subcontext receives an initial credit counter of three. The load balancer 422, or other hardware and / or software on the CPU 102, initializes the credit counter 434 for each subcontext with a value of three, indicating that each subcontext can simultaneously execute CTAs on up to three of the four TPCs. Upon initialization, the scheduler 410 receives the three tasks and sends them to the CWD 420. The load balancer 422 in the CWD 420 receives the tasks, with each task corresponding to a set of one or more CTAs.The load balancer 422 stores the received tasks in task table 436 and priority-sorted task table 438. Since there are no credits available in subcontext 0 and none of the TRTs 425 currently have a task, the load balancer 422 assigns the first received task for subcontext 0 to all TRTs 425 as a qualifying task. Each TRT 425 that has a qualifying task communicates with the corresponding TPCs 340 and receives a number of free slots. Assuming that all TPCs 340 have four free slots, each TRT 425 sends a resource offer to the load balancer 422 indicating that four execution slots are available. Upon receiving the resource offers, the load controller 422 selects one of the TRTs 425, based on these offers, to initiate the CTA to the corresponding TPC 340. The selected TRT 425 prepares a start packet for the CTA and sends the start packet to the TPC 340.If no local memory block has yet been allocated, which is the case at this point in the example, the TRT 425 allocates a local memory block for the CTA running on the TPC 340. The load balancer 422 decrements the credit counter 434 for subcontext 0 from 3 to 2. This process continues to select the next TRT 425 to start a CTA, causing the load balancer 422 to decrement the credit counter 434 for subcontext 0 from 2 to 1. This process continues with the selection of the next TRT 425 to start the CTA, causing the load balancer 422 to decrement the credit counter 434 for subcontext 0 from 1 to 0. At this point, the load distributor 422 may not be using the fourth TRT 425 for subcontext 0, so the assignment of the task to the fourth TRT 425 is removed, making the TRT 425 available for a task from a different subcontext.

[0072] The load balancer 422 now selects the first task for subcontext 1 from the priority-sorted task table 438. Since subcontext 0 has no remaining credits and there is one available TRT 425, the scheduler selects the task for subcontext 1 and assigns it to the remaining TRT 425. This TRT 425 then starts CTAs on the corresponding TPC 340, and the load balancer 340 decrements the credit counter 434 for subcontext 1 from 3 to 2. If subcontext 0 and subcontext 1 are non-exclusive, then, once all CTAs for the first task have been started and the assignment of the first task to the three TRTs 425 has been removed, the task for subcontext 1 can use up to two additional TRTs 425 besides the one currently assigned to the task.If subcontext 0 and subcontext 1 are exclusive, then subcontext 1 waits until at least one TPC 340 has completed the execution of all CTAs for subcontext 0, and can then have the task of subcontext 1 assigned to one of the first three TRTs 425 that have completed the execution of all CTAs for subcontext 0.

[0073] The load balancer 422 now selects the second task for subcontext 0 from the priority-sorted task table 438. Subcontext 0 has zero credits, so the second task for subcontext 0 waits until all CTAs for the first task for subcontext 0 have started, even if there is an available TRT 425. Once all CTAs for the first task for subcontext 0 have started, the task is released from the assignment to the TRTs 425, making them available for a new task.

[0074] When a TPC 340 completes the execution of CTAs for a specific task, the behavior of the load balancer 422, the TRTs 425, and the TPCs 340 depends on the current status of all tasks in the priority-sorted task table 438. Continuing the example above, consider a situation where three of the four TPCs 340 are executing CTAs for the first task for subcontext 0, while one of the four TPCs 340 is executing CTAs for the first task for subcontext 1. The current values ​​of the credit counters 434 for subcontext 0 and subcontext 1 are 0 and 2, respectively. The second TPC 340 of the three TPCs 340, the one executing CTAs for the first task for subcontext 0, terminates the execution of all CTAs.

[0075] In the first scenario, the first task for subcontext 0 has to initiate additional CTAs. In this first scenario, the TRT 425, which corresponds to the second TPC 340, initiates additional CTAs for the first task for subcontext 0 to the second TPC 340. In the second scenario, all CTAs for the first task for subcontext 0 have been initiated. Therefore, the first task for subcontext 0 has been removed from the priority-sorted task table 438. In this second scenario, the first task for subcontext 1 has to initiate additional CTAs. The load controller 422 increments the credit counter 435 for subcontext 0 from 0 to 1. The load controller 422 assigns the first task for subcontext 1 to all available TRTs 425. The load controller 422 selects one of the available TRTs 425. The selected TRT 425 can be the TRT 425 assigned to the second TPC 340, or alternatively, the TRT 425 assigned to any other TPC 340.The load controller 422 decrements the credit counter 435 for subcontext 1 from 2 to 1. The selected TRT 425 then issues CTAs for the first task for subcontext 1 to the corresponding TPC 340. Note that if the first task for subcontext 1 has no unstarted CTAs, the load controller 422 removes the assignment of the first task for subcontext 1 to the unselected TRTs 425. Otherwise, if the first task of subcontext 1 still has unstarted CTAs, the assignment of the first task for subcontext 1 to any TRT 425 is not removed until subcontext 1 subsequently has no remaining credits. In a third scenario, neither the first task for subcontext 0 nor the first task for subcontext 1 has to start any additional CTAs. Therefore, the first task for subcontext 0 and the first task for subcontext 1 have been removed from the priority-sorted task table 438.In this third scenario, the TRT 425, which corresponds to the second TPC 340, initiates CTAs for the second task for subcontext 0 to the second TPC 340. In this way, tasks and associated subcontexts migrate over time between the different TRTs 425 based on the dynamic load conditions of the associated TPCs 340.

[0076] In some embodiments, certain subcontexts may be prohibited from executing simultaneously on the same TPC 340. In some embodiments, the CWD 420 may maintain an exclusive allocation table with one bit per subcontext to indicate whether that subcontext is exclusive. If a particular subcontext is marked as exclusive, no other subcontext may execute on a given TPC 340 when the exclusive subcontext is executing on the same TPC 340. In some embodiments, the CWD 420 may maintain an exclusive allocation table that indicates whether certain pairs of subcontexts are exclusive. In such embodiments, the rows and columns of the exclusivity tables represent subcontexts. A '1' in a particular cell may indicate that the two corresponding subcontexts are pairwise exclusive.Two subcontexts that are pairwise exclusive may be prohibited from executing CTAs on the same TPC 340 simultaneously. If two subcontexts are pairwise exclusive with respect to each other, and the first subcontext is executing CTAs on a particular TPC 340, then the second subcontext waits until the CTAs for the first subcontext have finished executing before it starts CTAs on the same TPC 340.

[0077] Note that the system shown here is for illustrative purposes only and that deviations and modifications are possible. For example, in the described embodiments, a value of '1' means that certain elements are enabled, permitted, allocated, or assigned, while a value of '0' indicates that certain elements are disabled, prohibited, not allocated, or not assigned. Any other suitable values ​​may be used within the scope of the invention.

[0078] Fig. Figures 5A-5B illustrate a TPC release table 500 and an LMEM block index table 550 for static TPC partitioning according to various embodiments of the present invention. The TPC release table 500 and the LMEM block index table 550 operate essentially like the TPC release table 430 and the LMEM block index table 432 of Fig. 4, except as further described below. As shown, the TPC share table 500 contains subcontext rows 510 and TPC columns 520. Each cell in the TPC share table 500 is set to '1', indicating that any subcontext can execute on any TPC 340. The LMEM block index table 550 also contains subcontext rows 560 and TPC columns 570. Each cell in the LMEM block index table 550 identifies a virtual TPC identifier for a particular subcontext running on a particular TPC 340. The virtual TPC identifier contained in each cell of the LMEM block index table 550 further identifies a local memory block location for the subcontext when it executes CTAs on the corresponding TPC 340.

[0079] Fig. Figures 6A-6B illustrate a TPC release table 600 and an LMEM block index table 650 for static TPC partitioning according to other various embodiments of the present invention. The TPC release table 600 and the LMEM block index table 650 operate essentially like the TPC release table 430 and the LMEM block index table 432 of Fig. 4, except as described in more detail below. As shown, the TPC release table contains 600 subcontext rows 610 and TPC columns 620. Subcontext row 610 for subcontext 0 contains two cells set to '1', corresponding to TPC 0 and TPC 1. The remaining cells in subcontext row 610 for subcontext 0 are set to '0'. Consequently, subcontext 0 can execute CTAs on TPC 0 and TPC 1, but not on any other TPCs 340. Similarly, subcontext row 610 for subcontext 0 contains two cells set to '1', corresponding to TPC 1 and TPC 2. The remaining cells in subcontext row 610 for subcontext 1 are set to '0'. Consequently, subcontext 0 can execute CTAs on TPC 1 and TPC 2, but not on any other TPCs 340. The cells for the remaining subcontext rows 610 of the TPC release table 600 are set similarly.Therefore, each subcontext can execute on up to two specified TPCs 340, as indicated in the TPC release table 600, and each TPC is limited to executing CTAs for only two subcontexts.

[0080] Subcontext row 660 for subcontext 0 in the LMEM block index table 650 is set to '0' and '1' for TPC 0 and TPC 1, respectively. These values ​​indicate that virtual TPC identifier 0 for subcontext 0 corresponds to physical TPC 0, and virtual TPC identifier 1 for subcontext 0 corresponds to physical TPC 1. These virtual TPC identifiers also mark a local memory block location for subcontext 0 when it executes CTAs on the corresponding TPC 340. The remaining cells in subcontext row 660 for subcontext 0 are set to X (irrelevant), since subcontext 0 is not allowed to execute on any of the other TPCs 340.

[0081] Subcontext row 660 for subcontext 1 in the LMEM block index table 650 is set to '0' and '1' for TPC 1 and TPC 2, respectively. These values ​​indicate that virtual TPC identifier 0 for subcontext 1 corresponds to physical TPC 1, and virtual TPC identifier 1 for subcontext 1 corresponds to physical TPC 2. These virtual TPC identifiers also mark a local memory block location for subcontext 1 when it executes CTAs on the corresponding TPC 340. The remaining cells in subcontext row 660 for subcontext 1 are set to X (ignore), since subcontext 1 is not allowed to execute on any of the other TPCs 340. The cells for the remaining subcontext rows 660 of the LMEM block index table 650 are set similarly.

[0082] Fig. Figure 7 illustrates a TPC release table 700 for dynamic TPC partitioning according to various embodiments of the present invention. The TPC release table 700 essentially works like the TPC release table 430 in Fig. 4, except as described in more detail below. As shown, the TPC release table 700 contains subcontext rows 710 and TPC columns 720. Each cell in the TPC release table 700 is set to '1', indicating that any subcontext can execute on any TPC 340, subject to credit count restrictions. The LMEM block index table (not shown) is dynamically set by the load balancer 422 based on the current credit count stored in the credit counter 434 for each subcontext and the current execution load on each TPC 340, as described in more detail herein.

[0083] Fig. Figure 8 illustrates a TPC release table 800 for dynamic TPC partitioning according to other various embodiments of the present invention. The TPC release table 800 essentially works like the TPC release table 430 in Fig. 4, except as further described below. As shown, the TPC release table contains 800 subcontext rows 810 and TPC columns 820. Each subcontext row 810 for subcontexts 0 through 7 contains a '1' in the cells for TPCs 0 through 6 and a '0' in the remaining cells. Consequently, each of subcontexts 0 through 7 can execute CTAs on one or more of TPCs 0 through 6, subject to balance counting restrictions. Similarly, each subcontext row 810 for subcontexts 8 through 15 contains a '1' in the cells for TPCs 7 through 13 and a '0' in the remaining cells. Consequently, each of subcontexts 8 through 15 can execute CTAs on one or more of TPCs 7 through 13, subject to balance counting restrictions.The LMEM block index table (not shown) is dynamically set by the load distributor 422 based on the current credit count, which is stored in the credit counter 434 for each subcontext, and the current execution load on each TPC 340, as described in more detail herein.

[0084] Fig. Figures 9A-9C show a flowchart of process steps for allocating execution resources within a processor according to various embodiments of the present invention. Although the process steps are related to the systems of Fig. As described in Figures 1-8, the person skilled in the art will recognize that any system configured to carry out the process steps in any order falls within the scope of protection of the present invention.

[0085] As shown, a procedure 900 begins in step 902, in which the load balancer 422 in the CWD 420, an operating system (OS) running on CPU 102, or a hypervisor running on CPU 102 initializes a TPC share table that identifies the TPCs 340 that are allowed to execute CTAs for each subcontext. If static TPC partitioning is used, the load balancer 422, the operating system, or the hypervisor also initializes the local memory block index table (LMEM), which identifies the virtual TPC identifiers and local memory block locations for each subcontext when CTAs are executed on a TPC 340. If dynamic or hybrid TPC partitioning is used, the load balancer 422 initializes and updates the LMEM block index table. In step 904, the load balancer 422, the operating system (OS) or the hypervisor initializes the initial balance count in the CWD 420 for each subcontext.The initial credit count represents the maximum number of TPCs that can execute CTAs for a subcontext at any given time. For example, if the initial credit count is three, each subcontext can have up to three TPCs executing one or more CTAs at any given time. If each TPC can accommodate four CTAs, up to twelve CTAs could be executed concurrently at any given time.

[0086] In step 906, the scheduler 410 in task / work unit 207 receives tasks from various processes running on CPU 102, each task corresponding to one or more CTAs. Each process issues such tasks to one or more subcontexts running in PPU 202. In step 908, the scheduler 410 sends the received tasks to the load balancer 422 in CWD 420. In step 910, the load balancer 422 places each received task in an appropriate position on task table 436 and priority-sorted task table 438, according to the arrival time and / or the selected priority value. In step 912, the load balancer 422 selects a task from priority-sorted task table 438. The scheduler 410 can select any task for a subcontext that does not have a current balance of zero.In general, the load distributor 422 selects a task that also has the highest priority. In some embodiments, each TRT 425 can execute CTAs for different tasks, and each task can correspond to more than one CTA. The load distributor 422 can consider resource offers received from each TRT 425 when a TPC 340 is selected to execute CTAs for a particular task, as described in more detail herein.

[0087] In step 914, the load balancer 422 assigns the selected task to all available TRTs 425 corresponding to TPCs 340 that are authorized to execute the task's CTAs based on the TPC authorization table. In step 916, each TRT 425 sends a resource offer to the load balancer 422, with the resource offer applying to a corresponding TPC 340. The resource offer of a TRT 425 indicates the number of available CTA execution slots on the corresponding TPC 340. In step 918, the load balancer 422 receives resource offers from each of the available TRTs 425, with each resource offer from a TRT 425 containing the number of available CTA execution slots on the corresponding TPC 340. In step 920, the load distributor 422 selects the TRT 425 according to the TPC 340 that is least loaded, that is, the TPC 340 with the highest number of free CTA execution slots.If multiple TPCs 340 have the same number of available CTA execution slots, the load balancer 422 selects any TPC 340 from among those with the highest number of available CTA execution slots. In step 922, the load balancer 422 removes the task assignment to the unselected TRTs 425.

[0088] In step 924, the load controller 422 determines whether the TPC 340 is executing any CTAs for the selected TRT 425 in the subcontext associated with the task. If the TPC 340 is not executing any CTAs for the subcontext for the selected TRT 425, procedure 900 proceeds to step 926, in which the load controller 422 decrements the credit counter 434 of the subcontext corresponding to the task. The procedure then proceeds to step 928, which is described below. If, in step 924, the TPC 340 is executing CTAs for the subcontext for the selected TRT 425, procedure 900 proceeds to step 928, in which the load controller 422 sends a start signal to the TRT 425 corresponding to the selected TPC 340.

[0089] In step 930, the TRT 425 in the CWD 420 prepares a startup package containing the CTA. This package also includes the subcontext number, used to identify a corresponding page directory base address for the subcontext, and identifies the subcontext state data for the CTA. The startup package also contains local memory allocation information for the CTA, enabling the TPC 340 to locate the allocated local memory blocks.

[0090] In step 932, the TRT 425 sends the start packet to the corresponding TPC 340. In step 934, the TPC 340 executes the CTA. In step 936, the load controller 422 determines whether the TPC 340 has completed the execution of the last CTA executed on the TPC 340 for the subcontext. If the TPC 340 has completed the execution of the last CTA executed on the TPC 340 for the subcontext, procedure 900 continues with step 938, in which the load controller 422 increments the corresponding credit counter 434 for the specified subcontext by one. Procedure 900 then terminates. If the TPC 340 has not completed the execution of the last CTA executed on the TPC 340 for the subcontext in step 934, procedure 900 terminates.

[0091] Fig. Figures 10A-10B show a flowchart of process steps for allocating local memory resources within a processor according to various embodiments of the present invention. Although the process steps are related to the systems of Fig. As described in Figures 1-8, the person skilled in the art will recognize that any system configured to carry out the process steps in any order falls within the scope of protection of the present invention.

[0092] As shown, a procedure 1000 begins in step 1002, in which the Local Memory Element (LMEM) block index table is initialized. The LMEM block index table identifies the virtual TPC identifiers and local memory block locations for each subcontext when CTAs are executed on a TPC 340. If static TPC partitioning is used, the operating system or hypervisor initializes the LMEM block index table. If dynamic or hybrid TPC partitioning is used, the CWD 420 initializes and updates the LMEM block index table. In step 1004, the load balancer 422 in the CWD 420 determines that a TRT 425 is starting a CTA to execute on a TPC 340. Starting a CTA is independent of the allocation of a local memory block. A CTA can be started to execute before, at the same time as, or after the allocation of the corresponding local memory block to the TPC.In general, the TPC 340 to which the CTA is started is allocated a local memory block, where the local memory block is for the subcontext corresponding to the CTA. Typically, the local memory block is allocated to the TPC 340, not the CTA itself. The CTA does not begin execution until the appropriate local memory block is allocated. In step 1006, the load balancer 422 determines whether a local memory block has already been allocated to the TPC to which the CTA is started. If a local memory block has already been allocated, procedure 1000 continues to step 1022, as described below. If no local memory block has yet been allocated to the CTA that was just started, procedure 1000 continues to step 1008, in which the load balancer 422 determines whether static TPC partitioning is being used.

[0093] If static TPC partitioning is used, procedure 1000 continues with step 1010, in which the load balancer 422 retrieves the local memory block index from the LMEM block index table. The load balancer 422 identifies the position for the cell in the LMEM block index table that corresponds to the subcontext for the CTA just started and the TPC 340 selected to execute the CTA. The load balancer 422 retrieves the local memory block index from the identified cell in the LMEM block index table. In step 1012, the load balancer 422 allocates a local memory block according to the local memory block index. The load balancer 422 allocates the local memory block from the local memory pool for the subcontext. Once the local memory block is allocated, all CTAs running on all SMs 310 in the corresponding TPC 340 can access memory locations within the local memory block.In step 1014, the load balancer 422 updates the cell in the LMEM block index table to indicate that the local memory block has been allocated. Procedure 1000 then proceeds to step 1022, which is described below.

[0094] Returning to step 1008, if static TPC partitioning is not used in step 1008, then either dynamic TPC partitioning or hybrid static / dynamic TPC partitioning is used. In such cases, procedure 1000 continues with step 1016, in which the load balancer 422 determines the local memory block index for the local memory block to be allocated to the CTA just started. Local memory blocks can be allocated contiguously, randomly, or by any algorithm that does not allow the same local memory block to be used simultaneously on different TPCs 340. For example, if the CTA just started corresponds to a subcontext where local memory blocks have already been allocated for local memory block indices 0, 1, and 2, then the load balancer 422 could allocate the next local memory block for the subcontext using local memory block index 3.If local memory block 1 is subsequently released while local memory blocks 0, 2 and 3 remain allocated, the load balancer 422 can allocate the next local memory block for the subcontext using local memory block index 1.

[0095] In step 1018, the load distributor 422 can, in some embodiments, allocate a local memory block according to the local memory block index. The load distributor 422 allocates the local memory block from the local memory pool. Once the local memory block is allocated, all CTAs running on all SMs 310 in the corresponding TPC 340 can access memory locations within the allocated local memory block. In step 1020, the load distributor 422 updates the cell in the LMEM block index table according to the local memory block with the local memory block index to indicate that the local memory block has been allocated.

[0096] In step 1022, the load balancer 422 determines whether all CTAs accessing the local memory block have completed execution and no longer require access to the local memory block. If all CTAs accessing the local memory block have not yet completed execution, procedure 1000 terminates. If, however, all CTAs accessing the local memory block have completed execution, procedure 1000 proceeds to step 1024, in which, before the allocation of the local memory block is released, the load balancer 422 performs a memory lock operation to ensure that all local memory writes from the TPC 340 have reached the memory coherence point. Furthermore, the memory lock ensures that another TPC 340 accessing the same local memory block does not corrupt the local memory.In particular, the memory lock ensures that when the local memory block is reassigned, there are no pending write operations from previous TPCs that could modify the data in local memory. In step 1026, load balancer 422 releases the local memory block back into the local memory pool. In step 1028, load balancer 422 updates the cell in the LMEM block index table corresponding to the local memory block to indicate that the local memory block is no longer allocated. Procedure 1000 then terminates.

[0097] In summary, various techniques for partitioning execution and memory resources in a parallel processing system for the execution of cooperative thread arrays (CTAs) according to tasks are revealed. Specifically, execution resources in the form of texture processing clusters (TPCs) and memory resources in the form of local memory blocks are allocated to different subcontexts that execute within a single context. Each subcontext maintains a separate virtual address space and separate state data, but shares scheduling and context-switching resources.

[0098] In the first technique, TPCs and local memory blocks are statically allocated to subcontexts. With this first technique, each subcontext is restricted to executing CTAs on specific TPCs and allocating specific local memory blocks. In the second technique, TPCs and local memory blocks are dynamically allocated to subcontexts. With this second technique, each subcontext can execute CTAs on any of the TPCs and allocate any local memory blocks, subject to a maximum credit count for the given subcontext. In the third technique, the properties of static and dynamic resource allocation are combined in a hybrid method. With this third technique, each subcontext is forced to execute on a specific subset of the total number of TPCs.Within the specified subset of TPCs, the subcontext can execute CTAs on any of the TPCs contained in the subset of TPCs and allocate local memory blocks in local memory according to the same subset of TPCs, subject to a maximum credit count for the given subcontext.

[0099] At least one advantage of the disclosed techniques is that execution and local memory resources can be flexibly and efficiently allocated to subcontexts, corresponding to multiple processes within a parallel processing system. This increases the utilization of execution and local memory resources compared to previous methods. Another advantage of the disclosed techniques is that the maximum amount of execution and local memory resources that can be allocated or assigned to a subcontext is selectable and can be limited within the constraints of the number of available TPCs and local memory blocks, thus enabling more subcontexts to execute concurrently. A further advantage of the disclosed techniques is that all subcontexts execute within a single context but maintain separate virtual address spaces and separate state data.This allows TPCs to quickly switch from executing a CTA for one subcontext to executing a CTA for another subcontext without requiring a full context switch.

[0100] Another advantage of the disclosed techniques is that execution and local storage resources are allocated to different subcontexts corresponding to different CPU processes running on a server machine operating in a data center or cloud computing environment that provides scalable computing resources as a service over a network. Various services provided by the server machine can be delivered to end users via a cloud computing infrastructure. Cloud computing generally refers to the provision of scalable computing resources as a service over a network. More formally, cloud computing can be defined as a computing service that abstracts between the computing resource and its underlying technical architecture (e.g.,Cloud computing provides servers, storage, and networks, enabling convenient on-demand network access to a shared pool of configurable computing resources that can be rapidly provisioned and released with minimal administrative overhead or interaction with the service provider. Thus, cloud computing allows a user to access virtual computing resources (such as storage, data, applications, and even entire virtualized computer systems) in the "cloud," regardless of the underlying physical systems (or locations of those systems) used to provide those computing resources.

[0101] 1. In some embodiments, a computer-implemented method for allocating execution resources to groups of threads within a graphics processing unit comprises: receiving a notification from a process to start a first group of threads; determining that a first subcontext associated with the process has at least one processor credit; identifying a first processor contained in a plurality of processors that has a processing load less than or equal to the processor loads allocated to all other processors contained in the plurality of processors; and starting the first group of threads for execution on the first processor.

[0102] 2. The computer-implemented method of Section 1, wherein the first subcontext is contained within a plurality of subcontexts, and each subcontext contained within the plurality of subcontexts is assigned to a different process and a different virtual address space.

[0103] 3. The computer-implemented method of Section 1 or Section 2, wherein the first subcontext is contained within a plurality of subcontexts and each subcontext contained within the plurality of subcontexts is associated with other state data.

[0104] 4. The computer-implemented method of one of Sections 1-3, wherein the first subcontext is contained in a plurality of subcontexts; the first subcontext is associated with a first virtual address space; a second subcontext contained in the plurality of subcontexts is associated with the first virtual address space; and a third subcontext contained in the plurality of subcontexts is associated with a second virtual address space that is different from the first virtual address space.

[0105] 5. The computer-implemented method of one of Sections 1-4, which further comprises, in response to the starting of the first group of threads for execution on the first processor, decrementing a credit counter associated with the first subcontext.

[0106] 6. The computer-implemented procedure of one of Sections 1-5, which further comprises: determining that the first group of threads has completed execution on the first processor; and incrementing the credit counter associated with the first subcontext.

[0107] 7. The computer-implemented method of one of Sections 1-6, wherein identifying a first processor included in the plurality of processors comprises: determining a count of available execution slots for each processor included in the plurality of processors; identifying one or more processors included in the plurality of processors having the highest count of available execution slots; and selecting the first processor from the one or more processors.

[0108] 8. The computer-implemented method of one of Sections 1-7, which further comprises determining that the first subcontext is not exclusive with respect to any subcontext associated with a second group of threads currently running on the first processor.

[0109] 9. The computer-implemented method of one of Sections 1-8, further comprising: retrieving a first record associated with the first subcontext from a first table; and determining from the first record that each processor included in the plurality of processors is available with respect to executing at least one group of threads associated with the first subcontext.

[0110] 10. The computer-implemented method of one of Sections 1-9, wherein the first table is initialized by an operating system or hypervisor before any group of threads is started for execution on any processor included in the plurality of processors.

[0111] 11. The computer-implemented method of one of Sections 1-10, further comprising retrieving a second record associated with the first subcontext from a second table, wherein the second record specifies a different virtual processor identifier for each processor included in the plurality of processors and available with respect to executing at least one set of threads associated with the first subcontext; and identifying a virtual processor identifier for the first processor from the second record; wherein the first processor performs a memory address computation based on the virtual processor identifier.

[0112] 12. In some embodiments, a parallel processing system comprises: a scheduler that sends a plurality of tasks to a workload distributor; and a workload distributor that: selects a task contained in the plurality of tasks from a task list associated with a first subcontext, identifies a first group of threads associated with the task to be started, determines that the first subcontext has at least one processor credit, identifies a first processor contained in a plurality of processors and having a processing load less than or equal to the processing load allocated to all other processors contained in the plurality of processors, and starts the first group of threads for execution on the first processor.

[0113] 13. The parallel processing system of Section 12, wherein the workload distributor further determines that the first processor is available to execute the first group of threads.

[0114] 14. The parallel processing system of Section 12 or Section 13, wherein the workload distributor continues to decrement a credit counter associated with the first subcontext in response to the starting of the first group of threads for execution on the first processor.

[0115] 15. The parallel processing system of one of Sections 12-14, wherein the workload distributor further: determines that the first group of threads has completed execution on the first processor; and increments the credit counter associated with the first subcontext.

[0116] 16. The parallel processing system of one of Sections 12-15, wherein identifying a first processor included in the plurality of processors comprises: determining a count of available execution slots for each processor included in the plurality of processors; identifying one or more processors included in the plurality of processors that have the highest count of available execution slots; and selecting the first processor from the one or more processors.

[0117] 17. The parallel processing system of one of Sections 12-16, wherein the workload distributor further: creates a startup packet for the first processor containing the first group of threads; and sends the startup packet to the first processor.

[0118] 18. The parallel processing system of one of Sections 12-17, wherein the start package still contains a number corresponding to the first subcontext.

[0119] 19. The parallel processing system of one of Sections 12-18, wherein the parallel processing system is assigned to a server machine located in a data center.

[0120] 20. In some embodiments, a computer-implemented method for allocating execution resources to groups of threads within a graphics processing unit comprises: receiving a notification from a process to start a first group of threads, wherein the first subcontext is contained within a plurality of subcontexts, and each subcontext contained within the plurality of subcontexts is allocated to a different process and a different virtual address space; determining whether the first subcontext has at least one processor credit;and if the first subcontext has at least one processor credit, then: identify a first processor contained in a plurality of processors that has a processing load less than or equal to the processor loads allocated to all other processors contained in the plurality of processors, and start the first group of threads to execute on the first processor; or if the first subcontext does not have at least one processor credit, then: identify a subset of processors contained in the plurality of processors already allocated to the first subcontext, identify a second processor contained in the subset of processors that has a processing load less than or equal to the processor loads allocated to all other processors contained in the subset of processors;and starting the first group of threads on the second processor.

[0121] 21. In some embodiments, a computer-implemented method for allocating local memory to thread groups within a graphics processing unit comprises: receiving a notification that a first thread group associated with a first subcontext has been allocated for execution on a first processor; identifying a first record in a local memory block allocation table corresponding to the first subcontext; identifying a first local memory block that is currently unallocated; and storing a first value in the first record indicating that the first local memory block has been allocated to the first subcontext and the first processor.

[0122] 22. The computer-implemented method of Section 21, which further comprises storing a first index to the first local memory block in the first record.

[0123] 23. The computer-implemented method of Section 21 or Section 22, further comprising: receiving a notification that a second thread group associated with the first subcontext has been allocated to run on a second processor; identifying the first record in the local memory block allocation table corresponding to the first subcontext; identifying a second local memory block that is currently unallocated; storing a second value in the first record indicating that the second local memory block is allocated to the first subcontext and the second processor; and storing a second index to the second local memory block in the first record, the second index being greater than the first index.

[0124] 24. The computer-implemented method of one of Sections 21-23, wherein: the first processor and the second processor are contained in a processing cluster; and the first local memory block is accessible to both the first processor and the second processor.

[0125] 25. The computer-implemented method of one of Sections 21-24, wherein the first processor and the second processor are contained in a plurality of processors and the local memory block allocation table is initialized by an operating system or hypervisor before any group of threads is started for execution on any processor contained in the plurality of processors.

[0126] 26. The computer-implemented method of one of Sections 21-25, which further comprises: retrieving the first index to the first local memory block from the first record; and assigning the first local memory block to the first index.

[0127] 27. The computer-implemented method of one of Sections 21-26, which further comprises sending a message to the first processor containing the first index to the first local memory block.

[0128] 28. The computer-implemented procedure of one of Sections 21-27, which further comprises: determining that the first thread group has completed execution on the first processor; and storing a new value in the first record indicating that the first local memory block is not allocated to the first subcontext.

[0129] 29. The computer-implemented method of one of Sections 21-28, which further comprises: receiving a notification that a second thread group, associated with a second subcontext, has been allocated to be executed on a second processor; identifying a second record in the local memory block allocation table corresponding to the second subcontext and the second processor; and determining that a memory block has already been allocated to the second subcontext.

[0130] 30. The computer-implemented method of one of Sections 21-29, starting with a first thread group to execute on the first processor before storing the value in the first record indicating that the first local memory block is allocated to the first subcontext.

[0131] 31. In some embodiments, a parallel processing system comprises: a scheduler that sends a plurality of tasks to a workload distributor; and a workload distributor that: selects a task corresponding to a process from a task list associated with a first subcontext, identifies a first thread group associated with the first subcontext that has been allocated for execution on a first processor, determines that the first subcontext has at least one processor credit, identifies a first record in a local memory block allocation table corresponding to the first subcontext, identifies a first local memory block that is currently unallocated, and stores a first value in the first record indicating that the first local memory block is allocated to the first subcontext and the first processor.

[0132] 32. The parallel processing system of Section 31, wherein the workload distributor continues to store a first index to the first local memory block in the first record.

[0133] 33. The parallel processing system of Section 31 or Section 32, wherein the workload distributor further: identifies a second thread group associated with the first subcontext that has been allocated for execution on a second processor; identifies the first record in the local memory block allocation table corresponding to the first subcontext; identifies a second local memory block that is currently unallocated; stores a second value in the first record indicating that the second local memory block is allocated to the first subcontext and the second processor; and stores a second index to the second local memory block in the first record, the second index being greater than the first index.

[0134] 34. The parallel processing system of one of Sections 31-33, wherein: the first processor and the second processor are contained in a processing cluster; and the first local memory block is accessible to both the first processor and the second processor.

[0135] 35. The parallel processing system of one of Sections 31-34, wherein the first processor and the second processor are contained in a plurality of processors and the local memory block allocation table is initialized by an operating system or a hypervisor before any group of threads is started for execution on any processor contained in the plurality of processors.

[0136] 36. The parallel processing system of one of Sections 31-35, wherein the workload distributor further: retrieves the first index to the first local memory block from the first record; and assigns the first local memory block to the first index.

[0137] 37. The parallel processing system of one of Sections 31-36, wherein the first processor accesses the first local memory block via a virtual address space assigned to the first subcontext.

[0138] 38. The parallel processing system of one of Sections 31-37, wherein the workload distributor further: generates a start packet for the first processor containing a page directory base address allocated to the virtual address space; and sends the start packet to the first processor.

[0139] 39. The parallel processing system of one of Sections 31-38, wherein the startup package continues to contain local memory allocation information with respect to the first local memory block.

[0140] 40. In some embodiments, a computer-implemented method for allocating local memory to thread groups within a graphics processing unit comprises: receiving a notification that a first thread group associated with a first subcontext has been allocated for execution on a first processor; identifying a first record in a local memory block allocation table corresponding to the first subcontext; determining whether local memory blocks contained within a plurality of local memory blocks are statically allocated;and if the local memory blocks contained in the plurality of local memory blocks are statically allocated, then retrieve a first index associated with a first local memory block from the first record; or if the local memory blocks contained in the plurality of local memory blocks are not statically allocated, then identify a second local memory block that is currently unallocated; and store a second index associated with the second local memory block in the first record.

[0141] Any and all combinations of any of the claim elements specified in any of the claims and / or any elements described in this application fall in any way within the intended scope of protection of the present invention and the protection.

[0142] The descriptions of the various embodiments are given for illustrative purposes only and are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and deviations are apparent to the person skilled in the art without departing from the scope of protection and spirit of the described embodiments.

[0143] Aspects of the present embodiments can be implemented as a system, method, or computer program product. Accordingly, aspects of the present disclosure can take the form of a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.), or an embodiment that combines software and hardware aspects, all of which may herein be generally referred to as a "module" or "system." Furthermore, aspects of the present disclosure can take the form of a computer program product embodied in one or more computer-readable media with computer-readable program code embodied thereon.

[0144] Any combination of one or more computer-readable media can be used. The computer-readable medium can be a computer-readable signaling medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media would be: an electrical connection with one or more cables, a portable CD, a hard disk, random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable CD-ROM storage device, an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.In the context of this document, a computer-readable storage medium can be any physical medium that can contain or store a program for use by or in conjunction with a system, device or apparatus for carrying out instructions.

[0145] Aspects of the present disclosure are described above with reference to flowchart representations and / or block diagrams of processes, devices (systems), and computer program products according to embodiments of the disclosure. It is understood that each block of the flowchart representations and / or block diagrams, and combinations of blocks in the flowchart representations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a specialized computer, or other programmable data processing device to manufacture a machine such that the instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the functions / actions specified in the flowchart and / or block diagram block or blocks.Such processors can be, without restriction, general-purpose processors, special-purpose processors, application-specific processors, or field-programmable gate arrays.

[0146] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this respect, each block in the flowchart or block diagrams can represent a module, segment, or section of code comprising one or more executable instructions for implementing the specified logical function(s). Note also that in some alternative implementations, the functions specified in the block may occur out of the order shown in the figures. For example, two consecutively shown blocks may be executed essentially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functionality involved.Note also that each block of the block diagrams and / or flowchart representation and combinations of blocks in the block diagrams and / or flowchart representation can be implemented by hardware-based special systems that perform the specified functions or actions, or combinations of special hardware and computer instructions.

[0147] While the foregoing relates to embodiments of the present disclosure, other and further embodiments of the disclosure may be conceived without deviating from its basic scope of protection, and its scope of protection is determined by the following claims.

Claims

[1] Computer-implemented method for allocating execution resources to groups of threads within a graphics processing unit, wherein the method comprises: Receiving a notification from a process that a first group of threads, which is assigned to a first subcontext, is to be started; Determine, based on a count of a credit counter (434), whether at least one processor credit remains from an initial number of processor credits that was initially allocated to the first subcontext, wherein the initial number of processor credits specifies a maximum number of processors (340) on which the first subcontext can be processed concurrently, and wherein the credit counter is initially set to the initial number of processor credits and is decremented each time a group of threads allocated to the first subcontext is started for execution on a processor not yet allocated to the first subcontext; if at least one processor credit remains, then identify a first processor that is contained in a plurality of processors (340) and that has a processing load less than or equal to the processor loads allocated to all the other processors contained in the plurality of processors; Starting the first group of threads for execution on the first processor, or If no processor credits remain, then identify a second processor in a subset of processors already assigned to the first subcontext, where the second processor has a processor load less than or equal to the processor load of all other processors in the subset of processors, and Starting the first group of threads for execution on the second processor. [2] Computer-implemented method according to claim 1, wherein the first subcontext is contained in a plurality of subcontexts and each subcontext contained in the plurality of subcontexts is assigned to a different process and a different virtual address space. [3] Computer-implemented method according to claim 1 or 2, wherein the first subcontext is contained in a plurality of subcontexts and each subcontext contained in the plurality of subcontexts is associated with other state data. [4] Computer-implemented method according to any one of the preceding claims, wherein: the first subcontext is contained within a multitude of subcontexts; the first subcontext is assigned to a first virtual address space; a second subcontext, contained within the multitude of subcontexts, is assigned to the first virtual address space; and a third subcontext, contained within the multitude of subcontexts, is assigned to a second virtual address space that is different from the first virtual address space. [5] A computer-implemented method according to any of the preceding claims, further comprising decrementing the credit counter associated with the first subcontext in response to the starting of the first group of threads for execution on the first processor. [6] Computer-implemented method according to claim 5, further comprising: Determine that the first group of threads has completed execution on the first processor; and Incrementing the credit counter assigned to the first subcontext. [7] Computer-implemented method according to any of the preceding claims, comprising identifying a first processor included in the plurality of processors: Determining the number of available execution slots for each processor included in the multitude of processors; Identifying one or more processors that are included among the multitude of processors having the highest count of available execution slots; and Selecting the first processor from the one or more processors. [8] A computer-implemented method according to any of the preceding claims, further comprising determining that the first subcontext is not exclusive with respect to any subcontext associated with a second group of threads currently running on the first processor. [9] Computer-implemented method according to any one of the preceding claims, further comprising: Retrieving an initial record associated with the first subcontext from an initial table; and Determine from the first record that each processor included in the plurality of processors is available with respect to executing at least one group of threads associated with the first subcontext. [10] Computer-implemented method according to claim 9, wherein the first table is initialized by an operating system or a hypervisor before any group of threads is started for execution on any processor included in the plurality of processors. [11] Computer-implemented method according to claim 9 or 10, further comprising: Retrieving a second record associated with the first subcontext from a second table, wherein the second record specifies a different virtual processor identifier for each processor contained in the plurality of processors and available with respect to executing at least one set of threads associated with the first subcontext; and Identifying a virtual processor identifier for the first processor from the second recording; where the first processor performs a memory address calculation based on the virtual processor identifier. [12] Parallel processing system, comprising: a scheduler (410) that sends a large number of tasks to a workload distributor; and a workload distributor (420) who is trained as: Receiving a notification from a process that a first group of threads, which is assigned to a first subcontext, is to be started; Determine, based on a balance counter, whether at least one processor balance remains from an initial number of processor balances that was initially allocated to the first subcontext, where the initial number of processor balances specifies a maximum number of processors on which the first subcontext can be processed concurrently, and where the balance counter is initially set to the initial number of processor balances and is decremented each time a group of threads allocated to the first subcontext is started for execution on a processor that is still allocated to the first subcontext; If at least one processor credit remains, then identify a first processor that is contained in a plurality of processors and that has a processing load less than or equal to the processor loads allocated to all the other processors contained in the plurality of processors; Starting the first group of threads for execution on the first processor, or If no processor credits remain, then identify a second processor in a subset of processors already assigned to the first subcontext, where the second processor has a processor load less than or equal to the processor load of all other processors in the subset of processors, and Starting the first group of threads for execution on the second processor. [13] Parallel processing system according to claim 12, wherein the workload distributor further determines that the first processor is available to execute the first group of threads. [14] Parallel processing system according to claim 12 or 13, wherein the workload distributor further decrements the credit counter associated with the first subcontext in response to the starting of the first group of threads for execution on the first processor. [15] Parallel processing system according to claim 14, wherein the workload distributor further: determined that the first group of threads has completed execution on the first processor; and increments the credit counter assigned to the first subcontext. [16] Parallel processing system according to any one of claims 12 to 15, wherein identifying a first processor included in the plurality of processors comprises: Determining a count of available execution slots for each processor included in the multitude of processors; Identifying one or more processors that are included among the multitude of processors having the highest count of available execution slots; and Selecting the first processor from the one or more processors. [17] Parallel processing system according to any one of claims 12 to 16, wherein the workload distributor further comprises: a starter package is created for the first processor, containing the first group of threads; and sends the starter package to the first processor. [18] Parallel processing system according to claim 17, wherein the start package further contains a number corresponding to the first subcontext. [19] Parallel processing system according to claim 17 or 18, wherein the parallel processing system is assigned to a server machine located in a data center.

Images