Concurrent throttling while managing upstream resources

Abstract

Systems, apparatuses, and methods are disclosed for arbitrating threads in a computing system. The computing system includes a processor having a plurality of cores that are each capable of concurrently processing instructions of a plurality of threads. When a thread throttling unit receives an indication that a shared cache has a resource contention, the throttling unit sets a cache miss threshold for the cache. If the number of cache misses exceeds the threshold, the throttling unit notifies a particular upstream computing unit to throttle the processing of instructions of the thread. After a time period elapses, if the cache continues to exceed the threshold, the throttling unit notifies the upstream computing unit to more restrictively throttle the thread by performing one or more of decreasing a selection rate and increasing the time period. Otherwise, the unit notifies the upstream computing unit to less restrictively throttle the thread.

Description

BACKGROUND

[0001] Description of Related Art

[0002] Processors, such as general-purpose central processing units, utilize any number of pipeline stages to process instructions of a software application. These processors include hardware for multiple instruction issue, dispatch, execution, and retirement; additional routing and logic to determine data forwarding for multiple instructions that proceed simultaneously per clock cycle; and complex branch prediction schemes. Additionally, many processors replicate processor cores multiple times in order to improve parallel execution of software applications. To further exploit instruction-level parallelism, processor cores execute multiple threads simultaneously.

[0003] Depending on the workload, two or more threads in a processor core can contend for shared resources of the processor core, such as an instruction fetch unit (IFU), a branch prediction unit, a reservation station that issues instructions to an integer execution unit or a floating point unit (FPU), a load queue and a store queue within a load-store unit (LSU), a common data bus that transports results of executed instructions, etc. Additionally, threads across two or more processor cores can contend for shared resources, such as a level two (L2) or level three (L3) cache, a memory bus, input / output (I / O) devices.

[0004] Resource contention occurs when the number of computing units requesting access to a shared resource exceeds the number of units that the shared resource can support simultaneous access. Examples of computing units are hardware threads, processor cores, microprocessors, etc. Resource contention typically results in a multi-cycle stall. The duration of the resource contention stall depends on the amount of time granted to one or more other computing units that are currently accessing the shared resource. The stall resulting from resource contention reduces the benefit of replicating cores or other computing units that are capable of multithreaded execution.

[0005] In view of the above, it would be desirable to have an efficient method for arbitrating threads in a computing system. BRIEF DESCRIPTION OF DRAWINGS

[0006] The advantages of the methods and mechanisms described herein can be better understood by reference to the following description in conjunction with the accompanying drawings in which:

[0007] Figure 1 is a block diagram of one embodiment of a processor core.

[0008] Figure 2 is a block diagram of one embodiment of control logic for thread arbitration in a computing system.

[0009] Figure 3 is a flow diagram of one embodiment of a method for thread arbitration in a computing system.

[0010] Figure 4is a flow diagram of one embodiment of a method for thread arbitration in a computing system.

[0011] Figure 5 is a flow diagram of one embodiment of a method for thread arbitration in a computing system.

[0012] Figure 6 is a block diagram of one embodiment of a microprocessor.

[0013] While the application is susceptible to various modifications and alternative forms, specific embodiments are shown by way of example in the drawings and are herein described in detail. It should be understood however that the drawings and detailed description thereto are not intended to limit the application to the particular form disclosed but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the application as defined by the appended claims. DETAILED DESCRIPTION

[0014] In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, it will be apparent to those skilled in the art that the various embodiments can be practiced without such specific details. In some instances, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail, so as not to obscure the understanding of the described methods. It will be appreciated, for example, that computer program instructions can be loaded onto a computer, which can include a processor, to produce an apparatus for practicing the methods presented herein. It will be appreciated that the functions of the various elements shown in the figures can be carried out by either software implementing the embodiments or by hardware equivalent thereof, as both are intended to be encompassed by the embodiments presented herein. For the sake of presentation, the detailed description will be in the form of a representative structure

[0015] Various systems, apparatuses, methods, and computer-readable media for arbitrating threads in a computing system are disclosed. In various embodiments, a computing system includes multiple shared resources. An example of a resource is various types of compute units used in different instruction processing stages in a processor core. Stages include an instruction fetch stage, a decode stage, a dispatch stage, and an issue stage, among others. A multi-threaded operating system or hypervisor divides a software application into multiple processes, each having multiple threads. A scheduler in the operating system or hypervisor manages the allocation of these different software threads to hardware, such as specific cores of multiple processor cores of a computing system. Multiple threads in a specific core share compute units and data storage queues in the core. Another example of a resource is memory controllers and data storage arrays in a hierarchical memory subsystem shared by multiple cores. Cores include one or more local cache levels in a cache hierarchy that are shared by multiple threads in the core, and the computing system includes one or more global cache levels shared by multiple cores.

[0016] Resource contention occurs when the number of sources requesting access to a shared resource exceeds the number of sources that the shared resource supports for simultaneous access. Examples of sources are threads in a core, cores, etc. Sources seeking to utilize the shared resource but not authorized for access stall. To reduce the penalty in a computing system for stalling, one or more of a software monitor and a hardware monitor determine whether a particular source (e.g., thread, core, etc.) is eligible for throttling or reduced instruction processing at the shared resource. However, throttling the shared resource can cause other resources upstream of the shared resource to stall or reduce performance because the stalled source has already used the upstream resources. As used herein, a "software monitor" includes one of various software for monitoring resource utilization of software threads and identifying particular threads for throttling, such as an operating system, a hypervisor, a basic input output software (BIOS) controlled function, user-level software using an application programmer interface (API), and any other type of software capable of manipulating the configuration of a computing system.

[0017] Moving the throttling point upstream from the shared resource frees some upstream resources from resource contention, but if the point is moved too far upstream, the performance of the thread can be reduced below a threshold. Additionally, the original shared resource can continue to experience resource contention. In an embodiment, the original shared resource in contention is a core level two (L2) cache. Other levels of cache hierarchies are possible and contemplated. Throttling arbitration of threads upstream of instruction processing from the L2 cache does not guarantee a reduction in the number of memory requests delivered by the L2 cache. The L2 cache receives memory requests from the L1 data cache and the L1 instruction cache. Additionally, the L2 cache controller can generate prefetch out requests for one or more miss requests to send to lower level memory, such as a level three (L3) cache or system memory. To achieve a balance between supporting throttling of the local L2 cache and maintaining a particular memory bandwidth at the L2 cache, control logic of the core monitors the number of cache misses at the L2 cache and sets a threshold number of cache misses for a thread at a particular cache level, the L2 cache.

[0018] When the control logic of the core determines or receives an indication that the number of cache misses for a particular thread at the L2 cache is greater than the threshold number of cache misses, the control logic notifies a particular upstream computing unit to throttle instruction processing for the thread. In one embodiment, the upstream computing unit is a dispatch unit. In another embodiment, the upstream computing unit is an instruction fetch unit, a decode unit, a load / store unit (LSU), etc.

[0019] In various embodiments, the control logic of the core sets a throttling severity level for the upstream compute unit, which specifies a selection rate of instructions of a thread to be selected for processing on the compute unit. In addition, the severity level specifies a time period to maintain the selection rate. When that time period elapses, if the number of cache misses at the L2 cache is still greater than the threshold number of cache misses, the control logic updates the severity level to a more restrictive throttling level. However, if the number of cache misses at the particular cache level is not greater than the threshold number of cache misses, the control logic updates the severity level to a less restrictive throttling level.

[0020] Referring to Figure 1 One embodiment of a general-purpose processor core 100 that performs out-of-order execution is shown. In one embodiment, the processor core 100 processes two or more threads concurrently. An instruction cache (i-cache) 102 stores instructions of a software application, and a corresponding translation lookaside buffer (TLB) stores addresses for accessing the instructions. If there is no i-cache miss, an instruction fetch unit (IFU) 104 fetches multiple instructions from the i-cache 102 in each clock cycle. The IFU 104 includes a program counter that holds a pointer to the address of the next instruction to be fetched from the i-cache 102, which is compared to the addresses in the i-TLB. The IFU 104 also includes a branch prediction unit (not shown) that predicts the results of conditional instructions before the actual results are determined by the execution unit in subsequent pipeline stages.

[0021] A decode unit 106 decodes the opcodes of the multiple fetched instructions and allocates entries in an in-order retirement queue, such as a reorder buffer 118, reservation stations 108, and load / store units 114. In some embodiments, the decode unit 106 also performs register renaming on instruction operands to eliminate write-after-read (WAR) and write-after-write (WAW) hazards. In other embodiments, the reorder buffer 118 performs register renaming. When instructions are free of read-after-write (RAW) hazards, the instructions are allocated into the reservation stations 108. Allocation of an entry in the reservation stations 108 is considered dispatch. The reservation stations 108 act as instruction queues in which instructions wait until their operands are available. When the operands are available and hardware resources are also available, the logic of the reservation stations 108 issues the out-of-order instructions to the integer and floating point functional units 110 or the load / store units 114.

[0022] The functional units 110 include an arithmetic logic unit (ALU) for performing computer calculations such as addition, subtraction, multiplication, division, and square root. Logic can be included to determine the result of conditional instructions. The load / store unit (LSU) 114 receives issued memory accesses such as load and store operations. The load / store unit 114 includes queues and logic for executing memory access instructions. In an embodiment, the load / store unit 114 includes verification logic for ensuring that a load instruction receives forwarded data from a correct, most recent store instruction.

[0023] The functional units 110 and the load / store unit 114 present results on a common data bus 112. The reorder buffer 118 receives results from the common data bus 112. In one embodiment, the reorder buffer 118 is a first-in-first-out (FIFO) queue that ensures that instructions are retired in program order. Here, an instruction that has received its results is marked as retired. If the instruction is at the head of the queue, logic of the reorder buffer 118 sends its results to the register file 120. The register file 120 holds the architectural state of the general purpose registers of the processor core 100. Then, instructions in the reorder buffer 118 retire in order, and the logic updates its queue head pointer to point to the next instruction in program order.

[0024] Results on the common data bus 112 are also sent to the reservation station 108 to forward values to operands of instructions waiting for results. When these waiting instructions have values for their operands and hardware resources are available to execute the instructions, these waiting instructions can be issued out of the reservation station 108 to the appropriate resource in the functional units 110 or the load / store unit 114, out of order. Unissued or unretired memory access instructions have entries in the load / store unit. Forwarded data values from the most recent, unissued older store instruction for a running or unissued load instruction are placed on the common data bus 112 or routed directly to the appropriate entry in the load buffer in the load / store unit 114.

[0025] Load / store unit 114 sends memory access requests to on-chip Level 1 (L1) data cache (d-cache) 116. In some embodiments, core 100 also includes Level 2 (L2) cache 130 for servicing memory requests from L1 data cache 116 and instruction cache 102, which is also an L1 cache. Each of L1 data cache 116 and L2 cache 130 has its own TLB for address comparison with the received memory request. Cache controllers for L1 data cache 116 and L2 cache 130 search the data array serially or in parallel. If the requested memory line is not found in L1 data cache 116 or instruction cache 102, the corresponding cache controller sends a miss request to L2 cache 130.

[0026] If the requested memory line is not found in L2 cache 130, the L2 cache controller issues a miss request to access memory in lower-level memory (such as the L3 cache or system memory). In various implementations, core 100 shares lower-level memory with other external cores. Waiting for the requested memory line to arrive from lower-level memory typically requires a significant number of clock cycles. To mitigate the performance degradation caused by this wait, in some implementations, the L2 cache controller generates one or more prefetch requests for one or more miss requests. The L2 cache controller updates the number of prefetch requests based on the number of accesses within the address range, the amount of time since the address range was accessed, process identifier (ID) or thread ID, priority level, etc. This information is not known to the cache controllers of L1 data cache 116 and instruction cache 102, nor to load / store unit 114.

[0027] Multiple threads share multiple resources within core 100. For example, these multiple threads share... Figure 1 Each of blocks 102-130 is shown. In this implementation, hardware monitor 140 collects data to help tune and understand the behavior of the software application and the utilization of hardware resources on core 100. Additionally, some events are inconvenient to measure in software, such as peak memory contention or response time for calling interrupt handlers. The hardware (such as circuitry) in hardware monitor 140 expands the types and details of measurements that have little or no impact on application performance. Based on information provided by one or more of hardware monitor 140 and software monitor 142, thread rate limiting unit 150 manages instruction rate limiting for one or more of blocks 102-130.

[0028] In one embodiment, the hardware monitor 140 includes counters to count the number of processor cycles spent executing pipeline flushes, data cache snoop and snoop hits, cache and TLB misses and miss rates, read and write operations, data cache lines written back, branch operations, branch operations taken, number of instructions in the integer or floating point pipelines, bus utilization, and utilization of various queues. As previously described, the software monitor 142 includes one of various types of software to monitor resource utilization of software threads and identify specific threads for throttling. Examples of software monitor 142 are operating systems, hypervisors, functions controlled by basic input output software (BIOS), user-level software using application programmer interfaces (APIs), and any other type of software capable of adjusting the configuration of a computing system. For illustrative purposes, each of the hardware monitor 140, software monitor 142, and thread throttling unit 150 are shown as being located outside of the processing blocks 102-130 of the processor core 100. In other embodiments, the functionality of one or more of the hardware monitor 140, software monitor 142, and thread throttling unit 150 are integrated throughout the floor plan of the core 100. Alternatively, portions of the hardware monitor 140 and thread throttling unit 150 are both located inside and outside of the core 100. All such combinations are contemplated.

[0029] The functionality of the thread throttling unit 150 is implemented in hardware, such as circuitry, one of various types of software, or both. In various embodiments, the thread throttling unit 150 receives an indication specifying that one or more shared resources corresponding to one of the blocks 102-130 are being utilized by a particular thread above a threshold utilization. One or more of the hardware monitor 140 and software monitor 142 send the indication to the thread throttling unit 150. In one embodiment, the identified shared resource is the L2 cache 130, which has one or more queues shared by multiple threads. In an embodiment, the thread throttling unit 150 sets a threshold number of cache misses for the particular thread at the L2 cache 130. Then, when the thread throttling unit 150 determines or receives an indication specifying that the number of cache misses by the particular thread at the L2 cache is greater than the threshold number of cache misses, the thread throttling unit 150 notifies a particular upstream computing unit to throttle the processing of instructions for the particular thread. In one embodiment, the upstream computing unit is the reorder buffer (dispatch unit) 118. In another embodiment, the upstream computing unit is the instruction fetch unit 104, decode unit 106, load / store unit (LSU) 114, etc.

[0030] While a reorder buffer 118 is used in the following description, in other embodiments another upstream computing unit is used. In various embodiments, the thread throttling unit 150 sets a throttling severity level for the reorder buffer 118, which specifies a selection rate at which instructions of a selected thread are selected for processing on the reorder buffer 118. In addition, the severity level specifies a time period to maintain the selection rate. The severity level is also referred to as a throttling level. When the time period elapses, if the number of cache misses at the L2 cache 130 is still greater than the cache miss threshold number, the thread throttling unit 150 updates the severity level to a more restrictive throttling level. However, if the number of cache misses at the L2 cache 130 is not greater than the cache miss threshold number, the thread throttling unit 150 updates the severity level to a less restrictive throttling level. The more restrictive throttling level has a selection rate that is less than the current selection rate. In some embodiments, the time period corresponding to the more restrictive throttling level is greater than the current time period. Conversely, the less restrictive throttling level has a selection rate that is greater than the current selection rate. In some embodiments, the time period corresponding to the less restrictive throttling level is less than the current time period.

[0031] To enable access to the data by, for example, an operating system, another software monitor 142, or a software programmer, in one embodiment, one or more of the hardware monitor 140 and the thread throttling unit 150 include a monitor output pin. The output pin toggles after a predetermined event, counter overflow, status information, or otherwise. By connecting one of these pins to an interrupt pin, software reacts to the reported data. In another embodiment, specific instructions are included in the instruction set architecture (ISA) in order to disable and enable data collection, respectively, and to read one or more specific registers in one or more of the hardware monitor 140 and the thread throttling unit 150.

[0032] In some embodiments, accessing these registers requires kernel-level support. For example, a program needs to be in supervisor mode and execute a system call to access these registers. In yet another embodiment, the operating system provides one or more application programming interfaces (APIs) corresponding to one or more of the hardware monitor 140, the software monitor 142, and the thread throttling unit 150. These APIs allow programming configurable thresholds corresponding to resource contention severity levels, thread arbitration rates, and time periods.

[0033] Reference is now made to Figure 2Figure 2 shows a generalized block diagram illustrating one embodiment of a thread throttling unit 200. The thread throttling unit 200 (or unit 200) includes control and status registers 210, control logic 220, and a table 230. The control logic 220 is implemented in hardware (such as circuitry), software (such as firmware), or both. The control and status registers receive information from one or more of a software monitor and a hardware monitor. Examples of the information are an indication specifying a particular thread that needs throttling, an identification of a shared resource that is experiencing resource contention, a select rate and a time period to use during thread arbitration on the identified shared resource, a measured utilization or another indication of a resource contention severity level, other thresholds, etc.

[0034] In addition, the control and status registers 210 receive information from a particular shared resource. One example of a shared resource is a particular cache level of a multi-level cache hierarchy. The table 230 includes a plurality of entries 240A-240D. One of a variety of data storage structures is used to implement the table 230. Examples of structures are flip-flops, one of a variety of random access memories (RAMs), content addressable memories (CAMs), etc. As shown, each of the entries 240A-240D includes a plurality of fields. Although the fields 250-264 are shown in a contiguous manner, in other embodiments another number of fields is used, and the storage arrangement is different from that shown, and possibly non-contiguous storage is used between some of the fields.

[0035] When a thread is identified as needing throttling, the control logic 220 or control logic in the table 230 allocates an entry of the entries 240A-240D. The field 250 stores a thread identifier that identifies the thread, and the field 252 stores a process identifier of a process from which the thread originates. When the particular shared resource that is experiencing resource contention is a cache level, the field 254 stores a cache miss threshold number for that cache level. The field 256 stores an identifier that identifies a compute unit that is to be throttled. The compute unit handles instructions from upstream of the shared resource that is experiencing resource contention. In one embodiment, the shared resource that is experiencing resource contention is an L2 cache in a core, and the upstream compute unit to be throttled is a dispatch unit in the core. The field 258 stores an indication of an age of the table entry.

[0036] Field 260 stores a severity level that indicates how much throttling is needed for the thread. In one embodiment, the severity level is based on a utilization measurement of a shared resource, a throughput or bandwidth of the shared resource, and a duration of resource contention. One or more of an external software monitor, an external hardware monitor, and control logic 220 set the severity level. Field 262 stores a selection rate that selects threads for processing at a particular upstream compute unit. Field 264 stores an arbitration time period for maintaining the selection rate. Fields 262 and 264 are set by the severity level stored in field 260. In various embodiments, the values stored in fields 262 and 264 are read from registers in control and status registers 210.

[0037] Referring now to Figure 3 , one embodiment of a method 300 for arbitrating threads in a computing system is shown. For purposes of discussion, the steps in this embodiment (and in Figures 4-5 ) are shown in sequential order. However, it is to be appreciated that one or more of the steps described are performed concurrently, in a different order than described, or are omitted entirely, depending on the circumstances. Other additional steps are also performed as desired. Any of the various systems or apparatus described herein are configured to implement method 300.

[0038] A multi-threaded software application generates multiple software processes in the same application. A multi-threaded operating system divides the software application into multiple processes, each having multiple threads. A scheduler in the operating system manages assigning these different software threads to hardware, such as a particular core of multiple processor cores of a computing system. One or more of hardware and software monitors monitor the multiple threads being processed (block 302). For example, a scheduler of the operating system is capable of monitoring. Other types of software monitors include at least a hypervisor, a basic input output software (BIOS) controlled function, and user-level software using an application programmer interface (API). Similarly, a dedicated hardware monitor in a processor core is capable of measuring contention for a resource shared by multiple threads on the core.

[0039] One or more of the software monitor and the hardware monitor can determine whether a thread qualifies as a candidate for throttling. In embodiments, an operating system or other software defines a quality of service (QoS) goal for a particular thread on a memory bandwidth management architecture. The particular thread is derived from a process of a software application that needs to meet the QoS memory bandwidth requirement. When another thread consumes resources (such as queue occupancy or bus utilization) that cause the other particular thread to fail to meet the QoS memory bandwidth requirement, the operating system or other software identifies the resource-consuming thread as a candidate for throttling. Similarly, in some embodiments, the hardware monitor compares resource consumption (such as queue occupancy, bus utilization, etc.) to a threshold. When a particular thread exceeds one or more thresholds, the hardware monitor identifies the resource-consuming thread as a candidate for throttling. When one or more of the operating system and the hardware monitor determine that a particular thread qualifies as a candidate for throttling, an indication is sent to control logic in the core.

[0040] If no threads in the core are candidates for throttling (the "No" branch of conditional block 304), control flow of the method 300 returns to block 302. However, if a thread of the plurality of threads in the core is a candidate for throttling (the "Yes" branch of conditional block 304), control logic of the core sets a throttling severity level for the thread (block 306). The control logic sets a number of miss thresholds for the thread at a particular cache level based on the severity level (block 308). In embodiments, the particular cache level is a cache level for which the number of memory access requests generated by the particular cache level (particularly due to speculative prefetch out requests) is unknown to higher levels of the cache hierarchy. For example, for one or more miss requests, the cache controller generates one or more prefetch out requests, so the actual number of memory requests issued from the particular cache level is unknown to higher levels in the cache hierarchy.

[0041] In one embodiment, one of the software monitor and the hardware monitor determines the number of miss thresholds for the thread at the particular cache level. During processing of instructions of the thread, if the number of cache misses at the particular cache level is not greater than the number of thresholds (the "No" branch of conditional block 310), the control logic continues processing the thread while one or more of the control logic and the cache controller monitor the number of cache misses at the particular cache level (block 312). If the number of cache misses at the particular cache level is greater than the number of thresholds (the "Yes" branch of conditional block 310), the control logic causes the thread to undergo throttling processing at a processing stage upstream of the particular cache level (block 314).

[0042] Reference is made to Figure 4This illustrates one implementation of a method 400 for arbitrating threads in a computing system. The processor core processes instructions for multiple threads. The core's control logic identifies threads that are eligible as candidates for rate limiting and have not exceeded their miss threshold number at a specific cache level (box 402). This identifier in box 402 typically corresponds to ( Figure 3 The control flow from the "No" branch of box 310 in method 300 to box 312. If the control logic determines that the thread is no longer a candidate for rate limiting (the "No" branch of condition box 404), the control logic removes the cache miss threshold for the thread (box 406). For example, one or more of the control logic at a specific cache level and the cache controller resets the threshold to the default maximum value, or otherwise invalidates the threshold for the thread. In one implementation, one or more of the software monitor and the hardware monitor determine that the thread is no longer a candidate and notify the control logic. In another implementation, the control logic determines that the thread has not exceeded its cache miss threshold for the duration exceeding the limit. After removing the cache miss threshold for the thread, the kernel continues processing the thread (box 408).

[0043] If the control logic determines that the thread is still a candidate for rate limiting (the "Yes" branch of condition box 404), but the thread has already completed and is therefore no longer active (the "No" branch of condition box 410), then method 400 completes (box 412). If the control logic determines that the thread is still a candidate for rate limiting (the "Yes" branch of condition box 404) and the thread is still active (the "Yes" branch of condition box 410), then the logic determines whether the thread has generated more cache misses than its cache miss threshold at a particular cache level.

[0044] If the number of cache misses for a thread at a specific cache level is not greater than the miss threshold (the "No" branch of condition box 414), then the control flow of method 400 returns to box 402, where the thread's instructions continue to be processed, even though the thread has not yet exceeded its cache miss threshold at that specific cache level. If the number of cache misses for a thread at a specific cache level is greater than the miss threshold (the "Yes" branch of condition box 414), then the core performs rate limiting on that thread in the upstream processing phase of that specific cache level (box 416). Further details regarding rate limiting are provided in method 500.

[0045] Reference Figure 5embodiment of a method 500 for arbitrating threads in a computing system is shown. A processor core processes instructions of multiple threads. Control logic of the core identifies threads that are eligible to be candidates for throttling and exceed a number of miss threshold at a particular cache level (block 502). The control logic of the core selects a processing stage upstream of the particular cache level (block 504). A computing unit is referred to as an "upstream" computing unit when it processes instructions at an earlier instruction processing stage than the data is accessed at the particular cache level. In one embodiment, the control logic notifies a dispatch unit that is located upstream of instruction processing from the particular cache level. In another embodiment, the control logic notifies another upstream computing unit, such as an instruction fetch unit, a decode unit, a load / store unit (LSU), etc.

[0046] The control logic of the core selects an initial throttling severity level for the thread at the selected upstream processing stage (block 506). The control logic selects a selection rate (or arbitration rate) to use during arbitration of multiple threads for access to a shared resource. In addition, the control logic selects a time period based on the severity level (block 508). In an embodiment, the control logic sets the selection rate to a maximum of a first number of instructions in a second number of clock cycles. Each of the first number and the second number are stored in programmable configuration registers, such as control and status registers. In one embodiment, the second number of clock cycles is an exponential value based on the severity level. An example of the second number of clock cycles is a power of 2 indicated by a value of the severity level. Thus, a severity level of 1 provides a second number of 2Λ1 clock cycles or 2 cycles. A more restrictive severity level of 2 provides a second number of 2Λ2 clock cycles or 4 cycles. A yet more restrictive severity level of 3 provides a second number of 2Λ3 clock cycles or 8 cycles, etc. In an embodiment, the selected time period for use of the selected arbitration rate is an integer multiple of the second number. For a severity level of 3, the second number of clock cycles is 8 cycles, and the integer multiple is 5, then the time period is 8 times 5, or 40 cycles.

[0047] The core throttles the thread at the selected upstream processing stage based on the selected arbitration rate (block 510). If the time period has not elapsed (the "No" branch of conditional block 512), control flow of the method 500 returns to block 510, where the core continues to throttle the thread. If the time period has elapsed (the "Yes" branch of conditional block 512), but the thread has not produced a number of cache misses at the particular cache level that is greater than the cache miss threshold (the "No" branch of conditional block 514), the control logic selects a less restrictive throttling severity level for the thread at the selected upstream processing stage (block 518). Control flow of the method 500 then returns to block 508, where the control logic selects an arbitration rate and a time period based on the severity level.

[0048] If the time period has elapsed (the "Yes" branch of conditional block 512) and the thread has generated a number of cache misses at the particular cache level that is greater than the cache miss threshold number (the "Yes" branch of conditional block 514), then the control logic selects a more restrictive throttling severity level for the thread at the selected upstream processing stage (block 518). Control flow of the method 500 then returns to block 508, where the control logic selects an arbitration rate and a time period based on the severity level.

[0049] Referring to Figure 6 One embodiment of an exemplary microprocessor 600 is shown. The microprocessor 600 includes a memory controller 620 coupled to a memory 630, interface logic 640, one or more processing units 615A-615B including one or more processor cores 612A-612B and corresponding cache memory subsystems 614A-614B, cross-interconnect logic 616, a shared cache memory subsystem 618, and a shared graphics processing unit (GPU) 650. The memory 630 is shown to include operating system code 632. The operating system 632 includes a scheduler 634 for allocating software threads to hardware in the microprocessor 600. In addition, the memory 620 includes a software monitor 636. Examples of the software monitor 636 are a hypervisor, a basic input output software (BIOS) controlled function, user level software using an application programmer interface (API), and any other type of software capable of adjusting the configuration of the microprocessor 600. It is noted that various portions of the operating system code 632 and the software monitor 636 reside in the memory 630, one or more caches (614A-614B, 618) (stored on a non-volatile storage device such as a hard disk (not shown)), etc. In one embodiment, the exemplary functions of the microprocessor 600 are incorporated into a single integrated circuit.

[0050] The GPU 650 can include one or more graphics processor cores and data storage buffers dedicated to a graphics rendering device for a personal computer, workstation, or video game console. The interface 640 generally provides an interface for input / output (I / O) devices outside of the microprocessor 600 to the shared cache memory subsystem 618 and the processing units 615A-615B. The I / O devices include peripheral network devices such as printers, keyboards, monitors, cameras, card readers, hard or floppy disk drives or drive controllers, network interface cards, video accelerators, sound cards, modems, various data acquisition cards.

[0051] The I / O devices are shared by each of the processing units 615A-615B in the microprocessor 600 and the GPU 650. In addition, these I / O devices are shared by the processing units 615A-615B in the other microprocessors. The interface 640 is used to communicate with these other microprocessors and / or other processing nodes. Typically, the interface logic 640 includes buffers that are used to receive data packets from the corresponding link and to buffer data packets to be transmitted on the corresponding link. Any suitable flow control mechanism is used to transmit data packets to and from the microprocessor 600, and the interconnect logic 616 supports this mechanism.

[0052] The microprocessor 600 is connected to respective memories via respective memory controllers 620. The memories use any of various types of memory devices. The address space of the microprocessor 600 is divided among the multiple memories. Each microprocessor 600 or respective processing node that includes the microprocessor 600 uses a memory map that is used to determine which addresses map to which memories, and thus, which microprocessor 600 or processing node a memory request for a particular address should be routed to. In one embodiment, the coherency point for an address is the memory controller 620 that is connected to the memory that stores the byte corresponding to the address. The memory controller 620 includes control circuitry for interfacing with the memory and a request queue for queuing memory requests.

[0053] The cache memory subsystems 614A-614B and 618 use high speed cache memory to store data blocks. In some embodiments, the cache memory subsystems 614A-614B are integrated in the respective processor cores 612A-612B. Alternatively, the cache memory subsystems 614A-614B are connected to the processor cores 612A-612B in a laterally- epi or in-line configuration as needed. In various embodiments, the cache memory subsystems 614A-614B are implemented as a hierarchical structure of caches. If needed, the caches (closer to the processor cores 612A-612B (in the hierarchical structure)) are integrated into the processor cores 612. In one embodiment, the cache memory subsystems 614A-614B each represent an L2 cache structure, and the shared cache subsystem 618 represents an L3 cache structure.

[0054] The processor cores 612A-612B include circuitry for executing instructions according to a predefined general purpose instruction set, and the processor cores 612A-612B support simultaneous multi-threading. In various embodiments, the processor cores 612A-612B have the same architecture as the processor core 100 Figure 1) the same functionality. Similarly, thread throttling units 613A-613B (or TTUs 613A-613B) have the same functionality as thread throttling units 150( Figure 1 ) and thread throttling units 200( Figure 2 ). The multiple threads executed by processor cores 612A-612B share at least cache memory subsystem 618, graphics processing unit (GPU) 650, and coupled I / O devices.

[0055] In various embodiments, the previously described methods and / or mechanisms are implemented using program instructions of a software application. The program instructions describe the behavior of hardware in a high-level programming language, such as C. Alternatively, a hardware design language (HDL) is used, such as Verilog. The program instructions are stored on a non-transitory computer-readable storage medium. Numerous types of storage media are available. The storage media is accessible by a computing system during use to provide the program instructions and accompanying data for program execution by the computing system. The computing system includes at least one or more memories and one or more processors to execute the program instructions.

[0056] It should be emphasized that the above-described embodiments are merely non-limiting examples of implementations. Many variations and modifications will be apparent to those skilled in the art once given the benefit of the above disclosure. The appended claims intend to cover all such variations and modifications.

Claims

1. An apparatus for arbitrating threads, the apparatus comprising: a plurality of compute units each configured to process instructions; a cache configured to store data; and control circuitry configured to: receive an indication that a given thread is a candidate for throttling; in response to identifying the given thread of a plurality of threads as a candidate for throttling, set a cache miss threshold number for the given thread; and in response to determining that a cache miss number for the given thread is greater than the cache miss threshold number, throttle a number of instructions processed by the given thread at a given compute unit of the plurality of compute units.

2. The apparatus of claim 1, wherein the given compute unit processes instructions at an earlier instruction processing stage than data is accessed at the cache.

3. The apparatus of claim 1, wherein the control circuitry is further configured to set a throttling level comprising: a selection rate for selecting the instructions of the given thread for processing at the given compute unit; and a time period for maintaining the selection rate.

4. The apparatus of claim 3, wherein the control circuitry is further configured to update the throttling level to a more restrictive throttling level by performing one or more of decreasing the selection rate and increasing the time period in response to determining that: the time period has elapsed; and after the time period elapses, the cache miss number at the cache is greater than the cache miss threshold number.

5. The apparatus of claim 3, wherein the control circuitry is further configured to update the throttling level to a less restrictive throttling level by performing one or more of increasing the selection rate and decreasing the time period in response to determining that: the time period has elapsed; and after the time period elapses, the cache miss number at the cache is not greater than the cache miss threshold number.

6. The apparatus of claim 1, wherein the control circuitry is further configured to allow the cache to process a cache miss number greater than the cache miss threshold number.

7. The apparatus of claim 1, wherein the indication is generated based at least in part on a quality of service of one of the plurality of threads.

8. The apparatus of claim 2, wherein the instruction processing stage comprises one of an instruction fetch stage, a decode stage, a dispatch stage, and an issue stage.

9. A method for arbitrating threads, the method comprising: processing instructions by a plurality of compute units; storing data by a cache; receiving an indication that a given thread is a candidate for throttling; by control logic, in response to identifying the given thread of a plurality of threads as a candidate for throttling, setting a cache miss threshold number for the given thread; and in response to determining that a cache miss number for the given thread is greater than the cache miss threshold number, throttling a number of instructions processed by the given thread at a given compute unit of the plurality of compute units. ​ ​ throttling the number of instructions processed by the given thread at a given compute unit of the plurality of compute units in response to determining, by the control logic, that a number of cache misses for the given thread is greater than the cache miss threshold number.

10. The method of claim 9, further comprising processing, by the given compute unit, instructions at an instruction processing stage earlier than when the data was accessed in the cache.

11. The method of claim 9, further comprising setting a throttling level, the throttling level comprising: a selection rate for selecting the instructions of the given thread for processing at the given compute unit; and a time period for maintaining the selection rate.

12. The method of claim 11, further comprising updating, in response to determining one or more of: the time period has elapsed; and the number of cache misses at the cache is greater than the cache miss threshold number after the time period has elapsed, the throttling level to a more restrictive throttling level by performing one or more of decreasing the selection rate and increasing the time period.

13. The method of claim 11, further comprising updating, in response to determining one or more of: the time period has elapsed; and the number of cache misses at the cache is not greater than the cache miss threshold number after the time period has elapsed, the throttling level to a less restrictive throttling level by performing one or more of increasing the selection rate and decreasing the time period.

14. The method of claim 9, wherein the indication is generated based at least in part on a quality of service of one of the plurality of threads.

15. The method of claim 10, wherein the instruction processing stage comprises one of an instruction fetch stage, a decode stage, a dispatch stage, and an issue stage.

16. A non-transitory computer-readable storage medium storing program instructions configured to perform thread arbitration, wherein the program instructions are executable to: process instructions by a plurality of compute units; store data by a cache; receive an indication that a given thread is a candidate for throttling; in response to identifying the given thread of the plurality of threads as a candidate for throttling, set a cache miss threshold number for the given thread; and throttle the number of instructions processed by the given thread at a given compute unit of the plurality of compute units in response to determining that a number of cache misses for the given thread is greater than the cache miss threshold number.

17. The non-transitory computer-readable storage medium of claim 16, wherein the given compute unit processes instructions at an instruction processing stage earlier than when the data was accessed in the cache.

18. The non-transitory computer-readable storage medium of claim 16, wherein the program instructions are further executable to set a throttling severity level, the throttling severity level comprising: a selection rate for selecting the instructions of the given thread for processing at the given compute unit; and a time period for maintaining the selection rate.

19. The non-transitory computer-readable storage medium of claim 18, wherein the program instructions are further executable to update, in response to determining one or more of: the time period has elapsed; and the number of cache misses at the cache is greater than the cache miss threshold number after the time period has elapsed, the throttling level to a more restrictive throttling level by performing one or more of decreasing the selection rate and increasing the time period.

20. The non-transitory computer-readable storage medium of claim 18, wherein the program instructions are further executable to update, in response to determining one or more of: the time period has elapsed; and the number of cache misses at the cache is not greater than the cache miss threshold number after the time period has elapsed, the throttling level to a less restrictive throttling level by performing one or more of increasing the selection rate and decreasing the time period.

19. The non-transitory computer-readable storage medium of claim 18, wherein the program instructions are further executable to update the throttling level to a more restrictive throttling level by performing one or more of decreasing the selection rate and increasing the time period in response to determining that: the time period has elapsed; and the number of cache misses at the cache after the time period has passed is greater than the cache miss threshold number.

20. The non-transitory computer-readable storage medium of claim 18, wherein the program instructions are further executable to update the throttling level to a less restrictive throttling level by performing one or more of increasing the selection rate and decreasing the time period in response to determining that: the time period has elapsed; and the number of cache misses at the cache after the time period has passed is not greater than the cache miss threshold number.

Images