Memory request throttling to reduce memory bandwidth usage
Memory request throttling in processing systems addresses oversubscription issues by controlling memory requests based on latency and prefetch accuracy, enhancing system performance and bandwidth efficiency.
Patent Information
- Application Number
- JP2020531612
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2017-12-12
- Filing Date
- 2018-08-28
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2038-08-28
AI Technical Summary
The oversubscription or saturation of memory links between processor cores and local memory in processing systems leads to increased memory access latency and reduced overall system performance.
Implementing memory request throttling by monitoring access latency and prefetch accuracy metrics to control the rate of memory requests, adjusting prefetcher aggressiveness, and limiting the number of pending memory transactions, thereby reducing memory bandwidth consumption.
Improves memory link efficiency and overall system performance by reducing inaccurate prefetch speculation and limiting memory requests, ensuring optimal bandwidth utilization.
Smart Images

Figure 0007584298000002 
Figure 0007584298000003 
Figure 0007584298000004
Abstract
Description
[Background technology]
[0001] Many processing systems pool many resources such that multiple processor cores share the same memory. This often leads to the links between a set of processor cores and the corresponding memory becoming saturated with memory requests, increasing overall memory access latency. Furthermore, an oversubscribed memory link can reduce overall system performance compared to a link that is not fully saturated.
[0002] The present disclosure can be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the following drawings, in which: The use of the same reference numbers in different drawings indicates similar or identical items. [Brief description of the drawings]
[0003] [Figure 1] 1 is a block diagram illustrating a processing system using memory request throttling in accordance with some embodiments. [Diagram 2] 2 is a block diagram illustrating a processor core and corresponding cache hierarchy of the processing system of FIG. 1 in greater detail, in accordance with some embodiments. [Diagram 3] FIG. 2 is a flow diagram illustrating a method for dynamic memory request throttling, according to some embodiments. [Figure 4] FIG. 2 is a flow diagram illustrating a method for software configured memory request throttling according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0004] Oversubscription or saturation of the link between a computing complex and local memory adversely affects overall system performance. The techniques described herein improve memory link efficiency of a processing system by throttling the rate at which memory requests to memory accessed through the memory link can be issued by the memory resource consumer, thereby throttling memory bandwidth used by memory resource consumers of a computing complex. The memory resource consumers that are throttled in this manner may include individual processor cores of the computing complex or specific threads executed by processor cores of the computing complex. In some embodiments, access latency metrics and prefetch accuracy metrics of the memory resource consumers are monitored, and the specific memory request throttling implemented for the memory resource consumer is based on one or both of these metrics. Memory request throttling is accomplished, for example, by modifying prefetcher aggressiveness on behalf of the memory resource consumer, by modifying the maximum number of pending memory transactions available to the memory resource consumer, or by a combination of modifying prefetcher aggressiveness and the maximum number of pending memory transactions available. This approach therefore seeks to improve the effectiveness of memory requests issued by memory resource consumers while reducing inaccurate prefetch speculations and limiting the number of memory requests that a memory resource consumer may issue at a given time, thereby limiting the memory link bandwidth that the memory resource consumer may consume.
[0005] 1 illustrates a processing system 100 that utilizes memory request throttling to improve memory utilization efficiency, according to some embodiments. Processing system 100 includes one or more processing nodes, such as processing node 101. Processing node 101 includes one or more computer complexes, such as computer complexes 102, 104, 106, 108, an interconnect fabric 110, and a local memory 112. Each computer complex includes one or more processor cores, such as four processor cores 114, 115, 116, 117. The processor cores may include, for example, a central processing unit (CPU) core, a graphics processing unit (GPU) core, a digital signal processor (DSP) core, or a combination thereof. It will be appreciated that the number of processor cores in a computer complex may be less than four or more than four.
[0006] Each computer complex further includes a cache hierarchy 118 having one or more levels of cache, such as a first level (L1), a second level (L2), and a third level (L3) cache. While the illustrated example includes three levels, in other embodiments, the cache hierarchy 118 includes fewer or more than three levels. Each cache level includes one or more caches at that level. By way of example, computer complex 102 implements small private caches at L1 for each processor core (illustrated as L1 caches 121, 122, 123, 124, each L1 cache associated with a corresponding one of processor cores 114-117). Additionally, in some embodiments, computer complex also implements larger private caches at L2 for each processor core (illustrated as L2 caches 125, 126, 127, 128, each L2 cache corresponding to a corresponding one of processor cores 114-117). Each of the L2 caches 125-128 is private to a corresponding processor core, but the cache hierarchy 118 operates to maintain coherency between the L2 caches 125-128. In other embodiments, two or more L1 caches may share an L2 cache. For the L3 cache level, the cache hierarchy 118 implements an L3 cache 130 that is shared by the processor cores of the computer complex 102 and thus shared by at least the L2 caches 125-128. In other embodiments, the L3 cache level may include multiple L3 caches shared by the L2 caches 125-128 in various combinations.
[0007] The interconnect fabric 110 serves as an interface between the cache hierarchy 118 of each computer complex and the local memory 112, as an interface between the computer complexes 102, 104, 106, 108, and as an interface between the processing node 101 and any other processing node 132 of the processing system 100. The interconnect fabric 110 may be implemented, for example, as a HyperTransport fabric, an Infinity fabric, or the like. The local memory 112 includes one or more individual random access memory (RAM) components and, in some embodiments, includes multiple links between the RAM components and the interconnect fabric 110, each link being assigned to a separate computer complex (or a subset of the computer complexes may share a given link between the interconnect fabric 110 and the local memory 112).
[0008] In operation, an operating system (OS), hypervisor, or software application assigns instruction sets representing certain processes to processor cores of the computer complexes 102-108 for execution. Here, the execution of an instruction set by a particular processor core is referred to as a "thread." In the course of execution, these threads issue memory requests that operate the processor cores to load data from local memory 112 (or local memory of other processing nodes). Such data may include instructions for execution or operand data that is manipulated during execution of the instructions. As is known in the art, this data is typically temporarily cached in one or more levels of the cache hierarchy 118 to facilitate data access efficiency. Thus, when the cache hierarchy 118 issues memory requests on behalf of a thread for data that is validly stored in the cache hierarchy 118, the memory requests are efficiently serviced by the cache hierarchy 118. However, if a memory request is not satisfied by the cache hierarchy 118 (i.e., there is a "cache miss" for the entire cache hierarchy 118), then the L3 cache 130 of the computer complex executing the thread must forward the memory request through the interconnect fabric 110 to the local memory 112 in order to satisfy the memory request. Given the caching capabilities provided by the cache hierarchy 118, memory requests typically take one of two forms: demand-type memory requests, where a processor core executes an instruction that requires a particular block of data that is not currently present in the processor core's registers, and prefetch-type memory requests, where a cache prefetcher predicts what data the processor core will need in the near future and issues a speculative memory request to obtain the predicted data so that the predicted data will be cached if the processor core issues a demand-type memory request for the data.
[0009] If each processor core of a computing complex typically executes multiple threads simultaneously, and each of these threads frequently issues both demand-type and prefetch-type memory requests that are not satisfied by the cache hierarchy 118 and therefore must be satisfied by the local memory 112, the link between the computing complex and the local memory 112 may become congested or oversubscribed, and oversubscription of the link between the computing complex and the memory typically impacts the overall performance of the processing system 100. Thus, in at least one embodiment, the processing node 101 further includes a memory bandwidth limiting subsystem 134 configured to selectively throttle the rate at which memory resource consumers of the processing node 101 can issue memory requests to reduce the risk of oversubscription of the memory link. In some embodiments, the memory resource consumers that are so throttled are the individual threads of each processor core, and in other embodiments, the memory resource consumers are the individual processor cores as a whole. In still other embodiments, the memory resource consumers that are throttled by the memory bandwidth limiting subsystem 134 include a combination of individual threads and processor cores together.
[0010] As described in more detail below, in some embodiments, the memory bandwidth throttling subsystem 134 is implemented as separate components within the caches of the cache hierarchy 118 of each computer complex 102-108, with some components configured to monitor memory requests issued to the local memory 112 to ascertain a latency metric representative of the average access latency of memory requests issued to the local memory 112, and other components configured to monitor the prefetchers at one or more cache levels to ascertain a prefetch accuracy metric representative of the accuracy of the prefetchers at each memory resource consumer's cache. Other components of the memory bandwidth throttling subsystem 134 then use both the access latency metric and the prefetch accuracy metric of the memory resource consumer to determine a throttle level to apply to the memory resource consumer. The throttle level operates to set one or both of the prefetcher aggressiveness of one or more prefetchers issuing prefetched memory requests on behalf of the memory resource consumer and the maximum number of pending memory requests available or available to be issued by the resource consumer at a given time. In doing so, the memory bandwidth limiting subsystem 134 operates to appropriately throttle the memory bandwidth consumed by any given memory resource consumer, thereby promoting more efficient utilization of the link between the computer complex and the local memory 112.
[0011] In some embodiments, the memory bandwidth limiting subsystem 134 operates to dynamically set the throttle limit to improve overall system performance, while in other embodiments, the memory bandwidth limiting subsystem 134 operates to set the throttle limit according to a fixed target bandwidth limit imposed by the software application 136, e.g., for isolation or differentiation. Additionally, in other embodiments, the memory bandwidth limiting subsystem 134 operates to ensure that the software-imposed target bandwidth limit is met, dynamically adjusting the throttle level in addition to the software-imposed bandwidth limit. These approaches are described in further detail herein with reference to Figures 3 and 4.
[0012] 2 is a diagram illustrating a portion of cache hierarchy 118 for a corresponding processor core 200 of a computer complex, according to some embodiments. Processor core 200 represents one processor core of a computer complex of processing system 100, such as any of processor cores 114-117 of computer complex 102. The illustrated portion of cache hierarchy 118 supporting processor core 200 includes an L1 cache 202 (e.g., representing any of L1 caches 121-124), an L2 cache 204 (e.g., representing any of L2 caches 125-128), and an L3 cache 206 (e.g., representing L3 cache 130). Each of L1 cache 202, L2 cache 204, and L3 cache 206 includes a cache line array (cache line arrays 208, 210, and 212, respectively) and a cache controller (cache controllers 214, 216, and 218, respectively) that controls access to the cache line arrays. Each of the cache line arrays 208, 210, 212 includes a number of entries that store a corresponding cache line of data along with tag line information for the cache line, including a corresponding address, status bits, etc. Additionally, in this example, the L1 cache 202 includes a prefetcher 220 and the L2 cache 204 includes a prefetcher 222, which are configured to speculatively prefetch cache lines of their respective caches. The L2 cache 204 includes a miss information buffer (MIB) 224 that serves as a buffer for L2 read misses directed to the L3 cache 206 or the local memory 112. The MIB 224 includes a number of MIB entries 226, each configured to store information regarding a corresponding memory request that was an L2 read miss before processing the memory request issued by the cache controller 216 to the L3 cache 206 or directly to the local memory 112.
[0013] With respect to the illustrated portion of the processor core 200 and the cache hierarchy 118 associated with the processor core 200, the memory bandwidth throttling subsystem 134 includes a throttle controller 228, a latency tracker 230, and a prefetch accuracy tracker 232. In one embodiment, these components are implemented in the L2 cache 204 as shown in FIG. 2, although in other embodiments, some or all of these components may be implemented at other cache levels. The latency tracker 230 is configured to monitor the access latencies of memory requests issued by memory resource consumers associated with the processor core 200 and satisfied by the local memory 112, and from these latencies, to provide the local memory 112 with an access latency metric representative of a current latency in satisfying the memory requests. By way of example, in some embodiments, the latency tracker 230 maintains a tracker table 234 that tracks the time required to satisfy the memory requests issued to the local memory 112 for each of a sampled subset of memory requests. Table 234 thus includes a number of entries 236, each of which includes an address (ADDR) field 238 that stores an address or address portion associated with a sampled memory request, and a latency (LTCY) field 240 that stores a value representing the time or latency required to satisfy the corresponding sampled memory request. Latency tracker 230 determines an average or nominal access latency from the latency represented by the value of latency field 240 of entry 236, and provides this average access latency, or another value determined from this average access latency, to throttle controller 228 as an access latency metric for use by throttle controller 228 in throttling memory requests. The average access latency may be determined separately for each memory resource consumer associated with processor core 200.Because these memory resource consumers share the same link to memory, the average access latency determined for memory requests of processor core 200 is sufficient to represent the average access latency of any given memory resource consumer of processor core 200 (regardless of which memory resource consumer issued the memory request).
[0014] Prefetch precision tracker 232 is configured to monitor the prefetch precision of prefetchers operating on behalf of processor core 200, including prefetcher 220 of L1 cache 202 and prefetcher 222 of L2 cache 204. To this end, prefetch precision tracker 232 maintains a precision table 242 for each prefetcher operating on behalf of processor core 200. In one embodiment, each precision table 242 is used to track recent prefetched memory requests issued by the corresponding prefetcher on a per-thread basis, and to track whether each tracked prefetched memory request was an “accurate” prefetch, i.e., whether the prefetched cache line associated with the tracked prefetched memory request was ultimately the target of a thread-demanded memory request. Thus, in some embodiments, precision table 242 is implemented as a data structure having a matrix of entries 244, with each column of entries 244 associated with a corresponding one of N threads currently executing in processor core 200, and with each entry 244 in the column having an address (ADDR) field 246 that stores the address or other identifier of the corresponding prefetched memory request, and a precision (ACCY) field 248 that stores a value indicating whether the corresponding cache line was ultimately the target of a thread-demanded memory request (e.g., a one-bit value where "0" indicates that the prefetched cache line was not a precise speculative prefetch and "1" indicates that the prefetched cache line was a precise speculative prefetch). In this manner, prefetch precision tracker 232 monitors prefetches issued by prefetchers 220, 222 and populates the corresponding precision table 242 accordingly. The prefetch precision tracker 232 then periodically or continuously evaluates the prefetch precision indicated in field 248 of entry 244 in precision table 242 and, from this information, determines and updates a prefetch precision metric that represents the prefetch precision of the corresponding prefetcher.Because different threads are more susceptible to accurate prefetch guesses than others, in one embodiment, prefetch accuracy tracker 232 determines and maintains a separate prefetch accuracy metric for each thread, while in another embodiment, one prefetch accuracy metric is determined for all threads (or a subset of two or more threads) of the corresponding prefetcher.
[0015] The throttle controller 228 is configured to use one or both of the current access latency metric from the latency tracker 230 and one or more current prefetch accuracy metrics from the prefetch accuracy tracker 232 to set or throttle the rate at which a corresponding memory resource consumer associated with the processor core 200 is permitted to issue memory requests to be satisfied by the local memory 112, thereby modifying the current memory bandwidth utilization attributable to the memory resource consumer. As described in more detail below, in at least one embodiment, the latency tracker 230 throttles a memory resource consumer by selecting a particular throttle level for the memory resource consumer from a set of predefined throttle levels based on one or both of the latency metric and the corresponding prefetch accuracy metric. Each of the throttle levels comprises a different parameter that affects the number of memory requests that the memory resource consumer can issue, including setting a prefetcher aggressiveness parameter to control the rate at which a given prefetcher issues prefetched memory requests on behalf of the memory resource consumer, setting a maximum number of pending memory requests available to the memory resource consumer (which controls the total number of direct memory requests to memory issued by the memory resource consumer on behalf of the memory resource consumer), or a combination thereof. In at least one embodiment, the throttle levels are implemented as a throttle level table 250, which includes a data structure that is hard-coded, programmed via fuses or other one-time programmable elements at implementation time, programmed by an integrator or user via software visible registers, etc. This throttle setting process is described in more detail below with reference to FIG. 3.
[0016] In addition to or instead of dynamically adjusting memory request throttle levels based on monitoring prefetch accuracy and memory request latency, in some embodiments, a software application (e.g., software application 136 of FIG. 1) specifies a maximum memory bandwidth utilization metric for one or more threads of the software application, and throttle controller 228 is configured to switch between throttle levels to find a throttle level for each of the one or more threads that satisfies the specified maximum bandwidth utilization metric. Thus, in at least one embodiment, processor core 200 includes or has access to storage element 252 (e.g., registers), and the software application executes at least one instruction that causes storage element 252 to store a value representing a software-specified limit on the maximum memory bandwidth utilization of the corresponding thread, which is communicated to throttle controller 228 for execution of that thread. This software-defined throttling process is described in more detail below with reference to FIG. 4.
[0017] 3 illustrates an example method 300 for dynamically adjusting memory request throttle levels of particular threads, particular processor cores, or other particular memory resource consumers of a computing complex of a processing system 100, according to some embodiments. For ease of explanation, the method 300 will be described in conjunction with the example embodiment of the processing system 100 of FIG. 1 and the example embodiment of the cache hierarchy 118 and memory bandwidth throttling subsystem 134 of FIG. 2.
[0018] The method 300 includes three processes operating in parallel: an access latency monitoring process 301, a prefetch accuracy monitoring process 302, and a throttle level setting process 303. The latency monitoring process 301 begins at block 304, where the L2 cache 204 filters memory requests from the processor core 200 to identify direct memory requests to issue to the local memory 112, i.e., memory requests that cannot be satisfied by a cache in the cache hierarchy 118 of the computer complex implementing the processor core 200 and are sent to the local memory 112 to satisfy the memory request. Illustratively, when a memory request is issued from the processor core 200 to the cache hierarchy 118, one or more probes may be sent to each cache in the cache hierarchy 118 to determine whether a cache line having the requested data is present and valid in at least one cache. If the probe reveals that no such valid cache line is cached in the cache hierarchy 118, the L3 cache 206 forwards the memory request to the interconnect fabric 110 for fulfillment by the local memory 112. At the same time, the latency tracker 230 allocates an entry 236 in a tracker table 234 for the direct request to memory.
[0019] At block 306, latency tracker 230 tracks the time it takes for the memory request to be satisfied and stores a value representing this latency in the entry 236 allocated for the memory request. In one embodiment, latency tracker 230 stores a start timestamp obtained from a local clock source in latency field 240 when the memory request is issued from the computer complex, stores a stop timestamp obtained from a local clock source (e.g., a system time clock) when data from the memory request is received from the computer complex's local memory 112, calculates the latency as the difference between these two timestamps, and stores a value representing this difference in latency field 240.
[0020] The process of blocks 304 and 306 is repeated for some or all of the direct memory requests to memory issued to the processor core 200 so as to populate a plurality of entries 236 with latency values for the direct memory requests to memory. In some embodiments, each direct memory request to memory is monitored and recorded in the tracker table 234, while in other embodiments, the direct memory requests to memory are sampled at some fixed or dynamically adjustable rate (e.g., one in five direct access requests to memory are monitored, or only direct memory requests to memory to a particular address range are monitored, etc.) to determine a sample set of direct memory requests to memory to track, and this sample set is maintained and updated in the tracker table 234. The tracker table 234 has a limited number of entries 236, and therefore, to perform the sliding window analysis, the latency tracker 230 performs a least recently used (LRU) process to overwrite older entries 236 with newly identified direct memory requests to memory when the tracker table 234 is full.
[0021] In block 308, periodically or in response to some trigger, the latency tracker 230 evaluates the latency values in the latency fields 240 of the occupancy entries 236 (or a subset thereof) of the tracker table 234 to determine an updated access latency metric representing the latency of recent memory requests issued by the processor core 200 to the local memory 112. In some embodiments, the latency tracker 230 determines a current average or nominal latency value from the considered access latency values and provides a representation of this current average latency as the updated access latency metric. In other embodiments, the latency tracker 230 determines the updated access latency metric based on a median access latency, based on a maximum access latency, based on a minimum access latency, based on a trimmed average access latency, etc. The updated access latency metric is then made available to the throttle controller 228 and block 308 is repeated for the next update iteration.
[0022] Returning to the prefetch accuracy monitoring process 302, the process is performed in parallel for each prefetcher (e.g., prefetcher 220 of L1 cache 202 and prefetcher 222 of L2 cache 204) monitored by prefetch accuracy tracker 232. Additionally, as described above, prefetch accuracy may be monitored per thread or per core, depending on the configuration. For illustrative purposes, process 302 is described per thread, but this same approach is easily adapted to a per core approach using the guidelines described herein. Process 302 begins at block 310, where a target prefetcher performs a prefetch algorithm to identify a next cache line to speculatively prefetch in prediction that data in the cache line will eventually be requested by the corresponding thread, and issues a prefetched memory request to access that cache line from local memory 112. In response to issuing the prefetch type memory request, the prefetch precision tracker 232 assigns an entry 244 in the precision table 242 to the prefetch type memory request in association with the corresponding thread.
[0023] In block 312, prefetch precision tracker 232 determines the prefetch precision of the prefetch-type memory request indicated in precision table 242. To this end, prefetch precision tracker 232 monitors demand-type memory requests issued on behalf of a thread, and when a demand-type memory request is issued for a cache line that was the target of a prefetch-type memory request indicated in precision table 242, prefetch precision tracker 232 updates precision field 248 of entry 244 assigned to that prefetch-type memory request (e.g., writes “1” to field 248) to reflect that the prefetch of that cache line was a precise prefetch. Otherwise, if the prefetched cache line is not the target of a subsequent demand-type memory request within a certain period of time (i.e., the thread does not "use" the prefetched data), then prefetch precision tracker 232 updates precision field 248 to reflect that the prefetch of that cache line was inaccurate or incorrect (e.g., by writing or maintaining a "0" in field 248).
[0024] At block 314, periodically or in response to a trigger, prefetch precision tracker 232 updates the prefetch precision metric for the thread based on precision table 242, such that the prefetch precision metric represents the precision of the prefetcher in prefetching cache lines for the thread. Because prefetch precision can be expressed as a ratio of the total number of accurate prefetches for the thread to the total number of prefetches for the thread, in some embodiments, prefetch precision tracker 232 determines the total number of entries 244 for the thread in precision table 242 that indicate accurate prefetches (e.g., have a "1" in precision field 248 of the corresponding entry 244), determines the total number of entries 244 for the thread in precision table 242, and determines the current value of the prefetch precision metric based on the ratio of these two numbers. The process of block 314 is repeated for some or all of the threads monitored by prefetch precision tracker 232.
[0025] Concurrently with processes 301 and 302, an iteration of throttle level setting process 303 is performed by throttle controller 228 periodically or in response to other triggering events. An iteration of process 303 is represented by block 316, in which throttle controller 228 sets a throttle level for a thread, processor core, or other memory resource consumer based on one or both of the memory resource consumer's current access latency metric and prefetch accuracy metric in the most recent iteration of each of processes 301 and 302. For ease of explanation, the following assumes that the memory resource consumer is a thread of processor core 200, although the techniques described herein are readily adapted for use in embodiments in which the memory resource consumer is a processor core or other memory resource consumer, using the guidelines provided herein.
[0026] A given throttle level set for a thread operates to limit the rate at which memory requests are issued to the local memory 112 on behalf of the thread, thus limiting the amount of bandwidth the thread uses between the computing complex using the processor cores and the local memory 112. This limit may be achieved in any of a variety of ways, or a combination thereof. As shown in block 318, the throttle level may set or modify the number of pending memory requests available to the thread at any given time, thereby limiting the rate at which memory requests can be issued on behalf of the thread. As described above, the MIB 224 (FIG. 2) acts as a buffer for the L2 cache 204 that stores memory requests that miss in the cache hierarchy 118, awaiting issue to the local memory 112 to satisfy the memory request. Thus, in one embodiment, the cache controller 216 for the L2 cache 204 may modify the number of MIB entries 226 available to the thread based on the throttle level set for the thread. Once a thread has used all of the MIB entries 226 available to it, the thread cannot issue any more memory requests to the local memory 112; therefore, limiting the number of MIB entries 226 available to a thread acts to control the maximum number of memory requests that a thread can issue to the local memory 112.
[0027] As represented by blocks 320, 322, and 324, the throttle level set for a given thread may also modify the prefetcher aggressiveness in issuing prefetches for that thread. The prefetch algorithm implemented by the prefetcher is controlled by several factors, including prefetch confidence and prefetch distance. Prefetch confidence refers to a measure of confidence that the prefetched data will subsequently be used by the thread, and the prefetcher algorithm is typically configured such that prefetches that may have a prefetch confidence below a certain threshold are not permitted to be issued as speculatively prefetched memory requests. Prefetch distance reflects how far ahead the speculative memory request stream of a thread that the prefetcher has allowed to prefetch is, and thus effectively limits the rate at which prefetch memory requests are issued on behalf of the thread in combination with the prefetch confidence. Thus, the "aggressiveness" of the prefetcher is reflected by one or both of the prefetch confidence threshold and prefetch distance used by the prefetcher. Thus, in some embodiments, prefetcher aggressiveness is throttled by one or more of adjusting the prefetcher distance at block 320, adjusting the prefetcher minimum confidence threshold (block 322), and enabling or disabling prefetching globally. By varying the prefetcher aggressiveness in one or more of these ways, throttle controller 228 effectively varies the rate at which the prefetcher issues prefetched memory requests on behalf of a thread, which in turn effectively varies memory bandwidth utilization attributable to the thread.
[0028] As described above, in some embodiments, there are multiple prefetchers, such as prefetcher 220 for L1 cache 202 and prefetcher 222 for L2 cache 204, that operate on behalf of a thread and attempt to prefetch data for the thread. However, these prefetchers may have different prefetcher accuracy for a thread for various reasons. Thus, in some embodiments, throttle controller 228 sets different throttle levels for different prefetchers for the same thread based on each prefetcher accuracy. In other embodiments, throttle controller 228 selects a single throttle level, e.g., based on best or worst prefetch accuracy, and applies this same throttle level to each prefetcher for a thread.
[0029] The throttle controller 228 may select a throttle level using any of a variety of techniques. In some embodiments, the throttle controller 228 uses a throttle level table 250 that specifies corresponding throttle levels for corresponding pairs of access latency ranges and prefetch accuracy ranges. Thus, to determine the throttle level to set for a thread, the throttle controller 228 performs a lookup in the throttle level table 250 based on the thread's current access latency metric and current prefetch accuracy metric to determine the corresponding throttle level to set for the thread. Table 1 below illustrates an example embodiment of the throttle level table 250. [Table 1]
[0030] Note that throttle level 0 is no throttling, or the default throttling, for a thread.
[0031] In other embodiments, rather than using a particular throttle level for a given set of metrics, the throttle controller 228 iteratively increases the severity of the throttling for a thread until the thread's metrics meet a particular threshold. For example, in one embodiment, the throttle controller 228 may determine the average prefetch accuracy of the prefetchers for all threads it is tracking, and when a given thread's prefetch accuracy metric exceeds this average prefetch accuracy by some amount (e.g., 150% of the average prefetch accuracy), the throttle controller 228 may iteratively increase the severity of the thread's memory request throttling until the thread's updated prefetch accuracy metric falls below this threshold.
[0032] FIG. 4 illustrates a method 400 for providing a software-defined memory bandwidth limit to a thread, according to some embodiments. As described above, in some embodiments, a software application (e.g., software application 136 of FIG. 1) sets a target memory bandwidth utilization limit for a thread of the software application executed by processor core 200 of a computer complex. Thus, method 400 begins at block 402, where the software application sets this target limit, e.g., by executing one or more instructions that cause processor core 200 to store a value representing the limit in storage element 252 (FIG. 2). In some embodiments, this value represents a particular maximum memory request issuance rate of the thread (e.g., X direct memory requests to memory per microsecond). In response, throttle controller 228 sets and enforces an initial throttle level for the thread based on the software-defined target limit value. In some embodiments, the initial throttle level is set to a default throttle level (e.g., throttle level 0 or no throttling), while in other embodiments, the throttle controller 228 estimates a corresponding throttle level appropriate for the target bandwidth limit based on the current access latency metrics of the computer complex or processor core 200.
[0033] In block 406, the latency tracker 230 monitors the current actual memory bandwidth utilization by the thread, for example, by tracking the current rate of direct memory requests to memory issued on behalf of the thread. In block 408, the throttle controller 228 compares the monitored actual memory bandwidth utilization with a software-set target memory utilization bandwidth limit. If the software-defined target memory bandwidth utilization limit is met, in block 410, the throttle controller 228 adjusts the throttle level of the thread to a less restrictive throttle level and returns to monitoring the thread at the less restrictive throttle level in block 406. Otherwise, if the software-defined target limit is not met by the monitored actual memory bandwidth utilization, in block 412, the throttle controller 228 adjusts the throttle level of the thread to a more restrictive throttle level and returns to block 406 to monitor the thread at the more restrictive throttle level. In this way, the throttle controller 228 dials in a throttle level appropriate for the software-defined target limit. As an example using Table 1 above, assume that in block 404, the throttle controller 228 sets the initial throttle level for a thread to throttle level 5. If, at throttle level 5, the thread issues direct requests to memory at a rate less than the maximum value represented by the software-defined limit, then the throttle controller 228 may decrement the throttle level to throttle level 4 in block 410. Otherwise, if the rate of memory request issuance exceeds the maximum value, then the throttle controller 228 may increment the throttle level to throttle level 5 in block 412, thereby further throttling the rate of memory request issuance to satisfy the software-defined limit.
[0034] In some embodiments, memory bandwidth throttling subsystem 134 may implement both dynamic hardware controlled memory request throttling as represented by method 300 and software defined memory request throttling as represented by method 400. In such cases, the software defined target memory bandwidth utilization limit controls the least restrictive memory throttle level that may be used by throttle controller 228 according to method 400, and throttle controller 228 can implement a more restrictive throttle level according to dynamic throttle level setting method 300.
[0035] In some embodiments, some aspects of the above techniques may be implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored in or tangibly embodied on a non-transitory computer-readable storage medium. The software may include instructions and specific data that, when executed by one or more processors, operate the one or more processors to perform one or more aspects of the above techniques. The non-transitory computer-readable storage medium may include, for example, a magnetic or optical disk storage device, a solid-state storage device such as a flash memory, a cache, a random access memory (RAM), or other non-volatile memory device, etc. The executable instructions stored on the non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats that can be interpreted or executed by one or more processors.
[0036] In addition to the above, it should be noted that not all activities or elements described in the summary description are required, some of the specific activities or devices may not be required, one or more additional activities may be performed, and one or more additional elements may be included. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, those skilled in the art will appreciate that various changes and modifications can be made without departing from the scope of the invention as set forth in the claims. Thus, the specification and drawings should be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the invention.
[0037] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, the benefits, advantages, solutions to problems, and features by which any benefit, advantage, or solution may occur or be manifested are not to be construed as critical, essential, or essential features of any or all claims. Moreover, the specific embodiments described above are illustrative only, as the disclosed invention may be modified and practiced in different but similar manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as set forth in the appended claims. It is therefore apparent that the specific embodiments described above may be altered or modified, and all such variations are considered to be within the scope of the disclosed invention. Accordingly, the protection sought herein is set forth in the appended claims.
Claims
1. 1. A method for throttling memory requests in a processing system, comprising: determining an access latency metric representative of an average access latency of memory requests of a processor core of the processing system; determining a prefetch accuracy metric representative of an accuracy of a prefetcher for a cache associated with the processor core; throttling a rate of memory requests that can be issued by the processor core by setting a throttle level based on at least one of the access latency metric and the prefetch accuracy metric; setting the throttle level includes setting a maximum number of pending memory requests available to the processor core. method.
2. executing a software application on the processor core, the software application including at least one instruction for configuring the processor core to set a target memory utilization bandwidth limit for a thread of the software application; monitoring actual memory utilization bandwidth of said threads in a computer complex including said processor core; and incrementally modifying the throttle level set for the thread until the monitored actual memory utilization bandwidth meets the target memory utilization bandwidth limit; Incrementally modifying the throttle level includes configuring at least one of a prefetcher aggressiveness for the cache associated with the thread and a maximum number of pending memory requests available to the thread.
2. The method of claim 1.
3. Setting the throttle level comprises: setting a prefetcher aggressiveness; 2. The method of claim 1.
4. Setting the prefetcher aggressiveness includes: setting a maximum prefetch distance for the prefetcher; setting the prefetcher to be enabled or disabled; At least one of The method of claim 3.
5. Throttling a rate of memory requests that can be issued by the processor cores includes: setting a maximum number of pending memory requests available to the processor core; 2. The method of claim 1.
6. Determining the prefetch accuracy metric includes determining the prefetch accuracy metric for a thread executing on the processor core; throttling a rate of memory requests that can be issued by the processor core includes throttling a rate of memory requests that can be issued by the processor core for the thread.
2. The method of claim 1.
7. Determining the access latency metric comprises: Sampling a plurality of memory requests issued to a local memory associated with the processor core to generate a sample set of memory requests; measuring, for each memory request in the sample set, a corresponding access latency for satisfying the memory request; determining the access latency metric based on an average of access latencies measured for a sample set of the memory requests.
2. The method of claim 1.
8. Throttling a rate of memory requests that can be issued by the processor cores includes: accessing a data structure representing a plurality of throttle levels, each throttle level including at least one of a maximum number of pending memory requests available and a level of prefetcher aggressiveness, each throttle level being associated with at least one of a corresponding latency threshold and a corresponding prefetch accuracy threshold; selecting a throttle level to implement for the processor core based on at least one of a comparison of the access latency metric to a latency threshold for each of the plurality of throttle levels, and a comparison of the prefetch precision metric to a prefetch precision threshold for each of the plurality of throttle levels.
2. The method of claim 1.
9. an interconnect fabric connectable to a local memory; at least one computer complex connected to the interconnect fabric, The computer complex comprises: A processor core; a cache hierarchy; The cache hierarchy comprises: Multiple caches and a throttle controller configured to throttle a rate of memory requests that can be issued by the processor core by setting a throttle level based on at least one of an access latency metric and a prefetch accuracy metric to set a maximum number of pending memory requests available to the processor core; the access latency metric represents an average access latency for memory requests of the processor core; the prefetch accuracy metric represents an accuracy of a prefetcher of a cache in the cache hierarchy; Processing system.
10. Setting the throttle level comprises: Setting the prefetcher aggressiveness, The processing system of claim 9.
11. The throttle controller is setting a maximum prefetch distance for the prefetcher; setting the prefetcher to be enabled or disabled; and setting the prefetcher aggressiveness by at least one of the following: The processing system of claim 10.
12. the maximum number of pending memory requests is a maximum number of miss information buffer (MIB) entries available to the processor core; The processing system of claim 10.
13. The throttle controller is By varying the maximum number of pending memory requests available to said processor core, configured to throttle a rate of memory requests that can be issued by the processor core; The processing system of claim 9.
14. The cache hierarchy comprises: a prefetch accuracy tracker configured to determine the prefetch accuracy metrics for threads executing on the processor core; the throttle controller is configured to throttle a rate of memory requests that can be issued by the processor core for the thread; The processing system of claim 9.
15. The cache hierarchy comprises: Sampling a plurality of memory requests issued to a local memory associated with the processor core to generate a sample set of memory requests; measuring, for each memory request in the sample set, a corresponding access latency for satisfying the memory request; determining the access latency metric based on an average of access latencies measured for a sample set of memory requests; a latency tracker for determining the access latency metric by The processing system of claim 9.
Citation Information
Patent Citations
Data storage system service quality controller and service quality method
JP2006521640A
Method, apparatus, and a system for a software configurable prefetcher
US20070094453A1
Prefetch throttling
US20140108740A1
