Oldest operation wait time display input to Set Dueling

Abstract

A system, apparatus, and method are disclosed for dynamically adjusting cache policies to reduce latency for an execution core. A processor includes a cache subsystem. The cache subsystem includes one or more cache levels and one or more cache controllers. The cache controller divides the cache levels into two portions: a test portion and a remainder portion. The cache controller applies a first policy to the first test portion and a second policy to the second test portion. The cache controller determines the time the execution core spends waiting for access to the first test portion and the second test portion. If the measured latency is shorter for the first test portion than for the second test portion, the cache controller applies the first policy to the remainder portion. Otherwise, the cache controller applies the second policy to the remainder portion.

Description

[Background technology]

[0001] Most modern computing devices provide at least one level of cache memory (or cache) in addition to the computing device's main memory. In general, a cache is a small amount of fast-access memory used to store a limited number of copies of data and instructions used to perform various operations near the functional blocks of the computing device that perform the operations. A cache is usually implemented using high-speed memory circuits, such as static random access memory (SRAM) integrated circuits or other types of memory circuits. When a processor requests access to data stored in memory, the processor first determines whether a copy of the data is stored in the cache. If so, the processor accesses the cache to more efficiently access the data.

[0002] Advantages of the methods and mechanisms described herein may be better understood by referring to the following description in conjunction with the accompanying drawings. [Brief description of the drawings]

[0003] [Figure 1] FIG. 1 is a block diagram of one embodiment of a computing system. [Diagram 2] FIG. 2 is a block diagram of one embodiment of a processor. [Diagram 3] FIG. 2 is a block diagram of one embodiment of determining a performance-optimized policy for a cache. [Figure 4] FIG. 2 is a generalized flow diagram illustrating one embodiment of a method for determining a performance-optimizing policy for a cache. [Diagram 5] FIG. 2 is a generalized flow diagram illustrating one embodiment of a method for monitoring the oldest pending cache access. [Figure 6]FIG. 1 is a generalized flow diagram illustrating one embodiment of a method for dynamically adjusting cache policies to improve performance. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, those skilled in the art should appreciate that various embodiments may be practiced without these specific details. For example, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the approaches described herein. For simplicity and clarity of illustration, it should be understood that elements illustrated in the figures have not necessarily been drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements.

[0005] Various systems, apparatus, methods, and computer readable media for dynamically adjusting cache policies to reduce latency of an execution core are disclosed herein. In one embodiment, a processor includes at least an execution core and a cache subsystem. The cache subsystem includes one or more cache levels and one or more cache controllers. The cache controller divides the cache level into three parts, including two test parts and a third part different from the two test parts. The cache controller applies a first policy to the first test part and a second policy to the second test part. Each of the first and second policies specifies behaviors and attributes, such as cache line replacement and insertion policies, request priority, speculation hints, pre-emptive response generation, etc. For example, the pre-emptive response generation policy specifies whether the cache should preemptively send a response before the cache hit / miss status or the open / close status of the DRAM page is known. The cache controller also determines the time the execution core spends waiting to access the first and second test parts. In various embodiments, the cache controller itself monitors and / or measures this time. In other embodiments, another component monitors and / or measures this time, and the cache controller accesses the monitored and / or measured time (e.g., a time stored in a memory location, register, etc.). Various such embodiments are possible and contemplated. If the first test portion is shorter than the second test portion for the measured wait time, the cache controller applies the first policy to the third portion. Otherwise, if the second test portion is shorter than the first test portion for the measured wait time, the cache controller applies the second policy to the third portion. In some embodiments, the third portion represents the remainder of the cache (i.e., all of the remaining portion of the cache), and in some embodiments, the third portion represents less than the remaining portion of the cache.In the following description, reference is often made to the remainder of the cache for ease of explanation, but it should be understood that, depending on the embodiment, all of the remainder or less than all of the remainder is contemplated.

[0006] 1, a block diagram of one embodiment of a computing system 100 is shown. In one embodiment, computing system 100 includes at least processors 105A-105N, input / output (I / O) interface 120, bus 125, memory controller(s) 130A-130N, network interface 135, and memory device(s) 140A-140N. In other embodiments, computing system 100 includes other components (e.g., a display controller) and / or computing system 100 is configured differently. Processors 105A-105N represent any number of processors included in system 100.

[0007] In one embodiment, processor 105A is a general-purpose processor such as a central processing unit (CPU). In one embodiment, processor 105N is a data parallel processor with a highly parallel architecture. Data parallel processors include graphics processing units (GPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and the like. In one embodiment, processor 105A includes at least execution core(s) 110A, cache subsystem 115A, and memory controller(s) 130A. Similarly, processor 105N includes at least execution core(s) 110N, cache subsystem 115N, and memory controller(s) 130N. It should be noted that execution core(s) 110A-110N may be referred to herein as execution units. It should also be noted that processors 105A-105N may include other components that are not shown in order not to obscure the figure.

[0008] In one embodiment, cache subsystem 115A-115N includes control logic for monitoring the time that execution core(s) 110A-110N each spends waiting for a cache access. In one embodiment, the control logic applies different policies to different portions of a given cache level. Also in this embodiment, the control logic tracks the time that the execution core(s) spend waiting for access to two separate portions of a given cache level. If the time spent waiting for the first cache portion is less than the time spent waiting for the second cache portion, the control logic retrieves the policy applied to the first cache portion and applies this policy to the remaining portion of the given cache level (a third portion different from the first and second portions). Otherwise, if the time spent waiting for the second cache portion is less than the time spent waiting for the first cache portion, the control logic retrieves the policy applied to the second cache portion and applies this policy to the remaining portion of the given cache level. In other embodiments, the control logic monitors other numbers of portions of a given cache level while applying different policies to those portions.

[0009] Each of the memory controller(s) 130A-130N represents any number and type of memory controller coupled to any number and type of memory devices 140A-140N. Although each of the memory controller(s) 130A-130N is shown as being located within the processor 105A-105N, this illustrates just one embodiment. In another embodiment, each of the memory controller(s) 130A-130N is external to the processor 105A-105N. Also, it should be noted that a portion of each of the cache subsystems 115A-115N may be located within the execution core(s) 110A-110N, and another portion of the cache subsystems 115A-115N may be located external to the execution core(s) 110A-110N. For example, in one embodiment, a level 1 (L1) cache and a level 2 (L2) cache are located within the execution core(s) 110A-110N, while a last level cache (LLC) is located outside the execution core(s) 110A-110N.

[0010] Memory device(s) 140A-140N may represent any number and type of memory devices. For example, memory types in memory device(s) 140A-140N may include dynamic random access memory (DRAM), static random access memory (SRAM), NAND flash memory, NOR flash memory, or ferroelectric random access memory (FeRAM), etc. I / O interface 120 may represent any number and type of I / O interface, such as a Peripheral Component Interconnect (PCI) bus, PCI-Extended (PCI-X), PCI Express (PCIE) bus, Gigabit Ethernet (GBE) bus, or Universal Serial Bus (USB). Various types of peripheral devices (not shown) are coupled to I / O interface 120. Such peripheral devices may include, but are not limited to, a display, a keyboard, a mouse, a printer, a scanner, a joystick or other type of game controller, a media recording device, an external storage device, and a network interface card. The network interface 135 is used to send and receive network messages over the network.

[0011] In various embodiments, computing system 100 may be a computer, a laptop, a mobile device, a gaming console, a server, a streaming device, a wearable device, or any of a variety of other types of computing systems or computing devices. It should be noted that the number of components of computing system 100 may vary from embodiment to embodiment. For example, there may be more or fewer components than those shown in FIG. 1. It should also be noted that computing system 100 may include other components not shown in FIG. 1. Furthermore, in other embodiments, computing system 100 may be configured in a manner other than that shown in FIG. 1.

[0012] 2, a block diagram of one embodiment of a processor 200 is shown. In one embodiment, processor 200 includes at least an execution core 205, a cache subsystem 210, a control unit 215, and a memory subsystem 220. Note that processor 200 may also include other components not shown so as not to obscure the figure. Core 205 represents any number of processor cores for executing instructions of one or more programs and / or operating systems. In one embodiment, the circuitry of processor 200 is included in processor 105A and / or processor 105N (of FIG. 1).

[0013] In one embodiment, core 205 continuously transmits the physical address (or some other subset of identifiable information, such as a cache index) of the oldest pending cache access to control unit 215. For example, in one embodiment, the instruction cache or data cache transmits the oldest pending address of an access to a level 2 (L2) cache of cache subsystem 210 to control unit 215. In one embodiment, a valid bit is also transmitted to indicate that core 205 is waiting for this operation in a performance critical path. Control unit 215 monitors the IDs of these instructions, which control unit 215 divides into two or more buckets. Control unit 215 applies different policies to these buckets, such as cache replacement / insertion policies, request priorities across memory subsystem 220, hints on speculation on whether to send a preemptive response before a cache hit / miss condition or a DRAM page open / close condition is known. The control unit 215 counts the number of cycles that instructions in a given bucket spend waiting in the cache subsystem 210, and the control unit 215 uses the count to determine a performance-optimized policy. Note that while the control unit 215 is shown as being separate from the cache subsystem 210, in other embodiments the control unit 215 is integrated within the cache subsystem 210.

[0014] The policies assigned to different buckets affect different settings of the cache subsystem 210. For example, in one embodiment, there is an age associated with every cache line in an index, and that age is relative to other cache lines in that index. The age can be set to different values ​​when accessing the corresponding cache line or when accessing different cache lines. Also, the age is adjusted when performing different operations on the same index. Different policies can adjust these settings differently. For example, a line can be inserted into a cache and assigned an intermediate age with a first policy, while a second policy inserts the line as the latest age. With a third policy, the line may not be inserted into the cache, but instead the line may be inserted into the next level cache. For example, in one embodiment, the line is not allocated into the L2 cache, but the line is allocated into the level 3 (L3) cache. Different policies can adjust these settings as well as other settings and parameters associated with the cache, portions of the cache, cache lines, etc.

[0015] In one embodiment, a given level of cache subsystem 210 is divided into a first portion, a second portion, and a remaining portion. The first portion corresponds to a first subset of cache indexes, the second portion corresponds to a second subset of cache indexes, and the remaining portion corresponds to all other cache indexes that are not in the first subset or the second subset. In one embodiment, the first portion and the second portion are smaller than the remaining portion. In one embodiment, the control unit 215 applies a first policy to the first portion and a second policy to the second portion. The control unit 215 monitors the latency of instructions accessing cache indexes in the first portion and the second portion. For example, the control unit 215 calculates a first aggregate wait time for the first portion of the cache subsystem 210. Also, the control unit 215 calculates a second aggregate wait time for the second portion of the cache subsystem 210. If the first aggregate wait time is less than the second aggregate wait time, the control unit 215 applies the first policy to the remaining portion of the cache subsystem 210. Otherwise, if the second aggregate latency is less than the first aggregate latency, then the control unit 215 applies the first policy to the remainder of the cache subsystem 210. By tracking latency rather than tracking cache hit / miss measurements, the control unit 215 is able to better determine which behaviors are affecting performance, thereby applying a more performance-optimal policy to the remainder of the cache subsystem 210 than would be possible with traditional approaches.

[0016] Referring to FIG. 3, a block diagram of one embodiment of determining a performance-optimized policy for cache 300 is shown. Cache 300 represents a cache at any level in a cache hierarchy (e.g., cache subsystem 210 of FIG. 2). Cache 300 includes at least cache memory 310 and control unit 315. In one embodiment, cache memory 310 is divided into portion 320A, portion 320B, and remaining portion 320C. For example, in one embodiment, portion 320A includes a first subset of cache indexes, portion 320B includes a second subset of cache indexes, and remaining portion 320C includes all remaining cache indexes that are not in the first subset or the second subset. In another embodiment, portion 320A includes a first subset of cache indexes, portion 320B includes a second subset of cache indexes, and remaining portion 320C includes cache indexes that are not in the first subset or the second subset, but not necessarily all remaining cache indexes. Control unit 315 applies policy 330A to portion 320A, and control unit 315 applies policy 330B to portion 320B. For purposes of this description, it is assumed that policy 330A is different from policy 330B.

[0017] Depending on the embodiment, each of policies 330A and 330B specifies a different type of behavior and / or rules for the corresponding portion 320A, 320B. For example, policies 330A, 330B specify one or more replacement policies, insertion policies, request priorities, speculative hints, bypass decisions, etc. The control unit 315 counts the number of cycles that a given portion has an associated processor core wait time. For example, in one embodiment, wait time counter 325A counts the number of cycles that a request corresponding to a cache index in portion 320A spends waiting for a response from cache 300, and wait time counter 325B counts the number of cycles that a request corresponding to a cache index in portion 320B spends waiting for a response from cache 300. In one embodiment, the control unit 315 monitors the values ​​of wait time counter 325A and wait time counter 325B, and the counter with the lower value is deemed to have the better performance optimal policy. This better performance optimal policy is then applied to the remaining portion 320C. For example, if wait time counter 325A is less than wait time counter 325B after the predetermined test interval has elapsed, cache memory 310 applies policy 330A to the cache index of remaining portion 320C. Alternatively, if wait time counter 325B is less than wait time counter 325A after the predetermined test interval has elapsed, cache memory 310 applies policy 330B to the cache index of remaining portion 320C. If wait time counter 325B is equal to wait time counter 325A, control unit 315 extends the test interval and checks the values ​​of counters 325A-325B after the additional time has elapsed. Alternatively, instead of extending the test interval, in some embodiments, control unit 315 simply selects which policy to apply to remaining portion 320C.

[0018] In one embodiment, the control unit 315 resets the wait time counters 325A-325B on a predefined schedule, and then the control unit 315 runs the monitoring test again. In one embodiment, the control unit 315 uses the same policies 330A-330B as previously tested, while in another embodiment, the control unit 315 changes one or both of the policies 330A-330B for the new test. Note that the example with two separate portions 320A-320B and two corresponding policies 330A-330B merely illustrates one embodiment. In other embodiments, the cache memory 310 is divided into other numbers of portions, each portion having its own policy. The control unit 315 then selects the best policy to apply to the remaining portion 320C from these three or more policies based on which portion has the smallest wait time counter.

[0019] Referring to FIG. 4, one embodiment of a method 400 for determining a performance-optimized policy for a cache is shown. For purposes of explanation, the steps of this embodiment and the steps of FIGS. 5-6 are shown in sequence. However, it should be noted that in various embodiments of the described method, one or more of the elements described may occur simultaneously, may occur in a different order than that shown, or may be omitted entirely. Other additional elements may also occur as desired. Any of the various systems or devices described herein may be configured to perform the method 400.

[0020] The cache controller applies a first policy to the first cache portion, and the cache controller applies a second policy to the second cache portion (block 405). For purposes of this description, it is assumed that the second policy is different from the first policy. Each of the first and second policies specifies different types of settings and behaviors to follow for the corresponding cache portion. The policies specify behaviors such as cache replacement / insertion policies, request priorities across the memory subsystem, hints regarding speculation on whether to send a preemptive response before a cache hit / miss condition or a DRAM page open / close condition is known, etc. For example, in one embodiment, there is an age associated with every cache line at a given index, and the age is relative to other cache lines at the given index. Different policies can dictate what value the age is set to when accessing the corresponding cache line or when accessing a different cache line at the same index. In one scenario, when a line is inserted, the first policy sets the line to the middle age, and when a line is inserted, the second policy sets the line to the latest age. In this scenario, a third policy may specify that the line is not inserted into the cache, but instead is inserted into the next level cache.

[0021] The cache controller also monitors the time the execution core spends waiting to access the first and second cache portions (block 410). In one embodiment, the cache controller monitors the time from when the core issues a request to when the request is processed (e.g., when the core receives a response from the cache). For example, in one embodiment, a first time point is recorded when the core issues a request to the cache. Then, a second time point is recorded when the core receives a response from the cache. Then, in block 410, the difference between the second time point and the first time point is calculated and used as a measurement of time. In one embodiment, the difference is measured based on the number of clock cycles that have elapsed between the first time point and the second time point.

[0022] After the predetermined test interval has elapsed, the cache controller determines whether a first time that the execution core spends waiting for a request to access the first cache portion is greater than a second time that the execution core spends waiting for a request to access the second cache portion (block 415). In one embodiment, the first time is the average time per access that the execution core spends waiting in the first cache portion. In this embodiment, the second time is the average time per access that the execution core spends waiting in the second cache portion.

[0023] If the first time is greater than the second time (conditional block 420: "yes"), the cache controller applies the second policy to the remainder of the cache (block 430). Otherwise, if the first time is less than the second time (conditional block 420: "no"), the cache controller applies the first policy to the remainder of the cache (block 425). After blocks 425 and 430, the method 400 ends. If the first time is equal to the second time, the cache controller may extend the test interval. Alternatively, the cache controller may modify either the first policy or the second policy and rerun the test.

[0024] 5, one embodiment of a method 500 for monitoring the oldest pending cache access is shown. An execution unit transmits an identifier (ID) of the oldest pending cache access to a control unit (block 505). For example, in one embodiment, an instruction or data cache of an execution unit sends the ID of the oldest pending address to an L2 cache. The ID may be a physical address, a virtual address, a cache index, or some other attribute of the access. The execution unit also sends a valid bit with the ID to indicate whether a core is waiting for this access in a performance-critical path (block 510).

[0025] The control unit monitors the received IDs and classifies the IDs into two distinct groups (block 515). In some cases, the control unit groups the IDs into three or more distinct groups. The control unit applies a first behavior policy to the first group and the control unit applies a second behavior policy to the second group (block 520). The control unit then counts the number of cycles that accesses of each group cause the execution unit to wait (block 525). The control unit then applies the behavior policy of the group with the fewer wait cycles to the remainder of the cache (block 530). After block 530, the method 500 ends.

[0026] Referring to FIG. 6, one embodiment of a method 600 for dynamically adjusting a cache policy to improve performance is shown. The control unit tests a first policy of a first cache portion and tests a second policy of a second cache portion (block 605). The duration of the test period may vary depending on the embodiment. If the latency associated with the first cache portion is longer than the latency associated with the second cache portion (conditional block 610: "Yes"), the control unit applies the policy of the second cache portion to the remainder of the cache (block 630). Also, the control unit generates a new (third) policy different from the second policy and applies it to the second cache portion (block 620). In one embodiment, one or more attributes selected for the new policy are randomly generated. In another embodiment, one or more attributes are set to values predicted to improve the performance of the corresponding execution unit. Next, the control unit re-executes the test, in which case the previous policy is applied to the first cache portion and the new policy is applied to the second cache portion (block 625). After block 625, method 600 returns to conditional block 610.

[0027] If the wait time associated with the first cache portion is less than the wait time associated with the second cache portion (conditional block 610: "No"), the control unit applies the policy of the first cache portion to the remainder of the cache (block 615). The control unit also generates and applies a new policy to the first cache portion (block 635). The control unit then re-runs the test, where the new policy is applied to the first cache portion and the policy of the second cache portion remains unchanged (block 640). After block 640, the method 600 returns to conditional block 610. By using the method 600, the control unit can dynamically adjust the policy at run time to achieve improved performance. Rather than using a fixed or default policy, the policy can adapt to the operating conditions of the host computing system.

[0028] In various embodiments, program instructions of a software application are used to implement the methods and / or mechanisms described herein. For example, program instructions executable by a general purpose processor or a special purpose processor are contemplated. In various embodiments, such program instructions are expressed in a high level programming language. In other embodiments, the program instructions are compiled from the high level programming language into a binary, intermediate or other format. Alternatively, the program instructions are written to describe the operation or design of hardware. Such program instructions are expressed in a high level programming language such as C. Alternatively, a hardware design language (HDL) such as Verilog is used. In various embodiments, the program instructions are stored in any of a variety of non-transitory computer readable storage media. The storage media are accessible by the computing system during use to provide the program instructions to the computing system for program execution. Generally, such a computing system includes at least one or more memories and one or more processors configured to execute the program instructions.

[0029] It should be emphasized that the above-described embodiments are merely non-limiting examples of embodiments. Many variations and modifications will become apparent to those skilled in the art once the above disclosure is fully understood. It is intended that the following claims be interpreted to include all such variations and modifications.

Claims

1. 1. A cache subsystem, comprising: Cache and a cache controller coupled to the cache; The cache controller receiving a plurality of identifiers from an execution core corresponding to pending cache accesses; dividing the plurality of identifiers into a first bucket of identifiers and a second bucket of identifiers; applying a first policy to a first bucket of identifiers and a second policy to a second bucket of identifiers; partitioning the cache into a first cache portion corresponding to a first bucket of identifiers and a second cache portion corresponding to a second bucket of identifiers; monitoring a first time that the execution core spends waiting for an instruction to access the first cache portion; applying the first policy to a third cache portion in response to a predetermined condition being satisfied; 4. The method of claim 3, Cache subsystem.

2. the first policy specifies one or more of a cache line replacement policy, a cache line insertion policy, a request priority, a speculation hint generation, and a preemptive response generation for the first cache portion. The cache subsystem of claim 1.

3. each of the plurality of identifiers corresponds to a pending cache access identified by the execution core as being stale relative to other pending cache accesses; The cache subsystem of claim 1.

4. The cache controller monitoring a second time that the execution core spends waiting for an instruction to access the second cache portion; determining a condition where the first time period is less than the second time period; applying the second policy to the third cache portion in response to the first time being not less than the second time; 4. The method of claim 3, The cache subsystem of claim 3.

5. the cache controller, in response to the first time for the first cache portion being less than the second time for the second cache portion, generating a third policy; applying the third policy to the second cache portion and the first policy to the first cache portion; applying the third policy to the third cache portion in response to a third waiting time of the second cache portion being shorter than a fourth waiting time of the first cache portion; [0023] The cache subsystem of claim 4.

6. the cache controller includes a plurality of counters; each counter is configured to indicate a number of cycles that a request corresponding to a portion of the cache spends waiting for a response; The cache subsystem of claim 1.

7. each of the plurality of identifiers corresponds to an instruction identified by the execution core as being on a critical path; The cache subsystem of claim 1.

8. A method for detecting a cache access by a cache controller, comprising: receiving from an execution core a number of identifiers corresponding to pending cache accesses that are older than other pending cache accesses; the cache controller dividing the plurality of identifiers into a first bucket of identifiers and a second bucket of identifiers; the cache controller applying a first policy to a first bucket of identifiers and a second policy to a second bucket of identifiers; the cache controller partitioning the cache into a first cache portion corresponding to a first bucket of the identifiers and a second cache portion corresponding to a second bucket of the identifiers; monitoring a first time that the execution core spends waiting for an instruction to access the first cache portion; and applying the first policy to a third cache portion in response to a predetermined condition being satisfied. method.

9. the first policy specifies one or more of a cache line replacement policy, a cache line insertion policy, a request priority, a speculation hint generation, and a preemptive response generation for the first cache portion.

9. The method of claim 8.

10. each of the plurality of identifiers corresponds to a pending cache access identified by the execution core as being stale relative to other pending cache accesses; 9. The method of claim 8. monitoring a second time that the execution core spends waiting for an instruction to access the second cache portion; determining a condition where the first time period is less than the second time period; and applying the second policy to the third cache portion in response to the first time being not less than the second time. The method of claim 8.

12. The method comprises: in response to the first time for the first cache portion being less than the second time for the second cache portion; generating a third policy; applying the third policy to the second cache portion and the first policy to the first cache portion; and applying the third policy to the third cache portion in response to a third waiting time of the second cache portion being shorter than a fourth waiting time of the first cache portion.

12. The method of claim 11.

13. one or more attributes of the third policy are randomly generated.

13. The method of claim 12.

14. the first time period is equal to an average number of clock cycles that the execution core waits for access to the first cache portion; 9. The method of claim 8.

15. Memory, a processor coupled to the memory; A cache subsystem according to any one of claims 1 to 7. system.

Images