Allocation control for cache

Abstract

A technique for operating a cache is disclosed. The technique includes identifying a first allocation permission policy based on a workload change; operating the cache according to the first distribution permission strategy; identifying a second allocation permission policy based on the set sampling; and operating the cache according to the second allocation permission policy.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Non-Provisional Application No. 17 / 852,296, filed on June 28, 2022, the entire contents of which are hereby incorporated by reference. Background Art

[0003] Caching improves performance by storing copies of data that is considered likely to be accessed again in the future in low-latency cache memory. Improvements in cache technology are continually being made. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] A more detailed understanding may be obtained from the following description given by way of example in conjunction with the accompanying drawings, in which:

[0005] Figure 1 is a block diagram of an example computing device that may implement one or more features of the present disclosure;

[0006] Figure 2 Elements of a device associated with a last level cache are shown according to one example;

[0007] Figure 3 shows a collection sampling operation according to an example;

[0008] Figure 4 The operation of the system according to one example is shown; and

[0009] Figure 5 is a flow chart of a method for operating a cache according to one example. DETAILED DESCRIPTION

[0010] A technology for operating a cache is disclosed, which includes identifying a first allocation authority policy based on workload changes; operating the cache according to the first allocation authority policy; identifying a second allocation authority policy based on set sampling; and operating the cache according to the second allocation authority policy.

[0011] Figure 11 is a block diagram of an example computing device 100 that may implement one or more features of the present disclosure. In various examples, computing device 100 is, for example, but not limited to, one of a computer, a gaming device, a handheld device, a set-top box, a television, a mobile phone, a tablet computer, or other computing device. Device 100 includes, but is not limited to, one or more processors 102, a memory 104, one or more auxiliary devices 106, a storage device 108, and a last level cache (“LLC”) 110. An interconnect 112, which may be a bus, a combination of busses, and / or any other communication component, communicatively links one or more processors 102, memory 104, one or more auxiliary devices 106, a storage device 108, and last level cache 110.

[0012] In various alternatives, the one or more processors 102 include a central processing unit (CPU), a graphics processing unit (GPU), a CPU and a GPU on the same die, or one or more processor cores, where each processor core may be a CPU, a GPU, or a neural processor. In various alternatives, at least a portion of the memory 104 is located on the same die as one or more of the one or more processors 102, such as on the same chip or in an interposer arrangement, and / or at least a portion of the memory 104 is located separately from the one or more processors 102. The memory 104 includes volatile or non-volatile memory, such as random access memory (RAM), dynamic RAM, or cache.

[0013] The storage device 108 includes a fixed or removable storage device, such as, but not limited to, a hard disk drive, a solid state drive, an optical disk, or a flash drive. The one or more auxiliary devices 106 include, but are not limited to, one or more auxiliary processors 114 and / or one or more input / output ("IO") devices. The auxiliary processor 114 includes, but is not limited to, a processing unit capable of executing instructions, such as a central processing unit, a graphics processing unit, a parallel processing unit capable of performing compute shader operations in a single instruction multiple data form, a multimedia accelerator such as a video encoding or decoding accelerator, or any other processor. Any auxiliary processor 114 may be implemented as a programmable processor that executes instructions, a fixed function processor that processes data according to fixed hardware circuits, a combination thereof, or any other type of processor.

[0014] The one or more IO devices 116 include one or more input devices, such as a keyboard, a keypad, a touch screen, a touchpad, a detector, a microphone, an accelerometer, a gyroscope, a biometric scanner, or a network connection (e.g., a wireless LAN card for sending and / or receiving wireless IEEE 802 signals), and / or one or more output devices, such as a display, a speaker, a printer, a tactile feedback device, one or more lights, an antenna, or a network connection (e.g., a wireless LAN card for sending and / or receiving wireless IEEE 802 signals).

[0015] The last level cache 110 acts as a shared cache for various components of the device 100, such as the processor 102 and various auxiliary devices 106. In some implementations, other caches are present in the device 100. For example, in some examples, the processor 102 includes a cache hierarchy that includes different levels, such as level 1 and level 2. In some examples, each such cache level is specific to a particular logical partition of the processor 102, such as a processor core, or a processor chip, die, or package. In some examples, the hierarchy also includes other types of caches. In various examples, one or more of the auxiliary devices 106 include one or more caches.

[0016] In some examples, the last level cache 110 is "last level" in the sense that such cache is the last cache that the device 100 attempts to service a memory access request before servicing the request from the memory 104 itself. For example, if the processor 102 accesses data that is not stored in any of the cache levels of the processor 102, the processor exports a memory access request to be satisfied by the last level cache 110. The last level cache 110 determines whether the requested data is stored in the last level cache 110. If the data is within the last level cache 110, the last level cache 110 services the request by providing the requested data from the last level cache 110. If the data is not within the last level cache 110, the device 100 services the request from the memory 104. It can be seen that in some specific implementations, the last level cache 110 acts as the final cache level before the memory 104, which helps to reduce the total amount of memory access latency for accesses to the memory 104. Although techniques are described herein as involving the operation of last level cache 100, it should be appreciated that these techniques may alternatively be used with other types of caches or memories.

[0017] Figure 2Elements of the device 100 associated with the last level cache 110 are shown according to one example. These elements include the last level cache 110, the client 212, the cache controller 202, and the workload monitor 204.

[0018] The last level cache 110 is shared among many clients 212 of the device 100. As used herein, the term "client" refers to any element, such as an element of the device 100, that requests access to the last level cache 110. In various examples, the client 212 includes one or more software elements (e.g., an operating system, a driver, an application, a thread, a process, or firmware) executing on a processor such as the processor 102, one or more hardware elements such as the processor 102 or the auxiliary device 106, or a combination of software and hardware.

[0019] The last level cache 110 has a limited size. With more contention for the last level cache 110, cache thrashing may occur, resulting in reduced performance for the client 212. Therefore, in some cases, it is beneficial to allow certain types of clients 212, but not other types of clients 212, to allocate entries into the cache 110. Allocation occurs in response to a miss of the client 212. Specifically, in some cases, when a memory access request of the client 212 does not hit in the cache 110, the cache 110 allocates an entry within the cache 110, obtains the data targeted by the memory access request from the memory 104, and places the data into the allocated entry. In the case where there are no free entries in the cache 110, the cache 110 evicts the data from the entry to the memory 104 and allocates the entry to the new data. Cache thrashing occurs when there is too much cache contention, resulting in too many misses, with the result that the data does not stay in the cache for a long time, thereby reducing the effectiveness of the cache as a means of reducing memory access latency.

[0020] Therefore, the present invention provides a technique for reducing cache thrashing by limiting which clients 212 are allowed to be allocated to the cache 110 based on the operating conditions of the device 100. Similarly, in response to a miss occurring, allocation to the cache means designating an entry of the cache 110 to store the missed data obtained from the memory 104. In the event that there are no free (e.g., invalid) entries to store the requested data, the allocation results in eviction. In some specific implementations or operating modes, a client 212 that is not allowed to be allocated to the cache 110 is still allowed to access data in the cache in other ways, such as obtaining data already stored in the cache, or modifying data already stored in the cache. However, in the event of a miss, such a client 212 will not cause new data to be brought into the cache 110.

[0021] Techniques for limiting which clients 212 are allowed to allocate include determining which clients 212 are allowed to allocate into the cache 110, and allowing or denying allocations to those clients 212 accordingly. In some implementations, determining which clients 212 are allowed to allocate is based on input from a workload monitor 204 and a cache controller 202 that performs set sampling. The workload monitor 204 is one or more of software, hardware (e.g., circuitry), or a combination thereof. In some examples, at least a portion of the workload monitor 204 is part of a driver or operating system executed on the processor 102. In some examples, the workload monitor 204 is alternatively a hardware circuit or includes a hardware circuit. The cache controller 202 is similarly a hardware circuit, a software entity, or a combination thereof.

[0022] The workload monitor 204 monitors the workloads being executed within the device 100. In some examples, each different workload is defined by which applications are executing on the processor 102 and / or which clients 212 are "active." A client 212 is active if the client is powered on and performs at least a threshold amount of work, where the threshold can be predefined and / or dynamically adjusted. In some examples, a first type of workload, which is a gaming workload, includes a gaming application executing on the processor 102, and a graphics processing unit (which is an auxiliary processor 114 and client 212 of the LLC 110) is active. In another example, a second type of workload, which is a video playback workload, includes video player software executing in the processor 102, and a video decoder (which is an auxiliary processor 114 and client 212 of the LLC 110) is active. In some example workloads, multiple different clients 212 are active and therefore compete for the LLC 110.

[0023] The workload monitor 204 maintains allowed client data 206 that indicates, for each of a plurality of workloads, which clients 212 are denied allocation of entries in the LLC 212 while the device 100 is executing under the workload. For example, for a workload in which the processor 102 is executing a game and a client 212 including a graphics processing unit and an audio hardware device is active, the allowed client data 206 indicates that the graphics processing unit and the processor 102 are permitted to be allocated into the LLC 110, but the audio hardware device is not permitted to be allocated into the LLC 110. In another example, where audio playback software is being executed on the processor 102 and the audio hardware device is active, and no game is being executed, but the graphics processing unit is intermittently used (and therefore active), the allowed client data 206 indicates that the audio hardware is permitted to be allocated into the LLC 110. Similarly, for each of a plurality of workloads, the allowed client data 206 indicates which of the one or more clients 212 are permitted to be allocated into the LLC 110. In various examples, the allowed client data 206 is stored in a memory specifically associated with the workload monitor 204 (e.g., in some embodiments where the workload monitor 204 is a hardware unit, stored within the workload monitor 204), or in a different memory, such as system memory 104 or a different memory.

[0024] The cache controller 202 performs collective sampling within the cache 110. The cache controller 202 grants or denies clients 212 based on the monitored workload and the collective sampling. More specifically, the workload monitor 204 determines when a workload switch occurs. In response to the workload switch, the workload monitor 204 queries a set of data (allowed client data 206) that indicates which clients 212 are allowed to be allocated into the cache 110 for the current workload. The workload monitor 204 then grants or denies allocations to the clients 212 accordingly. During any particular period of time when the workload does not change, the cache controller 202 performs collective sampling to identify a set of clients 212 that should be allowed to be allocated and / or a set of clients 212 that should not be allowed to be allocated.

[0025] Generally speaking, set sampling involves reserving a small portion of the cache 110 for testing different configurations, and operating the cache 110 in a manner that is considered optimal based on the tested configurations. As is well known, a set associative cache is divided into a plurality of sets, each of which has one or more ways. Set sampling involves using a small portion of these sets (the "test set") to test different allocation techniques, and periodically adjusting the non-test set (i.e., the set of cache 110 other than the test set) to use the allocation technique that is considered optimal. "Allocation technique" refers to which clients 212 are allowed and which clients are not allowed to be allocated to the LLC 110. Set sampling provides the benefit of adjusting the operation of the cache 110 to take into account current operating conditions, but sometimes there will be delays because the cache controller 202 may need some time to "recognize" that a particular allocation technique is more optimal than the technique currently being used in the non-test.

[0026] Figure 3 A set sampling operation according to an example is shown. LLC 110 includes a plurality of non-test sets 304 and a plurality of test sets 306. Set 302 is a set in a set associative cache scheme. Such a scheme is a scheme in which data to be placed in the cache (i.e., in response to a miss) is placed in one of the ways in a particular set based on the address of the data. In one example, for a miss at a cache line with an address, the cache fetches a cache line from memory 104 and places the cache line in one of the ways in the set determined by some bits of the address.

[0027] For non-test sets 304, the cache controller 202 operates these sets according to the current allocation authority policy. The allocation authority policy indicates which clients 212 are allowed to be allocated to the last level cache 110. For test sets 306, the cache controller 202 operates these sets according to the candidate allocation authority policy. The cache controller 202 operates different test sets 306 according to different candidate allocation authority policies. Based on the performance measured with each test set 306, the cache controller 202 selects an allocation authority policy. In one example, the test set 306 selects an allocation authority policy for the test set 306 that is considered to exhibit the best performance. In one example, the test set 306 is considered to exhibit the best performance when it experiences the smallest miss percentage among all test sets 306, or experiences the highest hit percentage among all test sets 306. The miss percentage refers to the percentage of misses to the total number of accesses in a given time, and the hit rate refers to the percentage of hits to the number of accesses. In some examples, the cache controller 202 accumulates the hit percentage or the miss percentage over time, and after a period of time, selects a new allocation authority policy to operate the last level cache 110. The cache controller 202 then operates the last level cache 110 according to the policy. Operating according to the allocation authority policy means that allocations to the client 212 are not permitted as specified by the allocation authority policy.

[0028] Figure 4 The operation of the system 100 according to an example is shown. The last level cache 110 operates according to the current allocation rights policy at any given point in time. As described above, the allocation rights policy indicates which clients 212 are permitted to allocate entries into the LLC 110. Operating according to the current allocation rights policy means allowing or denying allocation to the client 212 according to the current allocation rights policy.

[0029] The cache controller 202 performs collective sampling on the test sets of the LLC 110. The cache controller 202 configures different test sets to operate with different allocation privilege policies. The cache controller 202 measures the performance of the test sets over time. At different times, the cache controller 202 selects the best performing test set and applies the allocation privilege policy of the test set to the non-test sets.

[0030] Aggregate sampling alone may inaccurately interpret aspects of the operation of the device 100. For example, when a workload on the device 100 switches, newly active clients 212 and / or newly executed software may utilize the cache 110 in a different manner than before the switch. However, aggregate sampling alone may not capture this new utilization pattern immediately or quickly. Therefore, the workload monitor 204 controls the allocation rights policy in the LLC 110 based on the monitored workload and the allowed client data 206. Therefore, the workload monitor 204 and the cache controller 202 operate together to select an allocation rights policy for operating the LLC 110. When the workload monitor 204 detects a workload change that will result in a different allocation rights policy, the workload monitor 204 causes the cache controller 202 to operate the LLC 110 based on the allocation rights policy. When the cache controller 202 determines that the LLC 110 should operate according to a new allocation rights policy based on aggregate sampling, the cache controller 202 causes the LLC 110 to operate according to the new rights policy.

[0031] Figure 5 is a flow chart of a method 500 for operating a cache according to one example. Figures 1 to 4 Although the system is described as a system for performing method 500, those skilled in the art will recognize that any system configured to perform the steps of method 500 in any technically feasible order falls within the scope of the present disclosure.

[0032] At step 502, a workload monitor 204 configured to monitor workload changes determines a workload change. Based on the workload change, the workload monitor 204 determines a new allocation authority policy. In some examples, the workload monitor 204 accesses allowed client data 206 to identify an allocation policy associated with the new workload. In some examples, the allowed client data 206 includes entries for each workload in a set of different workloads. For a specific workload, each entry indicates what allocation authority policy to use. In some examples, the workload monitor 204 communicates with hardware and / or software (e.g., an operating system or a driver) to determine the current workload. At step 504, in response to the workload change, the workload monitor 204 causes the allocation authority policy to change based on the new allocation authority policy.

[0033] At step 506, the cache controller 202 determines a new allocation permission policy based on the set sampling. In various examples, this determination occurs at various timing intervals, such as irregular or regular timing intervals. In each timing interval, the cache controller 202 collects test data indicating the performance of a given allocation permission policy in a number of test sets 306, and determines the allocation permission policy of the test set that is considered to perform optimally. In some examples, the test set 306 with the highest hit rate (e.g., the percentage of hits to total memory access requests) or the lowest miss rate (e.g., the percentage of misses to total memory access requests) is considered to perform optimally. At step 508, the cache controller 202 causes the LLC 110 to operate according to the selected allocation permission policy.

[0034] It should be understood that Figure 5 The order of the steps may be reversed or rearranged in any manner. Generally speaking, method 500 changes the current allocation permission policy in response to the emergence of a new workload, as the new workload is used to identify what the appropriate allocation permission policy should be. Then, based on set sampling, cache controller 202 uses set sampling to monitor actual cache performance in order to change the allocation permission policy when necessary, as actual operating conditions may indicate that a different allocation permission policy is needed. Changing the allocation permission policy in response to both actual performance and workload monitoring allows cache operation to be flexible and responsive to changing operating conditions.

[0035] Elements in the figures are embodied as software executing on a processor, a fixed function processor, a programmable processor, or a combination thereof, where appropriate. Processor 102, last level cache 110, interconnect 112, memory 104, storage 108, various auxiliary devices 106, client 212, cache controller 202, and workload monitor 204 include at least some hardware circuits and, in some implementations, include software executing on a processor within that component or within another component.

[0036] It should be understood that many variations are possible based on the disclosure herein. Although the above features and elements are described in particular combinations, each feature or element may be used alone without the other features and elements, or in various combinations with or without the other features or elements.

[0037] The provided methods may be implemented in a general purpose computer, a processor, or a processor core. Suitable processors include, by way of example, a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), and / or a state machine. Such processors may be manufactured by configuring a manufacturing process using the results of processed hardware description language (HDL) instructions and other intermediate data including a netlist (such instructions capable of being stored on a computer readable medium). The results of such processing may be masks, which are then used in a semiconductor manufacturing process to manufacture processors that implement the features of the present disclosure.

[0038] The methods or flow charts provided herein may be implemented in a computer program, software, or firmware incorporated into a non-transitory computer-readable storage medium for execution by a general-purpose computer or processor. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (such as internal hard disks and removable disks), magneto-optical media, and optical media (such as CD-ROM disks), and digital versatile disks (DVDs).

Claims

1. A method for operating a cache of a device, the method comprising: Based on the workload change, identifying a first allocation authority strategy; operating the cache according to the first assigned rights policy; Based on the set sampling, identifying a second allocation authority strategy; as well as The cache is operated according to the second assigned rights policy. 2 . The method of claim 1 , wherein the workload change comprises the device changing from operating according to a first workload to operating according to a second workload.

3. The method of claim 2, wherein the first workload comprises a first combination of an active client and software executing on the device; and the second workload comprises a second combination of an active client and software executing on the device. 4 . The method of claim 1 , wherein identifying the first allocation rights policy comprises identifying the first allocation rights policy based on currently active workloads by reference to a set of allowed client data.

5. The method of claim 1, wherein the collective sampling includes operating different test sets of the cache according to different allocation authority policies. 6 . The method of claim 5 , wherein identifying the second allocation rights policy comprises selecting an allocation rights policy for which observed performance is considered optimal.

7. The method of claim 5, wherein the different test sets comprise cache sets in a set associative scheme.

8. The method of claim 1, wherein the first allocation rights policy indicates which clients are allowed and which clients are not allowed to allocate entries in the cache.

9. The method of claim 1, wherein the cache is a last level cache.

10. A system, comprising: Cache; and A cache controller, the cache controller being configured to: Based on the workload change, identifying a first allocation authority strategy; operating the cache according to the first assigned rights policy; Based on the set sampling, identifying a second allocation authority strategy; as well as The cache is operated according to the second assigned rights policy.

11. The system of claim 10, wherein the workload change comprises the device changing from operating according to a first workload to operating according to a second workload.

12. The system of claim 11, wherein the first workload comprises a first combination of an active client and software executing on the device; and the second workload comprises a second combination of an active client and software executing on the device.

13. The system of claim 10, wherein identifying the first allocation rights policy comprises identifying the first allocation rights policy based on currently active workloads by referencing a set of allowed client data.

14. The system of claim 10, wherein the collective sampling includes different test sets operating the cache according to different allocation authority policies.

15. The system of claim 14, wherein identifying the second allocation rights policy comprises selecting an allocation rights policy for which observed performance is considered optimal.

16. The system of claim 14, wherein the different test sets comprise cache sets in a set associative scheme.

17. The system of claim 10, wherein the first allocation rights policy indicates which clients are allowed and which clients are not allowed to allocate entries in the cache.

18. The system of claim 10, wherein the cache is a last level cache.

19. A system, comprising: processor; a cache configured to service requests of the processor; and A cache controller, the cache controller being configured to: Based on the workload change, identifying a first allocation authority strategy; operating the cache according to the first assigned rights policy; Based on the set sampling, identifying a second allocation authority strategy; as well as The cache is operated according to the second assigned rights policy.

20. The system of claim 19, wherein the workload change comprises the device changing from operating according to a first workload to operating according to a second workload.

Images