Cache management based on access type priority

Abstract

A system, apparatus, and method for cache management based on access type priority are disclosed. The system includes at least a processor and a cache. During the execution phase of a program, certain access types are more likely to cause demand hits in the cache than other access types. Demand hits are load hits and store hits to the cache. A runtime profiling mechanism is used to find access types that are more likely to cause demand hits. Based on the profiling results, cache lines that are likely to be accessed in the future are retained based on their most recent access type. The goal is to increase demand hits, thereby improving system performance. An efficient cache replacement policy can reduce redundant data movement, thereby improving system performance and reducing energy consumption.

Description

[Background technology]

[0001] This invention was made with Government support under the PathForward Project awarded by Lawrence Livermore National Security, under Prime Contract No. DE-AC52-07NA27344, Subcontract No. B620717, awarded by the U.S. Department of Energy (DOE). The U.S. Government has certain rights in this invention.

[0002] Description of Related Art Computer systems typically use main memory formed from inexpensive, high-density dynamic random access memory (DRAM) chips. However, DRAM chips suffer from relatively long access times. To improve performance, data processors typically include at least one local, high-speed memory known as a cache. A cache stores blocks of data that are frequently accessed by the processor. As used herein, a "block" is a set of bytes stored in contiguous memory that are treated as a unit for coherency purposes. As used herein, the terms "cache block," "block," "cache line," and "line" are each interchangeable. In some embodiments, a block may be the unit of cache allocation and deallocation. The number of bytes in a block varies as a design choice and may be any size. Additionally, the terms "cache tag," "cache line tag," and "cache block tag" are each interchangeable.

[0003] Since the storage capacity of a cache is limited, a cache management policy determines which cache line is selected for replacement when the corresponding cache area is full. To improve the performance of an application, an efficient cache management policy is important. However, some conventional cache management policies, such as policies based on the "least recently used" (LRU) principle, may be inefficient when handling irregular accesses to cache lines or may have limited applicability due to the need for relatively complex circuit implementation.

[0004] Advantages of the methods and mechanisms described herein may be better understood by referring to the following description in conjunction with the accompanying drawings. [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 is a block diagram of one embodiment of a computing system. [Diagram 2] FIG. 1 is a block diagram of one embodiment of a computing system. [Diagram 3] FIG. 2 is a block diagram of one embodiment of a cache that tracks the last access type for each cache line. [Figure 4] FIG. 2 is a block diagram of one embodiment of a cache that maintains counters that track the number of demand hits caused by each access type. [Diagram 5] FIG. 2 is a generalized flow diagram illustrating one embodiment of a method for setting cache line priority based on last access type. [Figure 6] FIG. 2 is a generalized flow diagram illustrating one embodiment of a method for selecting cache lines eligible for eviction. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, those skilled in the art will recognize that various embodiments may be practiced without these specific details. In some instances, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the techniques described herein. For simplicity and clarity of illustration, it should be understood that elements illustrated in the figures have not necessarily been drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements.

[0007] Various systems, apparatuses, and methods are disclosed herein for cache management based on access type priority. In one embodiment, the system includes at least a processor and a cache. During the execution phase of a program, certain access types are more likely to cause demand hits in the cache than other access types. Demand hits are load and store hits to the cache. A run-time profiling mechanism is used to find the access types that are more likely to cause demand hits. Based on the profiling results, cache lines that are likely to be accessed in the future are retained based on their most recent access type. The goal is to increase demand hits, thereby improving system performance. An efficient cache replacement policy can reduce redundant data movement, thereby improving system performance and reducing energy consumption.

[0008] 1, a block diagram of one embodiment of a computing system 100 is shown. In one embodiment, computing system 100 includes at least processor(s) 110, fabric 120, input / output (I / O) interface(s) 125, memory interface 130, peripheral device(s) 135, and memory subsystem 140. In other embodiments, computing system 100 may include other components, computing system 100 may lack the components described, and / or computing system 100 may be configured differently. In one embodiment, each processor 110 includes a cache subsystem 115. Cache subsystem 115 may have any number of cache levels having any of a variety of types of caches, which may vary depending on the embodiment. In some cases, one or more caches in the cache hierarchy of cache subsystem 115 may be located elsewhere, external to processor(s) 110.

[0009] In one embodiment, one or more caches of cache subsystem 115 use a cache management scheme based on access type priorities. For example, in one embodiment, the cache controller determines which access types are more likely to cause demand hits. A demand hit is a hit to the cache caused by a load or store operation. During the profiling phase, the cache controller determines which access type, among the multiple access types, caused the fewest number of demand hits. Then, during replacement, the cache controller attempts to evict cache lines that have a record of a last access type that matches the access type among the multiple access types that caused the fewest number of demand hits. In other words, the cache controller dynamically determines the type of access that is most likely to precede a load or store hit to the cache line. The cache controller then protects cache lines that were recently accessed by the access type that is most likely to precede a load or store hit. Further details of the techniques used to manage cache replacement policies based on access type priorities are provided throughout the remainder of this disclosure.

[0010] Processor(s) 110 represent any number and type of processing units (e.g., central processing unit (CPU), graphic processing unit (GPU), digital signal processor (DSP), field programmable gate array (FPGA), application specific integrated circuit (ASIC)). Memory subsystem 140 includes any number and type of memory devices. For example, memory types of memory subsystem 140 may include high bandwidth memory (HBM), non-volatile memory (NVM), dynamic random access memory (DRAM), static random access memory (SRAM), NAND flash memory, NOR flash memory, ferroelectric random access memory (FeRAM), etc. I / O interface(s) 125 represent any number and type of I / O interfaces (e.g., peripheral component interconnect (PCI) bus, PCI extended (PCI-X), PCI Express (PCI E) bus, Gigabit Ethernet (GBE) bus, universal serial bus (USB)). Various types of peripheral device(s) 135 may be coupled to I / O interface(s) 125. Such peripheral device(s) 135 include (but are not limited to) a display, a keyboard, a mouse, a printer, a scanner, a joystick or other type of game controller, a media recording device, an external storage device, a network interface card, etc.

[0011] In various embodiments, computing system 100 is a computer, a laptop, a mobile device, a game console, a server, a streaming device, a wearable device, or any of a variety of other types of computing systems or devices. It should be noted that the number of components of computing system 100 may vary from embodiment to embodiment. For example, in other embodiments, there may be more or fewer components than those shown in FIG. 1. It should also be noted that in other embodiments, computing system 100 includes other components not shown in FIG. 1. Additionally, in other embodiments, computing system 100 is configured in a manner other than that shown in FIG. 1.

[0012] Referring to Figure 2, a block diagram of one embodiment of a computing system 200 is shown. As shown, system 200 represents chips, circuits, components, etc. of a desktop computer 210, a laptop computer 220, a server 230, a mobile device 240, etc. Other devices are possible and contemplated. In the illustrated embodiment, system 200 includes at least one instance of cache subsystem 115 (of Figure 1). Although not shown in Figure 2, system 200 may include any number and type of other components, such as one or more processors, one or more memory devices, one or more peripheral devices, etc. Cache subsystem 115 includes any number of cache levels that employ a cache management scheme based on access type priority.

[0013] 3, a block diagram of one embodiment of a cache 300 that tracks the last access type of each cache line is shown. In various embodiments, cache 300 is a low-latency, high-bandwidth memory that is separate from the system memory. In some embodiments, cache 300 is used as a last-level cache in a cache memory subsystem (e.g., cache subsystem 115 of FIG. 1). In other embodiments, cache 300 is another level within the cache memory subsystem.

[0014] When a read or write request is received by the cache 300, a lookup in the tag store 302 is performed using the tag of the address that is the subject of the request. If the lookup result is a hit and an access is performed to a cache line in the data store 304, the access type is stored in the last access type field 308 of the corresponding entry 306 in the tag storage 302. In one embodiment, the tag store 302 and the data store 304 are organized as an array, although other configurations are possible and contemplated. In some embodiments, each of the tag storage 302 and the data storage 304 are separate entities, while in other embodiments they are combined into a single storage entity (device). In either case, the tag 302 and the data 304 stores may be referred to simply as data stores or data storage devices. In one embodiment, the different possible access types include load, store, prefetch, and writeback. As used herein, a "load operation" or "load access" is defined as an operation that specifies a transfer of data from a memory location to a processor or execution unit. A "store operation" or "store access" is defined as an operation that specifies the transfer of data from a processor or execution unit to a storage location. A "prefetch operation" or "prefetch access" is defined as an operation that specifies the transfer of data from a storage location to a cache before the data is requested by a demand operation. The data targeted by a prefetch access may end up unused by an execution unit if it is based on an incorrect prediction. A "writeback operation" or "writeback access" is defined as the transfer of a dirty cache line to a lower level cache or storage location. A "dirty cache line" is defined as a cache line that has been modified and has not yet been written to a lower level cache or main memory.

[0015] If the lookup in the second tag store 302 fails and a cache line is allocated to the request, the cache controller 320 determines which cache line to evict so that a new cache line can be stored. In one embodiment, the cache controller 320 uses the last access type field 308 stored in each entry of the corresponding set of the second tag store 302 to determine which cache line to evict from the second data store 304. For example, in one embodiment, the cache controller 320 obtains a sorted priority 350 that lists a priority associated with each access type. Once the cache controller 320 obtains the sorted priority 350, the cache controller 320 searches for a cache line whose last access type has the lowest priority according to the sorted priority 350. It should be noted that the cache controller 320 may be referred to as a control unit or control logic.

[0016] In one embodiment, cache 300 includes counters 340 for tracking the number of demand hits for each access type. A demand hit refers to a load or store operation hitting an entry in last access type field 308. In one embodiment, counters 340 include a counter for each different access type. For example, counter 340 for prefetch access type is incremented when a demand access hits a cache line whose last access type field 308 is encoded with a prefetch encoding. In one embodiment, the value of counter 340 is used to determine a priority for each access type. For example, the higher the counter value, the higher the priority of the corresponding access type. In one embodiment, periodically, cache controller 320 retrieves the values ​​of counter 340 and sorts the values ​​in descending order. Cache controller 320 then generates sorted priorities 350 based on the sorted values ​​of counter 340, with the highest priority being assigned to the access type whose counter has the highest value. Other access types are assigned priorities in descending order based on the values ​​of the counters.

[0017] An example of a last access type encoding that may be used according to one embodiment is shown in last access type encoding table 330. For example, bits "00" indicate that the last access type was a load. For bits "01", this indicates that the last access type was a store. For bits "10", this indicates that the last access type was a prefetch. For bits "11", this indicates that the last access type was a writeback. In other embodiments, the last access type field 308 of an entry in second tag store 302 may have other numbers of bits other than two. Also, in other embodiments, other encodings different from those shown in last access type encoding table 330 may be used.

[0018] 4, a block diagram of one embodiment of a cache 400 that maintains counters to track the number of demand hits caused by each access type is shown. When a lookup of the cache 400 is performed for a given address, the tag 410, set 415, and offset 420 portions of the given address are used to access various structures of the cache 400. The set portion 415 of the address is used to select the given set of the cache 400, and then the tag portion 410 of the address is compared by a comparator 425 to the tags stored in the ways 405 of the given set. In the illustrated example, the set 400 of the given cache includes four ways 405 (way 0-way 3), although more or fewer ways may be implemented in other caches.

[0019] If a match is found in one of the ways 405 of the tag portion 410 of the address, the last access type is obtained from the corresponding last access type field 430. If the access is a demand hit (i.e., a load hit or a store hit), the last access type is provided to the comparison logic 440. Depending on the last access type, a corresponding counter is incremented. For example, if the last access type was a load, then the load counter 445 is incremented. If the last access type was a store, then the store counter 450 is incremented. If the last access type was a prefetch, then the prefetch counter 455 is incremented. Otherwise, if the last access type was a writeback, then the writeback counter 460 is incremented.

[0020] When the number of accesses reaches a certain programmable threshold number, the cache controller performs a sorting phase 465 to sort the counters 445, 450, 455, 460 in descending order according to their count values. The cache controller then performs a priority assignment phase 470 to assign priorities to the sorted counters 445, 450, 455, 460. The counter with the highest count value gets the highest priority, the counter with the next highest count value gets the next highest priority, and so on. The priorities are then used to determine the replacement policy when the cache controller searches for a cache line to evict. In other words, when the cache controller needs to evict a cache line, it evicts the cache line with the lowest priority.

[0021] Referring to Figure 5, one embodiment of a method 500 for setting cache line priority based on last access type is shown. For purposes of illustration, the steps of this embodiment and those of Figure 6 are shown in sequence. However, it should be noted that in various embodiments of the described method, one or more of the described elements may be performed simultaneously, in a different order than shown, or omitted entirely. Other additional elements may also be performed as desired. Any of the various systems or devices described herein may be configured to perform the method 500.

[0022] Counters are maintained by the cache controller for each different access type (block 505). In one embodiment, there are four counters for four distinct access types: load, store, prefetch, and writeback. In other embodiments, other numbers of counters can track other numbers of different access types. For example, in another embodiment, multiple counters can track multiple different types of loads, where a first counter can track loads from a first application, a second counter can track loads from a second application, etc. Other counters can track stores from each of the individual applications, prefetches from each of the individual applications, and writebacks from each of the applications. In other embodiments, other types of access types can be tracked by the cache controller. In one embodiment, all sets in the cache share the same group of access type counters. In another embodiment, there are multiple groups of counters, where each group of counters is shared by a portion of the cache sets. In a further embodiment, a single group of counters tracks several representative sets, rather than the entire cache.

[0023] If the request received by the cache results in a demand hit (e.g., load hit, store hit), the last access type field corresponding to the hit cache line is obtained (block 510). A counter corresponding to the access type specified in the obtained last access type field is then incremented (block 515). If fewer than a threshold number of cache accesses have occurred (conditional block 520: "no"), the method 500 returns to block 510. If the number of cache accesses has reached the threshold number (conditional block 520: "yes"), the access type counters are sorted in descending order and priorities are assigned to the access types based on the sorted order (block 525). The priorities are then used to determine a replacement policy for selecting a cache line for eviction (block 530). For example, if the prefetch access type counter has the lowest value among all counters, the cache controller prioritizes the selection of the eviction cache line that was last accessed by the prefetch access. In another example, if the load access type counter has the highest value of all counters, the cache controller attempts to retain the cache line last accessed by the load access. After block 530, method 500 ends. It should be noted that method 500 may be performed periodically or in response to detection of some event (e.g., the start of a new phase of an application) to update the priorities of different access types.

[0024] Referring to FIG. 6, one embodiment of a method 600 for selecting a cache line for eviction is shown. The cache controller detects a cache miss for a given request (block 605). In response to detecting the miss, the cache controller searches for a cache line to evict to make space for the data targeted by the given request (block 610). The cache controller then determines which access type has the lowest priority based on the most recent tracking interval (block 615). In various embodiments, the interval is a predetermined period of time, a predetermined number of clock cycles, a predetermined number of transactions or accesses, etc. An example of determining which access type has the lowest priority based on the most recent tracking interval is described in the discussion associated with method 500 (of FIG. 5). The cache controller then searches for a cache line whose last access type field matches the lowest priority access type (block 620).

[0025] If a cache line is found with the last access type field set to the lowest priority access type (conditional block 625: "yes"), the cache controller evicts the cache line (block 630). Otherwise, if a cache line with the last access field set to the lowest priority access type is not found (conditional block 625: "no"), the cache controller determines which of the other remaining access types has the lowest priority (block 640). If a cache line is found with the last access field set to the next lowest priority access type (conditional block 645: "yes"), the cache controller evicts the cache line (block 650). Otherwise, if a cache line with the last access field set to the next lowest priority access type is not found (conditional block 645: "no"), the method 600 returns to block 640 where the cache controller determines which of the other remaining access types has the lowest priority. After blocks 630 and 650, the cache controller stores the new cache line in the way of the evicted cache line (block 635). After block 650, method 600 ends.

[0026] In various embodiments, program instructions of a software application are used to implement the methods and / or mechanisms described herein. For example, program instructions executable by a general purpose or special purpose processor are contemplated. In various embodiments, such program instructions are expressed in a high level programming language. In other embodiments, the program instructions are compiled from the high level programming language into a binary, intermediate or other format. Alternatively, the program instructions are written to describe the operation or design of hardware. Such program instructions are expressed in a high level programming language such as C. Alternatively, a hardware design language (HDL) such as Verilog is used. In various embodiments, the program instructions are stored in any of a variety of non-transitory computer readable storage media. The storage medium is accessible by the computing system during use to provide the program instructions to the computing system for program execution. Generally, such a computing system includes at least one or more memories and one or more processors configured to execute the program instructions.

[0027] It should be emphasized that the above-described embodiments are merely non-limiting examples of embodiments. Many variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to include all such variations and modifications.

Claims

1. A processor; a cache, The cache includes: maintaining a last access type indication for each of a plurality of cache lines; maintaining a plurality of counters, each of the plurality of counters tracking a number of each of a plurality of different access types; assigning the lowest priority to a given access type corresponding to a counter with the fewest number of demand hits, a demand hit being a hit into the cache for a load or store operation; in response to detecting a cache miss, prioritizing the eviction of a cache line having a last access type field set to the predetermined access type assigned the lowest priority; 4. The method of claim 3, system.

2. In response to detecting a demand hit to the first cache line, The cache includes: in response to detecting that a prefetch access type is set in a last access type field of the first cache line, incrementing a first counter; incrementing a second counter in response to detecting that a writeback access type is set in a last access type field of the first cache line; 4. The method of claim 3, The system of claim 1.

3. The cache includes: assigning the highest priority to a first access type having a counter with the highest number of demand hits; giving preference to retaining cache lines having a last access type field set to the first access type; 4. The method of claim 3, The system of claim 1.

4. the cache is configured to sort the plurality of counters according to a number of demand hits per counter within a most recent time interval; The system of claim 1.

5. the cache is configured to store the last access type indication in a last access type field of a first tag entry of a corresponding cache line; The system of claim 1.

6. the predetermined access type is a prefetch access type; The system of claim 1.

7. the predetermined access type is a write-back access type; The system of claim 1.

8. the cache maintaining a last access type indication for each of a plurality of cache lines; maintaining a plurality of counters, each of the plurality of counters tracking a number of each of a plurality of different access types; assigning the lowest priority to a given access type corresponding to a counter with the fewest number of demand hits, a demand hit being a hit into the cache for a load or store operation; in response to detecting a cache miss, prioritizing the eviction of a cache line having a last access type field set to the predetermined access type assigned the lowest priority. method.

9. In response to detecting a demand hit to the first cache line, The method comprises: in response to detecting that a prefetch access type is set in a last access type field of the first cache line, incrementing a first counter; incrementing a second counter in response to detecting that a writeback access type is set in a last access type field of the first cache line. The method of claim 8.

10. assigning a highest priority to a first access type having a counter among the plurality of counters with a highest number of demand hits; and preferentially retaining cache lines having the first access type set in a last access type field. The method of claim 8.

11. sorting the plurality of counters according to a number of demand hits per counter within a most recent time interval. The method of claim 8.

12. storing the last access type indication in a last access type field of a first tag entry of a corresponding cache line; The method of claim 8.

13. the predetermined access type is a prefetch access type; The method of claim 8.

14. the predetermined access type is a write-back access type; The method of claim 8.

15. A data store; a cache controller, maintaining in the data store a last access type indication for each of a plurality of cache lines in the data store; maintaining a plurality of counters, each of the plurality of counters tracking a number of each of a plurality of different access types; assigning the lowest priority to a given access type corresponding to a counter with the fewest number of demand hits, a demand hit being a hit to the cache for a load or store operation; in response to detecting a cache miss, prioritizing the eviction of a cache line having a last access type field set to the given access type assigned the lowest priority; 4. The method of claim 3, cache.

16. In response to detecting a demand hit to the first cache line, The cache controller in response to detecting that a prefetch access type is set in a last access type field of the first cache line, incrementing a first counter; incrementing a second counter in response to detecting that a writeback access type is set in a last access type field of the first cache line; 4. The method of claim 3, 16. The cache of claim 15.

17. The cache controller assigning a highest priority to a first access type having a counter with the highest number of demand hits among the plurality of counters; giving preference to retaining cache lines having a last access type field set to the first access type; 4. The method of claim 3, 16. The cache of claim 15.

18. the cache controller is configured to sort the plurality of counters according to a number of demand hits per counter within a most recent time interval; 16. The cache of claim 15.

19. the predetermined access type is a prefetch access type; 16. The cache of claim 15.

20. the predetermined access type is a write-back access type; 16. The cache of claim 15.

Images