MANAGING A PROGRAMMABLE CACHE CONTROL MAPPING TABLE IN A SYSTEM-LEVEL CACHE

Abstract

Several embodiments include techniques for managing cache memory in a computing system. The disclosed techniques include a cache policy manager that monitors the activity of various components accessing a shared cache memory. The cache policy manager establishes cache rules that determine which data remains in the cache memory and which data is removed from the cache memory to make room for new data. If the activities of these components change over time, cache rules that worked well for a previous activity profile may no longer work well for the current activity profile. Therefore, the cache policy manager dynamically modifies the cache rules as the activity profile changes in order to select cache rules at any given time that work well with the current activity profile.These techniques are advantageous compared to traditional approaches that use static cache rules which only work well for specific activity profiles.

Description

BACKGROUND Area of ​​the various embodiments

[0001] Various embodiments relate generally to computing system architectures and, more particularly, to managing a programmable cache control mapping table in a system-level cache. Description of related technology

[0002] A computing system generally includes, among other things, one or more processing units, such as central processing units (CPUs) and / or graphics processing units (GPUs), and one or more memory systems. An interconnect facilitates communication among the various components of the computing system, including CPUs, GPUs, auxiliary processors, video and / or audio streaming devices, network adapters, and / or the like. These components of the computing system perform various processes, also referred to herein as work, on behalf of one or more application programs. The interconnect also facilitates communication between the components of the computing system and a shared memory system, referred to herein as system memory.In some embodiments, the system memory is implemented with dynamic random access memory (DRAM) devices and / or other similar memory devices. DRAM is a common type of memory that provides a relatively large amount of storage space and low power consumption, but with relatively low bandwidth and high access latency relative to other types of memory.

[0003] Control circuits within the computing system help conserve energy consumed by the computing system's components while maintaining an appropriate level of performance. To conserve energy, computing system components transition between high and low power states in varying usage scenarios. When a particular component is in heavy use while performing assigned work, the component is in a high power state. When a particular component is in light use or not used at all during application program execution, the component is in a low power state to reduce energy consumption.

[0004] Furthermore, when in use, the components of the computing system frequently access system memory, such as to fetch instructions from system memory, load data from system memory, load data into system memory, and / or the like. Because multiple components may access system memory, the limited bandwidth of system memory may limit the overall performance of the system. To minimize the system memory bandwidth consumed by the components of the computing system, a shared cache accessible to all components may be placed between the interconnect and the system memory, or alternatively, within the interconnect itself. This type of shared cache is referred to herein as a system-level cache (SLC) and typically includes a number of cache lines.In some embodiments, the SLC is implemented with static random access memory (SRAM) devices and / or other similar memory devices. An SRAM is a common type of memory that provides a relatively small amount of memory space and higher power consumption but with relatively high bandwidth and short access latency compared to other types of memory. An SLC included in the system can significantly reduce the number of memory operations directed to the system memory, thereby increasing the overall performance.

[0005] Additionally, due to the increased bandwidth provided by SLC, components can complete work faster, thereby allowing components to enter low-power states more frequently and remain in those low-power states for longer periods of time. Furthermore, when components access SLC more frequently and thus access system memory less frequently, DRAM devices can either operate at a lower frequency or enter a low-power state during periods when system memory is not being accessed.

[0006] Because the SLC is a relatively small memory, data that is likely to be accessed in the near future is preferentially retained within the SLC. In the prior art, control circuitry residing in or associated with the cache implements one or more cache replacement policies to determine which lines to evict when new requests are made to the cache. Advantageously, the control logic evicts data that has been accessed in the past but is unlikely to be accessed in the near future. Cache replacement policies attempt to optimize SLC performance by reducing the cache miss rate.

[0007] Many cache replacement policies have been deployed. For example, a random replacement (RR) policy randomly selects a cache line for eviction. While an RR policy does not require access history data and is therefore relatively easy to implement, an RR policy is not optimized for specific memory access patterns. Some cache replacement policies are based on access recency, which includes a first-in-first-out (FIFO) policy, a last-in-first-out (LIFO) policy, a least-recently-used (LRU) policy, and a most recently-used (MRU) policy. Similarly, some cache replacement policies are based on access frequency, which includes a least-frequently-used (LFU) policy.Access recency policies and access frequency policies involve storing specific access history data for each cache line, such as the time of storage, the time of first access, the time of last access, the frequency of accesses, and / or the like. Therefore, access recency policies and access frequency policies are more complex to implement than an RR policy, but they are optimized for specific memory access patterns. Other cache replacement policies rely on storing and using access history data to help predict when cache lines are likely to be re-referenced.These cache replacement policies include re-reference interval prediction (RRIP), static re-reference interval prediction (SRRIP), bimodal re-reference interval prediction (BRRIP), and dynamic re-reference interval prediction (DRRIP). Re-reference prediction policies are also more complex to implement than an RR policy and have different trade-offs between hardware and software complexity versus miss rate, but are again optimized for specific memory access patterns.

[0008] A problem with conventional cache replacement policies in an SLC is that different components of the computing system may benefit from different cache replacement policies. In some use cases, a first component that is likely to reference data stored in the cache before the data is evicted may benefit from allocating cache memory. However, a second component that is not likely to reference data stored in the cache before the data is evicted may not need to allocate any cache memory. In such use cases, selecting a cache replacement policy that allocates cache memory to the first component could result in unnecessary allocation of cache memory to the second component, thereby degrading the cache benefit for other components.On the other hand, allocating no cache memory to the second component might be appropriate, but would result in poor memory performance for the first component as well as other components that could benefit from allocating cache memory.

[0009] Dynamic cache replacement policies, such as DRRIP, can marginally improve cache utilization over static policies, but they are unable to make bold allocation decisions that can dramatically improve cache performance in specific use cases, and are unable to optimally handle cases such as streaming writes that are accessed once after an extended period of time. Furthermore, system state can change which caching decisions are optimal for a given set of client request streams. For example, when the CPU and other clients are active, an optimal solution may be to not allocate video client request streams into the cache, whereas in low-power use cases where video clients are the only active clients, such clients can benefit from allocating them into the cache to meet energy targets.The performance and energy of the computing system may therefore suffer compared to what would be possible if the caching behavior were better managed.

[0010] As the foregoing illustrates, what is needed in the art are more effective techniques for managing cache memory in a computing system by exploiting knowledge of the system state and memory access patterns of clients. SUMMARY

[0011] Various embodiments of the present disclosure set forth a computer-implemented method for managing cache memory in a computing system. The method includes monitoring telemetry data measuring a system state of the computing system. The method further includes determining whether the system state is consistent with a previously identified system state included in a set of previously identified system states.The method further comprises, if the system state is consistent with a first previously identified system state included in the set of previously identified system states, then: updating a mapping table with table entry indices appropriate for each combination of client type identifier and buffer type for the first previously identified system state, or if the system state is inconsistent with any previously identified system state included in the set of previously identified system states, then: updating the mapping table with the table entry indices appropriate for each combination of client type identifier and buffer type for a default system state. The method further comprises reading one or more cache control parameters from a table entry referenced by the table entry index.

[0012] Other embodiments include, without limitation, a system implementing one or more aspects of the disclosed techniques and one or more computer-readable media comprising instructions for performing one or more aspects of the disclosed techniques.

[0013] At least one technical advantage of the disclosed techniques relative to the prior art is that, with the disclosed techniques, cache replacement policy management and cache partition management are dynamically selected based on the memory access use case for different SLC clients. The techniques consider both the current system state and client-provided (sub-stream) caching information. The techniques can concurrently achieve the benefits of several prior approaches, such as DRRIP approaches, state-based approaches, and client hint-based approaches. As with DRRIP approaches, the techniques can infer whether or not certain client streams may benefit from caching data. As with state-based approaches, the techniques can configure cache policy decisions based on the current system state.As with client-hint-based approaches, the techniques can make different caching decisions for different types of requests from the same client. For example, the techniques can choose not to allocate cache for streaming data accesses from a client and allocate cache for data accesses from the same client that have a high probability of reuse. In this way, the techniques can make optimal caching decisions for each client type and access type, taking into account the current system state as well as the client-provided sub-stream information. As a result, the techniques can lead to improved cache utilization, cache performance, and cache power management relative to conventional approaches.Furthermore, the techniques are extensible, so that cache replacement policy management and cache partition management can be applied to specific client types and data access types and later expanded to apply to other client types and data access types. Furthermore, these techniques can be extended to detect and optimize additional use cases once these additional use cases are identified. These advantages represent one or more technological improvements over prior art approaches. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order that the above-listed features of the various embodiments may be understood in detail, a more particular description of the inventive concepts briefly summarized above may be obtained by reference to various embodiments, some of which are illustrated in the accompanying drawings. It should be noted, however, that the accompanying drawings illustrate only typical embodiments of the inventive concepts and are therefore in no way limiting in scope, and that other equally effective embodiments exist. Fig. 1 is a block diagram of a computing system configured to implement one or more aspects of the various embodiments; Fig. 2 is a block diagram of an SLC memory system used in the computing system of Fig. 1, according to various embodiments; Fig. 3 is a block diagram of the SLC policy manager for the SLC storage system of Fig. 2 according to various embodiments; Fig. 4 is a block diagram of the partition manager module included in the SLC policy manager of Fig. 3, according to various embodiments; Fig. 5 is a flowchart of process steps for managing the data stored in the computer system of Fig. 1 comprised system-level cache according to various embodiments; and Fig. 6 is a flow diagram of method steps for updating the programmable mapping table in the system-level cache used in the computing system of Fig. 1, according to various embodiments. DETAILED DESCRIPTION

[0015] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the various embodiments. However, one skilled in the art will appreciate that the inventive concepts may be practiced without one or more of these specific details. System overview

[0016] Fig. 1 is a block diagram of a computing system 100 configured to implement one or more aspects of the various embodiments. As shown, computing system 100 includes, without limitation, a central processing unit (CPU) 102, a system-level cache (SLC) 122, other SLC clients 140, and system memory 104 coupled to an auxiliary processing subsystem 112 via a memory bridge 105 and a communication path 113. Memory bridge 105 is further coupled to an I / O (input / output) bridge 107 via a communication path 106, and I / O bridge 107 is in turn coupled to a switch 116. The CPU 102 is coupled to the memory bridge 105, which enables the other SLC clients 140 to communicate with other SLC clients 140, the system-level cache 122, the system memory 104, the auxiliary processing subsystem 112, and the I / O bridge 107.Other SLC clients 140 are also coupled to the memory bridge 105, which enables the other SLC clients 140 to communicate with the CPU 102, the system-level cache 122, the system memory 104, the auxiliary processing subsystem 112, and the I / O bridge 107.

[0017] In operation, I / O bridge 107 is configured to receive user input information from input devices 108, such as a keyboard or a mouse, and forward the input information to CPU 102 for processing via communication path 106 and memory bridge 105. In some embodiments, input devices 108 are employed to verify the identities of one or more users to allow authorized users access to computing system 100 and deny unauthorized users access to computing system 100. Switch 116 is configured to provide connections between I / O bridge 107 and other components of computing system 100, such as a network adapter 118 and various add-in cards 120 and 121.

[0018] As also shown, the I / O bridge 107 is coupled to a system disk 114, which may be configured to store content and applications, as well as data for use by the CPU 102 and the auxiliary processing subsystem 112. As a general matter, the system disk 114 provides non-volatile storage for applications and data and may include fixed or removable hard disk drives, flash memory devices, and CD-ROM (compact disk read-only memory), DVD-ROM (digital versatile disk ROM), Blu-ray, HD-DVD (high-definition DVD), or other magnetic, optical, or solid-state storage devices. Finally, although not explicitly shown, other components, such as universal serial bus or other port connections, compact disk drives, digital versatile disk drives, movie recording devices, and the like, may also be connected to the I / O bridge 107.

[0019] In various embodiments, memory bridge 105 may be a northbridge chip and I / O bridge 107 may be a southbridge chip. Furthermore, communication paths 106 and 113, as well as other communication paths within computing system 100, may be implemented using any technically suitable protocols, including, without limitation, Peripheral Component Interconnect Express (PCIe), HyperTransport, or any other bus or point-to-point communication protocol known in the art.

[0020] In some embodiments, auxiliary processing subsystem 112 includes a graphics subsystem that provides pixels to a display device 110, which may be any conventional cathode ray tube, liquid crystal display, light-emitting diode display, or the like. In such embodiments, auxiliary processing subsystem 112 incorporates circuitry optimized for graphics and video processing, including, for example, video output circuitry. Such circuitry may be incorporated via one or more auxiliary processors included within auxiliary processing subsystem 112.An auxiliary processor includes any one or more processing units capable of executing instructions, such as a central processing unit (CPU), a parallel processing unit (PPU), a graphics processing unit (GPU), a direct memory access (DMA) unit, an intelligence processing unit (IPU), a neural processing unit (NAU), a tensor processing unit (TPU), a neural network processor (NNP), a data processing unit (DPU), a vision processing unit (VPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and / or the like.

[0021] In some embodiments, the auxiliary processing subsystem 112 incorporates circuitry optimized for general-purpose processing and / or computational processing. Such circuitry may, in turn, be incorporated via one or more auxiliary processors included within the auxiliary processing subsystem 112 that are configured to perform such general-purpose and / or computational operations. In yet other embodiments, the one or more auxiliary processors included within the auxiliary processing subsystem 112 may be configured to perform graphics processing, general-purpose processing, and computational processing operations.

[0022] The system-level cache 122 is a last-level cache. The system-level cache 122 can maintain hardware coherence between various coherent SLC clients in the computing system 100, which includes the CPU 102, the auxiliary processing subsystem 112, the network adapter 118, and / or the like. The system-level cache 122 exhibits low access latency relative to the system memory 104. The system-level cache 122 acts as a serialization point in that traffic between various SLC clients and the system memory 104 passes through the system-level cache 122. The system-level cache is managed by an SLC policy manager (in Fig. 1 not shown) as in conjunction with Fig. 2-4. The SLC policy manager is also referred to here as a cache memory policy manager.

[0023] In various embodiments, the auxiliary processing subsystem 112 may be integrated with one or more of the other elements of Fig. 1 to form a single system. For example, the auxiliary processing subsystem 112 may be integrated with the CPU 102, the memory bridge 105, the I / O bridge 107, and / or other components and interconnect circuits on a single chip to form a system-on-a-chip (SoC). Additionally or alternatively, these various components may be integrated together on multiple chips in the form of chiplets and / or dielets. As referred to herein, chiplets and dielets are small integrated circuits, with each chiplet and dielet performing a defined set of functions. Multiple chiplets and dielets may be combined together to form a multi-chip module or a hybrid integrated circuit that performs higher-level functions. For example, multiple chiplets and dielets may be combined together to form a computing system 100.Interconnect circuits on the SoC, chiplets, and / or dielets facilitate communication among the various components, including CPUs, GPUs, auxiliary processors, video and / or audio streaming devices, network adapters, and / or the like.

[0024] It is recognized that the system shown here is illustrative and that variations and modifications are possible. The interconnect topology, including the number and arrangement of bridges, the number of CPUs 102, and the number of auxiliary processing subsystems 112, may be modified as desired. In alternative topologies, the auxiliary processing subsystem 112 may be connected to the I / O bridge 107 or directly to the CPU 102 and not to the memory bridge 105. In still other embodiments, the I / O bridge 107 and the memory bridge 105 may be integrated into a single chip rather than existing as one or more discrete devices. Finally, in certain embodiments, one or more Fig. 1 may not be present. For example, switch 116 could be eliminated, and network adapter 118 and add-in cards 120, 121 would be connected directly to I / O bridge 107. Managing a System-Level Cache with Multiple Cache Clients

[0025] Various embodiments include an SLC policy manager that employs hardware-based techniques and software-based techniques for managing a cache in a computing system. The disclosed techniques dynamically detect various memory access use cases and select cache replacement policies well-suited to the detected use case. To assist in detecting memory access use cases and selecting cache replacement policies, components of the computing system are tagged and identified by an allocation / replacement identifier (ARID) value. Components of the computing system that access the system-level cache are referred to herein as SLC clients. In some embodiments, each individual SLC client may be assigned a unique ARID.Alternatively, a class of SLC clients that have similar allocation and / or replacement policies, that share the same SLC residency characteristics, and / or that have the same or similar memory access patterns may share the same ARID value. The tag data for each cache line stored in the SLC is tagged with the ARID of the SLC client that caused the corresponding cache line to be stored in the SLC. The disclosed techniques include client cache hints, a programmable mapping table, cache policy controllers, and SLC policy manager (SLCPM) software.

[0026] Fig. 2 is a block diagram of an SLC memory system 200 used in the computing system 100 of Fig. 1, according to various embodiments. As shown, the SLC storage system 200 includes, without limitation, an SLC policy manager 210, an operating system 220, a power manager 222, as well as the system-level cache 122 and the system memory 104 of Fig. 1. The SLC clients 140 communicate with the SLC storage system 200 via a system interconnect, such as the storage bridge 105 of Fig. 1. The system-level cache 122 includes a programmable mapping table 230 and a cache memory 232. In some embodiments, the programmable mapping table 230 is a single mapping table included in the SLC storage system 200. In some embodiments, the programmable mapping table 230 includes multiple mapping tables distributed among one or more SLC storage systems 200.

[0027] The SLC clients 140 generate memory requests according to the requirements of the application(s) supported by the SLC clients 140. The configurations of a given SLC client 140 and the stream of memory requests generated by the given SLC client 140 are based on one or more applications generated by the SLC client 140. The configuration and stream of memory requests are generally independent of the overall system state. In some examples, a video application including a video decoding engine that decodes a Motion Picture Experts Group (MPEG) stream may behave the same whether the video application is the only executing application or whether a CPU-intensive application is executing concurrently. With each memory request, an SLC client 140 generates and transmits a buffer type hint that encodes information about the memory request pattern for that stream.In some examples, the BufferType hint may identify a category of the memory request, such as whether memory addresses associated with the memory requests are to be re-referenced once, many times, or not at all. An SLC client 140 may generate memory request streams concurrently with different memory access patterns, with each memory request stream tagged with a corresponding BufferType value. The SLC client 140 further generates an ARID that identifies the type of SLC client 140. Each SLC client 140 that allocates cache lines in the system-level cache 122 is assigned an appropriate ARID by the SLC memory system 200. Memory requests generated by that SLC client 140 are tagged with the ARID corresponding to that SLC client 140.

[0028] The memory bridge 105 or other system interconnect forwards memory requests 270 from SLC clients 140 to the system-level cache 122 while arbitrating between memory request streams from the various SLC clients 140. The memory bridge 105 maintains the attributes associated with each memory request. These attributes include the buffer type value, the ARID, and / or the like. The memory bridge 105 transmits forwarded memory requests 272 to the system-level cache 122.

[0029] Concurrently, the SLC policy manager 210 executes received use case data 252 from the operating system 220, system power state 254 from the power manager 222, and / or the like as a background process. In some embodiments, the SLC policy manager may further receive activity data 250 from SLC clients 140, from system monitors, and / or the like. The SLC policy manager 210 may execute on the CPU 102, on the auxiliary processing subsystem 112, on an auxiliary processor, on an FPGA, on a microcoded processor, and / or the like. The SLC policy manager 210 determines the system state and transmits configuration commands via a configuration interface 260 based on the current system state. In some embodiments, the configuration interface 260 includes a standard register programming interface, a communication bus included in the computer system 100, and / or the like.When the SLC policy manager 210 determines that the system state has changed, the SLC policy manager 210 transmits updates 262 to the programmable mapping table 230 appropriate for the newly determined system state. The SLC policy manager 210 may also update global cache control parameters 264 as appropriate for the newly determined system state. The global cache control parameters 264 include parameters that apply to all SLC clients 140, such as cache partitioning data.

[0030] When a forwarded store request 272 arrives at the system-level cache 122, a PMT subsystem (not shown) included in the SLC memory system 200 creates a PMT entry index by concatenating the ARID and buffer type information together to form an index 274 in the programmable mapping table 230. This PMT subsystem is referred to as a mapping table controller. The programmable mapping table 230 selects appropriate cache control parameters 278 for the store request 276, which may include cache allocation parameters, cache replacement parameters, cache eviction parameters, and / or the like. The cache control parameters 278 determine whether the store request 276 is allocated to the cache 232 and how cache line attributes are set.Memory requests 276 that cannot be satisfied within the cache memory 232 are transferred via path 280 to the system memory 104 as read operations or write operations.

[0031] A system-level cache memory controller (not shown) associated with the system-level cache 122 uses the concatenation of ClientID and BufferType as the index into the programmable mapping table 230. In some embodiments, storing a complete set of cache control parameters 278 in each table entry of the programmable mapping table 230 may result in a programmable mapping table 230 that is impractically large. Accordingly, in some embodiments, the size of the programmable mapping table 230 may be reduced by mapping ClientIDs into a smaller set of ARIDs, where each ARID represents a subset of SLC clients 140 that have similar memory access patterns. In such embodiments, the concatenation of ARID and BufferType is smaller, and the size of the programmable mapping table 230 may be reduced.In some embodiments, the programmable mapping table 230 includes indices and non-explicit cache control parameters 278. These indices refer to a limited set of allocation classes, with each allocation class associated with a specific set of cache control parameters 278. In some embodiments, the number of indices is 16, so each table entry in the programmable mapping table 230 comprises only 4 bits, where the 4 bits identify the index to one of the 16 sets of cache control parameters 278. In this manner, each entry of the programmable mapping table 230 does not need to store the dozens or hundreds of bits required to explicitly store a full set of cache control parameters 278.The index stored in the table entries of the programmable mapping table 230 refers to a corresponding table entry in a table of actual cache control parameters 278. Such embodiments may implement the {ClientID, BufferType} mapping in cache control parameters 278 in a more efficient manner.

[0032] Specifically, the SLC storage system 200 employs at least four techniques to manage the system-level cache 122. These techniques include client cache hints, a programmable mapping table 230, cache policy controls, and an SLC policy manager 210.

[0033] A first technique, called client caching hints, is a hardware-based technique in which each SLC client specifies a memory access pattern associated with the SLC client. More specifically, SLC clients specify a buffer type along with the corresponding memory access request. The buffer type distinguishes different types of buffers used by the SLC clients, which may benefit from different cache allocation policies. For example, one buffer type may identify buffers containing weight parameters that are read from memory multiple times but rarely, if ever, written to memory. A second buffer type may identify buffers containing source data that is streamed into the processor once and then discarded. A third buffer type may identify buffers that are to be written to memory once, and so on.The buffer type does not directly specify what the cache allocation and replacement policy should be. Instead, the buffer type provides information about the data usage by a particular SLC client, which can be used in determining an appropriate allocation and replacement policy for that data in a particular use case. In some embodiments, SLC clients may include a buffer type index along with the corresponding memory access request. The buffer type index is a small numeric tag that the SLC client appends as an attribute to each memory access request. The buffer type index encodes the buffer type into a small multi-bit value, so that each different buffer type corresponds to a different buffer type index.

[0034] A second technique is a hardware-based technique that employs a programmable mapping table (PMT). The programmable mapping table maps the ARID and buffer type into the SLC's allocation and replacement controls. A PMT subsystem included in the SLC memory system 200 creates a PMT entry index by concatenating the ARID and buffer type values ​​together to form a vector {ARID, BufferType}. A system-level cache memory controller inserts the vector {ARID, BufferType} as an index into the programmable mapping table. Each entry in the programmable mapping table includes cache allocation and cache replacement parameters for memory requests from the specific SLC client employing the specified buffer type. The entries in the programmable mapping table are programmable, so the entries can be updated for any given use case.The cache allocation and cache replacement parameters determine whether and how data associated with memory requests is allocated in the SLC and criteria for when cache lines are replaced in the SLC.

[0035] In some embodiments, the number of SLC clients, buffer types, and cache parameters, such as allocation and replacement policy controls, may be substantially large. Accordingly, the size of the PMT may be substantially large in both the number of table entries and the data width of the table entries. In such embodiments, the overall size of the PMT may be reduced by splitting the PMT into two smaller tables. A first table maps the vector {ARID, BufferType} into a PMT index. A second table maps the PMT index into a set of cache parameters. The entries in the second PMT table include distinct allocation and replacement policy controls along with other cache parameters useful in computing system 100. This approach effectively quantizes the vector and compresses the storage space consumed by PMT table entries.The combined size of these two tables can be significantly smaller than a single PMT indexed directly by the vector {ARID, BufferType}.

[0036] A third technique, referred to as cache policy controls, is a hardware-based technique where the SLC implements one or more allocation and replacement controls that are applied to incoming requests as determined by the relevant PMT table entry.

[0037] An allocation and replacement control, referred to as cache line allocation, defines where in the LRU chain a particular memory request is allocated. The memory request can be allocated into different eviction classes, which can provide isolation and / or protection during victim selection, where a victim is a cache line selected for eviction.

[0038] Another allocation and replacement control, referred to as cache line promotion, promotes a cache line when the cache line is hit. The hit cache line may be promoted to the MRU position, to the next level in the LRU chain, to the LFU position, and / or the like. Further, the promotion technique may vary based on the SLC client that caused the cache line hit. In some embodiments, if the CPU 102 causes a cache line hit, the cache line may be promoted to the MRU position, whereas if the auxiliary processing subsystem 112 causes a cache line hit, the cache line may be promoted to the LFU position. This approach may be advantageous for use cases where memory accesses by the auxiliary processing subsystem 112 are of higher frequency relative to memory accesses by the CPU 102.

[0039] Yet another allocation and replacement control, called cache line demotion, demotes a cache line under certain conditions. For example, cache store hints from the client may indicate that a current memory request is the last use of the associated memory address by the client. Such a cache store hint may indicate that the corresponding cache line is no longer needed and can therefore be evicted from the cache.

[0040] Yet another allocation and replacement control, called cache line sacrifice, places newly allocated cache lines resulting from memory requests into different sacrifice levels. These sacrifice levels can serve as the final arbitration stage during a sacrifice selection.

[0041] Yet another allocation and replacement control is a replacement code, which indicates the probability that a cache line will be rereferenced. The replacement code may indicate whether a cache line is most likely to be rereferenced, more likely to be rereferenced, somewhat likely to be rereferenced, least likely to be rereferenced, and / or the like. Additionally or alternatively, the replacement code may indicate that there is no recommendation, in which case a preset rereference probability is selected.

[0042] Yet another allocation and replacement control is a read / write indicator for cases where the cache policy may differ for read operations versus write operations. Yet another allocation and replacement control is an eviction class code, which indicates whether the cache line should be evicted with high priority, medium priority, or low priority.

[0043] Yet another allocation and replacement control is a cooldown interval, which sets a period of time for demoting a cache line to a lower eviction class or for evicting a cache line. In this regard, in some embodiments, each cache line is provided with a cooldown counter. Further, some embodiments may provide a cooldown counter for a collection of cache lines or another mechanism to track the aging of memory requests. In such embodiments, if the cache line is re-referenced during the cooldown interval, the cooldown counter is reset to the initial value. If the cache line is re-referenced during the cooldown interval, the cooldown counter is reset to the initial value.If the cache line is not newly referenced during the decay interval, then the decay counter is decremented, the cache line is downgraded to a lower space class, the cache line is evicted from the system-level cache, and / or the like.

[0044] The cache allocation and replacement parameters may further include any other data that may improve caching behavior for requests from a given client with a given buffer type. Other embodiments may implement allocation and replacement controls in different variations and in different combinations. For example, some embodiments may index certain allocation and replacement controls in the programmable mapping table by ARID and buffer type {ARID, BufferType}, while other embodiments may index certain allocation and replacement controls in the programmable mapping table only by ARID {ARID}.

[0045] A fourth technique, called SLC Policy Manager (SLCPM), is a technique in which a software or firmware program called an SLC Policy Manager monitors the system state, detects and infers the memory access use case, and dynamically updates the programmable mapping table to optimize the use of the SLC.

[0046] The SLC policy manager controls the mapping of each vector {ARID, BufferType} into a corresponding PMT entry. The SLC policy manager controls this mapping to thereby control which memory buffers are allocated in the system-level cache and to control the priority for replacement for each of the memory buffers stored in the system-level cache. The SLC policy manager can reprogram the mapping into the PMT based on a detected use case and / or a detected scenario, where the detected scenario is based on the power state for various SLC clients, the activity level of various SLC clients, and / or the like. The SLC policy manager receives the data for detecting scenarios from the operating system, the hardware components, and / or the like.Based on the detected scenario, the SLC policy manager updates mappings in the table entries in the PMT to fine-tune the system-level cache operation for the detected scenario. The table entries in the PMT define the combination of different cache policies. The table entries in the PMT can cause changes in the cache policies based on detected scenarios, system requirements, and / or the like. In this way, the SLC policy manager updates the SLC allocation and replacement policy controls in the PMT to adapt the caching behavior of the system-level cache for different scenarios.

[0047] Fig. 3 is a block diagram of the SLC policy manager 210 for the SLC storage system 200 of Fig. 2 according to various embodiments. The SLC policy manager 210 includes, without limitation, a policy holder module 310, a generic event handling module 316, an event rate debounce module 318, a partition manager module 320, a Machine Learning Store 322, a statistics database 324, a garbage collection policy control 326, and a partition control interface (I / F) 330. The policy holder module 310 in turn includes, without limitation, a configuration data manager 312 and an SLC configuration 314. The components of the SLC policy manager 210 can be implemented in any combination with hardware components, firmware components, or software components.

[0048] The SLC policy manager 210 monitors the state of individual SLC clients 140, the system activity of the computing system 100, and the power state of individual SLC clients 140. Furthermore, the SLC policy manager 210 may receive information about the state of the operating system and about the applications running on the various SLC clients 140. In doing so, the SLC policy manager 210 monitors system events 364, which include system-related events and / or power-related events delivered in the form of interrupts, event notifications, and / or the like. The system events 364 represent telemetry data measuring various components of the computing system 100. The SLC policy manager 210 monitors this telemetry data.Based on this telemetry data, the SLC policy manager 210 identifies known use cases and / or scenarios, also previously identified here, that can be optimized for efficient access to the SLC 122. These known scenarios may include video streaming, audio streaming, audiovisual conferencing, intensive computations, and / or the like. Upon identifying a current scenario, the SLC policy manager 210 selects one or more corresponding scenario-based actions that modify one or more cache control parameters. The SLC policy manager 210 performs the actions that result in a cache state of the SLC 122 tuned for the identified scenario, including a high cache hit rate, a low eviction rate, and / or the like.If the SLC policy manager 210 does not identify a specific scenario, then the SLC policy manager 210 applies a generic or preset setting that best maximizes performance in a particular preset scenario across a range of scenarios. In some embodiments, the preset scenario is a high-performance, high-energy scenario. High-performance, high-energy use cases can be difficult to identify and / or categorize and may therefore be good candidates for the preset scenario. Furthermore, if the SLC policy manager 210 cannot identify a specific scenario that is tuned for a particular use case, then selecting a high-performance, high-energy scenario as a preset scenario does not unfairly penalize the SLC client 140.More generally, if the SLC policy manager 210 identifies a specific scenario, then the SLC policy manager 210 selects a PMT entry optimized for the previously identified system state associated with the specific scenario. If the SLC policy manager 210 does not identify a specific scenario, then the SLC policy manager 210 selects a PMT entry that specifies a default behavior compatible with, but not optimized for, the previously identified system state associated with the specific scenario.

[0049] The policy holder module 310 receives cache policy initial parameters 350 and runtime updates 352 and stores the initial parameters 350 and runtime updates 352 for other components of the SLC policy manager 210

[0050] The configuration data manager 312 receives initial parameters 350 and accepts runtime updates 352 to the parameters during operation. The configuration data manager 312 parses and categorizes initial parameters 350 and runtime updates 352. After parsing and categorizing, the configuration data manager 312 stores the processed data in the SLC configuration 314. If the configuration data manager 312 modifies data in the SLC configuration 314, the configuration data manager 312 generates one or more notifications of the modification and transmits the notifications to relevant components of the SLC policy manager 210.

[0051] During initial startup, the SLC 122 is fully shared by the SLC clients 140 via path 384. The policy holder module 310 may modify the initial startup configuration by receiving the initial parameters 350, parsing and categorizing the initial parameters 350, and storing the resulting configuration data 354 in the SLC configuration 314. The SLC configuration 314 transmits the configuration data to relevant components of the SLC policy manager 210. In this regard, the SLC configuration 314 transmits broadcast updates 356 to the generic event handler module 316, broadcast updates 358 to the event rate debounce module 318, and broadcast updates 360 and 362 to the partition manager module 320. The components of the SLC policy manager 210 begin execution using the configuration data in the SLC configuration 314.

[0052] Over time, the policy holder module 310 may modify the current configuration by receiving runtime updates 352, parsing and categorizing the runtime updates 352, and storing the resulting updated configuration data in the SLC configuration 314. The SLC configuration 314 transmits the updated configuration data to relevant components of the SLC policy manager 210. In this regard, the SLC configuration 314 transmits broadcast updates 356 to the generic event handler module 316, broadcast updates 358 to the event rate debounce module 318, and broadcast updates 360 and 362 to the partition manager module 320. The components of the SLC policy manager 210 continue execution using the updated configuration data in the SLC configuration 314.

[0053] Data received as the initial parameters 350 and runtime updates 352 includes: debounce timer intervals, hardware partition information, partition size and / or partition granule size information, hit and / or miss rate per SLC client 140, percentage of SLC 122 dedicated to each traffic class when multiple traffic classes are active, information regarding which SLC clients 140 are members of each traffic class, victim / eviction selection strategy, beneficiary selection strategy, streaming detection enablement, information identifying the partitioning technique used, and / or the like. In some embodiments, the partition size may be expressed as a number of granules, and the partition size may change in multiples of granules.The granule size can be determined either by hardware and / or by software that modifies the granule size set by hardware.

[0054] The generic event handler module 316 receives system events 364 relevant to system-level cache management and transmits the system events 364 to the event rate debouncing module 318. The generic event handler module 316 also receives broadcast updates 356 from the SLC configuration 314. Based on policy parameter settings received via broadcast updates 356, the generic event handler module 316 selects certain system events 364 to be transmitted to the event rate debouncing module 318 and suppresses the transmission of other system events 364. Certain system events 364, such as a power state change of SLC clients 140, are relevant to SLC management and would therefore be transmitted by the generic event handler module 316.Other system events 364, such as certain activity measures, may or may not be relevant to SLC management and may be transmitted or suppressed by the generic event handler module 316 depending on the SLC configuration 314. The generic event handler module 316 transmits filtered system events 366 to the event rate debounce module 318.

[0055] The event rate debounce module 318 receives filtered system events 366 from the generic event handler module 316 and broadcast updates from the SLC configuration 314. The event rate debounce module 318 stabilizes the state of the SLC 122, such as when an SLC client 140 has become unstable and is generating a high rate of system events 364. In some embodiments, rapid state changes of an SLC client 140 can cause a flood of cache allocations and evictions, effectively negating the benefits of cache management strategies such as cache partition size optimization. The event rate debounce module 318 protects the SLC 122 from excessive cache misses and evictions caused by frequent changes in which segments belong to which partitions.The event rate debouncing module 318 can perform global debouncing for all SLC clients 140 or specific debouncing for each SLC client 140, in any combination. The event rate debouncing module 318 receives a debouncing timer interval (DTI) from the SLC configuration, which defines a minimum time interval before a consecutive SLC client state change would generate a partition allocation state change. Setting the DTI to zero disables the debouncing function. The event rate debouncing module 318 transmits debounced system events 368 to the partition manager module 320.

[0056] The partition manager module 320 receives debounced system events 368 from the event rate debounce module 318 and broadcasts updates 360 and 362 from the SLC configuration. The partition manager module 320 manages partition sizes of the SLC 122 based on the debounced system events 368 and the applied policy, as received from the SLC configuration 314 via broadcast updates 360 and 362. The partition manager module 320 is used in conjunction with Fig. 4. Briefly, the partition manager module 320 may modify the cache control parameters by changing a partition boundary. Changing the partition boundary increases an amount of system-level cache memory allocated to a first system-level cache partition and decreases an amount of system-level cache memory allocated to a second system-level cache partition. In some embodiments, the first system-level cache partition and / or the second system-level cache partition may be allocated to a subset of two or more SLC clients 140.

[0057] Furthermore, the partition manager module 320 tunes eviction policy parameters 374 and transmits the eviction policy parameters 374 to the eviction policy controller 326. The partition manager module 320 generates cache partition data 378 and transmits the cache partition data 378 to the partition control interface 330. The partition manager module 320 generates partition training data 370 and transmits the partition training data 370 to the machine learning store 322. The partition manager module 320 generates statistical data 372 and transmits the statistical data 372 to the statistics database 324.

[0058] The eviction policy controller 326 receives eviction policy parameters 374 from the partition manager module 320. The eviction policy controller 326 dynamically changes the eviction policy of the SLC 122 over time in response to updated eviction policy parameters 374 received from the partition manager module 320.

[0059] The partition control interface 330 receives cache partition data 378 from the partition manager module 320. The partition control interface 330 dynamically changes the partition boundaries 382 of the SLC 122 over time in response to updated cache partition data 378 received from the partition manager module 320. The partition control interface 330 transmits the updated partition boundaries 382 to the SLC 122.

[0060] The statistics database 324 receives statistical data 372 from the partition manager module 320 and receives hit / miss data 386 from the SLC 122. The statistics database 324 stores statistical data 372 and hit / miss data 386 for subsequent analysis to determine the efficiency and effectiveness of the SLC policy manager 210. Furthermore, the statistics database 324 transmits statistical training data 388 to the machine learning store 322.

[0061] The machine learning store 322 receives partition training data 370 from the partition manager module 320 and receives statistical training data 388 from the statistics database 324. The machine learning store 322 stores partition training data 370 and statistical data 372 for subsequent training of a machine learning model to perform one or more of the techniques described herein.

[0062] Fig. 4 is a block diagram of the SLC policy manager 210 of Fig. 3, according to various embodiments. The partition manager module 320 includes, without limitation, a scenario table 410, partition state data 412, a scenario detection module 414, an action coordination module 416, and a repartition module 418. The components of the partition manager module 320 may be implemented with hardware components, firmware components, or software components in any combination.

[0063] The scenario table 410 receives parameters via broadcast updates 360 from the SLC configuration 314. The scenario table 410 formats these parameters and stores the parameters received via broadcast updates 360. The scenario table 410 also receives parameter updates 452 from the scenario recognition module 414 and updates the parameters stored in the scenario table 410 accordingly. The parameters are stored in the scenario table 410 in the form of a programmable mapping table described herein.

[0064] Partition state data 412 receives parameters via broadcast updates 362 from SLC configuration 314. Partition state data 412 stores the parameters received via broadcast updates 362. Partition state data 412 also receives parameter updates 454 from action coordination module 416 and updates the parameters stored in scenario table 410 accordingly.

[0065] The scenario detection module 414 receives debounced system events 368 from the event rate debounce module 318.

[0066] The scenario detection module 414 coalesces debounced system events 368 to detect low-energy scenarios for one or more SLC clients 140 in any combination. The scenario table 410 includes a set of known scenarios for which caching behavior can be optimized. The scenario detection module 414 accesses the scenario table 410 via path 450 to determine whether the current system state corresponds to any of the scenarios included in the scenario table 410. If the scenario detection module 414 determines that the current system state corresponds to a scenario included in the scenario table 410, the scenario detection module 414 determines appropriate action requests 458 based on the scenario. The scenario detection module 414 transmits the action requests 458 to the action coordination module 416.If a recognized low - energy scenario corresponds to a single table entry in the scenario table 410, then the scenario recognition module 414 transfers the action requests 458 specified by the table entry. If a recognized low - energy scenario largely corresponds to multiple table entries in the scenario table 410, then the scenario recognition module 414 generates a hybrid action request 458 based on a combination of the multiple table entries that largely correspond to the recognized low - energy scenario. In addition, the scenario recognition module 414 generates parameter updates 452 and transfers the parameter updates 452 to the scenario table 410.

[0067] In some embodiments, when the scenario detection module 414 receives a series of filtered and debounced system events 368 from the event rate debounce module 318, the scenario detection module 414 has data about the state of each SLC client 140. The state of each SLC client 140 may be represented as a bitmask. The scenario detection module 414 may use the bitmask as a direct key in a hash table of states included in the scenario table 410 to detect an accurate scenario. If the scenario detection module 414 finds a single table entry from the direct key, then the scenario detection module 414 determines that the table entry is correct.If the scenario recognition module 414 does not find a matching table entry from the direct key, the scenario recognition module 414 performs a second-order search to identify a combination of two or more table entries that, when combined, correspond to a table entry representing a similar scenario to the current state. When the scenario recognition module 414 receives such a second-order combination, the scenario recognition module 414 performs a triage operation to determine a set of actions based on the multiple table entries that formed the combination. The scenario recognition module 414 transmits the set of actions as action requests 458 to the action coordination module 416.

[0068] The action coordination module 416 receives action requests 458 from the scenario recognition module 414. Based on these action requests 458, the action coordination module 416 generates two types of actions. The first type of action is an action to set eviction policy parameters 374. The action coordination module 416 adjusts these eviction policy parameters 374 and transmits the eviction policy parameters 374 to the eviction policy controller 326. The second type of action is an action to set partition parameters. The action coordination module 416 generates partition settings 460 and transmits the partition settings 460 to the repartition module 418. In addition, the action coordination module 416 generates parameter updates 454 and transmits the parameter updates 454 to the partition state data 412.

[0069] The repartition module 418 receives partition settings 460 from the action coordination module 416 and receives partition parameters 456 from the SLC configuration 314. The repartition module 418 maintains the partition states of the SLC 122 and implements accesses to various partitioning techniques, such as partitioning by cache ways, partitioning by available cache percentage, and / or the like.

[0070] In some embodiments, when one or more SLC clients 140 are active, the repartition module 418 partitions the SLC 122 such that the entire SLC 122 is allocated for use by the active SLC clients 140. If another SLC client 140 subsequently leaves a low power state and transitions to an active state, the repartition module 418 decreases the partition sizes of the other active SLC clients 140 to free up space in the SLC 122 for the new active SLC client 140. If an SLC client 140 subsequently leaves an active state and transitions to a low power state, the repartition module 418 increases the partition sizes of the remaining active SLC clients 140 to utilize the space in the SLC 122 for the SLC client 140 that entered the low power state.

[0071] In some embodiments, the repartition module 418 organizes SLC clients 140 into traffic classes. In some systems, the number of SLC clients 140 may be in the dozens or hundreds. Instead of creating a partition for each individual SLC client 140, the repartition module 418 may create a partition for each group of SLC clients 140, where each group of SLC clients 140 represents a different traffic class. For example, all CPUs 102 could be members of a single CPU traffic class. Additionally or alternatively, the CPUs 102 could be organized into a small number of traffic classes, where each CPU traffic class comprises a different subset of the total number of CPUs 102. In a particular example, the repartition module 418 could organize SLC clients 140 into two traffic classes, i.e., a first traffic class that includes all CPUs 102 and a second traffic class that includes all other SLC clients 140.Over time, the repartition module 418 redraws the partition boundary between the partitions for these two traffic groups as needed.

[0072] In some embodiments, the repartition module 418 operates with certain allocation constraints. In this regard, the repartition module 418 may define a minimum allocation, which is the allocation value that each SLC client 140 is guaranteed to have when active. The repartition module 418 may further define an initial allocation, which defines the SLC size percentage value permanently assigned to each traffic class of the SLC client 140 upon initialization. The repartition module 418 may define the initial allocation to be identical to the minimum allocation. The repartition module 418 may further define a delta allocation, which represents a portion of an SLC partition that is above the initial allocation when all SLC clients 140 are considered active.

[0073] In some embodiments, the SLC policy manager 210 triggers updates in all affected modules of the SLC policy manager 210 concurrently. The SLC policy manager 210 may effect this concurrent update via double buffering of memory in the SLC configuration 314. With double buffering, once the configuration data manager 312 has finished storing all of the new cache policy data at the initial parameters 350 and / or runtime updates 352 in a secondary buffer of the SLC configuration 314, the configuration data manager 312 swaps the secondary buffer with the primary buffer of the SLC configuration 314. As a result, the former secondary buffer containing the new cache policy data is now accessed as the primary buffer. The former primary buffer containing the previous cache policy data is now accessed as the secondary buffer to collect a subsequent set of cache policy data.The configuration data manager 312 generates broadcast updates 356, 358, 360, and 362 to notify other modules in the SLC policy manager 210 of the buffer swap. In doing so, the configuration data manager 312 does not reset the state of the cache policy data to the initial configuration. Instead, the configuration data manager 312 stores the new cache policy data over the old cache policy data. Furthermore, the cache line contents are retained until evictions occur naturally.

[0074] In some embodiments, changes to SLC partition sizes are stateless, so that the previous history of partition sizes does not affect subsequent partition size distributions. As a result, for a given list of active SLC clients 140, the steady state of the SLC policy manager 210 typically repeatedly achieves the same partition size percentages whenever the same policy is reapplied.

[0075] Fig. 5 is a flowchart of process steps for managing the system-level cache used in the computing system of Fig. 1-4, according to various embodiments. The method steps may be performed by a combination of the SLC policy manager 210, the SLC clients 140, the programmable mapping table 230, and the system-level cache 122. Additionally or alternatively, certain method steps may be performed by one or more processing units and / or alternative auxiliary processors, including, without limitation, CPUs, GPUs, DMA units, IPUs, NPUs, TPUs, NNPs, DPUs, VPUs, ASICs, FPGAs, and / or the like in any technically feasible combination. Although the method steps in connection with the system of Fig. 1-4, those of ordinary skill in the art will understand that any system configured to perform the method steps in any order is within the scope of the present disclosure.

[0076] As shown, a method 500 begins at step 502, where a system-level cache policy manager 210 included in a computing system 100 identifies a system state of the computing system. The system-level cache policy manager 210 identifies the system state based on data from various hardware components, including system-level cache allocation and / or replacement controls, control circuitry that generates power state change notifications, system activity, bandwidth monitors that generate telemetry data, and / or the like.The system-level cache policy manager 210 further identifies a system state based on various software components, including a software mechanism for delivering component state change events to the system-level cache policy manager 210, a software mechanism for determining system configuration, operating system state and configuration information, and / or the like. The system-level cache policy manager 210 monitors the state of individual clients, the system activity of the computing system, and the power state of individual system-level cache clients. Furthermore, the system-level cache policy manager 210 may receive information about the state of the operating system and about the applications running on the various clients.The system-level cache policy manager 210 monitors system events, including system-related events and / or energy-related events delivered in the form of interrupts, event messages, and / or the like. System events represent telemetry data that measures various components of the computing system. Based on this telemetry data, the system-level cache policy manager 210 identifies known use cases and / or scenarios that can be optimized for efficient access to the system-level cache. These known scenarios may include video streaming, audio streaming, audiovisual conferencing, intensive computations, and / or the like.

[0077] At step 504, the system-level cache policy manager 210 stores cache control parameters specifying allocation behavior, eviction behavior, and / or the like in a programmable mapping table 230. This characterization data may be determined a priori based on the characterization and / or qualification of the various clients.

[0078] At step 506, each SLC client 140 executing in computing system 100 determines a memory access pattern associated with that particular SLC client 140. More specifically, SLC clients 140 specify a buffer type along with the corresponding memory access request. The buffer type distinguishes between different types of buffers used by the SLC clients 140, which may benefit from different cache allocation policies.

[0079] At step 508, the system-level cache 122 accesses the programmable mapping table 230 to determine a set of cache control parameters based on the system state and the memory access pattern. The system-level cache 122 accesses the programmable mapping table 230 based on the vector {ARID, BufferType} to determine the set of cache control parameters for each given memory request.

[0080] At step 510, logic within system-level cache 122 processes a memory request according to the set of cache control parameters retrieved from programmable mapping table 230. The parameters may cause the request to allocate a new cache line in system-level cache 122 or not to allocate a cache line. If a cache line is allocated, the parameters may determine the LRU status, eviction class, decay interval, and / or any other parameters beneficial for making optimal eviction decisions. Method 500 then proceeds to step 502 described above to continue dynamically managing system-level cache operation.

[0081] It should be noted that method 500 includes all of the steps to process a single memory request. In this regard, certain steps performed by the cache policy manager, such as step 502 and step 504, are performed less frequently than other steps, such as step 506, step 508, and step 510. In a specific example, step 502 and step 504 may be performed on the order of once per millisecond, whereas step 506, step 508, and step 510 may be performed for each memory request on the order of once per nanosecond or less. As a result, step 502 and step 504 may be performed independently of step 506, step 508, and step 510.

[0082] Fig. 6 is a flow diagram of method steps for updating the programmable mapping table 230 in the system-level cache 122 included in the computing system 100 of Fig. 1-4, according to various embodiments. The method steps may be performed by the SLC policy manager 210. Additionally or alternatively, the method steps may be performed by one or more processing units and / or alternative auxiliary processors, including without limitation CPUs, GPUs, DMA units, IPUs, NPUs, TPUs, NNPs, DPUs, VPUs, ASICs, FPGAs, and / or the like, in any combination. Although the method steps are described in connection with the system of Fig. 1-4, those of ordinary skill in the art will understand that any system configured to perform the method steps in any order is within the scope of the present disclosure.

[0083] As shown, a method 600 begins at step 602, wherein a system-level cache policy manager 210 included in a computing system 100 identifies a system state of the computing system. The system-level cache policy manager 210 identifies a system state based on data from various hardware components, including system-level cache allocation and / or replacement controls, control circuitry that generates power state change notifications, system activity, bandwidth monitors that generate telemetry data, and / or the like. The system-level cache policy manager 210 further identifies a system state based on various other conditions, as described herein in connection with step 502 of Fig. 5 described.

[0084] At step 604, the system-level cache policy manager 210 determines that the system state of the computing system has changed. The change in system state may be due to a system-level cache client transitioning between an active state and a low power state. Additionally or alternatively, the change in system state may be due to a system-level cache client initiating a streaming operation or ceasing a streaming operation. Additionally or alternatively, the change in system state may be due to a change in activities directed to the system-level cache by one or more system-level cache clients.

[0085] At step 606, the system-level cache policy manager 210 updates entries in the programmable mapping table 230 with cache control parameters appropriate for SLC clients 140 given the current system state. In doing so, the system-level cache policy manager 210 updates the cache allocation and replacement policy currently in effect for each {ARID, BufferType} combination, tuned for a previous system state, to a new cache allocation and replacement policy tuned for the current system state. As a result, cache performance is improved by updating the cache allocation policy and / or cache replacement policy as the system state changes over time. The method 600 then proceeds to step 602 described above to continue updating the programmable mapping table 230 as the system state changes.

[0086] In summary, various embodiments include techniques for managing cache memory in a computing system.

[0087] The disclosed techniques dynamically detect different memory access use cases and select cache replacement policies well-suited to the detected use case. The disclosed techniques include client cache hints, a programmable mapping table, cache policy controls, and an SLC policy manager. A first technique, referred to as client cache hints, is a hardware-based technique in which each SLC client specifies a memory access pattern associated with the SLC client. A second technique is a hardware-based technique employing a programmable mapping table (PMT). The programmable mapping table maps the ARID and buffer type into the SLC's allocation and replacement controls.A third technique, referred to as cache policy controls, is a hardware-based technique in which the SLC implements one or more allocation and replacement controls that are applied to incoming requests as determined by the relevant PMT table entry. A fourth technique is a technique in which a software or firmware program referred to as an SLC policy manager monitors system state, detects and infers the memory access use case, and dynamically updates the programmable mapping table to optimize SLC usage.

[0088] At least one technical advantage of the disclosed techniques relative to the prior art is that, with the disclosed techniques, cache replacement policy management and cache partition management are dynamically selected based on the memory access use case for different SLC clients. The techniques consider both the current system state and client-provided (sub-stream) caching information. The techniques can achieve the benefits of several prior approaches concurrently, such as DRRIP approaches, state-based approaches, and client hint-based approaches. As with DRRIP approaches, the techniques can infer whether or not certain client streams may benefit from caching data. As with state-based approaches, the techniques can configure cache policy decisions based on the current system state.As with client-hint-based approaches, the techniques can make different caching decisions for different types of requests from the same client. For example, the techniques can choose not to allocate cache memory for streaming data accesses from a client and allocate cache memory for data accesses from the same client that have a high probability of reuse. In this way, the techniques can make optimal caching decisions for each client and access type, taking into account the current system state as well as client-provided sub-stream information. As a result, the techniques can lead to improved cache utilization, performance, and power management relative to conventional approaches.Furthermore, the techniques are extensible, so that cache replacement policy management and cache partition management are applied to specific client types and data access types and later expanded to apply to other client types and data access types. Furthermore, these techniques can be extended to detect and optimize additional use cases as these additional use cases are identified. These advantages represent one or more technological improvements over prior art approaches.

[0089] Any and all combinations of any of the claim elements recited in any of the claims and / or elements described in this application are in all respects within the intended scope of the embodiments and protection.

[0090] The descriptions of the various embodiments have been presented for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

[0091] Aspects of the present embodiments may be embodied as a system, method, or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be generally referred to herein as a "module" or "system." Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon.

[0092] Any combination of one or more computer-readable media may be used. Each computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing.More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction-executing system, device, or apparatus.

[0093] Aspects of the present disclosure are described above with reference to flowchart illustrations and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, may be implemented using computer program instructions.

[0094] These computer program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing device to produce a machine such that the instructions, executed via the processor of the computer or other programmable data processing device, enable the implementation of the functions / acts specified in the flowchart and / or block diagram block or blocks. Such processors may be, but are not limited to, general-purpose processors, special-purpose processors, application-specific processors, or field-programmable gate arrays.

[0095] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code comprising one or more executable instructions for implementing the specific logical function(s). It should also be noted that in some alternative implementations, the functions recited in the block may occur in a different order than shown in the figures. For example, two blocks shown in sequence may actually execute substantially concurrently, or the blocks may sometimes execute in reverse order depending on the functionality involved.It should also be noted that each block of the block diagram and / or flowchart illustration and combinations of blocks in the block diagram and / or flowchart illustration may be implemented by means of special-purpose hardware-based systems that perform the specified functions or operations, or combinations of special-purpose hardware and computer instructions.

[0096] Although the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be formulated without departing from the basic scope thereof, and the scope thereof is determined by means of the following claims.

Claims

[1] A computer-implemented method for managing a cache memory in a computing system, the method comprising: Monitoring telemetry data that measures a system state of the computing system; Determining whether the system state is consistent with a previously identified system state included in a set of previously identified system states; if the system state is consistent with a first previously identified system state included in the set of previously identified system states, then: Updating a mapping table with table entry indexes appropriate for each combination of client type identifier and buffer type for the first previously identified system state, or if the system state is inconsistent with any previously identified system state included in the set of previously identified system states, then: Updating the mapping table with the table entry indexes appropriate for each combination of client type identifier and buffer type for a standard system state; and Reading one or more cache control parameters from a table entry referenced by the table entry index. [2] The computer-implemented method of claim 1, wherein data regarding a memory request pattern for the first client is encoded in the buffer type attribute associated with the first client. [3] A computer-implemented method according to claim 1 or 2, wherein the first client is identified by an identifier associated with a class of clients having similar policies to at least one allocation policy associated with the first client or replacement policy associated with the first client. [4] A computer-implemented method according to any preceding claim, wherein each client included in the plurality of clients is associated with a unique identifier. [5] A computer-implemented method according to any preceding claim, wherein the first client is associated with the same identifier as a second client included in the plurality of clients. [6] A computer-implemented method according to any preceding claim, wherein the table entry associated with the table entry index based on the default system state specifies a default behavior that is compatible with, but not optimized for, the previously identified system state. [7] A computer-implemented method according to any preceding claim, wherein the table entry associated with the table entry index based on the previously identified system state specifies a behavior optimized for the previously identified system state. [8] A computer-implemented method according to any preceding claim, wherein the table entry comprises at least one of cache allocation parameters or cache replacement parameters for memory requests from the first client employing a buffer type specified by the buffer type attribute. [9] A computer-implemented method according to any preceding claim, further comprising concatenating the buffer type attribute with an identifier associated with at least one of the allocation policy or the replacement policy. [10] A computer-implemented method according to any preceding claim, further comprising receiving the buffer type attribute from the first client along with a storage request. [11] System comprising: a memory; multiple cache storage clients coupled with the storage; a mapping table that maps a client type identifier and a buffer type associated with a request into a table entry index; a cache policy manager configured to: to monitor telemetry data that measures a system status of the system; to determine whether the system state is consistent with a previously identified system state included in a set of previously identified system states; and if the system state is consistent with a first previously identified system state included in the set of previously identified system states, then: Updating the mapping table with table entry indexes appropriate for each combination of client type identifier and buffer type for the first previously identified system state, or if the system state is inconsistent with a previously identified system state included in the set of previously identified system states, then: Updating the mapping table with the table entry indexes appropriate for each combination of client type identifier and buffer type for a default system state; and a cache memory controller configured to: read one or more cache control parameters from a table entry referenced by a first table entry index included in the table entry indexes. [12] The system of claim 11, wherein data regarding a memory request pattern for the first client is encoded in the buffer type attribute associated with the first client. [13] The system of claim 11 or 12, wherein the first client is identified by an identifier associated with a class of clients having similar policies to at least one of the allocation policy associated with the first client or the replacement policy associated with the first client. [14] System according to one of claims 11 to 13, wherein each client included in the plurality of clients is associated with a unique identifier. [15] System according to one of claims 11 to 14, wherein the first client is associated with the same identifier as a second client included in the plurality of clients. [16] The system of any one of claims 11 to 15, wherein the table entry associated with the table entry index based on a default system state specifies a default behavior that is compatible with, but not optimized for, the previously identified system state. [17] The system of any one of claims 11 to 16, wherein the table entry associated with the table entry index based on a default system state specifies behavior optimized for the previously identified system state. [18] The system of any one of claims 11 to 17, wherein the table entry comprises at least one of cache allocation parameters or cache replacement parameters for memory requests from the first client employing a buffer type specified by the buffer type attribute. [19] The system of any of claims 11 to 18, wherein the cache policy manager is further configured to concatenate the buffer type attribute with an identifier associated with at least one of the allocation policy and the replacement policy. [20] The system of any one of claims 11 to 19, wherein the cache policy manager is further configured to receive the buffer type attribute from the first client along with a storage request.

Images