System probe aware last level cache insertion skip

Abstract

Systems, apparatuses, and methods for last level cache insertion skip strategies with system probe filter awareness are disclosed. The system includes a plurality of processing nodes, a probe filter, and a shared cache. The probe filter monitors the rate of generated recall probes and, if the rate is greater than a first threshold, the system initiates a cache partition and monitors a phase of the shared cache. As a result, the cache is partitioned into two parts. If the hit rate of a first part is greater than a second threshold, a second part will have a non-skip insertion strategy because the cache is relatively useful in this scenario. However, if the hit rate of the first part is less than or equal to the second threshold, the second part will have a skip insertion strategy because the cache is less useful in this case.

Description

Background Technology

[0001] Related technical specifications

[0002] Computer systems use main memory typically formed from inexpensive and high-density dynamic random access memory (DRAM) chips. However, DRAM chips suffer from relatively long access times. To improve performance, data processors typically include at least one local high-speed memory called a cache. In multi-core data processors, each data processor core may have its own dedicated Level 1 (L1) cache, while other caches (e.g., Level 2 (L2) cache, Level 3 (L3) cache) are shared by the data processor cores.

[0003] A cache subsystem in a computing system includes a high-speed cache memory that stores blocks of data. As used herein, a “block” is a set of bytes stored in a contiguous memory location, which is treated as a unit for consistency purposes. As used herein, the terms “cache block,” “block,” “cache line,” and “line” are interchangeable. In some embodiments, a block can also be a unit of allocation and deallocation in the cache. The number of bytes in a block varies depending on design choices and can be of any size. Additionally, the terms “cache label,” “cache line label,” and “cache block label” are interchangeable.

[0004] In multi-node computer systems, special precautions must be taken to maintain the consistency of data being used by different processing nodes. For example, if a processor attempts to access data at a certain memory address, it must first determine whether the memory is stored in another cache and has been modified. To implement this cache coherence protocol, caches typically contain multiple status bits to indicate the status of cache lines, thereby maintaining data consistency across the entire system. A common coherence protocol is called the "MOESI" protocol. According to the MOESI protocol, each cache line includes status bits indicating which MOESI state the line is in. These status bits include bits indicating the following: the cache line has been modified (M); the cache line is exclusive (E) or shared (S); or the cache line is invalid (I). The occupied (O) status indicates that the line has been modified in one cache; a shared copy may exist in another cache; and the data in memory is outdated.

[0005] Probe filters are a key building block in high-performance, scalable systems. They are used to track cache lines currently in use by the system. Probe filters improve memory bandwidth and reduce probe bandwidth by executing memory requests or probe requests only when needed. Logically, probe filters reside at the master node of the cache line that enforces the cache coherence protocol. Probe filters operate on the principle of inclusivity (i.e., lines present in the central processing unit (CPU) cache must also be present in the probe filter).

[0006] Probe filters are typically designed to cover all caches for the expected traffic patterns, but they can encounter capacity issues with certain types of non-standard traffic. For example, traffic causing severe index conflicts at the probe filter can lead to capacity problems. Furthermore, traffic resulting in sparse accesses when probe filter entries track multiple cache lines can also cause capacity issues. With very large Level 3 (L3) and Last-Level Cache (LLC) caches, the capacity of the system probe filter can be strained, leading to recalls from the cache to make room for new cache lines within the probe filter. In extreme cases, the LLC can become useless because lines are evicted prematurely, and performance can degrade further if the system is not designed to support the maximum throughput recall flow. Attached Figure Description

[0007] The advantages of the methods and mechanisms described herein can be better understood by referring to the following description in conjunction with the accompanying drawings, in which:

[0008] Figure 1 It is a block diagram of one implementation method of a computing system.

[0009] Figure 2 This is a block diagram of one implementation method for handling nodes.

[0010] Figure 3 This is a block diagram of one implementation method for a multi-node system.

[0011] Figure 4 This is a block diagram of one implementation method for a part of a system-on-a-chip.

[0012] Figure 5 This is a general flowchart illustrating an implementation of a method for using a last-level cache insertion skip strategy that is aware of system probes.

[0013] Figure 6 This is a general flowchart illustrating an implementation of a method for determining an insertion strategy for a portion of the cache. Detailed Implementation

[0014] Numerous specific details are set forth in the following description to provide a thorough understanding of the methods and mechanisms presented herein. However, those skilled in the art will recognize that various implementations can be practiced without these specific details. In some cases, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the methods described herein. It should be understood that, for the sake of brevity and clarity, the elements shown in the figures are not necessarily drawn to scale. For example, the dimensions of some elements may be enlarged relative to other elements.

[0015] This document discloses various systems, apparatuses, methods, and computer-readable media for employing a last-level cache insertion skip strategy perceived by a system probe filter. In one implementation, the system includes multiple nodes, a probe filter, and a last-level cache (LLC). The probe filter monitors the rate at which recall probes are generated, and if the rate is greater than a first threshold, the system initiates a cache partitioning phase and monitors the shared cache. Therefore, the cache is partitioned into two parts. If the hit rate of the first part is greater than a second threshold, the second part will have a non-skip insertion strategy because the cache is useful in this scenario. However, if the hit rate of the first part is less than or equal to the second threshold, the second part will have a skip insertion strategy because the LLC is useless in this case. This helps reduce the number of recall probes generated when the LLC hit rate is low.

[0016] Now for reference Figure 1 The diagram illustrates a block diagram of one implementation of a computing system 100. In one implementation, the computing system 100 includes at least processing nodes 105A to 105N, an input / output (I / O) interface 120, a bus 125, one or more memory controllers 130, and a network interface 135. In other implementations, the computing system 100 may include other components, and / or the computing system 100 may be arranged differently. In one implementation, each processing node 105A to 105N includes one or more general-purpose processors, such as a central processing unit (CPU). It should be noted that "processing node" may also be referred to herein as a "core complex" or "CPU". In some implementations, one or more processing nodes 105A to 105N may include a data-parallel processor with a highly parallel architecture. Examples of data-parallel processors include a graphics processing unit (GPU), a digital signal processor (DSP), etc. Each processor core within the processing nodes 105A to 105N includes a cache subsystem with one or more tiers of cache. In one implementation, each processing node 105A to 105N includes a cache (e.g., a Level 3 (L3) cache) shared among multiple processor cores.

[0017] One or more memory controllers 130 represent any number and type of memory controllers accessible by processing nodes 105A to 105N. One or more memory controllers 130 are coupled to any number and type of memory devices (not shown). For example, the memory type in one or more memory devices coupled to one or more memory controllers 130 may include Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), NAND flash memory, NOR flash memory, ferroelectric random access memory (FeRAM), etc. I / O interface 120 represents any number and type of I / O interface (e.g., Peripheral Component Interconnect (PCI) bus, PCI Expansion (PCI-X), PCIE (PCI High Speed) bus, Gigabit Ethernet (GBE) bus, Universal Serial Bus (USB)). Various types of peripheral devices may be coupled to I / O interface 120. Such peripheral devices include (but are not limited to) monitors, keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, etc.

[0018] In various implementations, the computing system 100 can be any of a server, computer, laptop, mobile device, game console, streaming device, wearable device, or various other types of computing system or device. It should be noted that the number of components in the computing system 100 can vary depending on the implementation. There may be more components than in... Figure 1 The number of each component shown may be greater or less than the number shown. It should also be noted that the computing system 100 may include [the components shown in the diagram]. Figure 1 Other components not shown. Additionally, in other implementations, it can be achieved by connecting with... Figure 1 The diagram shows other ways to construct the computing system 100, different from the methods shown.

[0019] Now transferred to Figure 2 A block diagram of one implementation of a processing node 200 is shown. In one implementation, the processing node 200 includes four processor cores 210A to 210D. In other implementations, the processing node 200 may include other numbers of processor cores. It should be noted that "processing node" may also be referred to herein as a "core complex" or "CPU". In one implementation, the components of the processing node 200 include ( Figure 1 The processing nodes are 105A to 105N.

[0020] Each processor core 210A to 210D includes a cache subsystem for storing data and instructions retrieved from a memory subsystem (not shown). For example, in one implementation, each core 210A to 210D includes a corresponding Level 1 (L1) cache 215A to 215D. Each processor core 210A to 210D may include or be coupled to a corresponding Level 2 (L2) cache 220A to 220D. Additionally, in one implementation, the processing node 200 includes a Level 3 (L3) cache 230 shared by the processor cores 210A to 210D. The L3 cache 230 is coupled to the memory subsystem (not shown) via a texture (not shown). It should be noted that in other implementations, the processing node 200 may include other types of cache subsystems with different numbers of caches and / or other configurations with different cache levels.

[0021] Now for reference Figure 3 The diagram illustrates a block diagram of one implementation of a multi-node system 300. In one implementation, the system includes multiple processing nodes (not shown). The number of processing nodes in each system may vary depending on the implementation. In one implementation, each processing node is connected to a corresponding consistency master (e.g., consistency master 310). As used herein, a "consistency master" is defined as an agent that handles traffic flowing through an interconnect structure (e.g., bus / texture 320) and manages the consistency of connected nodes. To manage consistency, the consistency master receives and processes consistency-related messages and probes, and generates consistency-related requests and probes.

[0022] In one implementation, each processing node is coupled to a consistency slave device (e.g., consistency slave device 330) via a corresponding consistency master device and bus / texture 320. The consistency slave device 330 is coupled to a memory controller (not shown) and also to a probe filter 335, which includes entries for cache lines cached in system 300 for use with memory accessible through the corresponding memory controller. As used herein, a “consistency slave device” is defined as an agent that manages consistency by processing received requests and probes for the corresponding memory controller. Additionally, as used herein, a “probe” is defined as a message passed from a consistency point to one or more caches in the computer system to determine whether the cache has a copy of a data block and optionally indicates the state the cache should have for the data block. When the consistency slave device 330 receives a memory request for its corresponding memory controller, the consistency slave device 330 performs a lookup against the probe filter 335. If the lookup against the probe filter 335 is successful, the probe is sent to the owner of the cache line targeted by the memory request. Otherwise, if the lookup for probe filter 335 fails, a memory request is sent to memory instead of generating a probe. Depending on the insertion strategy of probe filter 335, a new entry can be added to probe filter 335 when a lookup fails.

[0023] Now go to Figure 4 This diagram illustrates a block diagram of one implementation of a portion of a System-on-Chip (SoC) 400. In one implementation, the SoC 400 includes at least a cache 410, a texture 425, a memory controller 430, and a probe filter 435. The cache 410 includes a cache memory 415 and a control unit 420, wherein the cache 410 represents any type of cache. For example, in one implementation, the cache 410 is a Level 3 (L3) cache, and the cache 410 is coupled to a Level 2 (L2) cache (not shown). In other implementations, the cache 410 is a cache of another level in a cache hierarchy. It should be noted that the cache 410 may also be referred to herein as the Last Level Cache (LLC).

[0024] Cache memory 415 includes memory capacities of arbitrary size, which vary depending on the implementation. In one implementation, in response to detecting a high stress level on probe filter 435, cache memory 415 is divided into portions 415A and 415B, where portion 415A is smaller than portion 415B. In one implementation, a “high stress level” is defined as probe filter 435 having a recall probe rate greater than a threshold. The recall probe rate refers to the number of recall probes generated within a given interval, where a recall probe is a message sent from probe filter 435 to cache 410 that causes cache 410 to clear a specific cache line. In other implementations, the “stress level” of probe filter 435 is determined by the recall probe rate and / or one or more other metrics. It should be understood that while portions 410A to 410B appear to be contiguous portions of cache 410, this is only shown for illustrative purposes. In another implementation, portion 410A is a number of randomly selected indices in cache 410, where these indices are distributed across non-contiguous locations within cache 410. In yet another implementation, several partitions can be independently created for various categories of cache traffic. For example, these categories could be based on instruction lines, data lines, Translation Backing Buffer (TLB) hardware table walker rows, various types of software and hardware prefetchers, traffic from various hardware threads or thread groups, etc. The control unit 420 then considers the hit rate of the cache lines for a specific category when deciding whether to apply a skip or non-skip insertion strategy. Other ways of dividing cache 410 into portions 410A to 410B are possible and envisioned.

[0025] In one implementation, control unit 420 applies a non-skip insertion policy to portion 415A while monitoring the hit rate of portion 415A. The non-skip insertion policy means that at least a portion of requests that fail in portion 415A will be allocated in portion 415A. Control unit 420 monitors the hit rate of portion 415A at given time intervals, and if the hit rate is greater than a threshold, control unit 420 applies the non-skip insertion policy to portion 415B. If the hit rate of portion 415A is higher than the threshold, this indicates that cache 410 is useful, and in this case, cache lines should be inserted into the remaining portion 415B. However, if the hit rate of portion 415A is less than or equal to the threshold, this indicates that cache 410 is not particularly useful for a given application executed by SoC 400. In this case, control unit 420 applies a skip insertion policy to portion 415B to allow requests to enter memory instead of being allocated in portion 415B. The skip-insertion policy means that any request that fails to find part 415B will not be allocated in part 415B. The skip-insertion policy helps reduce cache thrashing and the number of recall probes generated by probe filter 435. As used herein, the term "recall probe" is defined as a message sent from the probe filter to the cache that causes the cache to remove a specific cache line from the cache. It should be noted that the skip-insertion policy can be overridden by other mechanisms, such as measuring the hit count at a higher-level cache or based on other determinations that a cache line may be further reused.

[0026] Texture 425 represents any type of interconnect structure that connects the various components and / or agents of SoC 400 together. Although texture 425 is shown as a single unit, it should be understood that this is only one way of representing texture 425. In some implementations, texture 425 includes multiple components distributed throughout SoC 400, where these multiple components are coupled together to allow requests, probes, probe recalls, and other messages to be sent between various agents. Memory controller 430 is coupled to probe filter 435 and memory (not shown). Requests received by memory controller 430 for corresponding memory will check probe filter 435 to see if data is cached by cache 410.

[0027] In some cases, when a lookup by probe filter 435 fails, probe filter 435 clears existing entries to make room for new entries. To clear existing entries, probe filter 435 generates recall probes, which are sent to cache 410. In a configuration where a given probe filter entry tracks multiple cache lines, there can be multiple recall probes. In response to receiving a recall probe, cache 410 clears one or more corresponding cache lines because probe filter 435 is no longer able to track those specific cache lines. When probe filter 435 is issuing frequent recall probes, this can negatively impact system performance.

[0028] Therefore, to help prevent this scenario, in one implementation, probe filter 435 includes counter 440, which tracks the number of recall probes generated during a specific time interval. If the number of recall probes generated during the interval exceeds a threshold, probe filter 435 sends a message to control unit 420 of cache 410 to divide cache memory 415 into portions 415A to 415B and begin monitoring the hit rate of portion 415A. Otherwise, if the number of recall probes is less than or equal to a given threshold, cache 410 can continue its normal operation. Alternatively, in another implementation, control unit 420 monitors the number of received recall probes and compares said number to a threshold at given intervals.

[0029] Now for reference Figure 5 This illustrates one implementation of a method 500 for employing a system probe-aware last-level cache insertion skipping strategy. For discussion purposes, the steps in this implementation are shown in sequential order. Figure 6 The steps are as follows. However, it should be noted that in various implementations of the described method, they may be performed simultaneously, in a different order than shown, or one or more of the described elements may be omitted entirely. Other additional elements may also be performed as needed. Any of the various systems or apparatuses described herein is configured to implement method 500.

[0030] The probe filter monitors the number of recall probes generated within a given interval (box 505). Alternatively, in another implementation, the cache controller monitors the number of recall probes received within a given interval. Other metrics besides the number of probes alone can be monitored in box 505, such as the count of the number of hit probes in each cache level (e.g., L1, L2, L3), the MOESI status of the probe hits, etc. If the number of generated recall probes is greater than a first threshold (condition box 510, "Yes" branch), the probe filter instructs the cache (e.g., the last-level cache (LLC)) to initiate the monitoring phase (box 515). Otherwise, if the number of recall probes within a given interval is less than or equal to the first threshold (condition box 510, "No" branch), method 500 returns to box 505.

[0031] As part of the startup monitoring phase, the cache is divided into a first part and a second part (Box 520). In one implementation, the first part includes a certain number of cache indices, while the second part includes the remainder of the cache. In another implementation, the cache indices for the first part are randomly selected. In other implementations, other suitable methods for dividing the cache into first and second parts may be used.

[0032] Next, following box 520, the cache monitors the hit rate of the first part when a non-skip insertion policy is applied (box 525). In one implementation, the non-skip insertion policy causes cache lines to be allocated for requests that failed in the second part. In another implementation, the hit rate is calculated by dividing the number of cache hits by the total number of requests received by the cache. For example, if the cache receives 100 requests for the first part and only 12 of those requests hit in the first part, the hit rate is 12%. If the hit rate of the first part is less than a second threshold (condition box 530, "Yes" branch), the cache applies a skip insertion policy to the second part (box 540). Applying the skip insertion policy causes requests to not be allocated in the second part, which helps prevent cache thrashing and reduces pressure on the probe filter. If the hit rate of the first part is less than the second threshold, this indicates that the cache is not particularly useful for the current application. It should be noted that in another implementation, the cache includes multiple monitors that monitor the hit rate of cache traffic across many different categories. The cache then determines its skip or non-skip insertion strategy based on the hit rate of the target cache line for a specific category. After box 540, method 500 returns to box 505. Alternatively, method 500 may alternate between returning to box 525 after box 540 in some iterations and returning to box 505 after box 540 in other iterations.

[0033] Otherwise, if the hit rate of the first part is greater than or equal to the second threshold (condition box 530, "No" branch), the cache applies a non-skip insertion strategy to the second part (box 535). In this case, the cache is useful, and therefore the cache can be allocated for requests that fail in the second part. After box 535, method 500 returns to box 505. Alternatively, method 500 may alternate between returning to box 525 after box 535 in some iterations and returning to box 505 after box 535 in other iterations. It should be noted that a certain amount of hysteresis may be applied to the threshold of method 500 to prevent the cache from changing between the non-skip insertion strategy and the skip insertion strategy.

[0034] Turn now Figure 6 This illustrates one implementation of an insertion strategy for determining a portion of the cache. The cache receives an indication of the probe filter pressure level (box 605). In one implementation, the probe filter pressure level indication is a measure of the probe recall rate of the probe filter. In other implementations, other metrics of the probe filter pressure level are generated and these metrics are sent to the cache as indications. Furthermore, the cache monitors the hit rate of a first portion of the cache (box 610).

[0035] Next, the cache determines an insertion policy to be applied to the second part of the cache, where the insertion policy is based on both the probe filter pressure level and the hit rate of the first part of the cache (box 615). Then, the insertion policy determined in box 615 is applied to the second part of the cache (box 620). In one implementation, the cache determines an insertion rate based on a combination of the probe filter pressure level and the hit rate of the first part of the cache. For example, in one implementation, the higher the probe filter pressure level and the lower the hit rate of the first part, the higher the insertion rate applied to the cache when deciding whether to allocate a new cache line in the second part of the cache. A higher insertion rate can also be referred to as a relatively harder-to-distinguish cache insertion policy. Conversely, the lower the probe filter pressure level and the higher the hit rate of the first part, the lower the insertion rate applied to the cache when deciding whether to allocate a new cache line in the second part of the cache. A lower insertion rate can also be referred to as a relatively easier-to-distinguish cache insertion policy. Method 600 ends after box 620. It should be noted that method 600 can be repeated at certain intervals to update the insertion strategy based on changes in probe filter pressure levels and changes in the hit rate of the first part of the cache.

[0036] In various implementations, the methods and / or mechanisms described herein are implemented using program instructions from a software application. For example, program instructions executable by a general-purpose processor or a special-purpose processor are envisioned. In various implementations, such program instructions are represented by a high-level programming language. In other implementations, the program instructions are compiled from a high-level programming language into binary, intermediate, or other forms. Alternatively, program instructions describing the behavior or design of the hardware are written. Such program instructions are represented by a high-level programming language such as C. Alternatively, a hardware design language (HDL) such as Verilog is used. In various implementations, the program instructions are stored on any of a variety of non-transitory computer-readable storage media. During use, the computing system can access the storage media to provide the program instructions to the computing system for program execution. Generally, such a computing system includes at least one or more memories and one or more processors configured to execute the program instructions.

[0037] It should be emphasized that the above implementations are merely non-limiting examples. Many variations and modifications will be apparent to those skilled in the art once the above disclosure is fully understood. The appended claims are intended to cover all such variations and modifications.

Claims

1. A system comprising: a probe filter; and a cache; wherein the system is configured to: monitor a recall probe rate of the probe filter, wherein the recall probe rate corresponds to a number of messages sent from the probe filter to the cache that cause the cache to evict data within a given interval; in response to the recall probe rate being greater than a first threshold, divide the cache into two parts; apply a first insertion policy to a first part of the cache and monitor a hit rate of the first part; and apply a second insertion policy to a second part of the cache, wherein the second insertion policy is selected based on a comparison of the hit rate of the first part to a second threshold.

2. The system of claim 1, wherein the second insertion policy is a skip policy if the hit rate is less than the second threshold.

3. The system of claim 1, wherein the second insertion policy is a non-skip policy if the hit rate is greater than or equal to the second threshold.

4. The system of claim 1, wherein the first insertion policy is a non-skip policy.

5. The system of claim 1, wherein a size of the first part is less than a size of the second part.

6. The system of claim 1, wherein in response to the recall probe rate being less than or equal to the first threshold, the system is configured to apply a non-skip policy to the entire cache.

7. The system of claim 1, wherein the cache is shared by two or more processor cores.

8. A method comprising: monitoring, by a probe filter, a recall probe rate of the probe filter, wherein the recall probe rate corresponds to a number of messages sent from the probe filter to a cache that cause the cache to evict data within a given interval; in response to the recall probe rate being greater than a first threshold, dividing the cache into two parts; applying a first insertion policy to a first part of the cache and monitoring a hit rate of the first part; and applying a second insertion policy to a second part of the cache, wherein the second insertion policy is selected based on a comparison of the hit rate of the first part to a second threshold.

9. The method of claim 8, wherein the second insertion policy is a skip policy if the hit rate is less than the second threshold.

10. The method of claim 8, wherein the second insertion policy is a non-skip policy if the hit rate is greater than or equal to the second threshold.

11. The method of claim 8, wherein the first insertion policy is a non-skip policy.

12. The method of claim 8, wherein a size of the first part is less than a size of the second part.

13. The method of claim 8, wherein in response to the recall probe rate being less than or equal to the first threshold, the method further comprises applying a non-skip policy to the entire cache.

14. The method of claim 8, wherein the cache is shared by two or more processor cores.

15. An apparatus comprising: a processing node comprising a cache hierarchy, wherein the cache hierarchy comprises a given cache shared by a plurality of processor cores; a memory; a memory controller coupled to the memory; and a probe filter coupled to the memory controller; wherein the apparatus is configured to: monitor a recall probe rate of the probe filter, wherein the recall probe rate corresponds to a number of messages sent from the probe filter to a given cache that cause the given cache to evict data within a given interval; responsive to the recall probe rate being greater than a first threshold, partition the given cache into two portions; apply a first insertion policy to a first portion of the given cache and monitor a hit rate of the first portion; and apply a second insertion policy to a second portion of the cache, wherein the second insertion policy is selected based on a comparison of the hit rate of the first portion to a second threshold.

16. The apparatus of claim 15, wherein the second insertion policy is a skip policy if the hit rate is less than the second threshold.

17. The apparatus of claim 15, wherein the second insertion policy is a non-skip policy if the hit rate is greater than or equal to the second threshold.

18. The apparatus of claim 15, wherein the first insertion policy is a non-skip policy.

19. The apparatus of claim 15, wherein a size of the first portion is less than a size of the second portion.

20. The apparatus of claim 15, wherein responsive to the recall probe rate being less than or equal to the first threshold, the apparatus is configured to apply a non-skip policy to the entire given cache. ​ ​

Images