Computer implementation methods, computer programs, and systems (minimizing the impact of cache thrashing by changing the persistence policy of non-temporary workloads).

Abstract

This invention provides computer implementation methods, systems, and computer program products for minimizing the impact of cache thrashing associated with non-transient workloads. [Solution] Method 400 monitors the cache activity of a workload, including cache activity related to local and peer caches (e.g., the number of cache hits), 401 and determines whether a non-ephemeral workload is identified based on an analysis of metrics related to cache activity, 402. The determination is based on a comparison of metrics related to the monitored cache activity of the workload with thresholds. If a non-ephemeral workload is identified, 403 yes, the method identifies the cache lines associated with the non-ephemeral workload, 404 and modifies the persistence policy of the identified cache lines, 405 and shortens the lifespan of the cache lines.

Description

Technical Field

[0001] This disclosure generally relates to cache thrashing, and more particularly to minimizing the impact of cache thrashing by changing the persistence policy of non-temporary workloads.

Background Art

[0002] Cache thrashing is a defect in the cache mechanism that occurs when the cache (e.g., the CPU cache) is constantly updated with new data or when repeatedly accessing data that exceeds the cache size. This can lead to frequent cache misses or data eviction, which in turn forces the processor to access the slower main memory more frequently. This can have an adverse effect on the performance and efficiency of the processor.

[0003] Cache thrashing can occur when the workload streams a large amount of data from memory. Workload refers to the computational tasks, processes, or data transactions that need to be executed by a program. Examples of such workloads that stream a large amount of data from memory include real-time data such as location information, stock prices, monitoring of information technology systems, fraud detection, retail inventory, sales, customer activities, etc.

[0004] Workloads that stream a large amount of data from memory (e.g., main memory) can affect the overall system performance by causing thrashing at multiple cache levels (e.g., constantly updating the cache with new data or repeatedly accessing data that exceeds the cache size). As a result, the number of cache misses or data evictions may increase, which may cause the processor to access the slower main memory to fetch the requested data.

[0005] Furthermore, such workloads can consume system resources to hold data that is not rereferenced in the cache (meaning data is stored in the cache for a certain period of time). As a result, the number of cache misses or data evictions increases, which may cause the processor to access slower main memory to fetch the requested data.

[0006] Techniques have been developed to address cache thrashing. For example, one such technique uses different cache levels or partitions to reduce cache contention.

[0007] Another example involves attempting to optimize cache size. For instance, after determining the cache size, large amounts of data can be divided into smaller blocks to fit into the cache.

[0008] Another approach involves optimizing policies, such as tailoring cache replacement policies to the workload.

[0009] Furthermore, the cache-aside pattern may be used to address cache thrashing. For example, a microservice check may be used to determine whether data is available in the cache before accessing it. [Overview of the project] [Problems that the invention aims to solve]

[0010] Despite these techniques for addressing cache thrashing, they fail to address the impact of cache thrashing in situations where the workload streams large amounts of data from memory (e.g., main memory). In this case, thrashing occurs at multiple cache levels, and the consumption of system resources due to data that is not rereferenced being held in the cache negatively impacts overall system performance. [Means for solving the problem]

[0011] In one embodiment of the present disclosure, a computer implementation method for minimizing the impact of cache thrashing comprises the step of monitoring the cache activity of a workload. The method further comprises the step of identifying non-ephemeral workloads based on the monitored cache activity of the workload. The method further comprises the step of identifying one or more cache lines associated with the non-ephemeral workloads. The method further comprises the step of modifying the persistence policy of one or more cache lines associated with the non-ephemeral workloads.

[0012] Other embodiments of the computer implementation method described above exist in system and computer program products.

[0013] Therefore, embodiments of this disclosure minimize the impact of cache thrashing from non-transient workloads.

[0014] The foregoing outlines the features and technical advantages of one or more embodiments of this disclosure in some general terms, in order to facilitate a better understanding of the detailed description of this disclosure that follows. Further features and advantages of this disclosure are described below, which may form the subject matter of the claims of this disclosure. [Brief explanation of the drawing]

[0015] A more thorough understanding of this disclosure can be obtained by considering the following detailed explanation in conjunction with the following drawings.

[0016] [Figure 1] This figure illustrates one embodiment of the disclosure relating to a computing environment for implementing the principles of the disclosure.

[0017] [Figure 2] This figure illustrates an example of a distributed symmetric multiprocessing (SMP) system utilizing cache persistence according to one embodiment of the present disclosure.

[0018] [Figure 3] This is a diagram of a software component used by a cache controller to minimize the impact of cache thrashing associated with non-transient workloads, according to one embodiment of the present disclosure.

[0019] [Figure 4] This is a flowchart illustrating a method for minimizing the impact of cache thrashing associated with non-temporary workloads according to one embodiment of the present disclosure. [Modes for carrying out the invention]

[0020] In one embodiment of the present disclosure, a computer implementation method for minimizing the impact of cache thrashing comprises the step of monitoring the cache activity of a workload. The method further comprises the step of identifying non-ephemeral workloads based on the monitored cache activity of the workload. The method further comprises the step of identifying one or more cache lines associated with the non-ephemeral workloads. The method further comprises the step of modifying the persistence policy of one or more cache lines associated with the non-ephemeral workloads.

[0021] In this way, the impact of cache thrashing by non-temporary workloads is minimized.

[0022] Furthermore, in certain embodiments of the present disclosure, the persistence policy is changed to shorten the duration of one or more cache lines.

[0023] In this way, by shortening the length of time that cache lines associated with non-temporary workloads are stored, the impact of cache thrashing by non-temporary workloads is minimized.

[0024] Furthermore, in certain embodiments of the present disclosure, the persistence policy is changed to shorten the duration of one or more cache lines by setting specific values for hint bits.

[0025] In this way, by shortening the length of time that cache lines associated with non-temporary workloads are stored via hint bits, the impact of cache thrashing by non-temporary workloads is minimized.

[0026] Furthermore, in certain embodiments of the present disclosure, the persistence policy is changed to directly evict one or more cache lines to memory.

[0027] In this way, by evicting cache lines associated with non-temporary workloads to memory, the impact of cache thrashing by non-temporary workloads is minimized.

[0028] Furthermore, in certain embodiments of the present disclosure, the method further comprises analyzing metrics regarding monitored cache activity of the workload.

[0029] In this way, non-temporary workloads are identified based on analysis of metrics regarding monitored cache activity of the workload.

[0030] Furthermore, in some embodiments of this disclosure, metrics relating to monitored cache activity of a workload include one or more of the following, selected from the group consisting of cache hit / miss ratio, data source of cache misses, fetch / castout ratio, and percentage of castouts with changed data:

[0031] In this way, non-transient workloads are identified based on an analysis of the workload's monitored cache activity, such as the cache hit / miss ratio, the data source of cache misses, the fetch / cast-out ratio, and / or the percentage of cast-outs with modified data.

[0032] Furthermore, in one embodiment of the present disclosure, the cache activity of a workload is monitored in association with a peer cache, and metrics for the monitored cache activity of the workload include the cast-in / fetch hit ratio provided by the peer cache.

[0033] In this way, non-transient workloads are identified based on monitoring of cache activity provided by the peer cache (e.g., cast-in / fetch hit ratio).

[0034] Furthermore, in some embodiments of this disclosure, the cache activity of a workload is monitored in association with the local cache and the peer cache.

[0035] In this way, non-transient workloads are identified based on cache activity metrics for local and peer caches.

[0036] Furthermore, in one embodiment of this disclosure, the workload streams data from memory.

[0037] In this way, non-transient workloads that stream data from memory can be identified.

[0038] As mentioned above, cache thrashing can occur when a workload streams large amounts of data from memory. A workload refers to a computational task, process, or data transaction that needs to be performed by a program. Examples of such workloads that stream large amounts of data from memory include streaming real-time data such as location information, stock prices, information technology system monitoring, fraud detection, retail inventory, sales, and customer activity.

[0039] Workloads that stream large amounts of data from memory (e.g., main memory) can negatively impact overall system performance by causing thrashing at multiple cache levels (e.g., constantly refreshing the cache with new data or repeatedly accessing data that exceeds the cache size). This can result in an increased number of cache misses or data evictions, which may cause the processor to access slower main memory to fetch the requested data.

[0040] Furthermore, such workloads can consume system resources to hold data that is not rereferenced in the cache (meaning data is stored in the cache for a certain period of time). As a result, the number of cache misses or data evictions increases, which may cause the processor to access slower main memory to fetch the requested data.

[0041] Techniques have been developed to address cache thrashing. For example, one such technique uses different cache levels or partitions to reduce cache contention.

[0042] Another example involves attempting to optimize cache size. For instance, after determining the cache size, large amounts of data can be divided into smaller blocks to fit into the cache.

[0043] Another approach involves optimizing policies, such as tailoring cache replacement policies to the workload.

[0044] Furthermore, the cache-aside pattern may be used to address cache thrashing. For example, a microservice check may be used to determine whether data is available in the cache before accessing it.

[0045] Despite these techniques for addressing cache thrashing, they fail to address the impact of cache thrashing in situations where the workload streams large amounts of data from memory (e.g., main memory). In this case, thrashing occurs at multiple cache levels, negatively impacting overall system performance, and system resources are consumed as data that is not rereferenced is held in the cache.

[0046] Embodiments of this disclosure provide means for minimizing the impact of cache thrashing by modifying the persistence policy for non-ephemeral workloads. In some embodiments, cache activity metrics for local and peer caches are monitored and analyzed to determine whether a workload streaming data from memory is a non-ephemeral workload. A “non-ephemeral workload,” as used herein, refers to a workload that retains data that is not immediately reused (re-read) until, for example, it is evictioned (cast out). As a result, there is no benefit in retaining such data in the cache, and it may be detrimental if the stored data evictes other useful data from the cache. In some embodiments, such non-ephemeral workloads may be identified based on an analysis of cache activity metrics for local and peer caches (e.g., cache hit / miss ratio, data source of cache misses, fetch / cast-out ratio, and percentage of cast-outs with modified data). For example, the lower the cache hit / miss ratio, the higher the fetch / cast-out ratio, and the higher the percentage of cast-outs involving modified data (casting out a modified data item means that the modified data item is read from the cache structure and written to permanent storage without being removed from the cache structure), the more likely it is that the data held in the cache corresponds to non-ephemeral workload data. Furthermore, the data source of a cache miss may indicate that the data stored in the cache corresponds to non-ephemeral data from a non-ephemeral workload. For example, algorithms that perform transformations on a data stream, such as reading data once from one location and writing it once to another, may result in data not being reused. In another example, algorithms that operate on datasets that do not fit in the cache may result in data not being reused.When a workload streaming data from memory is determined to be a non-ephemeral workload, the cache line associated with that non-ephemeral workload is identified. A cache line, as used herein, refers to a block of memory that the processor loads into the cache when it accesses a portion of memory that is not yet stored in the cache. Cache lines contain the actual data fetched from main memory, as well as directory stores, state information, and valid memory addresses. The persistence policy of cache lines associated with non-ephemeral workloads can be modified to minimize the impact of cache thrashing, for example, by shortening the cache line's lifetime (the length of time data is stored in the cache) or by directly casting the cache line to memory. In this way, the impact of cache thrashing associated with non-ephemeral workloads is minimized. Further explanation of these and other features is provided below.

[0047] In some embodiments of this disclosure, the disclosure comprises computer implementations, systems, and computer program products for minimizing the impact of cache thrashing associated with non-ephemeral workloads. In some embodiments of the disclosure, the cache activity of a workload (such as cache activity related to local and peer caches) is monitored. Examples of such activity include the number of cache hits and cache misses against the cache, the number of fetches and cast-outs, and the amount of data that the cache downloaded but did not add to its cache. Based on an analysis of metrics regarding such monitored cache activity, a determination is made as to whether a non-ephemeral workload is identified. A “non-ephemeral workload” as used herein refers to a workload that retains data that is not immediately reused (re-read) until, for example, it is evictioned (cast out). As a result, there is no benefit in retaining such data in the cache, and it may be detrimental if the stored data evictes other useful data from the cache. In some embodiments, such a determination is based on a comparison of metrics regarding the monitored cache activity of the workload with a user-specifiable threshold. For example, the cache hit / miss ratio, fetch / cast-out ratio, and / or the percentage of cast-outs with modified data may be compared to user-specified thresholds. For instance, if the cache hit / miss ratio falls below a threshold, it may be inferred that the workload corresponds to a non-ephemeral workload. Once a non-ephemeral workload is identified, the cache lines associated with that non-ephemeral workload are identified. Next, the persistence policy of the identified cache lines is modified. The persistence policy, as used herein, defines the rules used to determine how long data or cache lines can be retained or stored in the cache.In one embodiment, the persistence policy of a cache line associated with a non-ephemeral workload is modified by shortening the lifespan of such a cache line in the cache (the length of time data is stored in the cache). By shortening the length of time a cache line associated with a non-ephemeral workload is stored in the cache, such a cache no longer retains data that is not rereferenced for a long period of time, thereby reducing the number of cache misses or evictions. The reduction in the number of cache misses or evictions minimizes the impact of cache thrashing. In another embodiment, the persistence policy of a cache line associated with a non-ephemeral workload is modified by casting out (evictioning) the cache line associated with the non-ephemeral workload. By casting out a cache line from the cache associated with a non-ephemeral workload, such a cache no longer retains data that is not rereferenced, thereby reducing the number of cache misses or evictions. The reduction in the number of cache misses or evictions minimizes the impact of cache thrashing. In this way, the impact of cache thrashing by non-ephemeral workloads is minimized.

[0048] The following description includes many specific details to ensure a full understanding of the disclosure. However, it will be apparent to those skilled in the art that the disclosure can be implemented without such specific details. In other instances, well-known circuits are shown in block diagram form to avoid obscuring the disclosure with unnecessary details. In most cases, details considering timing considerations, etc., are omitted because such details are not necessary for a full understanding of the disclosure and are within the scope of the skills of those skilled in the art.

[0049] Referring here to the figures in detail, Figure 1 illustrates one embodiment of the present disclosure relating to a computing environment 100 for implementing the principles of the present disclosure.

[0050] Various aspects of this disclosure are described in the explanatory text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of computer program products (CPPs). With respect to any flowchart, depending on the relevant technology, operations may be performed in a different order than those shown in a particular flowchart. For example, also depending on the relevant technology, two operations shown in consecutive blocks of a flowchart may be performed in reverse order, as a single integrated step, simultaneously, or with at least partial time overlap.

[0051] The term "CPP embodiment" ("CPP embodiment" or "CPP") is used in this disclosure to describe any set of one or more storage media (also referred to as "multiple media") that collectively comprise a set of one or more storage devices containing machine-readable code corresponding to instructions and / or data for performing computer operations defined in a given CPP claim. "Storage device" is any tangible device capable of holding and storing instructions used by a computer processor. Computer-readable storage media can be, but are not limited to, electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, mechanical storage media, or any suitable combination thereof. Some known types of storage devices, including these media, include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices (e.g., punch cards, or pits / lands formed on the main surface of a disk), or any appropriate combination of the aforementioned devices. Computer-readable storage media, as used in this disclosure, should not be interpreted as storing signals that are essentially transient, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, light pulses passing through optical fiber cables, electrical signals transmitted through wires, and / or other transmission media.As those skilled in the art will understand, data is typically moved at several irregular points during the normal operation of a storage device, for example, during access, defragmentation, or garbage collection. However, this does not mean that the storage device is transient. This is because data is not transient while it is stored there.

[0052] The computing environment 100 includes an example of an environment for executing at least a portion of the computer code (stored in block 125) related to the execution of the method of the present invention (e.g., minimizing the impact of cache thrashing associated with non-temporary workloads). In addition to block 125, the computing environment 100 includes, for example, a computer 101, a network 124 such as a wide area network (WAN), an end user device (EUD) 102, a remote server 103, a public cloud 104, and a private cloud 105. In this embodiment, the computer 101 includes a processor set 106 (including processing circuits 107 and a cache 108), a communication fabric 109, volatile memory 110, persistent storage 111 (including an operating system 112 and the block 125 shown above), a peripheral device set 113 (including a user interface (UI) device set 114, storage 115, and an Internet of Things (IoT) sensor set 116), and a network module 117. The remote server 103 includes a remote database 118. The public cloud 104 includes a gateway 119, a cloud orchestration module 120, a host physical machine set 121, a virtual machine set 122, and a container set 123.

[0053] Computer 101 can take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device, currently known or to be developed in the future, that is capable of running programs, accessing networks, or querying databases such as remote database 118. As is well understood in the field of computer technology, and depending on the technology, the execution of a computer implementation may be distributed across multiple computers and / or multiple locations. On the other hand, in this description of the computing environment 100, in order to keep the description as concise as possible, we will focus on and discuss in detail a single computer, specifically computer 101. Computer 101 may be located in the cloud, even if it is not shown in the cloud in Figure 1. On the other hand, it is not necessary for computer 101 to be located in the cloud, except to the extent definitively shown.

[0054] The processor set 106 includes one or more computer processors of any type currently known or to be developed in the future. The processing circuits 107 may be distributed across multiple packages, for example, across multiple interconnected integrated circuit chips. The processing circuits 107 can implement multiple processor threads and / or multiple processor cores. The cache 108 is memory located within the processor chip package and is typically used for data or code that should be available for high-speed access by threads or cores operating on the processor set 106. The cache memory is typically divided into multiple levels depending on its relative proximity to the processing circuits. Alternatively, some or all of the cache for the processor set may be located "off-chip". In some computing environments, the processor set 106 may operate with qubits and be designed to perform quantum computing.

[0055] Computer-readable program instructions are typically loaded into computer 101, causing the processor set 106 of computer 101 to execute a series of operational steps, thereby performing the computer implementation method. As a result, the instructions thus executed instantiate the methods (collectively referred to as the "Methods of the Invention") specified in the flowcharts and / or explanatory texts of the computer implementation methods contained herein. These computer-readable program instructions are stored in various types of computer-readable storage media, for example, the cache 108 and other storage media described later. The program instructions and associated data are accessed by the processor set 106 to control and direct the execution of the Methods of the Invention. In computing environment 100, at least some of the instructions for executing the Methods of the Invention may be stored in block 125 of persistent storage device 111.

[0056] The communication fabric 109 is a signal transmission path that allows various components of the computer 101 to communicate with each other. Typically, this fabric is made up of switches and conductive paths, such as buses, bridges, and physical input / output ports. Other types of signal communication paths, such as optical fiber communication paths and / or wireless communication paths, may also be used.

[0057] Volatile memory 110 is any type of volatile memory currently known or to be developed in the future. Examples include dynamic random-access memory (RAM) or static RAM. Volatile memory typically features random access, but this is not mandatory unless explicitly stated. In computer 101, the volatile memory 110 is located within a single package and resides inside computer 101, but alternatively or additionally, the volatile memory may be distributed across multiple packages and / or located outside computer 101.

[0058] The persistent memory device 111 is any currently known or future-developed form of non-volatile memory for a computer. Non-volatility of this memory means that the stored data is maintained regardless of whether power is supplied to the computer 101 and / or directly to the persistent memory device 111. The persistent memory device 111 can be read-only memory (ROM), but typically at least a portion of the persistent memory device allows for data writing, data deletion, and data rewriting. Some well-known forms of persistent memory devices include magnetic disks and solid-state devices. The operating system 112 can take several forms, for example, various known proprietary operating systems, or open-source POSIX (Portable Operating System Interface) type operating systems using kernels. The code contained in block 125 typically includes at least a portion of computer code related to the execution of the method of the present invention.

[0059] The peripheral device set 113 includes a set of peripheral devices for the computer 101. Data communication connections between the computer 101's peripheral devices and other components can be implemented in various ways, such as Bluetooth® connections, Near-Field Communication (NFC) connections, cable connections (such as universal serial bus (USB) type cables), insertable connections (e.g., secure digital (SD) cards), connections via local area communication networks, and connections via wide area networks such as the internet. In various embodiments, the UI device set 114 may include components such as a display screen, speakers, microphones, wearable devices (such as goggles and smartwatches), keyboards, mice, printers, touchpads, game controllers, and haptic devices. The storage device 115 is an external storage device such as an external hard drive, or an insertable storage device such as an SD card. The storage device 115 can be persistent and / or volatile. In some embodiments, the storage device 115 may take the form of a quantum computing storage device that stores data in the form of qubits. In embodiments where computer 101 requires mass storage (for example, computer 101 locally stores and manages a large database), this storage may be provided by peripheral storage devices designed to store vast amounts of data, for example, by a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 116 consists of sensors that can be used in Internet of Things applications. For example, one sensor may be a thermometer and another may be a motion detector.

[0060] The network module 117 is a collection of computer software, hardware, and firmware that enables computer 101 to communicate with other computers via the WAN 124. The network module 117 may include hardware such as a modem or Wi-Fi® signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for exchanging data over the internet. In some embodiments, the network control and network forwarding functions of the network module 117 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing Software-Defined Networking (SDN)), the control and forwarding functions of the network module 117 are performed on physically separate devices, resulting in multiple different network hardware devices being managed by the control function. Computer-readable program instructions for performing the method of the present invention can typically be downloaded from an external computer or external storage device to computer 101 via a network adapter card or network interface included in the network module 117.

[0061] WAN124 is any wide area network (e.g., the Internet) that enables the exchange of computer data over long distances using any currently known or future-developed technology for exchanging computer data. In some embodiments, a WAN may be replaced and / or complemented by a local area network (LAN), such as a Wi-Fi® network, designed to exchange data between devices located in a local area. WANs and / or LANs typically include computer hardware such as copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0062] The end-user device (EUD) 102 is any computer system used and managed by an end-user (e.g., a customer of the company operating computer 101) and can take any of the forms described above in relation to computer 101. The EUD 102 typically receives useful data from the operation of computer 101. For example, in a hypothetical case where computer 101 is designed to provide recommendations to the end-user, these recommendations would typically be transmitted from computer 101's network module 117 to the EUD 102 via the WAN 124. In this way, the EUD 102 can display or otherwise present the recommendations to the end-user. In some embodiments, the EUD 102 can be a client device such as a thin client, heavy client, mainframe computer, or desktop computer.

[0063] The remote server 103 is any computer system that supplies at least some data and / or functionality to computer 101. The remote server 103 may be managed and used by the same entity that operates computer 101. The remote server 103 represents a machine that collects and stores useful data that is useful for use by other computers, such as computer 101. For example, in a hypothetical case where computer 101 is designed and programmed to make recommendations based on historical data, this historical data may be provided to computer 101 from the remote database 118 of the remote server 103.

[0064] The public cloud 104 is an arbitrary computer system available for use by multiple entities, providing on-demand availability of computer system resources and / or other computer functions, particularly data storage (cloud storage) and computing power, without requiring direct user management. Cloud computing typically leverages resource sharing to achieve consistency and economies of scale. Direct and voluntary management of the computing resources of the public cloud 104 is performed by the computer hardware and / or software of the cloud orchestration module 120. The computing resources provided by the public cloud 104 are typically implemented by virtual computing environments running on various computers that make up the host physical machine set 121. The host physical machine set refers to all physical computers present in and / or available for the public cloud 104. Virtual computing environments (VCEs) typically take the form of virtual machines from the virtual machine set 122 and / or containers from the container set 123. These VCEs can be stored as images and may be transferred as images within and between various physical machine hosts, or after the VCE is instantiated. The cloud orchestration module 120 manages the transfer and storage of images, deploys new instances of VCEs, and manages active instances resulting from VCE deployments. The gateway 119 is a collection of computer software, hardware, and firmware that enables the public cloud 104 to communicate over the WAN 124.

[0065] Here, we will provide some further explanation of virtual computing environments (VCEs). A VCE can be stored as an "image." A new active instance of a VCE can be instantiated from that image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature where the kernel enables the existence of multiple isolated user-space instances called containers. These isolated user-space instances typically behave like actual computers from the perspective of the programs running within them. Computer programs running on a normal operating system can utilize all of that computer's resources, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running inside a container can only use the contents of the container and the devices allocated to the container; this feature is known as containerization.

[0066] The private cloud 105 is similar to the public cloud 104, but differs in that its computing resources are available only for use by a single company. While the private cloud 105 is illustrated in other embodiments as interacting with the WAN 124, the private cloud may be completely isolated from the internet and accessible only via a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types), each often implemented by a different vendor. Each of the multiple clouds remains a separate and isolated entity, but in larger hybrid cloud architectures, they are integrated by standardized or proprietary technologies, enabling orchestration, management, and / or data / application portability across the multiple clouds that comprise them. In this embodiment, both the public cloud 104 and the private cloud 105 are part of a larger hybrid cloud.

[0067] Block 125 further includes software components that minimize the effects of cache thrashing associated with non-transient workloads, which are described herein in reference to Figure 3. In some embodiments, such components may be implemented as hardware. The functions performed by such components are not general-purpose computer functions. For this reason, computer 101 becomes a specific machine as a result of implementing specific non-general-purpose computer functions.

[0068] In one embodiment, the functionality of such a software component of computer 101 may include a function to minimize the effects of cache thrashing associated with non-transient workloads, and may be embodied as an application-specific integrated circuit.

[0069] One embodiment of computer 101 that implements a distributed symmetric multiprocessing (SMP) system utilizing cache persistence is described later in relation to Figure 2.

[0070] Figure 2 illustrates a distributed symmetric multiprocessing (SMP) system 200 utilizing cache persistence according to one embodiment of the present disclosure. In one embodiment, such a system 200 corresponds to a processing circuit 107 and a cache 108 of a computer 101.

[0071] Referring to Figure 2, the SMP system 200 includes processing units ("PUs") 201A to 201H. Processing units 201A to 201H may be referred to collectively or individually as multiple processing units (PUs) 201 or processing unit (PU) 201.

[0072] In one embodiment, each processing unit 201 includes eight microprocessor (CP) chips 202A-202H. The CP chips 202A-202H may be referred to collectively or individually as multiple CP chips 202 or CP chip 202. In one embodiment, a CP chip 202 corresponds to a single semiconductor chip integrating the functional units of a computer, such as arithmetic / logic, control, storage, input, and output.

[0073] In one embodiment, each CP chip 202 includes eight cores 203A to 203H. The cores 203A to 203H may be referred to collectively or individually as multiple cores 203 or core 203. In one embodiment, the cores 203 correspond to components of a computer processing unit (e.g., PU201) that executes instructions and processes data.

[0074] In one embodiment, each CP chip 202 further includes L1 (Level 1) caches 204A-204H (including both instruction caches and data caches). In one embodiment, the L1 caches 204A-204H are supported by L2 (Level 2) caches 205A-205H. The L1 caches 204A-204H may be referred to collectively or individually as multiple L1 caches 204 or L1 cache 204. The L2 caches 205A-205H may be referred to collectively or individually as multiple L2 caches 205 or L2 cache 205. In one embodiment, multiple L2 caches 205 interact with each other to provide a chip-mounted virtual L3 (Level 3) cache. In one embodiment, each PU 201 includes up to eight CP chips 202, and a virtual L4 (Level 4) cache is provided by a fully connected topology. Virtual L3 and virtual L4 caches can be implemented by a set of chip caching techniques that cluster independent physical L2 caches 205 within the CP chip 202 and PU201 to function as an integrated shared victim cache.

[0075] In one embodiment, a virtual L3 / L4 cache may be implemented by defining groups / clusters of L2 caches 205 residing in each of the CP chip 202, groups of CP chips 202, and / or PU201, in order to evict cache lines from peer caches. That is, cache lines are evicted from the first L2 cache 205 to peer L2 caches 205 residing in the defined groups / clusters of L2 caches 205, according to a defined substitution policy. A “peer cache” (also called a “lateral cache”), as used herein, refers to a cache used to hold data (e.g., cache lines) that has been evicted or cast out from another cache. For example, such a peer cache may be divided into clusters of caches (e.g., L2 caches 205A-205H in CP202A of PU201A form a first cluster, and L2 caches 205A-205H in CP202A of PU201B form a second cluster). Here, a cache line within the cache (for example, L2 cache 205A in CP202A of PU201A) is evictionated to and held in a peer cache within the cluster (for example, L2 cache 205A in CP202A of PU201B). If such a cache line is further evictionated, it may be held in another cluster until the evictionated cache line reaches the last cluster and can be evictionated to main memory 207 (for example, memory 110 in Figure 1).

[0076] Eviction (also called cast-out), as used herein, refers to moving a cache line out of the cache (e.g., L2 cache 205A). A cache line, as used herein, refers to a block of memory that the processor loads into the cache when it accesses a portion of memory that is not yet stored in the cache. Cache lines contain actual data fetched from main memory, as well as directory stores, state information, and valid memory addresses.

[0077] Furthermore, as shown in Figure 2, the system 200 includes a cache controller 206 connected to the PU 201 and the main memory 207. In one embodiment, lateral persistence and replacement policies are implemented using the cache controller 206, which manages cache evictions between cache clusters and evictions to the main memory 207. In one embodiment, the cache controller 206 may reside locally in the PU 201 or may be a distributed element residing in an instance for each cache cluster.

[0078] In one embodiment, the cache controller 206 is configured to minimize the impact of cache thrashing associated with non-ephemeral workloads. In one embodiment, the cache controller 206 monitors and analyzes cache activity metrics for local and peer caches (e.g., L1 cache 204, L2 cache 205) to determine whether a workload streaming data from memory (e.g., main memory 207) is a non-ephemeral workload. As used herein, a “non-ephemeral workload” refers to a workload that retains data that is not immediately reused (re-read) until, for example, it is evictioned (cast out). As a result, there is no benefit to retaining such data in the cache, and it may be detrimental if the stored data evictes other useful data from the cache.

[0079] In one embodiment, such non-ephemeral workloads may be identified based on an analysis of cache activity metrics (e.g., cache hit / miss ratio, data source of cache misses, fetch / cast-out ratio, and percentage of cast-outs with modified data) for local and peer caches (e.g., L1 cache 204, L2 cache 205). For example, the lower the cache hit / miss ratio, the higher the fetch / cast-out ratio, and the higher the percentage of cast-outs with modified data (casting out a modified data item means that the modified data item is read from the cache structure and written to permanent storage without the data item being removed from the cache structure), the more likely it is that the data held in the cache corresponds to non-ephemeral workload data. Furthermore, the data source of a cache miss may indicate that the data stored in the cache corresponds to non-ephemeral data from a non-ephemeral workload. For example, an algorithm that performs transformations on a data stream, such as reading data once from one location and writing it once to another location, may result in a state where data is not reused. In another example, an algorithm operating on a dataset that does not fit in the cache may result in a state where data is not reused.

[0080] When the cache controller 206 determines that a workload streaming data from memory is a non-transient workload, it identifies the cache line associated with that non-transient workload. A cache line, as used herein, refers to a block of memory that the processor loads into the cache when it accesses a portion of memory that is not yet stored in the cache. The cache line contains actual data fetched from main memory (e.g., main memory 207), as well as directory stores, state information, and valid memory addresses.

[0081] In one embodiment, in response to determining that a workload streaming data from memory is a non-ephemeral workload, the cache controller 206 modifies the persistence policy of the cache line associated with the non-ephemeral workload to minimize the impact of cache thrashing, for example, by shortening the lifetime of the cache line (the length of time the data is stored in the cache) or by directly casting the cache line to main memory 207.

[0082] A description of the software components that the cache controller 206 uses to minimize the impact of cache thrashing associated with non-transient workloads is provided below in relation to Figure 3.

[0083] Figure 3 shows a software component used by the cache controller 206 to minimize the impact of cache thrashing associated with non-transient workloads, according to one embodiment of the present disclosure.

[0084] Referring to Figure 3 in conjunction with Figures 1 and 2, the cache controller 206 includes a monitoring engine 301 configured to monitor the cache activity of the workload and to obtain metrics of such monitored cache activity. In one embodiment, the monitoring engine 301 monitors cache activity and obtains metrics of such monitored cache activity in the local cache and peer caches (e.g., L1 cache 204, L2 cache 205).

[0085] As used herein, a workload refers to a computational task, process, or data transaction that needs to be performed by a program. Such tasks may involve streaming data from main memory 207. If data fetched from main memory 207 is not yet stored in the cache and is likely to be accessed in the near future, a cache line containing the actual data fetched from main memory 207 is loaded into the cache (e.g., L1 cache 204, L2 cache 205) for future access. As used herein, a cache line refers to a block of memory that the processor loads into the cache when it accesses a portion of memory that is not yet stored in the cache. A cache line contains the actual data fetched from main memory 207, as well as directory stores, state information, and valid memory addresses.

[0086] In one embodiment, various cache activities are monitored and measured by the monitoring engine 301. For example, such activity is supplied data, which corresponds to the total amount of data supplied by the cache (e.g., L1 cache 204, L2 cache 205). Another example of monitored and measured cache activity is the amount of data that the cache (e.g., L1 cache 204, L2 cache 205) downloaded but did not add to its cache. Further examples of monitored and measured cache activity are supplied data from the origin, for example, the amount of data that the cache (e.g., L1 cache 204, L2 cache 205) downloaded from the internet. In addition, monitored and measured cache activity may include supplied data from peers, for example, the amount of data that the cache (e.g., L1 cache 204, L2 cache 205) downloaded from either of its peer caches. Another example of monitored and measured cache activity is supplied data to clients. This could correspond to the amount of data supplied by the cache (e.g., L1 cache 204, L2 cache 205) to the client computer / device. Another example of cache activity to be monitored and measured is data supplied to peers. This could correspond to the amount of data supplied to any of those peer caches. An additional example of cache activity is cache pressure. This corresponds to how urgently the cache (e.g., L1 cache 204, L2 cache 205) needs more disk space.

[0087] Other examples of cache activity monitored and measured by the monitoring engine 301 include the number of cache hits and cache misses of the cache (e.g., L1 cache 204). A cache hit, as used herein, refers to a situation where a data request can be satisfied without the cache needing to retrieve data from main memory 207 or elsewhere. That is, if there is a cache hit, the data is already stored in the cache and can be delivered to the user quickly and efficiently. A cache miss, as used herein, refers to a situation where there is a request to retrieve data from the cache, but the requested data is not currently in the cache. Furthermore, in some embodiments, in connection with monitoring cache misses, the monitoring engine 301 identifies the data source of the cache miss. Upon detecting a cache miss, the monitoring engine 301 can use various software tools to identify the data source of the cache miss. Such tools may include, but are not limited to, the Linux® Perf tool and VTune®.

[0088] Another example of cache activity monitored and measured by the monitoring engine 301 is the monitoring and measurement of the number of fetches and cast-outs. Fetch, as used herein, refers to retrieving data from a source, such as main memory 207. Cast-out, as used herein, refers to eviction, movement, or flushing data (such as a cache line) from the cache, which may be written to memory (e.g., main memory 207). Furthermore, in some embodiments, the monitoring engine 301 monitors and measures the percentage of cast-outs involving modified data. Cast-out of a modified data item refers to reading that data from the cache (e.g., L1 cache 204) and writing it to memory (e.g., main memory 207). However, even when a data item is cast out of the cache (e.g., L1 cache 204), the data item is not deleted from the cache (e.g., L1 cache 204); the data item remains in the cache (e.g., L1 cache 204).

[0089] Other examples of cache activity monitored and measured by the monitoring engine 301 include cast-ins and fetch hits. A cast-in, as used herein, refers to loading data such as a cache line into a cache (e.g., L1 cache 204). A fetch hit, as used herein, refers to a fetch request that fetches or retrieves data from a cache (e.g., L1 cache 204) and results in a cache hit.

[0090] In one embodiment, the monitoring engine 301 monitors and measures the various cache activities described above using various software tools. Such tools may include, but are not limited to, Apple's Activity Monitor, IBM's Integrated Analytics System, Amazon's ElastiCache, Linux's Perf tool, and VTune.

[0091] The cache controller 206 further includes an analysis engine 302 configured to analyze measurements or metrics related to monitored cache activity for the workload.

[0092] In one embodiment, the analysis engine 302 performs an analysis of metrics related to the monitored cache activity of the workload to determine whether the workload is non-transient. A “non-transient workload,” as used herein, refers to a workload that holds data that is not immediately reused (re-read) until, for example, it is evictioned (cast out). As a result, there is no benefit in holding such data in the cache, and it may be detrimental if the stored data evictes other useful data from the cache. In one embodiment, such a non-transient workload may be identified based on an analysis of cache activity metrics for the local cache and peer cache (e.g., cache hit / miss ratio, data source of cache misses, fetch / cast-out ratio, and percentage of cast-outs with changed data).

[0093] The cache hit / miss ratio refers to the ratio of cache hits to cache misses in the cache in question (e.g., L1 cache 204). The cache miss data source refers to the data source of cache misses identified by the monitoring engine 301. The fetch / cast-out ratio refers to the ratio of fetches to cast-outs in the cache in question. The percentage of cast-outs with changed data refers to the percentage of cast-outs associated with changed data.

[0094] As mentioned above, analyzing such metrics can indicate whether a workload is non-ephemeral. For example, a lower cache hit / miss ratio, a higher fetch / cast-out ratio, and a higher percentage of cast-outs involving modified data (casting out a modified data item means that the modified data item is read from the cache structure and written to permanent storage without being removed from the cache structure) are more likely to indicate that the data held in the cache corresponds to non-ephemeral workload data. Furthermore, the data source of a cache miss may indicate that the data stored in the cache (e.g., L1 cache 204) corresponds to non-ephemeral data from a non-ephemeral workload. For example, algorithms that perform transformations on a data stream, such as reading data once from one location and writing it once to another, may result in data not being reused. In another example, algorithms that operate on datasets that do not fit in the cache may result in data not being reused.

[0095] Furthermore, in one embodiment, when analyzing metrics related to monitored cache activity of a workload to determine whether the workload is non-transient, metrics resulting from feedback provided by peer caches, such as cast-ins and fetch hits, may be included. Such metrics may be analyzed by the analysis engine 302 as a cast-in / fetch hit ratio. This ratio corresponds to the ratio of cast-ins to fetch hits in the cache in question. A lower cast-in / fetch hit ratio indicates a higher probability that the data held in the cache corresponds to data from a non-transient workload.

[0096] In one embodiment, the analysis engine 302 determines whether a workload is non-temporary based on a comparison of such indicator values ​​with a threshold. This threshold can be user-specified. For example, the cache hit / miss ratio, the fetch / cast-out ratio, and / or the percentage of cast-outs with modified data can be compared with a user-specified threshold. For example, if the cache hit / miss ratio is below the threshold, it can be inferred that the workload corresponds to a non-temporary workload. If the fetch / cast-out ratio is above the threshold, it can be inferred that the workload corresponds to a non-temporary workload. If the percentage of cast-outs with modified data is above the threshold, it can be inferred that the workload corresponds to a non-temporary workload.

[0097] In another example, the cast-in / fetch-hit ratio metric can be compared to a threshold. If the cast-in / fetch-hit ratio falls below the threshold, it can be inferred that the workload corresponds to a non-temporary workload.

[0098] In one embodiment, the analysis engine 302 analyzes metrics related to the monitored cache activity of the workload described above using various software tools. Such tools may include, but are not limited to, Apple's Activity Monitor, IBM's Integrated Analytics System, Amazon's ElastiCache, Linux's Perf tool, and VTune.

[0099] Furthermore, the cache controller 206 includes a tracking engine 303 configured to identify cache lines associated with non-ephemeral workloads.

[0100] In one embodiment, the tracking engine 303 tracks data streamed from memory (e.g., main memory 207) by the workload. Copies of such data are stored or cached in a cache (e.g., L1 cache 204) as cache lines. The cache lines contain the actual data fetched from main memory, as well as directory stores, state information, and valid memory addresses. By tracking the data streamed from memory (e.g., main memory 207) by the workload and the cache lines loaded into the cache (e.g., L1 cache 204), the tracking engine 303 can identify cache lines associated with workloads, such as non-ephemeral workloads.

[0101] In one embodiment, the tracking engine 303 performs such tracking using various software tools. Such tools may include, but are not limited to, the Linux® Perf tool, Datadog®, Redis®, and others.

[0102] Furthermore, the cache controller 206 includes a persistence policy engine 304 configured to modify persistence policies, for example, the persistence policy of a cache line associated with a non-ephemeral workload. A persistence policy, as used herein, defines rules used to determine how long data or a cache line can be retained or stored in the cache (e.g., the L1 cache 204).

[0103] In one embodiment, the persistence policy engine 304 modifies the persistence policy of a cache line associated with a non-ephemeral workload. For example, it shortens the lifetime (the length of time data is stored in the cache) of such a cache line in a cache (e.g., the L1 cache 204). In one embodiment, the persistence policy engine 304 shortens the lifetime of a cache line associated with a non-ephemeral workload by setting a specific value in a hint bit (a bit that can indicate the lifetime of the associated cache line). By shortening the length of time a cache line associated with a non-ephemeral workload is stored in the cache, such a cache no longer holds data that is not rereferenced for a long time, thereby reducing the number of cache misses or evictions. The reduction in the number of cache misses or evictions minimizes the impact of cache thrashing.

[0104] In another embodiment, the persistence policy engine 304 modifies the persistence policy of cache lines associated with non-ephemeral workloads to cast out (eviction) those cache lines. By casting out cache lines from caches associated with non-ephemeral workloads (e.g., L1 cache 204), such caches no longer store data that will not be rereferenced, thereby reducing the number of cache misses or evictions. Reducing the number of cache misses or evictions minimizes the impact of cache thrashing.

[0105] In one embodiment, the persistence policy engine 304 modifies the persistence policy using various software tools. Such tools may include, but are not limited to, CloudFront®, Amazon®'s ElastiCache®, and Intel®'s Cache Acceleration Software.

[0106] A description of how to minimize the impact of cache thrashing caused by non-transient workloads is provided below in relation to Figure 4.

[0107] Figure 4 is a flowchart of a method 400 for minimizing the impact of cache thrashing associated with non-transient workloads, according to one embodiment of the present disclosure.

[0108] Referring to Figure 4 in conjunction with Figures 1 to 3, in step 401, the monitoring engine 301 of the cache controller 206 monitors the cache activity of the workload and obtains measurements of the monitored cache activity.

[0109] As described above, in one embodiment, the monitoring engine 301 monitors cache activity and obtains metrics regarding such monitored cache activity in the local cache and peer caches (e.g., L1 cache 204, L2 cache 205).

[0110] As used herein, a workload refers to a computational task, process, or data transaction that needs to be performed by a program. Such tasks may involve streaming data from main memory 207. If data fetched from main memory 207 is not yet stored in the cache and is likely to be accessed in the near future, a cache line containing the actual data fetched from main memory 207 is loaded into the cache (e.g., L1 cache 204, L2 cache 205) for future access. As used herein, a cache line refers to a block of memory that the processor loads into the cache when it accesses a portion of memory that is not yet stored in the cache. A cache line contains the actual data fetched from main memory 207, as well as directory stores, state information, and valid memory addresses.

[0111] In one embodiment, various cache activities are monitored and measured by the monitoring engine 301. For example, such activity is supplied data, which corresponds to the total amount of data supplied by the cache (e.g., L1 cache 204, L2 cache 205). Another example of monitored and measured cache activity is the amount of data that the cache (e.g., L1 cache 204, L2 cache 205) downloaded but did not add to its cache. Further examples of monitored and measured cache activity are supplied data from the origin, for example, the amount of data that the cache (e.g., L1 cache 204, L2 cache 205) downloaded from the internet. In addition, monitored and measured cache activity may include supplied data from peers, for example, the amount of data that the cache (e.g., L1 cache 204, L2 cache 205) downloaded from either of its peer caches. Another example of monitored and measured cache activity is supplied data to clients. This could correspond to the amount of data supplied by the cache (e.g., L1 cache 204, L2 cache 205) to the client computer / device. Another example of cache activity to be monitored and measured is data supplied to peers. This could correspond to the amount of data supplied to any of those peer caches. An additional example of cache activity is cache pressure. This corresponds to how urgently the cache (e.g., L1 cache 204, L2 cache 205) needs more disk space.

[0112] Other examples of cache activity monitored and measured by the monitoring engine 301 include the number of cache hits and cache misses of the cache (e.g., L1 cache 204). A cache hit, as used herein, refers to a situation where a data request can be satisfied without the cache needing to retrieve data from main memory 207 or elsewhere. That is, if there is a cache hit, the data is already stored in the cache and can be delivered to the user quickly and efficiently. A cache miss, as used herein, refers to a situation where there is a request to retrieve data from the cache, but the requested data is not currently in the cache. Furthermore, in some embodiments, in connection with monitoring cache misses, the monitoring engine 301 identifies the data source of the cache miss. Upon detecting a cache miss, the monitoring engine 301 can use various software tools to identify the data source of the cache miss. Such tools may include, but are not limited to, the Linux® Perf tool and VTune®.

[0113] Another example of cache activity monitored and measured by the monitoring engine 301 is the monitoring and measurement of the number of fetches and cast-outs. Fetch, as used herein, refers to retrieving data from a source, such as main memory 207. Cast-out, as used herein, refers to eviction, movement, or flushing data (such as a cache line) from the cache, which may be written to memory (e.g., main memory 207). Furthermore, in some embodiments, the monitoring engine 301 monitors and measures the percentage of cast-outs involving modified data. Cast-out of a modified data item refers to reading that data from the cache (e.g., L1 cache 204) and writing it to memory (e.g., main memory 207). However, even when a data item is cast out of the cache (e.g., L1 cache 204), the data item is not deleted from the cache (e.g., L1 cache 204); the data item remains in the cache (e.g., L1 cache 204).

[0114] Other examples of cache activity monitored and measured by the monitoring engine 301 include cast-ins and fetch hits. A cast-in, as used herein, refers to loading data such as a cache line into a cache (e.g., L1 cache 204). A fetch hit, as used herein, refers to a fetch request that fetches or retrieves data from a cache (e.g., L1 cache 204) and results in a cache hit.

[0115] In one embodiment, the monitoring engine 301 monitors and measures the various cache activities described above using various software tools. Such tools may include, but are not limited to, Apple's Activity Monitor, IBM's Integrated Analytics System, Amazon's ElastiCache, Linux's Perf tool, and VTune.

[0116] In step 402, the analysis engine 302 of the cache controller 206 analyzes metrics related to the monitored cache activity of the workload.

[0117] As described above, in one embodiment, the analysis engine 302 performs an analysis of metrics related to the monitored cache activity of the workload to determine whether the workload is non-transient. A “non-transient workload,” as used herein, refers to a workload that holds data that is not immediately reused (re-read) until, for example, it is evictioned (cast out). As a result, there is no benefit in holding such data in the cache, and it may be detrimental if the stored data evictes other useful data from the cache. In one embodiment, such a non-transient workload may be identified based on an analysis of cache activity metrics for the local cache and peer cache (e.g., cache hit / miss ratio, data source of cache misses, fetch / cast-out ratio, and percentage of cast-outs with changed data).

[0118] The cache hit / miss ratio refers to the ratio of cache hits to cache misses in the cache in question (e.g., L1 cache 204). The cache miss data source refers to the data source of cache misses identified by the monitoring engine 301. The fetch / cast-out ratio refers to the ratio of fetches to cast-outs in the cache in question. The percentage of cast-outs with changed data refers to the percentage of cast-outs associated with changed data.

[0119] As mentioned above, analyzing such metrics can indicate whether a workload is non-ephemeral. For example, a lower cache hit / miss ratio, a higher fetch / cast-out ratio, and a higher percentage of cast-outs involving modified data (casting out a modified data item means that the modified data item is read from the cache structure and written to permanent storage without being removed from the cache structure) are more likely to indicate that the data held in the cache corresponds to non-ephemeral workload data. Furthermore, the data source of a cache miss may indicate that the data stored in the cache (e.g., L1 cache 204) corresponds to non-ephemeral data from a non-ephemeral workload. For example, algorithms that perform transformations on a data stream, such as reading data once from one location and writing it once to another, may result in data not being reused. In another example, algorithms that operate on datasets that do not fit in the cache may result in data not being reused.

[0120] Furthermore, in one embodiment, when analyzing metrics related to monitored cache activity of a workload to determine whether the workload is non-transient, metrics resulting from feedback provided by peer caches, such as cast-ins and fetch hits, may be included. Such metrics may be analyzed by the analysis engine 302 as a cast-in / fetch hit ratio. This ratio corresponds to the ratio of cast-ins to fetch hits in the cache in question. A lower cast-in / fetch hit ratio indicates a higher probability that the data held in the cache corresponds to data from a non-transient workload.

[0121] In one embodiment, the analysis engine 302 analyzes metrics related to the monitored cache activity of the workload described above using various software tools. Such tools may include, but are not limited to, Apple's Activity Monitor, IBM's Integrated Analytics System, Amazon's ElastiCache, Linux's Perf tool, and VTune.

[0122] In step 403, the analysis engine 302 of the cache controller 206 determines whether a non-transient workload has been identified based on an analysis of metrics regarding the monitored cache activity of the workload.

[0123] In one embodiment, such a determination is made by the analysis engine 302 based on a comparison of one or more metrics relating to the monitored cache activity of the workload with a user-specified threshold. For example, the cache hit / miss ratio, the fetch / cast-out ratio, and / or the percentage of cast-outs with modified data may be compared with a user-specified threshold. For example, if the cache hit / miss ratio falls below a threshold, it may be inferred that the workload corresponds to a non-temporary workload. If the fetch / cast-out ratio exceeds a threshold, it may be inferred that the workload corresponds to a non-temporary workload. If the percentage of cast-outs with modified data exceeds a threshold, it may be inferred that the workload corresponds to a non-temporary workload.

[0124] In another example, the cast-in / fetch-hit ratio metric can be compared to a threshold. If the cast-in / fetch-hit ratio falls below the threshold, it can be inferred that the workload corresponds to a non-temporary workload.

[0125] In a further example, a data source causing a cache miss may indicate that the data stored in the cache (e.g., L1 cache 204) corresponds to non-ephemeral data from a non-ephemeral workload. For instance, an algorithm that performs transformations on a data stream, such as reading data once from one location and writing it once to another, may result in data not being reused. Another example is an algorithm operating on a dataset that does not fit in the cache, which will result in data not being reused.

[0126] If non-transient workloads are not identified based on the analysis of metrics regarding the monitored cache activity of the workload, in step 401, the monitoring engine 301 of the cache controller 206 continues to monitor the cache activity of the workload and obtains metrics regarding the monitored cache activity.

[0127] However, if a non-ephemeral workload is identified based on an analysis of metrics regarding the monitored cache activity of the workload, in step 404, the tracking engine 303 of the cache controller 206 identifies the cache line associated with the non-ephemeral workload.

[0128] As described above, in one embodiment, the tracking engine 303 tracks data streamed from memory (e.g., main memory 207) by the workload. Copies of such data are stored or cached in a cache (e.g., L1 cache 204) as cache lines. The cache lines contain the actual data fetched from main memory, as well as directory stores, state information, and valid memory addresses. By tracking the data streamed from memory (e.g., main memory 207) by the workload and the cache lines loaded into the cache (e.g., L1 cache 204), the tracking engine 303 can identify cache lines associated with workloads, such as non-ephemeral workloads.

[0129] In one embodiment, the tracking engine 303 performs such tracking using various software tools. Such tools may include, but are not limited to, the Linux® Perf tool, Datadog®, Redis®, and others.

[0130] In step 405, the persistence policy engine 304 of the cache controller 206 modifies the persistence policy, for example, the persistence policy for a cache line associated with a non-ephemeral workload. A persistence policy, as used herein, defines the rules used to determine how long data or a cache line can be retained or stored in the cache (e.g., the L1 cache 204).

[0131] As described above, in one embodiment, the persistence policy engine 304 modifies the persistence policy of a cache line associated with a non-ephemeral workload. For example, it shortens the lifetime (the length of time data is stored in the cache) of such a cache line in a cache (e.g., the L1 cache 204). In one embodiment, the persistence policy engine 304 shortens the lifetime of a cache line associated with a non-ephemeral workload by setting a specific value in a hint bit (a bit that can indicate the lifetime of the associated cache line). By shortening the length of time a cache line associated with a non-ephemeral workload is stored in the cache, such a cache no longer holds data that is not rereferenced for a long time, thereby reducing the number of cache misses or evictions. The reduction in the number of cache misses or evictions minimizes the impact of cache thrashing.

[0132] In another embodiment, the persistence policy engine 304 modifies the persistence policy of cache lines associated with non-ephemeral workloads to cast out (eviction) those cache lines. By casting out cache lines from caches associated with non-ephemeral workloads (e.g., L1 cache 204), such caches no longer store data that will not be rereferenced, thereby reducing the number of cache misses or evictions. Reducing the number of cache misses or evictions minimizes the impact of cache thrashing.

[0133] In one embodiment, the persistence policy engine 304 modifies the persistence policy using various software tools. Such tools may include, but are not limited to, CloudFront®, Amazon®'s ElastiCache®, and Intel®'s Cache Acceleration Software.

[0134] In this way, the impact of cache thrash caused by non-transient workloads is minimized.

[0135] Furthermore, the principles of this disclosure improve the techniques or technologies related to cache thrashing in cache mechanisms.

[0136] As mentioned above, cache thrashing can occur when a workload streams large amounts of data from memory. A workload refers to a computational task, process, or data transaction that needs to be performed by a program. Examples of such workloads that stream large amounts of data from memory include streaming real-time data, such as location information, stock prices, information technology system monitoring, fraud detection, retail inventory, sales, and customer activity. Workloads that stream large amounts of data from memory (e.g., main memory) can negatively impact overall system performance by causing thrashing at multiple cache levels (e.g., constantly refreshing the cache with new data or repeatedly accessing data that exceeds the cache size). This can result in an increased number of cache misses or data evictions, which may cause the processor to access slower main memory to fetch the requested data. Furthermore, such workloads can consume system resources to hold data that will not be rereferenced in the cache (meaning storing data in the cache for a certain period of time). This can result in an increased number of cache misses or data evictions, which may cause the processor to access slower main memory to fetch the requested data. Techniques have been developed to address cache thrashing. For example, one such approach uses different cache levels or partitions to reduce cache contention. Another example attempts to optimize cache size, for instance, by dividing large amounts of data into smaller blocks after determining the cache size, so that it can fit into the cache. Yet another approach optimizes policies, such as by tailoring the cache replacement policy to the workload. Furthermore, the cache-aside pattern may be used to address cache thrashing, for example, by using microservice checks to determine whether data is available in the cache before accessing it.Despite these techniques for addressing cache thrashing, they fail to address the impact of cache thrashing in situations where the workload streams large amounts of data from memory (e.g., main memory). In this case, thrashing occurs at multiple cache levels, negatively impacting overall system performance, and system resources are consumed as data that is not rereferenced is held in the cache.

[0137] Embodiments of this disclosure improve such techniques by monitoring the cache activity of a workload (such as cache activity related to local and peer caches). Examples of such activity include the number of cache hits and cache misses, the number of fetches and castouts, and the amount of data that the cache downloaded but did not add to its cache. Based on the analysis of metrics related to such monitored cache activity, a determination is made as to whether a non-ephemeral workload is identified. A “non-ephemeral workload,” as used herein, refers to a workload that retains data that is not immediately reused (re-read) until, for example, it is evictioned (cast out). As a result, there is no benefit in retaining such data in the cache, and it may be detrimental if the stored data evictes other useful data from the cache. In some embodiments, such a determination is based on a comparison of metrics related to the monitored cache activity of a workload with user-specified thresholds. For example, the cache hit / miss ratio, the fetch / castout ratio, and / or the percentage of castouts with modified data may be compared with user-specified thresholds. For example, if the cache hit / miss ratio falls below a threshold, it may be inferred that the workload corresponds to a non-ephemeral workload. When a non-ephemeral workload is identified, the cache line associated with that non-ephemeral workload is identified. Next, the persistence policy of the identified cache line is modified. A persistence policy, as used herein, defines the rules used to determine how long data or a cache line can be retained or stored in the cache. In one embodiment, the persistence policy of a cache line associated with a non-ephemeral workload is modified by shortening the lifespan of such a cache line in the cache (the length of time the data is stored in the cache).By reducing the length of time that cache lines associated with non-ephemeral workloads are stored in the cache, such caches no longer retain data that is not rereferenced for extended periods, thereby reducing the number of cache misses or evictions. The reduction in cache misses or evictions minimizes the impact of cache thrashing. In another embodiment, the persistence policy of cache lines associated with non-ephemeral workloads is modified by casting out (evictioning) the cache lines associated with the non-ephemeral workloads. By casting out cache lines from the cache associated with the non-ephemeral workloads, such caches no longer retain data that is not rereferenced, thereby reducing the number of cache misses or evictions. The reduction in cache misses or evictions minimizes the impact of cache thrashing. In this way, the impact of cache thrashing caused by non-ephemeral workloads is minimized. Furthermore, this method brings about improvements in the technical aspects related to cache thrashing in caching mechanisms.

[0138] The descriptions of the various embodiments of this disclosure are presented for illustrative purposes only and are not intended to be exhaustive or limitful to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the embodiments described. The terms used herein have been selected to best describe the principles, practical applications, or technical improvements to the technologies available on the market of the embodiments, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer implementation method that minimizes the effects of cache thrashing, The stage of monitoring workload caching activity; A step of identifying non-transient workloads based on the monitored cache activity of the said workload; The step of identifying one or more cache lines associated with the non-temporary workload; and The step of modifying the persistence policy of one or more cache lines associated with the aforementioned non-temporary workload. A method for providing this.

2. The method according to claim 1, wherein the persistence policy is modified to shorten the lifespan of one or more cash lines.

3. The method according to claim 2, wherein the persistence policy is modified to shorten the lifespan of one or more cache lines by setting a specific value to a hint bit.

4. The method according to claim 1, wherein the persistence policy is modified to cast out one or more cache lines directly to memory.

5. The step of analyzing metrics related to the monitored cache activity of the workload. The method according to any one of claims 1 to 4, further comprising:

6. The method according to claim 5, wherein the metrics relating to the monitored cache activity of the workload include one or more of the following, selected from the group consisting of cache hit / miss ratio, data source of cache misses, fetch / castout ratio, and percentage of castouts with changed data.

7. The method according to claim 5, wherein the cache activity of the workload is monitored in association with a peer cache, and the metrics relating to the monitored cache activity of the workload include the cast-in / fetch hit ratio provided by the peer cache.

8. The method according to any one of claims 1 to 4, wherein the cache activity of the workload is monitored in association with local cache and peer cache.

9. The method according to any one of claims 1 to 4, wherein the workload streams data from memory.

10. A computer program for minimizing the effects of cache thrashing, wherein the computer program comprises program code, and the program code is Steps to monitor workload cache activity; A procedure for identifying non-transient workloads based on the monitored cache activity of the said workload; A procedure for identifying one or more cache lines associated with the non-temporary workload; and Procedure for modifying the persistence policy of one or more cache lines associated with the aforementioned non-temporary workload. A computer program that includes program instructions that cause a processor to execute.

11. The computer program according to claim 10, wherein the persistence policy is modified to shorten the lifespan of one or more cache lines.

12. The computer program according to claim 11, wherein the persistence policy is modified to shorten the lifespan of one or more cache lines by setting a specific value to a hint bit.

13. The computer program according to claim 10, wherein the persistence policy is modified to cast out one or more cache lines directly to memory.

14. The aforementioned program code further, Procedure for analyzing metrics related to the monitored cache activity of the workload. A computer program according to any one of claims 10 to 13, comprising program instructions relating to the following.

15. The computer program according to claim 14, wherein the metrics relating to the monitored cache activity of the workload include one or more of the following, selected from the group consisting of cache hit / miss ratio, data source of cache misses, fetch / castout ratio, and percentage of castouts with changed data.

16. The computer program according to claim 14, wherein the cache activity of the workload is monitored in association with a peer cache, and the metrics relating to the monitored cache activity of the workload include the cast-in / fetch hit ratio provided by the peer cache.

17. The computer program according to any one of claims 10 to 13, wherein the cache activity of the workload is monitored in association with local cache and peer cache.

18. Memory for storing computer programs to minimize the effects of cache thrashing; and A processor connected to the memory, where the processor Steps to monitor workload cache activity; A procedure for identifying non-transient workloads based on the monitored cache activity of the said workload; A procedure for identifying one or more cache lines associated with the non-temporary workload; and Procedure for modifying the persistence policy of one or more cache lines associated with the aforementioned non-temporary workload. The computer program is configured to execute program instructions, including those of the computer program. A system equipped with these features.

19. The system according to claim 18, wherein the persistence policy is modified to shorten the lifespan of one or more cash lines.

20. The system according to claim 19, wherein the persistence policy is modified to shorten the lifespan of one or more cache lines by setting a specific value to a hint bit.

21. The system according to claim 18, wherein the persistence policy is modified to cast out one or more cache lines directly to memory.

22. The program instructions of the aforementioned computer program further, Procedure for analyzing metrics related to the monitored cache activity of the workload. The system according to any one of claims 18 to 21, including the system described in any one of claims 18 to 21.

23. The system according to claim 22, wherein the metrics relating to the monitored cache activity of the workload include one or more of the following, selected from the group consisting of cache hit / miss ratio, data source of cache misses, fetch / castout ratio, and percentage of castouts with changed data.

24. The system according to claim 22, wherein the cache activity of the workload is monitored in association with a peer cache, and the metrics relating to the monitored cache activity of the workload include the cast-in / fetch hit ratio provided by the peer cache.

25. The system according to any one of claims 18 to 21, wherein the cache activity of the workload is monitored in association with local cache and peer cache.

Images