Multilevel global cache

Abstract

This disclosure relates to a multi-tier global cache. One aspect of the disclosure provides a cache system for a computer cluster comprising a plurality of nodes. The cache system can comprise a global cache distributed across a subset of the nodes of the computer cluster. Respective nodes in the subset contribute a portion of local storage resources to the global cache and run a cache manager process to: receive a request for a data segment; determine whether the requested data is present in the global cache; responsive to locating the requested data in the portion of local storage resources contributed to the global cache by a contributing node, retrieve the requested data by accessing the local storage resources of the contributing node; and use the retrieved data to respond to the request.

Description

[0001] Related applications This application claims the rights of the inventors, Clarette Riana Krasta, John L. Byrne, Abishek Dwaraki, David R. Emberson, H. Kuno, Li Shiquan, Sharad Sinhar, Ramya Bangalore Ahobalalao, Shreyas Vineka Basri KS, Amita Chandrachari, Chimmai Ghosh, Rishi Keshkumar Rajak, Sriram Ravishanka, and Bono Som, to Indian Provisional Application No. 202511034562, Attorney General’s Case No. HPE-P176945INPRV, entitled “Globally Distributed Client-Side Cache for DAOS”, filed on 8 April 2025. Technical Field

[0002] This disclosure relates to a caching system. More specifically, this disclosure relates to a multi-tiered global caching system shared among applications running across a cluster of computers. Background Technology

[0003] High-performance computing (HPC) and artificial intelligence (AI) workloads require efficient processing of petabyte-scale (PB) data, where file system access latency can severely impact performance. Distributed workloads face unique challenges because they rely on centralized storage systems to share intermediate results across nodes. To address latency issues, modern file systems employ various caching methods, such as server-side caching and client-side caching. Server-side caching stores frequently accessed data in fast storage (e.g., dynamic random access memory (DRAM)) on the file server, but is limited by server memory capacity and network bandwidth. Client-side caching stores frequently accessed data locally (using, for example, solid-state drives or DRAM). However, traditional client-side caching typically restricts data reuse to the local node, thus limiting the advantages of high-speed cache sharing across the entire cluster. Attached Figure Description

[0004] Figure 1 An example computer cluster implementing a global cache according to one aspect of this disclosure is shown.

[0005] Figure 2 An example functional block diagram of a cache manager according to one aspect of this disclosure is shown.

[0006] Figure 3 A flowchart illustrating a sample application read operation processed by a global cache according to one aspect of this application is shown.

[0007] Figure 4 A computer system for facilitating the implementation of a global cache, according to one aspect of this disclosure, is shown.

[0008] Figure 5 A computer-readable medium (CRM) facilitating the implementation of a multi-layered distributed global cache according to one aspect of this disclosure is shown.

[0009] In the accompanying drawings, similar reference numerals refer to the same elements. Detailed Implementation

[0010] Existing caching systems are typically siloed—optimized for specific use cases such as application, file system, or object storage acceleration—leading to fragmented resource allocation across clusters with heterogeneous workloads and diverse data needs. As accelerators and GPUs become more prevalent, traditional CPU-centric caches are becoming increasingly inadequate. This disclosure addresses these challenges by introducing a multi-tiered global cache on the client side that unifies various memory and storage technologies, facilitates data location lookups, and allows for the co-deployment of compute tasks with cached data, thereby reducing latency and improving resource utilization.

[0011] As the number of compute nodes in an HPC cluster increases, the aggregated resources available for client-side caching (such as DRAM capacity and network bandwidth) also grow proportionally. Providing a shared client-side cache across multiple nodes becomes increasingly advantageous for optimizing data access and reducing redundant storage. According to some aspects of this disclosure, a client-side global cache can aggregate various storage resources (e.g., DRAM, Non-Volatile Memory High Speed ​​(NVMe), High Bandwidth Memory (HBM), or Storage Class Memory (SCM)) contributed by multiple nodes within the cluster, including both compute nodes and dedicated memory-server nodes.

[0012] Figure 1 An example computer cluster implementing a global cache according to one aspect of this disclosure is shown. Computer cluster 100 may include multiple interconnected nodes, such as compute nodes 102 and 104, memory servers 106 and 108, and storage servers 110 and 112.

[0013] As used herein, the term "node" can refer to one or more computing devices configured to provide the functions described herein. A "computing device" can be a server, server cluster, storage array, computer device, workstation, desktop computer, portable computer, switch, router, or any other processing device or device that includes processing resources. A node may include processing resources communicatively coupled to at least one non-transitory computer-readable storage medium including instructions that, when executed by the processing resource, cause the node to perform certain operations and functions described herein.

[0014] Compute nodes may include dedicated servers optimized for performing computational tasks, such as data processing, simulation, or application execution. Additionally, they may provide a pool of memory to the client-side global cache. Memory servers may include dedicated servers specifically designed to provide a large, shared pool of memory (e.g., distributed DRAM or SCM) to other nodes in the cluster and to the client-side global cache. Compute nodes and memory servers in the cluster may include different types of storage resources, including but not limited to: persistent memory (e.g., non-volatile dual in-line memory modules (NVDIMMs) and SCMs), SSDs, DRAM, GPU memory (e.g., HBM), and any type of computer-readable storage media that provides high data access speeds. The term "storage resource" may refer to memory, disk space, or storage devices (such as HDDs, SSDs, or NVMe drives) available on a specific node within the cluster.

[0015] Different types of storage resources can be organized into multiple tiers, with the highest tier providing the highest speed. In one example, the highest tier (or tier 1) may include DRAM and GPU memory, the second tier (or tier 2) may include persistent memory such as SCM and NVDIMM, the third tier (or tier 3) may include SSDs, and the lowest tier (or tier 4) may include large-capacity persistent storage (e.g., hard disk drives). In further examples, each type of storage resource can form different tiers within a cache. Storage resources can be used to store instructions, data, application files, temporary files, logs, backups, and any information related to node operations.

[0016] exist Figure 1 In the example shown, compute node 102 includes GPU 114 and local memory 116. Compute node 102 can contribute a portion of local memory 116 to a global cache; this contribution... Figure 1The global cache portion 118 is represented as residing on compute node 102. Compute node 104 is similar to compute node 102 but includes an additional local storage module 122, which may include an SSD. Compute node 104 may contribute its local memory and / or a portion of local storage module 122 to the global cache. In some examples, when the global cache exhausts its DRAM capacity, the space within local storage module 122 (e.g., a local SSD) can be used as an overflow region. SSDs provide a good balance between high-speed, capacity-limited DRAM (i.e., the highest layer) and a larger-capacity parallel distributed persistent storage layer (i.e., the lowest layer). By leveraging the fast parallel I / O of SSDs, frequently accessed data can be stored and retrieved with reduced latency, thereby improving compute throughput. Incorporating local SSDs into the global cache also achieves cost-effectiveness, as SSDs offer a lower cost per byte than DRAM while maintaining performance closer to memory speeds.

[0017] Memory server 106 includes local memory 124 and local storage module 126. Memory server 106 may contribute a portion of local memory 124 and / or local storage module 126 to the global cache; this contribution is represented as a global cache portion 128 residing on memory server 106. Memory server 108 is similar to memory server 106 and includes local memory, but does not have a local storage module. Memory server 108 may contribute a portion of its local memory to the global cache.

[0018] Storage servers are designed to store, manage, and provide access to large amounts of data, and typically use multiple SSDs and hard disk drives (HDDs). Figure 1 In the example shown, storage servers 110 and 112 each include an array of local storage modules (e.g., module 132). The storage servers can provide distributed object storage (e.g., distributed asynchronous object storage (DAOS) or a distributed file system).

[0019] The global cache can support high-speed data movement within and across applications via a Remote Direct Memory Access (RDMA) network 140 that provides low-latency memory access. As previously mentioned, when DRAM capacity from individual nodes is exceeded, the global cache can seamlessly offload data to locally connected SSDs, thereby reducing overall DRAM requirements while maintaining performance.

[0020] A global cache can be used as an application cache, allowing data processed by one application to be retained in a cache for use by other applications, thereby improving efficiency and reducing redundant data movement. A global cache can also be used as a file system or object storage cache with minimal modifications. Different caching strategies can be implemented based on the configuration of the global cache. A global cache can provide compatibility with various file systems, such as Distributed Asynchronous Object Storage (DAOS) or the Lustre file system. This flexibility allows the cache to be used as a unified resource for applications running across a cluster.

[0021] A global cache can be shared across applications running on different nodes within a cluster (e.g., compute nodes 102 and 104). More specifically, the global cache can be partitioned, and portions of its total capacity can be allocated to individual applications based on several factors, including, for example, a target Quality of Service (QoS) value, a measured QoS value, a target hit rate, and a measured hit rate of the application over a given time interval. Allocating cache capacity to different applications is beyond the scope of this disclosure.

[0022] According to some aspects of this disclosure, a global cache can be managed by a distributed cache manager. This manager is responsible for integrating per-node resources across multiple tiers into a unified cache, maintaining awareness of the capacity and access latency associated with each tier, and implementing appropriate caching strategies and data placement algorithms. Taking application caching as an example, the distributed cache manager can be linked to the application and includes processes running on each node where the application runs. Figure 1 In the example shown, each node contributing to the global cache can run a cache manager process. For example, compute node 102 runs cache manager process 120, storage server 106 runs cache manager process 130, and so on. Cache manager processes running on different nodes can be synchronized via cache manager metadata, which can be stored in a shared storage location accessible to all nodes running the cache manager processes.

[0023] The cache manager is responsible for coordinating cache capacity and usage across different nodes (e.g., nodes 102-108) and memory / storage tiers (e.g., DRAM and SSD). The cache manager can monitor cache usage, handle configuration changes, and dynamically adjust cache capacity to optimize performance and resource utilization. Figure 2An example functional block diagram of a cache manager according to one aspect of this disclosure is shown. The cache manager 200 may include data access functionality 202, data consistency and correctness functionality 204, data location lookup functionality 206, cache eviction functionality 208, application programming interface (API) functionality 210, and cache manager metadata functionality 212. These functions may be implemented through software, hardware, firmware, or a combination thereof.

[0024] Data access function 202 can employ various mechanisms to access data across different storage and memory tiers. For example, data access function 202 can use operating system or file system interfaces to access data stored in SSDs, use RDMA to access data in CPU or GPU memory, and use native APIs to interface with distributed file systems (e.g., Lustre) or object storage (e.g., DAOS). Furthermore, data access function 202 can employ advanced data migration strategies to dynamically move data between memory and storage tiers based on real-time access patterns and application performance requirements. Frequently accessed data can be moved to faster memory tiers (e.g., DRAM or GPU memory), while less frequently used data can be moved to slower, more cost-effective storage tiers (e.g., SSDs or HDDs). When accessing GPU memory, data access function 202 can utilize GPUDirect features (e.g., GPUDirect RDMA or GPUDirect Memory, if supported) for efficient data transfer between GPU memory and the global cache.

[0025] In several respects, data access function 202 can integrate and adapt functions developed for network-attached memory (FAM) systems (e.g., functions provided by the OpenFAM framework) into the cache layer. OpenFAM provides efficient access to network-attached (also known as “network-attached”) memory by using RDMA on various high-performance networks, including Ethernet, Slingshot, and InfiniBand. Within the OpenFAM architecture, each node runs a memory manager responsible for managing its local memory contribution to the global FAM pool. Such a memory manager can be used to manage the portion of local memory contributed to the global cache.

[0026] The Data Consistency and Correctness feature 204 is responsible for ensuring data consistency and correctness to allow the global cache to operate according to the expected semantics of the underlying object storage or file system. The Data Consistency and Correctness feature 204 can use various mechanisms to enforce data consistency, including but not limited to: dirty bit tracking, cache invalidation, and metadata updates.

[0027] When the global cache is configured as an application cache, all data accesses can be handled by the cache manager 200 without invalidation, because the cache manager has complete control over all data. However, when the global cache is configured as a file system cache, the underlying data may change due to external access from outside the application or cache manager 200. In this case, invalidation mechanisms are provided to maintain data consistency.

[0028] In several respects, the data consistency and correctness function 204 can provide invalidation mechanisms tailored to application needs or the underlying storage system. More specifically, the data consistency and correctness function 204 can provide a variety of configurable invalidation mechanisms, such as mechanisms that automatically invalidate cached data in response to notifications of changes in the underlying storage, and mechanisms that invalidate cached data based on a time-to-live (TTL) policy, which ensures that stale data is refreshed periodically. These cache invalidation mechanisms ensure that the global cache operates according to the intended semantics of the object storage or file system, thereby providing reliable data access in heterogeneous and distributed environments.

[0029] The data location query function 206 can handle queries about the location of application data within the global cache. By providing accurate and up-to-date information about the location where specified data resides, the data location query function 206 allows computational tasks to be co-located with their corresponding data. This data-aware scheduling can significantly reduce data movement, lower access latency, and improve overall computational efficiency across the cluster.

[0030] The cache eviction function 208 can enforce a variety of flexible and configurable eviction policies, ranging from basic methods such as Least Recently Used (LRU) and First-In-First-Out (FIFO) to advanced policies such as Adaptive Replacement Cache (ARC). Furthermore, the system is built to adapt to future new policies. Applications can specify their desired caching policy using configuration parameters or API calls.

[0031] API function 210 can be responsible for providing multiple APIs for managing the global cache. These APIs enable efficient data access, caching, and management across multiple nodes and memory / storage tiers. Examples of these APIs may include, but are not limited to, object management APIs, data access APIs, metadata APIs, data location APIs, and directory management APIs.

[0032] The object management API may include at least the Open Object API (used to open objects in the global cache, retrieve their metadata, and initialize object handles for subsequent operations), the Close Object API (used to close objects, release associated resources, and update the cache state), and the Delete Object API (used to delete objects from the cache and persistent storage, thereby ensuring that all associated metadata and resources are cleared).

[0033] A data access API can include at least three APIs: a data read API, a data write API, and a data save API. The data read API is used to read data from a global cache or underlying storage into a user-provided buffer. Cached data can be dynamically retrieved from the appropriate tier (e.g., DRAM, SSD, or persistent storage) based on availability and access patterns. The data write API is used to write data to the global cache and update the appropriate memory tier and metadata. It ensures data consistency and handles block-level updates. The data save API is used to flush dirty data from the cache to persistent storage, thereby ensuring data persistence and consistency.

[0034] The metadata API can include a get cache configuration API (used to retrieve the configuration of the global cache, including eviction policy, invalidation mechanism, block size and other parameters) and a set cache configuration API (used to set the configuration of the global cache).

[0035] The Data Location API includes the Get Location API, which retrieves information about the location of data blocks within the global cache. This data location information allows data-consuming applications to optimize data access by coordinating computation and data placement. For example, a task scheduler can use the Get Location API to obtain the location of data in the global cache and then schedule computation tasks accordingly. Computation tasks that need to access certain data are scheduled to the node that caches that data.

[0036] The directory management API may include a create directory API for creating directories in the persistent storage layer and a delete directory API for deleting directories from the persistent storage layer.

[0037] The APIs described above are accessible to various frameworks that leverage the global cache, such as runtime systems or file system client components. APIs for configuring the global cache (e.g., selecting a caching strategy) are user-visible. These APIs provide a robust framework for managing a distributed, multi-tiered global cache, enabling efficient data access, consistency, and scalability in heterogeneous HPC and AI environments. Furthermore, the configurable APIs allow for customization of the global cache and its integration with various applications and storage systems, adapting it to a wide range of use cases and environments.

[0038] The cache manager metadata function 212 is responsible for managing cache manager metadata, which can be stored in a shared location accessible to all nodes running the cache manager process. Cache manager metadata can include information about the contents of the global cache, such as data size and location. Cache manager metadata is updated each time the cache manager process modifies the global cache (e.g., adding new data to the cache or changing the cache capacity).

[0039] Figure 3 An example application read operation, illustrating one aspect of this application, is processed by a global cache. The global cache may be distributed across a subset of nodes in a computer cluster, with each node in the subset contributing a portion of its local storage resources to the global cache. The global cache may include storage resources that may be organized into multiple tiers (e.g., faster or slower tiers) based on the type of these storage resources, including but not limited to DRAM, HBM, SCM, SSD, or any other existing or future memory or storage medium.

[0040] During operation, the cache manager process can receive read requests associated with data segments (Operation 302). The cache manager process can run on a compute node belonging to a cluster of computers running one or more applications, such as HPC or AI applications. The cache manager process can use APIs to intercept read requests from applications.

[0041] Upon receiving the read request, the cache manager process can determine whether the requested data resides in the global cache (Operation 304). The global cache can be shared by multiple applications running across a cluster of nodes. Cached data can reside on the same node where the cache manager process intercepted the read request, or on different nodes that contribute to the global cache. In some examples, the cache manager process can read cache manager metadata that tracks the contents and distribution of the global cache. If the cache manager metadata indicates that the requested data exists in the global cache, the cache manager can identify the specific node where the data resides.

[0042] In response to the location of the requested data in the storage resources of the contributing node, the cache manager process can retrieve the data by accessing the storage resources of that contributing node (operation 306). Depending on the type of storage resource, the cache manager process can use different data access mechanisms. In one example, the cached data is stored in DRAM on a remote node, and the cache manager process can use an RDMA read operation to retrieve the cached data. If the cached data is stored on a remote node in a tier that is not accessible via RDMA operations (e.g., on an SSD), the cache manager process handling the read request may need to communicate with the cache manager process running on that remote node to gain access to the data. In one example, the cached data is stored in distributed object storage, and the cache manager process can use the object storage API to access the cached data.

[0043] If the requested data is not in the global cache, the cache manager process can initiate a search across multiple memory and storage tiers in the cluster (Operation 308). The cache manager process can search DRAM, SSDs (e.g., flash drives) attached to compute nodes in the cluster, and persistent storage (e.g., persistent object storage) provided by storage servers. In some examples, the cache manager process may search faster storage tiers (e.g., DRAM and SSDs) before searching slower storage tiers (e.g., HD).

[0044] Subsequently, the cache manager process can retrieve the requested data based on the search results (Operation 310). Depending on the type of storage resource storing the data, the cache manager process can use different data access mechanisms to retrieve the data. If the requested data resides on an SSD, the cache manager process can access the data using the operating system or file system interface.

[0045] The cache manager process can then cache the data retrieved from the tiered storage into the global cache (Operation 312). Depending on the usage of the global cache, this data may be cached on the current node (i.e., the node that intercepted the read request) or on different nodes within the cluster. In some examples, the cache manager process may first determine if the global cache on the current node running the cache manager process is full. If the current node has unused global cache capacity, the cache manager process can cache the data in the global cache on the current node, allowing the cached data to be cached close to the computation requesting that data. If the global cache on the current node is full, the cache manager process may need to cache the data in the global cache on a remote node. When caching data, the cache manager process can use various mechanisms to enforce data consistency, including but not limited to dirty bit tracking, cache invalidation, and metadata updates. The cache manager process can also enforce a variety of flexible and configurable eviction policies depending on whether the cache is configured as an application cache, a file system cache, or an object storage cache.

[0046] After caching, the cache manager process can update the cache manager metadata indicating the location of the cached data (operation 314). Timely updates to the cache manager metadata ensure that further requests can be processed efficiently. The cache manager process can respond to the read request using the retrieved data (operation 316). If the read request originates from an application, the retrieved data can be returned to the application via an API.

[0047] For example, Figure 3 All or any part of the operation shown can be performed by Figure 2 The various functions shown are used to perform this. Although Figure 3 The example flow in the flowchart illustrates a specific order for performing certain operations, but the flow is not limited to these orders. The operations shown sequentially in the flowchart can be performed in different orders, simultaneously, partially simultaneously, or in combination.

[0048] Figure 4A computer system for facilitating the implementation of a global cache, according to one aspect of this disclosure, is illustrated. The computer system 400 includes processing resources 402, memory 404, and storage device 406. A portion of memory 404 and / or storage device 406 may contribute to a multi-tiered global cache. Furthermore, the computer system 400 may also be coupled to peripheral I / O user devices 410 (e.g., display 412, keyboard 414, and pointing device 416). Storage device 406 includes a non-transitory computer-readable storage medium and stores instructions of operating system 418, a global cache system 420, and data 440. According to some aspects, the computer system 400 may be implemented on nodes in a cluster of multiple nodes, such as… Figure 1 The computing node 102 or 104, or storage server 106 or 108, shown in the diagram. Computer system 400 may include... Figure 4 The fewer or more entities shown.

[0049] The global caching system 420 may include any number of software units, hardware units, and firmware units that work together to achieve the goal of implementing a distributed, multi-tiered global cache on the client side. According to some aspects, the global caching system 420 may include instructions that, when executed by the processing resource 402, cause the processing resource 402 to perform the methods and / or processes described in this disclosure.

[0050] In the examples described herein, processing resources may include, for example, one or more processors, either in a single computing device or distributed across multiple computing devices. As used herein, a “processor” may be at least one of a central processing unit (CPU), a semiconductor-based microprocessor, a graphics processing unit (GPU), a field-programmable gate array (FPGA) configured to retrieve and execute instructions, other electronic circuitry suitable for retrieving and executing instructions stored on a computer-readable storage medium, or combinations thereof. In the examples described herein, processing resources may acquire, decode, and execute instructions stored on a storage medium to perform the functions described with respect to the instructions stored on the computer-readable medium. In other examples, the functions described with respect to any instructions described herein may be implemented in the form of electronic circuitry, in the form of executable instructions encoded on a computer-readable medium, or a combination thereof. The computer-readable storage medium may be located either within the computing device executing these instructions or remotely accessible to the computing device (e.g., via a computer network) for execution. In the examples illustrated herein, a node may be implemented by one or more computer-readable storage media.

[0051] According to some aspects, the global cache system 420 may include instructions 422 for establishing a global cache by configuring a subset of nodes within the computer cluster so that each node contributes a portion of its local storage resources to the global cache. Local storage resources may include different types of computer-readable media organized into multiple tiers, including but not limited to DRAM, HBM, SCM, SSD, or any other existing or future computer-readable storage media. The computer cluster may run HPC or AI applications that require data exchange between different nodes within the cluster.

[0052] The global cache system 420 may include instructions 424 for configuring the cache manager process running on the corresponding node to receive requests for data segments, as described above. Figure 3 As described in operation 302, the cache manager process can exist on each node running the application and can intercept read requests from that application.

[0053] The global cache system 420 may include instructions 426 for configuring the cache manager process to determine whether the requested data exists in the global cache, as described above. Figure 3 As described in operation 304. Instruction 426 may include instructions for reading cache manager metadata that tracks the contents and distribution of the global cache.

[0054] The global cache system 420 may include instructions 428 for configuring the cache manager process to retrieve data by accessing the storage resources of a contributing node in response to the location of requested data in the storage resources of that contributing node, as described above. Figure 3 As described in operation 306 shown. If the data is cached in the DRAM of a remote node, instruction 428 may include RDMA instructions. If the data is cached in distributed object storage, instruction 428 may configure the cache manager process to retrieve the cached data via the object storage API.

[0055] The global cache system 420 may include instructions 430 for configuring the cache manager process to search for requested data across multiple memories and storage tiers in the cluster in response to the requested data not being in the global cache, as described above. Figure 3 As illustrated in operation 308, the cache manager process can be configured to search faster storage layers before searching slower ones. For example, instruction 430 can configure the cache manager process to search the DRAM / SSD layer before searching the HD layer.

[0056] The global cache system 420 may include instructions 432 for configuring the cache manager process to retrieve requested data based on search results, as described above. Figure 3 As shown in operation 310. Instruction 432 may include RDMA instructions or instructions that use an operating system or file system interface to retrieve data.

[0057] The global cache system 420 may include instructions 434 for caching data retrieved from hierarchical storage into the global cache, as described above. Figure 3 As described in operation 312. Instruction 434 may include instructions for enforcing data consistency using various mechanisms such as dirty bit tracking, cache invalidation, and metadata updates, as well as instructions for enforcing a variety of flexible and configurable eviction policies.

[0058] The global cache system 420 may include instruction 436 for updating cache manager metadata, as mentioned above. Figure 3 As illustrated in operation 314, the cache manager metadata is stored in a shared location accessible to all nodes in the cluster. The cache manager process can update the cache management metadata each time it modifies the global cache (e.g., caches new data, evicts stale data, or updates cache capacity).

[0059] The global cache system 420 may include instructions 438 for responding to requests using retrieved data, as described above. Figure 3 As shown in operation 316. Instruction 438 may include instructions for using the API to return data.

[0060] The global cache system 420 can include more than Figure 4 The instructions shown are further examples of those instructions. For instance, the global cache system 420 may include: instructions for receiving a query from a task scheduler for the location of a data segment associated with a computation task; instructions for reading cache manager metadata to determine the location of the data based on the cache manager metadata; and instructions for responding to the query with the determined location to allow the task scheduler to schedule the computation task based on the access latency associated with the determined location.

[0061] Figure 5A computer-readable medium (CRM) facilitating the implementation of a multi-layered distributed global cache according to one aspect of this disclosure is illustrated. The CRM 500 may be a non-transitory computer-readable medium or device storing instructions that, when executed by a computer or processing resource, cause the computer or processing resource to perform a method. As used herein, a “computer-readable storage medium” can be any electronic, magnetic, optical, or other physical storage device to include or store information such as executable instructions, data, etc. For example, any computer-readable storage medium described herein may be RAM, EEPROM, volatile memory, non-volatile memory, flash memory, storage drives (e.g., HDD, SSD), any type of storage disk (e.g., optical disc, DVD, etc.), or similar devices, or combinations thereof. Furthermore, any computer-readable storage medium described herein may be non-transitory.

[0062] CRM 500 can store instruction 510 for establishing a global cache by configuring a subset of nodes in a computer cluster to have each node contribute a portion of its local storage resources to the global cache; and instruction 520 for configuring the cache manager process running on the corresponding node to receive requests for data fragments, as described above. Figure 3 The operation shown in the diagram; instruction 530, is used to configure the cache manager process to determine whether the requested data exists in the global cache, as described above regarding... Figure 3 As described in operation 304; instruction 540 is used to configure the cache manager process to retrieve data by accessing the storage resources of a contributing node in response to locating that the requested data is located in the storage resources of that contributing node, as described above regarding... Figure 3 As described in operation 306; instruction 550 is used to configure the cache manager process to search for requested data across multiple memories and storage tiers in the cluster in response to the requested data not being in the global cache, as described above regarding Figure 3 Operation 308 is shown; instruction 560 is used to configure the cache manager process to retrieve the requested data based on search results, as described above regarding... Figure 3 Operation 310 is shown; instruction 570 is used to cache data retrieved from hierarchical storage into a global cache, as described above regarding... Figure 3 Operation 312 is shown; instruction 580 is used to update cache manager metadata, as described above regarding... Figure 3 Operation 314 is shown; and instruction 590 is used to respond to the request using the retrieved data, as described above regarding Figure 3 As described in operation 316.

[0063] CRM 500 can include more Figure 5 The instructions shown are further examples of those instructions. For instance, the CRM 500 may include: instructions for receiving a query from the task scheduler for the location of a data segment associated with a computation task; instructions for reading cache manager metadata to determine the location of the data based on that cache manager metadata; and instructions for responding to the query with the determined location to allow the task scheduler to schedule the computation task based on the access latency associated with the determined location.

[0064] In summary, this disclosure provides a solution to the client-side caching problem for heterogeneous applications running on a computer cluster with different storage and data access requirements. A multi-tiered distributed global client-side caching system can scale with the number of compute nodes in the cluster by aggregating client-side storage resources. This global caching system can include multiple nodes, each contributing a portion of its local storage resources to a global cache shared by applications running on different nodes across the cluster. Each node can run a cache manager process to access data from different tiers of the global cache, application caching strategies and algorithms for data placement, and ensure data consistency. A task scheduler can query the cache manager process for data locations and schedule compute tasks in proximity to that data to reduce data access latency.

[0065] In the examples described herein, a function described as being executed by “instructions” can be understood as a function that can be executed by these instructions when the processed resource is executed. In other examples, the function described with respect to instructions can be implemented by one or more logical units, which can be any combination of hardware and programming for implementing the function of that logical unit.

[0066] One aspect of this disclosure provides a caching system for a computer cluster comprising multiple nodes. The caching system may include a global cache distributed across a subset of nodes in the computer cluster. Relevant nodes in the subset contribute a portion of their local storage resources to the global cache and run a cache manager process to: receive a request for a data segment; determine if the requested data exists in the global cache; in response to locating the requested data in the portion of the local storage resources contributed to the global cache by the contributing node, retrieve the requested data by accessing the contributing node's local storage resources; and respond to the request using the retrieved data.

[0067] According to one variant of this approach, the storage resources contributed to the global cache by a subset of nodes are organized into multiple layers based on the type of these storage resources.

[0068] According to further variations, the type of storage resource may include one or more of the following: Dynamic Random Access Memory (DRAM); High Bandwidth Memory (HBM); Storage Class Memory (SCM); or Solid State Drive (SSD).

[0069] According to one variant of this, the cache manager process is further configured to: search for additional storage locations associated with the computer cluster in response to determining that the requested data is not in the global cache; retrieve the located data from the storage location in response to locating the requested data at the storage location; and cache the data retrieved from the storage location into the global cache.

[0070] According to a further variant, cached data may include: caching data in the global cache on the corresponding node in response to determining that the global cache on the corresponding node has unused capacity, and caching data in the global cache on a remote node in response to determining that the global cache on the corresponding node is full.

[0071] According to a further variant, the cache manager process is used to update cache manager metadata accessible to a subset of nodes, which indicates the location of cached data.

[0072] In a further variant, the cache manager process reads cache manager metadata when determining whether the requested data exists in the global cache.

[0073] In a further variant, the cache manager process is used to: receive a query from the task scheduler for the location of a second data segment associated with a computation task; determine the location of the second data segment based on cache manager metadata; and respond to the query with the determined location. The task scheduler is used to schedule computation tasks based on the access latency associated with the determined location.

[0074] In one variation, the global cache is configured as an application cache, a file system cache, or an object storage cache. The cache manager process is used to implement caching strategies based on the configuration of the global cache.

[0075] One aspect of this disclosure provides a method for implementing a caching system for a computer cluster comprising multiple nodes. The method includes establishing a global cache by configuring a subset of nodes within the computer cluster such that each node contributes a portion of its local storage resources to a global cache, and configuring cache manager processes running on the respective nodes within the subset to: receive a request for a data segment; determine whether the requested data exists in the global cache; in response to locating the requested data in the portion of local storage resources contributed to the global cache by the contributing node, retrieve the requested data by accessing the local storage resources of that contributing node; and respond to the requesting node with the retrieved data.

[0076] One aspect of this disclosure provides a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform a method for implementing a cache system for a computer cluster comprising multiple nodes. The method includes establishing a global cache by configuring a subset of nodes within the computer cluster, each node contributing a portion of its local storage resources to a global cache, and configuring cache manager processes running on the respective nodes within the subset of nodes to: receive a request for a data segment; determine whether the requested data exists in the global cache; in response to locating the requested data in the portion of local storage resources contributed to the global cache by the contributing node, retrieve the requested data by accessing the local storage resources of the contributing node; and respond to the request using the retrieved data.

[0077] The methods and processes described in the detailed description section may be embodied in code and / or data, which may be stored in a computer-readable storage medium as described above. When a computer system reads and executes the code and / or data stored on the computer-readable storage medium, the computer system executes the methods and processes embodied in data structures and code and stored in the computer-readable storage medium.

[0078] The methods and processes described above may be incorporated into hardware modules or devices. Hardware modules or devices may include, but are not limited to, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), dedicated or shared processors that execute specific software modules or code slices at specific times, and other programmable logic devices now known or developed thereafter. When a hardware module or device is activated, it executes the methods and processes contained therein.

[0079] The above description is presented to enable any person skilled in the art to make and use these aspects and examples, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed aspects will be apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects and applications without departing from the spirit and scope of this disclosure. Therefore, the aspects described herein are not limited to those shown, but are given the widest scope consistent with the principles and features disclosed herein.

[0080] Furthermore, descriptions of the foregoing aspects have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the aspects described herein to the disclosed forms. Therefore, many modifications and variations will be apparent to those skilled in the art. Additionally, the foregoing disclosure is not intended to limit the aspects described herein. The scope of the aspects described herein is defined by the appended claims.

Claims

1. A caching system for a computer cluster comprising multiple nodes, the caching system comprising: A global cache, distributed across a subset of nodes in the computer cluster, wherein each node in the subset contributes a portion of its local storage resources to the global cache and runs a cache manager process, in order to: Receive requests for data segments; Determine if the requested data exists in the global cache; In response to locating the requested data in the portion of the local storage resources contributed by the contributing node to the global cache, the requested data is retrieved from the local storage resources of the contributing node; as well as Use the retrieved data to respond to the request.

2. The caching system of claim 1, wherein the storage resources contributed to the global cache by a subset of the nodes are organized into multiple layers based on the type of the storage resources.

3. The cache system of claim 2, wherein the type of storage resource includes one or more of the following: Dynamic Random Access Memory (DRAM); High-bandwidth memory (HBM); Storage-type memory (SCM); or Solid-state drives (SSDs).

4. The caching system of claim 1, wherein the cache manager process is further configured to: In response to determining that the requested data is not in the global cache, search for additional storage locations associated with the computer cluster; In response to locating the requested data at a storage location, retrieving the located data from the storage location; and The data retrieved from the storage location will be cached in the global cache.

5. The cache system of claim 4, wherein caching the data comprises: In response to determining that the global cache on the corresponding node has unused capacity, the data is cached in the global cache on the corresponding node; as well as In response to determining that the global cache on the corresponding node is full, the data is cached in the global cache on the remote node.

6. The cache system as described in claim 5, wherein, The cache manager process is further used to update cache manager metadata accessible to a subset of the nodes, wherein the cache manager metadata indicates the location of the cached data.

7. The cache system as described in claim 6, wherein, When determining whether the requested data exists in the global cache, the cache manager process further reads the cache manager metadata.

8. The cache system as described in claim 6, in, The cache manager process is further used for: Receive a query from the task scheduler regarding the location of the second data segment associated with the computation task; The location of the second data is determined based on the cache manager metadata; The query is responded to using the determined location; as well as The task scheduler is used to schedule the computing tasks based on the access latency associated with the determined location.

9. The cache system as described in claim 1, wherein, The global cache is configured as an application cache, a file system cache, or an object storage cache, and the cache manager process is used to further implement caching strategies based on the configuration of the global cache.

10. A method for implementing a caching system for a computer cluster comprising multiple nodes, the method comprising: A global cache is established by configuring a subset of nodes within the computer cluster such that each node contributes a portion of its local storage resources to the global cache. as well as Configure the cache manager process to run on the corresponding nodes within the subset, in order to: Receive requests for data segments; Determine if the requested data exists in the global cache; In response to locating the requested data in the portion of the local storage resources contributed by the contributing node to the global cache, the requested data is retrieved by accessing the local storage resources of the contributing node; as well as Use the retrieved data to respond to the request.

11. The method of claim 10, wherein, The storage resources contributed to the global cache by a subset of the nodes are organized into multiple layers based on the type of the storage resources, wherein the type of the storage resources includes one or more of the following: Dynamic Random Access Memory (DRAM); High-bandwidth memory (HBM); Storage-type memory (SCM); or Solid-state drives (SSDs).

12. The method of claim 10, further comprising configuring the cache manager process to: In response to determining that the requested data is not in the global cache, search for additional storage locations associated with the computer cluster; In response to locating the requested data at a storage location, the located data is retrieved from the storage location; as well as The data retrieved from the storage location will be cached in the global cache.

13. The method of claim 12, wherein caching the data comprises: In response to determining that the global cache on the corresponding node has unused capacity, the data is cached in the global cache on the corresponding node; as well as In response to determining that the global cache on the corresponding node is full, the data is cached in the global cache on the remote node.

14. The method of claim 13, further comprising configuring the cache manager process to update cache manager metadata accessible to a subset of the nodes, wherein, The cache manager metadata indicates the location of cached data, and determining whether the requested data exists in the global cache includes reading the cache manager metadata.

15. The method of claim 14, further comprising: Configure the cache manager to: Receive a query from the task scheduler regarding the location of the second data segment associated with the computation task; The location of the second data is determined based on the cache manager metadata; as well as The query is responded to using the determined location; as well as The task scheduler schedules the computational tasks based on the access latency associated with the determined location.

16. The method of claim 10, wherein, The global cache is configured as an application cache, a file system cache, or an object storage cache, and the method further includes configuring the cache manager process to implement a caching strategy based on the configuration of the global cache.

17. A non-transitory computer-readable storage medium storing instructions, said instructions, when executed by a processor, causing the processor to perform a method for implementing a caching system for a computer cluster comprising multiple nodes, said method comprising: A global cache is established by configuring a subset of nodes within the computer cluster to contribute a portion of their local storage resources to the global cache. as well as Configure the cache manager process to run on the corresponding nodes within the subset, in order to: Receive requests for data segments; Determine if the requested data exists in the global cache; In response to locating the requested data in the portion of the local storage resources contributed by the contributing node to the global cache, the requested data is retrieved by accessing the local storage resources of the contributing node; as well as Use the retrieved data to respond to the request.

18. The non-transitory computer-readable storage medium of claim 17, wherein, The storage resources contributed to the global cache by a subset of the nodes are organized into multiple layers based on the type of the storage resources, wherein the type of the storage resources includes one or more of the following: Dynamic Random Access Memory (DRAM); High-bandwidth memory (HBM); Storage-type memory (SCM); or Solid-state drives (SSDs).

19. The non-transitory computer-readable storage medium of claim 17, wherein the method further comprises configuring the cache manager process to: In response to determining that the requested data is not in the global cache, search for additional storage locations associated with the computer cluster; In response to locating the requested data at a storage location, the located data is retrieved from the storage location; as well as The data retrieved from the storage location will be cached in the global cache; as well as Update the cache manager metadata accessible to a subset of the nodes, wherein the cache manager metadata indicates the location of the cached data.

20. The non-transitory computer-readable storage medium of claim 19, wherein the method further comprises: Configure the cache manager to: Receive a query from the task scheduler regarding the location of the second data segment associated with the computation task; The location of the second data is determined based on the cache manager metadata; as well as The query is responded to using the determined location; as well as The task scheduler schedules the computational tasks based on the access latency associated with the determined location.

Images