Public cloud-oriented plug-and-play dynamic sharing last-stage cache management system and method

Abstract

A public cloud-oriented plug-and-play dynamic sharing last-stage cache management system comprises a hardware compatible layer, a type identification and scheduler and an LLC allocation strategy, the hardware compatible layer encapsulates a hardware performance event and an LLC allocation technology, and resource isolation is ensured; the type identification and dispatcher identifies the type of the workload through a sliding window algorithm, and performs cross-node dispatching to optimize the performance; according to the LLC allocation strategy, a coarse-grained strategy is adopted for intermittent tasks, and resource allocation is dynamically adjusted; and a fine-grained strategy is implemented for a persistent task, testing and optimal configuration are carried out, and stable system performance is ensured. According to the method, intermittent tasks and persistent tasks can be effectively distinguished, appropriate resource allocation strategies are customized for the intermittent tasks and the persistent tasks, performance isolation between different workloads is improved, and the stability and response speed of cloud tenant application are ensured. Meanwhile, the allocation of LLC resources is dynamically adjusted according to the actual demand of the workload by using an automatic state machine algorithm and a fine-grained strategy, so that the waste and excessive allocation of the resources are avoided.

Description

Technical Field

[0001] This invention relates to the fields of cloud computing and processor caching architecture. Specifically, it relates to a plug-and-play dynamic shared last-level cache management system and method for public clouds. Background Technology

[0002] Multi-tenant cloud architectures, with their high cost-effectiveness and near-unlimited resources, such as flexible virtual machine resource rental, greatly improve the efficiency of application deployment and operation, significantly enhance overall resource utilization, and reduce energy consumption. Under the Infrastructure as a Service (IaaS) model, users can easily deploy and run various applications and services, covering a wide range of scenarios from lightweight static web page services to resource-intensive long-term machine learning model (such as LLM) training. However, with the increasing density of virtual machines in multi-tenant environments, performance interference issues arising from shared resources (especially CPU and last-level cache LLC) among workloads running in parallel on the same physical server are becoming increasingly prominent.

[0003] Currently, although Virtual Machine Monitors (VMMs) can effectively manage CPU time allocation and ensure fairness among virtual machines, challenges remain in managing LLC resources. Different types of workloads (such as memory-intensive and compute-intensive workloads) have significantly different demands for LLC resources, leading to frequent resource contention. Specifically, memory-intensive applications (such as Memcached) may excessively consume LLC resources, forcing compute-intensive applications (such as CPU-based LLM inference) to frequently encounter data being evicted to main memory, thereby increasing data loading latency and LLC miss rates, resulting in a two-way performance degradation effect.

[0004] Existing solutions for LLC management mostly focus on optimizing continuous tasks, neglecting the intermittent task characteristics prevalent in public cloud environments. This results in poor performance in mixed workload scenarios and limited generalization capabilities. Furthermore, these solutions often rely on complex hyperparameter configurations, requiring manual intervention and adjustments. They are ill-suited to the reality that workloads in multi-tenant cloud environments are treated as black boxes, meaning the management system cannot directly access detailed user-level information. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention proposes a plug-and-play dynamic shared last-level cache management system for public clouds. This system is independent of user-level information and has fewer hyperparameters, making it easy for cloud service providers to deploy on any public cloud server, significantly reducing configuration complexity and dependencies. It can intelligently identify and distinguish between intermittent and continuous tasks, implementing more refined resource allocation strategies. This achieves efficient performance isolation while improving the overall flexibility and adaptability of the system, providing a novel solution for resource optimization management in multi-tenant cloud environments.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] On the one hand, this invention provides a plug-and-play dynamic shared last-level cache management system for public clouds, characterized in that it includes:

[0008] The hardware compatibility layer is used to encapsulate the performance events and last-level cache allocation technology provided by the hardware. The performance event collection is achieved by parsing and processing data from the processor's performance monitoring unit (PMU), collecting and recording workload behavior indicators including the last-level cache (LLC) reference count and instruction cycle count (IPC). The LLC allocation is achieved by using the pqos library to dynamically configure the LLC size and CPU core mapping according to the type of workload and performance requirements, ensuring the isolation of LLC resources and avoiding overlap of service classes (COS) between different cores.

[0009] The type identification and scheduler is used to identify the type of workload and schedule CPU nodes. The workload type identification analyzes the fluctuation of IPC through a sliding window-based heuristic algorithm to divide the workload into intermittent tasks and continuous tasks. The CPU cross-node scheduling allocates different tasks to different NUMA slots according to the type of workload to optimize LLC utilization efficiency and performance isolation.

[0010] LLC allocation strategies, based on the characteristics of persistent and intermittent tasks, include coarse-grained and fine-grained strategies. The coarse-grained strategy is designed for intermittent tasks, including filtering non-memory-intensive tasks, task sorting based on LLC access counts, and using an automatic state machine algorithm to switch between balanced, exclusive, and isolated modes to dynamically allocate LLC resources. The fine-grained strategy is designed for persistent tasks, which tests different configurations and selects the configuration with the highest average total IPC as the globally optimal configuration. It also determines whether to re-evaluate the configuration based on real-time IPC data to maintain system performance stability.

[0011] Preferably, the hardware compatibility layer uses the msr-tools toolkit to perform read and write operations on the PMU to collect and process performance events.

[0012] Preferably, the sliding window heuristic algorithm in the type identification and scheduler automatically identifies the hot and cold characteristics of workloads by maintaining an IPC queue and calculating the average IPC value, thereby accurately classifying workload types.

[0013] Preferably, the granularity strategy includes:

[0014] - Filtering steps to identify and exclude non-memory-intensive tasks;

[0015] - Sorting steps: Sort intermittent tasks based on metrics such as LLC access count;

[0016] - Decision-making steps: Based on the ranking results, an automatic state machine algorithm is used to switch between balanced mode, exclusive mode and isolation mode to dynamically allocate LLC resources.

[0017] Preferably, the automatic state machine algorithm in the coarse-grained strategy dynamically adjusts the LLC allocation mode based on the number of LLC accesses of the workload and preset threshold conditions to ensure performance balance and isolation between different workloads.

[0018] Preferably, the fine-grained strategy includes:

[0019] - Configure testing and optimization: Test each configuration, pause and collect IPC data, and select the configuration with the highest total IPC average as the global optimal configuration;

[0020] - Configure a decision-making mechanism to determine whether a new globally optimal configuration needs to be re-evaluated and applied based on real-time changes in workload IPC data.

[0021] On the other hand, the present invention also provides a method for utilizing the above-mentioned plug-and-play dynamic shared last-level cache management system for public clouds, characterized in that the method includes:

[0022] S1. Collect and analyze workload performance events through the hardware compatibility layer;

[0023] S2. Use type identification and scheduler to classify workloads and schedule them to appropriate CPU nodes;

[0024] S3. Dynamically allocate LLC resources to workloads according to LLC allocation policies to optimize overall performance and resource utilization, and ensure that cloud tenants obtain better performance isolation.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] 1) Traditional methods often struggle to accurately distinguish between intermittent and continuous tasks. However, this invention introduces a sliding window algorithm to track and analyze IPC fluctuations in real time, enabling accurate identification of workload types, improving the system's adaptability to complex workloads, and providing strong support for subsequent resource allocation.

[0027] 2) This invention uses an automatic state machine algorithm to dynamically adjust the allocation of LLC resources. It can automatically switch between balanced mode, exclusive mode and isolation mode according to the number of LLC accesses of the workload and preset threshold conditions to optimize overall performance and resource utilization. This not only reduces the need for manual intervention, but also improves the intelligence level of the system.

[0028] 3) This invention designs fine-grained and coarse-grained strategies for different types of tasks. For example, the fine-grained strategy optimizes the performance of persistent tasks by testing different configurations and selecting the optimal solution; while the coarse-grained strategy optimizes resource allocation for intermittent tasks by filtering non-memory-intensive tasks and using sorting-based algorithms. This combination of strategies ensures both the overall performance of the system and fairness among different tasks.

[0029] 4) By encapsulating hardware toolkits such as msr-tools and pqos libraries, interface encapsulation for hardware performance events and LLC allocation technology is implemented. This design enables the system to interact with the underlying hardware more efficiently, improving system stability and reliability. Simultaneously, the modular design of the hardware compatibility layer makes the system easier to maintain and upgrade.

[0030] 5) This invention, through LLC allocation strategies and type recognition technology, effectively distinguishes between intermittent and continuous tasks, and customizes appropriate resource allocation strategies for them respectively. This significantly improves performance isolation between different workloads, reduces mutual interference, and ensures the stability and response speed of cloud tenant applications. Simultaneously, by utilizing automatic state machine algorithms and fine-grained strategies, the allocation of LLC resources is dynamically adjusted according to the actual needs of the workload, avoiding resource waste and over-allocation. This dynamic adjustment mechanism enables the system to utilize limited LLC resources more efficiently, improving overall resource utilization.

[0031] 6) It reduces the dependence on hyperparameters and adapts to different workloads and cloud environments through heuristic algorithms and automatic adjustment mechanisms, thereby reducing the complexity of system deployment and maintenance and making it easier for cloud service providers to integrate the system into existing public cloud servers.

[0032] 7) The hardware compatibility layer design enables SpiderSense to be compatible with multiple hardware platforms and processor architectures, providing a foundation for the system's widespread application. Meanwhile, the modular design of the type recognition and scheduler, as well as the LLC allocation strategy, makes the system easy to expand and customize to meet the needs of different cloud tenants and application scenarios. Attached Figure Description

[0033] Figure 1 This is an architecture diagram of a plug-and-play dynamic shared last-level cache management system for public clouds, based on the present invention.

[0034] Figure 2 Schematic diagram of workload identification and cross-CPU node scheduling Specific implementation methods

[0035] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings. These embodiments are implemented based on the technical solution and architecture of the present invention, and detailed implementation methods and specific operation processes are provided. However, the applicable platform is not limited to the following examples. A specific deployment example is a regular server consisting of two NUMA nodes. The present invention is not limited by the type or number of servers and can be deployed on any existing multi-node server.

[0036] This invention mainly comprises three sub-parts: a hardware compatibility layer, a type identification and scheduler, and an LLC allocation strategy, such as... Figure 1 This demonstrates the overall architecture of the system.

[0037] The hardware compatibility layer is primarily responsible for encapsulating the interfaces of hardware-provided performance events and last-level cache allocation techniques, as described below:

[0038] 1. Performance Event Collection: Workload behavior is recorded and analyzed by analyzing the CPU running the workload and collecting metrics such as LLC reference counts and IPC. The hardware compatibility layer utilizes msr-tools to encapsulate read and write operations to the PMU to collect various processor performance events.

[0039] 2. LLC Allocation: The hardware compatibility layer utilizes the Intel Platform Quality of Service (pqos) library to dynamically configure LLC size and CPU core mapping. The management system dynamically sets the Service Class (COS) based on the type and policy of each workload and binds the COS to the core running these workloads. To ensure LLC isolation, the system prevents COS overlap between different cores, ensuring a one-to-one correspondence between workloads and COS.

[0040] The type identification and scheduler is responsible for two tasks: identifying the type of workload and scheduling CPU nodes, as explained below:

[0041] 1. Workload Type Identification: A sliding window-based heuristic algorithm is used to label workload types. To track IPC changes over time, each workload maintains a queue and an average IPC, with the queue length corresponding to the sliding window length. The algorithm runs in two phases. In the initial phase, due to insufficient data for analysis, each workload is defaulted to a persistent task, and the average IPC is updated. Subsequently, SpiderSense adds IPC data to the queues every second. Once the duration exceeds the sliding window and there is sufficient data for analysis in each queue, the relationship between the current IPC and the average IPC is calculated after each queue update. If alternating hot and cold periods are observed, the workload is marked as an intermittent task, and the average IPC is updated.

[0042] 2. CPU cross-node scheduling: such as Figure 2 We implemented different LLC management methods on an Intel Xeon processor equipped with two NUMA sockets, each with 12-way LLCs. We then ran six actual workloads. The system, having identified the task types, could assign them to different NUMA sockets. In this example, the scheduler marked Redis, Memcached, and ClickHouse as persistent tasks, while marking Kernel Build, Llama, and Stable Diffusion as intermittent tasks. The TypeScheduler then scheduled them to NUMA socket 0 and socket 1, respectively.

[0043] LLC allocation strategies are categorized into coarse-grained and fine-grained strategies based on the characteristics of continuous and intermittent tasks. The coarse-grained strategy is custom-designed for intermittent tasks and is explained below:

[0044] 1. Filtering: The first stage determines whether intermittent tasks are memory-intensive. To ensure that memory-intensive tasks receive sufficient LLC, non-memory-intensive applications are first filtered out. SpiderSense compares the memory load / store ratio of each workload to a predefined threshold. If the ratio is below the threshold, the task is considered not memory-intensive and is allocated minimal LLC resources.

[0045] 2. Sorting: The second stage sorts the LLC access counts for each intermittent task. There are three main reasons for avoiding threshold-based allocation methods: First, to develop systems with strong generalization capabilities, the demand for hyperparameters must be minimized. Second, in cloud computing environments, thresholds are difficult to configure statically. Third, the LLC access counts differ significantly between different tasks. Sorting-based algorithms can more effectively reflect the memory requirements of different workloads. SpiderSense extracts the following four metrics: maximum LLC access count (M-LA), minimum LLC access count (L-LA), second largest LLC access count (SM-LA), and second smallest LLC access count (SL-LA).

[0046] 3. Decision-Making: The third phase employs a heuristic algorithm using an automatic state machine. SpiderSense defines the following three modes and makes decisions based on the ranking results: Balanced Mode: All workloads are evenly distributed with the same number of LLCs. This mode is primarily used when the LLC requirements of all workloads are similar. SpiderSense uses the balanced mode when the difference between M-LA and S-LA is no more than one order of magnitude. Exclusive Mode: One workload is allocated the vast majority of LLCs, while the remaining workloads share the fewest LLCs. When one workload is in a cold phase while other workloads are in a hot phase, it indicates that the workload in the hot phase has a higher demand for LLCs. In this case, SpiderSense uses the exclusive mode, which is enabled when M-LA exceeds SM-LA by one order of magnitude. Isolation Mode: One workload is allocated the fewest LLCs, while other workloads are allocated the same number of LLCs. When one workload is in a hot phase while other workloads are in a cold phase, it indicates that the workload in the hot phase has the lowest demand for LLCs. In this case, SpiderSense uses the isolation mode, which is enabled when SL-LA exceeds L-LA by one order of magnitude. SpiderSense will choose isolation mode when both exclusive and isolation modes are met, because ensuring the performance floor of all workloads is more important than prioritizing the performance of individual workloads.

[0047] Fine-grained strategies are custom-designed for persistent tasks, as explained below:

[0048] 1. For each configuration in the configuration set, SpiderSense pauses for a period of time after applying the configuration, and then collects the average IPC for each workload to determine the overall average IPC for the entire system. SpiderSense then selects the configuration with the highest overall average IPC as the globally optimal configuration.

[0049] 2. Beyond the strategy, each time IPC data for the workload is collected, SpiderSense decides whether to use an algorithm to reconfigure or maintain the existing configuration. For relatively stable, continuous tasks, there is little need to frequently change the globally optimal configuration. Therefore, spending a few seconds to determine the optimal configuration is worthwhile.

[0050] In this embodiment, the system is configured on a machine equipped with The server uses a Gold 5317 processor. Each socket of this processor has 12 cores with a base frequency of 3.0 GHz and 18 MB of last-level cache (LLC), distributed across 12 lanes with 1.5 MB of LLC per lane. The host runs Qemu 6.2.0 to emulate a KVM Hypervisor-based guest operating system. The host and guest operating systems are Linux 6.5 and Linux 5.15, respectively. To maintain consistency with a real-world multi-tenant cloud environment, hyper-threading was enabled in the experiment.

[0051] This invention proposes the SpiderSense dynamic shared last-level cache management system, specifically designed for public cloud environments to enhance performance isolation for cloud tenants. This system is independent of user-level information, reduces hyperparameter dependencies, and facilitates deployment by cloud service providers on any public cloud server. The system architecture encompasses a hardware compatibility layer, a type identification and scheduler, and an LLC allocation strategy. The type identification and scheduler adds workload type identification and cross-CPU node scheduling, such as... Figure 2 LLC allocation strategies are based on task characteristics, employing both coarse-grained (for intermittent tasks) and fine-grained (for continuous tasks) strategies to ensure efficient resource utilization and performance optimization.

[0052] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A plug-and-play dynamic shared last-level cache management system for public clouds, characterized in that, include: The hardware compatibility layer is used to encapsulate the performance events and last-level cache allocation technology provided by the hardware. The performance event collection is achieved by parsing and processing data from the processor's performance monitoring unit (PMU), collecting and recording workload behavior indicators including the last-level cache (LLC) reference count and instruction cycle count (IPC). The LLC allocation is achieved by using the pqos library to dynamically configure the LLC size and CPU core mapping according to the type of workload and performance requirements, ensuring the isolation of LLC resources and avoiding overlap of service classes (COS) between different cores. The type identification and scheduler is used to identify the type of workload and schedule CPU nodes. The workload type identification analyzes the fluctuation of IPC through a sliding window-based heuristic algorithm to divide the workload into intermittent tasks and continuous tasks. The CPU cross-node scheduling allocates different tasks to different NUMA slots according to the type of workload to optimize LLC utilization efficiency and performance isolation. LLC allocation strategies, based on the characteristics of persistent and intermittent tasks, include coarse-grained and fine-grained strategies. The coarse-grained strategy is designed for intermittent tasks, including filtering non-memory-intensive tasks, task sorting based on LLC access counts, and using an automatic state machine algorithm to switch between balanced, exclusive, and isolated modes to dynamically allocate LLC resources. The fine-grained strategy is designed for persistent tasks, which tests different configurations and selects the configuration with the highest average total IPC as the globally optimal configuration. It also determines whether to re-evaluate the configuration based on real-time IPC data to maintain system performance stability.

2. The plug-and-play dynamic shared last-level cache management system for public clouds according to claim 1, characterized in that, The hardware compatibility layer uses the msr-tools toolkit to perform read and write operations on the PMU, enabling the collection and processing of performance events.

3. The plug-and-play dynamic shared last-level cache management system for public clouds according to claim 1, characterized in that, The sliding window heuristic algorithm in the type identification and scheduler automatically identifies the alternating hot and cold characteristics of workloads by maintaining an IPC queue and calculating the average IPC value, thereby accurately classifying workload types.

4. The plug-and-play dynamic shared last-level cache management system for public clouds according to claim 1, characterized in that, The granularity strategy includes: - Filtering steps to identify and exclude non-memory-intensive tasks; - Sorting steps: Sort intermittent tasks based on metrics such as LLC access count; - Decision-making steps: Based on the ranking results, an automatic state machine algorithm is used to switch between balanced mode, exclusive mode and isolation mode to dynamically allocate LLC resources.

5. The plug-and-play dynamic shared last-level cache management system for public clouds according to claim 4, characterized in that, The automatic state machine algorithm in the coarse-grained strategy dynamically adjusts the LLC allocation mode based on the number of LLC accesses of the workload and preset threshold conditions to ensure performance balance and isolation between different workloads.

6. The plug-and-play dynamic shared last-level cache management system for public clouds according to claim 1, characterized in that, The fine-grained strategy includes: - Configure testing and optimization: Test each configuration, pause and collect IPC data, and select the configuration with the highest total IPC average as the global optimal configuration; - Configure a decision-making mechanism to determine whether a new globally optimal configuration needs to be re-evaluated and applied based on real-time changes in workload IPC data.

7. A method for a plug-and-play dynamic shared last-level cache management system for public clouds according to any one of claims 1 to 5, characterized in that, The method includes: S1. Collect and analyze workload performance events through the hardware compatibility layer; S2. Use type identification and scheduler to classify workloads and schedule them to appropriate CPU nodes; S3. Dynamically allocate LLC resources to workloads according to LLC allocation policies to optimize overall performance and resource utilization, and ensure that cloud tenants obtain better performance isolation.

Images