K8s and Slurm hybrid deployment management and control method

Abstract

The invention aims to provide a K8s and Slurm hybrid deployment management and control method, and particularly relates to a Kubernetes and Slurm hybrid deployment management and control method based on dynamic resource awareness. According to the method, an intelligent node agent (Node Agent) is introduced, trend prediction is carried out according to the real-time water level and historical load characteristics, and therefore refined resource arbitration is carried out on K8s (online) and Slurm (offline) on the Linux Cgroup level. According to the method, the problems of resource conflict and invisibility of heterogeneous scheduling systems (K8s and Slurm) during hybrid deployment at the same node are solved, the problem of performance interference between online services and offline operations is solved, the resource utilization rate is maximized, and meanwhile, the quality of service (QoS) of the online services is ensured. The system has higher stability and safety and anti-interference performance at a micro-architecture level.

Description

Technical Field

[0001] This invention belongs to the fields of cloud computing, big data, edge computing, and artificial intelligence. Specifically, it relates to a hybrid deployment and management method of Kubernetes and Slurm, and more particularly to a hybrid deployment and management method of Kubernetes and Slurm based on dynamic resource awareness. Background Technology

[0002] In current data center architectures, there are typically two types of cluster management systems and services: one type is services represented by microservices, including various online and offline processing services, which usually run in Kubernetes clusters; the other type is offline batch processing jobs, represented by scientific computing and AI training, which usually run in HPC clusters such as Slurm.

[0003] Current cluster management and job scheduling technologies mainly fall into the following categories:

[0004] (1) Non-intrusive hosting solution (such as patent CN115237547B): This patent describes a non-intrusive HPC cluster hosting method that converts Kubernetes workloads into executable scripts or instructions for HPC (such as Slurm) by customizing resources and configurators, thereby achieving unified task distribution.

[0005] (2) Heterogeneous job scheduling and plug-in driven solution (e.g., patent CN116661979A): This patent describes a method for connecting multiple computing clusters (K8s, Slurm, etc.) through a virtual node controller and plug-in bus. It abstracts different computing resources into virtual nodes and allocates tasks to the corresponding backend clusters through a scheduler.

[0006] (3) Template-based job scheduling scheme (such as patent CN117093352A): This patent describes a method to adapt to different computing frameworks by calling job templates, which solves the problem of parameter configuration complexity when users submit jobs in different clusters.

[0007] (4) Deep learning-specific scheduling scheme (such as patent CN108920259A): This patent describes a container scheduling method for deep learning jobs, which optimizes container creation and task submission by monitoring container status and job identifier.

[0008] Analysis of the existing technologies revealed the following shortcomings in scenarios involving deep integration of K8s and Slurm:

[0009] (1) Single-machine resource "split-brain" and conflict issues:

[0010] Existing technologies (such as patents CN115237547B and CN116661979A) focus on implementing "task subcontracting" or "unified management" at the control plane, allowing Kubernetes to act as an entry point for managing Slurm. However, at the physical node level, if a Kubernetes Pod and a Slurm Job are running simultaneously on the same machine, the two schedulers are unaware of each other's real-time CPU / memory usage. The drawback of this approach is that it can lead to severe resource overselling or conflicts; Slurm jobs may preempt resources from critical Kubernetes services (online services), causing OOM (Out of Memory) errors or a significant increase in latency. This invention proposes a NodeAgent mechanism that does not rely on control plane synchronization but instead establishes dynamic arbitration directly at the physical node kernel level through Cgroup V2.

[0011] (2) The resource view is static and the utilization rate is uneven:

[0012] Existing technologies (such as patent CN108920259A) typically employ a static resource request model. To ensure online business security, a large amount of idle resources are usually reserved. The drawback is that this "static partitioning" results in resources not being fully utilized by Slurm operations during periods of low business activity, leading to high overall costs. This invention introduces a heuristic dynamic watermark algorithm to calculate real-time security quotas. This method dynamically lends "fragmented, instantaneous" idle resources to Slurm based on the historical fluctuation characteristics of Kubernetes services, significantly improving resource utilization.

[0013] (3) Lack of interference detection at the microarchitecture level:

[0014] Existing technologies (such as patent CN117093352A) only optimize at the job submission and logical scheduling level, failing to address contention in underlying hardware (such as L3 cache and memory bandwidth). A drawback is that when offline computing jobs (Slurm) perform high-intensity computations, even if CPU utilization is not full, memory bus pressure can slow down online service (K8s) responses. This invention introduces an emergency circuit breaker mechanism based on PSI (Pressure Stall Information). When a microsecond-level blockage is detected in online services, the Node Agent immediately freezes the Slurm process, providing higher service reliability.

[0015] The purpose of this invention is to solve the resource conflict and invisibility problems when heterogeneous scheduling systems (K8s and Slurm) are deployed together on the same node, as well as the performance interference problem between online services and offline jobs, so as to maximize resource utilization and ensure the quality of service (QoS) of online services.

[0016] English-Chinese bilingual edition:

[0017] Container orchestration system (Kubernetes, K8s), high-performance computing scheduler (Slurm, Simple Linux Utility for Resource Management), control groups (Cgroups), online services (Latency Sensitive Services, LS), offline services (Best Effort / Batch Jobs, BE), quality of service (QoS), and node agent (NA). Summary of the Invention

[0018] The purpose of this invention is to propose a method for managing the hybrid deployment of Kubernetes and Slurm, particularly a method based on dynamic resource awareness. This invention introduces an intelligent node agent that performs trend prediction not only based on real-time resource levels but also on historical load characteristics, thereby enabling fine-grained resource arbitration for Kubernetes (online) and Slurm (offline) at the Linux Cgroup level. This solves the resource conflicts and invisibility issues when heterogeneous scheduling systems (Kubernetes and Slurm) are deployed on the same node, as well as the performance interference between online services and offline jobs, maximizing resource utilization while ensuring the Quality of Service (QoS) of online services.

[0019] This invention proposes a hybrid deployment and management method for Kubernetes and Slurm. The management method is implemented by a management system, which adopts a three-layer, dual-stack architecture, including a control plane layer, a node proxy layer, and a kernel isolation layer.

[0020] The control plane is located at the top layer, with the K8s control plane on one side and the Slurm controller on the other side. Logically, the two do not communicate with each other. The K8s control plane sends scheduling instructions to the middle layer, and the Slurm controller sends jobs to the middle layer.

[0021] The node agent layer is located in the middle layer, deploying a node agent on each physical node. The node agent includes a data collector, a heuristic engine, and an actuator. The output of the data collector connects to the input of the heuristic engine, and the output of the heuristic engine connects to the input of the actuator via quota calculation. The data collector interfaces with the Kubelet Summary API and / sys / fs / cgroup to collect second-level resource metrics. The heuristic engine maintains historical time-series data windows and calculates dynamic safety levels. The actuator directly manipulates the Cgroup v2 interface (cpu.max, memory.high) to implement suppression.

[0022] The kernel isolation layer is located at the bottom layer and includes high priority and low priority; high priority corresponds to online services, and low priority corresponds to offline services; the Kubelet in the middle layer receives scheduling instructions from the top-level Kubernetes control plane, and transmits the information that meets the requirements to the high priority layer at the bottom layer by setting restrictions; the resource executor in the middle layer connects to the low priority layer at the bottom layer through the Cgroup v2 interface; the kernel isolation layer performs physical allocation of hardware resources according to the Cgroup parameters;

[0023] The control method is an infinite loop control circuit, and the specific steps are as follows:

[0024] (1): State snapshot: Node Agent every Retrieve the most recent data at intervals (e.g., 5 seconds) K8s resource usage records for each cycle;

[0025] (2): Heuristic prediction: Calculate the predicted resource demand values ​​for future moments in K8s based on historical data. and volatility ;

[0026] (3): Water level calculation: Based on the predicted value and volatility, calculate the safe quota reserved for Slurm. ;

[0027] (4): Strategy issued, current resource usage is :

[0028] like Relaxing Slurm Cgroup restrictions;

[0029] like Shrink Slurm Cgroup restrictions;

[0030] (5) Interference detection: Check the PSI (Pressure Stall Information) index; if a sudden increase in the PSI of online services is found, skip the calculation step and trigger step (6) to perform "emergency circuit breaker";

[0031] (6) Emergency Circuit Breaker: Freeze or terminate the Slurm process;

[0032] (7) Maintenance strategy: Maintain the current scheduling strategy unchanged.

[0033] In this invention, to avoid resource waste or untimely response caused by a simple fixed threshold in step (4), this invention designs a dynamic buffering algorithm based on volatility awareness, as follows:

[0034] The Node Agent does not directly use the current instantaneous utilization rate, but instead calculates the predicted peak.

[0035]

[0036] in: The maximum amount of resources allocated to Slurm in this cycle; Total physical resources of the node (e.g., number of CPU cores); Trend item; Fluctuation buffer term Static minimum guarantee;

[0037] Trend Item The load baseline of K8s is calculated using an exponentially weighted moving average.

[0038]

[0039] Compared to simple averaging, EMA places greater weight on the most recent data points, enabling it to respond more quickly to load spikes in Kubernetes.

[0040] Fluctuation buffer : It is the standard deviation of load within the historical window, representing the "jitter" of Kubernetes services. It is the confidence coefficient (e.g., taking...) or The heuristic logic is as follows: if the online business is very stable (StdDev is small), a small buffer is reserved, allowing Slurm to use more resources; if the online business fluctuates wildly (StdDev is large), the buffer is automatically expanded to reserve more resources in case of unforeseen circumstances.

[0041] Static guarantee Minimum reserved water level (e.g., 5%) to prevent instantaneous overselling during cold starts.

[0042] This shows that although K8s was only used at 50% in scenario B, due to its large fluctuations, the system intelligently reserved a large amount of buffer, sacrificing some offline resources in exchange for the absolute security of online services.

[0043] In this invention, the PSI-driven emergency circuit breaker mechanism described in step (6) uses a heuristic algorithm to handle millisecond / second-level resource allocation. For microsecond-level microarchitecture contention (L3 Cache / Memory Bandwidth), the PSI circuit breaker mechanism is as follows:

[0044] The monitoring metrics are read from / proc / pressure / cpu and / proc / pressure / memory, specifically the metric (partial blocking time). When the PSI (avg10) of a K8s Cgroup exceeds the threshold (e.g., 20ms), a severe resource contention is identified, triggering subsequent actions. These actions include ignoring the heuristic calculation results and directly reducing the cpu.max of the SlurmCgroup to 10% or triggering a cgroup.freeze suspension operation until the PSI returns to normal.

[0045] The beneficial effects of this invention are as follows:

[0046] 1. Enhanced stability and security: Compared to the simple fixed-ratio partitioning in existing technologies, this application introduces volatility as a calculation factor. This gives the system a stronger predictive ability when facing sudden bursts of Kubernetes traffic, significantly reducing the risk of online business memory overflow or response timeouts due to untimely resource switching.

[0047] 2. Micro-architecture level anti-interference: Combining the PSI index, it solves the hidden interference problem of low CPU utilization but high system load.

[0048] 3. In universities and research institutes, many still retain the habit of using Slurm. However, with the rapid development of cloud-native applications and the increasing use of Kubernetes (K8s), the ability to use Slurm and K8s simultaneously without conflicts is a common need. Therefore, Slurm has broad application value in university and research institute settings. Attached Figure Description

[0049] Figure 1 This is a system architecture diagram.

[0050] Figure 2 This is a flowchart of the closed-loop process for resource regulation. Detailed Implementation

[0051] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0052] Example 1: Hybrid Deployment System Architecture Based on Node Agents

[0053] like Figure 1 As shown, this embodiment provides a hybrid deployment and management system for container clusters and high-performance computing based on dynamic resource awareness. The system is deployed in a data center environment containing multiple physical nodes, each of which has hardware resources including a CPU (Central Processing Unit), memory, and L3 cache.

[0054] In terms of software architecture, this physical node uses the Linux operating system and implements hierarchical resource management based on the Cgroup V2 mechanism. The system is divided into three layers from bottom to top: kernel layer, agent layer, and application layer.

[0055] 1. Application Layer and Control Plane:

[0056] Container orchestration systems (Kubernetes / K8s): Used to deploy online services, such as web services or microservice gateways. K8s manages Pods on nodes through the Kubelet component.

[0057] The high-performance computing system (Slurm) is used to deploy offline jobs, such as AI model training or scientific computing tasks. Slurm manages jobs on nodes through the Slurmd component.

[0058] Configuration characteristics: In the Slurm configuration file slurm.conf, the node's resources (such as the number of CPU cores) are configured to 100% of the total physical hardware (e.g., 128 cores), meaning the Slurm view considers it to own all resources. This overlaps with the resource view in Kubernetes, but in this invention, this "logical resource conflict" is expected and resolved through the following proxy layer.

[0059] 2. Proxy layer:

[0060] In this embodiment, a separate daemon process—NodeAgent—is deployed on each physical node. This agent runs independently of Kubelet and Slurmd and has CAP_SYS_RESOURCE permissions.

[0061] The Node Agent consists of three core modules:

[0062] Data Collector: Used to obtain real-time resource usage of online services through the Kubelet Summary API and to obtain the system's PSI (Pressure Stall Information) metric through the / proc / pressure interface.

[0063] Heuristic Engine: Built-in heuristic algorithms are used to calculate the security resource level of offline services based on historical data.

[0064] Actuator: Used to directly manipulate the Cgroup file system interface and dynamically adjust resource quotas for offline services.

[0065] 3. Kernel isolation layer:

[0066] The system creates two top-level slices in the Cgroup root directory:

[0067] `kubepods.slice`: Used to assign all Kubernetes Pod processes. Setting a high CPU weight (e.g., `cpu.weight = 1000`) ensures priority scheduling under extreme contention.

[0068] `slurm.slice`: Used to manage all job processes belonging to the Slurmd daemon and its spawned processes. Set a low CPU weight (e.g., `cpu.weight = 10`) and enable hard limit management using `cpu.max`.

[0069] Example 2: A Dynamic Resource Management Method Based on Heuristic Algorithms

[0070] like Figure 2 As shown, this embodiment, based on the system architecture of Embodiment 1, elaborates on the specific process of resource management by the Node Agent. This process is a continuously running closed-loop control loop, with a default cycle of It lasts for 5 seconds.

[0071] Step S1: Status Acquisition and History Window Update

[0072] At the start of each cycle, the Node Agent performs the following operations:

[0073] (1) Call the Kubelet API to get the total CPU usage of all Kubernetes Pods on the current node. (Unit: Core)

[0074] (2) Read / proc / pressure / cpu, obtain the avg10 value of the line some, and record it as .

[0075] (3) Maintain a historical data sliding window of length N (e.g., N=60). .

[0076] Step S2: Heuristic Calculation of Safety Water Level

[0077] The decision engine does not directly use the current instantaneous usage. Instead of calculating the "predicted peak," it combines trend and volatility to calculate the "predicted peak." The specific calculation logic is as follows:

[0078] Calculate the trend term (EMA): Calculate the baseline load of online services using an exponentially weighted moving average algorithm.

[0079] Among them, the smoothing coefficient The value is set to 0.2, making the algorithm more sensitive to recent load changes.

[0080] (1) Calculate the fluctuation term (StdDev): Calculate the sliding window within the standard deviation This value reflects the severity of fluctuations in online business operations.

[0081] (2) Calculate the offline quota: Calculate the maximum resource limit allocated to Slurm according to the following formula. :

[0082]

[0083] in: Total number of physical cores (e.g., 100 cores). Confidence coefficient, taken in this embodiment This indicates that a fluctuation range of 2 standard deviations is reserved. Static minimum reserve, taking 5% of the total number of cores (i.e., 5 cores).

[0084] Specific calculation example: Assume the physical machine has 100 cores.

[0085]

[0086] Where: C is 100, K is 2, and the static minimum margin is 5.

[0087] In scenario A (late at night): K8s load is 20% (load baseline calculated by exponential weighted moving average), i.e., EMA is 20, and it is very stable (StdDev=1).

[0088] calculate: Cores. That is, Slurm can obtain 73 cores.

[0089] Scenario B (Daytime Promotion): K8s load is 50% (load baseline calculated by exponential weighted moving average), i.e., EMA is 50, but fluctuates greatly (StdDev=10).

[0090] calculate: Cores. That is, Slurm only gets 25 cores.

[0091] Step S3: Policy Issuance (Execution Control)

[0092] The actuator will calculate the result The value is converted into a microsecond time slice (Quota) and written to the cpu.max file of the Cgroup where Slurm resides.

[0093] For example, if If the CFS period is 100ms, then 6100000 100000 will be written.

[0094] In this way, the Linux kernel strictly limits the CPU utilization of the Slurm process group to no more than 6100%, thereby physically freeing up resources for Kubernetes.

[0095] Example 3: Emergency Circuit Breaker Mechanism Based on PSI

[0096] In addition to the conventional adjustments in Example 2, this example also includes a high-priority interference detection and circuit breaker mechanism to handle covert interferences such as CPU cache (L3 cache) contention or memory bandwidth saturation.

[0097] 1. Interference Detection: The Node Agent monitors the PSI metric of the Cgroup where Kubernetes resides in real time. A threshold is set. If detected This means that online services have been blocked for more than 20ms in the past 10 seconds due to resource waiting, which is considered a serious microarchitectural disruption.

[0098] 2. Circuit Breaker Operation: Once interference is detected, the Node Agent immediately skips the calculation in step S2 and directly executes the circuit breaker operation:

[0099] Level 1 Circuit Breaker (Degradation): Forces the cpu.max of the Slurm Cgroup to a minimum value (e.g., 1 core) for 1 minute.

[0100] Level 2 Circuit Breaker (Freeze): If the PSI does not recover after the Level 1 circuit breaker, write 1 to the cgroup.freeze file of slurm.slice to suspend all offline job processes until the online workload decreases.

[0101] Effect Analysis:

[0102] Through the above implementation methods, this invention utilizes a Node Agent to build a "resource arbitration court" for Kubernetes and Slurm at the kernel layer. Compared with existing static partitioning schemes, this scheme utilizes... The algorithm can increase the cluster's resource utilization rate from the traditional 20%-30% to 50%-70% while ensuring the QoS of online services, thus maximizing the utilization of computing resources.

Claims

1. A method for hybrid deployment and management of Kubernetes and Slurm, characterized in that... The control method is implemented by a control system, which adopts a three-layer dual-stack architecture, including a control plane layer, a node proxy layer, and a kernel isolation layer, wherein: The control plane is located at the top layer, with the K8s control plane on one side and the Slurm controller on the other side. Logically, the two do not communicate with each other. The K8s control plane sends scheduling instructions to the middle layer, and the Slurm controller sends jobs to the middle layer. The node agent layer is located in the middle layer, deploying node agents on each physical node. Each node agent includes a data collector, a heuristic decision engine, and a resource executor. The output of the data collector connects to the input of the heuristic decision engine, and the output of the heuristic decision engine connects to the input of the resource executor via quota calculation. The data collector interfaces with the KubeletSummary API and / sys / fs / cgroup to collect second-level resource metrics. The heuristic decision engine maintains historical time-series data windows and calculates dynamic safety levels. The resource executor directly operates the Cgroup v2 interface (cpu.max, memory.high) to implement suppression. The kernel isolation layer is located at the bottom layer and includes high priority and low priority; high priority corresponds to online services, and low priority corresponds to offline services; the Kubelet in the middle layer receives scheduling instructions from the top-level Kubernetes control plane, and transmits the information that meets the requirements to the high priority layer at the bottom layer by setting restrictions; the resource executor in the middle layer connects to the low priority layer at the bottom layer through the Cgroup v2 interface; the kernel isolation layer performs physical allocation of hardware resources according to the Cgroup parameters; The control method is an infinite loop control circuit, and the specific steps are as follows: (1): State snapshot: Node Agent every Periodically pull the most recent K8s resource usage records for each cycle; (2): Heuristic prediction: Calculate the predicted resource demand values ​​for future moments in K8s based on historical data. and volatility ; (3): Water level calculation: Based on the predicted value and volatility, calculate the safe quota reserved for Slurm. ; (4): Strategy issued, current resource usage is : like Relaxing Slurm Cgroup restrictions; like Shrink Slurm Cgroup limits; (5) Interference detection: Check the PSI index; if a sudden increase in the PSI of online services is found, skip the calculation step and trigger step (6) to perform "emergency circuit breaker"; (6) Emergency Circuit Breaker: Freeze or terminate the Slurm process; (7) Maintenance strategy: Maintain the current scheduling strategy unchanged.

2. The method for hybrid deployment and management of K8s and Slurm according to claim 1, characterized in that... Step (4) employs a volatility-aware dynamic buffer algorithm, as detailed below: The Node Agent does not directly use the current instantaneous utilization rate, but instead calculates the predicted peak. ； in: The maximum amount of resources allocated to Slurm in this cycle; Total physical resources of the node; Trend item; Fluctuation buffer term Static minimum guarantee; Trend Item The load baseline of K8s is calculated using an exponentially weighted moving average. ； Compared to simple averaging, EMA places greater weight on the most recent data points, enabling it to respond more quickly to load spikes in Kubernetes. Fluctuation buffer : It is the standard deviation of load within the historical window, representing the "jitter" level of Kubernetes services. It is a confidence coefficient. The heuristic logic is as follows: if the online business is very stable, that is, StdDev is small, leave a small buffer and let Slurm use more resources; if the online business fluctuates, that is, StdDev is large, automatically expand the buffer and reserve more resources to prevent unforeseen events. Static guarantee Minimum reserved water level to prevent instantaneous overselling during cold starts.

3. The method for hybrid deployment and management of K8s and Slurm according to claim 1, characterized in that... The PSI-driven emergency circuit breaker mechanism described in step (6) uses a heuristic algorithm to handle resource allocation at the millisecond / second level. For microsecond-level microarchitecture contention, the PSI circuit breaker mechanism is as follows: The monitoring metrics, specifically the blocking time of some metrics in / proc / pressure / cpu and / proc / pressure / memory, are read. When the PSI of a K8s Cgroup exceeds the threshold, a severe resource contention is determined, triggering subsequent actions. These actions include ignoring the above heuristic calculation results and directly reducing the cpu.max of the Slurm Cgroup to 10% or triggering a cgroup.freeze suspension operation until the PSI returns to normal.

Images