Load testing method and system for large model task scheduling system based on k8s cluster
By deploying a large-scale task scheduling system on a real Kubernetes cluster and building a simulation environment, the problem that existing load testing tools cannot simulate the resource consumption and scheduling latency of large-scale tasks is solved. This enables efficient and automated performance evaluation and bottleneck location, and generates detailed load testing reports.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG LAB
- Filing Date
- 2026-05-25
- Publication Date
- 2026-06-26
AI Technical Summary
Existing load testing tools cannot accurately simulate the high resource consumption characteristics and pod complexity of large-scale model tasks, making it difficult to discover resource allocation bottlenecks and scheduling latency in the scheduling system. They also suffer from limited test scenarios, unrealistic test data, and low automation, failing to meet the performance evaluation requirements of large-scale model scheduling systems.
Deploy a large-scale task scheduling system on a real Kubernetes cluster, build a simulated Kubernetes cluster and simulate real node resources, monitor and collect task scheduling metrics in real time through task load management tools and data collection tools, generate stress test reports, support custom task parameters and multiple scheduling strategies, and achieve one-click stress testing and data analysis.
It achieves highly realistic large-scale computing resource simulation, can accurately evaluate the performance of the scheduling system, supports testing of various scheduling strategies and complex scenarios, improves the degree of automation, accurately locates performance bottlenecks, and generates valuable stress test reports.
Smart Images
Figure CN122285459A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of cloud computing and container orchestration technology, and in particular to a stress testing method and system for a large-scale task scheduling system based on a K8s cluster. Background Technology
[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) are increasingly widely used in fields such as natural language processing and computer vision. Kubernetes, as a container orchestration platform, has become the mainstream infrastructure for deploying and managing large model tasks due to its powerful resource scheduling and elastic scaling capabilities. As the number of parameters in large models increases, the demand for computing resources such as GPUs for large model tasks is also increasing, and the requirements for the scale of Kubernetes clusters in large model task scheduling systems are also increasing. This brings many challenges to the scheduling system: resource contention, scheduling latency, and uneven node load may occur during large-scale concurrent task scheduling; different GPU / CPU resource allocation strategies have a significant impact on task execution efficiency; and bottlenecks such as network bandwidth and storage I / O may become constraints on system performance under high-pressure scenarios.
[0003] In existing technologies, performance testing tools for Kubernetes clusters, such as Kubemark, are mainly geared towards load testing of regular container applications, lacking dedicated load testing solutions for large-scale scheduling systems and large-scale tasks. Large-scale scheduling systems are characterized by large cluster size and numerous managed resources, while large-scale tasks are characterized by high resource requirements, complex execution processes, long execution times, and computational intensity. Traditional load testing methods struggle to simulate the load characteristics of real-world large-scale clusters and large-scale tasks, making it impossible to accurately assess the performance of scheduling systems in large-scale scenarios. Furthermore, existing load testing tools have shortcomings in areas such as test metric collection, performance bottleneck identification, and test scenario configurability, failing to meet the refined performance evaluation requirements of large-scale scheduling systems.
[0004] Therefore, the existing technology has the following problems:
[0005] 1. Lack of targeted load testing tools: Existing load testing tools cannot accurately simulate the high resource consumption characteristics and pod complexity of large-scale model tasks;
[0006] 2. Performance bottlenecks are difficult to identify: It is impossible to identify resource allocation bottlenecks, scheduling delays, task backlogs and other issues in the scheduling system before deployment;
[0007] 3. Limited testing scenarios: It is difficult to cover the performance of large-scale model tasks under different loads and scheduling strategies;
[0008] 4. Inaccurate test data: The characteristics of stress testing tasks differ greatly from those of actual production tasks, limiting the reference value of test results;
[0009] 5. Low level of automation: The testing process relies on manual configuration, making it impossible to achieve continuous performance monitoring and regression testing.
[0010] There is an urgent need for a stress testing method and system specifically designed for large-scale task scheduling systems based on Kubernetes clusters. This system should be able to simulate real large-scale Kubernetes clusters and large-scale task loads, comprehensively evaluate various performance indicators of the scheduling system, and provide a reliable basis for system optimization and capacity planning. Summary of the Invention
[0011] To address the shortcomings of existing technologies, the present invention aims to provide a stress testing method and system for a large-scale task scheduling system based on a K8s cluster.
[0012] To achieve the above objectives, the technical solution adopted by this invention is as follows: a stress testing method for a large-scale task scheduling system based on a K8s (Kubernetes) cluster, comprising the following steps:
[0013] On a real Kubernetes cluster, the first large-scale model task scheduling system was deployed to collect load testing task datasets.
[0014] On a simulated Kubernetes cluster, a second large model task scheduling system is deployed, and simulated nodes are built, the computing power resource specifications of which are the same as those of real nodes;
[0015] The task Pods in the load testing task dataset are scheduled to the simulation nodes to perform load testing. The task scheduling metrics, cluster resource metrics, and performance metrics of the second-largest model task scheduling system are monitored and collected in real time. Abnormal events (such as task failure reasons, resource shortage events, and scheduler error logs) are recorded.
[0016] Generate a stress test report, including performance metric statistics, bottleneck analysis, and optimization suggestions.
[0017] Furthermore, the load testing task dataset is collected in the following ways:
[0018] Based on the characteristics of large model tasks, a task template library is constructed, which supports custom task parameters.
[0019] Based on the task template library, a task data acquisition tool is used to collect the load testing task dataset in the first large model task scheduling system; or, a public dataset conforming to the task template library is used as the load testing task dataset.
[0020] Furthermore, the large model task characteristics include CPU resource requirements, GPU resource requirements, memory usage, storage I / O mode, task creation time, and task duration; the custom task parameters include scheduling strategy configurations such as resource request volume, priority, node affinity, and tolerance.
[0021] Furthermore, the simulation node is deployed using the simulation node tool Kubemark. The execution logic of the Pod on the simulation node is rewritten as follows: the Kubemark source code is modified to add the parameter fakeRemoteRuntime to the ProbeManager dependency of Hollow Kubelet. This allows ProbeManager to call the StopContainer function through fakeRemoteRuntime, enabling the Pod running on the simulation node to skip the container initialization execution phase and directly enter the container execution phase.
[0022] Furthermore, the simulated Kubernetes cluster is also equipped with a task load management tool and a task data acquisition tool (API Server Controller). The task load management tool is used to create or terminate load testing tasks, and the task data acquisition tool is used to monitor and collect task scheduling metrics (average scheduling latency, scheduling success rate, task timeout rate), simulated Kubernetes cluster resource metrics (CPU / GPU utilization, memory utilization, network bandwidth usage), and the performance metrics of the second-largest model task scheduling system (API Server response time, etcd read / write latency, node health status) in real time.
[0023] Furthermore, the task data acquisition tool continuously monitors the cluster status based on the Kubernetes API Server Controller mechanism.
[0024] Furthermore, the task load management tool is implemented by the following sub-modules:
[0025] The task data loading module is used to load the load testing task dataset, which includes the resource specifications and runtime of the task. The resource specifications include the number of CPUs, memory, hard disks, GPUs, and nodes, and the runtime includes the task creation time and end time.
[0026] The task load creation module is used to configure load testing tasks in the second large model task scheduling system according to the load testing task dataset and execute the load testing tasks.
[0027] The task load termination module is used to monitor the task scheduling information of the load testing task and terminate the task load according to the task execution time.
[0028] This invention also provides a stress testing system for a large-scale task scheduling system based on a Kubernetes cluster, comprising:
[0029] The real environment configuration module is used to deploy the first major model task scheduling system on a real Kubernetes cluster;
[0030] The simulation environment configuration module is used to deploy the second large model task scheduling system on the simulated Kubernetes cluster and build simulation nodes, the computing power resource specifications of which are the same as those of the real nodes;
[0031] The data acquisition module is used to collect load testing task datasets from the first large model task scheduling system, monitor and collect task scheduling indicators, cluster resource indicators, and performance indicators of the second large model task scheduling system in real time during the execution of load testing tasks, and record abnormal events.
[0032] The load management module is used to schedule the task Pods in the load testing task dataset to the simulation node for load testing, and to create and terminate load testing tasks.
[0033] Furthermore, the load management module is implemented by the following sub-modules:
[0034] The task data loading module is used to load the load testing task dataset, which includes the resource specifications and runtime of the task. The resource specifications include the number of CPUs, memory, hard disks, GPUs, and nodes, and the runtime includes the task creation time and end time.
[0035] The task load creation module is used to configure load testing tasks in the second large model task scheduling system according to the load testing task dataset and execute the load testing tasks.
[0036] The task load termination module is used to monitor the task scheduling information of the load testing task and terminate the task load according to the task execution time.
[0037] Furthermore, the system includes a data analysis module, which generates a stress test report based on task scheduling metrics, cluster resource metrics, performance metrics of the second-largest model task scheduling system, and abnormal events. The report includes performance metric statistics, bottleneck analysis, and optimization suggestions.
[0038] The beneficial effects of this invention are:
[0039] 1. High simulation accuracy: The stress testing environment can accurately simulate large-scale computing resource clusters, and the stress testing tasks can accurately simulate the resource consumption characteristics and scheduling behavior of real large-scale model tasks, making the test results more valuable for reference.
[0040] 2. Comprehensive coverage: Supports testing of various scheduling strategies (priority, affinity, preemption, etc.) and complex scenarios (mixed task types, burst traffic);
[0041] 3. High degree of automation: Enables one-click execution of load testing, data acquisition, and report generation, reducing manual operation costs;
[0042] 4. Precise bottleneck identification: Through multi-dimensional indicator monitoring and in-depth analysis, performance bottlenecks at the levels of scheduling system, resource allocation, network storage, etc., can be quickly identified;
[0043] 5. High scalability: The system adopts a modular design, supports custom task templates, monitoring metrics, and report templates, and adapts to the needs of different business scenarios;
[0044] 6. Production environment friendly: The load testing tasks adopt standard Kubernetes resource definitions, which are compatible with the production environment and support canary testing and pre-release environment verification. Attached Figure Description
[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0046] Figure 1 This is an architecture diagram of a stress testing system for a large-scale task scheduling system based on a K8s cluster, provided in an embodiment of the present invention.
[0047] Figure 2 The flowchart shows a stress testing method for a large-scale task scheduling system based on a K8s cluster, as provided in another embodiment of the present invention. Detailed Implementation
[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0049] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.
[0050] like Figure 1As shown, this embodiment of the invention provides a stress testing system for a large-scale task scheduling system based on a Kubernetes cluster. It involves two environments deployed on a Kubernetes cluster: a real environment to provide real large-scale task load information; and a simulation environment to perform stress testing and collect key performance indicators. The system specifically includes:
[0051] The real environment configuration module is used to deploy the first major model task scheduling system on a real Kubernetes cluster.
[0052] The simulation environment configuration module is used to deploy the second large model task scheduling system on the simulated Kubernetes cluster and build simulation nodes, the computing resources of which are the same as those of the real nodes.
[0053] The data acquisition module is used to collect load testing task datasets from the first large model task scheduling system, monitor and collect task scheduling indicators, cluster resource indicators, and the performance indicators of the second large model task scheduling system in real time during the execution of load testing tasks, and record abnormal events.
[0054] The load management module is used to schedule the task Pods in the load testing task dataset to the simulation node for load testing, and to create and terminate load testing tasks.
[0055] Preferably, the task load management tool is implemented by the following sub-modules:
[0056] The task data loading module is used to load the load testing task dataset, which includes the resource specifications and runtime of the task. The resource specifications include the number of CPUs, memory, hard disks, GPUs, and nodes, and the runtime includes the task creation time and end time.
[0057] The task load creation module is used to configure load testing tasks in the second large model task scheduling system based on the load testing task dataset and execute the load testing tasks.
[0058] The task load termination module is used to monitor the task scheduling information of the load testing task and terminate the task load according to the task execution time.
[0059] like Figure 2 As shown in the figure, this embodiment of the invention also provides a stress testing method for a large-scale task scheduling system based on a K8s cluster, including the following steps:
[0060] On a real Kubernetes cluster, the first large-scale model task scheduling system was deployed to collect load testing task datasets.
[0061] On a simulated Kubernetes cluster, a second large model task scheduling system is deployed, and simulated nodes are built, with the same computing resources as the real nodes.
[0062] The task Pods in the load testing task dataset are scheduled to the simulation nodes to perform load testing. The task scheduling metrics, cluster resource metrics, and performance metrics of the second-largest model task scheduling system are monitored and collected in real time. Abnormal events (such as task failure reasons, resource shortage events, and scheduler error logs) are recorded.
[0063] Generate a stress test report, including performance metric statistics, bottleneck analysis, and optimization suggestions.
[0064] As a preferred embodiment, this invention provides a task data acquisition tool. This tool is implemented based on the Kubernetes API Server Controller mechanism. After the API Server processes Kubernetes resource change requests, it obtains resource change information in an Informer manner. After monitoring Job resources, it can obtain template data for large-scale model tasks from the first large-scale model task scheduling system deployed in a real Kubernetes cluster, thereby obtaining the task dataset required for load testing. The template data mainly includes task resource scale (CPU, memory, and GPU count), task creation time, start time, and end time.
[0065] Furthermore, the task data acquisition tool is also used to monitor Job resources in the second-largest model task scheduling system deployed in the simulated Kubernetes cluster, and to collect statistics on task creation time, start time and end time. Through calculation, data such as cluster computing power resource fragmentation rate and average task scheduling time are obtained for evaluation of test results after stress testing.
[0066] In a preferred embodiment, the simulation node is deployed using simulation tools within a simulated Kubernetes cluster. The size of the simulation node is configured according to requirements; large-scale model tasks in practice require substantial computing resources.
[0067] For example, taking the simulation of 10,000 cards and one machine with eight cards as an example, the total number of simulation nodes = 10,000 / 8 = 1250, so 1250 simulation nodes need to be created.
[0068] When a simulation node joins the cluster, it needs to have node tags, taints, and image pull configurations added according to the requirements of the scheduling system. These settings can be configured in the simulation node configuration file. The simulation tool Kubemark consists of two parts: HollowKubelet and Hollow proxy. Hollow Kubelet is responsible for reporting node and Pod information to the API Server.
[0069] In a large-scale task Pod, the container lifecycle is described as: Pod creation → Init container creation → Init container execution → Init container termination → Normal container creation → Normal container execution → Normal container termination → Pod termination. The task Pod can only enter the running state after the Init container terminates. The original Kubemark does not support terminating the Init container; therefore, this invention improves the KubemarkHollow Kubelet by rewriting the Init container phase logic. Instead of keeping the Init container in a running state, it switches to a terminated state, allowing the Pod and task to enter the running state, generating task scheduling information that can be collected by task data acquisition tools. The Kubemark rewrite process is as follows: The Kubemark source code is modified to add the parameter `fakeRemoteRuntime` to the Hollow Kubelet's dependency `ProbeManager`, enabling `ProbeManager` to call the `StopContainer` function via `fakeRemoteRuntime`.
[0070] As a preferred embodiment, the present invention provides a task load management tool that relies on the RESTful API interface for creating and terminating tasks provided by the second-largest model task scheduling system. After obtaining the task dataset, for each task in the dataset, when creating a task, the task information is configured according to the task resource scale and other parameters in the load testing task dataset. Using the task creation time in the dataset as a relative time, a thread pool is used to periodically call the task creation RESTful API interface. When a task Pod is scheduled onto a simulation node and all Pods are in the running state, the task enters the running state, and the task data acquisition tool detects the change in task status. When terminating a task, upon receiving the actual start time of the task from the task data acquisition tool, the actual end time of the task is calculated by combining it with the running time of the task in the dataset (end time - start time). The task is then periodically called to terminate the task using the task termination RESTful API interface.
[0071] After all load testing tasks are completed, the load testing process can be analyzed and the results evaluated from both task and system perspectives, based on the data obtained from the task data acquisition tools and monitoring system.
[0072] Preferably, the present invention also deploys Prometheus and Grafana on the second large model task scheduling system to continuously collect key performance indicators of the system and resources. These indicators mainly include API Server response time, API Server 5XX error count, etcd read / write latency, and node resource utilization.
[0073] This invention simulates large-scale computing resources at a low cost and uses data from real scheduling systems as test data. It completes the simulation of each task through the process of "creating tasks on a timed basis → changing task status → calculating end time → ending tasks on a timed basis", thereby obtaining the performance of the simulation environment in stress testing. This allows scheduling problems to be exposed or reproduced in advance, providing a reference for in-depth problem investigation, evaluating the scheduling performance in the real environment, and guiding the optimization of scheduling performance.
[0074] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.
Claims
1. A stress testing method for a large-scale task scheduling system based on a Kubernetes cluster, characterized in that, Includes the following steps: On a real Kubernetes cluster, the first large-scale model task scheduling system was deployed to collect load testing task datasets. On a simulated Kubernetes cluster, a second large model task scheduling system is deployed, and simulated nodes are built, the computing power resource specifications of which are the same as those of real nodes; The task Pods in the load testing task dataset are scheduled to the simulation nodes to perform load testing. Task scheduling metrics, cluster resource metrics, and performance metrics of the second-largest model task scheduling system are monitored and collected in real time, and abnormal events are recorded.
2. The method according to claim 1, characterized in that, The load testing task dataset was collected in the following ways: Based on the characteristics of large model tasks, a task template library is constructed, which supports custom task parameters. Based on the task template library, a task data acquisition tool is used to collect the load testing task dataset in the first large model task scheduling system; or, a public dataset conforming to the task template library is used as the load testing task dataset.
3. The method according to claim 2, characterized in that, The large model task features include CPU resource requirements, GPU resource requirements, memory usage, storage I / O mode, task creation time, and task duration; the custom task parameters include resource request volume, priority, node affinity, and tolerance.
4. The method according to claim 1, characterized in that, The simulation nodes are deployed using the simulation node tool Kubemark. The execution logic of the Pods on the simulation nodes is rewritten so that the Pods running on the simulation nodes skip the container initialization execution phase and directly enter the container execution phase.
5. The method according to claim 1, characterized in that, The simulated Kubernetes cluster is also equipped with a task load management tool and a task data acquisition tool. The task load management tool is used to create or terminate load testing tasks, and the task data acquisition tool is used to monitor and collect task scheduling metrics, simulated Kubernetes cluster resource metrics, and performance metrics of the second-largest model task scheduling system in real time.
6. The method according to claim 5, characterized in that, The task data acquisition tool continuously monitors the cluster status based on the Kubernetes API Server Controller mechanism.
7. The method according to claim 5, characterized in that, The task load management tool is implemented by the following sub-modules: The task data loading module is used to load the load testing task dataset, which includes the resource specifications and runtime of the task. The resource specifications include the number of CPUs, memory, hard disks, GPUs, and nodes, and the runtime includes the task creation time and end time. The task load creation module is used to configure load testing tasks in the second large model task scheduling system according to the load testing task dataset and execute the load testing tasks. The task load termination module is used to monitor the task scheduling information of the load testing task and terminate the task load according to the task execution time.
8. A stress testing system for a large-scale task scheduling system based on a Kubernetes cluster, characterized in that, include: The real environment configuration module is used to deploy the first major model task scheduling system on a real Kubernetes cluster; The simulation environment configuration module is used to deploy the second large model task scheduling system on the simulated Kubernetes cluster and build simulation nodes, the computing power resource specifications of which are the same as those of the real nodes; The data acquisition module is used to collect load testing task datasets from the first large model task scheduling system, monitor and collect task scheduling indicators, cluster resource indicators, and performance indicators of the second large model task scheduling system in real time during the execution of load testing tasks, and record abnormal events. The load management module is used to schedule the task Pods in the load testing task dataset to the simulation node for load testing, and to create and terminate load testing tasks.
9. The system according to claim 8, characterized in that, The load management module is implemented by the following sub-modules: The task data loading module is used to load the load testing task dataset, which includes the resource specifications and runtime of the task. The resource specifications include the number of CPUs, memory, hard disks, GPUs, and nodes, and the runtime includes the task creation time and end time. The task load creation module is used to configure load testing tasks in the second large model task scheduling system according to the load testing task dataset and execute the load testing tasks. The task load termination module is used to monitor the task scheduling information of the load testing task and terminate the task load according to the task execution time.
10. The system according to claim 8, characterized in that, The system includes a data analysis module, which generates a stress test report based on task scheduling metrics, cluster resource metrics, performance metrics of the second-largest model task scheduling system, and abnormal events.