Load balancing scheduling method, system and equipment

Abstract

A load balancing scheduling method, system and device relate to the technical field of resource scheduling, and the method comprises the following steps: setting a node label for each working node in a cluster to label a hardware resource type, and obtaining a load index of each working node; receiving a task request, obtaining a task stage and a task type, and forwarding the request to the Service corresponding to the task stage and the task type; task resource requirements, Pod topological constraints and resource utilization rates of scheduling nodes are obtained to construct a QUBO model, a task-to-node allocation matrix is obtained through a quantum annealing algorithm, and target nodes are determined; and starting the pod at the target node through the pre-deployed KubeRay, and when a preset elastic expansion and contraction condition is triggered, performing elastic expansion and contraction. According to the method, the hardware resource type is labeled through the node label, and the task demand and the node resource are dynamically matched in combination with the QUBO model and the quantum annealing algorithm, so that the resource utilization rate is effectively improved.

Description

Technical Field

[0001] This application relates to the field of resource scheduling technology, specifically to a load balancing scheduling method, system, and device. Background Technology

[0002] With the rapid iteration of machine learning and language models, the demand for dynamic scheduling of underlying resource architectures for related distributed training and inference tasks is increasing. Integrating the distributed training framework Ray onto native Kubernetes (K8s) (KubeRay) has become a viable solution. KubeRay is an operator with custom components such as RayCluster, RayJob, and RayService, enabling end-to-end lifecycle management of LLM and ML models.

[0003] In KubeRay, the computational resource requirements for different training tasks vary significantly. For example, the memory bandwidth requirements of the prefill and decode stages in training and inference can differ by 3-5 times, and their serial computation leads to low utilization of the cluster's computational resources. Related technologies employ pipelined computing to deploy the two stages across different hardware resource pools, improving training and inference computational capabilities and resource utilization.

[0004] However, current Kubernetes load balancing strategies (Schedulers) are based on round-robin allocation, which cannot perceive the heterogeneous resource requirements of GPUs (Graphics Processing Units), such as memory and computing cores, based on training tasks. The current Kubernetes Ingress controller uses the Round Robin scheduling algorithm by default, which cannot perceive dynamic load indicators of cluster resources such as GPU memory utilization, computing unit load, and resource utilization. This can lead to high-priority tasks being scheduled to high-load nodes, causing task blocking and latency, while low-priority training tasks experience resource overflow. Furthermore, current Ingress controllers expose services using ClusterIP or LoadBalancer, which mostly only support domain name / path matching and lack the ability to coordinate scheduling based on model training task type and resource status. Summary of the Invention

[0005] This application provides a load balancing scheduling method, system, and device, which can solve the technical problem of low utilization of cluster computing resources in the prior art.

[0006] Firstly, this application provides a load balancing scheduling method, the method comprising: Set node labels for each worker node in the cluster to indicate the type of hardware resources, and obtain the load metrics for each worker node; the load metrics include the resource utilization rate of hardware resources. Upon receiving a task request, the task stage and task type are obtained based on the request path and request header fields. The request is then forwarded to the Service corresponding to the task stage and task type. The Service is associated with the scheduling node through the node tag. The task resource requirements, Pod topology constraints, and resource utilization of the aforementioned scheduling nodes are obtained to construct a QUBO model. The task-to-node allocation matrix is ​​obtained through the quantum annealing algorithm to determine the target node. The pod is started on the target node by pre-deploying KubeRay, and elastic scaling is performed when the preset elastic scaling conditions are triggered.

[0007] In conjunction with the first aspect, in one implementation, the task stage and task type are obtained based on the request path and request header fields of the aforementioned request, specifically including: The task stage of the request is determined by matching the request path using regular expressions. Parse the above request header fields to extract the task identifier, which is used to distinguish the task type.

[0008] In conjunction with the first aspect, in one implementation, after obtaining the load metrics for each working node, the method further includes: Based on preset resource alarm rules, the weight coefficients of each node are adjusted, thereby modifying the canary-weight of the Ingress controller; The above resource alarm rules include: when any load metric exceeds its corresponding first metric threshold and the duration exceeds the first time threshold, a resource alarm is triggered.

[0009] In conjunction with the first aspect, in one implementation, a QUBO model is constructed, and a task-to-node allocation matrix is ​​obtained through a quantum annealing algorithm, including: Encode the above task resource requirements, Pod topology constraints, and resource utilization; Construct the Hamiltonian, which is the product of the resource gap weight and the task resource requirement code, plus the communication delay weight, the Pod topology constraint code, and the resource utilization code. With the goal of minimizing the Hamiltonian, the QUBO model is mapped onto the qubits of the quantum annealing machine, and the above allocation matrix is ​​output.

[0010] In conjunction with the first aspect, in one implementation, the objective of minimizing the Hamiltonian specifically includes: When the above request is a preprocessing request, the communication delay weight is gradually increased, but not exceeding the first preset multiple of the resource gap weight, and the communication delay weight is taken when the Hamiltonian is the smallest. When the above request is an inference request, the resource gap weight is gradually increased, but not exceeding the second preset multiple of the resource gap weight, and the resource gap weight is taken when the Hamiltonian is the smallest.

[0011] In conjunction with the first aspect, in one implementation, the above-mentioned elastic expansion / shrinkage capacity operation when a preset elastic expansion / shrinkage condition is triggered specifically includes: When elastic scaling conditions are triggered, the configuration file of Ray Service is automatically modified to adjust the resource quota or quantity of Pods, and a blue-green deployment strategy is adopted to achieve seamless migration of tasks.

[0012] In conjunction with the first aspect, in one implementation, the above-mentioned elastic expansion and contraction conditions include expansion conditions and contraction conditions. The above expansion conditions include: the duration for which any load metric exceeds its corresponding second metric threshold exceeds the second time threshold; The above-mentioned scaling-down conditions include: the duration for which any load metric is below its corresponding third metric threshold exceeds the third time threshold; The threshold of the second indicator corresponding to any indicator is greater than the threshold of its corresponding third indicator.

[0013] In conjunction with the first aspect, in one implementation, when performing flexible expansion and contraction, the method further includes: Control the number of Pods started on each node between the preset maximum and minimum number.

[0014] Secondly, this application provides a load balancing scheduling system, the system comprising: The cluster management module is used to set node labels for each worker node in the cluster to indicate the type of hardware resources, and to obtain the load indicators of each worker node; the load indicators include the resource utilization rate of hardware resources. The routing gateway receives task requests, obtains the task stage and task type based on the request path and request header fields, and forwards the request to the corresponding Service for the task stage and task type. The Service is associated with the scheduling node through the node label. The scheduling module is used to obtain task resource requirements, Pod topology constraints, and the resource utilization of the aforementioned scheduling nodes to build a QUBO model and obtain the task-to-node allocation matrix through the quantum annealing algorithm to determine the target node. The aforementioned cluster management module is also used to start pods on the target nodes via pre-deployed KubeRay, and to perform elastic scaling when preset elastic scaling conditions are triggered.

[0015] Thirdly, this application provides a load balancing scheduling device, which includes a processor, a memory, and a load balancing scheduling program stored in the memory and executable by the processor, wherein when the load balancing scheduling program is executed by the processor, it implements the steps of the load balancing scheduling method described above.

[0016] The beneficial effects of the technical solution provided in this application include: By assigning node tags to each worker node in the cluster to indicate hardware resource types and obtaining load metrics for each worker node (including hardware resource utilization), the system receives task requests, obtains the task stage and type based on the request path and header fields, and forwards the request to the corresponding Service. The Service is then associated with the scheduling node through the node tags. The system obtains task resource requirements, Pod topology constraints, and the resource utilization of the scheduling node to construct a QUBO model. A quantum annealing algorithm is used to obtain the task-to-node allocation matrix to determine the target node. The pod is then started on the target node using a pre-deployed KubeRay instance, and elastic scaling is performed when preset elastic scaling conditions are triggered. By tagging hardware resource types with node tags, combined with the QUBO model and quantum annealing algorithm, task requirements and node resources are dynamically matched, effectively improving resource utilization and avoiding the resource idleness or overload and low resource utilization problems caused by static allocation in related technologies. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating an embodiment of the load balancing scheduling method of this application; Figure 2 This is a timing diagram of the load balancing scheduling method according to an embodiment of this application; Figure 3 This is a flowchart illustrating another embodiment of the load balancing scheduling method of this application; Figure 4 This is a schematic diagram of the architecture of an embodiment of the load balancing scheduling system of this application; Figure 5 This is a schematic diagram of the architecture of another embodiment of the load balancing scheduling system of this application; Figure 6 This is a schematic diagram of the hardware structure of the load balancing scheduling device involved in the embodiments of this application. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0019] Firstly, embodiments of this application provide a load balancing scheduling method.

[0020] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the load balancing scheduling method of this application. The load balancing scheduling method includes: S1. Set node labels for each worker node in the cluster to indicate the type of hardware resources, and obtain the load metrics for each worker node; the load metrics include the resource utilization rate of hardware resources; S2. Receive a task request, obtain the task stage and task type based on the request path and request header fields, and forward the request to the Service corresponding to the task stage and task type; the Service is associated with the scheduling node through the node tag. S3. Obtain the task resource requirements, Pod topology constraints, and resource utilization of the aforementioned scheduling nodes to construct the QUBO model, and obtain the task-to-node allocation matrix through the quantum annealing algorithm to determine the target node; S4. Start the pod on the target node using the pre-deployed KubeRay, and perform elastic scaling when the preset elastic scaling conditions are triggered.

[0021] In this embodiment, node labels are set for each worker node in the cluster to indicate the type of hardware resources, and the load indicators of each worker node are obtained, including the resource utilization rate of hardware resources. Task requests are received, and the task stage and type are obtained based on the request path and request header fields. The requests are then forwarded to the corresponding Service for the task stage and type. The Service is associated with the scheduling node through the node labels. Task resource requirements, Pod topology constraints, and the resource utilization rate of the scheduling node are obtained to construct a QUBO model. A task-to-node allocation matrix is ​​obtained using the quantum annealing algorithm to determine the target node. The pod is started on the target node using a pre-deployed KubeRay, and elastic scaling is performed when preset elastic scaling conditions are triggered. By labeling hardware resource types with node labels, combined with the QUBO model and quantum annealing algorithm, task requirements and node resources are dynamically matched, effectively improving resource utilization and avoiding the problems of resource idleness or overload and low resource utilization caused by static allocation in related technologies.

[0022] Furthermore, in this embodiment, before setting node labels for each worker node in the cluster, the following steps are also included: Assign role information to each node in the cluster, including head node and worker node.

[0023] Preferably, the aforementioned hardware resource is a GPU or a CPU (Central Processing Unit). The type of hardware resource, either GPU or CPU, is indicated by setting node labels.

[0024] In this embodiment, the resource components of the cluster need to be deployed to enable subsequent scheduling and monitoring. Specifically, by deploying heterogeneous GPU and CPU nodes in the cluster, node tag information is used to distinguish resource types. Prometheus and Grafana are used to visualize the cluster resources, including resource utilization, computing unit load, and bandwidth. At the same time, an RDMA network policy is configured for the GPU nodes to reduce the latency of cross-node data transmission.

[0025] In addition, the built-in Dashboard component of KubeRay can be activated to monitor the granularity of inference training tasks in real time.

[0026] Furthermore, in one embodiment, step S2 above, obtaining the task stage and task type based on the request path and request header fields of the above request, specifically includes: First, the task stage of the request is determined by matching the request path using regular expressions.

[0027] Then, the above request header fields are parsed to extract the task identifier, which is used to distinguish the task type.

[0028] In some embodiments, traffic distribution via static paths is supported through the Nginx Ingress controller, which cannot identify the characteristics of internal task stages. Therefore, in this embodiment, based on defined resource control and scheduling rules, the task stage is obtained according to the request path of the aforementioned request, and the task type is obtained according to the request header fields of the aforementioned request, thereby achieving accurate distribution of request traffic for different task stages and types.

[0029] Furthermore, in one embodiment, after obtaining the load metrics of each working node, the method further includes: Based on preset resource alarm rules, the weight coefficients of each node are adjusted, and then the canary-weight of the Ingress controller is modified according to the weight coefficients of each node.

[0030] The above resource alarm rules include: when any load metric exceeds its corresponding first metric threshold and the duration exceeds the first time threshold, a resource alarm is triggered.

[0031] Preferably, for GPU nodes, the load metrics include GPU memory utilization, GPU bandwidth utilization, memory utilization, and the number of queued task requests. For CPU nodes, the load metrics include CPU memory utilization, CPU memory utilization, memory utilization, and the number of queued task requests.

[0032] Optionally, the aforementioned first time threshold is 60 seconds.

[0033] In this embodiment, for a GPU node, a resource alarm is triggered when the GPU memory utilization rate is greater than its corresponding first memory utilization rate threshold for 60 seconds, the GPU memory utilization rate is greater than its corresponding first memory utilization rate threshold for 60 seconds, the GPU bandwidth utilization rate is greater than its corresponding first bandwidth utilization rate threshold for 60 seconds, the memory utilization rate is greater than its corresponding first memory utilization rate threshold for 60 seconds, or the number of task request queues is greater than its corresponding first queue number threshold for 60 seconds.

[0034] Similarly, for a CPU node, a resource alarm is triggered when the CPU memory utilization rate is greater than its corresponding first memory utilization rate threshold for 60 seconds, the CPU memory utilization rate is greater than its corresponding first memory utilization rate threshold for 60 seconds, the memory utilization rate is greater than its corresponding first memory utilization rate threshold for 60 seconds, or the number of task request queues is greater than its corresponding first queue number threshold for 60 seconds.

[0035] In this embodiment, the first video memory utilization threshold, the first memory utilization threshold, the first bandwidth utilization threshold, the first memory usage threshold, and the first queue number threshold can all be set according to actual needs.

[0036] Optionally, the first video memory utilization threshold, the first memory utilization threshold, the first bandwidth utilization threshold, and the first memory usage threshold are all 80%.

[0037] In this embodiment, the Ingress controller can forward the above requests to the corresponding Service based on the canary weight, and then associate them with the corresponding scheduling node. In addition, when performing elastic scaling through the KubeRay framework, the target node can be scaled up or down based on the canary weight of the Ingress controller to further ensure the correct distribution of tasks.

[0038] In this embodiment, by extending the KubeRay custom components and combining the Ingress controller's dynamic routing and load balancing strategies, the scheduling of different task stages in distributed training can be effectively optimized.

[0039] Furthermore, in one embodiment, step S3 above, which involves constructing a QUBO model and obtaining the task-to-node allocation matrix using a quantum annealing algorithm, specifically includes: First, the resource requirements, Pod topology constraints, and resource utilization of the above tasks are encoded. Then, the Hamiltonian is constructed, which is the product of the resource gap weight and the task resource requirement code, plus the communication delay weight, the Pod topology constraint code, and the resource utilization code. Finally, with the goal of minimizing the Hamiltonian, the QUBO model is mapped onto the qubits of the quantum annealing machine, and the above allocation matrix is ​​output.

[0040] For GPU nodes, resource utilization includes GPU memory utilization, GPU RAM utilization, and GPU bandwidth utilization. For CPU nodes, resource utilization includes CPU memory utilization and CPU RAM utilization.

[0041] In this embodiment, by converting the scheduling problem in the cluster into a QUBO model for analysis, dynamic scheduling of cluster resources can be achieved according to the needs of different stages of the sensing task.

[0042] Preferably, minimizing the Hamiltonian is the objective, specifically including: When the above request is a preprocessing request, the communication delay weight is gradually increased, but not exceeding the first preset multiple of the resource gap weight. The communication delay weight with the smallest Hamiltonian is taken to obtain the minimum Hamiltonian. When the above request is an inference request, the resource gap weight is gradually increased, but not exceeding the second preset multiple of the resource gap weight. The resource gap weight with the smallest Hamiltonian is taken to obtain the minimum Hamiltonian.

[0043] In this embodiment, when the communication delay weight is gradually increased, the resource gap weight can remain unchanged; when the resource gap weight is gradually increased, the communication delay weight can remain unchanged.

[0044] Preferably, the first preset multiple and the second preset multiple are the same, both being 3 times.

[0045] In this embodiment, the weight of resource gap or communication delay is dynamically adjusted to focus on task submission at different stages. Specifically, in the inference stage, the weight of communication delay can be increased to reduce latency as a priority. In the preprocessing stage, the weight of resource gap can be increased to improve resource utilization as a priority.

[0046] Furthermore, in one embodiment, the above-mentioned elastic expansion / shrinkage capacity operation when a preset elastic expansion / shrinkage condition is triggered specifically includes: When elastic scaling conditions are triggered, the configuration file of Ray Service is automatically modified to adjust the resource quota or quantity of Pods, and a blue-green deployment strategy is adopted to achieve seamless migration of tasks.

[0047] Furthermore, in this embodiment, the above-mentioned elastic expansion and contraction conditions include expansion conditions and contraction conditions; The above expansion conditions include: the duration for which any load metric exceeds its corresponding second metric threshold exceeds the second time threshold; The above-mentioned scaling-down conditions include: the duration for which any load metric is below its corresponding third metric threshold exceeds the third time threshold; The threshold of the second indicator corresponding to any indicator is greater than the threshold of its corresponding third indicator.

[0048] In this embodiment, the second time threshold and the third time threshold may be the same or different.

[0049] Preferably, the second time threshold and the third time threshold are the same, both being 120s.

[0050] In this embodiment, when any load metric exceeds its corresponding second metric threshold for a duration exceeding the second time threshold, a scaling condition is triggered. At this time, scaling is performed by increasing the resource quota of the Pod or the number of Pods.

[0051] Optionally, for a GPU node, the expansion condition is triggered when the GPU memory utilization rate is greater than its corresponding second memory utilization rate threshold for 120 seconds, the GPU memory utilization rate is greater than its corresponding second memory utilization rate threshold for 120 seconds, the GPU bandwidth utilization rate is greater than its corresponding second bandwidth utilization rate threshold for 120 seconds, the memory utilization rate is greater than its corresponding second memory utilization rate threshold for 120 seconds, or the number of task request queues is greater than its corresponding second queue number threshold for 120 seconds.

[0052] For a CPU node, the expansion condition is triggered when the CPU memory utilization rate is greater than its corresponding second memory utilization rate threshold for 120 seconds, the CPU memory utilization rate is greater than its corresponding second memory utilization threshold for 120 seconds, the memory utilization rate is greater than its corresponding second memory utilization threshold for 120 seconds, or the number of task request queues is greater than its corresponding second queue number threshold for 120 seconds.

[0053] Similarly, when any load metric is below its corresponding third metric threshold for a duration exceeding the third time threshold, a scaling-down condition is triggered. In this case, scaling down is achieved by reducing the resource quota of the Pod or reducing the number of Pods.

[0054] Optionally, for a GPU node, a scaling-down condition is triggered when the GPU memory utilization is less than its corresponding third memory utilization threshold for 120 seconds, the GPU memory utilization is less than its corresponding third memory utilization threshold for 120 seconds, the GPU bandwidth utilization is less than its corresponding third bandwidth utilization threshold for 120 seconds, the memory utilization is less than its corresponding third memory utilization threshold for 120 seconds, or the number of task requests in the queue is less than its corresponding third queue number threshold for 120 seconds.

[0055] For a CPU node, the scaling-down condition is triggered when the CPU memory utilization is less than its corresponding third memory utilization threshold for 120 seconds, the CPU memory utilization is less than its corresponding third memory utilization threshold for 120 seconds, the memory utilization is less than its corresponding third memory utilization threshold for 120 seconds, or the number of task requests in the queue is less than its corresponding third queue number threshold for 120 seconds.

[0056] In this embodiment, each of the above thresholds can be set according to actual needs.

[0057] Optionally, the second video memory utilization threshold, the second memory utilization threshold, the second bandwidth utilization threshold, and the second memory usage threshold are all 80%.

[0058] Optionally, the aforementioned third video memory utilization threshold, third memory utilization threshold, third bandwidth utilization threshold, and third memory usage threshold are all 20%.

[0059] Furthermore, in this embodiment, when performing elastic scaling, the method also includes controlling the number of Pods started on each node to be between a preset maximum and minimum number.

[0060] In this embodiment, by setting the maximum and minimum number of Pods that can be started on the same node, excessive expansion and fragmentation of node resources can be effectively avoided.

[0061] like Figure 2 As shown, in one embodiment, the above-mentioned load balancing scheduling method specifically includes: Step 1: Deploy cluster resources and monitoring components to enable subsequent scheduling and monitoring.

[0062] First, assign role information (ray-component:head / worker) to the Kubernetes cluster nodes for subsequent KubeRay task scheduling and elastic scaling. Then, label the worker nodes with their hardware resource types: GPU nodes (accelerator:gpu) and CPU nodes (accelerator:cpu), and set a weight coefficient for each node (default 50%) for subsequent Ingress request routing, enabling accurate identification of node types during subsequent scheduling and monitoring.

[0063] Subsequently, Prometheus and Grafana were used to visualize cluster resources, including GPU / CPU utilization, compute unit load, and bandwidth. Simultaneously, an RDMA network policy was configured for worker nodes to reduce latency in cross-node data transmission. Furthermore, KubeRay's built-in Dashboard component was activated for real-time monitoring of inference and training tasks at various granularities.

[0064] Step 2: Extend the Ingress controller and Service rules to achieve accurate distribution of request traffic of different task stages and types through stage identification and routing enhancement in HTTP request headers.

[0065] The routing rules are as follows: Preprocessing stage: preprocessing / prefill stage ( / preprocess / prefill), preprocessing / decoding stage ( / preprocess / decode); Inference stage: inference / prefill stage ( / inference / prefill), inference / decoding stage ( / preprocess / decode). The Ingress controller categorizes paths: CPU node Service associated path: / preprocess( / |$)(.*) (matching all task requests in the preprocessing stage); GPU node Service associated path: / inference( / |$)(.*) (matching all task requests in the inference stage). By parsing the task identifier field in the HTTP request header and combining it with the path prefix of the Ingress controller, regular expression enhancement matching is achieved for different tasks, enabling precise location and routing of task requests.

[0066] Meanwhile, to improve cluster resource utilization, preprocessing tasks with low GPU dependency are routed to CPU nodes, while inference task requests that require high GPU memory are routed to GPU nodes. Furthermore, to monitor request routing in real time, Service Mesh is integrated with Prometheus and Grafana to collect and display load metrics (GPU / CPU memory utilization, GPU / CPU memory utilization, GPU bandwidth utilization) for relevant nodes, and related resource alert rules are set (e.g., triggering an alert if GPU / CPU memory utilization > 80%, GPU / CPU memory utilization > 80%, or GPU bandwidth utilization > 80% for 60 seconds).

[0067] The weight coefficients of nodes are dynamically adjusted based on alarm rules and load metrics (gradually increasing the weight coefficients of low-load, high-bandwidth nodes to an upper limit of 70%, and gradually decreasing the weight coefficients of high-load, memory-insufficient nodes to a lower limit of 30%). Specific implementation method: The canary-weight of the Ingress controller is modified according to the different weight coefficients of the nodes to achieve correct task distribution. Specifically, in alarm status, the node weight coefficients are modified every minute, optionally with an update ratio of 0.1 (i.e., an update step size of 10%). In normal status, the weight coefficients are checked every five minutes to achieve load balancing and maximize resource utilization, avoiding overload of individual node resources.

[0068] Step 3: Integrate dynamic scheduling strategies to achieve dynamic scheduling of cluster resources based on the needs of different stages of the perception task, in order to solve the problem that traditional scheduling strategies based on simple polling mechanisms cannot cope with the submission of training tasks under multi-dimensional constraints.

[0069] In this embodiment, task resource requirements and Pod topology constraints are obtained through task requests and platform configurations. Then, the task resource requirements, Pod topology constraints, and resource utilization of scheduling nodes are encoded into a QUBO model. The quantum annealing algorithm is used to solve for the optimal node allocation scheme, balancing the resource utilization and latency requirements of different tasks. The scheduling problem in the cluster is transformed into a QUBO model for analysis. First, the quantum scheduler `build_hamiltonian` function is defined to construct the Hamiltonian H: H = α × (Task resource requirements) + β × (Pod topology constraints) × (Resource utilization rate) With the goal of minimizing Hamiltonian, the QUBO model is mapped to the qubits of the quantum annealing machine, and the optimal allocation matrix for distributing requesting tasks to nodes is obtained through the quantum_scheduler function.

[0070] Furthermore, by dynamically adjusting the values ​​of α or β, the focus is on task submission at different stages. In the inference stage, increasing the weight of β to reduce latency is the priority goal, while in the preprocessing stage, increasing the weight of α to improve resource utilization is the priority goal.

[0071] Step 4: Deploy KubeRay and cluster scheduling strategies to enable elastic scaling of the cluster for tasks at different stages. This addresses the issue that existing cluster and Ray elastic scaling strategies rely solely on time slice theory for resource scheduling, failing to recognize the resource requirements of tasks at different stages.

[0072] In this embodiment, CRD components such as RayCluster and RayService are used to achieve task distribution and elastic scaling of cluster resources. The Head node (ray-component:head) is responsible for cluster node management, task distribution, and resource coordination; the worker node (ray-component:worker) executes the distributed tasks, and elastic scaling adapts to the needs of different tasks.

[0073] The elastic scaling rules are as follows: In the `autoscaling_config` configuration file on RayService, relevant elastic scaling conditions are set as trigger conditions. Specifically, scaling is triggered when GPU / CPU resource utilization > 80%, task request queues are too high, or memory usage is too high, and any of these conditions persists for 120 seconds. The maximum and minimum number of Pods that can be started on the same node are also set to avoid over-scaling and fragmentation of node resources. When monitoring metrics such as resource utilization and usage are high, worker nodes are vertically scaled up, and Pod resource quotas are increased. When the request queue backlog exceeds a threshold, Pods are quickly added for horizontal scaling. Similarly, when cluster resource utilization is low, task request queues are too few, or memory usage is too low, and any of these conditions persists for 120 seconds, triggering scaling down, the number of Pods is quickly reduced or resource quotas are lowered to release idle resources.

[0074] Furthermore, by collecting node load metrics (GPU, CPU, memory, request queue) through Prometheus, the Ray Service configuration file is automatically modified after a threshold is triggered. A blue-green deployment strategy is employed (creating new task queues and gradually migrating tasks to avoid interrupting task requests) to enable synchronized elastic scaling of old and new nodes, ensuring seamless backlog switching of task queues. The relevant configurations in the RayService file include: autoscaling_config configuration parameters, and resource quota information for workers and heads.

[0075] like Figure 3 As shown, in one embodiment, the above method includes: A1. Ingress Dynamic Routing: By extending the Ingress controller request path and identifying the model training task stage and type, dynamic weight adjustment and path regular expression matching are achieved to distribute requests to the corresponding Service; A2. K8s Resource Monitoring: By building a heterogeneous data resource pool, that is, dividing the K8s cluster nodes into high-priority (K8s high node) and low-priority resource nodes (K8s low node), and dynamically matching Pods according to the node weight coefficient, dynamic load balancing that is perceived by the model at different stages is achieved. A3. The RayCluster component is deployed to add or remove worker nodes from Pods. Through scheduling resource monitoring, integrating Prometheus and Grafana to monitor cluster resource pool metrics in real time, it drives dynamic routing decisions and optimizes load balancing across cluster nodes.

[0076] The load balancing scheduling method in this embodiment is based on the distributed training framework KubeRay and KubernetesIngress. By extending the Ingress path routing strategy, heterogeneous resource-aware scheduling, and dynamic load balancing algorithm, it solves problems such as low GPU resource utilization, coarse task stage scheduling, and low cross-node communication efficiency, and realizes dynamic matching between training task stages and hardware resources, thereby improving the efficiency of KubeRay distributed training and inference.

[0077] Secondly, embodiments of this application also provide a load balancing scheduling system.

[0078] In one embodiment, reference is made to Figure 4 , Figure 4 This is a functional module diagram of an embodiment of the load balancing scheduling system of this application. The load balancing scheduling system includes a cluster management module, a routing gateway, and a scheduling module.

[0079] The aforementioned cluster management module is used to set node labels for each worker node in the cluster to indicate the type of hardware resources, and to obtain the load indicators of each worker node; the load indicators include the resource utilization rate of hardware resources.

[0080] The aforementioned routing gateway is used to receive task requests, obtain the task stage and task type based on the request path and request header fields, and forward the request to the Service corresponding to the task stage and task type; the aforementioned Service is associated with the scheduling node through the aforementioned node label.

[0081] The aforementioned scheduling module is used to obtain task resource requirements, Pod topology constraints, and the resource utilization of the aforementioned scheduling nodes to construct a QUBO model, and to obtain the task-to-node allocation matrix through the quantum annealing algorithm to determine the target node.

[0082] The aforementioned cluster management module is also used to start pods on the target nodes via pre-deployed KubeRay, and to perform elastic scaling when preset elastic scaling conditions are triggered.

[0083] like Figure 5As shown, in one embodiment, the client sends a task request to the routing gateway. The routing gateway receives the task request by integrating an Ingress controller and forwards the request to the corresponding Service based on the parsing results of the request path and request header fields. The Service is associated with the scheduling node through the aforementioned node tags. The scheduling module obtains the task resource requirements, Pod topology constraints, and the resource utilization of the aforementioned scheduling nodes to construct a QUBO model. It then uses the quantum annealing algorithm to obtain the task-to-node allocation matrix and determine the target node. The cluster management module runs on the Kubernetes platform to start pods on the aforementioned target node through a pre-deployed KubeRay and performs elastic scaling when preset elastic scaling conditions are triggered.

[0084] Furthermore, in one embodiment, the routing gateway is also used for: The task stage of the request is determined by matching the request path using regular expressions. Parse the above request header fields to extract the task identifier, which is used to distinguish the task type.

[0085] Furthermore, in one embodiment, the cluster management module is also used for: Based on preset resource alarm rules, the weight coefficients of each node are adjusted, thereby modifying the canary-weight of the Ingress controller; The above resource alarm rules include: when any load metric exceeds its corresponding first metric threshold and the duration exceeds the first time threshold, a resource alarm is triggered.

[0086] Furthermore, in one embodiment, the scheduling module is also used for: Encode the above task resource requirements, Pod topology constraints, and resource utilization; Construct the Hamiltonian, which is the product of the resource gap weight and the task resource requirement code, plus the communication delay weight, the Pod topology constraint code, and the resource utilization code. With the goal of minimizing the Hamiltonian, the QUBO model is mapped onto the qubits of the quantum annealing machine, and the above allocation matrix is ​​output.

[0087] Furthermore, in one embodiment, the scheduling module is also used for: When the above request is a preprocessing request, the communication delay weight is gradually increased, but not exceeding the first preset multiple of the resource gap weight, and the communication delay weight is taken when the Hamiltonian is the smallest. When the above request is an inference request, the resource gap weight is gradually increased, but not exceeding the second preset multiple of the resource gap weight, and the resource gap weight is taken when the Hamiltonian is the smallest.

[0088] Furthermore, in one embodiment, the cluster management module is also used for: When elastic scaling conditions are triggered, the configuration file of Ray Service is automatically modified to adjust the resource quota or quantity of Pods, and a blue-green deployment strategy is adopted to achieve seamless migration of tasks.

[0089] In this embodiment, the above-mentioned elastic expansion and contraction conditions include expansion conditions and contraction conditions; The above expansion conditions include: the duration for which any load metric exceeds its corresponding second metric threshold exceeds the second time threshold; The above-mentioned scaling-down conditions include: the duration for which any load metric is below its corresponding third metric threshold exceeds the third time threshold; The threshold of the second indicator corresponding to any indicator is greater than the threshold of its corresponding third indicator.

[0090] Furthermore, in one embodiment, the cluster management module is also used for: Control the number of Pods started on each node between the preset maximum and minimum number.

[0091] The functions of each module in the above-mentioned load balancing scheduling system correspond to the steps in the above-mentioned load balancing scheduling method embodiment, and their functions and implementation processes will not be described in detail here.

[0092] Thirdly, embodiments of this application provide a load balancing scheduling device, which can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.

[0093] Reference Figure 6 , Figure 6 This is a schematic diagram of the hardware structure of the load balancing scheduling device involved in the embodiments of this application. In the embodiments of this application, the load balancing scheduling device may include a processor, a memory, a communication interface, and a communication bus.

[0094] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0095] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting devices within the load balancing scheduling equipment, as well as interfaces used for interconnecting the load balancing scheduling equipment with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0096] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0097] The processor can be a general-purpose processor, which can call a load balancing scheduler stored in memory and execute the load balancing scheduling method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the load balancing scheduler is called can be referred to in the various embodiments of the load balancing scheduling method of this application, and will not be repeated here.

[0098] Those skilled in the art will understand that Figure 6 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0099] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0100] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0101] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0102] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0103] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0105] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A load balancing scheduling method, characterized in that, The method includes: Set node labels for each worker node in the cluster to indicate the type of hardware resources, and obtain the load index of each worker node; the load index includes the resource utilization rate of hardware resources; Upon receiving a task request, the task stage and task type are obtained based on the request path and request header fields, and the request is forwarded to the Service corresponding to the task stage and task type; the Service is associated with the scheduling node through the node tag; The task resource requirements, Pod topology constraints, and resource utilization of the scheduling nodes are obtained to construct a QUBO model, and the task-to-node allocation matrix is ​​obtained through the quantum annealing algorithm to determine the target node. The pod is launched on the target node via a pre-deployed KubeRay and will perform elastic scaling when the preset elastic scaling conditions are triggered.

2. The load balancing scheduling method as described in claim 1, characterized in that, The task stage and task type are obtained based on the request path and request header fields, specifically including: The request path is matched using regular expressions to determine the task stage of the request; The request header fields are parsed to extract the task identifier, which is used to distinguish the task type.

3. The load balancing scheduling method as described in claim 1, characterized in that, After obtaining the load metrics for each worker node, the process also includes: Based on preset resource alarm rules, the weight coefficients of each node are adjusted, thereby modifying the canary-weight of the Ingress controller; The resource alarm rule includes: when any load metric exceeds its corresponding first metric threshold and the duration exceeds the first time threshold, a resource alarm is triggered.

4. The load balancing scheduling method as described in claim 1, characterized in that, Construct a QUBO model and obtain the task-to-node assignment matrix using the quantum annealing algorithm, including: The task resource requirements, Pod topology constraints, and resource utilization are encoded. Construct the Hamiltonian, which is the product of the resource gap weight and the task resource requirement code, plus the communication delay weight, the Pod topology constraint code, and the resource utilization code. With the goal of minimizing the Hamiltonian, the QUBO model is mapped onto the qubits of the quantum annealing machine, and the allocation matrix is ​​output.

5. The load balancing scheduling method as described in claim 4, characterized in that, The objective is to minimize the Hamiltonian, specifically including: When the request is a preprocessing request, the communication delay weight is gradually increased, but not exceeding a first preset multiple of the resource gap weight, and the communication delay weight is taken when the Hamiltonian is the smallest. When the request is an inference request, the resource gap weight is gradually increased, but not exceeding a second preset multiple of the resource gap weight, and the resource gap weight is taken when the Hamiltonian is the smallest.

6. The load balancing scheduling method as described in claim 1, characterized in that, The step of performing elastic expansion / shrinkage when a preset elastic expansion / shrinkage condition is triggered specifically includes: When elastic scaling conditions are triggered, the configuration file of Ray Service is automatically modified to adjust the resource quota or quantity of Pods, and a blue-green deployment strategy is adopted to achieve seamless migration of tasks.

7. The load balancing scheduling method as described in claim 6, characterized in that, The elastic expansion and contraction conditions include expansion conditions and contraction conditions; The expansion conditions include: the duration for which any load metric exceeds its corresponding second metric threshold exceeds a second time threshold. The reduction conditions include: the duration for which any load index is lower than its corresponding third index threshold exceeds the third time threshold. The threshold of the second indicator corresponding to any indicator is greater than the threshold of its corresponding third indicator.

8. The load balancing scheduling method as described in claim 6, characterized in that, When performing flexible expansion and contraction, it also includes: Control the number of Pods started on each node between the preset maximum and minimum number.

9. A load balancing scheduling system, characterized in that, The system includes: The cluster management module is used to set node labels for each worker node in the cluster to indicate the type of hardware resources, and to obtain the load indicators of each worker node; the load indicators include the resource utilization rate of hardware resources. A routing gateway is used to receive task requests, obtain the task stage and task type based on the request path and request header fields, and forward the request to the Service corresponding to the task stage and task type; the Service is associated with the scheduling node through the node tag; The scheduling module is used to obtain task resource requirements, Pod topology constraints, and resource utilization of the scheduling nodes to build a QUBO model and obtain the task-to-node allocation matrix through the quantum annealing algorithm to determine the target node. The cluster management module is also used to start pods on the target node via pre-deployed KubeRay, and to perform elastic scaling when preset elastic scaling conditions are triggered.

10. A load balancing scheduling device, characterized in that, The load balancing scheduling device includes a processor, a memory, and a load balancing scheduling program stored in the memory and executable by the processor, wherein when the load balancing scheduling program is executed by the processor, it implements the steps of the load balancing scheduling method as described in any one of claims 1 to 8.

Images