Traffic distribution method and system based on heterogeneous computing power adaptive load balancing

Abstract

This invention discloses a traffic distribution method and system based on heterogeneous computing power adaptive load balancing, belonging to the field of electronic digital data processing technology. The method includes: calculating the load score of a node based on the normalized value of resource indicators and the weighted sum of indicator weights; calculating the scheduling weight of traffic requests based on the load score; generating a random interval based on the sum of the scheduling weights of multiple nodes in the computing power cluster, and generating a random number within the random interval; generating a node interval based on the scheduling weight; and matching a target node based on the random number and the node interval. By normalizing resource indicators, the private indicators of heterogeneous computing power are converted into a unified standard dimension; and the load score is calculated based on the resource load of the node; the scheduling weight is calculated by combining the traffic request and the load score; and traffic scheduling is performed based on the random number generated by the scheduling weight, avoiding a sudden influx of a large number of requests into a single node and achieving a smooth transition of cluster load.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, specifically to a traffic distribution method and system based on heterogeneous computing power adaptive load balancing. Background Technology

[0002] With the explosive growth of artificial intelligence (AI) and deep learning applications, cloud computing data centers have evolved from simple CPU computing to hybrid clusters incorporating various heterogeneous computing powers such as GPUs, NPUs (Neural Processing Units), and DCUs (Deep Computing Units). In microservice architectures, load balancing distributes traffic to backend instances. However, existing load balancing technologies face the following significant challenges: 1. Heterogeneous resources are treated as a "black box." Traditional load balancing strategies (such as round-robin, random, and least connections) only focus on network layer connection status or CPU load, failing to perceive the actual load on backend nodes on heterogeneous hardware, such as GPU memory and NPU tensor cores. This can lead to computing requests being distributed to nodes that have exhausted their memory, causing service crashes or high latency.

[0003] 2. Significant differences in business resource sensitivity. Different types of businesses have drastically different resource dependencies. For example, AI inference is sensitive to GPU memory, big data computing is sensitive to memory bandwidth, and web services are sensitive to CPU. Traditional traffic distribution technologies use a "one-size-fits-all" scheduling logic, which cannot dynamically adjust the priority of resource weights based on business characteristics.

[0004] 3. The traditional strategy of distributing traffic to the least loaded node can easily cause all new requests to rush to the same low-load node at once, causing that node to be overloaded momentarily, triggering system instability and creating a herding effect. Summary of the Invention

[0005] To address the aforementioned technical problems in the existing technology, this invention provides a traffic distribution method and system based on adaptive load balancing of heterogeneous computing power. The method distributes traffic for heterogeneous computing power requests based on adaptive load balancing, thereby improving the overall utilization rate of heterogeneous computing power resources.

[0006] This invention discloses a traffic distribution method based on heterogeneous computing power adaptive load balancing, comprising the following steps: collecting resource indicators of the computing power cluster; normalizing the resource indicators to obtain normalized values; matching the indicator weights of traffic requests based on service type; calculating the load score of a node based on the weighted sum of the normalized values ​​of the resource indicators and the indicator weights; calculating the scheduling weight of the traffic request based on the load score; generating a random interval based on the sum of the scheduling weights of multiple nodes in the computing power cluster, and generating a random number within the random interval; generating a node interval based on the node's scheduling weight; matching a target node based on the random number and the node interval; and distributing the traffic request to the target node.

[0007] Preferably, the collected resource indicator data are preprocessed based on the exponentially weighted moving average method; after preprocessing, the resource indicators are normalized. Business types include AI inference and web services. For AI inference business types, the weight of video memory is greater than the weight of GPU computing power, and the weight of GPU computing power is greater than the weight of CPU computing power.

[0008] Preferably, the normalized value is calculated based on the ratio of the current usage value to the maximum value of the resource indicator; Resource metrics are categorized into computational metrics and storage metrics. The normalized calculation formula for computational metrics is expressed as follows: V compute = current flops / max flop 100 （1）； in, V compute To calculate the normalized value of the indicator; current flops The computing resources used for calculating the metrics. max flop This represents the maximum computational resource required to calculate the metrics. The normalized formula for the storage metric is expressed as: V memory = Used Memory / Total memory 100 (2); in, V memory To store the normalized values ​​of the metrics, Used Memory For used storage resources ,Total memory This represents the total amount of storage resources.

[0009] Preferably, the load score is calculated as follows: (3); in, s i For nodes i The load score, M ij Represented as nodes i The j The normalized value of each resource indicator,n This represents the total number of resource indicators. W j For the first j The weight of each resource indicator; a This is a penalty factor.

[0010] Preferably, if the normalized values ​​of all resource indicators of the current node are less than the safety threshold, the penalty factor is set to 1. If the normalized value of any resource indicator of the current node is greater than or equal to the safety threshold, the penalty factor is calculated exponentially, which is expressed as: , in, Threshold safe This is a safety threshold; k It is expressed as a sensitivity coefficient.

[0011] Preferred scheduling weight P i Represented as: (6); in, To adjust the power exponent for sensitivity; To prevent extremely small constants with a denominator of zero.

[0012] Preferably, random numbers r Represented as: ; in, for m The sum of the scheduling weights of each node. It is a random interval.

[0013] Preferably, the method for generating node intervals includes: Based on the preset starting point X 0. Establish a node sequence based on the node's scheduling weight. The node sequence is represented as: { X 0 , X 1 , X 2 , X i , ..., X m}, X i = X i-1 + P i ; Based on the current node in the node sequence i Establish node intervals by taking the sequence value and the previous sequence value.

[0014] Preferably, the node range of the current node is represented as follows: [ X i-1 , X i )or( X i-1 , X i ] or[ X i-1 , X i ].

[0015] The present invention also provides a distribution system for implementing the above-described traffic distribution method, comprising a data acquisition module, a load analysis module, and a traffic scheduling module; The acquisition module is used to collect resource indicators of the computing power cluster; The load analysis module is used to normalize resource indicators to obtain normalized values; match the indicator weights of traffic requests based on business patterns; calculate the load score of a node based on the weighted sum of the normalized values ​​of resource indicators and indicator weights; and calculate the scheduling weight of traffic requests based on the load score. The traffic scheduling module is used to generate a random interval based on the sum of the scheduling weights of multiple nodes; generate a random number within the random interval; generate a node interval based on the scheduling weights of the nodes; match the target node based on the random number and the node interval; and distribute the traffic request to the target node.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: by normalizing the private indicators of heterogeneous computing power, the private indicators of heterogeneous computing power are transformed into a unified standard dimension; and the load score is calculated according to the resource load of the node; the scheduling weight is calculated by combining the traffic request and the load score; and the traffic scheduling is performed based on the random number of the scheduling weight, so as to avoid a large number of requests flooding into a single node at any time and achieve a smooth transition of cluster load. Attached Figure Description

[0017] Figure 1 This is a logic block diagram of the traffic distribution method based on heterogeneous computing power adaptive load balancing of the present invention; Figure 2 This is a logic block diagram of the distribution system of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The present invention will now be described in further detail with reference to the accompanying drawings: Example 1 provides a traffic distribution method based on heterogeneous computing power adaptive load balancing, such as... Figure 1 As shown, it includes the following steps: Step S1: Collect resource metrics of the computing cluster.

[0020] Specifically, the collected basic resource metrics include CPU load, memory usage, and disk I / O; heterogeneous resource metrics include GPU SM utilization, GPU memory usage, NPU Cube utilization, and NPU HBM utilization; application metrics include Requests Per Second (QPS) and Response Time (RT); but are not limited to these. The computing cluster consists of multiple nodes.

[0021] Step S2: Normalize the resource indicators to obtain normalized values.

[0022] Through normalization, heterogeneous resource indicators are mapped to standard values ​​of 0-100. V norm To align with the differences between different hardware.

[0023] Step S3: Match the metric weights of traffic requests based on the business type.

[0024] Step S4: Calculate the node's load score based on the normalized values ​​of the resource indicators and the weighted sum of the indicator weights. s i .

[0025] Step S5: Calculate the scheduling weight of traffic requests based on the load score.

[0026] Step S6: Generate a random interval based on the sum of the scheduling weights of multiple nodes; and generate random numbers within the random interval.

[0027] Step S7: Generate node intervals based on the scheduling weights of the nodes.

[0028] Step S8: Match the target node based on the random number and node range.

[0029] Step S9: Distribute the traffic request to the target node.

[0030] By normalizing resource metrics, the private metrics of heterogeneous computing power are transformed into a unified standard dimension; and a load score is calculated based on the resource load of the nodes; a scheduling weight is calculated by combining traffic requests and load scores; and traffic scheduling is performed based on random numbers of scheduling weights to avoid a large number of requests flooding into a single node at once, thus achieving a smooth transition of cluster load.

[0031] In step S1, resource metrics can be collected based on a full-stack resource probe (Metric Agent): deployed on backend service nodes, utilizing low-level interfaces such as eBPF, NVML (NVIDIA Management Library), DCMI (DCU Management Interface), and Ascend-SMI, to collect full-stack resource metrics such as CPU, memory, GPU (video memory / computing power), and NPU in real time. Besides proactively reporting using the Agent deployed on the node, a pull method using Prometheus / Metrics Server can also be used. Alternatively, a Sidecar mode (such as Istio Envoy) can be employed, where a Sidecar agent intercepts traffic and collects cgroup information from local containers for reporting.

[0032] This invention can also smooth the collected data based on the exponentially weighted moving average (EWMA): ; in, This is a smoothing factor (e.g., 0.3). Vraw t for t The collected values ​​at each moment, M t for t The values ​​are smoothed over time. This effectively avoids traffic fluctuations caused by jumps in a single metric, making the cluster load curve smoother.

[0033] In step S2, the normalized value is calculated based on the ratio of the current usage value to the maximum value of the resource indicator. Specifically, the resource indicators include computational indicators and storage indicators. The normalized calculation formula for computational indicators is expressed as follows: V compute = currentflops / max flop 100 (1); in, V compute To calculate the normalized value of the indicator; current flops The computing resources used for calculating the metrics. max flop This represents the maximum computational resources required to calculate the metrics.

[0034] The normalized formula for the storage metric is expressed as: V memory = Used Memory / Total memory 100 (2); in, V memory To store the normalized values ​​of the metrics, Used Memory For storage resources already in use, Total memory This represents the total amount of storage resources.

[0035] In step S3, the weights of the metrics can be based on empirical values. For example, in AI inference, the weight of GPU memory is greater than the weight of GPU computing power, and the weight of GPU computing power is greater than the weight of CPU computing power. More specific weights can be assigned to GPU memory utilization. W vram = 0.6, GPU computing power weight W Gpu_compute = 0.3, CPU computing power weight W Cpu =0.1; In this mode, memory utilization dominates the score. In web services, CPU computing power is weighted. W Cpu =0.5, memory utilization weight W mem = 0.3, Network Resource Weight W conn = 0.2. But not limited to this. The weight of the metric can be matched with the business model and template based on the metadata (Header / Path) of the traffic request.

[0036] In step S4, the load score is calculated as follows: (3); in, s i For nodes iThe load score, M ij Represented as nodes i The j The normalized value of each resource indicator, n This represents the total number of resource indicators. W j For the first j The weight of each resource indicator; a This is a penalty factor.

[0037] If the normalized value of any resource indicator of the current node is greater than or equal to the safety threshold, the penalty factor is calculated exponentially, which is expressed as: , Diff = M ij – Threshold safe (4), in, Threshold safe This is a safety threshold; Diff This represents the amount by which a resource indicator exceeds a safe threshold. k This is expressed as a sensitivity coefficient, which is a constant, for example, 0.5, used to control the steepness of the growth of the penalty factor. k The larger the value, the faster the score rises above the limit, and the more decisive the circuit breaker will be.

[0038] In another embodiment, the penalty factor can also be calculated as follows: (5). The penalty factor uses an exponential function to achieve a non-linear "soft circuit breaker" effect. Compared to linear penalties (which grow too slowly) or hard threshold cutoffs (which are too abrupt), the exponential function has the characteristic of "explosive growth after the critical point." For example, when the load exceeds the safety threshold by 1%, the penalty factor... a Only a slight increase, with a slight decrease in traffic; when it exceeds 5%, the penalty factor... a The bandwidth increases dramatically, almost completely blocking traffic. This ensures system security under high load while retaining some throughput capacity in extreme situations. Effect: Once resource metrics such as GPU memory approach overflow, the load score... s i The probability of the node being selected approaches zero due to the exponential increase in the number of nodes, thus achieving "soft circuit breaker" protection.

[0039] In step S5, scheduling weights P i Represented as: (6); in, To adjust the power exponent of sensitivity, it is usually set to 2; To prevent extremely small constants with a denominator of zero, such as 10 -6 .

[0040] In step S6, random number r Represented as: (7); in, for m The sum of the scheduling weights of each node. It is a random interval.

[0041] The method for generating node intervals in step S7 includes: Step 701: Based on the preset starting point X 0. Establish a node sequence based on the node's scheduling weight. The node sequence is represented as: { X 0 , X 1 , X 2 , X i , ..., X m}, X i = X i-1 + P i .

[0042] Step 702: Based on the current node in the node sequence i sequence values X i and the previous sequence value X i-1 Establish node intervals.

[0043] The node range of the current node is represented as: [ X i-1 , X i )or( X i-1 , X i ] or[ X i-1 , X i (8).

[0044] For example, the scheduling weight of the first node is 70, the scheduling weight of the second node is 30, the scheduling weight of the third node is 20, and the sum of the scheduling weights is 120. (Starting point...) X0 If the value is zero, the node range for the first node is [0, 70), the node range for the second node is [70, 100), and the node range for the third node is [100, 120]. The larger the scheduling weight value, the larger the range of the node's node range, and the greater the probability that the random number will fall within that range. If the random number is generated... r =45, falling within the node range of the first node, then the traffic request is distributed to the first node; if r If the value is 85, it will be distributed to the second node. The first node has the lowest load, but it will not monopolize all traffic requests, thus achieving a smooth transition of cluster load and avoiding a single point of failure.

[0045] In another embodiment, the calculation method for indicator weights and scheduling weights can also employ a prediction model based on neural networks (such as MLP) or fuzzy logic controllers for prediction and calculation.

[0046] To verify the effectiveness of the traffic distribution method, a stress test comparison of Stable Diffusion inference service was conducted in a hybrid heterogeneous cluster containing 10 nodes (5 T4 GPU cards and 5 Ascend 910 NPU cards). The comparison results are shown in Table 1.

[0047] Scenario setup: Simulate high-concurrency requests and continuously increase QPS until the cluster is saturated.

[0048] Table 1 The traffic distribution method of the present invention has the following effects: 1. Significantly improve the stability of AI services: By incorporating GPU memory into core decision factors and combining it with a non-linear circuit breaker mechanism, the service crash problem caused by memory overflow (OOM) is completely solved, ensuring the continuity of large model inference services.

[0049] 2. Maximize resource utilization: It achieves fine-grained scheduling of heterogeneous resources, and "computing power sensitive" and "memory sensitive" tasks are rationally allocated to corresponding advantageous nodes, increasing the overall throughput of the cluster by more than 20%.

[0050] 3. Compatible with heterogeneous ecosystems: The solution does not depend on specific hardware manufacturers. Through a unified and normalized interface, it can seamlessly manage mixed computing resources from multiple manufacturers, reducing the complexity of operation and maintenance.

[0051] 4. Eliminate herding effect: Based on EWMA smoothing and probabilistic scheduling algorithms, it effectively avoids traffic fluctuations caused by single index jumps, making the cluster load curve smoother.

[0052] This invention converts proprietary metrics of different hardware architectures (GPU / NPU / DCU) into a unified standard dimension (0-100%), particularly focusing on the alignment logic for "computing power" and "video memory capacity". It dynamically matches and switches weight coefficients based on business type and dynamically adjusts the load score calculation model; and introduces a non-linear penalty factor. a It grows exponentially to handle critical resource states (such as the risk of memory overflow); it uses scheduling weights to perform probability transformations of random numbers to avoid herding effects.

[0053] Example 2 provides a distribution system for implementing the above traffic distribution method, such as... Figure 2 As shown, it includes a data acquisition module 1, a load analysis module 2, and a traffic scheduling module 3; The acquisition module 1 is used to collect resource indicators of the computing power cluster; Load analysis module 2 is used to normalize resource indicators to obtain normalized values; match the indicator weights of traffic requests based on business patterns; calculate the load score of nodes based on the weighted sum of the normalized values ​​of resource indicators and indicator weights; and calculate the scheduling weight of traffic requests based on the load score. The traffic scheduling module 3 is used to generate a random interval based on the sum of the scheduling weights of multiple nodes; generate a random number within the random interval; generate a node interval based on the scheduling weights of the nodes; match the target node based on the random number and the node interval; and distribute the traffic request to the target node.

[0054] Example 3 provides a distribution device, including a processor and a memory, wherein the memory stores code, and when the code is processed by the processor, the above-described traffic distribution method is executed.

[0055] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A traffic distribution method based on heterogeneous computing power adaptive load balancing, characterized in that, Includes the following steps: Collect resource metrics of the computing cluster; The resource indicators are normalized to obtain normalized values; Weighting of traffic requests based on business type; The load score of a node is calculated based on the normalized value of the resource indicator and the weighted sum of the indicator weights. The scheduling weight of the traffic request is calculated based on the load score; Based on the sum of the scheduling weights of multiple nodes in the computing power cluster, a random interval is generated, and random numbers are generated within the random interval. Node intervals are generated based on the scheduling weights of the nodes; Match the target node based on the random number and node range; The traffic request is distributed to the target node.

2. The traffic distribution method according to claim 1, characterized in that, The collected resource indicator data are preprocessed using an exponentially weighted moving average method; after preprocessing, the resource indicators are normalized. Business types include AI inference and web services. For AI inference business types, the weight of video memory is greater than the weight of GPU computing power, and the weight of GPU computing power is greater than the weight of CPU computing power.

3. The traffic distribution method according to claim 1, characterized in that, Calculate the normalized value based on the ratio of the current usage value to the maximum value of the resource indicator; Resource metrics are categorized into computational metrics and storage metrics. The normalized calculation formula for computational metrics is expressed as follows: V compute = current flops / max flop 100 ； in, V compute To calculate the normalized value of the indicator; current flops The computing resources used for calculating the metrics. max flop This represents the maximum computational resource required to calculate the metrics. The normalized formula for the storage metric is expressed as: V memory = Used Memory / Total memory 100 ； in, V memory To store the normalized values ​​of the metrics, Used Memory For used storage resources Total memory This represents the total amount of storage resources.

4. The traffic distribution method according to claim 1, characterized in that, The load score is calculated as follows: ； in, s i For nodes i The load score, M ij Represented as nodes i The j The normalized value of each resource indicator, n This represents the total number of resource indicators. W j For the first j The weight of each resource indicator a This is a penalty factor.

5. The traffic distribution method according to claim 4, characterized in that, If the normalized values ​​of all resource metrics of the current node are less than the safety threshold, the penalty factor is set to 1. If the normalized value of any resource indicator of the current node is greater than or equal to the safety threshold, the penalty factor is calculated exponentially, which is expressed as: ， in, Threshold safe This is a safety threshold; k It is expressed as a sensitivity coefficient.

6. The traffic distribution method according to claim 4, characterized in that, Scheduling weight P i Represented as: （6）； in, To adjust the power exponent for sensitivity; To prevent extremely small constants with a denominator of zero.

7. The traffic distribution method according to claim 6, characterized in that, random numbers r Represented as: ； in, for m The sum of the scheduling weights of each node. It is a random interval.

8. The traffic distribution method according to claim 7, characterized in that, Methods for generating node intervals include: Based on the preset starting point X 0. Establish a node sequence based on the node's scheduling weight. The node sequence is represented as: { X 0 , X 1 , X 2 , X i , ..., X m }, X i = X i-1 + P i ; Based on the current node in the node sequence i Establish node intervals by taking the sequence value and the previous sequence value.

9. The traffic distribution method according to claim 8, characterized in that, The node range of the current node is represented as: [ X i-1 , X i )or( X i-1 , X i ] or[ X i-1 , X i ] ; in, X i For the current node i sequence values, X i-1 It is the value of the previous sequence.

10. A distribution system for implementing the traffic distribution method as described in any one of claims 1-9, characterized in that, It includes a data acquisition module, a load analysis module, and a traffic scheduling module; The acquisition module is used to collect resource indicators of the computing power cluster; The load analysis module is used to normalize resource metrics and obtain normalized values. Based on the business model, match the indicator weights of traffic requests; calculate the load score of the node based on the normalized value of the resource indicator and the weighted sum of the indicator weights. Based on the load score, calculate the scheduling weight of the traffic request; The traffic scheduling module is used to generate a random interval based on the sum of the scheduling weights of multiple nodes; A random number is generated within a random interval; a node interval is generated based on the node's scheduling weight; a target node is matched based on the random number and the node interval; and the traffic request is distributed to the target node.

Images