Asymmetric segmentation scheduling system and method in heterogeneous GPU cluster

Abstract

The invention provides an asymmetric segmentation scheduling system and method in a heterogeneous GPU cluster, and the method comprises the steps: S1, checking the features of an incoming request, and grouping the incoming request into different request buckets according to the token length; s2, in a heterogeneous GPU cluster environment, optimizing a large language model reasoning instance by adopting a double-layer strategy; and S3, calculating the matching degree of the request buckets and the big language model reasoning instances, and scheduling different request buckets to the big language model reasoning instance with the highest matching degree according to a calculation result. According to the method provided by the invention, the model layer can be asymmetrically segmented according to the computing power and the video memory capacity of each GPU on the premise of satisfying the model parallelism degree constraint, the dynamic balance of the execution duration between stages is realized, assembly line cavitation bubbles are fundamentally reduced, and the overall throughput rate is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of large language models, in particular to an asymmetric partitioning scheduling system and method in a heterogeneous GPU cluster, and more particularly to an asymmetric partitioning scheduling system and method for large language model inference in a heterogeneous GPU cluster. BACKGROUND

[0002] Large language models (LLMs), such as GPT-4 and Llama-3, have revolutionized artificial intelligence by exhibiting cutting-edge performance in natural language understanding and content generation. Their deployment primarily adopts two paradigms: online inference services (requiring low-latency responses) and offline batch inference (focusing on high-throughput). Online inference supports real-time applications, such as chatbots, intelligent customer service, and personal assistants, which require sub-second response times and high availability. On the other hand, offline inference is designed for large-scale, non-interactive tasks, such as document summarization, data set preprocessing, and automatic report generation. Although the priorities of online and offline scenarios differ, they both face a core challenge: coping with the rising costs of increasingly large LLMs. The deployment of GPUs in current production systems is characterized by diversity. As shown in FIG. 1, in three representative production clusters, each cluster contains approximately 10,000, 5,000, and 2,000 GPUs, respectively, of which less than 50% belong to the A100 level, and the rest are older models such as V100, T4, and A10, or consumer-grade devices such as the RTX40 series. This diversity is due to phased upgrades, budget constraints, and the reallocation of resources across clusters. For example, compared to mid-range or consumer-grade devices (such as V100, T4), high-end accelerators such as A100 and H200 not only cost 2 to 5 times more per hour, but are also in short supply. Figure 1

[0003] Deploying LLM services on heterogeneous clusters helps reduce hardware costs, but also presents two key challenges.

[0004] ​First, hardware heterogeneity leads to significant performance disparity. High-end GPUs (e.g., H20) can provide more than 300 TFLOPS of computing power, while low-end devices (e.g., A10 and T4) can only reach 40 to 120 TFLOPS, leading to a performance gap of 3 to 8 times. Memory disparity further exacerbates this imbalance: A100 has 80 GB HBM, while A10 has only 24 GB, which greatly limits the number of model layers or key-value cache entries that each device can host. In addition, the difference in intra-node links (whether NVLink or PCIe) can significantly affect certain parallel strategies, such as tensor parallelism. Second, the diversity of workloads further exacerbates the complexity of scheduling. LLM services differ in multiple dimensions, including model size, input sequence length, and request arrival pattern, all of which interact with hardware constraints in extraordinary ways. This variability makes it difficult to efficiently configure heterogeneous resources while ensuring high throughput in different scenarios.

[0005] Existing LLM service engines can generally be divided into two categories. The first category includes throughput-oriented batch engines, such as BlendServe and BatchLLM, which emphasize efficient memory management and request-level scheduling to maximize utilization. The second category includes latency-oriented online engines, such as SarathiServe and AlphaServe, which employ techniques such as speculative execution and fine-grained scheduling to reduce response latency. Although these systems differ from each other, they have a common limitation: they are typically designed with homogeneous resources in mind, adopting a symmetric model partitioning that distributes layers evenly across devices. However, in a heterogeneous cluster, this approach can result in significantly low efficiency due to the straggler effect. SUMMARY

[0006] In view of the defects in the prior art, the purpose of the present application is to provide an asymmetric partitioning scheduling method and system in a heterogeneous GPU cluster.

[0007] According to the asymmetric partitioning scheduling system in a heterogeneous GPU cluster provided by the present application, the system comprises: A length-aware request bucketizer is configured to inspect the characteristics of incoming requests and group them into different request buckets according to token length. A heterogeneous-aware partitioning engine is configured to optimize large language model inference instances in a heterogeneous GPU cluster environment using a two-layer strategy. A bipartite matching scheduler is configured to calculate the matching degree of request buckets and large language model inference instances, and to schedule different request buckets to the large language model inference instance with the highest matching degree according to the calculation results.

[0008] Preferably, the length-aware request bucketer uses online kernel density estimation to model the empirical distribution of token lengths and determines the optimal bucketing boundary by analyzing the properties of the distribution.

[0009] Preferably, online kernel density estimation includes: Enable the length-aware request bucket to maintain the most recently observed request length. A sliding window is used to continuously estimate the empirical density. ; For the sample Construct a continuous empirical density function:

[0010] in, For Gaussian kernel function, The bandwidth parameter was selected based on Silverman's rule of thumb. Length-aware request buckets detect density gradients. The optimal segmentation point is dynamically identified by analyzing the changing regions within the localization density estimation; candidate segmentation points are determined by analyzing the inflection points in the localization density estimation.

[0011] in It is a preset gradient magnitude threshold; Candidate set according to Sort in descending order, select the first The candidate set is used as the optimal hierarchical breakpoint; breakpoint Regular updates and These represent the observed minimum and maximum lengths, respectively. A boundary value is generated. Each internal element bucket, new requests will be based on the interval boundaries. Assigned to batch ; For each containing length of The requested storage bucket Calculate the empirical average as a representative length measure: .

[0012] Preferably, the two-layer strategy includes: In the first phase, GPUs of different levels are combined to configure large language model instances; In the second phase, the large language model layer within each instance is partitioned to keep the workload consistent with the heterogeneous GPU combination, while taking into account both computing power and memory capacity.

[0013] Preferably, a heterogeneous cluster of device types is formalized as a GPU assignment problem of the form: A GPU assignment problem of a heterogeneous cluster of device types is formalized as:

[0014] where each tuple represents a GPU type and its available quantity; Given a parallel configuration:

[0015] where , and denote the parallelism of data, tensor and pipeline respectively.

[0016] Preferably, the bipartite matching scheduler maps request buckets to large language model instances and determines the batch size for each instance.

[0017] According to the asymmetric partitioning scheduling method in the heterogeneous GPU cluster provided by the application, comprising: Step S1: Check the characteristics of incoming requests, and group them into different request buckets according to token length; Step S2: In the heterogeneous GPU cluster environment, adopt a double-layer strategy to optimize large language model inference instances; Step S3: Calculate the matching degree of the request bucket and the large language model inference instance, and schedule different request buckets to the large language model inference instance with the highest matching degree according to the calculation result.

[0018] Preferably, the step S1 includes modeling the empirical distribution of token length using online kernel density estimation, and determining the optimal bucketing boundary by analyzing the properties of the distribution.

[0019] Preferably, the online kernel density estimation includes: The length-aware request bucketizer maintains a sliding window of the latest observed request lengths and continuously estimates the empirical density ; For samples a continuous empirical density function is constructed:

[0020] where, is a Gaussian kernel function, is a bandwidth parameter selected according to Silverman's empirical rule; The length-aware request bucketizer dynamically identifies the optimal split point by detecting regions of changing density gradient ; candidate split points are determined by locating inflection points in the density estimate:

[0021] wherein is a preset gradient magnitude threshold value; candidate set is arranged in descending order, and the first candidate set is selected as the best hierarchical breakpoint; breakpoint is updated regularly, and and respectively represent the observed minimum and maximum lengths, boundary values are generated internal element buckets, and new requests will be assigned to batches according to interval boundaries . ; For each storage bucket containing requests with a length of , the empirical mean is calculated as a representative length measure: .

[0022] Preferably, the double-layer strategy comprises: a first stage, combining different levels of GPUs to configure large language model instances; a second stage, partitioning large language model layers within each instance to keep workloads consistent with heterogeneous GPU combinations, while taking into account computing power and memory capacity.

[0023] Preferably, the GPU allocation problem of a heterogeneous cluster containing device types is formalized as:

[0024] where each tuple represents a GPU type and its available number; Given a parallel configuration:

[0025] where , and represent the parallelism of data, tensor and pipeline, respectively.

[0026] Preferably, the bipartite matching scheduler maps request buckets to large language model instances and determines the batch size of each instance.

[0027] Compared with the prior art, the present application has the following beneficial effects: ​1. The application evaluates the deficiencies of symmetric partitioning in hardware heterogeneity, workload patterns and parallel strategies for hybrid GPUs.

[0028] 2. The application proposes a request bucketing mechanism, which obtains the probability density function by maintaining the sliding window statistics of the length of the request token, using online kernel density estimation (KDE), determines the bucketing boundary according to the inflection point of the density gradient, and forms micro-batches by grouping requests with similar lengths.

[0029] 3. The application designs a heterogeneous-aware partitioning method, which maximizes the overall utilization under the constraints of data / tensor / pipeline parallelism by using heuristic instance provisioning and asymmetric layer cutting methods.

[0030] 4. The application reduces the performance imbalance and high latency problems existing in the LLM inference stage in the heterogeneous GPU cluster through the segmentation method; based on the asymmetric segmentation and heterogeneous-aware scheduling LLM inference optimization method, by analyzing the inference request characteristics and GPU resource differences, a length-aware request bucketing module, a heterogeneous-aware partitioning engine and a bipartite graph matching scheduling mechanism are designed, which realizes efficient scheduling and execution of large language model inference tasks in a heterogeneous cluster, significantly reduces the first token latency and average response time, and improves the GPU utilization and overall performance of the system.

[0031] 5. The method provided by the application can asymmetrically cut the model layers according to the computing power and video memory capacity of each GPU under the premise of meeting the model parallelism constraint, realize dynamic balancing of execution time between stages, and fundamentally reduce the pipeline void and improve the overall throughput.

[0032] 6. The application establishes a supply and demand matching model of request bucketing and inference instances, combines the available video memory and communication cost of instances, predicts the request distribution and realizes dynamic batch size adjustment, effectively solving the problem of uneven resource allocation in the inference process. BRIEF DESCRIPTION OF DRAWINGS

[0033] Other features, objects and advantages of the application will become more apparent through reading the following detailed description of the non-limiting embodiments with reference to the accompanying drawings: Figure 1 GPU distribution diagram in three production clusters.

[0034] Figure 2 The framework structure diagram of the application.

[0035] Figure 3 The flowchart of the length-aware request bucketing device.

[0036] Figure 4 The pseudo-code diagram of the heuristic-based instance configuration.

[0037] Figure 5 Pseudocode diagram for asymmetric partitioning strategy.

[0038] Figure 6 Flowchart for binary matching scheduler.

[0039] Figure 7 Pseudocode diagram for binary matching scheduler. DETAILED DESCRIPTION

[0040] The application will be described in detail below with specific embodiments. The following examples will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that for those skilled in the art, without departing from the concept of the application, a number of changes and improvements can be made. These are within the scope of the present application.

[0041] As shown in Figure 2 An asymmetric partitioning scheduling system in a heterogeneous GPU cluster includes: A length-aware request bucketizer for inspecting the characteristics of incoming requests and grouping them into different request buckets according to token length; A heterogeneous-aware partitioning engine for optimizing large language model inference instances in a heterogeneous GPU cluster environment using a two-layer strategy; A binary matching scheduler for calculating the matching degree of request buckets and large language model inference instances, and scheduling different request buckets to the large language model inference instance with the highest matching degree according to the calculation result.

[0042] In one embodiment, a framework is designed that leverages asymmetric partitioning to improve the efficiency of heterogeneous resources while ensuring service level objectives (SLOs). The core idea of the framework is to proportionally utilize the contribution of each GPU tier, resulting in overall efficiency that exceeds that of using only a select set of high-end GPUs. To achieve this, the present application dynamically manages request batching, model partitioning, and resource scheduling through three synergistic components: Length-aware request bucketizer (LRB), heterogeneous-aware partitioning engine (HPE), and binary matching scheduler (BMS).

[0043] These three components work synergistically through a closed-loop design to maintain high throughput.

[0044] First, by inspecting the characteristics of incoming requests, the LRB uses density estimation of token length distribution to group requests into micro-batches of similar length. This approach reduces padding overhead and ensures more consistent execution time across batches. Second, the HPE improves utilization of various resources through a two-layer strategy, thereby eliminating performance bottlenecks: (1) Determine the instance number and its GPU combination according to the device performance, LLM (Large Language Model) metadata and parallel strategy; (2) Divide the LLM into multiple slices according to the FLOP and memory capacity of the GPU of the given instance.

[0045] Thirdly, BMS improves throughput by calculating the relevance score between the request storage bucket and the LLM instance, and schedules each batch to the instance that best meets its computing needs, thereby promoting load-aware execution.

[0046] It is worth noting that the present application enhances the performance of vLLM and mainstream LLM service framework by integrating heterogeneous perception model division and adaptive scheduling. This method improves throughput, reduces latency and improves SLO compliance in heterogeneous GPU clusters while retaining the original optimization.

[0047] Next, the components in the system will be described in detail: 1. Length-aware request bucketizer (LRB) The current LLM inference service system faces two fundamental challenges that significantly affect performance and efficiency. First, the inherent differences in token length in incoming requests require a large amount of padding in batch processing. This leads to low resource utilization, as the processing time of each batch is determined by the longest sequence, while shorter requests experience unnecessary delays, ultimately reducing latency and throughput. Second, this difference leads to systematic imbalance in memory and computing resource allocation. Requests with longer sequences disproportionately consume resources, resulting in suboptimal batch processing efficiency and reducing overall performance. These challenges are even more severe in a heterogeneous environment.

[0048] Token frequency follows the Zipf law, however, the token length of incoming requests is unknown before execution. To address this issue, Figure 3 The LRB is presented, which uses an adaptive bucketing mechanism. The LRB groups requests with similar token lengths in two steps. First, it models the empirical distribution of token lengths using online kernel density estimation (KDE). Then, the LRB determines the optimal bucketing boundary by analyzing the properties of the distribution. By minimizing the padding overhead between different sequence lengths, the LRB effectively alleviates the inefficiency problem caused by traditional batch processing methods.

[0049] (3) Length distribution estimation based on KDE The key technology of length distribution estimation lies in analyzing the probability density function (PDF) of token length to identify natural partition points. The LRB maintains a recent observed request length sliding window and continuously estimate the empirical density by online KDE .

[0050] Specifically, the sample is processed by KDE and a continuous empirical density function is constructed:

[0051] where is the Gaussian kernel function, is the bandwidth parameter chosen according to Silverman's rule of thumb.

[0052] (2) Density-guided lexical unit length partitioning The LRB dynamically identifies the optimal partitioning points by detecting regions of rapid change in the density gradient , which naturally correspond to transitions between different request patterns (e.g., short dialog turns vs. long document processing).

[0053] Specifically, candidate partitioning points are determined by locating inflection points in the density estimate, as follows:

[0054] where is a preset gradient magnitude threshold of 0.01, which effectively filters out spurious local fluctuations while preserving major modal transitions. This dual condition captures locations where the distribution curvature is maximized ( ) and the slope is significantly ( ) steep, corresponding to rapid transitions between request length modalities.

[0055] The candidate set is then sorted in descending order to prioritize the most apparent modal boundaries. Finally, the top candidates are selected as the optimal hierarchical breakpoints. These breakpoints are periodically updated to adapt to changes in the workload. Here and denote the observed minimum and maximum lengths (external boundaries), respectively. Consequently, boundary values yield internal element buckets. New requests are assigned to batches according to the interval boundaries .

[0056] For each storage bucket containing requests of length , the empirical mean is computed as a representative length measure:

[0057] In addition, to mitigate the volatility of , a smoothing term, Exponential Weighted Moving Average (EWMA), is introduced:

[0058] where is the average estimation value of the current window , is the last smoothing result, is the parameter.

[0059] 2. Heterogeneity-aware partitioning engine (HPE) Modern GPU clusters exhibit significant heterogeneity in three key dimensions: compute capability, memory capacity, and interconnect bandwidth. This diversity presents challenges to LLM services, as simple grouping of a single type of GPU either wastes premium resources or exacerbates the underdog effect. To address this issue, HPE adopts a two-stage optimization scheme. In the first stage, it configures LLM instance configurations by reasonably combining GPUs of different tiers, balancing between maximizing overall utilization and alleviating bottlenecks caused by weaker devices. In the second stage, it partitions LLM layers within each instance, aligning workloads with heterogeneous GPU combinations while considering both compute capability and memory capacity. This design ensures that heterogeneous resources are effectively utilized rather than marginalized.

[0060] (1) Problem formalization The GPU allocation problem for a heterogeneous cluster containing device types is formalized as , where each tuple represents a GPU type and its available quantity. Given a parallel configuration , where , , and denote the parallelism of data, tensor, and pipeline, respectively. Allowing one or more components of to be zero indicates that the corresponding parallelization strategy is not applied. The goal is to determine the number of LLM instances and their GPU composition to achieve performance targets by prioritizing high-throughput configurations.

[0061] This problem inherently involves combinatorial optimization, as each instance requires a specific GPU arrangement to suit its parallelization strategy. For example, TP typically requires allocated GPUs to be located within the same node and of the same type to ensure high-bandwidth interconnects within the node; while DP allows for more flexible resource allocation across different GPU types while considering load balancing. PP often imposes constraints on computational power at each stage, making the allocation process more complex. Given a set of mixed GPUs and the multidimensional nature of parallel configurations, this problem is computationally challenging and NP-hard, similar to the generalized multi-resource scheduling problem. Therefore, a heuristic approach is proposed, prioritizing feasible resource allocations.

[0062] (2) Heuristic Instance Configuration HPE employs an adaptive LLM instance configuration strategy to maximize resource utilization under heterogeneous hardware constraints, which is reflected in two aspects. First, GPU selection strictly follows the requirements of the chosen parallelization strategy, ensuring that each instance is feasible and efficient under TP, PP, or DP. Second, this strategy aims to minimize resource fragmentation, thereby avoiding excessively low GPU utilization and improving overall throughput.

[0063] Figure 4 The pseudocode demonstrates the heuristic instance configuration process, which is divided into two main phases.

[0064] In step 1, a given resource pool and parallel configuration The algorithm first generates candidate configurations by grouping GPUs in a manner that satisfies parallelism constraints. (Line 2). For each candidate According to LLM requirements To ensure feasibility, among which This can be estimated as the sum of parameter storage, intermediate activation, and key-value caching for attention. Then, feasible candidate solutions are evaluated using a performance score, as shown in the formula:

[0065] This formula balances normalized FLOPs and memory capacity (lines 3-8), where It is a weighting factor. Candidate solutions are sorted in descending order to ensure that solutions with higher configurations are given priority (line 9).

[0066] In step 2, the algorithm determines each candidate solution. The number of instances that can be instantiated. For each instantiated instance The algorithm records the data assigned to each pipeline stage. GPU type and quantity, that is Get and Get And due to the bandwidth limitations of TP, each PP stage uses only the same type of GPU on a single node. Then... From available pool Remove from the pool. This process is repeated until the DP requirement is met or no more instances are available (lines 10-17). If there are still GPUs remaining after meeting the required DP instances, more instances can be generated by expanding PP or TP (lines 18-22). Since each candidate instance is scored and sorted once, the computational complexity of this algorithm is O(n log n). In fact, The size depends on the chosen parallel strategy The limitations are that, due to the moderate nature of typical parallel configurations (e.g., small TP or PP orders), the number of candidate nodes remains limited, making the overall overhead negligible.

[0067] (3) An asymmetric partitioning strategy suitable for GPU heterogeneity Heterogeneous GPU clusters face challenges in symmetric partitioning due to inherent differences in computing power, memory size, and interconnect bandwidth. To overcome these limitations, an asymmetric partitioning strategy is introduced that dynamically adapts to hardware heterogeneity while maintaining parallel execution efficiency. This strategy follows two fundamental principles. First, TensorP requires high-bandwidth inter-GPU communication for efficient partitioning and synchronization of tensor operations. To minimize communication latency and avoid cross-node bottlenecks, all GPUs within a tensor parallel group are restricted to the same node. This ensures full utilization of high-speed interconnects such as NVLink, thereby maximizing computational throughput. Second, TensorP inherently supports hardware heterogeneity by adjusting the computational load in stages. Layer allocation in pipeline stages follows a proportional distribution relative to the computing power of the allocated GPUs, ensuring that faster devices are allocated more layers proportionally, thus reducing idle latency (i.e., mitigating pipeline bubbles).

[0068] Figure 5 This demonstrates an asymmetric partitioning strategy that deploys the LLM across different GPU sets through two hierarchical steps. First, it configures the pipeline stages (PPs), and then forms TP groups within each stage. This hierarchical design conforms to the coarse-to-fine architecture of model parallelism, where the PPs determine which layers are executed together, while the TPs determine how computations within a stage are shared among the GPUs. In step 1, for each pipeline stage... The GPU group discussed in the algorithm is represented as ,in It is a GPU type. It is a stage The number of such GPUs. Then, the computation stage. Total video memory capacity ,Right now The sum of the video memory capacity of all GPUs in the system. In step 2, given a... Each layer has a video memory requirement of [number] layers. LLM The algorithm will scan each layer sequentially. It maintains a layer memory accumulator, which increments whenever the accumulated demand approaches the capacity. When a split point is reached, it is recorded. For all Repeat this operation in each stage to obtain the set of split points. At each stage, the workload is... The algorithm distributes resources evenly across the GPUs to perform tensor parallelism. The time complexity of this algorithm is O(log n). ,in This is the total number of GPU types. This refers to the number of LLM layers.

[0069] 3. Binary Matching Scheduler (BMS) To bridge the gap between LRB and HPE, BMS was introduced. BMS is a dynamic request scheduling mechanism that optimally routes incoming requests to the most suitable LLM instance based on the request token length. The BMS process consists of two steps: first, mapping request buckets to LLM instances; second, determining the batch size for each instance.

[0070] (1) Allocation strategy for storage buckets and instances LRB Construction Request buckets The average token length per bucket is And adopt real-time workload distribution ,in yes The arrival rate of requests. In addition, HPE deployed a set of LLM instances. Each instance has an overall capacity profile. ,like Figure 6 As shown. The main challenge facing BMS is the difference between different requests and different LLM instances. If misallocated, short requests may waste resources on high-end GPUs, while long requests may be limited by weaker instances, resulting in SLO violations.

[0071] As Figure 6 shown, the left partition in the bipartite graph corresponds to the request buckets , each with a weight of , where is the arrival rate of , and is its average token length. The right partition corresponds to the deployed LLM instances , each with a weight of , where denotes the total FLOP of all GPUs in instance , obtained by profiling each instance with different sequence lengths and averaging the measured computation time for each generated token , denotes its average computation time per token. The edge weight represents the expected processing efficiency of assigning a request bucket to an instance , as follows:

[0072] where estimates the end-to-end communication overhead of assigning a request to an instance, is a tunable trade-off parameter, is a smoothing constant to prevent the denominator from being zero, ensuring the stability of numerical calculations. These values follow general settings to balance computation efficiency and communication cost. In addition, BMS enforces the capacity limit of each instance during scheduling, thereby achieving load balancing across heterogeneous instances.

[0073] Figure 7 The pseudocode of BMS is shown, and the time complexity of BMS is mainly determined by the weighted bipartite matching step, which builds a graph in time and solves the minimum cost flow (or assignment) problem in time, which is practical because and are usually small in inference workloads.

[0074] (2) Adaptive batching mechanism In each selected instance, the batching strategy needs to adapt to different request lengths and memory limits. An adaptive batching mechanism is introduced, which determines the batching size of each storage bucket according to its average request length and the available memory of the target instance Since request lengths often exhibit a long-tailed distribution, containing a large number of short requests and a small number of long sequences with large computation, the batching mechanism dynamically adjusts to the current resource limit and fluctuating request pattern. In implementation, the batch size is jointly determined by the available GPU memory of the allocated instance and the average request length . This dynamic batching mechanism provides three main benefits: (1) Reducing padding by uniform-length batching; (2) Stabilizing resource utilization by EWMA length estimation; (3) Inherent adaptability to changing workload patterns.

[0075] The application further provides an asymmetric partitioning scheduling method in a heterogeneous GPU cluster, comprising: Step S1: checking the characteristics of incoming requests, and grouping into different request buckets according to token length; Step S2: in a heterogeneous GPU cluster environment, using a double-layer strategy to optimize large language model inference instances; Step S3: calculating the matching degree of the request bucket and the large language model inference instance, and scheduling different request buckets to the large language model inference instance with the highest matching degree according to the calculation result.

[0076] Those skilled in the art know that, in addition to implementing the system provided by the application and each device, module, unit thereof in a pure computer readable program code manner, the system provided by the application and each device, module, unit thereof can also be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. by logically programming the method steps to achieve the same function. Therefore, the system provided by the application and each device, module, unit thereof can be considered as a hardware component, and the devices, modules, units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, units for implementing various functions can also be considered as both software modules implementing methods and structures within hardware components.

[0077] The specific embodiments of the application are described above. It should be understood that the application is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the application. In the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. An asymmetric partitioning scheduling system in a heterogeneous GPU cluster, the system comprising: include: The length-aware request bucket is used to examine the characteristics of incoming requests and group them into different request buckets based on the token length. The heterogeneous perceptual partitioning engine is used to optimize large language model inference instances in a heterogeneous GPU cluster environment using a two-layer strategy. The binary matching scheduler calculates the matching degree between request buckets and large language model inference instances, and schedules different request buckets to the large language model inference instance with the highest matching degree based on the calculation results.

2. The system of claim 1, wherein, The length-aware request bucketing machine uses online kernel density estimation to model the empirical distribution of token lengths and determines the optimal bucketing boundary by analyzing the properties of the distribution.

3. The system of claim 2, wherein, Online kernel density estimation includes: causing the length-aware request bucketizer to maintain a sliding window of the most recently observed request lengths and continually estimate the empirical density ; To the sample Constructing a continuous empirical density function: wherein, is a Gaussian kernel function, is a bandwidth parameter chosen according to Silverman's rule of thumb; Length-aware request bucketizer by detecting density gradient dynamically identify the optimal split point by locating the inflection point in the density estimate; determine candidate split points by locating the inflection point in the density estimate: wherein is a preset gradient magnitude threshold value; Candidate set according to Sort in descending order, select the first The candidate set is used as the optimal hierarchical breakpoint; breakpoint regularly updated, and min and max length observed, respectively, boundary value generation internal element buckets, new requests will be assigned to batches according to interval boundaries allocation to batches ; For each bucket of requests containing files of length the empirical mean is computed as a representative length measure: ​ 。 4. The system of claim 1, wherein, The two-layer strategy includes: In the first phase, GPUs of different levels are combined to configure large language model instances; In the second phase, the large language model layer within each instance is partitioned to keep the workload consistent with the heterogeneous GPU combination, while taking into account both computing power and memory capacity.

5. The asymmetric partitioning scheduling system in a heterogeneous GPU cluster according to claim 4, characterized in that, A device type is included The GPU allocation problem for a heterogeneous cluster of device types is formalized as: where each tuple represents a GPU type and its available number; Given a parallel configuration: wherein , and represent the parallelism of data, tensor and pipeline, respectively.

6. The system of claim 1, wherein, The binary matching scheduler maps request buckets to large language model instances and determines the batch size for each instance.

7. A method for asymmetric partition scheduling in a heterogeneous GPU cluster, the method comprising: include: Step S1: Examine the characteristics of the incoming requests and group them into different request buckets based on the token length; Step S2: In a heterogeneous GPU cluster environment, a two-layer strategy is adopted to optimize large language model inference instances; Step S3: Calculate the matching degree between the request bucket and the large language model inference instance, and schedule different request buckets to the large language model inference instance with the highest matching degree according to the calculation results.

8. The method of asymmetric partitioning scheduling in a heterogeneous GPU cluster of claim 7, wherein, Step S1 includes modeling the empirical distribution of token length using online kernel density estimation and determining the optimal bucket boundary by analyzing the properties of the distribution.

9. The method of asymmetric partitioning scheduling in a heterogeneous GPU cluster of claim 8, wherein, Online kernel density estimation includes: causing the length-aware request bucketizer to maintain a sliding window of the most recently observed request lengths and continually estimate the empirical density ; To the sample Constructing a continuous empirical density function: wherein, is a Gaussian kernel function, is a bandwidth parameter chosen according to Silverman's rule of thumb; Length-aware request bucketizer by detecting density gradient dynamically identify the optimal split point by locating the inflection point in the density estimate; determine candidate split points by locating the inflection point in the density estimate: wherein is a preset gradient magnitude threshold value; Candidate set according to Sort in descending order, select the first The candidate set is used as the optimal hierarchical breakpoint; breakpoint regularly updated, and denote the minimum and maximum length observed, respectively, boundary value generation internal element buckets, new requests will be assigned to batches according to interval boundaries assigned to batches ; For each bucket of requests containing files of length the empirical mean is computed as a representative length measure: ​ 。 10. The method of asymmetric partitioning scheduling in a heterogeneous GPU cluster of claim 7, wherein, The two-layer strategy includes: In the first phase, GPUs of different levels are combined to configure large language model instances; In the second phase, the large language model layer within each instance is partitioned to keep the workload consistent with the heterogeneous GPU combination, while taking into account both computing power and memory capacity.

Images