A heterogeneous cluster distributed inference collaboration method and system based on VLLM

Abstract

The application discloses a VLLM-based heterogeneous cluster distributed reasoning cooperation method and system, and the method comprises the following steps: S1, generating a heterogeneous device performance image signal through a performance probe arranged in each computing unit in the cluster; S2, performing non-uniform task division according to the heterogeneous device performance image signal and combining a calculation graph structure of a to-be-reasoned model to form an optimal task scheduling strategy signal for a heterogeneous environment; S3, generating a task allocation and loading instruction signal based on the optimal task scheduling strategy signal; S4, coordinating each heterogeneous computing unit to perform reasoning calculation in a distributed reasoning execution process to generate a distributed cooperative reasoning signal; and S5, integrating and post-processing the distributed cooperative reasoning signal to output a final reasoning result. The VLLM-based heterogeneous cluster distributed reasoning cooperation method and system can solve the problems of low resource utilization and poor reasoning efficiency when a large model is run on a heterogeneous cluster.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and distributed computing technology, specifically to a heterogeneous cluster distributed inference collaboration method and system based on VLLM. Background Technology

[0002] With the rapid increase in the parameter scale of large language models and other artificial intelligence models, their enormous computational and memory consumption far exceeds the capacity of a single computing device, making distributed inference an inevitable choice. Against this backdrop, inference engines such as VLLM, specifically designed for large model inference, have emerged. Through innovative attention mechanisms and memory management techniques, they significantly improve throughput when handling tasks on homogeneous computing clusters. However, existing technologies are primarily designed and optimized around homogeneous cluster environments, assuming that all computing devices in the cluster are identical or similar in computing power, memory capacity, and communication bandwidth. This idealized assumption is often difficult to hold in real-world production environments. Actual computing clusters are typically built and upgraded gradually, inevitably containing various architectures and models of computing units, such as different generations of graphics processors, dedicated AI processors, and even central processing units, forming a typical heterogeneous computing environment. When inference methods designed for homogeneous clusters are directly applied to such heterogeneous environments, many serious problems are exposed. First, due to a lack of fine-grained perception of device performance differences, task allocation often adopts an even distribution strategy. This leads to high-performance computing units having to wait for low-performance computing units to complete their tasks, resulting in a significant "weakest link" effect. The overall inference speed is dragged down by the slowest device, causing idle and wasted computing resources on high-performance devices. Second, the memory hierarchy among heterogeneous devices is not effectively utilized, and intelligent scheduling between high-speed and low-speed memory cannot be performed based on the hot / cold characteristics of data. Furthermore, static and rigid task scheduling strategies cannot adapt to the complexity of uneven device performance and dynamic load changes in heterogeneous clusters. This not only makes inference latency difficult to predict but also fails to effectively utilize low-cost or existing hardware resources to reduce overall operating costs. Therefore, there is an urgent need for a new method that can deeply perceive the characteristics of heterogeneous clusters and, based on this, perform intelligent task collaboration and resource scheduling to fully unleash the potential computing power of heterogeneous clusters. Summary of the Invention

[0003] In view of the shortcomings of the prior art, the purpose of this invention is to provide a heterogeneous cluster distributed inference collaboration method and system based on VLLM, to solve the problems of low resource utilization and poor inference efficiency when running large models in heterogeneous clusters. This invention dynamically establishes real-time performance profiles for computing units with different performance levels in the cluster, and accordingly performs non-uniform intelligent partitioning and mapping of the model computation graph, allocating the most suitable tasks to the most matched computing units. Simultaneously, it extends the memory management mechanism of the inference engine, constructs a logically unified heterogeneous memory pool, and optimizes cross-device communication synchronization. Finally, it continuously optimizes the scheduling strategy through execution feedback, thereby achieving efficient collaboration of heterogeneous resources and maximizing overall inference performance.

[0004] This invention provides a heterogeneous cluster distributed inference collaboration method based on VLLM, comprising: S1: By using performance probes deployed in each computing unit of the cluster, benchmark test indicators are collected to generate heterogeneous device performance profile signals; S2: Based on the performance profile signal of heterogeneous devices, and combined with the computation graph structure of the model to be inferred, non-uniform task segmentation is performed to form the optimal task scheduling strategy signal for heterogeneous environments. S3: Based on the optimal task scheduling strategy signal, the model operator and KV Cache (key-value cache) memory pages are dynamically allocated to the corresponding heterogeneous computing units, and task allocation and loading instruction signals are generated; S4: During the distributed inference execution process, based on the task allocation and loading instruction signals, the heterogeneous computing units are coordinated to perform inference calculations, and intermediate results are synchronized through the collaborative management mechanism to generate distributed collaborative inference signals. S5: Integrates and post-processes distributed collaborative inference signals, outputs the final inference result, and collects performance data of this inference and feeds it back to the performance profile signal to optimize subsequent scheduling decisions.

[0005] In one embodiment of the present invention, generating a heterogeneous device performance profile signal specifically includes: dynamically updating the performance profile signal by continuously monitoring the real-time load, computing throughput, and memory usage of the computing unit when performing inference tasks; the benchmark test indicators include at least computing power indicators, memory capacity and bandwidth indicators, and intra-node and inter-node communication bandwidth and latency indicators for different precision and different model structures; the dynamically updated performance profile signal provides a real-time and multi-dimensional cluster capability view for subsequent task scheduling.

[0006] In one embodiment of the present invention, forming an optimal task scheduling strategy signal for heterogeneous environments specifically includes: parsing the computation graph structure of the model to be inferred and abstracting it into a data flow graph containing operator dependencies; combining the heterogeneous device performance profile signal, using a graph partitioning algorithm to divide the data flow graph into multiple subgraphs, and allocating appropriate computation units to each subgraph according to the computational characteristics of the computational units and the current load, thereby generating a task mapping scheme aimed at minimizing overall inference latency and communication overhead.

[0007] In one embodiment of the present invention, generating task allocation and loading instruction signals specifically includes: loading different operators or layers of the model into their assigned computing units according to the optimal task scheduling strategy signal; for the model's attention key-value cache, performing non-uniform paging allocation and management based on the memory characteristics of each computing unit, generating specific memory allocation instructions and model parameter loading instructions, ensuring that each heterogeneous computing unit can efficiently and collaboratively undertake the overall inference computing task.

[0008] In one embodiment of the present invention, generating distributed collaborative reasoning signals specifically includes: during the reasoning execution process, coordinating each heterogeneous computing unit to execute its assigned subtasks according to the dependencies of the data flow graph; when the execution process requires cross-device data exchange, starting an efficient serialization and communication library to transmit intermediate computation result tensors, and synchronizing the execution status of each computing unit, thereby ensuring the correctness and consistency of the entire distributed reasoning process.

[0009] In one embodiment of the present invention, outputting the final inference result and collecting performance data feedback specifically includes: after completing an inference request, the system summarizes the final output results from each computing unit, integrates them through a possible post-processing module, and returns them to the user; at the same time, it records the actual execution time, resource consumption, and communication overhead data of this inference task on each computing unit, and inputs these performance data as feedback signals into the performance profile generation process for adaptive adjustment and optimization of future task scheduling strategies.

[0010] In one embodiment of the present invention, the graph partitioning algorithm specifically includes: adopting a load-balanced graph partitioning strategy, which aims to minimize critical path latency and cross-device communication volume, to segment the data flow graph; during partitioning, the topology of the physical connections between computing units is fully considered, and communication-intensive operator subgraphs are preferentially placed on computing unit groups with high-speed internal connections, thereby pre-optimizing communication efficiency during the task partitioning stage.

[0011] In one embodiment of the present invention, non-uniform paging allocation and management of the attention key-value cache of the model specifically includes: establishing a logically unified but physically distributed heterogeneous memory pool, preferentially retaining active attention key-value cache pages on high-speed memory computing units, while swapping inactive or historical attention key-value cache pages out to large-capacity but relatively low-speed memory computing units for storage, and predictively performing data prefetching based on access patterns to balance the needs of memory capacity and access speed.

[0012] In one embodiment of the present invention, activating an efficient serialization and communication library to transmit intermediate computation result tensors specifically includes: dynamically selecting the optimal communication protocol and compression algorithm for intermediate result tensors of different sizes and urgency based on the communication bandwidth and latency indicators contained in the heterogeneous device performance profile signal, and achieving overlap between computation and communication during data transmission to hide some communication latency and improve pipeline parallel efficiency.

[0013] This invention also includes a heterogeneous cluster distributed inference collaborative system based on VLLM, comprising: The parsing module receives the raw communication message file uploaded by the user through the data access module, and uses the document parsing module to extract structured data signals from the message file; The traversal module retrieves structured data signals from the database through the code execution module, runs an anomaly detection algorithm, and uses preset parameter threshold signals and continuous anomaly time window signals to traverse the structured data signals to identify potential anomaly signals. Based on the persistence judgment logic, it filters the potential anomaly signals and generates valid anomaly event signals. The execution module inputs valid abnormal event signals and communication quality statistics signals derived from structured data signals into the large language model service. It generates professional analysis text signals through structured prompt words and automatically generates visualization chart signals based on communication quality statistics signals by calling the chart library through the code execution module. The analysis module integrates professional analytical text signals and visual chart signals into a predefined report template through the report rendering module to generate analytical report signals; The presentation module presents the analysis report signals to the user through the user interface module.

[0014] This invention provides a heterogeneous cluster distributed inference collaboration method and system based on VLLM. It dynamically establishes real-time performance profiles for computing units with different performance levels within the cluster, and accordingly performs non-uniform intelligent partitioning and mapping of the model computation graph, assigning the most suitable tasks to the most matched computing units. Simultaneously, it extends the inference engine's memory management mechanism, constructs a logically unified heterogeneous memory pool, and optimizes cross-device communication synchronization. Finally, it continuously optimizes the scheduling strategy through execution feedback, thereby achieving efficient collaboration of heterogeneous resources and maximizing overall inference performance. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a heterogeneous cluster distributed inference collaboration method based on VLLM; Figure 2 This is a system architecture diagram of a heterogeneous cluster distributed inference collaborative system based on VLLM. Detailed Implementation

[0017] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0018] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0019] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0020] Please see Figure 1-2 The image illustrates a VLLM-based heterogeneous cluster distributed inference collaboration method and system according to the present invention. The VLLM-based heterogeneous cluster distributed inference collaboration method of the present invention includes: S1: Collecting benchmark test indicators by using performance probes deployed in each computing unit of the cluster to generate a heterogeneous device performance profile signal; S2: Based on the heterogeneous device performance profile signal, performing non-uniform task partitioning in conjunction with the computation graph structure of the model to be inferred, forming an optimal task scheduling strategy signal for the heterogeneous environment; S3: Based on the optimal task scheduling strategy signal, dynamically allocating model operators and KV Cache memory pages to the corresponding heterogeneous computing units, generating task allocation and loading instruction signals; S4: During distributed inference execution, coordinating the execution of inference computation by each heterogeneous computing unit according to the task allocation and loading instruction signals, and synchronizing intermediate results through a collaborative management mechanism to generate a distributed collaborative inference signal; S5: Integrating and post-processing the distributed collaborative inference signal, outputting the final inference result, and collecting performance data from this inference and feeding it back to the performance profile signal to optimize subsequent scheduling decisions.

[0021] like Figure 1As shown, this method begins with a comprehensive and dynamic performance evaluation of various computing units in a heterogeneous computing cluster. This step is the foundation for all subsequent intelligent scheduling and collaborative work, and its core lies in generating a performance profile signal that can realistically and in real-time reflect the comprehensive capabilities of each heterogeneous computing unit in the cluster. Specifically, this is achieved through lightweight software probes deployed on each computing node in the cluster. These probes execute a series of carefully designed benchmark programs that simulate the computational, memory access, and communication patterns in typical large language model inference tasks. The collected benchmark metrics are a multi-dimensional set, far exceeding the device's theoretical peak computing power. In the computational dimension, it measures the actual effective computing power of the computing unit when performing calculations of different precisions, such as floating-point 16-bit format and integer 8-bit format, as well as the efficiency in handling tensor operations of different forms. In the memory dimension, it accurately measures the capacity and bandwidth of the computing unit's local memory, such as video memory or cache, which is crucial for determining the model size that can be supported. In terms of communication, it systematically measures the bandwidth and latency characteristics between computing units, whether communicating via high-speed interconnects within the same physical node or via remote direct memory access or Ethernet communication across network nodes, thus mapping the communication topology and performance profile within the cluster. All these metrics are not measured once and then fixed, but are tracked by a continuous monitoring system. This system collects real-time data on the dynamic load of each computing unit, instantaneous fluctuations in computing throughput, and changes in memory usage during inference task execution. This real-time data is continuously input and updated into the initial performance profile, transforming it from a static reference label into a dynamic signal reflecting the current health and available resources of the cluster. This dynamic, multi-dimensional performance profile signal acts like a detailed "capability map" for the entire heterogeneous cluster, accurately identifying where there is strong computing power, where there is ample memory space, and the actual passability of the "roads"—communication links—connecting these computing islands. This provides indispensable, high-quality data input for subsequent refined task planning and resource scheduling. After obtaining accurate performance profiles of heterogeneous devices, the method enters its core decision-making phase, namely, forming optimal task scheduling strategy signals for heterogeneous environments. This process can be compared to a general who knows the strengths and current status of each soldier under his command, formulating a precise battle plan after receiving a combat mission. Here, the "combat mission" is the model to be reasoned. The system first loads and parses the computational graph structure of the model. This computational graph, in the form of a directed graph, clearly defines the dependencies and data flows between all operators within the model, such as matrix multiplication, activation functions, and attention mechanisms.Subsequently, an intelligent scheduler is activated, receiving two key inputs: an abstracted model data flow graph and the dynamic performance profile signal generated in the preceding steps. The scheduler's core mission is to solve the complex optimization problem of "how to break down this massive computation graph and allocate its fragments to the most suitable locations within the heterogeneous cluster." Internally, it runs an advanced graph partitioning algorithm whose objective function is not simply to achieve load uniformity, but rather to minimize the overall inference task completion time. The key to this is balancing the computational load with the resulting cross-device communication overhead. Therefore, the algorithm non-uniformly partitions the data flow graph based on the computational capabilities of each computing unit identified in the performance profile signal. It tends to allocate computationally intensive subgraphs, such as large multilayer perceptron modules, to computing units with strong floating-point computing capabilities; while operations with special memory access characteristics or irregular computational patterns may be assigned to coprocessors with other matching characteristics. Simultaneously, the algorithm pays close attention to the communication boundaries between the partitioned subgraphs. Referencing communication bandwidth and latency data from the performance profile, it prioritizes placing adjacent subgraphs requiring frequent and large-scale data exchange on pairs of computational units with high-speed physical connections and low latency, such as different accelerator cards within the same server. This minimizes data transmission bottlenecks at the source of task partitioning. Ultimately, the output of this complex decision-making process is an optimal task scheduling strategy signal. This signal is a concrete, executable task mapping scheme that explicitly specifies which computational unit in the cluster will execute each part of the computation graph, and how these subtasks should be executed sequentially or in parallel. Essentially, this strategy signal intelligently and optimally binds the abstract model computation graph to the physical, heterogeneous cluster resources.

[0022] Furthermore, once the optimal task scheduling strategy signal is generated, the method enters the execution preparation phase, namely, generating task allocation and loading instruction signals. This step is the key link in transforming the aforementioned "paper plan" into concrete "action instructions." Guided by the scheduling strategy signal, the system initiates a distributed task loading and resource allocation process. Specifically, the control node sends precise instructions to each computing unit specified in the strategy signal. For computing units assigned model computation tasks, the instructions contain the parameters of the model operators or layers they need to load. These parameters are precisely retrieved from shared storage or parameter servers and loaded into their local memory, ensuring that each computing unit clearly understands its role in the entire inference task and the "tools" it needs. More importantly, this step innovatively manages the attention key-value cache, which is crucial in large language model inference. Traditional homogeneous systems typically perform uniform paging in high-speed memory, while in heterogeneous environments, this method constructs a logically unified but physically distributed heterogeneous memory pool. The generated task allocation and loading instruction signals include an intelligent allocation scheme for key-value cache memory pages. This scheme performs non-uniform allocation based on the memory characteristics reflected in the performance profile signals of each computing unit. For example, it prioritizes and retains active, frequently accessed key-value cache pages in the current inference sequence on computing units with high-speed, low-latency memory to ensure the speed of core attention computation. Inactive, historical, or background key-value cache pages can be swapped out and stored in computing units with large but relatively slow memory, such as main memory or even non-volatile memory. This hierarchical storage management based on data access frequency allows limited fast memory resources to be used for the most critical data, thereby expanding the manageable model size and context length overall. Therefore, the generated task allocation and loading instruction signals constitute a comprehensive set of instructions that drives the entire heterogeneous cluster's collaborative action. It ensures that each heterogeneous computing unit is not only assigned the correct computational tasks but also configured with the memory resources needed to efficiently complete those tasks, laying a solid execution foundation for the upcoming distributed collaborative inference.

[0023] In one embodiment of the invention, after completing fine-grained task allocation and resource loading, the method enters the core execution phase, namely, generating distributed collaborative inference signals. This process is the pivotal point for the entire system to transition from static planning to dynamic operation. Its core lies in coordinating numerous heterogeneous computing units, making them function as an organic whole, collaboratively completing complex model inference calculations. Specifically, a collaborative controller in the system issues start execution commands to each computing unit according to a previously determined task mapping scheme. However, a simple "start" command is far from sufficient to cope with the complexity of the heterogeneous environment. The controller must strictly follow the operator dependencies defined in the model data flow graph to orchestrate the entire execution process. This means that when a computing unit completes the computation of its subtask, the intermediate result tensor it generates may be the prerequisite input required for another or more computing units to begin their computation. At this point, the system activates an efficient and intelligent communication layer. This communication layer does not use a fixed communication mode, but dynamically selects the most suitable communication protocol and data serialization format based on the size and urgency of the intermediate result tensor to be transmitted, as well as the specific data on inter-computational communication bandwidth and latency provided in the performance profile signal. For large-scale data transfers, remote direct memory access (RDBMI) technology might be used to reduce latency; while for small-scale synchronization signals, lightweight message passing mechanisms might be employed. More advanced systems strive to overlap computation and communication, where a computational unit transmits some of its already generated results while it is still performing its current computation, thus hiding communication time within computation time—a crucial aspect of highly parallel pipelined execution. Throughout the inference process, the co-controller continuously monitors the execution status of all computational units, ensuring they remain globally synchronized and preventing the entire inference task from stalling due to unexpected delays in any single unit. Therefore, this distributed collaborative inference signal is essentially a dynamic, continuous execution coordination flow that ensures the efficient, orderly, and correct interweaving of computation, communication, and synchronization across heterogeneous clusters, ultimately driving the massive model inference task step by step towards completion.

[0024] like Figure 1As shown, once the distributed collaborative inference process is successfully completed, the method enters a closed-loop phase of output and self-optimization, namely, outputting the final inference result and collecting performance data feedback. This step marks the completion of a single inference request, but more importantly, it provides the system with a valuable opportunity for continuous learning and performance improvement. At the end of the inference computation, the final output tensors generated by each computational unit are aggregated by the system to one or more designated aggregation nodes. These outputs may be an embedding vector of a complete text sequence or a probability distribution after the final linear transformation. The aggregation nodes perform necessary integration and post-processing on these outputs from different computational units, such as performing beam search decoding, probability sampling, or simple tensor concatenation, ultimately generating a unified inference result that meets the user's expectations and returning it to the caller. At the same time, an equally crucial background process is quietly running: a sophisticated monitoring system is comprehensively collecting the massive amounts of performance data generated by this inference task throughout its entire lifecycle. These data cover multiple dimensions, both micro and macro, including but not limited to: the actual precise time consumed by each computing unit to execute its assigned subtasks, fluctuations in the utilization of its computing cores and memory, the total communication overhead generated during execution, and potential cache hit rates or misses. This real-world performance data provides invaluable comparison with the performance profile signal predicted through benchmark testing in step one. The system uses this real-time performance data as feedback signals, systematically inputting it into the performance profile update module. Through an intelligent evaluation and analysis mechanism, the system can determine whether the current performance profile accurately reflects the cluster's real behavior. If a significant and persistent deviation is found between the actual performance of certain computing units and the profile prediction, the system adaptively triggers a profile recalibration process, using the latest measured data to correct and enrich the original performance model. This process establishes a powerful feedback loop, enabling the system to move beyond static reliance on the initial configuration and instead perceive changes caused by cluster load, temperature, and even hardware performance degradation, thereby dynamically optimizing its core decision-making basis—the performance profile signal. This means that each execution of an inference task makes the system more intelligent to some extent, and its ability to schedule heterogeneous clusters will continue to evolve as the runtime increases, thus achieving a qualitative leap from "static scheduling" to "adaptive scheduling".

[0025] Furthermore, this paper delves into the graph partitioning algorithm used in forming the optimal task scheduling strategy signal. Its core essence lies in employing a refined partitioning strategy guided by load balancing and deeply integrated with communication topology awareness. The algorithm's core optimization objective is not a single, local metric, but a complex multi-objective function. Its primary task is to minimize the critical path latency of the entire inference task from start to finish. The critical path refers to the longest execution path in the data flow graph that offers no room for parallel optimization; it directly determines the shortest task completion time. To achieve this objective, the algorithm must simultaneously and meticulously balance computational load and cross-device communication volume. It analyzes performance profile signals to estimate the execution cost of each operator node in the data flow graph on a specific type of computing unit, and estimates the transmission cost of each edge (representing a data flow) on different communication links. Based on this, the algorithm uses advanced heuristics or metaheuristic methods to attempt to divide the computation graph into several subgraphs matching the number of heterogeneous computing units. This division is not equal but non-uniform and tailored. It aims to ensure that the total computational cost of the subgraph allocated to each computing unit is proportional to its own computing power, thereby achieving dynamic load balancing at the computational level. More ingeniously, the algorithm deeply incorporates considerations of the physical connection topology between computing units during partitioning. It identifies communication-intensive regions in the data flow graph, i.e., groups of operators that need to exchange large amounts of tensor data. When partitioning boundaries, it prioritizes placing these communication-intensive operator subgraphs as integrally as possible on internally connected high-speed computing unit groups, such as multiple accelerators sharing the same high-speed bus or computing nodes connected via dedicated interconnect networks. This topology-based affinity scheduling minimizes data traffic traversing relatively low-speed networks (such as cross-rack Ethernet) at the source of task partitioning, confining most communication within high-speed internal channels. Therefore, this graph partitioning process is a complex decision-making process seeking Pareto optimality between computational load balancing and minimizing communication overhead. The task scheduling strategy signal it produces is a blueprint for fundamentally improving the efficiency of distributed inference.

[0026] Specifically, the non-uniform paging allocation and management of the model's attention key-value cache is implemented by constructing and maintaining an intelligent, hierarchical heterogeneous memory pool architecture. During autoregressive inference in large language models, the attention mechanism needs to maintain a continuously growing key-value cache to store historical sequence information, which puts significant pressure on memory capacity. In heterogeneous environments, the memory resources of different computing units vary significantly in capacity, speed, and cost. This method first logically abstracts a unified, contiguous memory address space for the model's attention layer to access, allowing upper-layer applications to disregard the specific physical location of the data. However, at the physical level, this massive key-value cache is dynamically and non-uniformly distributed across the heterogeneous memory resources of the entire cluster. An intelligent memory manager is responsible for executing specific data placement strategies. The core of this strategy is based on data access frequency. Key-value cache pages that are very active in the current generation step and frequently queried by the attention mechanism are preferentially placed by the manager on computing units with ultra-high-speed, low-latency memory, such as the video memory of a high-performance graphics processor, to ensure the speed and responsiveness of core computation when generating each new token. Conversely, for key-value cache pages that have become historical background and are unlikely to be accessed at the current time step, or cache pages that are preloaded to support long contexts but are not currently the focus of generation, the memory manager migrates or swaps them out to computational units with large capacity but relatively slow access speeds, such as the system's main memory or even non-volatile memory extensions. To further optimize performance and avoid significant latency when accessing historical data, the memory manager also integrates a predictive data prefetching mechanism. This mechanism analyzes the model's access patterns, such as the principle of locality of sequence, to predict which cache pages currently stored in slow memory are likely to be reactivated in the next few generation steps. Then, it moves these "pre-warmed" pages from slow storage to fast storage in advance and asynchronously, attempting to hide the data access latency within the computation time. This dynamic data tiering and prefetching mechanism based on access frequency cleverly balances the scarcity of high-speed memory with the abundance of large-capacity low-speed memory, enabling the system to efficiently support the continuous expansion of model size and the continuous growth of context length under limited hardware constraints.

[0027] like Figure 1As shown, efficient transmission of intermediate computation result tensors is crucial for maintaining overall system performance during distributed collaborative inference. The specific practices in this method regarding the initiation of efficient serialization and communication libraries for transmitting these tensors embody a highly context-aware and dynamically adaptive communication optimization philosophy. Instead of employing a static communication strategy, the system constructs an intelligent communication decision layer. When an intermediate result tensor needs to be transmitted, this layer simultaneously receives input information from multiple dimensions: first, the physical properties of the tensor itself, including its data size, data type precision, and the urgency implied by its logical position in the computation graph; second, real-time quality assessment of the communication path from performance profile signals, particularly quantitative data on the currently available communication bandwidth and basic latency between the source and target computation units. Based on this fusion analysis, the system dynamically selects the optimal combination of communication protocol and data compression algorithm from its communication library for this transmission task. For large, high-priority tensors with strong computational path dependencies, if high-speed interconnects are available, a zero-copy remote direct memory access protocol may be directly selected to minimize latency. For smaller control signals or gradient data with less stringent real-time requirements, a highly optimized message passing interface standard based on the Transmission Control Protocol (TCP) can be used. On bandwidth-constrained network links, the system intelligently employs lossless or lossy compression algorithms, making a shrewd trade-off between computational and communication resources, significantly alleviating communication bottlenecks with minimal computational overhead. Even more ingeniously, the system deeply implements the design principle of overlapping computation and communication. Before the computational unit has fully generated all output data, it may initiate an asynchronous data transmission process, sending ready data blocks to the target computational unit's preparation buffer in advance. This operating mode, especially in pipelined parallel execution strategies, effectively hides some or all of the communication time behind the continuous computation time, enabling smoother connections between heterogeneous computational units. This greatly improves the vertical parallel efficiency of the entire inference pipeline and the overall hardware utilization, transforming communication—a traditional performance bottleneck—into an effectively managed background process.

[0028] Furthermore, the system's ability to self-improve and continuously evolve is fundamentally reflected in its sophisticated closed-loop feedback mechanism, which records the actual execution data of each inference task and uses it as a feedback signal. This mechanism establishes a comprehensive and operational performance evaluation system, which not only includes the overall completion time and throughput at the task level, but also delves into the sub-task execution time breakdown of each computing unit, the peak and fluctuation of memory allocation and release, cache hit rate statistics, and the time and data volume consumed by all cross-device communication operations. After each inference task is completed, this system automatically collects this massive amount of fine-grained actual operational data. Subsequently, a core analysis and comparison process is triggered: the system compares these actual measurements with the theoretical execution data predicted based on performance profile signals before task scheduling. This comparison is not a simple difference calculation, but involves judging the patterns and trends of deviations at the statistical significance level. The system has a pre-defined dynamic, learnable threshold judgment logic. When it detects that the actual execution time of a computing unit on a specific type of operator consistently and significantly deviates from the predicted value in its performance profile, or when the actual bandwidth of a communication link is confirmed to be far lower than its benchmark value, it will automatically trigger a recalibration process for the performance profile signal. This process is not a crude replacement of data, but a gradual data fusion with a forgetting factor. For example, it may use algorithms such as sliding window averaging or exponentially weighted moving average to integrate new measured data into the historical profile with appropriate weights. This allows the performance profile to both track long-term drift in cluster performance (such as due to hardware aging or driver updates) and filter out short-term, occasional performance fluctuations. Through this continuous learning mechanism, the system no longer rigidly relies on an initial, potentially outdated performance model. Instead, it can adaptively perceive the performance dynamics within the cluster caused by changes in load, temperature, resource contention, or hardware status. Based on this, it dynamically and accurately adjusts its most fundamental decision-making basis, thereby ensuring that the task scheduling strategy is always based on the most realistic and reliable understanding of the cluster's capabilities, achieving continuous optimization of system efficiency and robust operation.

[0029] like Figure 2As shown, this invention also provides a VLLM-based heterogeneous cluster distributed inference collaborative system, comprising: a data acquisition module, which collects benchmark test indicators by using performance probes deployed in each computing unit of the cluster, and generates a heterogeneous device performance profile signal; a scheduling module, which performs non-uniform task partitioning based on the heterogeneous device performance profile signal and the computation graph structure of the model to be inferred, forming an optimal task scheduling strategy signal for the heterogeneous environment; an execution module, which dynamically allocates model operators and KV Cache memory pages to the corresponding heterogeneous computing units based on the optimal task scheduling strategy signal, generating task allocation and loading instruction signals; an analysis module, which coordinates the execution of inference computation by each heterogeneous computing unit according to the task allocation and loading instruction signals during the distributed inference execution process, and synchronizes intermediate results through a collaborative management mechanism, generating a distributed collaborative inference signal; and an inference module, which integrates and post-processes the distributed collaborative inference signal, outputs the final inference result, and collects the performance data of this inference and feeds it back to the performance profile signal to optimize subsequent scheduling decisions.

[0030] Specifically, when facing a surge of batch inference requests, this method demonstrates an enhanced level of intelligence. Its core lies in a strategy of dynamically batching requests based on performance profile signals from heterogeneous devices. This strategy aims to simultaneously optimize two seemingly contradictory goals: ensuring the response latency and service level agreement (SLA) for individual requests, and maximizing the overall computational throughput of the entire cluster. When multiple inference requests arrive at the system simultaneously, the scheduler does not simply adopt a first-come, first-served or static batching strategy. Instead, it initiates a complex dynamic decision-making process. First, the scheduler examines the real-time load status of all available computing units in the current cluster and their inherent processing capabilities as defined in their performance profiles. Then, it evaluates the requests in the queue, considering their model structure, input sequence length, and potential SLA requirements. Based on this information, the scheduler allocates appropriate, potentially different, local batch sizes to each computing unit in the cluster, and even to different model instances running on the same computing unit. For example, for a high-performance computing unit with powerful computing capabilities that is currently idle, the scheduler might allocate a larger batch size, allowing it to process multiple requests at once, thereby fully utilizing its parallel computing capabilities and improving the unit's individual throughput. For a computing unit with slightly lower performance or already under high load, smaller batches or even processing requests one by one may be allocated to avoid it becoming a system bottleneck and ensure that requests assigned to it are processed in a timely manner. To coordinate these requests processed in different batches on different performance units, the system employs a sophisticated pipelined parallel architecture. It divides the entire inference process into multiple stages, such as preprocessing, model execution, and post-processing, and processes different request batches through these stages like an assembly line. Computationally powerful units handle the computationally intensive model execution stage, while less powerful units or the central processing unit may handle stages such as logic control, data preprocessing, or result packaging. By precisely controlling the processing rhythm and buffer queue depth of each stage in the pipeline, the system ensures smooth pipeline filling between heterogeneous computing units, allowing fast units to avoid waiting for slow units for extended periods, and ensuring that slow units continue to have tasks to perform. This dynamic batch processing and pipeline coordination based on heterogeneous capability awareness ultimately enables the system to push the overall throughput of the heterogeneous cluster to its theoretical limit while meeting the latency expectations of the vast majority of requests, achieving a high level of balance between service quality and resource efficiency.

[0031] This invention discloses a heterogeneous cluster distributed inference collaboration method and system based on VLLM. It dynamically establishes real-time performance profiles for computing units with different performance levels within the cluster, and accordingly performs non-uniform intelligent partitioning and mapping of the model computation graph, allocating the most suitable tasks to the most matched computing units. Simultaneously, it extends the inference engine's memory management mechanism, constructs a logically unified heterogeneous memory pool, and optimizes cross-device communication synchronization. Finally, it continuously optimizes the scheduling strategy through execution feedback, thereby achieving efficient collaboration of heterogeneous resources and maximizing overall inference performance.

[0032] Therefore, the heterogeneous cluster distributed inference collaboration method and system based on VLLM of the present invention can solve the problems of low resource utilization and poor inference efficiency when running large models in heterogeneous clusters.

[0033] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A heterogeneous cluster distributed inference collaboration method based on VLLM, characterized in that, include: S1: By using performance probes deployed in each computing unit of the cluster, benchmark test indicators are collected to generate heterogeneous device performance profile signals; S2: Based on the heterogeneous device performance profile signal, and combined with the computation graph structure of the model to be inferred, non-uniform task segmentation is performed to form an optimal task scheduling strategy signal for heterogeneous environments. S3: Based on the optimal task scheduling strategy signal, dynamically allocate model operators and KV Cache memory pages to the corresponding heterogeneous computing units, and generate task allocation and loading instruction signals; S4: During the distributed inference execution process, based on the task allocation and loading instruction signals, the heterogeneous computing units are coordinated to perform inference calculations, and intermediate results are synchronized through a collaborative management mechanism to generate distributed collaborative inference signals; S5: Integrate and post-process the distributed collaborative inference signals, output the final inference result, and collect the performance data of this inference and feed it back to the performance profile signal to optimize subsequent scheduling decisions.

2. The heterogeneous cluster distributed inference collaboration method based on VLLM according to claim 1, characterized in that, The generation of heterogeneous device performance profile signals specifically includes: dynamically updating the performance profile signals by continuously monitoring the real-time load, computing throughput, and memory usage of the computing units when performing inference tasks; the benchmark test indicators include at least computing power indicators, memory capacity and bandwidth indicators, and intra-node and inter-node communication bandwidth and latency indicators for different precision and different model structures; the dynamically updated performance profile signals provide a real-time and multi-dimensional cluster capability view for subsequent task scheduling.

3. The heterogeneous cluster distributed inference collaboration method based on VLLM according to claim 1, characterized in that, The formation of the optimal task scheduling strategy signal for heterogeneous environments specifically includes: parsing the computation graph structure of the model to be inferred and abstracting it into a data flow graph containing operator dependencies; combining the heterogeneous device performance profile signal, using a graph partitioning algorithm to divide the data flow graph into multiple subgraphs, and allocating appropriate computation units to each subgraph according to the computational characteristics of the computational units and the current load, thereby generating a task mapping scheme aimed at minimizing overall inference latency and communication overhead.

4. The heterogeneous cluster distributed inference collaboration method based on VLLM according to claim 1, characterized in that, The generation of task allocation and loading instruction signals specifically includes: according to the optimal task scheduling strategy signal, loading different operators or layers of the model into their assigned computing units; for the model's attention key-value cache, performing non-uniform paging allocation and management based on the memory characteristics of each computing unit, generating specific memory allocation instructions and model parameter loading instructions, ensuring that each heterogeneous computing unit can efficiently and collaboratively undertake the overall inference computing task.

5. The heterogeneous cluster distributed inference collaboration method based on VLLM according to claim 1, characterized in that, The generation of distributed collaborative reasoning signals specifically includes: during the reasoning execution process, coordinating each heterogeneous computing unit to execute its assigned subtasks according to the dependencies of the data flow graph; when the execution process requires cross-device data exchange, starting an efficient serialization and communication library to transmit intermediate computation result tensors, and synchronizing the execution status of each computing unit, thereby ensuring the correctness and consistency of the entire distributed reasoning process.

6. The heterogeneous cluster distributed inference collaboration method based on VLLM according to claim 1, characterized in that, The process of outputting the final inference result and collecting performance data feedback specifically includes: after completing an inference request, the system summarizes the final output results from each computing unit, integrates them through possible post-processing modules, and returns them to the user; at the same time, it records the actual execution time, resource consumption, and communication overhead data of this inference task on each computing unit, and inputs these performance data as feedback signals into the performance profile generation process to adaptively adjust and optimize future task scheduling strategies.

7. The heterogeneous cluster distributed inference collaboration method based on VLLM according to claim 2, characterized in that, The graph partitioning algorithm specifically includes: adopting a load-balanced graph partitioning strategy, which aims to minimize critical path latency and cross-device communication volume, to segment the data flow graph; during partitioning, the topology of physical connections between computing units is fully considered, and communication-intensive operator subgraphs are preferentially placed on high-speed computing unit groups with internal connections, thereby pre-optimizing communication efficiency during the task partitioning stage.

8. The heterogeneous cluster distributed inference collaboration method based on VLLM according to claim 3, characterized in that, The non-uniform paging allocation and management of the model's attention key-value cache specifically includes: establishing a logically unified but physically distributed heterogeneous memory pool; prioritizing the retention of active attention key-value cache pages on high-speed memory computing units; swapping out inactive or historical attention key-value cache pages to large-capacity but relatively low-speed memory computing units for storage; and predictively prefetching data based on access patterns to balance the needs of memory capacity and access speed.

9. A heterogeneous cluster distributed inference collaboration method based on VLLM according to claim 4, characterized in that, The specific steps of launching an efficient serialization and communication library to transmit intermediate computation result tensors include: dynamically selecting the optimal communication protocol and compression algorithm for intermediate result tensors of different sizes and urgency based on the communication bandwidth and latency indicators contained in the heterogeneous device performance profile signal, and achieving overlap between computation and communication during data transmission to hide some communication latency and improve pipeline parallel efficiency.

10. A system using the VLLM-based heterogeneous cluster distributed inference collaboration method according to any one of claims 1-9, characterized in that, include: The acquisition module collects benchmark test indicators by using performance probes deployed in each computing unit of the cluster, and generates heterogeneous device performance profile signals. The scheduling module performs non-uniform task partitioning based on the heterogeneous device performance profile signal and the computation graph structure of the model to be inferred, forming an optimal task scheduling strategy signal for the heterogeneous environment. The execution module dynamically allocates model operators and KV Cache memory pages to corresponding heterogeneous computing units based on the optimal task scheduling strategy signal, and generates task allocation and loading instruction signals. The analysis module, during the distributed inference execution process, coordinates the execution of inference calculations by each heterogeneous computing unit according to the task allocation and loading instruction signals, and synchronizes intermediate results through a collaborative management mechanism to generate distributed collaborative inference signals; The inference module integrates and post-processes the distributed collaborative inference signals, outputs the final inference result, and collects the performance data of this inference and feeds it back to the performance profile signal to optimize subsequent scheduling decisions.

Images