A GPU / NPU heterogeneous resource pooling scheduling method for large model inference

By constructing a computing power pool abstraction layer to uniformly address GPUs and NPUs, dynamically translating instructions and optimizing communication, the problem of static binding being unable to adapt to differences in computing power types and instruction incompatibility in large model inference is solved, thereby improving resource utilization and throughput.

CN123068060APending Publication Date: 2026-11-10SHANDONG DACHENG ZHONGKE IND CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610912384.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-06-24
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

In existing technologies, static binding cannot dynamically adapt to the different computing power requirements of different stages of large model inference. Instructions are incompatible between heterogeneous devices, and cross-device communication and parallel scheduling are inefficient, resulting in insufficient resource utilization and inference throughput of GPUs and NPUs.

Method used

An abstraction layer for the computing power pool is constructed to unify the addressing of GPUs and NPUs, provide device-independent interfaces, dynamically translate computing instructions, maintain address mapping, adopt a hybrid parallel strategy, schedule the NPU to perform the pre-filling stage to generate key-value caches, the GPU to perform the decoding stage, and optimize cross-device communication through PCIe P2P direct memory access.

Benefits of technology

It improves the utilization rate of heterogeneous computing resources and the throughput of large model inference, enhances execution efficiency, solves the problems of static binding being unable to adapt dynamically and instruction incompatibility, and optimizes cross-device communication latency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123068060A_ABST
    Figure CN123068060A_ABST
Patent Text Reader

Abstract

This invention discloses a GPU / NPU heterogeneous resource pooling scheduling method for large model inference, belonging to the field of resource scheduling technology. The method includes: constructing a computing power pool abstraction layer, uniformly addressing GPUs and NPUs, providing device-independent interfaces, and dynamically translating instructions and maintaining a unified address mapping; for large model inference, pre-filling is performed by the NPU, and decoding is performed by the GPU with superior memory bandwidth, employing a hybrid parallel strategy of inter-layer parallelism and intra-layer tensor parallelism; during scheduling, GPUs are allocated to reduce cross-domain communication latency, and adjacent layers are allocated according to inter-layer parallelism; after completion, data is transferred from the NPU to the GPU memory. This invention solves the technical problems in existing technologies, such as the inability of static binding to dynamically adapt to the differentiated computing power requirements of different stages of large model inference, the incompatibility of instructions between heterogeneous devices, and the low efficiency of cross-device communication and parallel scheduling. It achieves the technical effects of improving the utilization rate of heterogeneous computing power resources, the throughput of large model inference, and the execution efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of resource scheduling technology, and in particular to a GPU / NPU heterogeneous resource pooling scheduling method for large model inference. Background Technology

[0002] Current mainstream large-scale model inference typically employs the Transformer architecture, whose computation process can be decomposed into two key stages: the prefill stage and the decode stage. The prefill stage is primarily a computationally intensive task, requiring parallel processing of all tokens in the input sequence to generate a key-value cache (KV Cache). The decode stage, on the other hand, generates output token by token, making it a memory-intensive task, highly sensitive to GPU memory bandwidth and memory access latency. In existing AI servers, GPUs (Graphics Processing Units) and NPUs (Neural Processing Units) are usually deployed with a fixed ratio to CPUs in a static hardware binding manner. For example, a server may be configured with a fixed number of GPUs and a fixed number of NPUs. These are independent at the hardware level and bound to specific computational tasks. This statically bound architecture struggles to adapt to the diverse computational needs of different models. For instance, some models experience high computational pressure during the prefill stage, while others have memory-intensive decoding stages. A fixed ratio cannot achieve dynamic, on-demand allocation of computational resources. Furthermore, GPUs and NPUs differ significantly in their instruction sets and programming models, making it impossible for traditional inference frameworks to transparently schedule heterogeneous computational resources, resulting in low development efficiency and difficulty in guaranteeing performance. Furthermore, large model inference is extremely sensitive to communication latency. When using hybrid parallel strategies such as tensor parallelism, the exchange of intermediate results across devices is frequent. The PCIe topology affects the bandwidth and latency of cross-device communication, reducing inference throughput.

[0003] At present, the relevant technologies have technical problems such as static binding being unable to dynamically adapt to the different computing power requirements of different stages of large model inference, as well as the incompatibility of instructions between heterogeneous devices and the low efficiency of cross-device communication and parallel scheduling. Summary of the Invention

[0004] This application provides a GPU / NPU heterogeneous resource pooling scheduling method for large model inference, which solves the technical problems in the prior art where static binding cannot dynamically adapt to the different computing power requirements of different stages of large model inference, as well as the incompatibility of instructions between heterogeneous devices and the low efficiency of cross-device communication and parallel scheduling. It achieves the technical effect of improving the utilization rate of heterogeneous computing power resources, the throughput of large model inference, and the execution efficiency.

[0005] This application provides a GPU / NPU heterogeneous resource pooling scheduling method for large-scale model inference. The method includes: constructing a computing power pool abstraction layer, uniformly addressing and registering the GPUs and NPUs within the server node to the same heterogeneous computing power resource pool, with the computing power pool abstraction layer providing a unified device-independent interface to the upper-layer inference framework; intercepting and dynamically translating computation instructions at the driver level of the computing power pool abstraction layer, and maintaining the address mapping relationship between the respective device memory of the GPU and NPU and the unified virtual address space; in response to a large language model inference request, decoupling the inference process into a prefill stage and a decoding stage, wherein the prefill stage schedules the NPUs in the heterogeneous computing power resource pool to perform forward computation of each Transformer layer to generate a key-value cache, and the decoding stage schedules the GPUs with dominant memory bandwidth in the heterogeneous computing power resource pool to perform autoregressive token generation, and the decoding stage adopts a hybrid parallelism strategy combining inter-layer parallelism and intra-layer tensor parallelism; querying the PCIe topology information of the server motherboard to identify the PCIe of each GPU and NPU. The Switch domain allocates GPUs participating in tensor parallelism within the same Decode stage layer to the same PCIe Switch domain to reduce cross-domain communication latency. GPUs and NPUs in adjacent Transformer layers are allocated according to their inter-layer parallelism. After the NPU completes the computation of each layer and generates the KV Cache in the Prefill stage, it transfers the KV Cache from the NPU device memory to the GPU memory responsible for the Decode stage through PCIe P2P direct memory access (DMA) enabled between the GPU and NPU.

[0006] In a possible implementation, the driver-level interception and dynamic translation of computation instructions in the computing pool abstraction layer includes: mapping CUDA operator calls from upper-layer applications to CANN operator instructions adapted to the NPU, and translating the CANN operator instructions into equivalent NPU-executable operators by querying a preset operator mapping table according to the target device type; mapping CANN operator calls from upper-layer applications to CUDA operator instructions adapted to the GPU, and translating the CUDA operator instructions into equivalent GPU-executable operators by querying a preset operator mapping table according to the target device type; marking operators without corresponding mapping relationships in the operator mapping table as fallback operators, and reassigning the fallback operators to another type of device for execution.

[0007] In a possible implementation, the Prefill phase schedules the NPUs in the heterogeneous computing resource pool to perform forward computation of each Transformer layer to generate a key-value (KV) cache. This includes: sequentially allocating all Transformer layers of the large language model to one or more NPUs in the heterogeneous computing resource pool as Prefill execution devices; each Prefill execution device sequentially performs forward computation of the allocated Transformer layer, generating a key-value (KV) cache corresponding to each layer, and temporarily storing the key-value (KV) cache in the NPU device memory of the Prefill execution device for subsequent transmission.

[0008] In a possible implementation, the Decode phase schedules GPUs with dominant memory bandwidth in the heterogeneous computing resource pool to perform autoregressive token generation, including: sequentially allocating all Transformer layers of the large language model to one or more GPUs in the heterogeneous computing resource pool as Decode execution devices; each Decode execution device performs multi-head attention computation and feedforward neural network computation layer by layer based on the key-value KV Cache received from the Prefill execution device, generating a decoded output sequence for each token.

[0009] In a possible implementation, the Decode stage employs a hybrid parallel strategy combining inter-layer parallelism and intra-layer tensor parallelism, including: dividing the weights of the same Transformer layer into multiple weight slices according to the hidden dimension, and allocating the multiple weight slices to multiple GPUs within the same PCIe Switch domain for intra-layer tensor parallel computation, so that each GPU holds one of the weight slices and collaboratively completes the forward computation of the Transformer layer; allocating multiple consecutive Transformer layers to different Prefill execution devices or different Decode execution devices in an inter-layer pipeline manner, so that the Prefill execution device in the Prefill stage is executed in a layer-by-layer pipelined manner, and the Decode execution device in the Decode stage is executed in a layer-by-layer pipelined manner, wherein the Prefill stage and the Decode stage are connected through the KV Cache to form a heterogeneous pipeline.

[0010] In a possible implementation, querying the PCIe topology information of the server motherboard and identifying the PCIe switch domain to which each GPU and NPU belongs includes: obtaining the PCI bus number and NUMA node information of each GPU and NPU through the operating system bus enumeration interface, parsing the PCIe bridge device affiliation to determine the uplink PCIe switch physical port of each acceleration device; constructing a mapping table between the device and the PCIe switch domain based on the uplink PCIe switch physical port for the scheduler to query, and allocating GPUs participating in tensor parallelism within the same Decode stage layer to the same PCIe switch domain according to the mapping table.

[0011] In one possible implementation, the KV Cache is transferred from the NPU device memory to the GPU memory, which is responsible for the Decode stage, through PCIe P2P direct memory access (DMA) enabled between the GPU and NPU. This includes: establishing a peer-to-peer memory mapping between the GPU and NPU at the computing power pool abstraction layer, making the physical address of the KV Cache on the NPU side visible to the GPU side, triggering a PCIe P2P DMA transfer operation, and writing the KV Cache directly from the NPU device memory to the GPU memory without going through the host-side CPU memory copy.

[0012] In a possible implementation, the method further includes a dynamic load-aware rescheduling process: periodically collecting the length of the NPU Prefill queue corresponding to the Prefill stage and the length of the GPU Decode queue corresponding to the Decode stage; when the length of the NPU Prefill queue exceeds a first preset threshold and the GPU idle rate corresponding to the GPU Decode queue is higher than a second preset threshold, marking some Transformer layers executed in the Prefill stage as Prefill layers to be migrated, and temporarily scheduling the Prefill layers to be migrated to idle GPUs for execution.

[0013] In a possible implementation, the dynamic load-aware rescheduling process further includes: when the length of the GPU Decode queue exceeds a third preset threshold and the NPU idle rate corresponding to the NPU Prefill queue is higher than a fourth preset threshold, marking some of the Transformer layers executed in the Decode phase as Decode layers to be migrated, and temporarily scheduling the Decode layers to be migrated to idle NPUs for execution; after the rescheduling is completed, restoring the initial allocation relationship between the NPU execution scheduled in the Prefill phase and the GPU execution scheduled in the Decode phase.

[0014] In a possible implementation, the Prefill layer to be migrated is temporarily scheduled to be executed on an idle GPU, or the Decode layer to be migrated is temporarily scheduled to be executed on an idle NPU. This includes: recording the real-time load status of each device in the computing power pool abstraction layer; when a rescheduling condition is triggered, the computing power pool abstraction layer migrates the computing tasks and weight parameters corresponding to the Prefill layer or the Decode layer to be migrated from the original device to the target device, and resumes computing execution on the target device; after the migration is completed, the device task mapping table in the computing power pool abstraction layer is updated.

[0015] This application proposes a GPU / NPU heterogeneous resource pooling scheduling method for large model inference. It constructs a computing power pool abstraction layer, unifies the addressing of GPUs and NPUs, provides device-independent interfaces, and dynamically translates instructions and maintains a unified address mapping. For large model inference, pre-filling is performed by the NPU, while decoding is performed by the GPU with superior memory bandwidth, employing a hybrid parallel strategy of inter-layer parallelism and intra-layer tensor parallelism. During scheduling, GPU allocation reduces cross-domain communication latency, and adjacent layers are allocated according to inter-layer parallelism. After completion, data is transferred from the NPU to the GPU memory. This method solves the technical problems of static binding failing to dynamically adapt to the differentiated computing power requirements at different stages of large model inference, as well as the technical issues of instruction incompatibility between heterogeneous devices and low efficiency in cross-device communication and parallel scheduling. It achieves the technical effects of improving the utilization rate of heterogeneous computing power resources, the throughput of large model inference, and execution efficiency. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. Flowcharts are used in this application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from these processes.

[0017] Figure 1 This is a flowchart illustrating a GPU / NPU heterogeneous resource pooling scheduling method for large model inference, provided in an embodiment of this application.

[0018] Figure 2 This is a flowchart illustrating the translation of computation instructions in a GPU / NPU heterogeneous resource pooling scheduling method for large model inference provided in an embodiment of this application. Detailed Implementation

[0019] To further illustrate the technical means and effects adopted by the present invention in order to achieve the intended purpose, the following detailed description is provided in conjunction with the accompanying drawings and preferred embodiments, based on the specific implementation methods, structures, features and effects of the present invention.

[0020] This application provides a GPU / NPU heterogeneous resource pooling scheduling method for large model inference, such as... Figure 1 As shown, the method includes: Step S100: Construct a computing power pool abstraction layer, uniformly address the graphics processing unit (GPU) and neural processing unit (NPU) within the server node and register them to the same heterogeneous computing power resource pool. The computing power pool abstraction layer provides a unified device-independent interface to the upper-layer inference framework.

[0021] Preferably, within the server node, a logical address range is allocated to each GPU and each NPU. The logical addresses of all GPUs and NPUs are integrated into a single global address mapping table. This table records the correspondence between each logical address range and the physical device and its internal physical memory address. The driver reads the device identifier, memory capacity, computing power, and other attributes of all GPUs and NPUs in the server node and registers them in a resource pool management structure maintained by the computing pool abstraction layer. This management structure stores metadata such as the handle, status (online / offline), and available memory address range for each device in the form of a list or hash table. The computing pool abstraction layer encapsulates the differences between underlying devices and exposes a set of interface functions that do not distinguish between GPUs and NPUs. When the upper-layer inference framework calls this unified interface, the computing pool abstraction layer looks up the address mapping table based on the passed logical address to determine the physical device type and the physical memory offset on that device. Then, the computing pool abstraction layer calls the native interface of the corresponding device driver to complete the actual operation. When allocating device memory, the computing pool abstraction layer selects an unused logical address from the unified address space, binds it to a block of physical memory of the selected physical device, and records the binding relationship in the address mapping table. The upper-layer inference framework always accesses memory through this logical address, and the computing pool abstraction layer performs address translation each time it accesses the memory.

[0022] In step S200, the driver level of the computing power pool abstraction layer intercepts and dynamically translates computing instructions, and maintains the address mapping relationship between the respective device memory of the GPU and NPU and the unified virtual address space.

[0023] Furthermore, such as Figure 2As shown, step S200 further includes step S210, mapping CUDA operator calls from upper-layer applications to CANN operator instructions adapted to NPU, and translating the CANN operator instructions into equivalent NPU-executable operators by querying a preset operator mapping table according to the target device type; step S220, mapping CANN operator calls from upper-layer applications to CUDA operator instructions adapted to GPU, and translating the CUDA operator instructions into equivalent GPU-executable operators by querying a preset operator mapping table according to the target device type; step S230, marking operators that do not have a corresponding mapping relationship in the operator mapping table as fallback operators, and reassigning the fallback operators to another type of device for execution.

[0024] Preferably, the computing power pool abstraction layer driver-level component captures CUDA API calls or CANN API calls directly issued by the upper-layer inference framework or application. When a CUDA operator call is intercepted, but the scheduling decision allocates the operator to the NPU for execution, the computing power pool abstraction layer parses the name, parameter signature, and kernel function attributes of the CUDA operator, and uses this information as a key to query a pre-built CUDA to CANN operator mapping table. Each row in this mapping table records the equivalent correspondence between a CUDA operator and a CANN operator. When a CANN operator call is intercepted, but the scheduling decision allocates the operator to the GPU for execution, the computing power pool abstraction layer parses the name, parameter signature, and kernel function attributes of the CANN operator, and uses this information as a key to query a pre-built CANN to CUDA operator mapping table.

[0025] Preferably, for the CUDA-to-NPU mapping, the corresponding CANN operator name and parameter conversion rules are retrieved from the mapping table. The parameters of the original CUDA operator, such as tensor shape, data type, stride, and reduction dimension, are mapped to the parameter format required by the CANN operator according to the conversion rules, generating a complete CANN operator call instruction sequence. This instruction sequence can be executed directly on the NPU, and the computation result is equivalent to that of the original CUDA operator. For the CANN-to-GPU mapping, the corresponding CUDA operator name and parameter conversion rules are retrieved from the mapping table. The parameters of the original CANN operator are converted to the parameter format required by the CUDA operator, generating a complete CUDA kernel function startup instruction. This instruction can be executed on the GPU, and the computation result is equivalent.

[0026] Preferably, when querying the operator mapping table, if no record matching the original operator is found, for example, if a CUDA operator does not have a corresponding CANN operator in the mapping table, the computing power pool abstraction layer marks the operator as a "rollback operator" and records its original call context, including the operator name, parameters, device memory address, etc. For CUDA operators marked as rollback operators, the computing power pool abstraction layer cancels the allocation to the NPU and directly sends the reserved original CUDA call to the currently idle GPU for execution. For CANN operators marked as rollback operators, the computing power pool abstraction layer directly sends the reserved original CANN call to the currently idle NPU for execution. No instruction translation or conversion is performed during the dispatch process.

[0027] Preferably, the computing power pool abstraction layer creates a global virtual address pool on the host side. This address pool covers the entire 64-bit virtual address range, is not bound to any physical device memory, and only serves as a logical address space referenced by the upper-layer inference framework. For the physical memory on each GPU device, the computing power pool abstraction layer selects an unoccupied continuous virtual address range from the unified virtual address space and records the starting address of this virtual address range, along with the GPU device identifier, the physical memory base address on the GPU device, and the range length, in an address mapping table. For the physical memory on each NPU device, the computing power pool abstraction layer selects an unoccupied continuous virtual address range from the unified virtual address space and records the starting address of this virtual address range, along with the NPU device identifier, the physical memory base address on the NPU device, and the range length, in the same address mapping table. This mapping table uses the same data structure as the mapping table used by the GPU. In the address mapping table, each record contains four fields: the unified virtual address start value, length, device type (GPU or NPU), and the device's internal physical address start value. The computing pool abstraction layer continuously maintains this table during runtime, adding records when new device memory is allocated and deleting corresponding records when device memory is released.

[0028] In step S300, in response to the large language model inference request, the inference process is decoupled into a prefill stage and a decoding stage. In the prefill stage, the NPU in the heterogeneous computing power resource pool is scheduled to perform forward computation of each Transformer layer to generate a key-value cache. In the decoding stage, the GPU with dominant video memory bandwidth in the heterogeneous computing power resource pool is scheduled to perform autoregressive token generation. The decoding stage adopts a hybrid parallelism strategy that combines inter-layer parallelism and intra-layer tensor parallelism.

[0029] Preferably, the computing power pool abstraction layer or the upper-layer inference framework receives a complete large language model inference request. This request includes an input token sequence, such as a token ID sequence obtained after user-input text is segmented, model weight parameters, and generation parameters, such as maximum generation length and temperature coefficient. The computing power pool abstraction layer parses the inference request and divides the entire inference process into two sequentially executed stages. The Prefill stage processes all tokens in the input token sequence, executes all layers of the Transformer model, calculates key vectors and value vectors for each token position in the input sequence, and generates a key-value cache (KV Cache). This stage is characterized by high computational load, high parallelism, and the computational load is determined by the number of input tokens. The Decode stage generates an output sequence token by token based on the KV Cache generated in the Prefill stage. Each time a token is generated, it is added to the input sequence, the KV Cache is updated, and the prediction of the next token continues until the maximum generation length is reached or a terminator is encountered. This stage is characterized by processing only one token at a time, memory access intensive, and frequent KV Cache read operations. The computing pool abstraction layer does not submit the Prefill and Decode phases as consecutive computing tasks to the same device. Instead, it encapsulates the Prefill phase as the first independent computing task and the Decode phase as the second independent computing task. The two tasks are connected through a KV cache as a data dependency relationship, rather than through an internal device synchronization mechanism.

[0030] Furthermore, step S300 also includes sequentially allocating all Transformer layers of the large language model to one or more NPUs in the heterogeneous computing power resource pool as Prefill execution devices; each Prefill execution device sequentially executes the forward computation of the allocated Transformer layer, generates the corresponding key-value KV Cache for each layer, and temporarily stores the key-value KV Cache in the NPU device memory of the Prefill execution device for subsequent transmission.

[0031] Preferably, the computing power pool abstraction layer loads the large language model configuration file, parses the model structure, and obtains a sequential list of all Transformer layers in the model. Each layer contains a multi-head attention module and a feedforward neural network module, and there is a sequential dependency between layers, that is, the output of the i-th layer is the input of the i+1-th layer. The computing power pool abstraction layer queries the status of all NPU devices in the heterogeneous computing power resource pool, obtains the device identifier, available memory capacity, current load, and other attributes of each NPU, and filters out NPUs with an "online" status and sufficient free memory to accommodate the storage required for at least one Transformer layer computation as candidate Prefill execution devices. Then, the computing power pool abstraction layer performs allocation decisions based on the number of layers, the computational load per layer, the computational power of each NPU, and the memory capacity. If only one NPU is allocated, all Transformer layers (layer 1 to layer N) are sequentially allocated to that NPU, forming a sequential execution chain. If multiple NPUs are allocated, several consecutive layers are divided into a group, and each group is allocated to one NPU. For example, layers 1 to k are allocated to NPU A, and layers k+1 to m are allocated to NPU A. B, and so on, ensure that the dependency between adjacent layers is not broken when dividing, that is, the internal layers of each group are continuous, and the output of the last layer of the previous group can be transmitted and used as the input of the first layer of the next group.

[0032] Preferably, for a set of consecutive layers assigned to a certain NPU, the NPU performs forward computation in ascending order of layer index. Specifically, the input tensor of the i-th layer is input, such as the output of the previous NPU or the original input token sequence of the first layer in the group. The NPU performs matrix multiplication, softmax, and other operations in the i-th layer, and outputs the output tensor of the i-th layer as the input of the (i+1)-th layer. This process is repeated until all layers in the group have been executed. During the forward computation of each Transformer layer, when the multi-head attention module is computed, the NPU performs a linear transformation on the input tensor of the current layer to generate a query matrix Q, a key matrix K, and a value matrix V. The key matrix K is stored as the key cache of the layer, and the value matrix V is stored as the value cache of the layer. Each layer independently stores the K cache and V cache. After generating the corresponding KV cache for each layer, the NPU writes the KV cache data into the physical memory of the current NPU device. The computing power pool abstraction layer is this KV cache. The cache allocates a contiguous device memory region, records the starting address, length, and corresponding layer index of the region, and saves it to the address mapping table. This KV cache is kept in the NPU memory, waiting for the unified PCIe P2P transmission operation after the subsequent Prefill stage is completed.

[0033] Furthermore, step S300 also includes sequentially allocating all Transformer layers of the large language model to one or more GPUs in the heterogeneous computing power resource pool as Decode execution devices; each Decode execution device performs multi-head attention calculation and feedforward neural network calculation layer by layer based on the key-value KV Cache received from the Prefill execution device, and generates a decoding output sequence per token.

[0034] Preferably, the computing power pool abstraction layer loads the large language model configuration file, parses the model structure, and obtains a sequential list of all Transformer layers in the model. This list is the same as the layer sequence used in the Prefill stage. The computing power pool abstraction layer queries the status of all GPU devices in the heterogeneous computing power resource pool, obtains the device identifier, memory capacity, memory bandwidth, current load, and other attributes of each GPU, and filters out GPUs with an "online" status and sufficient memory to accommodate the storage required for at least one Transformer layer computation as candidate Decode execution devices. GPUs with higher memory bandwidth are given priority during the filtering. The computing power pool abstraction layer performs allocation decisions based on the number of layers, the amount of computation per layer, the computing power of each GPU, and the memory capacity. If only one GPU is allocated, all Transformer layers (layer 1 to layer N) are sequentially allocated to that GPU, forming a sequential execution chain. If multiple GPUs are allocated, several consecutive layers are divided into a group, and each group is allocated to one GPU. For example, layers 1 to k are allocated to GPU A, and layers k+1 to m are allocated to GPU A. B, and so on, ensure that the dependency between adjacent layers is not broken when dividing, that is, the internal layers of each group are continuous, and the output tensor of the last layer of the previous group is used as the input of the first layer of the next group after transmission.

[0035] Preferably, before the Decode phase starts, the Prefill phase has generated the KV Cache for each of the N layers and written the KV Cache from the NPU device memory to the GPU memory of each GPU via PCIe P2P DMA transfer. The computing power pool abstraction layer records the storage location of each KV Cache in the GPU memory in the address mapping table. Before each GPU starts computing, it obtains the memory address of the KV Cache corresponding to the layer it is responsible for from the computing power pool abstraction layer. For a set of consecutive layers allocated to a GPU, the GPU performs the Decode phase computing in ascending order of layer index. For each layer i, the GPU reads the Key Cache and Value Cache from the KV Cache corresponding to that layer. The GPU receives the embedding vector of the current Token, performs a dot product calculation between the query vector of the Token and the Key Cache in the KV Cache to obtain the attention score, and then performs a weighted summation with the Value Cache to obtain the output of the multi-head attention module.

[0036] Preferably, after completing the multi-head attention calculation for the current layer i, the GPU continues to execute the feedforward neural network calculation for the same layer. The GPU inputs the multi-head attention output tensor into the first fully connected layer of the feedforward neural network, performs matrix multiplication and activation functions, and then inputs the output of the first fully connected layer into the second fully connected layer, performs matrix multiplication, and the tensor output by the feedforward neural network becomes the final output of the current layer i. After the calculation of all Transformer layers, the GPU sends the output tensor of the last layer into the output layer, the output layer calculates the probability distribution of each word, and the GPU selects the word with the highest probability as the output of the current token and appends it to the decoding sequence. If the maximum generation length is not reached and no terminator is encountered, the embedding vector of the token is used as the input for the next round of the Decode stage, and the process is repeated to generate the next token. If the maximum generation length is reached or a terminator is encountered, the Decode stage ends.

[0037] Furthermore, step S300 also includes: dividing the weights of the same Transformer layer into multiple weight slices according to the hidden dimension, and allocating the multiple weight slices to multiple GPUs in the same PCIe Switch domain for intra-layer tensor parallel computation, so that each GPU holds one of the weight slices and collaboratively completes the forward computation of the Transformer layer; allocating multiple consecutive Transformer layers to different Prefill execution devices or different Decode execution devices in an inter-layer pipeline manner, so that the Prefill execution device executes in a layer-by-layer pipeline in the Prefill stage, and the Decode execution device executes in a layer-by-layer pipeline in the Decode stage, wherein the Prefill stage and the Decode stage are connected through the KV Cache to form a heterogeneous pipeline.

[0038] Preferably, the computing power pool abstraction layer loads the weight parameters of a Transformer layer in the large language model, including the query linear transformation weight matrix, key linear transformation weight matrix, value linear transformation weight matrix, and output linear transformation weight matrix in the multi-head attention module, as well as the weight matrices of the first and second fully connected layers in the feedforward neural network module. For each weight matrix, it is segmented along the hidden dimension to obtain multiple weight slices. The computing power pool abstraction layer queries the server motherboard PCIe topology information, identifies the uplink PCIe Switch physical port of each GPU, and groups all GPUs connected to the same PCIe Switch physical port into the same PCIe Switch domain. Multiple GPUs are selected from this domain as execution devices for tensor parallelism within the layer, and the segmented weight slices are then transferred to the memory of the selected GPUs respectively, with each GPU storing only one slice. For one forward computation of the Transformer layer, the input tensor is sent to all GPUs in a block manner. Each GPU uses its own weight slice and input tensor to perform partial computation. The GPUs exchange output matrices through set communication operations, and finally, they are concatenated along the hidden dimension to obtain the complete output matrix.

[0039] Preferably, the computing power pool abstraction layer divides all Transformer layers of the large language model into multiple consecutive layer groups in sequence. Each layer group contains several consecutive layers, called a pipeline stage. For example, stage 0 contains layers 1 to k, stage 1 contains layers k+1 to m, and so on. The number of divisions is equal to the number of available execution devices. Stage 0 is assigned to NPU A, stage 1 is assigned to NPU B, and so on. In the Prefill stage, the computing power pool abstraction layer assigns each pipeline stage to a different NPU. After the Prefill stage starts, NPU A starts executing stage 0, completes all forward computations of this stage, generates the KV Cache corresponding to stage 0, and temporarily stores it in the memory of NPU A. After stage 0 completes some layers, NPU A transmits the output tensor of the last layer of stage 0 to NPU B. NPU B immediately starts executing stage 1, which is carried out simultaneously with NPU A continuing to execute subsequent micro-batches. Multiple NPUs work in parallel in a pipeline manner. At the same time, NPU A processes the t-th micro-batch, and NPU B processes the t-1-th micro-batch.

[0040] Preferably, during the Decode phase, the computing power pool abstraction layer assigns each pipeline stage to a different GPU: stage 0 to GPU A, stage 1 to GPU B, and so on. After the Decode phase starts, during the token-by-token generation process, for each token, GPU A executes stage 0, reads the corresponding KV Cache, performs multi-head attention and feedforward neural network computations, and transmits the output tensor to GPU B. GPU B then executes stage 1, simultaneously with other stages in the token generation process. Multiple GPUs process the computations of each layer of the same token in parallel using a pipelined approach. After the Prefill phase is fully completed, the computing power pool abstraction layer transfers the KV Caches of all layers from the memory of each NPU device to the video memory of the corresponding GPU. The Prefill phase is the first stage of the overall pipeline, and the Decode phase is the second stage. The two stages are connected through a unique dependency relationship of KV Caches. After the Prefill phase processes the entire input sequence, it triggers the KV Cache transfer. After the transfer is completed, the Decode phase starts, processing token by token. The two stages achieve parallelism between heterogeneous devices through a task-level pipeline.

[0041] Step S400: Query the PCIe topology information of the server motherboard, identify the PCIe Switch domain to which each GPU and NPU belongs, allocate GPUs participating in the same Decode stage layer tensor parallelism to the same PCIe Switch domain to reduce cross-domain communication latency, and allocate GPUs and NPUs in adjacent Transformer layers according to the inter-layer parallelism relationship.

[0042] Step S400 further includes obtaining the PCI bus number and NUMA node information of each GPU and NPU through the operating system bus enumeration interface, parsing the PCIe bridge device affiliation to determine the uplink PCIe Switch physical port of each acceleration device; constructing a mapping table between the device and the PCIe Switch domain based on the uplink PCIe Switch physical port for the scheduler to query, and allocating GPUs participating in tensor parallelism within the same Decode stage layer to the same PCIe Switch domain according to the mapping table.

[0043] Preferably, the computing power pool abstraction layer calls the PCI bus enumeration function provided by the operating system. By reading device entries, it returns a list of all PCI devices in the server node. It then traverses the PCI device list, reads the category code and vendor ID of each device, and records the PCI bus number for each selected device in the format "domain:bus:device:function" (e.g., 0000:01:00.0). For each selected GPU or NPU device, the computing power pool abstraction layer reads the NUMA node information, indicating the NUMA node number to which the device is connected. If -1 is returned, it indicates that the device is not bound to a specific NUMA node.

[0044] Preferably, the computing power pool abstraction layer starts from the root controller and recursively reads the type field in the configuration space of each PCI device. If the type field is a PCIe bridge device, it enters the downstream port of that bridge and continues to traverse. For each GPU or NPU device, the computing power pool abstraction layer reads the upstream bus number field from its PCI configuration space, traces back towards the root based on this field, and sequentially passes through the configuration space of each bridge device until it reaches the PCIe Switch device, recording the port number of the last bridge device in the path. After identifying the PCIe Switch, the computing power pool abstraction layer reads the downstream port number connected to the GPU or NPU device from the configuration space of the PCIe Switch. This is usually the value in the port register, which is the upstream PCIe Switch physical port of the acceleration device. Based on the uplink PCIe Switch physical port, a mapping table between devices and PCIe Switch domains is constructed for the scheduler to query. Specifically, the computing pool abstraction layer collects the uplink PCIe Switch physical port numbers of all GPU and NPU devices. For each different port number, a unique domain identifier is assigned. All devices connected to the same physical port are grouped into the same PCIe Switch domain. The computing pool abstraction layer creates a mapping table in memory. This table is a key-value pair structure, where the key is the device identifier and the value is a combination of the domain identifier and the uplink port number. For each GPU and NPU device, the computing pool abstraction layer inserts a record into the mapping table and provides a query interface to the upper-layer scheduler. When the scheduler allocates devices, it calls this interface, passes in the device identifier, and the interface returns the PCIe Switch domain identifier to which the device belongs.

[0045] Preferably, GPUs participating in intra-layer tensor parallelism in the same Decode stage are allocated to the same PCIe Switch domain according to the mapping table. Specifically, the computing power pool abstraction layer reads the configuration parameters of the large language model to obtain the intra-layer tensor parallelism, i.e., the number of weight slices after the same Transformer layer is divided, and the corresponding number of GPUs need to hold one slice respectively; it iterates through all available GPUs in the heterogeneous computing power resource pool, and for each GPU, calls the mapping table query interface to obtain its domain identifier, and groups all GPUs according to the domain identifier, with the GPUs in each group connected to the same PCIe Switch physical port; then it checks the number of available GPUs in each domain group, selects the first group with the number of available GPUs greater than or equal to the parallelism, and arbitrarily selects multiple GPUs from this group. If the number of available GPUs in all groups is less than the parallelism, the parallelism is reduced or the device is waited for release; finally, the selected multiple GPUs are marked as execution devices for intra-layer tensor parallelism, and their device identifiers are passed to the operator executor for allocating weight slices and initiating set communication, thereby enabling mutual communication in intra-layer tensor parallel computation. Since they are in the same PCIe... Switch domains have lower latency because their communication paths do not involve a round trip from the uplink port to the root controller.

[0046] In step S500, after the NPU completes the calculations of each layer and generates the KV Cache during the Prefill stage, the KV Cache is transferred from the NPU device memory to the GPU memory responsible for the Decode stage through the PCIe P2P direct memory access DMA enabled between the GPU and the NPU.

[0047] Step S500 further includes establishing a peer-to-peer memory mapping between the GPU and NPU in the computing power pool abstraction layer, making the physical address of the KV Cache on the NPU side visible to the GPU side, triggering a PCIe P2P DMA transfer operation, and writing the KV Cache directly from the NPU device memory to the GPU video memory without going through the host-side CPU memory copy.

[0048] Preferably, after the NPU completes the calculations at each layer and generates the KV Cache during the Prefill phase, the computing power pool abstraction layer queries the NPU driver for the physical address start value, length, and the bus number and device number of the NPU device on the PCIe bus of the KV Cache memory region. The computing power pool abstraction layer calls the peer-to-peer memory mapping interface provided by the GPU driver, passing in the PCIe bus number and device number of the target NPU. The GPU driver establishes a mapping record in the GPU's memory management unit, marking the physical address range of the NPU-side KV Cache as remotely accessible. Then, the computing power pool abstraction layer simultaneously calls the interface provided by the NPU driver, passing in the PCIe bus number and device number of the GPU initiating the access. The NPU driver sets access permissions in the NPU's memory management unit, allowing PCIe read requests from that GPU to access the specified KV Cache address range in the NPU device's memory. The computing power pool abstraction layer records the GPU-NPU peer-to-peer mapping relationship in the address mapping table, which contains the physical address range of the NPU memory, the corresponding GPU virtual address range, and the mapping status flag.

[0049] Preferably, after enabling peer-to-peer mapping, the GPU driver generates an address handle that can be directly referenced by the GPU. The computing pool abstraction layer saves this address handle and associates it with the original physical address of the NPU-side KV Cache. The computing pool abstraction layer constructs a descriptor structure containing the GPU-visible address handle, the KV Cache length, and the Transformer layer index corresponding to the KV Cache. This structure is then passed to the GPU executing the Decode stage via the GPU's command queue or memory-mapped register. When the GPU executes a kernel function or the DMA engine receives this address handle, the GPU's memory management unit recognizes that the address points to remote NPU memory. The GPU hardware resolves this address into the target bus number and physical address in the PCIe transaction layer data packet, without triggering a page fault or access error.

[0050] Preferably, before the Decode phase begins, the computing power pool abstraction layer allocates a receive buffer on the GPU to store the KV Cache to be transmitted. The GPU driver constructs a DMA transfer control block, which includes the GPU-visible address handle of the NPU-side KV Cache, the physical address of the receive buffer in the GPU's local video memory, the total number of bytes in the KV Cache, and the PCIe device read direction. Then, the DMA transfer control block is submitted to the DMA engine on the GPU. The DMA engine is triggered to start execution by writing to the GPU's command doorbell register. The GPU DMA engine generates a PCIe memory read request transaction layer data packet based on the source address in the DMA control block. The destination address field is filled with the physical address of the NPU-side KV Cache, and the destination bus number field is filled with the NPU's PCIe bus number. The GPU DMA engine sends the data packet to the PCIe link. Then, it is directly routed to the downlink port where the NPU is located via the PCIe Switch, without passing through the host memory controller in the Root Complex, and without being processed by the host CPU's cache system. During the entire transmission process, the number of read and write operations on the host memory is zero, and no temporary buffer is requested from the host memory when scheduling the transmission. The transmission command is executed autonomously by the GPU DMA engine, and the CPU is only responsible for submitting the DMA control block and does not participate in data transfer.

[0051] Preferably, the KV Cache is written directly from the NPU device memory to the GPU memory. Specifically, when a PCIe read request from the GPU DMA engine arrives at the NPU, the NPU's PCIe endpoint receives the request. The NPU's internal memory controller reads the KV Cache data from the NPU device memory according to the physical address in the request, packages the read KV Cache data into a PCIe completion data packet, and sends it back to the requesting GPU via the PCIe link. This packet contains the original KV Cache byte sequence read from the NPU memory. The GPU's PCIe endpoint receives the returned data packet, and the GPU DMA engine writes its payload to the target address specified in the DMA control block. This address points to the receive buffer in the GPU's local memory. After the write is complete, the KV Cache is completely moved from the NPU device memory to the GPU memory. After the transfer is complete, the GPU DMA engine sends an interrupt signal to the host or updates the completion flag. After the computing power pool abstraction layer detects the transfer completion event, it updates the status field in the address mapping table to mark that the KV Cache is available, and finally starts the execution of the Decode stage, thereby ensuring zero-copy transfer of the KV Cache.

[0052] Furthermore, the method also includes a dynamic load-aware rescheduling process: periodically collecting the NPU Prefill queue length corresponding to the Prefill stage and the GPU Decode queue length corresponding to the Decode stage; when the NPU Prefill queue length exceeds a first preset threshold and the GPU idle rate corresponding to the GPU Decode queue is higher than a second preset threshold, marking some Transformer layers executed in the Prefill stage as Prefill layers to be migrated, and temporarily scheduling the Prefill layers to be migrated to idle GPUs for execution.

[0053] Preferably, the computing power pool abstraction layer starts a timer thread to perform a collection operation at a fixed time interval (e.g., every 100 milliseconds), calls the interface function provided by the NPU driver, passes in the command queue handle dedicated to the Prefill stage, queries the internal task queue of each NPU device driver, and the driver interface returns the number of computing tasks currently waiting to be executed in the queue, thus determining the length of the NPU Prefill queue. Similarly, the length of the GPU Decode queue is determined by querying. During initialization, the computing power pool abstraction layer reads the configuration file to obtain a first preset threshold and a second preset threshold. The first preset threshold is the maximum allowed length of the NPU Prefill queue (e.g., 10 tasks), and the second preset threshold is the minimum allowed value of GPU idle rate (e.g., 0.7, indicating 70% idle). The computing power pool abstraction layer reads the latest NPU Prefill queue length value from the collected results and performs a numerical comparison. If the NPU Prefill queue length exceeds the first preset threshold, the computing power pool abstraction layer reads the latest GPUDecode queue length value and obtains the maximum concurrency capability of the GPU, i.e., the upper limit of the number of Decode tasks that the GPU can process at the same time. The idle rate is calculated as 1 - (current queue length / maximum concurrency capability). If there are multiple GPUs, the idle rate of each GPU is calculated separately or the average idle rate of all GPUs is calculated.

[0054] Preferably, the GPU idle rate corresponding to the GPU Decode queue is compared with the second preset threshold execution data. If it is higher than the second preset threshold, some Transformer layers executed in the Prefill stage are marked as Prefill layers to be migrated. Specifically, the computing power pool abstraction layer calculates the number of layers to be migrated based on the extent to which the current NPU Prefill queue length exceeds the threshold. The migration number = min(number of tasks exceeding the threshold / number of tasks that the idle GPU can accommodate). For example, if the queue length is 15, the threshold is 10, and there are 5 tasks exceeding the threshold, then the migration number is 5. Then, the list of currently queued tasks in the NPU Prefill queue is read, and tasks at the end of the queue or layers with smaller computational load are selected first. The first N tasks (N equals the migration number) are taken out from the queue. Each selected task contains a series of consecutive Transformer layer indices. A flag bit is set in the control block of each selected task with the value "Prefill layer to be migrated". At the same time, the original target device and task size are recorded in the control block and will be logically removed from the NPU Prefill queue.

[0055] Preferably, the Prefill layer to be migrated is temporarily scheduled to be executed on an idle GPU. Specifically, the computing power pool abstraction layer queries the status of all GPUs in the heterogeneous computing power resource pool, filters out GPUs with an idle rate higher than a second preset threshold, and selects the device with the lowest load and sufficient video memory to accommodate the computation of the Prefill layer to be migrated. The selection criteria include the smallest current Decode queue length, the largest available video memory, and the optimal PCIe topology distance. The target device field in the task control block marked "Prefill layer to be migrated" is changed from the original NPU to the selected GPU. At the same time, a record is added to the device task mapping table of the computing power pool abstraction layer to record the GPU identifier to which the task is currently migrated. Then, the task submission interface of the GPU driver is called to submit the computation kernel function and parameters corresponding to the Prefill layer to the command queue of the target GPU. The GPU's native CUDA instructions are used instead of being translated. The computing power pool abstraction layer needs to translate the original Prefill computation logic into equivalent operators that can be executed by the GPU. The target GPU executes the migrated Prefill layer computation, completes the forward propagation of each Transformer layer, and generates KV Cache layer by layer, which is stored in the target GPU's local video memory. The computing power pool abstraction layer updates the storage location record of the KV Cache in the address mapping table, changing the mapping entry that originally pointed to the NPU device memory to point to the GPU video memory. At the same time, it records that the source device type of the KV Cache is GPU. When the Decode phase starts, the computing power pool abstraction layer directly uses the KV Cache in the GPU's local video memory. The Decode phase tasks continue to be executed using the originally allocated GPU.

[0056] Furthermore, the method also includes, when the length of the GPU Decode queue exceeds a third preset threshold and the NPU idle rate corresponding to the NPU Refill queue is higher than a fourth preset threshold, marking some of the Transformer layers executed in the Decode phase as Decode layers to be migrated, and temporarily scheduling the Decode layers to be migrated to idle NPUs for execution; after the rescheduling is completed, restoring the initial allocation relationship between the NPU execution scheduled in the Prefill phase and the GPU execution scheduled in the Decode phase.

[0057] Preferably, during initialization, the computing pool abstraction layer reads the configuration file and obtains two preset values. The third preset threshold is the maximum allowed length of the GPU Decode queue (e.g., 8 tasks), and the fourth preset threshold is the minimum allowed NPU idle rate (e.g., 0.8, indicating 80% idle). The computing pool abstraction layer reads the latest GPU Decode queue length value from periodically collected results and performs a numerical comparison. If the GPU Decode queue length exceeds the third preset threshold, it reads the latest NPU prefill queue length value and simultaneously obtains the maximum concurrency capability of the NPU, i.e., the upper limit of the number of prefill tasks the NPU can process simultaneously. It calculates the idle rate as 1 - (current queue length / maximum concurrency capability). If multiple NPUs exist, the idle rate of each NPU is calculated separately, or the average idle rate of all NPUs is calculated. Then, the NPU idle rate is compared with the fourth preset threshold. If the NPU idle rate exceeds the fourth preset threshold, some Transformer layers executed during the Decode stage are marked as Decode layers to be migrated. Specifically, based on the current GPU... If the length of the decode queue exceeds a threshold, calculate the number of layers that need to be migrated. The number of migrations is calculated as min(number of tasks exceeding the threshold / number of tasks that the idle NPU can accommodate). For example, if the GPU queue length is 12 and the threshold is 8, and there are 4 tasks exceeding the threshold, then the number of migrations is 4. Read the list of currently queued tasks in the GPU decode queue and prioritize tasks at the end of the queue or layers with lower computational load and lower memory access bandwidth requirements. Take out the first N tasks (N equals the number of migrations). Each selected task contains multiple consecutive Transformer layer indices and the corresponding KV Cache read locations. The computing power pool abstraction layer sets a flag in the control block of each selected task with the value "Decode layer to be migrated". At the same time, the control block records the original target device, the storage location of the KV Cache on which the decode layer depends in the GPU memory, and the task size.

[0058] Furthermore, the method also includes recording the real-time load status of each device in the computing power pool abstraction layer. When a rescheduling condition is triggered, the computing power pool abstraction layer migrates the computing tasks and weight parameters corresponding to the Prefill layer or Decode layer to be migrated from the original device to the target device, and resumes computing execution on the target device. After the migration is completed, the device task mapping table in the computing power pool abstraction layer is updated.

[0059] Preferably, the status of all NPUs in the heterogeneous computing power resource pool is queried, and NPUs with idle rates higher than the fourth preset threshold are selected, i.e., devices with the smallest current Prefill queue length, the largest available memory, and the optimal PCIe topology distance. The target device field in the task control block marked "Decode layer to be migrated" is changed from the original GPU to the selected NPU. At the same time, a record is added to the device task mapping table of the computing power pool abstraction layer, recording the NPU identifier to which the task is currently migrated. Then, the address mapping table is queried to obtain the physical address and length of the KV Cache required by the Decode layer in the GPU video memory. A peer-to-peer memory mapping is established between the NPU and the GPU, enabling the NPU to read the KV in the GPU video memory via PCIe P2P. The process involves: first, transferring the KVCache from GPU memory to NPU device memory; then, calling the task submission interface of the NPU driver to submit the computation kernel function and parameters corresponding to the Decode layer to the command queue of the target NPU, translating the original Decode computation logic into an equivalent operator executable by the NPU; next, processing layer by layer and finally generating the output probability distribution of the next Token, with the NPU selecting the word with the highest probability from the distribution as the output of the current Token; finally, updating the Decode context state of the inference request in the computing power pool abstraction layer, recording the current step number, the generated Token sequence, and the updated KVCache, while updating the storage location record of the KVCache in the address mapping table, changing the mapping entry that originally pointed to GPU memory to point to NPU device memory.

[0060] Preferably, the computing power pool abstraction layer continuously monitors the lengths of the NPU Prefill queue and the GPU Decode queue. If the NPU Prefill queue length is less than or equal to a first preset threshold and the GPU Decode queue length is less than or equal to a third preset threshold, a recovery process is triggered, pausing the allocation of new Decode tasks to the temporarily migrated NPU. The computing power pool abstraction layer queries the completion status of the Decode tasks executed on the migrated NPU. For migration tasks that have not yet been completed, it waits for them to finish. If the KV Cache is updated and stored in the NPU device memory during the execution of the migration task, the computing power pool abstraction layer will update the KV Cache after the task is completed. The cache transfers data from the NPU device memory back to the GPU memory, then iterates through the device task mapping table, searching for all task records marked "migrated." For each record, the target device field in the task control block is modified from the temporarily allocated NPU back to the initially allocated GPU, and the "Decode layer to be migrated" flag in the task control block is cleared. Then, all temporary records related to this migration are deleted from the device task mapping table, restoring the table to its pre-migration state. The computing pool abstraction layer reverts to the default scheduling policy, meaning that newly arriving Prefill stage tasks are scheduled to be executed on the NPU, and Decode stage tasks are scheduled to be executed on the GPU. The computing pool abstraction layer continues to periodically collect queue length and idle rate data, repeating the entire load awareness and rescheduling process.

[0061] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A GPU / NPU heterogeneous resource pooling scheduling method for large model inference, characterized in that, The method includes: A computing power pool abstraction layer is constructed to uniformly address and register the graphics processing unit (GPU) and neural processing unit (NPU) in the server node to the same heterogeneous computing power resource pool. The computing power pool abstraction layer provides a unified device-independent interface to the upper-layer inference framework. The driver-level intercepts and dynamically translates computation instructions in the computing power pool abstraction layer, and maintains the address mapping relationship between the device memory of the GPU and NPU and the unified virtual address space. In response to the large language model inference request, the inference process is decoupled into a prefill stage and a decoding stage. In the prefill stage, the NPU in the heterogeneous computing power resource pool is scheduled to perform forward computation of each Transformer layer to generate key-value cache. In the decoding stage, the GPU with dominant video memory bandwidth in the heterogeneous computing power resource pool is scheduled to perform autoregressive token generation. The decoding stage adopts a hybrid parallelism strategy that combines inter-layer parallelism and intra-layer tensor parallelism. Query the PCIe topology information of the server motherboard, identify the PCIe switch domain to which each GPU and NPU belongs, allocate GPUs participating in the same Decode stage layer tensor parallelism to the same PCIe switch domain to reduce cross-domain communication latency, and allocate GPUs and NPUs in adjacent Transformer layers according to the inter-layer parallelism relationship. After the NPU completes the calculations for each layer and generates the KV Cache during the Prefill phase, it transfers the KV Cache from the NPU device memory to the GPU memory responsible for the Decode phase through PCIe P2P direct memory access (DMA) enabled between the GPU and the NPU.

2. The GPU / NPU heterogeneous resource pooling scheduling method for large model inference as described in claim 1, characterized in that, The driver level of the computing power pool abstraction layer intercepts and dynamically translates computing instructions, including: CUDA operator calls originating from upper-layer applications are mapped to CANN operator instructions adapted to the NPU, and the CANN operator instructions are translated into equivalent operators executable by the NPU by querying a preset operator mapping table according to the target device type. CANN operator calls originating from upper-layer applications are mapped to CUDA operator instructions adapted to the GPU, and the CUDA operator instructions are translated into equivalent GPU-executable operators by querying a preset operator mapping table according to the target device type. Operators that do not have a corresponding mapping relationship in the operator mapping table are marked as backoff operators, and the backoff operators are reassigned to another type of device for execution.

3. The GPU / NPU heterogeneous resource pooling scheduling method for large model inference as described in claim 1, characterized in that, The Prefill phase schedules the NPUs in the heterogeneous computing resource pool to perform forward computation of each Transformer layer to generate a key-value cache, including: All Transformer layers of the large language model are sequentially assigned to one or more NPUs in the heterogeneous computing power resource pool as Prefill execution devices. Each of the Prefill execution devices sequentially performs the forward computation of the allocated Transformer layer, generates the corresponding key-value cache for each layer, and temporarily stores the key-value cache in the NPU device memory of the Prefill execution device for subsequent transmission.

4. The GPU / NPU heterogeneous resource pooling scheduling method for large model inference as described in claim 3, characterized in that, The Decode phase schedules GPUs with dominant memory bandwidth in the heterogeneous computing power resource pool to perform autoregressive token generation, including: All Transformer layers of the large language model are sequentially allocated to one or more GPUs in the heterogeneous computing power resource pool as Decode execution devices. Each of the Decode execution devices performs multi-head attention computation and feedforward neural network computation layer by layer based on the key-value KV Cache received from the Prefill execution device, generating a decoded output sequence token by token.

5. The GPU / NPU heterogeneous resource pooling scheduling method for large model inference as described in claim 4, characterized in that, The Decode phase employs a hybrid parallelism strategy combining inter-layer parallelism and intra-layer tensor parallelism, including: The weights of the same Transformer layer are divided into multiple weight slices according to the hidden dimension, and the multiple weight slices are allocated to multiple GPUs in the same PCIe Switch domain for intra-layer tensor parallel computation, so that each GPU holds one of the weight slices and works together to complete the forward computation of the Transformer layer. Multiple consecutive Transformer layers are assigned to different Prefill execution devices or different Decode execution devices in an inter-layer pipeline manner, so that the Prefill execution device in the Prefill stage is executed in a layer-by-layer pipeline, and the Decode execution device in the Decode stage is executed in a layer-by-layer pipeline. The Prefill stage and the Decode stage are connected through the KV Cache to form a heterogeneous pipeline.

6. The GPU / NPU heterogeneous resource pooling scheduling method for large model inference as described in claim 1, characterized in that, Query the server motherboard's PCIe topology information to identify the PCIe switch domain to which each GPU and NPU belongs, including: The PCI bus number and NUMA node information of each GPU and NPU are obtained through the operating system bus enumeration interface, and the PCIe bridge device affiliation is analyzed to determine the uplink PCIe switch physical port of each acceleration device. The scheduler will query a mapping table between the device and the PCIe Switch domain based on the uplink PCIe Switch physical port and allocate GPUs participating in tensor parallelism within the same Decode stage layer to the same PCIe Switch domain according to the mapping table.

7. The GPU / NPU heterogeneous resource pooling scheduling method for large model inference as described in claim 1, characterized in that, The KV Cache is transferred from the NPU device memory to the GPU video memory, which is responsible for the Decode stage, via PCIe P2P direct memory access (DMA) enabled between the GPU and NPU, including: A peer-to-peer memory mapping is established between the GPU and NPU in the computing power pool abstraction layer, making the physical address of the KV Cache on the NPU side visible to the GPU side, triggering a PCIe P2P DMA transfer operation, and writing the KV Cache directly from the NPU device memory to the GPU video memory without going through the host-side CPU memory copy.

8. The GPU / NPU heterogeneous resource pooling scheduling method for large model inference as described in claim 1, characterized in that, The method also includes a dynamic load-aware rescheduling process: The length of the NPU Prefill queue corresponding to the Prefill stage and the length of the GPU Decode queue corresponding to the Decode stage are periodically collected. When the length of the NPU Prefill queue exceeds the first preset threshold and the GPU idle rate corresponding to the GPU Decode queue is higher than the second preset threshold, some Transformer layers executed in the Prefill stage are marked as Prefill layers to be migrated, and the Prefill layers to be migrated are temporarily scheduled to be executed on idle GPUs.

9. The GPU / NPU heterogeneous resource pooling scheduling method for large model inference as described in claim 8, characterized in that, The dynamic load-aware rescheduling process also includes: When the length of the GPU Decode queue exceeds the third preset threshold and the NPU idle rate corresponding to the NPU Prefill queue is higher than the fourth preset threshold, some Transformer layers executed in the Decode stage are marked as Decode layers to be migrated, and the Decode layers to be migrated are temporarily scheduled to be executed on an idle NPU. After rescheduling is completed, the initial allocation relationship between the NPU execution scheduled in the Prefill phase and the GPU execution scheduled in the Decode phase is restored.

10. The GPU / NPU heterogeneous resource pooling scheduling method for large model inference as described in claim 9, characterized in that, Temporarily scheduling the Prefill layer to be migrated to an idle GPU for execution, or temporarily scheduling the Decode layer to be migrated to an idle NPU for execution, includes: The real-time load status of each device is recorded in the computing power pool abstraction layer. When a rescheduling condition is triggered, the computing power pool abstraction layer migrates the computing tasks and weight parameters corresponding to the Prefill layer or Decode layer to be migrated from the original device to the target device, and resumes computing execution on the target device. After the migration is completed, the device task mapping table in the computing power pool abstraction layer is updated.