A heterogeneous large-model inference adaptation and computing power scheduling method and system for domestically produced GPUs

By defining a unified GPU backend interface and load balancing strategy, the high development and maintenance costs of domestically produced GPUs in the financial industry's LLM inference stack were solved, enabling efficient operation and computing power scheduling of domestically produced GPUs, and improving inference performance and system availability.

CN123070313APending Publication Date: 2026-11-10SHANGHAI HENGSHI TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611238757.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-15
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

In existing technologies, the LLM inference stack in the financial industry lacks native support for domestic GPUs, resulting in high development and maintenance costs. The computing power management tools lack unified visualization monitoring and scheduling capabilities, making it difficult to achieve unified interface adaptation and computing power scheduling for various heterogeneous GPUs.

Method used

This paper presents a method for heterogeneous large-scale model inference adaptation and computing power scheduling for domestic GPUs. By defining a unified GPU backend interface, it enables efficient operation of multiple heterogeneous platforms and uses a load balancing strategy and monitoring system for computing power pooling and automatic fault recovery.

Benefits of technology

It has achieved reduced development costs for domestically produced GPUs, optimized inference performance, unified scheduling of heterogeneous computing power, low-overhead real-time monitoring and high availability, support for domestic IT innovation platforms, improved throughput and reduced latency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123070313A_ABST
    Figure CN123070313A_ABST
Patent Text Reader

Abstract

This invention discloses a method and system for heterogeneous large-model inference adaptation and computing power scheduling for domestically produced GPUs. The method includes: defining a unified GPU backend interface comprising five core primitives: memory allocation, host-to-device copying, device-to-host copying, kernel startup, and computation flow synchronization; implementing multiple backend instances based on CUDA, MXN, HALO, and CPU; reusing operator abstractions at the inference engine operator execution layer, replacing only the hardware execution part with the corresponding backend instance, and automatically selecting the optimal backend according to preset priorities; directly mapping quantized weight tensors to device memory; organizing multiple GPU nodes into a computing power pool and routing inference requests based on a load balancing strategy; monitoring and alarming through telemetry data collection via a daemon process; and automatic failover in case of node failure. This invention implements a single API for multiple GPU backends, supporting efficient inference and unified scheduling of large models on domestically produced GPUs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence inference acceleration and heterogeneous computing power management technology. Specifically, it relates to a method, system, and computer-readable storage medium for heterogeneous large model inference adaptation and computing power scheduling for domestically produced GPUs. Background Technology

[0002] With the accelerated adoption of Large Language Models (LLM) in industries such as finance, the demand for inference computing power is growing exponentially. However, existing technologies face the following challenges:

[0003] (1) CUDA ecosystem lock-in and high migration costs: The existing LLM inference stacks in the financial industry (such as llama.cpp, vLLM, TensorRT-LLM) are deeply tied to the NVIDIA CUDA ecosystem. Although llama.cpp supports CPU and Metal backends, it lacks native support for domestic GPUs (MUXI MXN, Pingtouge HALO). Reimplementing the inference backend for each domestic GPU requires porting thousands of lines of CUDA kernels, which is a huge development workload and difficult to maintain.

[0004] (2) Fragmentation of domestic GPU software stacks: Muxi provides MXN-SDK, Pingtouge provides HALO platform, Hygon provides ROCm compatible stack, and Cambricon provides Cambricon Neuware. The programming models, API naming and memory management methods of each SDK are different. Developers need to learn multiple programming frameworks at the same time, resulting in high development and maintenance costs.

[0005] (3) Lack of computing power management tools: The management tools provided by domestic GPU manufacturers (such as MXN-SMI and HALO Monitor) usually only support command line queries and lack unified visual monitoring, alarm and scheduling capabilities. In hybrid deployment scenarios, managing NVIDIA and domestic GPUs at the same time requires switching between multiple sets of tools, and there is a lack of GPU scheduling optimization for LLM inference scenarios.

[0006] Against the backdrop of export controls on high-end NVIDIA GPUs and the accelerated promotion of domestic substitution in the information technology innovation field, how to adapt to various heterogeneous domestic GPUs with a unified interface, achieve "one API, multiple backends" operation for LLM inference, and complete the unified scheduling and monitoring of heterogeneous computing power has become an urgent technical problem to be solved. Summary of the Invention

[0007] The purpose of this invention is to overcome at least one of the above-mentioned defects in the prior art and provide a method, system and storage medium for heterogeneous large model inference adaptation and computing power scheduling for domestic GPUs. This invention enables the efficient operation of large language models on various heterogeneous platforms such as CUDA, MXN, HALO and CPU through a unified GPU backend abstraction, and provides a complete solution for computing power pooling, load balancing and monitoring.

[0008] The first aspect of this invention provides a method for heterogeneous large-model inference adaptation and computing power scheduling for domestically produced GPUs, comprising the following steps:

[0009] Step S1: Define a unified GPU backend interface. The unified GPU backend interface includes at least five core primitives: memory allocation, host-to-device data copy, device-to-host data copy, kernel startup, and compute stream synchronization.

[0010] Step S2: Implement a unified GPU backend interface based on CUDA, MXN, HALO and CPU respectively to obtain multiple backend instances. MXN is the heterogeneous computing programming framework of Muxi GPU, and HALO is the programming framework of Pingtouge Hanguang NPU.

[0011] Step S3: In the operator execution layer of the large model inference engine, the general operator abstraction is mapped to a unified GPU backend interface. At runtime, the available backend instances in the current environment are automatically detected, and the optimal backend instance is selected according to the preset priority to execute the operator operation.

[0012] Step S4: The quantized weight tensor of the large model is directly mapped to the device memory corresponding to the selected backend instance for operator operation.

[0013] Step S5: Organize multiple GPU nodes into a computing power pool, and route inference requests to the target GPU nodes in the computing power pool based on a load balancing strategy.

[0014] Step S6: Collect telemetry data of the GPU devices corresponding to each backend instance through an independent daemon process and output it to the monitoring system to monitor and alert on GPU utilization, memory, temperature, power consumption and inference service indicators.

[0015] Step S7: When a GPU node failure is detected, the pending inference requests are rerouted to healthy nodes, and automatically re-added to the computing power pool after the failed node recovers.

[0016] The second aspect of this invention provides a heterogeneous large model inference adaptation and computing power scheduling system for domestic GPUs, including a backend abstraction layer module, an inference adaptation module, a model management module, an inference task scheduling module, a computing power pool management module, and a computing power monitoring module.

[0017] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] 1. One API, multiple GPU backends: By shielding the differences between underlying GPUs through a unified GPU backend interface, the upper-layer inference code only depends on the unified API. The runtime automatically detects and selects the optimal backend, which greatly reduces the development cost of integrating domestic GPUs (Moxi MXN, Pingtouge HALO) into the LLM inference stack.

[0020] 2. Inference Performance Optimization: In the GGML framework of llama.cpp, only the hardware execution part is replaced, operator abstraction and memory management are reused, and the quantization inference path is kept consistent with the CUDA backend to ensure lossless accuracy; key computations (matrix multiplication, attention, etc.) are accelerated by calling the vendor's high-performance SDK libraries (such as mxblas, HALO NPU operators). llama-3-8B-Q4_K_M achieves an inference speed of 50+ tokens / s (single card) on the Muxi C500.

[0021] 3. Unified scheduling of heterogeneous computing power: computing power pooling + four load balancing strategies (least connections / round-robin / weighted round-robin / memory priority) + dynamic batch processing + priority queue, supporting hot loading and hot switching of multiple models, improving throughput and reducing latency.

[0022] 4. Low-overhead real-time monitoring: The Rust daemon directly collects telemetry data through the vendor's SDK and transmits it to the acquisition end via a lock-free circular buffer in shared memory, with an impact on GPU performance of less than 1%.

[0023] 5. High availability and compatibility with domestic IT innovation: Automatic transfer and recovery of nodes after failure; full support for Kylin V10, Tongxin UOS and ARM64 domestic IT innovation platforms such as Phytium, Kunpeng, and Yitian 710. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0025] Figure 1 A schematic diagram of the overall architecture of a heterogeneous large-model inference adaptation and computing power scheduling system for domestic GPUs provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the architecture of the GPU backend abstraction layer provided in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the inference task scheduling and dynamic batch processing provided in an embodiment of the present invention. Figure 4 This is a flowchart illustrating the computing power monitoring and alarm process provided in an embodiment of the present invention. Figure 5 This is a schematic diagram of the computing power pooling and load balancing process provided in an embodiment of the present invention.

[0026] in, Figure 1 The system architecture shown includes an access layer (FastAPI REST API, WebSocket streaming interface), a scheduling layer (request queue, scheduler, load balancer), an inference layer (llama.cpp inference engine, GPU backend abstraction layer, model manager), a GPU backend layer (CUDA / MXN / HALO / CPU backend), and a monitoring layer (mxmon daemon, Prometheus, Grafana).

[0027] Figure 2 The GPU backend abstraction layer shown includes a unified API layer (GpuBackend interface) and four backend implementations (CUDABackend for NVIDIA A100 / V100 / T4, MXN Backend for Muxi C500 / C series, HALO Backend for Pingtouge Hanguang 800, and CPU Backend for x86_64 / ARM NEON). Each backend calls the underlying SDK (CUDA 12.xToolkit, MXN-SDK 2.x, HALO Platform SDK, and OpenBLAS / ARM Compute Library).

[0028] Figure 3 The scheduling process shown is as follows: Inference requests enter the Redis List request queue → the scheduler retrieves them in batches → grouping by model name → merging the same model into a batch → the GPU executes once → the result is written back to the corresponding request's Future.

[0029] Figure 4 The monitoring process shown is as follows: GPU device → mxmon daemon process (collects telemetry data through the vendor's SDK C interface and writes it to a shared memory circular buffer) → Prometheus → Grafana dashboard. Alarm rules include GPU temperature greater than 85 degrees Celsius, VRAM usage greater than 90%, inference P99 latency greater than 5 seconds, and GPU disconnection.

[0030] Figure 5 The computing power pool process shown is as follows: unified pooling management of multiple GPU nodes → node selection by load balancing strategy → health check → marking as unavailable after 3 consecutive failures → rerouting pending requests → automatic rejoining after node recovery. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0032] System Overall Architecture

[0033] Reference Figure 1 The heterogeneous large-model inference adaptation and computing power scheduling system for domestically produced GPUs in this embodiment includes:

[0034] Access Layer: FastAPI REST API and WebSocket streaming interface, receiving inference requests and supporting SSE streaming output; Scheduling Layer: Request queue based on Redis List, Scheduler, and LoadBalancer; Inference Layer: llama.cpp inference engine, GPU backend abstraction layer, and ModelManager; GPU Backend Layer: Four backend instances: CUDA, MXN, HALO, and CPU; Monitoring Layer: mxmon daemon, Prometheus, and Grafana.

[0035] Unified GPU backend interface definition (Step S1)

[0036] Reference Figure 2 The unified GPU backend interface (GpuBackend) defines the following core primitives:

[0037] Interface methods illustrate parameter allocate Allocate video memory on the GPU byte count copy_host_to_device Host → GPU data copy Host pointer + Device pointer + Size copy_device_to_host GPU → Host Data Copy Device pointer + host pointer + size launch_kernel Start the GPU computing kernel kernel name + parameters synchronize Synchronous GPU computation flow - get_device_info Obtain device information -

[0038] The interface is implemented using a dual-track approach: Rust Trait (Rust component side) and C++ pure virtual class (llama.cpp side). The two layers exchange data and synchronize states through the C FFI boundary. The Rust side is responsible for device discovery, resource querying, and monitoring data collection, while the C++ side is responsible for high-performance inference computation.

[0039] Implementation and automatic detection of multiple backend instances (steps S2 and S3)

[0040] Implement four backend instances respectively:

[0041] CUDA Backend: Adapted for NVIDIA A100 / V100 / A10 / T4, dependent on CUDA 12.x Toolkit, suitable for x86_64 architecture; MXN Backend (ggml-mxn.cpp): Adapted for Muxi C500 / C200 / C100, dependent on MXN-SDK 2.x, rewriting the CUDA kernel (matrix multiplication, attention calculation, LayerNorm, RoPE, etc.) with MXN-SDK heterogeneous computing primitives, using mxblas for matrix multiplication acceleration, and using mxAlloc / mxFree for memory management; HALO Backend (ggml-halo.cpp): Adapted for Pingtouge Hanguang 800 / Hanguang 600, dependent on HALO Platform SDK, using the Hanguang 800 matrix computing unit (MAC Array) to accelerate FP16 / INT8 inference, with the ARM CPU side (Etian 710) responsible for scheduling and data transfer, supporting ARM. NEON instruction set optimization; CPU backend: adapted to x86_64 (OpenBLAS) and ARM64 (ARM Compute Library) scenarios without GPU fallback.

[0042] The backend automatic detection logic is as follows: try loading the CUDA backend in sequence → if successful, use it; if it fails, try the MXN backend → if successful, use it; if it fails, try the HALO backend → if successful, use it; if all fail, fall back to the CPU backend.

[0043] At the operator execution layer, GGML operator abstraction and memory management are reused, with only the hardware execution part replaced by the corresponding backend implementation. The quantization inference path is consistent with the CUDA backend to ensure lossless accuracy.

[0044] Model loading and quantization weight memory mapping (step S4)

[0045] The model management module supports registration, hot loading, and hot unloading of GGUF format models. After loading, the model remains resident in GPU memory, supporting parallel multi-model services. It supports quantization formats such as Q4_0 (4-bit), Q4_K_M (4-bit), Q5_K_M (5-bit), and Q8_0 (8-bit). The single-card inference speed (batch_size=1) for each llama-3-8B quantization format on the Muxi C500 and Pingtouge Hanguang GPUs is as follows:

[0046] Quantization format Model size (7B) Muxi C500 reasoning speed Honey Badger's reasoning speed Q4_0 3.6 GB 55 tok / s 35 tok / s Q4_K_M 4.1 GB 50 tok / s 32 tok / s Q5_K_M 4.7 GB 42 tok / s 28 tok / s Q8_0 6.7 GB 28 tok / s 18 tok / s

[0047] The quantized weight tensor (GGUF format) is directly mapped to the device memory of the selected backend instance, avoiding additional copy overhead.

[0048] Inference Task Scheduling and Dynamic Batch Processing

[0049] Reference Figure 3 The scheduler uses asyncio asynchronous coroutines to implement non-blocking scheduling. The scheduling loop is as follows: retrieve requests in batches from RedisList → group by model name → perform inference for each group → write the results back to the Future of the corresponding request.

[0050] Priority queue definitions: high (high priority queue, real-time dialogue / interactive applications, timeout 5 seconds); normal (normal queue, batch analysis / report generation, timeout 30 seconds); batch (batch queue, offline processing / data preprocessing, timeout 300 seconds).

[0051] Dynamic batching: In each inference loop, the scheduler checks the queue for multiple requests with the same data type. If found, they are merged into a single batch and executed on the GPU at once. The batching window is configurable (default 50ms). Inference continues even if the maximum batch size has not been reached after the timeout, balancing latency and throughput.

[0052] Computing power pooling and load balancing (step S5)

[0053] Unified pooling management of multiple GPU nodes, providing four load balancing strategies:

[0054] Least Connections: Selects the node with the fewest active requests, suitable for scenarios with large differences in request processing time; Round Robin: Assigns nodes sequentially, suitable for scenarios with uniform node performance; Weighted Round Robin: Assigns nodes according to weight ratios, suitable for scenarios with large differences in node performance; Memory Priority: Selects the node with the most available memory, suitable for large model inference scenarios.

[0055] Computing power monitoring and alerts (step S6)

[0056] Reference Figure 4 The mxmon daemon (implemented in Rust) collects telemetry data (MXN-SMI, HALO Monitor) through the GPU vendor's SDK C interface. It uses a shared memory circular buffer (lock-free SPSC queue) to pass the data to the Python side, avoiding serialization overhead. The sampling frequency is 1Hz, with CPU overhead of less than 0.5% and an impact on GPU performance of less than 1%.

[0057] Monitoring metrics include: GPU utilization, VRAM usage, GPU temperature, GPU power consumption, inference QPS, token generation rate, inference latency (P50 / P95 / P99), and queue depth. Alarm rules include: GPU temperature greater than 85 degrees Celsius (critical), VRAM usage as a percentage of total VRAM greater than 0.90 (warning), inference P99 latency greater than 5 seconds (warning), and GPU utilization at 0 for 5 consecutive minutes (critical).

[0058] Automatic failover of nodes (step S7)

[0059] Perform a health check (memory test + small model inference test) on GPU nodes every 30 seconds. If three consecutive health checks fail: mark the node as "unavailable" → reroute pending requests to healthy nodes → send an alarm notification → automatically rejoin the computing pool after the node recovers.

[0060] Compatible with domestic IT innovation environment

[0061] It fully supports the Kylin V10, UnionTech UOS, Ubuntu 22.04 LTS, and CentOS 7.9 operating systems; it supports domestic x86 CPUs such as Hygon C86 and Zhaoxin KH-40000, as well as domestic ARM CPUs such as Phytium S2500, Kunpeng 920, and Pingtouge Yitian 710; it provides native ARM64 compilation support and ensures environment consistency through Docker containerization.

[0062] Application scenarios

[0063] This embodiment can be applied to: financial institutions migrating LLM inference workloads from NVIDIA GPUs to domestic GPU platforms; private inference deployment of financial AI applications (intelligent investment research, compliance analysis, RAG retrieval enhancement) (data not leaving the domain); domestic GPU backend support for multi-model inference platforms (Qwen / Llama / ChatGLM, etc., switchable on demand); unified management and monitoring of heterogeneous computing power in NVIDIA + domestic GPU hybrid environments; and replacement of AI inference infrastructure under the requirements of domestic IT innovation.

[0064] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for heterogeneous large-scale model inference adaptation and computing power scheduling for domestically produced GPUs, characterized in that, Includes the following steps: Step S1: Define a unified GPU backend interface, which includes at least five core primitives: memory allocation, host-to-device data copying, device-to-host data copying, kernel startup, and computation flow synchronization. Step S2: Implement the unified GPU backend interface based on CUDA, MXN, HALO, and CPU respectively, resulting in multiple backend instances. MXN is the heterogeneous computing programming framework for Muxi GPU, and HALO is the programming framework for Pingtouge Hanguang NPU. Step S3: In the operator execution layer of the large model inference engine, reuse the general operator abstraction and memory management of the inference engine, replacing only the hardware execution part with the implementation of the corresponding backend instance. The newly added backend instances are accessed as independent backend files. The general operator abstraction includes matrix multiplication, attention calculation, layer normalization, and rotation position encoding operators. Matrix multiplication is based on the BLA provided by the corresponding SDK of each backend. The S-library accelerates the process, with memory allocation and release based on the memory allocators provided by each backend SDK. During runtime, it automatically detects available backend instances in the current environment and selects the optimal backend instance for operator computation according to preset priorities. Step S4 involves directly mapping the quantized weight tensor of the large model to the memory of the selected backend instance for operator computation. Step S5 organizes multiple GPU nodes into a computing power pool, routing inference requests to the target GPU nodes within the pool based on a load balancing strategy. Step S6 collects telemetry data from the GPU devices corresponding to each backend instance through an independent daemon process and outputs it to the monitoring system, monitoring and alerting on GPU utilization, memory usage, temperature, power consumption, and inference service metrics. Step S7, when a GPU node failure is detected, the pending inference requests are rerouted to healthy nodes, and the failed node is automatically re-added to the computing power pool after recovery.

2. The method according to claim 1, characterized in that, Step S2 also includes backend automatic detection logic: try loading the CUDA backend, MXN backend, and HALO backend in sequence. When a backend is successfully loaded, it is selected. When all GPU backends fail to load, it falls back to the CPU backend.

3. The method according to claim 1, characterized in that, The quantized weight tensor in step S4 is a quantized tensor in GGUF format, and the quantization format includes Q4_0, Q4_K_M, Q5_K_M and Q8_0.

4. The method according to claim 1, characterized in that, The load balancing strategies mentioned in step S5 include the least connections strategy, the round-robin strategy, the weighted round-robin strategy, and the memory priority strategy. The memory priority strategy selects the GPU node with the most available memory for large model inference requests.

5. The method according to claim 1, characterized in that, It also includes an inference task scheduling step: inference requests are entered into a high-priority queue, a normal queue, or a batch queue according to their priority and are assigned different timeout periods; the scheduler checks within a preset batch processing window whether there are multiple inference requests of the same type in the request queue, and if so, merges them into a batch and executes them on the GPU at once. The batch processing window is configurable.

6. The method according to claim 1, characterized in that, The daemon process described in step S6 collects telemetry data through the C interface of each GPU vendor's SDK and transmits the telemetry data to the monitoring and acquisition terminal through a shared memory ring buffer, wherein the shared memory ring buffer is a lock-free single producer single consumer queue.

7. The method according to claim 1, characterized in that, The specific steps for detecting GPU node failures in step S7 are as follows: perform periodic health checks on the GPU node, and mark the node as unavailable when the health checks fail N times in a row, where N is a preset positive integer.

8. The method according to claim 1, characterized in that, The large model inference engine is the llama.cpp inference engine. The operator execution layer reuses the GGML operator abstraction, only replacing the hardware execution part with the corresponding backend implementation. The newly added MXN backend and HALO backend are connected as independent backend files.

9. A heterogeneous large-model inference adaptation and computing power scheduling system for domestically produced GPUs, characterized in that, include: The backend abstraction layer module defines a unified GPU backend interface containing five core primitives: memory allocation, host-to-device data copying, device-to-host data copying, kernel startup, and compute stream synchronization. It also provides multiple backend instances for CUDA, MXN, HALO, and CPU. The inference adaptation module reuses the general operator abstraction and memory management of the inference engine at the operator execution layer of the large model inference engine, replacing only the hardware execution part with the implementation of the corresponding backend instance, and automatically selects the optimal backend instance at runtime according to a preset priority. The model management module directly maps the quantized weight tensor to the device memory of the selected backend instance and manages the hot loading and hot unloading of the model. The inference task scheduling module is used to manage request queues, request priorities, dynamic batch processing, and route inference requests to target GPU nodes. The computing power pool management module is used to organize multiple GPU nodes into a computing power pool, route inference requests based on load balancing strategies, and automatically transfer nodes when they fail and automatically rejoin them after recovery. The computing power monitoring module is used to collect telemetry data from each backend GPU through a daemon process and output it to the monitoring system.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the heterogeneous large model inference adaptation and computing power scheduling method for domestic GPUs as described in any one of claims 1 to 8.