Large model heterogeneous reasoning method and system

Abstract

The invention discloses a large model heterogeneous reasoning method and system. The method comprises the steps that target components to be unloaded and reserved components to be unloaded are determined according to calculation and storage characteristics for a large model, the target components are components which are densely stored and have the calculation strength lower than a set standard, and the reserved components are calculation-intensive components; and unloading the target component to a heterogeneous processing unit different from the GPU to execute related tasks, allocating the related tasks of the reserved component to the GPU to execute, and performing acceleration processing on the tasks executed by the heterogeneous processing unit in cooperation with a parallel scheduling strategy and a quantitative storage strategy so as to obtain a reasoning result. According to the method, the dependence on the GPU video memory and the reasoning cost are reduced, and the throughput is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and more particularly to a large model heterogeneous inference method and system. BACKGROUND

[0002] In recent years, large language models (LLMs) have shown excellent performance in natural language processing, intelligent question answering, code generation and other tasks, and have promoted the rapid development of the model-as-a-service (MaaS) mode. However, as the model size continues to expand and the context window continues to grow (such as Kimi supporting long text input and DeepSeek-R1 supporting long sequence inference), inference services face serious challenges at the system level. For example, a large amount of Key-Value Cache (KV Cache) needs to be cached during LLM inference, especially in the long context scenario, and the KV Cache occupies much more video memory than the model parameters itself. This video memory bottleneck becomes the main factor restricting the concurrent throughput. For another example, the computing intensity of different operators in the Transformer structure is significantly different (such as Attention computing-intensive and FFN memory-intensive), and the traditional GPU-centered architecture is difficult to efficiently coordinate the computing and storage resources between CPU and GPU. In addition, the GPU memory is expensive and has limited capacity, while the CPU memory and SSD (solid state drive) have large capacity but low bandwidth and computing power. How to balance the cost and performance in the whole system range, such as "storage for calculation" and "storage for intensive calculation", becomes the key to reducing the inference cost.

[0003] In the prior art, there are some solutions for optimizing large model inference, mainly GPU-centric inference systems and heterogeneous inference systems based on offloading. The traditional GPU-centric inference system highly depends on the high computing power and high bandwidth characteristics of the GPU, and places all model parameters and KV Cache in the GPU memory. Typical systems such as NVIDIA Triton and TensorRT perform excellently in computing efficiency, but are limited by the memory capacity and difficult to support high-concurrency long-context inference scenarios, with poor scalability. To break through the memory limit, some heterogeneous inference systems based on offloading attempt to offload part of the model parameters or KV Cache to CPU memory or SSD, such as llama.cpp. llama.cpp is an open source project that supports CPU and hybrid CPU / GPU inference, providing typical "partial layer offloading" functions. The basic mechanism of this type of technology is that users can specify a parameter (such as -ngl 40) to indicate that the first 40 layers of the model will be executed by the GPU. The remaining layers are completely calculated by the CPU; during the inference process, after each GPU layer is calculated, its output tensor needs to be transmitted to the CPU memory through the PCIe bus as the input of the next CPU layer; conversely, the results of CPU layer calculation may also need to be transmitted back to the GPU for subsequent GPU layer use. The characteristics of this type of technology are: the model layer is taken as the scheduling granularity, that is, the smallest unit of offloading is the entire layer or module (such as FFN layer, feedforward network) of the model, and the scheduling strategy is simple and crude; before the inference starts, the allocation of computing tasks between the GPU and the CPU is determined, and this static division cannot be dynamically adjusted according to the current computing load or data characteristics; the data exchange between the CPU and the GPU is frequent and explicit, and this explicit data migration method introduces a lot of PCIe communication overhead.

[0004] Through analysis, the prior art mainly has the following defects:

[0005] 1) DeepSeek full system, Kimi-K2, Qwen3, LLAMA4, GLM4.5, Smallthinker, etc. The model parameter quantity is as high as hundreds of billions, and the single card memory occupation is often more than 100GB, which must rely on multi-GPU cluster parallel. By introducing model parallel, tensor segmentation, pipeline parallel and other parallel strategies, relying on NCCL, MPI and other underlying communication libraries, the engineering implementation difficulty and maintenance cost increase significantly. And distributed inference has high requirements for network bandwidth and communication delay, and the hardware procurement and maintenance cost is high.

[0006] 2) The demand for resources causes the MOE (expert hybrid) large model to be unable to achieve local deployment on consumer platforms, and the cost and complexity brought by horizontal expansion (increasing nodes) of the server platform do not proportionally increase the benefits, thereby causing limited flexibility and scalability.

[0007] 3) There are also pure CPU inference frameworks such as llama.cpp on the market, which can run without GPU, but are subject to the parallel computing capability and memory bandwidth of the CPU, and the processing speed of the model Prefill stage is unbearable. Although the Token generation speed in the Decode stage can reach single digits, it is far from meeting the low-latency demand of real-time dialogue and large-scale concurrent services due to performance limitations. SUMMARY

[0008] The purpose of the present application is to overcome the defects of the prior art described above, and to provide a large model heterogeneous inference method and system.

[0009] According to a first aspect of the present application, a large model heterogeneous inference method is provided. The method comprises the following steps:

[0010] Step S1, for a large model, determining target components and reserved components to be offloaded according to the calculation and storage characteristics, the target components being storage-intensive and having a calculation intensity lower than a set standard, and the reserved components being calculation-intensive;

[0011] Step S2, offloading the target components to a heterogeneous processing unit different from the GPU to perform related tasks, assigning related tasks of the reserved components to the GPU for execution, and cooperatively performing accelerated processing on the tasks executed by the heterogeneous processing unit using a parallel scheduling strategy and a quantized storage strategy, thereby obtaining an inference result.

[0012] In one embodiment, the large model comprises an expert hybrid network, a multi-head attention module, and an input embedding module, and the heterogeneous processing unit is a CPU, and step S2 comprises the following sub-steps:

[0013] Offloading the expert network to the CPU, wherein all expert weights are resident in the host DRAM; in the filling stage, using x86 AMX instructions to batch accelerate the set size matrix-matrix operation of the expert network; in the decoding stage, switching back to AVX instructions to perform vector-matrix operation;

[0014] Performing parallel scheduling based on non-uniform memory access awareness, wherein the expert calculation tasks are bound to the corresponding memory nodes and CPUs to maximize the local bandwidth and multi-core parallelism;

[0015] Performing quantized storage, wherein the expert weights are stored using INT8 or INT4 quantization, and the quantized matrix is used for calculation during the inference process.

[0016] In one embodiment, the target component is all expert networks contained in the large model or multiple linear layers in a single expert network, or a feedforward network contained in the large model.

[0017] In one embodiment, the quantization storage strategy is to use different quantization storage precisions for different layers in the expert network.

[0018] In one embodiment, FP16 is used for the first linear layer of the expert network, the activation function is quantized to INT8 after the calculation of the second linear layer, and then dequantized to FP16 for output.

[0019] In one embodiment, the quantization storage strategy is to assign different quantization levels according to the importance of the expert weights.

[0020] In one embodiment, the large model includes DeepSeek series, Kimi-K2, Qwen3, LLAMA4, GLM4.5, and Smallthinker.

[0021] According to a second aspect of the present application, a large model heterogeneous inference system is provided. The system comprises:

[0022] A task offloading analysis module is configured to determine, for a large model, target components and reserved components to be offloaded according to the calculation and storage characteristics, the target components being storage-intensive and having a calculation intensity lower than a set standard, and the reserved components being calculation-intensive;

[0023] A cooperative acceleration module is configured to offload the target components to a heterogeneous processing unit different from the GPU to perform related tasks, assign related tasks of the reserved components to the GPU for execution, and cooperatively perform parallel scheduling and quantization storage strategies to accelerate the processing of the tasks performed by the heterogeneous processing unit, thereby obtaining an inference result.

[0024] Compared with the prior art, the large model heterogeneous inference system provided by the present application breaks through the high dependence of existing mainstream MOE large models (such as DeepSeek series, Kimi-K2, Qwen3, LLAMA4, GLM4.5, and Smallthinker) on computing power and video memory, reduces the deployment threshold, and enables expert hybrid models with hundreds of billions or even thousands of billions of parameters to complete local inference and fine-tuning on a single consumer-level video card (such as a high-end game card or workstation card with 24 GB of video memory).

[0025] Other features and advantages of the present application will become apparent from the following detailed description of exemplary embodiments thereof, which description should be taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0027] Figure 1 is a flow chart of a large model heterogeneous reasoning method according to one embodiment of the application;

[0028] Figure 2 is an architecture diagram of a large model heterogeneous reasoning system according to one embodiment of the application. DETAILED DESCRIPTION

[0029] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangements, numerical expressions, and numerical values of the components and steps set forth in these embodiments do not limit the scope of the present application unless otherwise specifically stated.

[0030] The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended to limit the scope of the application, its application, or uses.

[0031] Techniques, methods, and apparatus known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered part of the specification as appropriate.

[0032] In all of the compositions and methods shown and discussed herein, any specific values should be interpreted as merely exemplary, and are not to be interpreted as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0033] It should be noted that like numbers and letters refer to like items throughout the drawings, and once an item is defined in one drawing, it is not necessary to discuss it further in subsequent drawings.

[0034] To more clearly understand the present application, a simple introduction to the relevant professional terms is first provided.

[0035] Large Language Models (LLMs) are a class of pre-trained models based on deep neural networks, which learn from massive amounts of text data in a self-supervised manner to obtain deep representations of language context. Unlike traditional natural language processing systems based on rules or shallow feature engineering, LLMs typically have hundreds of millions or even tens of billions of parameters, enabling rapid transfer and application in various downstream tasks at the level of zero-shot, few-shot, or fine-tuning. The core architecture of LLMs usually adopts the Transformer structure, which captures long-range dependencies through the Multi-Head Self-Attention mechanism, and has the ability of text generation, summarization, translation, and question answering through layered encoder-decoder or pure decoder architecture. LLMs can be flexibly controlled through "Prompt Engineering" during inference, without the need for retraining to complete new tasks. At the same time, through continuous online learning and incremental updating, LLMs can continuously absorb new knowledge and enhance their continuous adaptability and scalability.

[0036] Attention is a neural network mechanism that involves intensive computation, which dynamically allocates computational resources based on the relevance between input positions when processing sequence data. The attention mechanism maps the features of each position into three representations: "Query", "Key", and "Value", calculates the similarity scores between Query and all Keys, and then weights and sums the Values based on these scores to generate a new context representation.

[0037] Mixture-of-Experts (MoE) is a sparse activation neural network extension technology that significantly increases model capacity while ensuring computational and memory efficiency. MoE inserts multiple "expert" sub-networks (Experts) into a regular Transformer or other deep network layer and introduces a dynamic routing gating mechanism to automatically select a few experts for activation and computation based on input features. Compared with dense models with full activation, MoE only calls a few experts for each input sample or each position, allowing the parameter size to grow linearly while the computational complexity increases only by a constant factor, enabling the design of models with over 100 billion or even 1 trillion parameters. During inference, MoE-based models generate different expert scores, allowing independent monitoring and optimization of the weights and routing strategies of each expert. At the same time, dynamic routing also supports sparse updates, expert replacement, and expert expansion, providing more fine-grained model evolution and maintainability.

[0038] Prefill (filling stage) refers to a complete forward calculation of the model on the "context" text (Prompt) provided by the user before the generation task starts. The core role of Prefill is: calculate the self-attention key-value pair (KV) cache: for each existing token in the prompt, the Transformer will calculate the corresponding Key and Value at each layer through the self-attention module and store them in the cache; generate context representation: at the same time, the model will output the hidden state representation (Hidden States) of each layer, providing rich context information for the subsequent generation stage; complete the most expensive part at one time: since the length of the Prompt is usually longer, Prefill needs to perform complete attention calculation for each position, so this is the most time-consuming step in the inference process. The processing speed of the Prefill stage affects the Time To First Token (TTFT) of the model.

[0039] After completing the prefill, the model enters the Decode (decoding) stage, which is the process of generating new tokens step by step. Each time a new token is generated, the model only needs to perform attention calculation on the Query of this position and the previously cached KV, without repeating the calculation of the entire context, thereby reducing the calculation amount of each step from O(n 2 ) to O(n) (n is the number of generated tokens). The newly generated Token is converted into the corresponding KV and appended to the cache for use in the next decoding step. At the same time, the hidden state of the last layer is updated to follow the changes in the context.

[0040] AMX (Advanced Matrix Extensions) is a set of hardware-level matrix operation instruction set extensions introduced by Intel in the fourth generation Xeon server, optimized for deep learning and high-performance computing scenarios. By dividing large-scale matrices into fixed-size tiles, AMX can load, calculate, and accumulate data for the entire tile in parallel at the hardware level, thereby completing multiple multiplication and accumulation operations in a single cycle. Compared with traditional AVX vector instructions, AMX has higher algorithmic throughput and lower instruction scheduling overhead when performing matrix-matrix multiplication, which can greatly improve the matrix operation performance per core. With AMX, developers can achieve comparable dense matrix computation acceleration on CPUs as GPU tensor cores without manually splitting loops or writing complex block scheduling code.

[0041] NUMA-Aware (Non-Uniform Memory Access Aware) refers to the parallel scheduling and memory allocation strategy optimized for Non-Uniform Memory Access (NUMA) topology in a multi-socket multi-core server architecture. In a typical NUMA system, different CPU sockets or physical core groups have their own local memory nodes, and cross-node access will cause significant access delay and bandwidth bottleneck. NUMA-Aware strategies include: 1) binding threads or processes to physical CPU cores and their local memory nodes; 2) allocating frequently accessed data structures (such as MoE expert weights, activation cache) to the corresponding local memory; 3) preferentially reusing local resources and avoiding cross-node exchange when scheduling. These strategies can maximize local memory bandwidth utilization and reduce access delay. For the expert hybrid model, NUMA-Aware parallelism can reasonably distribute a large number of sparse activation expert computing tasks to each NUMA node, enabling efficient multi-threaded parallelism and significantly improving overall throughput and stability.

[0042] Referring to Figure 1 As shown, in general, the provided large model heterogeneous inference method includes: step S110, for a large model, determining target components and reserved components to be offloaded according to computing and storage characteristics, the target components are storage-intensive and have a computing intensity lower than a set standard, and the reserved components are computing-intensive components; step S120, offloading the target components to a heterogeneous processing unit different from the GPU to perform related tasks, assigning related tasks of the reserved components to the GPU for execution, and cooperating with parallel scheduling strategies and quantization storage strategies to accelerate processing of tasks performed by the heterogeneous processing unit, thereby obtaining an inference result.

[0043] The large model can be a MoE structure model or a non-MOE structure model, etc. The following mainly takes the MoE structure model as an example for illustration. Figure 2 is the basic architecture of the large model, including Add&Norm (residual connection and layer normalization) module, MOE (expert hybrid network) module, Multi-Head Attention (multi-head attention) module, and Input Embedding (input embedding module), etc. For this basic architecture, the invention designs a heterogeneous inference method to offload the computation of the MOE module from the GPU to the CPU for execution. In this way, the Attention module can fully utilize the computing power of the graphics card, and the MOE module with higher storage requirements but lower computing power requirements can be executed on the CPU. The activation after CPU computation is transmitted back to the GPU for subsequent computation.

[0044] Specifically, when the input text sequence length reaches a certain scale, the calculation of the Attention module becomes the performance bottleneck of the model. The Attention module needs to read and write the same batch of KV cache multiple times, which has a very high requirement for the memory bandwidth. Moreover, the calculation mode is a large-scale matrix multiplication (GEMM), which is very suitable for SIMT (Single Instruction Multiple Thread) parallelism and tensor core on GPU, and has high acceleration efficiency and is easy to optimize. The MoE module activates only top-k experts each time, and the calculation amount is much lower than that of Attention, but the parameter scale is much larger than that of Attention, and the demand for storage resources is extremely large. In view of the above characteristics, the application designs a technical solution for supporting the MOE model using GPU and CPU cooperative calculation, which mainly improves the following aspects:

[0045] 1. MoE offloading to CPU

[0046] (a) Resident host DRAM for all expert weights, completely releasing GPU memory;

[0047] (b) Prefill stage using x86 AMX instructions to batch accelerate small-scale matrix-matrix operations of the expert network;

[0048] (c) Switch back to AVX instructions to perform vector-matrix operations in the Decode stage, taking into account the low-latency incremental inference performance.

[0049] 2. NUMA-Aware parallel scheduling

[0050] Based on the NUMA topology, the expert calculation task is bound to the corresponding memory node and CPU core, maximizing the local bandwidth and multi-core parallelism;

[0051] 3. Low-precision quantization optimization

[0052] INT8 / INT4 quantization storage is used for expert weights, and quantized matrices are directly used for calculation during inference, further reducing memory occupation and improving memory bandwidth utilization;

[0053] Through the above cooperative strategies, the advantages of GPU in dense calculation are fully utilized, and the large storage and AMX acceleration capability of CPU are also utilized, realizing efficient deployment and inference of a hundred billion MoE model on a single consumer-level platform.

[0054] It should be noted that the above embodiments can be appropriately changed or modified by those skilled in the art without departing from the spirit and scope of the application. For example, in addition to offloading the MoE module to the CPU for execution and using the AMX instruction set for acceleration, the following acceleration schemes can also be considered.

[0055] 1. Alternative solution for calculation offloading granularity

[0056] The preferred solution of the present application takes the expert network (Expert) as the basic unit of offloading, and can also use finer or coarser granularity for offloading, which can also achieve the purpose of releasing GPU memory and heterogeneous collaboration. For example:

[0057] (a) Sub-expert level offloading: For a single expert with a large number of parameters, it can be further split, such as deploying multiple linear layers in a single expert network on GPU or CPU independently, and performing more extreme fine-grained scheduling according to the computing intensity of each layer. The layers with high computing intensity are retained on the GPU, while the layers with low computing intensity but large parameter quantity are offloaded to the CPU.

[0058] (b) Intra-layer operator level offloading: Offload the feedforward network (FFN) module in the Transformer model as a whole. This is a coarser granularity than the expert level. For dense models without MoE structure, this solution is particularly suitable. The entire FFN module (including two linear layers and activation functions) can be offloaded to the CPU, while the Attention module is retained on the GPU.

[0059] 2. Alternative solutions for collaborative computing hardware platforms

[0060] The preferred solution of the present application uses the hardware combination of x86 CPU (equipped with AMX instruction set) + NVIDIA GPU. Other types of coprocessors or computing units can also be used to perform offloaded computing tasks as long as they have large capacity memory and corresponding computing power.

[0061] For example, the target platform for offloading is not limited to x86 architecture CPU, but also ARM architecture server-level CPU (such as Neoverse series) can be used to perform expert computing using its considerable memory bandwidth and parallel processing capability. Or use RISC-V architecture CPU, with its dedicated vector computing extension instruction set.

[0062] For another example, use a dedicated acceleration card, i.e. offload the MoE module to other dedicated computing units, such as offloading to FPGA (Field Programmable Gate Array), by customizing the logic circuit, designing an efficient data flow and computing pipeline for expert network computation. Or offload to AI coprocessor units (such as AI acceleration cards based on ASIC), which are optimized for low-precision, high-concurrency matrix operations.

[0063] 3. Alternative solutions for precision and quantization schemes

[0064] The preferred scheme of the present application adopts INT8 / INT4 quantization. Alternative schemes can adopt other precision formats or quantization strategies. For example, a mixed precision strategy is adopted, different precisions are adopted for different layers in the expert network, such as FP16 is used for the first linear layer, the activation function is immediately quantized to INT8 for the calculation of the second linear layer, and finally the output is dequantized to FP16, so as to achieve a better balance between precision and performance. For another example, non-uniform quantization is adopted, that is, more advanced non-uniform quantization or sparse quantization method is adopted, and more quantization levels are allocated to more important parts of the expert weight, so as to maintain the model performance as much as possible at a very low bit rate (such as INT2, INT1).

[0065] Correspondingly, the present application also provides a large model heterogeneous inference system for implementing one or more aspects of the above method. For example, the system includes: a task offloading analysis module: for a large model, determining target components and reserved components to be offloaded according to the calculation and storage characteristics, the target component is a storage-intensive component with a calculation intensity lower than a set standard, and the reserved component is a calculation-intensive component; a cooperative acceleration module: for offloading the target component to a heterogeneous processing unit different from the GPU to perform related tasks, assigning related tasks of the reserved component to the GPU for execution, and cooperatively scheduling the tasks performed by the heterogeneous processing unit with a parallel scheduling strategy and a quantized storage strategy to accelerate the processing of the tasks, thereby obtaining an inference result.

[0066] In order to further verify the effect of the present application, an experimental test is carried out. Table 1 shows the benchmark performance index of DeepSeek-V3 processing 1024 Tokens on a test platform equipped with Intel Xeon 8488C CPU, NVIDIA RTX 4090 GPU and 16-channel DDR5-4400MT / s memory.

[0067] Table 1: Comparison of large model inference performance

[0068]

[0069] As can be seen from Table 1, the Prefill speed of the present application can reach tens of times of that of a pure CPU platform, and the Decode rate is also improved to a certain extent. The test results show that the present application not only meets the local deployment needs of consumer platforms, but also can achieve linear performance improvement at the lowest cost and simplest configuration when small-scale horizontal expansion is required, and fully solves the bottleneck of traditional MoE large models in flexibility and scalability.

[0070] In summary, the large model heterogeneous reasoning system provided by the application can identify components with different computing and storage characteristics in the Transformer model, and offload components with high storage density and relatively low computing intensity (such as MoE expert network, FFN, etc.) to a large-capacity memory managed by a CPU or other coprocessor for execution, while retaining the computing-intensive components (such as Attention) on the GPU, thereby reducing the dependence on GPU video memory, controlling the reasoning cost, and improving the system throughput.

[0071] The application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0072] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a globally addressable array, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0073] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0074] Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0075] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0076] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or nonvolatile memory, or a suitable combination of the different types of computer readable storage media. The computer readable program instructions can also be downloaded to a computer, other programmable data processing apparatus, or other device from a computer readable storage medium or to an external computer or external storage device via a data signal that can be transmitted for example via a wired medium or a wireless medium such as the Internet or Wireless Application Protocol (WAP) signaling.

[0077] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0078] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0079] Embodiments of the application have been described above, and the description is intended to be illustrative of the embodiments of the application and not exhaustive. Numerous modifications and adaptations thereof will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The scope of the application is defined by the appended claims.

Claims

1. A large model heterogeneous inference method, comprising the following steps: Step S1, for a large model, determining target components and reserved components to be offloaded according to computing and storage characteristics, the target components being storage-intensive and having a computing intensity lower than a set standard, and the reserved components being computing-intensive; Step S2, offloading the target components to a heterogeneous processing unit different from a GPU to perform related tasks, assigning related tasks of the reserved components to the GPU for execution, and performing accelerated processing on the tasks performed by the heterogeneous processing unit in cooperation with a parallel scheduling strategy and a quantized storage strategy, thereby obtaining an inference result.

2. The method of claim 1, wherein, The large model comprises an expert hybrid network module and a multi-head attention module, the expert hybrid network module comprising one or more expert networks, and the heterogeneous processing unit is a CPU, and step S2 comprises the following sub-steps: offloading the expert networks to the CPU, wherein all expert weights are resident in the host DRAM; in a padding phase, using x86 AMX instructions to batch accelerate the set-scale matrix-matrix operations of the expert networks; switching back to AVX instructions to perform vector-matrix operations in a decoding phase; performing parallel scheduling based on non-uniform memory access awareness, wherein the expert computing tasks are bound to corresponding memory nodes and CPUs to maximize local bandwidth and multi-core parallelism; performing quantized storage, wherein the expert weights are stored in INT8 or INT4, and the quantized matrix is used for calculation during inference.

3. The method of claim 1, wherein, The target components are all expert networks contained in the large model or multiple linear layers in a single expert network, or a feedforward network contained in the large model.

4. The method of claim 2, wherein, The quantized storage strategy is to use different quantized storage precisions for different layers in the expert network.

5. The method of claim 4, wherein, For the first linear layer of the expert network, FP16 is used, and after the activation function, the result is quantized to INT8 for the calculation of the second linear layer, and finally dequantized to FP16 for output.

6. The method of claim 2, wherein, The quantized storage strategy is to assign different quantization levels according to the importance of the expert weights.

7. The method of claim 1, wherein, The large model comprises DeepSeek, Kimi-K2, Qwen3, LLAMA4, GLM4.5, and Smallthinker.

8. A large model heterogeneous inference system, comprising: a task offloading analysis module for determining target components and reserved components to be offloaded according to computing and storage characteristics for a large model, the target components being storage-intensive and having a computing intensity lower than a set standard, and the reserved components being computing-intensive; a cooperative acceleration module for offloading the target components to a heterogeneous processing unit different from a GPU to perform related tasks, assigning related tasks of the reserved components to the GPU for execution, and performing accelerated processing on the tasks performed by the heterogeneous processing unit in cooperation with a parallel scheduling strategy and a quantized storage strategy, thereby obtaining an inference result.

9. A computer readable storage medium having stored thereon a computer program, wherein, The computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

10. A computer device comprising a memory and a processor, having stored on the memory a computer program capable of running on the processor, characterized in that, The processor, when executing the computer program, implements the steps of the method according to any one of claims 1 to 7.

Images