Light-weight large model reasoning acceleration method and system

Abstract

The invention discloses a reasoning acceleration method and system for a lightweight large model, and the method comprises the following steps: S1, carrying out complex domain enhanced ultra-low bit quantization, including complex domain representation conversion of real number parameters, dynamic bit distribution of task perception, and complex domain quantization and precision calibration; and S2, constructing a three-layer architecture comprising a resource layer, a scheduling layer and an application layer, inputting the quantized real number weight parameter, the user reasoning request and the hardware state information in the step S13, completing pre-filling and decoding stage task execution through three-layer collaborative scheduling, performing heterogeneous collaborative reasoning based on PD separation, and outputting a reasoning result and a task execution state. Belongs to the technical field of artificial intelligence, and realizes high precision maintenance under ultra-low bit quantization through combination of complex domain representation conversion and dynamic bit allocation precision calibration, effectively reduces the occupation of a large model video memory, solves the technical problem of overlarge precision loss of low bit quantization, and breaks through the limitation of traditional real number domain quantization.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically a lightweight method and system for accelerating inference of large models. Background Technology

[0002] With the development of large model technology, large models such as GLM-4 and Llama3 have demonstrated excellent performance in natural language processing tasks. However, the sheer size of these models, typically reaching billions to hundreds of billions of parameters, leads to problems such as high memory consumption, long inference latency, and high hardware costs during inference, limiting their industrial application. To address these issues, existing technologies mainly focus on two directions: model compression (e.g., quantization, pruning) and inference architecture optimization (e.g., PD separation). Low-bit quantization techniques: Mainstream solutions such as GPTQ, AWQ, and TensorRT-LLM quantization are limited to the real number domain and employ uniform or simple non-uniform quantization strategies. In ultra-low-bit scenarios of INT4 and below, discretization errors cause a sharp drop in model performance, with accuracy loss generally exceeding 5%; moreover, existing solutions mostly use static bit allocation and do not dynamically adjust according to task type, making it difficult to balance accuracy and efficiency. Low-bit quantization refers to converting high-precision weight parameters (such as 32-bit floating-point numbers) of the model into low-precision representations (such as INT4, INT2) to reduce memory usage and computational overhead.

[0003] PD Separation Architecture: Existing solutions (such as vLLM and Text Generation Inference) only achieve physical separation of the pre-fill-decode split (PD split). PD split is an architectural optimization technique for large model inference, dividing the inference process into a pre-fill stage (P stage) and a decoding stage (D stage). The P stage mainly completes the token embedding of the input text and the initial attention calculation, which is a computationally intensive task; the D stage mainly completes the generation of subsequent tokens, relying on KV cache to store intermediate results, which is a storage-intensive task. Existing solutions use homogeneous hardware (such as pure GPUs) for deployment, without fine-tuning the resource requirements of the two stages; the scheduling strategy is static batch scheduling, and the latency fluctuation exceeds 15% in high-concurrency scenarios, resulting in poor stability.

[0004] Heterogeneous inference architecture: Existing solutions are mostly simple collaborations between CPU and GPU, without integrating dedicated quantization operation hardware units, and without deep integration with model compression methods, making it difficult to reach the upper limit of hardware performance.

[0005] Existing technology has three major flaws: Ultra-low bit quantization suffers significant precision loss and struggles to break through the performance ceiling of real number field quantization. The PD separation architecture has a rough resource adaptation, fails to achieve targeted matching of heterogeneous hardware, and has low resource utilization. In high-concurrency scenarios, scheduling lacks dynamic load awareness, resulting in large latency fluctuations and making it difficult to meet the real-time requirements of industrial applications.

[0006] Therefore, there is an urgent need to develop a lightweight inference acceleration method for large models to solve the problems in existing technologies. Summary of the Invention

[0007] The purpose of this invention is to provide a lightweight inference acceleration method and system for large models, which can solve the technical problems of large precision loss in ultra-low bit compression, rough system resource adaptation, and performance fluctuation in high-concurrency scenarios. It is also simple in structure and easy to use, thus solving the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A lightweight method for accelerating inference in large models includes the following steps: S1: Perform complex-domain enhanced ultra-low bit quantization, including complex-domain representation transformation of real parameters, task-aware dynamic bit allocation, and complex-domain quantization and precision calibration, specifically including: S11: Input the real weight parameters and parameter distribution statistics of the Transformer layer of the large model, convert the real weight parameters into complex domain parameters containing amplitude and phase components through the complex domain mapping function, introduce a regularization term to constrain the mapping process to ensure the integrity of parameter information, and output the complex domain parameters, amplitude components and phase components. S12: Construct a bit allocation model based on reinforcement learning, inputting the network layer type, task type, amplitude and phase distribution characteristics of S11, and hardware resource utilization rate, and outputting the quantization bit configuration scheme of each Transformer layer; S13: Input the amplitude component and phase component of S11, the quantization bit configuration scheme of S12 and the verification dataset, quantize the amplitude and phase respectively and then dequantize to obtain real number weight parameters, evaluate the accuracy and parameter distribution differences in combination with the verification dataset and dynamically adjust the quantization granularity, and output the quantized real number weight parameters and the final quantization configuration scheme. S2: Construct a three-layer architecture including a resource layer, a scheduling layer, and an application layer. Input the quantized real-valued weight parameters from S13, the user inference request, and the hardware status information. Complete the pre-filling and decoding stage tasks through three-layer collaborative scheduling, perform heterogeneous collaborative inference based on PD separation, and output the inference results and task execution status.

[0009] By adopting the above technical solutions, the core method for accelerating lightweight large model inference is realized as a whole. High-precision compression of model weights is achieved through complex domain enhanced ultra-low bit quantization. Then, combined with the three-layer architecture of resource layer, scheduling layer and application layer, heterogeneous collaborative inference with PD separation is completed. A collaborative link of quantization optimization and heterogeneous inference is constructed to achieve efficient inference with low precision loss.

[0010] As a further aspect of the present invention: in S11, the transformation logic of the complex domain mapping function includes: taking the ratio of the absolute value of the real number weight parameter to the maximum value of the absolute value as the amplitude component; taking the ratio of the mean of each row of the real number weight parameter to the mean of each column through arctangent operation to obtain the phase component; and obtaining the regularization term by combining the ratio of the L2 norm of the complex domain parameter to the original real number weight parameter with a preset regularization coefficient, which is used to constrain the energy difference of the parameters before and after mapping.

[0011] By adopting the above technical solution, the specific transformation logic of the complex domain mapping function is clarified, ensuring the information integrity and energy stability when transforming real parameters to the complex domain, laying the foundation for high precision in subsequent ultra-low bit quantization, and improving the reliability and rationality of complex domain representation transformation.

[0012] As a further aspect of the present invention: in S12, the training process of the bit allocation model includes: Collect diverse task datasets and corresponding large model weight parameters, distribution characteristics, accuracy loss rate and resource consumption rate under different bit configurations to construct a training sample set; An improved deep Q-network architecture is adopted, which encodes the network layer type and task type and concatenates them with the distribution characteristics and the normalized result of resource utilization as the input layer; The model parameters are optimized using the temporal difference loss function, and training is completed using a preset learning rate, batch size, and number of iterations. The reward function is a weighted sum of the accuracy weight, the efficiency weight, the accuracy guarantee coefficient, and the resource saving coefficient, respectively.

[0013] By adopting the above technical solutions, the training process and core design of the bit allocation model are refined to ensure that the model can accurately learn the matching relationship between task type, network layer features and hardware status, output the optimal quantized bit configuration, realize task-aware dynamic bit allocation, and balance accuracy and resource consumption.

[0014] As a further aspect of the present invention: In S13, the quantization process includes non-uniform quantization of the amplitude component and uniform quantization of the phase component: the amplitude component is statistically divided into high-value and low-value regions, and fine-grained and coarse-grained quantization are used respectively; the phase component is uniformly quantized after determining the quantization interval according to the value range and quantization level, and the quantization logic is that the ratio of the component to the quantization interval is rounded and then multiplied by the quantization interval.

[0015] By adopting the above technical solution, a differentiated quantization strategy for amplitude and phase components is determined. By combining non-uniform and uniform quantization, the accuracy loss of key components is reduced while ensuring quantization efficiency, thereby improving the overall effect of complex domain quantization.

[0016] As a further aspect of the present invention: In S13, the accuracy calibration logic includes: obtaining the real number weight parameters by quantizing the amplitude component, the maximum absolute value of the original real number weight parameters, and the sign dequantization; calculating the distribution difference and inference accuracy between the original real number weight parameters and the dequantized parameters; if the difference and accuracy loss exceed a preset threshold, adjusting the quantization granularity and requantizing until the requirements are met.

[0017] By adopting the above technical solution, the core logic of accuracy calibration is clarified. Through inverse quantization verification and threshold judgment, the dynamic adjustment of quantization granularity is achieved, ensuring that the inference accuracy and parameter distribution stability of the quantized model meet the requirements and avoiding the sharp drop in accuracy caused by ultra-low bit quantization.

[0018] As a further aspect of the present invention: in S2, the three-layer architecture collaborative scheduling logic includes: The resource layer receives scheduling instructions, deploys pre-filled tasks to the GPU cluster and decoding tasks to the FPGA cluster, transmits data through the high-speed interconnect module and feeds back the hardware status; The scheduling layer constructs a dynamic scheduling model based on reinforcement learning, inputs the application layer's inference tasks and the resource layer's hardware status, and achieves load scheduling through task classification and sorting, batch adaptive adjustment, and elastic resource allocation. The application layer receives user requests, calls the quantization tool to execute S1, forwards the inference task, and feeds back the inference results and abnormal alarm information.

[0019] By adopting the above technical solutions, the collaborative scheduling logic of the three-layer architecture is refined, enabling heterogeneous hardware-oriented deployment at the resource layer, load adaptation scheduling at the scheduling layer, and task flow control at the application layer. This ensures the orderly and efficient execution of PD separate inference and improves the smoothness and controllability of task processing.

[0020] As a further aspect of this invention: the dynamic scheduling model adopts the PPO reinforcement learning architecture. The input layer is a feature concatenation of task type, priority, batch size, hardware status, and task execution status. The parameters are optimized through a hybrid loss function containing policy loss, value loss, and entropy loss. The batch adaptive adjustment dynamically adjusts the batch size in the pre-filling stage according to the GPU utilization rate. In high-concurrency scenarios, resource elastic allocation enables dynamic migration of tasks between the FPGA and CPU clusters based on the FPGA cache occupancy rate.

[0021] By adopting the above technical solutions, the architecture and core control strategies of the dynamic scheduling model are clarified, enabling precise scheduling based on hardware load and task characteristics. Through batch adaptation and elastic resource migration, the inference stability and resource utilization in high-concurrency scenarios are improved.

[0022] This invention also discloses a lightweight inference acceleration system for large models, comprising a complex domain quantization module, a heterogeneous resource layer, a dynamic scheduling module, and an application layer, with each module working together to form a closed-loop system. The complex domain quantization module is used to perform the complex domain enhanced ultra-low bit quantization step of S1 above. It takes real number weight parameters, parameter distribution statistics and verification dataset as input, and outputs the quantized real number weight parameters and the final quantization configuration scheme. The heterogeneous resource layer, including GPU clusters, FPGA clusters and high-speed interconnect modules, is used to receive scheduling instructions to complete the pre-filling and decoding stage tasks, and output inference intermediate results, final inference token sequence and hardware status information. The dynamic scheduling module builds a scheduling model based on reinforcement learning, takes application layer inference tasks and heterogeneous resource layer hardware status information as input, and outputs scheduling instructions to achieve load-aware dynamic scheduling. The application layer is used to receive user inference requests, call the complex domain quantization module to execute the quantization process, forward inference tasks and provide feedback on inference results, task execution reports and abnormal alarm information. The output of the complex domain quantization module is transmitted to the heterogeneous resource layer and the dynamic scheduling module. The hardware status information of the heterogeneous resource layer is transmitted to the dynamic scheduling module, forming a logical closed loop.

[0023] By adopting the above technical solutions, a closed-loop system is constructed to realize the aforementioned inference acceleration method. Through multi-domain quantization, heterogeneous resources, dynamic scheduling, and application layer module collaboration, it accepts user requests and completes the entire quantization inference process, ensuring the feasibility and systematic nature of the method's implementation.

[0024] As a further aspect of the present invention: the complex domain quantization module includes a complex domain mapping unit, a bit allocation unit, and a quantization calibration unit: the complex domain mapping unit performs complex domain representation transformation of real number parameters and outputs complex domain parameters, amplitude, and phase components; the bit allocation unit deploys the bit allocation model and outputs a quantization bit configuration scheme; the quantization calibration unit performs amplitude non-uniform quantization and phase uniform quantization, and adjusts the quantization granularity through inverse quantization and accuracy evaluation.

[0025] By adopting the above technical solutions, the internal unit division of the complex domain quantization module is refined, making the functional decomposition of complex domain mapping, bit allocation and quantization calibration clearer, ensuring the orderly execution of the complex domain enhanced ultra-low bit quantization steps, and improving the controllability and efficiency of module operation.

[0026] As a further aspect of the present invention: the FPGA cluster of the heterogeneous resource layer integrates a dedicated complex domain quantization operation unit, which supports multi-bit complex multiplication and addition operations; the high-speed interconnect module adopts PCIe and InfiniBand interconnect technology combined with data compression algorithms to achieve low-latency data transmission between the GPU and the FPGA cluster.

[0027] By adopting the above technical solutions, the hardware configuration of the heterogeneous resource layer is optimized. By combining dedicated computing units with high-speed interconnect technology, the computing efficiency and data transmission rate of multi-domain quantization inference are improved, ensuring low latency and high reliability of heterogeneous collaborative inference.

[0028] Compared with the prior art, the beneficial effects of the present invention are: This invention achieves high precision preservation under ultra-low bit quantization by combining complex domain representation transformation with dynamic bit allocation and precision calibration, effectively reducing the memory usage of large models, solving the technical pain point of excessive precision loss in low bit quantization, and breaking through the limitations of traditional real number field quantization.

[0029] This invention, based on a three-layer heterogeneous architecture with separate product and processing (PD) layers and a reinforcement learning scheduling strategy, achieves precise matching between hardware resources and task requirements, improves inference speed and stability in high-concurrency scenarios, reduces hardware deployment costs, and solves the problems of low inference efficiency and coarse scheduling in traditional homogeneous architectures. Other features and advantages of the present invention will be disclosed in detail in the following detailed description and accompanying drawings. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the overall structure of a lightweight large model inference acceleration method according to an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Example 1 In this embodiment of the invention, a lightweight method for accelerating inference of large models is described, see [link to relevant documentation]. Figure 1 As shown, it includes two core components: a complex-domain enhanced ultra-low bit quantization method and a heterogeneous collaborative inference architecture and intelligent scheduling system based on PD separation. The two work together to form a complete technical system of quantization optimization, heterogeneous scheduling, and end-to-end acceleration.

[0033] The specific steps are as follows: Step 1: Complex Domain Representation Transformation of Real Parameters The core objective of this step is to map the real weight parameters of a large model to the complex domain, expand the parameter expression space through two degrees of freedom of amplitude and phase, and reduce the accuracy loss of ultra-low bit quantization.

[0034] Input: Real weight parameters W of the Transformer layer of the large model, with dimensions of... Parameter distribution statistics, including the maximum value of the weights. Row mean and column mean, etc. In one feasible embodiment, when When the value is 4096, the weight parameter W is a 4096×4096 matrix.

[0035] Implementation details: Introducing complex domain mapping functions Convert the real number parameter W into a complex field parameter. , where A is the amplitude component and B is the phase component.

[0036] The specific calculation method is as follows: Amplitude A calculation: The method normalizes the absolute value of the original real parameter by dividing it by the maximum value of the absolute value of the weight, resulting in an amplitude value within the range of [0,1], which represents the intensity information of the parameter. The innovation of this calculation method is that it ensures the numerical stability of the amplitude component through normalization, avoiding discretization errors caused by excessively large numerical ranges during subsequent quantization.

[0037] Phase B calculation: ,in Let W be the mean of each row of the weight matrix. Let W be the mean of each column of the weight matrix; calculate the phase angle by the ratio of the row and column means, construct the phase dimension, and expand the parameter expression space; when When the value is 0, to avoid the denominator being 0, set... ;when When the value is 0, set B=0.

[0038] Mapping constraints: Introducing L2 regularization terms This is used to constrain the integrity of parameter information during complex domain mapping, avoiding energy loss of parameters after mapping; where λ=0.01, this value is the optimal regularization coefficient verified through extensive experiments; the calculation logic of the regularization term is as follows: first calculate the L2 norm of the complex domain parameter Z and the L2 norm of the original real parameter W respectively, then calculate the ratio of the two and multiply it by the regularization coefficient λ; the calculation method of the L2 norm of the complex domain parameter Z is as follows. The L2 norm of the original real parameter W is calculated as follows: This constraint ensures that the energy difference of the parameters before and after mapping is controlled within 1%.

[0039] Output: Complex domain parameter Z, amplitude component A, and phase component B, where the complex domain parameter Z has the same dimension as the weight parameter W. The complex matrix; both amplitude component A and phase component B are A real matrix.

[0040] Step 2: Task-Aware Dynamic Bit Allocation This step constructs a bit allocation model based on reinforcement learning, dynamically allocating the optimal quantized bits according to the task type and network layer type. The input is the complex domain parameter information and task characteristics output from step 1, and the output is the bit configuration scheme for each layer.

[0041] Core component: Task-aware bit allocation reinforcement learning model, or bit allocation model for short; Input: Current network layer type, including attention layer, FeedForward layer and layer normalization layer; task type, including classification, question answering and generation; distribution characteristics of amplitude A and phase B output from step 1, such as the proportion of high value regions and variance; hardware resource utilization, including the computing power utilization of GPU and FPGA and the utilization of video memory.

[0042] Training process of bit allocation model: 1) Training data preparation: Collect diverse task datasets, including the GLUE dataset for text classification, the MMLU dataset for question answering, and the CNN / Daily Mail summarization task for text generation.

[0043] For each task, GLM-4-9B and Llama3-8B were selected as benchmark models. The weight parameters W, parameter distribution characteristics, accuracy loss rate and resource utilization rate under different bit configurations of each Transformer layer were extracted to construct a training sample set. In one feasible implementation, the number of training samples is no less than 100,000.

[0044] Model Structure: The model employs an improved Deep Q-Network architecture. The input layer has a dimension of 128, constructed by one-hot encoding the network layer type and task type, normalizing the parameter distribution features and resource utilization, and then concatenating the results. The total dimension is 12, specifically 3D layer type, 3D task type, 4D distribution features, and 2D resource utilization. The hidden layers consist of two fully connected layers with dimensions of 256 and 128 respectively. The output layer has a dimension of 3, corresponding to three bit selections: 1 bit, 2 bits, and 4 bits.

[0045] Training parameter settings: learning rate is 0.001, batch size is 64, training iterations are 10,000, experience replay pool size is 100,000, and target network update frequency is once every 100 iterations.

[0046] Loss function: Temporal difference loss function is used. Where r is the reward function value, and γ is the discount factor with a value of 0.9. The Q value of the current network. The Q-value of the target network.

[0047] Reward function design: , where α=0.7, as the accuracy weight; β=0.3, as the efficiency weight.

[0048] The innovation of this design lies in prioritizing inference accuracy while also considering resource efficiency.

[0049] The accuracy loss rate is calculated as the absolute value of the ratio of (quantized model accuracy - original model accuracy) to the original model accuracy; the resource utilization rate is calculated as the ratio of (quantized memory utilization + quantized computing power utilization) to (original model memory utilization + original model computing power utilization).

[0050] Training process: Initialize the parameters of the current network and the target network; A batch of samples (s, a, r, s') is randomly sampled from the training sample set, where s is the current state, including layer type, task type, etc.; a is the bit selection action; r is the reward value; and s' is the next state. Input s into the current network and get ; Input s' into the target network to obtain ; Calculate the loss function L and optimize the current network parameters using gradient descent; The target network parameters are updated every 100 iterations; Repeat steps (2) to (6) until the training iterations are completed and the reward value on the validation set converges. The convergence condition is that the reward value fluctuation over 100 consecutive iterations does not exceed 0.01.

[0051] Dynamic bit allocation implementation: After offline training is complete, the bit allocation model is deployed to the inference process; During the inference phase, the current layer type, task type, distribution characteristics of amplitude A and phase B, and hardware resource utilization are input in real time, i.e., state s; The bit allocation model outputs the Q value of each candidate bit, including 1 bit, 2 bits, and 4 bits, and selects the bit with the largest Q value as the quantization bit of the current layer. Candidate bit range constraints are set for different layers. The candidate bits for the attention layer are 2 bits and 4 bits. Since the attention layer has a large impact on the accuracy, 1-bit quantization is prohibited. The candidate bits for the layer normalization layer are 1 bit and 2 bits. Since the parameters of the layer normalization layer have a small impact on the accuracy, a lower bit range can be used. The candidate bits for the FeedForward layer are 1 bit, 2 bits and 4 bits.

[0052] Output: The quantization bit configuration scheme for each Transformer layer. In one feasible embodiment, the configuration scheme is 4 bits for the attention layer, 2 bits for the FeedForward layer, and 1 bit for the layer normalization layer.

[0053] Step 3: Complex Domain Quantization and Accuracy Calibration This step quantizes the amplitude A and phase B output from step 1 according to the bit configuration scheme output from step 2, and then adjusts the quantization granularity through a precision calibration feedback mechanism to ensure that the precision loss meets the requirements.

[0054] Among them, accuracy calibration refers to the optimization process of adjusting quantization parameters to control accuracy loss through inverse quantization and performance verification.

[0055] Input: Amplitude A and phase B from step 1; bit configuration scheme for each layer from step 2; validation dataset for a large model, such as the validation dataset for GLUE.

[0056] Specific implementation: 1) Amplitude quantification: For amplitude A, first determine the high-value area and low-value area through statistics, with the high-value area accounting for 20% and the low-value area accounting for 80%.

[0057] The criterion for determining high-value regions is A≥0.8×max(A). This threshold can accurately classify high-value parameters that are critical to accuracy. This conclusion is verified through sample statistics.

[0058] High-value regions employ fine-grained quantization. In one feasible embodiment, when the bit configuration is 4 bits, the quantization level is 16 levels, and the quantization interval is... The low-value region is A < 0.8 × max(A), and coarse-grained quantization is used. In one feasible embodiment, when the bit configuration is 1 bit, the quantization level is 2 levels and the quantization interval is... The quantification formula is: ,in This represents the quantization interval for the corresponding region, and round is the rounding function.

[0059] 2) Phase Quantization: Since phase B has a relatively low impact on accuracy, a uniform quantization strategy is adopted. In one feasible embodiment, when the bit configuration is 2 bits, the quantization level is 4 levels, and the quantization interval is... The range of phase B, calculated using arctan, is: The quantification formula is: After quantization, it is mapped to a set of discrete values. This set corresponds to 4 quantization levels.

[0060] 3) Complex domain quantization parameter output: Outputs the quantized amplitude. and phase Recombined into quantized complex-domain parameters .

[0061] 4) Accuracy calibration feedback mechanism: ① Dequantization: Dequantizing the complex domain parameters Through the inverse mapping function Convert to real number parameters The inverse mapping function is ,in The sign of the original real number parameter W is set to ensure that the sign of the parameter after dequantization is consistent with the original parameter.

[0062] ②Accuracy assessment: Substitute the data into the large model, perform inference on the validation dataset, and calculate the inference accuracy; simultaneously calculate the original real parameter W and the dequantized parameter. The KL divergence is used to measure the difference in parameter distributions, and it is calculated as follows: ,when The limit is 0.

[0063] ③ Feedback Adjustment: Set the KL divergence threshold to 0.05. This threshold was determined through a large number of experiments and corresponds to an accuracy loss of no more than 2%. If the KL divergence exceeds the threshold or the accuracy loss on the validation set exceeds 2%, adjust the quantization granularity of the corresponding layer. For example, increase the quantization bits in the high-value region by 1 level, changing 1 bit to 2 bits, and re-execute the quantization process in step 3. If the KL divergence does not exceed 0.05 and the accuracy loss does not exceed 2%, then the quantization is complete.

[0064] Output: Quantized real-valued weight parameters and the final scheme for quantization configuration at each layer, including the quantized real-valued weight parameters. It can be used directly for large model inference.

[0065] This invention provides a lightweight inference acceleration system for large models, specifically comprising the following components: The P / D phase achieves heterogeneous resource-oriented adaptation and dynamic load scheduling through a three-layer architecture consisting of a resource layer, a scheduling layer, and an application layer. The three layers collaborate from top to bottom: the scheduling layer receives task requests from the application layer and performs scheduling based on the hardware status of the resource layer; the resource layer executes tasks according to the scheduling instructions. The specific architecture and implementation are as follows: 1. Resource Layer: Targeted Deployment of Heterogeneous Hardware The core objective of this layer is to deploy heterogeneous hardware in a targeted manner based on the differences in resource requirements between the computationally intensive P phase and the storage-intensive D phase, thereby achieving precise resource matching.

[0066] Input: Task allocation instructions issued by the scheduling layer, which specify that tasks in the P stage should be allocated to the GPU cluster and tasks in the D stage should be allocated to the FPGA cluster.

[0067] In this embodiment, the specific implementation includes: 1) P-stage pre-filled deployment: NVIDIA A100 GPU cluster is used. This cluster is an 8-card cluster with 80GB of video memory per card, which is suitable for computing-intensive needs.

[0068] A single GPU is responsible for token embedding and initial attention calculation of batch text. It supports tensor parallel acceleration between GPUs and is implemented using the NCCL communication library, with a communication bandwidth of no less than 300GB / s.

[0069] The specific task allocation is to process 16 batches of pre-filling tasks per GPU, with a batch size of 32, meaning that each GPU can process 32 × 16 = 512 texts of pre-filling simultaneously.

[0070] The input is the original text token sequence, and the output is the pre-filled KV cache initial value and the initial hidden state. The KV cache refers to the intermediate result cache that stores the keys and values ​​in the attention mechanism, which is used to accelerate the inference process in the decoding stage.

[0071] 2) Decoding deployment in stage D: Xilinx Alveo U55C FPGA dedicated hardware is used, which is an 8-FPGA cluster adapted to storage-intensive requirements.

[0072] The FPGA integrates a dedicated complex domain quantization operation unit. Specifically, the complex number operation module is designed using Verilog language, supporting complex multiplication and addition operations of INT1, INT2, and INT4 bits with an operation delay of no more than 1ns. The quantized KV pairs output from step 3 are cached using the FPGA's highly parallel storage unit, which has a capacity of no less than 1TB.

[0073] The specific task allocation is to process 32 decoding tasks for each FPGA. The input is the initial KV buffer and initial hidden state output from the P stage, and the output is the decoded token sequence.

[0074] 3) High-speed interconnect module: PCIe 5.0 and InfiniBand HDR high-speed interconnect module are adopted to realize low-latency data transmission between GPU cluster and FPGA cluster.

[0075] PCIe 5.0 has a transfer rate of 32GT / s, while InfiniBand HDR has a transfer rate of 200GB / s. Data compression further reduces transmission latency by using the LZ4 compression algorithm with a compression ratio of no less than 2:1, ensuring that the data transmission latency in the P / D stage does not exceed 1μs.

[0076] Output: The initial KV cache and initial hidden state of the P stage are transmitted to the D stage; the decoded token sequence of the D stage is transmitted to the application layer; hardware status information, including the computing power utilization of GPU and FPGA, video memory and storage occupancy, and network bandwidth, is transmitted to the scheduling layer.

[0077] 2. Scheduling Layer: Load-Aware Dynamic Scheduling Algorithm This layer uses reinforcement learning to build a dynamic scheduling model, which collects the hardware status of the resource layer and the task requests of the application layer in real time, and dynamically adjusts the task allocation strategy to ensure low latency and stability in high-concurrency scenarios.

[0078] Input: Inference tasks issued by the application layer, including task type, priority, and batch size; hardware status information reported by the resource layer, including GPU computing power utilization, FPGA cache utilization, network bandwidth, task queue length, and single task latency.

[0079] Core component: Load-aware dynamic scheduling reinforcement learning model In this embodiment, the training process of the load-aware dynamic scheduling reinforcement learning model includes: 1) Training data preparation: Construct a load dataset that simulates a high-concurrency scenario, containing inference tasks of different priorities.

[0080] High-priority tasks are real-time dialogues with a latency requirement of no more than 50ms; medium-priority tasks are batch summarization with a latency requirement of no more than 500ms; and low-priority tasks are text generation with a latency requirement of no more than 1s.

[0081] In one feasible embodiment, the number of tasks is no less than 100,000.

[0082] Simulate task execution under different hardware conditions, including GPU utilization of 30%-100% and FPGA cache utilization of 20%-90%. Record data such as task allocation strategy, hardware status, task completion latency, and latency fluctuation to build a training sample set.

[0083] 2) Model structure: The PPO (Proximal Policy Optimization) reinforcement learning architecture is adopted. The state input layer has a dimension of 20. Its construction method is a feature concatenation of 3D task type, 3D priority, 1D batch size, 1D GPU utilization, 1D FPGA cache utilization, 1D network bandwidth, 3D task queue length, 3D single task latency, and 3D latency fluctuation.

[0084] The hidden layer is a 2-layer LSTM with a dimension of 256. The output layers are a policy network and a value network. The policy network outputs the task assignment probability, such as the probability of assigning to GPU1 in the P stage and the probability of assigning to FPGA2 in the D stage. The value network outputs the state value.

[0085] 3) Training parameter settings: learning rate is 0.0003, batch size is 256, training iterations are 5000, and discount factor is [not specified]. , gae_lambda=0.95, clip_range=0.2, where clip_range is the clipping range for policy updates.

[0086] 4) Loss function: The PPO hybrid loss function is adopted. ,in For strategic losses, For the loss of value, The entropy loss is set to 0.01, which is used to encourage exploration.

[0087] 5) Training process: ① Initialize the parameters of the policy network and value network; ② Collect a batch of trajectories (s,a,r,s'), where s is the state, a is the action, r is the reward, and s' is the next state; ③ Calculate the advantage function A and the return G; ④ Update the policy network based on the advantage function, and avoid excessive policy update magnitude through a pruning mechanism; ⑤ Update the value network to minimize value loss; ⑥ Repeat steps ② to ⑤ until the training iteration is completed and the average latency fluctuation on the validation set does not exceed 5%.

[0088] In this embodiment, the implementation of the dynamic scheduling strategy includes: Task classification and priority sorting: After receiving inference tasks from the application layer, tasks are classified according to task type and latency requirements. Task types include dialogue, summary, and generation. Priority is marked as high, medium, and low. Priority queue management is adopted, and high-priority tasks are queued first for execution.

[0089] Batch adaptive adjustment: The batch size in the P stage is dynamically adjusted according to the GPU utilization. When the GPU utilization is below 60%, the batch size is increased from 32 to 64; when the GPU utilization is above 90%, the batch size is split from 32 to 16; when the GPU utilization is between 60% and 90%, the batch size is kept at 32. The innovation of this adjustment strategy is to balance the GPU utilization and task latency through dynamic batch adjustment.

[0090] Elastic resource allocation: In high-concurrency scenarios, i.e., when QPS is not less than 1000, the FPGA cache utilization rate is monitored in real time. If the utilization rate exceeds 85%, some low-priority D-stage tasks are temporarily migrated to the CPU cluster. The CPU cluster uses Intel Xeon Gold 6330 CPUs, which support AVX-512 instruction set acceleration, to ensure the FPGA resource utilization of high-priority tasks. When the FPGA cache utilization rate is less than 60%, the tasks migrated to the CPU are migrated back to the FPGA to improve execution efficiency.

[0091] MoE Model Adaptation: A new dynamic expert network distribution module has been added. The MoE model, or Hybrid Expert Model, refers to a model structure containing multiple expert networks and a gating network. The gating network is responsible for distributing input to different expert networks for processing. This module allocates the expert networks of the MoE model to corresponding hardware based on the task type: computationally intensive experts are assigned to GPU clusters (e.g., text generation experts), while storage-intensive experts are assigned to FPGA clusters (e.g., text classification experts). An expert load balancing algorithm ensures balanced load across hardware. The core logic of this algorithm is to calculate the task execution time of each expert and dynamically adjust the hardware allocation for each expert.

[0092] Output: Task allocation instructions, batch size adjustment instructions, and expert network hardware allocation instructions. The task allocation instructions specify the P-stage GPU node and D-stage FPGA or CPU node for each task.

[0093] 3. Application Layer: Integrated Inference Scheduling Platform This layer integrates four major modules to realize the reception of inference tasks, the invocation of quantification tools, accuracy monitoring, and operation and maintenance management. It is the core layer for user interaction with the system.

[0094] Input: The user's inference request, including the original text, task type, and latency requirements; the task execution status reported by the scheduling layer; and the inference accuracy data collected by the accuracy monitoring module.

[0095] Specific module implementation: The quantization tool module provides a one-click interface for complex-domain quantization, supporting automatic quantization of mainstream large models. After the user inputs the original model file, the module automatically calls the three-step quantization process described in Section 3.2.1 and outputs the quantized model file. The interface format is a RESTful API, supporting batch quantization task submission.

[0096] Task scheduling module: Receives user inference requests, parses task type and latency requirements, marks priorities and sends them to the scheduling layer; receives task execution status from the scheduling layer and returns inference results and task execution latency to the user.

[0097] Accuracy monitoring module: Real-time acquisition of accuracy data of inference results, classification task acquisition accuracy, generation of task acquisition BLEU value, and comparison with preset accuracy threshold. In one feasible embodiment, the accuracy threshold is set to an accuracy rate not lower than 98% of the original model's accuracy rate. If the accuracy loss exceeds the threshold, an accuracy calibration feedback mechanism is automatically triggered to readjust the quantization granularity and send a resource adjustment request to the scheduling layer, such as increasing the hardware resource priority of the corresponding task.

[0098] The operations and maintenance management module provides a hardware status monitoring interface, displaying real-time utilization, temperature, and voltage of GPUs, FPGAs, and CPUs; it offers task log query functionality, recording the execution time, resource consumption, and accuracy loss of each task; and it provides fault alarm functionality, sending alerts via email or SMS when hardware failures or task latency exceed limits. This module supports elastic cluster scaling, automatically adding GPU and FPGA nodes when the task load increases and releasing nodes when the task load decreases.

[0099] Output: User inference results, task execution report; accuracy anomaly alarm information; hardware fault alarm information.

[0100] The complete technical system of this invention forms a closed-loop logic of user request, application layer processing, scheduling layer scheduling, resource layer execution, quantitative optimization, and result feedback, specifically including: The user submits an inference request to the application layer, with the input being the original text and the task type; The application layer calls the quantization tool module to perform complex-domain ultra-low bit quantization on the large model. The input is the original model, and the output is the quantized model. The application layer sends the quantized inference task to the scheduling layer, with the input being the quantized model, task type, and priority; The scheduling layer collects the hardware status of the resource layer, generates task allocation instructions through a dynamic scheduling model, and outputs the allocation instructions. The GPU cluster in the resource layer performs the P-stage pre-filling, with the input being the quantized model and the original text, and the output being the initial KV buffer; the FPGA cluster performs the D-stage decoding, with the input being the initial KV buffer and the output being the decoded token sequence. The resource layer returns the decoding result to the application layer, and at the same time feeds back the hardware status to the scheduling layer; The application layer's precision monitoring module verifies the precision of the decoding result. The inputs are the decoding result and the verification dataset, and the output is the precision data. If the precision is abnormal, quantization optimization is triggered. The application layer returns the inference results to the user, with the output being the decoded token sequence and execution latency.

[0101] This invention provides a lightweight inference acceleration method for large models, which can solve the technical problems of large precision loss in ultra-low bit compression, rough system resource adaptation, and performance fluctuation in high-concurrency scenarios, and has high reliability.

[0102] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0103] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A lightweight method for accelerating inference in large models, characterized in that, Includes the following steps: S1: Perform complex-domain enhanced ultra-low bit quantization, including complex-domain representation transformation of real parameters, task-aware dynamic bit allocation, and complex-domain quantization and precision calibration, specifically including: S11: Input the real weight parameters and parameter distribution statistics of the Transformer layer of the large model, convert the real weight parameters into complex domain parameters containing amplitude and phase components through the complex domain mapping function, introduce a regularization term to constrain the mapping process to ensure the integrity of parameter information, and output the complex domain parameters, amplitude components and phase components. S12: Construct a bit allocation model based on reinforcement learning, inputting the network layer type, task type, amplitude and phase distribution characteristics of S11, and hardware resource utilization rate, and outputting the quantization bit configuration scheme of each Transformer layer; S13: Input the amplitude component and phase component of S11, the quantization bit configuration scheme of S12 and the verification dataset, quantize the amplitude and phase respectively and then dequantize to obtain real number weight parameters, evaluate the accuracy and parameter distribution differences in combination with the verification dataset and dynamically adjust the quantization granularity, and output the quantized real number weight parameters and the final quantization configuration scheme. S2: Construct a three-layer architecture including a resource layer, a scheduling layer, and an application layer. Input the quantized real-valued weight parameters from S13, the user inference request, and the hardware status information. Complete the pre-filling and decoding stage tasks through three-layer collaborative scheduling, perform heterogeneous collaborative inference based on PD separation, and output the inference results and task execution status.

2. The lightweight large model inference acceleration method according to claim 1, characterized in that, In S11, the transformation logic of the complex domain mapping function includes: taking the ratio of the absolute value of the real number weight parameter to the maximum value of the absolute value as the amplitude component; taking the ratio of the mean of each row of the real number weight parameter to the mean of each column through arctangent operation to obtain the phase component; and taking the regularization term by combining the ratio of the L2 norm of the complex domain parameter to the original real number weight parameter with a preset regularization coefficient, which is used to constrain the energy difference of the parameters before and after the mapping.

3. The lightweight large model inference acceleration method according to claim 1, characterized in that, In S12, the training process of the bit allocation model includes: Collect diverse task datasets and corresponding large model weight parameters, distribution characteristics, accuracy loss rate and resource consumption rate under different bit configurations to construct a training sample set; An improved deep Q-network architecture is adopted, which encodes the network layer type and task type and concatenates them with the distribution characteristics and the normalized result of resource utilization as the input layer; The model parameters are optimized using the temporal difference loss function, and training is completed using a preset learning rate, batch size, and number of iterations. The reward function is a weighted sum of the accuracy weight, the efficiency weight, the accuracy guarantee coefficient, and the resource saving coefficient, respectively.

4. The lightweight large model inference acceleration method according to claim 1, characterized in that, In S13, the quantization process includes non-uniform quantization of the amplitude component and uniform quantization of the phase component: the amplitude component is statistically divided into high-value and low-value regions, and fine-grained and coarse-grained quantization are used respectively. Phase components are uniformly quantized after the quantization interval is determined according to the value range and quantization level. The quantization logic is to round the ratio of the component to the quantization interval and then multiply it by the quantization interval.

5. The lightweight large model inference acceleration method according to claim 1, characterized in that, In S13, the accuracy calibration logic includes: obtaining the real number weight parameters by quantizing the amplitude component, the maximum absolute value of the original real number weight parameters, and the sign dequantization; calculating the distribution difference and inference accuracy between the original real number weight parameters and the dequantized parameters; if the difference or accuracy loss exceeds a preset threshold, adjusting the quantization granularity and requantizing until the requirements are met.

6. The lightweight large model inference acceleration method according to claim 1, characterized in that, In S2, the three-layer architecture collaborative scheduling logic includes: The resource layer receives scheduling instructions, deploys pre-filled tasks to the GPU cluster and decoding tasks to the FPGA cluster, transmits data through the high-speed interconnect module and feeds back the hardware status; The scheduling layer constructs a dynamic scheduling model based on reinforcement learning, inputs the application layer's inference tasks and the resource layer's hardware status, and achieves load scheduling through task classification and sorting, batch adaptive adjustment, and elastic resource allocation. The application layer receives user requests, calls the quantization tool to execute S1, forwards the inference task, and feeds back the inference results and abnormal alarm information.

7. The lightweight large model inference acceleration method according to claim 6, characterized in that, The dynamic scheduling model adopts the PPO reinforcement learning architecture. The input layer is a feature concatenation of task type, priority, batch size, hardware status and task execution status. The parameters are optimized through a hybrid loss function containing policy loss, value loss and entropy loss. The batch adaptive adjustment dynamically adjusts the batch size in the pre-filling stage according to the GPU utilization. In high-concurrency scenarios, resource elastic allocation enables dynamic migration of tasks between FPGA and CPU clusters based on FPGA cache utilization.

8. A lightweight inference acceleration system for large models, characterized in that, It includes a complex domain quantization module, a heterogeneous resource layer, a dynamic scheduling module, and an application layer. These modules work together to form a closed-loop system. The complex domain quantization module is used to execute the complex domain enhanced ultra-low bit quantization step as described in any one of claims 1-5, inputting real number weight parameters, parameter distribution statistics and verification dataset, and outputting quantized real number weight parameters and final quantization configuration scheme. The heterogeneous resource layer, including GPU clusters, FPGA clusters and high-speed interconnect modules, is used to receive scheduling instructions to complete the pre-filling and decoding stage tasks, and output inference intermediate results, final inference token sequence and hardware status information. The dynamic scheduling module builds a scheduling model based on reinforcement learning, takes application layer inference tasks and heterogeneous resource layer hardware status information as input, and outputs scheduling instructions to achieve load-aware dynamic scheduling. The application layer is used to receive user inference requests, call the complex domain quantization module to execute the quantization process, forward inference tasks and provide feedback on inference results, task execution reports and abnormal alarm information. The output of the complex domain quantization module is transmitted to the heterogeneous resource layer and the dynamic scheduling module. The hardware status information of the heterogeneous resource layer is transmitted to the dynamic scheduling module, forming a logical closed loop.

9. The lightweight large-model inference acceleration system according to claim 8, characterized in that, The complex domain quantization module includes a complex domain mapping unit, a bit allocation unit, and a quantization calibration unit: the complex domain mapping unit performs complex domain representation transformation of real parameters and outputs complex domain parameters, amplitude, and phase components; The bit allocation unit deploys the bit allocation model and outputs the quantization bit configuration scheme; the quantization calibration unit performs amplitude non-uniform quantization and phase uniform quantization, and adjusts the quantization granularity through inverse quantization and accuracy evaluation.

10. The lightweight large-model inference acceleration system according to claim 8, characterized in that, The FPGA cluster in the heterogeneous resource layer integrates a dedicated complex domain quantization arithmetic unit, supporting multi-bit complex multiplication and addition operations; the high-speed interconnect module uses PCIe and InfiniBand interconnect technologies combined with data compression algorithms to achieve low-latency data transmission between the GPU and the FPGA cluster.

Images