Residual quantization-based self-inference decoding large model inference method and system

Abstract

The application provides a residual quantization-based self-guessing decoding large model inference method and system, and belongs to the technical field of large language model inference optimization. The method comprises the following steps: S1: loading floating-point weights, smoothing abnormal values, and residual quantization to obtain shared weights and exclusive residual weights; S2: generating candidate tokens for a draft model; S3: using the shared weights and the residual weights, a target model verifies the candidate tokens in parallel using a fusion quantization operator; S4: if all the candidate tokens are valid, the verification result is written into a KV cache; if an invalid token appears, the KV cache is rolled back to a position before the first invalid token, and is regenerated. The steps S2-S4 are repeated until the entire inference process is completed. The method can reduce the system memory occupation, improve the inference speed and accuracy, and has good hardware friendliness and universality.

Description

Technical Field

[0001] This invention belongs to the field of large language model reasoning optimization technology, and specifically relates to a self-inferential decoding large model reasoning method and system based on residual quantization. Background Technology

[0002] In recent years, large language models have achieved remarkable application results in fields such as natural language processing. The emergence of various pre-trained models has also promoted the rapid development of natural language understanding and generation tasks. However, the massive increase in the number of model parameters has also brought many core challenges to its inference deployment. Large models require a large amount of computing card resources during inference, and the storage space required for their runtime often exceeds the dedicated memory space of a single computing acceleration device. Special design of model weight bit width or scheduling methods is required to achieve deployment. At the same time, large models adopt an autoregressive iterative operation mode, generating only one output in each decoding process. The inference speed is inherently slow, and this iterative mode further increases the inference time exponentially, making it difficult to meet the rapid response requirements of real-time application scenarios such as real-time machine translation and speech recognition.

[0003] To address the aforementioned issues, model quantization and self-predictive decoding have become mainstream techniques for optimizing large-scale model inference. Model quantization, as a lossy compression method, quantizes the high-precision floating-point weights of the model into a low-bit-width form, effectively reducing memory usage and improving inference speed. 4-bit (GPTQ, AWQ) and 8-bit (SmoothQuant, llm.int8) are currently widely used quantization bit widths in the industry. However, the lower the quantization bit width, the greater the loss of model precision. Furthermore, existing quantization solutions lack differentiated designs for self-predictive decoding application scenarios and cannot adapt to the different inference needs of draft and target models. Self-predictive decoding, on the other hand, constructs a homologous draft model by lightweighting the target model. The draft model then autoregressively generates multiple candidate tokens, which are then validated in parallel by the target model. This transforms the autoregressive inference of large models into a partially parallel form, thereby accelerating inference. The end-to-end inference performance of this technology mainly depends on two core factors: the inference speedup ratio of the draft model relative to the target model and the probability that the tokens generated by the draft model are accepted by the target model. Taking QSpec (Speculative Decoding with Complementary Quantization Schemes) as an example, its draft model uses the W4A4 weight quantization scheme, while the target model uses the W4A16 quantization scheme. It attempts to accelerate inference using the NVIDIA graphics card INT4 TensorCore computing core. However, experiments have shown that in low-concurrency scenarios, the bottleneck for inference is not computing power, but memory access. Therefore, simply accelerating computation through quantization cannot achieve the expected results. Furthermore, the bit width of its target model is still 4 bits, which is prone to quantization errors.

[0004] Existing solutions combining self-predictive decoding and quantization techniques still have significant technical shortcomings, making it difficult to achieve efficient inference acceleration for large models on edge devices. Firstly, existing solutions do not separate the weight bit widths of the draft and target models, failing to leverage the lower bit weights of the draft model to increase the speedup ratio between the draft and target models. Experiments show that in model-based paper-cutting self-predictive decoding, the draft model's inference time accounts for over 75% of the total inference time. Furthermore, quantizing both weights to the same low bit width introduces significant quantization errors, leading to a loss of model inference accuracy. Increasing the quantization bit width, however, narrows the performance gap between self-predictive decoding and traditional autoregressive decoding, diminishing the speedup benefits of self-predictive decoding. While reducing the bit width of the draft model can accelerate its speed relative to the target model, the presence of outliers in the weights during quantization can cause the quantization scale to be biased by a few extreme values. Larger end values ​​compress most weight information into a smaller representation range, causing a shift in the output distribution between the draft model and the target model. This significantly reduces token acceptance rate, increasing the overhead of backoff recalculation and offsetting the acceleration effect. Furthermore, separating the weights of the draft and target models incurs additional GPU memory usage. To reduce this, the target model weights store the quantization error of the draft model weights, thus preserving the accuracy of the target model while avoiding additional GPU memory consumption. However, this separate weight storage requires the target model to call both quantized weights separately during the verification phase. Multiple operator calls introduce additional overhead such as kernel startup and repeated memory accesses, directly reducing the end-to-end efficiency of the entire inference process. Therefore, a method for accelerating inference of large language models based on residual quantization and self-predictive decoding is urgently needed. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a self-predictive decoding method for large model inference based on residual quantization, comprising the following steps:

[0006] Step S1: Load the floating-point weights of the original model, perform outlier smoothing and residual quantization in sequence to obtain shared weights and exclusive residual weights, and load the fusion quantization operator to initialize the shared KVcache space.

[0007] Step S2: The draft model uses the shared weights to quickly generate several candidate tokens in an autoregressive manner. During the generation process, historical data in the shared KVcache is directly called and the candidate tokens are passed into the target model.

[0008] Step S3: The target model uses the shared weights and residual weights to call the fusion quantization operator to perform parallel verification of the candidate tokens, output the verification results, and determine the validity of each candidate token;

[0009] Step S4: If all candidate tokens are valid, write the verification result of the target model directly to the shared KVcache; if an invalid token appears, roll back the KVcache to the position before the first invalid token, and let the draft model generate a valid token from that position through autoregression; repeat steps S2 to S4 until the target model generates an end symbol or reaches the preset maximum generation length to complete the entire inference process.

[0010] Beneficial effects:

[0011] 1. Effectively reduce memory usage: The residual quantization strategy designed in this invention uses shared weights and exclusive residual weights as parameters, eliminating the need to introduce an additional independent model for the draft model. The draft and target models share some weights, and low bit width quantization is used throughout, which greatly reduces the overall memory usage of the model and adapts to the memory limitations of edge devices.

[0012] 2. Ensure near-lossless inference accuracy: This invention restores the complete weight features close to the original model by superimposing the shared weights and exclusive residual weights of the target model. Combined with the weight outlier smoothing mechanism, it effectively controls the quantization error, ensures high similarity between the draft and the target model, improves the token acceptance rate, and makes the inference accuracy very close to the original large model, reducing the accuracy loss caused by low bit width quantization.

[0013] 3. Significantly Improved Inference Speed: The residual quantization strategy of this invention maximizes the performance gap between the draft and the target model. The fused quantization operator significantly reduces the computational overhead during the target model verification stage. The unified KVcache management mechanism reduces the cost of rollback and recalculation. These optimizations collectively improve the end-to-end inference speed of self-predictive decoding, meeting the requirements of real-time inference on the edge. Experiments demonstrate that on NVIDIA RTX4090 hardware, the inference performance of the Llama-3-8B model is improved by 104% compared to autoregressive inference.

[0014] 4. High hardware friendliness: The integrated quantization operator of this invention makes full use of the computing units and cache resources of the edge GPU, improving cache reuse efficiency and computing resource utilization. The entire technical solution is based on mainstream deep learning frameworks and operator libraries, and can be directly deployed on mainstream edge computing acceleration devices such as NVIDIA RTX4090 without the need for dedicated hardware support, making deployment easy.

[0015] 5. High versatility: This invention does not require retraining or fine-tuning of the original large model. Weight quantization and integration with the self-predictive decoding framework can be completed through post-processing alone. It can be adapted to mainstream large language models such as Llama and Qwen, supports flexible adjustment of different quantization bit widths and candidate token generation numbers, and can be quickly migrated to different edge inference application scenarios. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the process of a self-predictive decoding large model inference method based on residual quantization according to the present invention;

[0017] Figure 2 This is a schematic diagram of the system architecture of the method of the present invention;

[0018] Figure 3 This is a block diagram of a self-speculatory decoding large model inference system based on residual quantization according to the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0020] Example 1:

[0021] This invention uses an NVIDIA RTX 4090 GPU as the hardware platform and an Llama3-8B as the original model. The weights of the original model are smoothed and quantized into shared weights, which are then used as the weights of the draft model. The difference between the original weights and the dequantized shared weights is quantized to obtain residual weights. The residual weights and shared weights are used together as the weights of the target model. The draft model and the target model share some weights, thereby achieving fast inference of the draft model and higher-precision verification of the target model without increasing memory usage. A 4-bit quantization bit width is used for the shared weights and the exclusive residual weights. The draft model generates 10 candidate tokens each time. The XSUM summary dataset is selected as the performance test dataset, and inference throughput and end-to-end latency are used as inference performance evaluation metrics.

[0022] like Figure 1 As shown in the figure, an embodiment of the present invention provides a self-predictive decoding large model inference method based on residual quantization, which includes the following steps:

[0023] Step S1: Load the floating-point weights of the original model, perform outlier smoothing and residual quantization in sequence to obtain shared weights and exclusive residual weights, and load the fusion quantization operator to initialize the shared KVcache.

[0024] KVcache, a widely used method for accelerating decoding of large models, stores historical key-value projection results in GPU memory as a cache, thereby speeding up decoding.

[0025] Step S2: The draft model uses shared weights to quickly generate several candidate tokens in an autoregressive manner. During the generation process, historical data in the shared KVcache is directly called and the candidate tokens are passed into the target model.

[0026] Step S3: The target model uses shared weights and residual weights to call the fusion quantization operator to perform parallel verification of candidate tokens, output the verification results and determine the validity of each candidate token;

[0027] Step S4: If all candidate tokens are valid, write the verification result of the target model directly to the shared KVcache; if an invalid token appears, roll back the KVcache to the position before the first invalid token, and let the draft model generate a valid token from that position through autoregression; repeat steps S2 to S4 until the target model generates an end symbol or reaches the preset maximum generation length to complete the entire inference process.

[0028] In one embodiment, step S1 above: loading the floating-point weights of the original model, sequentially performing outlier smoothing and residual quantization to obtain shared weights and exclusive residual weights, and simultaneously loading the fused quantization operator and initializing the shared KVcache, specifically includes:

[0029] Step S11: Weighted outlier smoothing, including the following steps:

[0030] Step S111: Select the calibration dataset and input it into the original model to capture the input activation value of each linear operator;

[0031] This embodiment of the invention first loads the original FP16 floating-point weights of the original Llama3-8B model. Based on the model characteristics, the 4-bit quantization grouping granularity is set to 64. Simultaneously, 1000 samples are selected from the XSUM and CNN summarization datasets as a calibration dataset to complete the pre-quantization preparation. During the forward propagation process of the model, hook functions are attached to each linear operator in the model to capture the input activation values ​​of each linear operator.

[0032] Step S112: Perform distribution statistical analysis on the activation values ​​and corresponding weights to locate the model layers and channels where outliers are concentrated;

[0033] The captured activation values ​​and their corresponding weights are subjected to distribution statistical analysis, including: calculating the absolute maximum value, quantiles, mean and variance of the two, so as to locate the model layer and channel where the outliers are mainly concentrated.

[0034] Step S113: Introduce a smoothing coefficient to redistribute the scale of activation values ​​and corresponding weights, reducing the dominance of outliers on the quantization scale; making the quantization error more evenly distributed across the grouping dimension and reducing the loss of weight information.

[0035] Step S114: Select the smoothing coefficient that makes the pseudo-quantized model output most similar to the original model output through grid search, and apply it to the subsequent quantization process;

[0036] By using a grid search method, smoothing coefficients are traversed within a preset interval with a fixed step size. A corresponding scaling vector is generated for each candidate coefficient. The activation values ​​and weights are rescaled and pseudo-quantization is performed. The similarity between the pseudo-quantized model output and the original model output is calculated. The smoothing coefficient with the highest similarity is selected as the optimal coefficient and applied to the actual quantization process.

[0037] To address the issues of quantization errors leading to output distribution misalignment between the draft and target models and low token acceptance rates in existing technologies, this invention designs a weight outlier smoothing mechanism. This mechanism effectively solves the quantization error problem caused by weight outliers, reduces the output distribution misalignment between the draft and target models, ensures high consistency between the two, thereby maintaining a high token acceptance rate and preventing the overhead of backoff recalculation from offsetting the acceleration benefits of self-predictive decoding, thus providing a guarantee for improving end-to-end inference speed.

[0038] Step S12: Residual quantization, including the following steps:

[0039] Step S121: Perform the first low-bit-width quantization on the smoothed floating-point weights of the original model to obtain the shared weights; at the same time, save the quantization parameters such as quantization scale and zero point generated during the quantization process of the shared weights to prepare for subsequent dequantization operations;

[0040] Step S122: Dequantize the shared weights back to floating-point form, and calculate the difference between the restored weights and the smoothed floating-point weights to obtain the residual weights;

[0041] The quantization parameters of the shared weights are dequantized and restored to FP16 floating-point precision. Then, the difference between the restored weights and the floating-point weights of the original model is calculated element by element to obtain the residual weights.

[0042] Step S123: Perform a second low-order quantization operation on the residual weights to obtain the exclusive residual weights; among them, the shared weights are used as the only weights for inference of the draft model, and the shared weights and exclusive residual weights are used together for the target model verification calculation;

[0043] The second low-order quantization operation, performed in the middle, involves a 4-bit quantization operation with the same bit width as the first quantization, resulting in exclusive residual weights. Simultaneously, the corresponding quantization parameters are saved, thus generating two sets of 4-bit shared weights and exclusive residual weights. The shared weights are assigned to the draft model as the sole inference weights, while the shared weights and exclusive residual weights are jointly assigned to the target model for computation during the validation phase.

[0044] This invention designs a residual quantization strategy adapted to self-predictive decoding scenarios. For the first time, it constructs a parameter form of shared weights + exclusive residual weights by using two low-bit-width quantizations of the same bit width. This enables the weight differentiation design between the draft model and the target model, maximizing the inference performance difference between the two without increasing memory usage. This solves the technical problems of poor adaptability of traditional quantization schemes to self-predictive decoding scenarios and difficulty in balancing performance and accuracy.

[0045] In actual inference, the draft model uses only shared weights to complete fast autoregressive inference, while the target model uses exclusive residual weights on top of the shared weights to complete the verification calculation of candidate tokens. This strategy allows the draft model to achieve high-speed inference by relying on a single set of low-bit-width weights, while the target model restores the weight features close to the original model by superimposing the two sets of weights, ensuring verification accuracy. At the same time, it does not require the introduction of an additional independent draft model. Without increasing the GPU memory usage, it maximizes the inference performance gap between the draft model and the target model and fully leverages the acceleration advantage of auto-speculation decoding.

[0046] Step S13: Construct the fusion quantization operator, the calculation process of which includes:

[0047] Step S131: Load all the input data required for inference into the GPU at once, including: activation values, low-bit-width quantized data of shared weights and exclusive residual weights, and their respective quantization scaling factors;

[0048] This invention defines the input data type and dimension of the operator. The input includes FP16 precision activation values, 4-bit precision shared weights, 4-bit precision exclusive residual weights, and their respective quantization scales, zero points, and other quantization parameters. Then, all input data is loaded into the GPU memory at once.

[0049] Step S132: In the kernel of the fusion quantization operator, the shared weight and the exclusive residual weight are dequantized sequentially. Then, the two dequantized floating-point weights are accumulated element by element to obtain a complete floating-point weight that is close to the original floating-point weight.

[0050] The shared weights and exclusive residual weights with 4-bit precision are dequantized to convert them to FP16 floating-point precision. Then, in the kernel of the fused quantization operator, the two dequantized floating-point weights are accumulated element-wise to obtain the complete floating-point weights that are close to the original model.

[0051] Step S133: Execute the matrix multiplication of activation value and full floating-point weight directly in the kernel of the fused quantization operator. After the calculation is completed, write the result directly back to the GPU memory to complete all calculations for a single operator call.

[0052] The fusion quantization operator, developed through the above steps, is integrated into the PyTorch deep learning framework to achieve seamless integration between the fusion quantization operator and the model inference process, and can be directly called by the target model.

[0053] The fusion quantization operator designed in this invention integrates computational steps that originally required multiple executions into a single operator kernel, enabling all verification calculations to be completed in a single operator call, significantly reducing computational overhead. The kernel of this fusion quantization operator supports loading all input data required for inference at once, including FP16 precision activation values, low-bit-width quantized data of shared weights and exclusive residual weights, and their respective quantization scales, zero points, and other quantization parameters. After loading, the kernel sequentially performs dequantization and element-wise accumulation operations on the two weights, restoring a complete weight feature close to the original weights of the large model. Then, the kernel directly performs matrix multiplication of the activation values ​​and the accumulated complete weights, and outputs the verification results directly after the calculation is complete.

[0054] The fusion quantization operator designed in this invention eliminates the overhead of multiple operator startups, repeated memory accesses, and intermediate result storage in traditional schemes. This effectively reduces the number of kernel startups and memory access costs, improves cache reuse efficiency and computational resource utilization, and significantly accelerates the computation speed during the target model verification phase, solving the technical problem of low efficiency in the verification phase. Compared to sequential execution, it reduces one kernel startup time and one matrix multiplication calculation.

[0055] In one embodiment, step S2 above: The draft model uses shared weights to quickly generate several candidate tokens in an autoregressive manner. During the generation process, historical data in the shared KVcache is directly called and the candidate tokens are passed into the target model.

[0056] The process of building a draft model also includes:

[0057] A Bayesian optimization method is introduced to maximize the end-to-end throughput of self-inferential decoding inference. The optimal pruning ratio and pruning level are searched to determine the pruning strategy of the draft model.

[0058] The kernel startup process is optimized by combining the pruned draft model with CUDA Graph technology.

[0059] This invention introduces a Bayesian optimization pruning method. Aiming to maximize the throughput of the self-predictive decoding system, it searches for the optimal pruning ratio and pruning level to determine the pruning strategy for the draft model, thereby further accelerating the draft model and improving throughput efficiency. Furthermore, after pruning the draft model, CUDA Graph technology is used to optimize the kernel startup process, further reducing startup overhead.

[0060] In one embodiment, step S3 above: The target model uses shared weights and residual weights to call the fusion quantization operator to perform parallel verification of candidate tokens, output the verification results and determine the validity of each candidate token;

[0061] After receiving a candidate token, the target model calls the fusion quantization operator to load the activation values, two sets of 4-bit weights and corresponding quantization parameters required for inference at once. The fusion quantization operator completes the dequantization, weight accumulation and matrix multiplication in the kernel, performs a parallel forward verification of the candidate token, and determines the validity of each token based on the output logits.

[0062] In one embodiment, step S4 above is as follows: if all candidate tokens are valid, the verification result of the target model is directly written to the shared KVcache; if an invalid token appears, the KVcache is rolled back to the position before the first invalid token, and the draft model generates a valid token from that position through autoregression; steps S2 to S4 are repeated until the target model generates an end symbol or reaches the preset maximum generation length, thus completing the entire inference process.

[0063] The unified KVcache management mechanism designed in this invention allows the draft model and the target model to share the same cache space. When the draft model is generated, the historical cache is directly called. After the target model is verified, only valid data is updated. When invalid data is found, it is simply rolled back to the specified position, minimizing the overhead of rollback and recalculation, and further improving end-to-end inference efficiency. This mechanism is fully adapted to the resource-constrained characteristics of edge devices and can be directly deployed on mainstream edge computing acceleration devices.

[0064] Verification of the effectiveness of this invention:

[0065] Under the experimental conditions described above, the Llama3-8B model was comprehensively tested in this embodiment of the invention, and the following results were obtained: In terms of memory usage, the overall memory usage of the entire model is approximately 8.7GB, which is 42% lower than that of the FP16 inference scheme (15GB). It can run stably on the NVIDIA RTX4090 edge GPU without memory overflow issues. In terms of inference performance, the fused quantization operator designed in this invention improves the inference throughput by more than 10% compared to two independent 4-bit operators. The end-to-end inference speed of the entire inference system is improved by more than 80% compared to the traditional autoregressive decoding scheme, with an average improvement of up to 104% on the XSUM dataset, effectively improving the inference efficiency of large models on edge devices.

[0066] This invention provides a method for accelerating inference of large language models based on residual quantization and self-predictive decoding. By designing a residual quantization strategy adapted to self-predictive decoding, it achieves weight differentiation between the draft model and the target model, reducing memory usage while widening the performance gap between the two. By optimizing the quantization error problem, it ensures high similarity between the draft and target models, maintaining a high token acceptance rate. By designing a dedicated fusion quantization operator, it reduces operator calls and memory access overhead during the target model verification stage. Ultimately, it achieves low memory usage, high accuracy, and high speed inference acceleration of large language models on edge devices, solving the technical problem of difficulty in balancing memory usage, inference accuracy, and inference speed in existing technologies.

[0067] Figure 2 This is a schematic diagram of the system architecture of the method of the present invention.

[0068] Example 2:

[0069] like Figure 3 As shown, this embodiment of the invention provides a self-predictive decoding large model inference system based on residual quantization, comprising the following modules:

[0070] Initialization module 51 is used to load the floating-point weights of the original model, sequentially complete the smoothing of outlier weights and residual quantization, obtain shared weights and exclusive residual weights, load the fusion quantization operator, and initialize the shared KVcache.

[0071] The candidate token generation module 52 is used by the draft model to quickly generate several candidate tokens in an autoregressive manner using the shared weights. During the generation process, historical data in the shared KVcache is directly called and the candidate tokens are passed into the target model.

[0072] The parallel verification module 53 is used by the target model to use the shared weights and residual weights to call the fusion quantization operator to perform parallel verification of candidate tokens, output the verification results and determine the validity of each candidate token;

[0073] The KVcache update and rollback module 54 is used to directly write the verification result of the target model into the shared KVcache if all candidate tokens are valid; if an invalid token is found, the KVcache is rolled back to the position before the first invalid token, and the draft model generates a valid token from that position through autoregression; the candidate token generation module, parallel verification module, and KVcache update and rollback module are repeatedly executed until the target model generates an end symbol or reaches the preset maximum generation length, thus completing the entire inference process.

[0074] A self-speculating decoding large model inference device based on residual quantization includes one or more electronic devices, wherein the one or more electronic devices are used to implement the self-speculating decoding large model inference method based on residual quantization.

[0075] An electronic device includes: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a self-speculating decoding large model inference method based on residual quantization.

[0076] A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to implement a self-speculating decoding large model inference method based on residual quantization.

[0077] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a self-speculating decoding large model inference method based on residual quantization.

[0078] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A self-predictive decoding method for large model inference based on residual quantization, characterized in that, include: Step S1: Load the floating-point weights of the original model, perform outlier smoothing and residual quantization in sequence to obtain shared weights and exclusive residual weights, and load the fusion quantization operator to initialize the shared KVcache. Step S2: The draft model uses the shared weights to quickly generate several candidate tokens in an autoregressive manner. During the generation process, historical data in the shared KVcache is directly called and the candidate tokens are passed into the target model. Step S3: The target model uses the shared weights and residual weights to call the fusion quantization operator to perform parallel verification of the candidate tokens, output the verification results, and determine the validity of each candidate token; Step S4: If all candidate tokens are valid, write the verification result of the target model directly to the shared KVcache; If an invalid token is found, the KVcache is rolled back to the position before the first invalid token, and the draft model generates a valid token from that position through autoregression. Steps S2 to S4 are repeated until the target model generates an end symbol or reaches the preset maximum generation length, thus completing the entire inference process.

2. The self-predictive decoding large model inference method based on residual quantization according to claim 1, characterized in that, The weighted outlier smoothing process in step S1 specifically includes: Step S111: Select the calibration dataset and input it into the original model to capture the input activation value of each linear operator; Step S112: Perform distribution statistical analysis on the activation values ​​and corresponding weights to locate the model layers and channels where outliers are concentrated; Step S113: Introduce a smoothing coefficient to redistribute the scale of activation values ​​and corresponding weights, thereby reducing the dominance of outliers on the quantization scale. Step S114: Select the smoothing coefficient that makes the pseudo-quantized model output most similar to the original model output through grid search, and apply it to the subsequent quantization process.

3. The self-predictive decoding large model inference method based on residual quantization according to claim 1, characterized in that, The residual quantization in step S1 specifically includes: Step S121: Perform the first low-bit-width quantization on the smoothed floating-point weights of the original model to obtain the shared weights; Step S122: Dequantize the shared weights back to floating-point form, and calculate the difference between the restored weights and the smoothed floating-point weights to obtain the residual weights; Step S123: Perform a second low-order quantization operation on the residual weights to obtain exclusive residual weights; wherein, the shared weights serve as the sole weights for inference in the draft model, and the shared weights and the exclusive residual weights are used together for the target model validation calculation.

4. The self-predictive decoding large model inference method based on residual quantization according to claim 3, characterized in that, The calculation process of the fusion quantization operator in step S1 includes: Step S131: Load all the input data required for inference into the GPU at once, including: activation values, low-bit-width quantized data of shared weights and exclusive residual weights, and their respective quantization scaling factors; Step S132: In the kernel of the fusion quantization operator, the shared weight and the exclusive residual weight are dequantized sequentially. Then, the two dequantized floating-point weights are accumulated element by element to obtain a complete floating-point weight that is close to the original floating-point weight. Step S133: Execute the matrix multiplication of activation value and full floating-point weight directly in the kernel of the fused quantization operator. After the calculation is completed, write the result directly back to the GPU memory to complete all calculations for a single operator call.

5. The self-predictive decoding large model inference method based on residual quantization according to claim 4, characterized in that, The draft model in step S2 also includes: A Bayesian optimization method is introduced to maximize the end-to-end throughput of self-inferential decoding inference. The optimal pruning ratio and pruning level are searched to determine the pruning strategy of the draft model. The kernel startup process is optimized by combining the pruned draft model with CUDA Graph technology.

6. A self-predictive decoding large model inference system based on residual quantization, characterized in that, Includes the following modules: The initialization module is used to load the floating-point weights of the original model, perform outlier smoothing and residual quantization in sequence to obtain shared weights and exclusive residual weights, load the fusion quantization operator, and initialize the shared KVcache. The candidate token generation module is used by the draft model to quickly generate several candidate tokens in an autoregressive manner using the shared weights. During the generation process, historical data in the shared KVcache is directly called and the candidate tokens are passed into the target model. The parallel verification module is used by the target model to call the fusion quantization operator to perform parallel verification of candidate tokens using the shared weights and residual weights, output the verification results and determine the validity of each candidate token; The KVcache update and rollback module is used to write the verification result of the target model directly to the shared KVcache if all candidate tokens are valid. If an invalid token is found, the KVcache will be rolled back to the position before the first invalid token, and the draft model will generate a valid token from that position through autoregression. Repeatedly execute the candidate token generation module, the parallel verification module, and the KVcache update and rollback module until the target model generates an end symbol or reaches the preset maximum generation length, thus completing the entire inference process.

7. A self-predictive decoding large model inference device based on residual quantization, characterized in that, It includes one or more electronic devices, wherein the one or more electronic devices are used to implement the method of any one of claims 1 to 5.

8. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1 to 5.

10. A non-transitory computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 5.

Images