An efficient reasoning method and system for large language models based on parallel decoding

Abstract

The present invention proposes a large language model efficient reasoning method and system based on parallel decoding, which belongs to the field of artificial intelligence technology, including: obtaining user questions and preprocessing; decoding the preprocessed user questions in parallel, and the parallel decoding includes the answer framework generation stage, the argument expansion stage and the result integration stage; in parallel decoding, the large language model reasoning process is deeply optimized by combining mixed precision calculation and KV cache quantization technology, and the details of each argument in the answer framework are expanded in parallel, so that the content of each part is generated at the same time to obtain the expanded content; the parallel processing capability of the large model is used to synchronously expand these key points. This process not only speeds up the information enrichment step, but also ensures the integrity of the final answer; it can greatly shorten the reasoning time and improve the practical application efficiency of the large language model under the premise of ensuring the quality and logical coherence of the generated text.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a large language model efficient reasoning method and system based on parallel decoding. Background Art

[0002] With the continuous breakthroughs in large language model technology, its innovative potential in diversified application scenarios is reshaping the value ecology of various industries. These models generally rely on the Transformer architecture, which has become an industry standard due to its outstanding performance and wide applicability in natural language processing and generation tasks. Its core lies in the revolutionary introduction of the self-attention mechanism, which realizes direct and efficient relationship modeling between any elements in the sequence, captures every part of the input sequence from a global perspective, and dynamically adjusts the attention to ensure that the generated text not only accurately fits the context logic, but also maintains the fluency and coherence of the language. It can perform well in both complex language structure analysis and context-adaptive generation.

[0003] However, this advantage of the Transformer architecture is also accompanied by challenges. Especially in the inference phase of the model, since the generation of each new token depends on the calculation results of the entire sequence, this leads to a high demand for memory access, and the limitation of memory access speed creates a significant I / O bottleneck. This serial processing feature has become a key obstacle to the realization of instant interactive performance of large language models. Therefore, how to overcome this bottleneck and improve the reasoning efficiency and interactive response speed of the model has become a technical problem that needs to be solved urgently.

[0004] To solve these problems, researchers have proposed a variety of improvement strategies. For example, using mixed precision computing to reduce memory and computing resource usage; introducing knowledge distillation technology to train a smaller student model to simulate the behavior of a large model, thereby reducing the amount of computation in the inference phase; and developing new optimization algorithms and hardware accelerators to improve the speed and efficiency of model inference. However, although these strategies have alleviated the problem of inference efficiency to a certain extent, they have not fundamentally solved the bottleneck caused by serial processing. Summary of the invention

[0005] In view of the problems in the prior art, the present invention proposes a parallel decoding strategy, specifically a large language model efficient reasoning method and system based on parallel decoding, which realizes concurrent processing of the answer generation link, thereby achieving a significant reduction in reasoning time.

[0006] In order to achieve the above purpose, the technical solution adopted in this application is:

[0007] In a first aspect, the present invention provides an efficient reasoning method for a large language model based on parallel decoding, comprising:

[0008] Obtain user questions and perform preprocessing;

[0009] The pre-processed user questions are decoded in parallel. The parallel decoding includes the answer framework generation stage, argument expansion stage and result integration stage. In the parallel decoding, the mixed precision calculation and KV cache quantization technology are combined to deeply optimize the large language model reasoning process. By expanding the details of each argument in the answer framework in parallel, the content of each part is generated simultaneously to obtain the expanded content.

[0010] Integrate each extended content into a coherent and complete final answer, and output the final answer to the user.

[0011] Furthermore, the preprocessing includes text cleaning, word segmentation and normalization.

[0012] Furthermore, the answer framework generation stage includes: inputting the preprocessed user questions into the big model, generating a preliminary answer framework through the big model, and the preliminary answer framework includes the main structure and arguments.

[0013] Furthermore, the argument expansion stage includes: extracting each argument of the preliminary answer frame, combining the user question and the preliminary answer frame into a prompt; packaging multiple prompts into a batch, sending them to a large language model or concurrently calling an API to expand the argument to obtain an extended answer frame.

[0014] Furthermore, the result integration stage includes: after the argument expansion stage is completed, all prompt generated contents are collected to obtain result contents, which are detailed contents of multiple arguments generated by the large language model in the parallel decoding process.

[0015] Furthermore, mixed-precision computing balances computing accuracy and performance by combining single-precision and 8-bit integer operations.

[0016] Furthermore, the KV cache quantization technology adopts an asymmetric quantization method to convert the KV cache from the Float16 type to the more compact int8 type. For the key cache, a channel grouping quantization method is adopted, and for the value cache, a word-based quantization strategy is adopted.

[0017] In a second aspect, the present invention provides a large language model efficient reasoning system based on parallel decoding, comprising:

[0018] The input module is configured to: obtain user questions and perform preprocessing;

[0019] The parallel decoding module is configured to: decode the pre-processed user questions in parallel, including the answer framework generation stage, argument expansion stage and result integration stage; in parallel decoding, the mixed precision calculation and KV cache quantization technology are combined to deeply optimize the large language model reasoning process, and the details of each argument in the answer framework are expanded in parallel, so that the content of each part is generated simultaneously to obtain the expanded content;

[0020] The output module is configured to: integrate the various extended contents into a coherent and complete final answer, and output the final answer to the user.

[0021] In a third aspect, the present invention provides a computer-readable storage medium, the computer-readable storage medium including a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the efficient reasoning method for a large language model based on parallel decoding as described in the first aspect.

[0022] In a fourth aspect, the present invention provides an electronic device comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, the electronic device is triggered to execute the efficient reasoning method for a large language model based on parallel decoding as described in the first aspect.

[0023] Compared with the prior art, the advantages and positive effects of the present invention are:

[0024] 1. The present invention adopts an efficient reasoning method of a large language model based on parallel decoding. First, a macro framework of the answer is generated, and then the core points in the framework are organized into batches, and the parallel processing capabilities of the large model are used to synchronously expand the content of these key points. This process not only speeds up the information enrichment step, but also ensures the integrity of the final answer. Finally, these parallel processed fragments are summarized and assembled into a coherent and comprehensive final answer.

[0025] 2. In order to further improve the model generation efficiency, the present invention uses mixed precision calculation and KV cache quantization technology in all stages of parallel decoding. This method not only greatly improves the reasoning efficiency and optimizes the time cost, but also makes full use of distributed computing resources and reduces the pressure on a single computing node. It can greatly shorten the reasoning time while ensuring the quality and logical coherence of the generated text, and improve the practical application efficiency of large language models, especially in scenarios that require rapid response. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0027] Figure 1 It is a flowchart of the efficient reasoning method of a large language model based on parallel decoding of the present invention. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0030] Embodiment 1, as Figure 1 As shown, this embodiment provides an efficient reasoning method for a large language model based on parallel decoding, including: obtaining user questions and performing preprocessing;

[0031] The pre-processed user questions are decoded in parallel. The parallel decoding includes the answer framework generation stage, argument expansion stage and result integration stage. In the parallel decoding, the mixed precision calculation and KV cache quantization technology are combined to deeply optimize the large language model reasoning process. By expanding the details of each argument in the answer framework in parallel, the content of each part is generated simultaneously to obtain the expanded content.

[0032] Integrate each extended content into a coherent and complete final answer, and output the final answer to the user.

[0033] This embodiment aims to solve the problem of high latency in the serial decoding process of large models. The traditional serial decoding process of large models has obvious efficiency bottlenecks. Specifically, in the serial decoding process, each newly generated token depends on the calculation results of all previously generated tokens, which means that the model must gradually generate each element in the sequence and calculate it in turn. This step-by-step generation process not only requires continuous access and updating of memory, but also requires frequent calculation and data transmission, resulting in high latency and waste of resources.

[0034] In each decoding step, the model calculates the probability distribution of the current token based on the current input sequence and context information through the self-attention mechanism and feedforward neural network, and then selects a token as the output. Next, this newly generated token will be added to the input sequence as the input for the next decoding step, and the above calculation process will be repeated until the complete output sequence is generated. This step-by-step calculation and dependence on the previous token means that the generation of each step must wait for the result of the previous step, thus forming a strict sequential dependency.

[0035] Since the generation of each token requires complex calculations and access to the contextual information of the entire sequence, the amount of calculation and memory access requirements increase as the sequence length increases. This not only leads to a linear increase in inference time, but also causes memory bottlenecks, especially when processing long sequences. Traditional optimization methods, such as hardware acceleration and model compression, can alleviate these problems to a certain extent, but cannot fundamentally solve the inherent limitations of serial decoding.

[0036] In order to meet these challenges, the present invention proposes a method based on parallel decoding, which decomposes the answer generation process into multiple independent tasks for parallel processing, thereby significantly improving the reasoning efficiency. Specifically, the method of the present invention includes three main stages: first, by generating a preliminary answer framework, the overall structure and arguments are captured; then, the details of each argument in the preliminary answer framework are expanded in parallel, so that the content of each part can be generated at the same time; finally, the expanded contents are integrated into a coherent and complete answer. Compared with autoregressive reasoning, this process not only reduces the reasoning time, but also improves resource utilization and avoids the bottleneck problem caused by traditional serial decoding.

[0037] Mixed precision computing and KV cache quantization techniques are used in various stages of model reasoning. Mixed precision computing is a technique that uses floating-point numbers of different precisions during deep learning training and reasoning. It aims to improve computing efficiency, reduce memory usage, and reduce energy consumption while maintaining model performance. Typically, mixed precision computing combines half-precision floating point numbers (FP16) and 8-bit integers (INT8). INT8 can significantly reduce memory bandwidth and storage requirements, while FP16 is used to ensure computing stability and accuracy.

[0038] The application of mixed precision computing and KV cache quantization in parallel decoding of large language models can be carried out in the following steps:

[0039] Step 1.1, model weight optimization: convert the model weights from the initial FP16 format to INT8 and keep the model's activation values ​​in FP16 format to maintain computational stability;

[0040] Step 1.2, input data processing: pack multiple prompts into a batch and convert them into FP16 format;

[0041] Step 1.3, model forward propagation: Use INT8 for most computing operations, such as matrix multiplication; maintain FP16 precision for key operations (such as LayerNorm) to ensure numerical stability.

[0042] Step 1.4, attention mechanism optimization: quantize the KV Cache generated by the attention mechanism into INT8 format, which can significantly reduce memory usage and improve parallel decoding efficiency.

[0043] In the reasoning process of a large language model, the next generated text needs to be predicted one by one, and the generation of each new text depends on the previous generated text, which makes the whole process very slow. Therefore, most reasoning frameworks use KV cache technology to optimize this process. In the Transformer architecture, the self-attention layer calculates the attention score by multiplying the query with the key, and then generates a weighted value vector matrix. By caching these key and value information, the model does not need to repeatedly calculate the key and value of each word, but directly retrieves the previous calculation results from the cache and splices them with the current vector. However, for long text or large batch size reasoning, this method will face memory bottleneck problems, and considering that the parallel decoding method proposed in this patent requires a larger batch for reasoning, we quantized the stored KV cache. KV cache quantization reduces the memory usage of the model when processing long sequence data by reducing the storage requirements of key-value pairs in the cache. By converting floating-point parameters to a lower bit width representation, the storage size of model parameters is reduced while maintaining model performance. The application of this technology not only improves the running speed of the model, but also reduces the dependence on hardware resources, enabling the model to run more efficiently in a resource-constrained environment.

[0044] The specific application steps of KV cache quantization in parallel decoding are:

[0045] Step 2.1, Initialize KV Cache: Create an initial KV cache for each prompt in the batch; the initial cache usually uses FP16 format;

[0046] Step 2.2, quantize the KV cache: quantize the initialized KV cache in FP16 format to INT8 format; record the quantization scale factor for subsequent dequantization;

[0047] Step 2.3, parallel decoding process: decode each prompt in the batch in parallel; each time a new token is generated, its KV cache is quantized and saved.

[0048] Step 2.4, update the KV cache: calculate the K and V vectors for the newly generated token; quantize the new K and V vectors and add them to the cache.

[0049] Step 2.5, dequantization: When KV cache is needed, dequantize the INT8 format to FP16; assuming that there is a batch containing 5 prompts, each prompt has an initial length of 50 tokens, and the model hidden dimension is 1024.

[0050] The unquantized KV cache size is as follows: each token's K and V each require 1024 FP16 numbers (2 bytes / number); the KV cache size of a single prompt: 50 * 1024 * 2 * 2 = 204,800 bytes;

[0051] Total KV cache size of 5 prompts: 204,800 * 5 = 1,024,000 bytes ≈ 1M;

[0052] Quantized KV cache size: Using INT8 format, each number only needs 1 byte; Quantized KV cache size of a single prompt: 50 * 1024 * 2 * 1 = 102,400 bytes; Total quantized KV cache size of 5 prompts: 102,400 * 5 = 512,000 bytes = 0.5MB;

[0053] Through quantization, the storage size of the KV cache can be reduced by 50%.

[0054] The present invention introduces mixed precision calculation and KV cache quantization in the process of parallel decoding, which can greatly shorten the reasoning time and improve the practical application efficiency of large language models while ensuring the quality and logical coherence of generated text, especially in scenarios that require fast response.

[0055] Example 2

[0056] This embodiment provides a large language model efficient reasoning system based on parallel decoding, including:

[0057] The input module is configured to: obtain user questions and perform preprocessing;

[0058] The parallel decoding module is configured to: decode the pre-processed user questions in parallel, including the answer framework generation stage, argument expansion stage and result integration stage; in parallel decoding, the mixed precision calculation and KV cache quantization technology are combined to deeply optimize the large language model reasoning process, and the details of each argument in the answer framework are expanded in parallel, so that the content of each part is generated simultaneously to obtain the expanded content;

[0059] The output module is configured to: integrate the various extended contents into a coherent and complete final answer, and output the final answer to the user.

[0060] Specifically, the overall system architecture includes an input module, a parallel decoding module, and an output module. The entire system achieves an efficient parallel decoding process by coordinating these three modules.

[0061] Input module: responsible for receiving natural language questions or tasks input by users and converting them into an input format suitable for model processing. This module includes preprocessing steps such as word segmentation and denoising to ensure the quality and consistency of input data.

[0062] Parallel decoding module: This is the core part of the system, responsible for parallel decoding of input data. This module includes the framework generation submodule and the argument expansion submodule.

[0063] Output module: responsible for summarizing the output answers of all sub-modules and outputting the final answer to the user.

[0064] In the parallel decoding process, the input module is responsible for receiving user input questions and first performs preprocessing, including: text cleaning, such as removing redundant spaces and special characters; word segmentation, such as dividing the input text into words or sub-word units; normalization, such as converting words to standard forms, such as lowercase English letters, etc.

[0065] The parallel decoding process includes three main stages: answer framework generation stage, argument expansion stage and result integration stage.

[0066] Answer framework generation stage: In this stage, the big model is required to generate a rough answer framework based on the input question. This framework contains the main structure and key points of the answer. Through targeted prompt design, the system can quickly generate frameworks for different types of questions. The following is a simple prompt example, the input question is "Please generate an article about climate change":

[0067] "Please generate an answer framework based on the following questions, including the main structure and key points. Each structure must contain 1 to 2 key points. There is no need to fully describe the key points, just give the key argument title. Question: Please generate an article about climate change." After inputting the above prompt into GPT4, the answer framework obtained is as follows:

[0068] “Answer Framework:

[0069] Introduction: Definition of climate change; Historical background of climate change;

[0070] Main causes: Greenhouse gas emissions; Deforestation;

[0071] Impact: Environmental impact; Economic impact;

[0072] Response measures: policies and regulations; individual and community action;

[0073] Future prospects; the potential of scientific and technological innovation; the need for global cooperation. ”

[0074] In the argument expansion phase, after the main framework is generated, the system extracts each key point in the framework and then combines the question and answer framework into a new prompt. Next, these prompts will be packaged into a batch and fed into a large language model or concurrently call the API for argument expansion.

[0075] First, the system extracts each argument in the generated answer framework, and these key points will serve as the basis for the subsequent argument expansion stage. For example, the new prompt example composed of key points extracted from the generated climate change article framework and combined with the question and answer framework is as follows:

[0076] “You are responsible for continuing to write an argument within the general framework of the following question.

[0077] Question: Please generate a detailed article about climate change.

[0078] frame:{answer frame};

[0079] The argument you are responsible for writing is: {1. Introduction: Definition of climate change; Historical background of climate change}"

[0080] The above answer framework can construct 5 prompts based on the arguments, and then package them into a batch and call them according to different model deployment methods.

[0081] 1. For large language models that can only be accessed using APIs, such as Kimi, BaiChuan4, or large model services deployed using VLLM, multiple API call requests can be processed in parallel. Although this will increase the number of API requests, it can significantly reduce end-to-end latency, thereby improving overall response speed. By processing multiple requests in parallel, the system can generate complete answers faster. Although the cost has increased, the improvement in performance and efficiency is crucial for efficient reasoning.

[0082] 2. For the locally deployed large language model, these 5 prompts can be packaged into a batch and sent to the large language model for parallel decoding. Through this batch processing method, the system can efficiently process multiple prompts in a single reasoning process, making full use of hardware resources such as GPU or TPU, so that multiple reasoning tasks can be performed simultaneously at the same time. Compared with the serial method of processing tasks one by one, batch processing can complete more reasoning tasks in the same time, thereby significantly reducing the overall end-to-end latency. By processing multiple prompts at one time, the system can generate the details of each argument more quickly, improving the overall processing speed.

[0083] In the result integration phase, after the argument expansion phase is completed, the system collects the content generated by all prompts, which are the detailed contents of multiple arguments generated by the large language model in the parallel decoding process, and passes these contents to the output module.

[0084] The output module first collects the output results of each subtask from the parallel decoding module. And integrates the collected results according to the predefined answer framework. The output module needs to combine the contents of each part in an orderly manner according to the structure of the original prompt to ensure that the generated answer is logically clear and the content is coherent. For example, when generating an article about climate change, the output module will integrate the contents of "Definition of Climate Change", "Historical Background of Climate Change", "Greenhouse Gas Emissions" and other parts into a complete article. During the integration process, the output module can also perform some basic content optimization operations, such as: adjusting the format of the output content as needed, such as the spacing between paragraphs, the style of the title, etc., to make it more beautiful and easy to read. Finally, the optimized content is integrated to generate the final output result.

[0085] Mixed precision computing and KV cache quantization, in order to further improve the generation efficiency of the model, the present invention adopts mixed precision computing technology in all stages of parallel decoding. Mixed precision computing effectively balances computing accuracy and performance by combining single precision (FP32) and 8-bit integer (INT8) operations. While ensuring the quality of model output, it reduces the consumption of computing resources and speeds up the reasoning speed of the model. Especially in the Transformer architecture involving large-scale matrix operations, mixed precision computing significantly reduces the demand for memory bandwidth and overall energy consumption.

[0086] In addition, this embodiment also introduces KV cache quantization technology to further optimize the storage and access efficiency of the model. An asymmetric quantization method strategy is adopted to convert the KV cache from the Float16 type to the more compact int8 type. Specifically, for the key cache (K Cache), a channel grouping quantization method is adopted. This method can independently perform quantization processing for the data characteristics within each channel. This allows the model to capture the data differences between different channels, thereby retaining more information during the quantization process. For the value cache (V Cache), a word-based quantization strategy is adopted. This method takes into account the data distribution at the word-level, so that the quantization operation can more accurately adapt to the uniqueness of each word. In this way, even under low-bit width quantization, the integrity of key information in the value cache can be maintained. This quantization strategy effectively reduces memory usage, thereby solving the memory bottleneck problem and improving reasoning efficiency.

[0087] By combining mixed precision computing and KV cache quantization technology, the method of the present invention achieves deep optimization of the reasoning process of large language models while ensuring the quality and logical coherence of the generated text. This not only greatly shortens the reasoning time and improves the response speed of the model, but also improves resource utilization and reduces the demand for computing and storage resources, thereby achieving more efficient reasoning performance.

[0088] Embodiment 3, a computer-readable storage medium, the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the method of embodiment 1.

[0089] Embodiment 4, the present invention provides an electronic device, including a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method of embodiment 1.

[0090] The electronic device may include: a processor, a memory and a communication unit. These components communicate via one or more buses. Those skilled in the art will appreciate that the structure of the electronic device does not limit the embodiments of the present invention, and it may be a bus structure or a star structure, or a combination of certain components, or different component arrangements.

[0091] The communication unit is used to establish a communication channel so that the electronic device can communicate with other devices, receive user data sent by other devices or send user data to other devices.

[0092] The processor is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor can only include a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0093] The memory is used to store the execution instructions of the processor. The memory can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0094] When the execution instructions in the memory are executed by the processor, the electronic device is enabled to execute part or all of the steps of Embodiment 1.

[0095] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. An efficient inference method for a large language model based on parallel decoding, characterized in that: include: Obtain user questions and perform preprocessing; The pre-processed user questions are decoded in parallel. The parallel decoding includes the answer framework generation stage, argument expansion stage and result integration stage. In the parallel decoding, the mixed precision calculation and KV cache quantization technology are combined to deeply optimize the large language model reasoning process. By expanding the details of each argument in the answer framework in parallel, the content of each part is generated simultaneously to obtain the expanded content. Integrate each extended content into a coherent and complete final answer, and output the final answer to the user; The answer framework generation stage includes: inputting the pre-processed user questions into the big model, and generating a preliminary answer framework through the big model. The preliminary answer framework includes the main structure and arguments; The argument expansion phase includes: extracting each argument of the preliminary answer framework, combining the user question and the preliminary answer framework into a prompt; packaging multiple prompts into a batch, feeding them into a large language model or concurrently calling the API to expand the argument to obtain an extended answer framework; The result integration stage includes: after the argument expansion stage is completed, all prompt generated contents are collected to obtain result contents, which are detailed contents of multiple arguments generated by the large language model in the parallel decoding process; Mixed precision computing balances computing accuracy and performance by combining single precision and 8-bit integer operations. The application of mixed precision computing and KV cache quantization in parallel decoding of large language models can be carried out in the following steps: Step 1.1, model weight optimization: convert the model weights from the initial FP16 format to INT8 and keep the model's activation values ​​in FP16 format to maintain computational stability; Step 1.2, input data processing: pack multiple prompts into a batch and convert them into FP16 format; Step 1.3, model forward propagation: Use INT8 for most computational operations, using matrix multiplication; maintain FP16 precision for key operations to ensure numerical stability; Step 1.4, attention mechanism optimization: quantize the KV Cache generated by the attention mechanism into INT8 format, which can significantly reduce memory usage and improve parallel decoding efficiency; KV cache quantization technology adopts the strategy of asymmetric quantization method to convert KV cache from Float16 type to more compact int8 type. For key cache, the quantization method of grouping by channel is adopted, and for value cache, the strategy of quantization by word element is adopted. The specific application steps of KV cache quantization in parallel decoding are as follows: Step 2.1, Initialize KV Cache: Create an initial KV cache for each prompt in the batch; the initial cache usually uses FP16 format; Step 2.2, quantize the KV cache: quantize the initialized KV cache in FP16 format to INT8 format; record the quantization scale factor for subsequent dequantization; Step 2.3, parallel decoding process: decode each prompt in the batch in parallel; each time a new token is generated, its KV cache is quantized and saved; Step 2.4, update the KV cache: calculate the K and V vectors for the newly generated token; quantize the new K and V vectors and add them to the cache; Step 2.5, dequantization: When KV cache is needed, dequantize the INT8 format to FP16; assuming that there is a batch containing 5 prompts, each prompt has an initial length of 50 tokens, and the model hidden dimension is 1024.

2. According to the method of claim 1, the method is characterized in that: The preprocessing includes text cleaning, word segmentation and normalization.

3. An efficient reasoning system for large language models based on parallel decoding, characterized in that: include: The input module is configured to: obtain user questions and perform preprocessing; The parallel decoding module is configured to: decode the pre-processed user questions in parallel, including the answer framework generation stage, argument expansion stage and result integration stage; in parallel decoding, the mixed precision calculation and KV cache quantization technology are combined to deeply optimize the large language model reasoning process, and the details of each argument in the answer framework are expanded in parallel, so that the content of each part is generated simultaneously to obtain the expanded content; The output module is configured to: integrate the various extended contents into a coherent and complete final answer, and output the final answer to the user; The answer framework generation stage includes: inputting the pre-processed user questions into the big model, and generating a preliminary answer framework through the big model. The preliminary answer framework includes the main structure and arguments; The argument expansion phase includes: extracting each argument of the preliminary answer framework, combining the user question and the preliminary answer framework into a prompt; packaging multiple prompts into a batch, feeding them into a large language model or concurrently calling the API to expand the argument to obtain an extended answer framework; The result integration stage includes: after the argument expansion stage is completed, all prompt generated contents are collected to obtain result contents, which are detailed contents of multiple arguments generated by the large language model in the parallel decoding process; Mixed precision computing balances computing accuracy and performance by combining single precision and 8-bit integer operations. The application of mixed precision computing and KV cache quantization in parallel decoding of large language models can be carried out in the following steps: Step 1.1, model weight optimization: convert the model weights from the initial FP16 format to INT8 and keep the model's activation values ​​in FP16 format to maintain computational stability; Step 1.2, input data processing: pack multiple prompts into a batch and convert them into FP16 format; Step 1.3, model forward propagation: Use INT8 for most computational operations, using matrix multiplication; maintain FP16 precision for key operations to ensure numerical stability; Step 1.4, attention mechanism optimization: quantize the KV Cache generated by the attention mechanism into INT8 format, which can significantly reduce memory usage and improve parallel decoding efficiency; KV cache quantization technology adopts the strategy of asymmetric quantization method to convert KV cache from Float16 type to more compact int8 type. For key cache, the quantization method of grouping by channel is adopted, and for value cache, the strategy of quantization by word element is adopted. The specific application steps of KV cache quantization in parallel decoding are as follows: Step 2.1, Initialize KV Cache: Create an initial KV cache for each prompt in the batch; the initial cache usually uses FP16 format; Step 2.2, quantize the KV cache: quantize the initialized KV cache in FP16 format to INT8 format; record the quantization scale factor for subsequent dequantization; Step 2.3, parallel decoding process: decode each prompt in the batch in parallel; each time a new token is generated, its KV cache is quantized and saved; Step 2.4, update the KV cache: calculate the K and V vectors for the newly generated token; quantize the new K and V vectors and add them to the cache; Step 2.5, dequantization: When KV cache is needed, dequantize the INT8 format to FP16; assuming that there is a batch containing 5 prompts, each prompt has an initial length of 50 tokens, and the model hidden dimension is 1024.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the large language model efficient reasoning method based on parallel decoding as described in any one of claims 1-2.

5. An electronic device, characterized in that: It includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the large language model efficient reasoning method based on parallel decoding as described in any one of claims 1-2.

Images