Inference acceleration method and device

Abstract

The invention relates to the technical field of artificial intelligence, and provides a reasoning acceleration method and device. The method comprises the following steps: distributing a plurality of received concurrent requests to corresponding draft models; based on the target candidate lexical element quantity corresponding to each draft model, obtaining a reasoning result output by each draft model; based on a target verification big language model, performing parallel verification on the plurality of reasoning results to obtain verification results of the reasoning results; according to the current load condition of the target verification large language model, performing re-reasoning processing or large language model decoding on the reasoning result which does not pass the verification to obtain the reasoning result which passes the verification; and after it is determined that the reasoning results of all the draft models pass verification, completing reasoning tasks of the multiple concurrent requests. According to the reasoning acceleration method and device provided by the invention, the reasoning efficiency is improved, and the deployment cost is reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a reasoning acceleration method and apparatus. Background Technology

[0002] The deployment of existing Large Language Model (LLM) inference applications faces numerous technical challenges. On one hand, traditional Transformer-based inference engines are divided into Prefill and Decode stages. The former processes data in parallel to generate a Key-Value (KV) cache, while the latter decodes data serially to generate a new KV cache. This results in issues such as serial inference processes, low efficiency, and uneven computational density. On the other hand, the time and speed at which LLM generates its first token significantly impact user experience; delays exceeding 100 milliseconds are perceptible, and delays exceeding 2 seconds can disrupt the user's thought process. Furthermore, the deployment of ultra-large-scale Mixture of Experts (MoE) models involves a large number of parameters and extremely high requirements for GPU memory bandwidth, leading to significant KV cache memory consumption during long-sequence inference.

[0003] Therefore, there is an urgent need for a reasoning acceleration method and device to solve the above problems. Summary of the Invention

[0004] To address the problems existing in the above-mentioned technologies, this application provides a reasoning acceleration method and apparatus.

[0005] Firstly, this application provides a method for accelerating inference, including: Distribute the received concurrent requests to the corresponding draft models; Based on the number of target candidate lexical units corresponding to each of the draft models, the inference results output by each of the draft models are obtained; Based on the target-validation large language model, multiple inference results are validated in parallel to obtain the validation results of each inference result; Based on the target verification of the current load of the large language model, perform re-inference processing or large language model decoding on the inference results that fail verification to obtain the inference results that pass verification; After verifying that the inference results of all the draft models have passed the verification, the inference task for multiple concurrent requests is completed.

[0006] In one embodiment, the step of assigning multiple received concurrent requests to corresponding draft models includes: Based on global routing scheduling and the current load of the draft model, the draft model corresponding to each concurrent request is determined, and each concurrent request is added to the request queue.

[0007] In one embodiment, the method further includes: Each of the inference results is written into a multi-level cache system; The target-based large language model performs parallel verification on multiple inference results to obtain verification results for each inference result, including: Based on the target verification large language model, the inference results in the multi-level caching system are verified in parallel to obtain the verification results of each inference result.

[0008] In one embodiment, obtaining the inference results output by each draft model based on the number of target candidate lexical units corresponding to each draft model includes: Based on the historical verification status, current computational load, and available bandwidth information of the draft model, the number of target candidate lexical units corresponding to the draft model is determined, wherein the available bandwidth information represents the available bandwidth of the draft model when accessing the multi-level caching system. Based on the draft model, the corresponding concurrent requests are inferred to generate the target number of candidate lexical units; The inference result is constructed based on the number of candidate lexical units of the target candidate lexical unit.

[0009] In one embodiment, the parallel verification of multiple inference results based on the target-based large language model to obtain verification results for each inference result includes: Multiple concurrent requests and multiple inference results are concatenated to obtain a concatenated sequence; The concatenated sequence is input into the target validation large language model to obtain the probability distribution of the next word element corresponding to each position in the concatenated sequence; Based on the probability distribution and the preset sampling strategy, the inference results in the spliced ​​sequence are sampled; The selected inference results from the sampling are determined as valid inference results, and the unselected inference results are determined as invalid inference results.

[0010] In one embodiment, the large language model decoding includes: If the current load of the target validation large language model is determined to be less than the preset large model load threshold, the inference results that failed the validation are decoded based on the target validation large language model to obtain the inference results that passed the validation. The re-inference process includes: If the current load of the target validation large language model is greater than or equal to the preset large model load threshold, the inference result output by the re-inference draft model is obtained based on the number of new candidate lexical units corresponding to the re-inference draft model. The number of new candidate lexical units is obtained by reducing the number of target candidate lexical units in the re-inference draft model. The re-inference draft model represents the draft model corresponding to the inference result that failed the validation. Based on the target validation large language model, the reasoning results output by the re-inference draft model are re-validated to obtain the valid reasoning results.

[0011] In one embodiment, the method further includes: If it is determined that the reasoning result output by the re-inference draft model fails the verification, the concurrent request corresponding to the re-inference draft model is switched to another draft model for reasoning, or, based on the target verification large language model, the reasoning result output by the re-inference draft model is decoded.

[0012] Secondly, this application provides a reasoning acceleration device, comprising: The allocation module is used to allocate multiple received concurrent requests to the corresponding draft models; An adaptive inference module is used to obtain the inference results output by each of the draft models based on the number of target candidate lexical units corresponding to each of the draft models. The large model verification module is used to verify the large language model based on the target, and to perform parallel verification of multiple inference results to obtain the verification results of each inference result; The large model processing decision module is used to verify the current load of the large language model according to the target, perform re-inference processing or large language model decoding on the inference results that fail the verification, and obtain the inference results that pass the verification. The inference output module is used to complete the inference tasks of multiple concurrent requests after determining that the inference results of all the draft models have passed the verification.

[0013] Thirdly, this application provides an electronic device including a processor and a memory storing a computer program, wherein the processor executes the program to implement the steps of the inference acceleration method described in the first aspect.

[0014] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the inference acceleration method described in the first aspect.

[0015] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the inference acceleration method described in the first aspect.

[0016] The inference acceleration method and apparatus provided in this application improve inference efficiency and reduce deployment costs by allocating multiple concurrent requests to corresponding draft models, obtaining inference results, and then using a target verification large language model for parallel verification. Based on the load of the large language model, the failed results are re-inferred or decoded, and finally all inference results pass the verification to complete the inference task. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating the inference acceleration method provided in this application.

[0019] Figure 2 A schematic diagram of the autonomous selection mechanism for the sampling strategy provided in this application.

[0020] Figure 3 The overall flowchart of inference acceleration based on multi-level caching provided in this application.

[0021] Figure 4 A schematic diagram illustrating the hardware interaction between the inference node and the acceleration node provided in this application.

[0022] Figure 5 A schematic diagram of the inference acceleration device provided in this application.

[0023] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] As the scale of large language models continues to expand, inference efficiency and deployment costs have become key bottlenecks restricting the widespread adoption of AI applications.

[0026] Traditional large-scale distributed LLM inference engines based on the transformer architecture consist of two phases: Prefill and Decode. The Prefill phase processes user input prompts in parallel and generates a key-value (KV) cache, making it computationally sensitive. The Decode phase, based on the KV cache from the Prefill phase, uses an autoregressive approach to serially decode and generate a KV cache of related output tokens. Because it can only predict the next token at a time and requires historical outputs as input for the next decoding step, the inference process is serial, inefficient, and computationally intensive, making it storage sensitive. Psychological and industry experiments consistently show that the time to generate the first token (TTFT) in an LLM, and the generation rate per second, directly determines the user's "waiting experience." A wait exceeding 100 milliseconds is perceptible, and a wait exceeding 2 seconds can interrupt thinking.

[0027] To address these issues, the industry has attempted various optimization methods, including model compression, quantization, and incremental decoding, but with limited effectiveness. This is particularly true in ultra-large-scale MoE model deployments, where the number of parameters can easily reach tens or hundreds of billions, placing extremely high demands on memory bandwidth, computing power, and internet bandwidth.

[0028] In the field of large language model inference optimization, speculative inference is a promising technical approach. By introducing lightweight models or external knowledge data, it generates inference drafts for large language models, allowing multiple tokens to be inferred at once during the decoding stage, thus improving computational density. Taking the DeepSeekV3 / R1 model as an example, it introduces an MTP (Multi-Token Prediction) speculative layer, effectively implementing speculative inference technology.

[0029] Speculative reasoning, also known as speculative decoding, is based on the idea of ​​using a small model (also called a speculative model or draft model) with a computationally lower cost than the large language model to make predictions about possible outputs. The large language model then verifies these predictions, thus achieving parallel reasoning and improving overall reasoning speed. However, perfectly combining the speculative model and the main model faces numerous challenges, including balancing prediction accuracy with generation efficiency, adapting to batch reasoning scenarios, and effectively integrating with techniques such as KVCache and quantization.

[0030] Figure 1 A flowchart illustrating the inference acceleration method provided in this application is shown below. Figure 1 As shown, this application provides a method for accelerating inference, including: Step 101: Distribute the received concurrent requests to the corresponding draft models.

[0031] In this application, multiple simultaneous raw Prompt input requests are first received, followed by global routing scheduling. During scheduling, a suitable draft model is selected for different requests within the same batch based on the current load of each draft model. For example, if request 1 is assigned to draft model 1, it can be represented as R1M1. After selecting the draft model, the request is pushed into the corresponding request queue so that subsequent draft models can pull tasks for processing.

[0032] Step 102: Based on the number of target candidate lexical units corresponding to each draft model, obtain the inference results output by each draft model.

[0033] In this application, multiple independently deployed draft models pull inference tasks from a request queue. During inference, the number of tokens generated by the draft model is not fixed, but dynamically adjusted based on various factors, namely the number of target candidate tokens (M). These factors include the historical validation adoption rate of the target validation large language model (i.e., the proportion of the draft model's output that was accepted by the target validation large language model in the past), its current load (such as the draft model's computational resource consumption), and the available bandwidth for accessing the multi-level caching system (which affects data reading speed and thus inference efficiency).

[0034] After adjusting the number of terms, the draft model performs Prefill and Decoding operations in parallel, and writes the KV (Key-Value pairs, used to store intermediate information during inference) generated during the inference process into a multi-level cache system, and finally outputs the inference result.

[0035] Step 103: Based on the target validation large language model, perform parallel validation on multiple inference results to obtain the validation results of each inference result.

[0036] In this application, the target validation large language model retrieves the key-value (KV) values ​​output by speculative inference from multiple draft models in a multi-level caching system, and combines these KV values ​​into a large batch. The batch size can be adjusted according to actual needs.

[0037] Then, the target validation large language model performs prefill parallel processing (i.e., validation operation) on this large batch. During the validation process, the target validation large language model autonomously selects and adjusts its sampling strategy based on factors such as user-defined expected preferences (e.g., the user's desired output to have a specific style or theme), historical usage preferences (the degree to which users have accepted similar outputs in the past), and feedback interactions with the final output (users' previous feedback on the output). These sampling strategies are used to validate the inference results of the draft model, thereby obtaining validation results for each inference result and determining whether it passes validation.

[0038] Step 104: Based on the current load of the target verification large language model, perform re-inference processing or large language model decoding on the inference results that fail verification to obtain the inference results that pass verification.

[0039] In this application, the target validation large language model will decide how to handle the inference results that fail validation based on its current load.

[0040] If the target validation large language model itself is under light load and the user allows it, it will enable the Decode engine to decode the unadopted speculative inference results generated by a specific draft model (the number of which can be combined with the load and the user-defined range), which is the relevant processing of large language model decoding.

[0041] If the target validation model is overloaded or does not meet other relevant conditions, the original draft model will be required to reduce the number of speculative outputs M and perform speculative inference again, i.e., the re-inference process. If, after multiple (the specific number can be set) speculative inferences, the result still fails the validation of the target validation language model, the target validation language model will select different processing methods according to the user configuration. For example, it may switch to another draft model for speculative inference, perform the Decode operation itself, and finally obtain a validated inference result, or return a failure message.

[0042] Step 105: After confirming that the inference results of all the draft models have passed the verification, complete the inference task for multiple concurrent requests.

[0043] In this application, once it is confirmed that the inference results of all draft models have passed the verification of the target large language model, it means that the inference task of multiple concurrent requests has been successfully completed, and the final inference result can be returned to the corresponding requester to complete the entire inference process.

[0044] The inference acceleration method provided in this application improves inference efficiency and reduces deployment costs by distributing multiple concurrent requests to corresponding draft models, obtaining inference results, and then using a target verification large language model for parallel verification. Based on the load of the large language model, the inference results that fail are re-inferred or decoded. Finally, all inference results pass the verification and the inference task is completed.

[0045] Based on the above embodiments, the step of allocating multiple received concurrent requests to corresponding draft models includes: Based on global routing scheduling and the current load of the draft model, the draft model corresponding to each concurrent request is determined, and each concurrent request is added to the request queue.

[0046] In this application, multiple raw prompt input requests are received simultaneously. These requests may be instructions from the user to have the model perform a specific task, such as generating text, answering questions, or creating content. They arrive at the system almost simultaneously, forming a scenario of concurrent requests.

[0047] Upon receiving these concurrent requests, a global routing and scheduling mechanism is responsible for coordinating and allocating these requests to ensure that they can be processed appropriately.

[0048] In this application, the current load information of each draft model is collected in real time. The load of the draft model can include multiple aspects, such as the number of requests being processed, the utilization rate of computing resources (such as CPU and GPU utilization), and memory usage. This information reflects the current working status and workload of the draft model.

[0049] Furthermore, based on the collected draft model load information, the global routing scheduling mechanism selects the most suitable draft model for different concurrent requests within the same batch. The selection principle is to maintain a balanced load across the draft models as much as possible, avoiding situations where some draft models are overloaded while others are idle. For example, if there are three concurrent requests (request 1, request 2, and request 3) and three draft models (draft model 1, draft model 2, and draft model 3), if draft model 1 currently has a light load, request 1 may be assigned to draft model 1; if draft model 2 has a moderate load, request 2 may be assigned to draft model 2; and if draft model 3 also has a light load, request 3 may be assigned to draft model 3. In this application, this allocation relationship can be recorded using a format similar to R1M1 (representing selecting draft model 1 for request 1), R2M2 (representing selecting draft model 2 for request 2), and R3M3 (representing selecting draft model 3 for request 3).

[0050] A request queue is a data structure used to manage and organize pending requests. It arranges concurrent requests in a certain order, waiting for the draft model to pull and process them sequentially. Request queues ensure orderly processing of requests and avoid conflicts and chaos between them.

[0051] After assigning a draft model to each concurrent request, these requests are added to the corresponding request queues according to their assigned draft model. For example, all requests assigned to draft model 1 are added to the request queue for draft model 1, requests assigned to draft model 2 are added to the request queue for draft model 2, and so on. In this way, each draft model can pull requests from its own request queue for processing, achieving distributed request processing and load balancing.

[0052] Based on the above embodiments, the method further includes: Each of the inference results is written into a multi-level cache system; The target-based large language model performs parallel verification on multiple inference results to obtain verification results for each inference result, including: Based on the target verification large language model, the inference results in the multi-level caching system are verified in parallel to obtain the verification results of each inference result.

[0053] In this application, after receiving concurrent requests allocated by global routing scheduling, the draft model performs inference calculations based on its own algorithm and parameters.

[0054] During the inference process, the draft model gradually generates intermediate and final inference results, which contain rich information. For example, in text generation tasks, it may contain generated word sequences and semantic representations; in question answering tasks, it may contain answer content and relevant evidence.

[0055] A multi-level caching system is a storage structure composed of different levels of caching, including memory caching and disk caching. Different levels of caching have different characteristics: memory caching offers fast access speeds but relatively small capacities; disk caching offers large capacities but relatively slower access speeds. The purpose of a multi-level caching system is to improve data access efficiency, reduce frequent accesses to the original data source (such as databases and file systems), and thus speed up the overall system processing.

[0056] After generating inference results, the draft model writes these results into a multi-level caching system according to certain rules and strategies. Specifically, results are prioritized for writing to the fast-access memory cache for quick subsequent retrieval. If the memory cache space is insufficient, some infrequently used data is moved out of the memory cache according to the cache eviction policy, and the inference results are written to the disk cache. When writing to the multi-level caching system, corresponding identification information, such as request ID and draft model ID, is also assigned to each inference result to facilitate accurate subsequent lookup and retrieval.

[0057] In this application, the main task of the target validation large language model is to validate and evaluate the inference results generated by the draft model, and determine whether these results meet the expected quality standards and requirements. For example, in text generation tasks, it verifies the grammatical correctness, semantic coherence, and logical rationality of the generated text; in question answering tasks, it verifies the accuracy and completeness of the answers.

[0058] When performing validation, the target validation large language model reads the inference results generated by the draft model from a multi-level caching system. Since the inference results are already stored in the multi-level caching system, the target validation large language model can quickly retrieve this data without waiting for the draft model to be regenerated, thus improving validation efficiency. When reading the inference results, it accurately locates the inference result that needs to be validated based on previously set identification information (such as request ID, draft model ID, etc.).

[0059] In this application, to improve verification efficiency, the target verification large language model employs parallel verification to process multiple inference results simultaneously. Parallel verification can utilize the computing power of multi-core CPUs or multi-GPUs to perform verification calculations on multiple inference results at the same time. For example, if multiple inference results need to be verified, the target verification large language model can divide these results into multiple batches, with each batch assigned to a different computing core or GPU for processing. During parallel verification, the target verification large language model independently verifies each inference result according to preset verification rules and algorithms, determining whether it passes verification.

[0060] After parallel validation, the target validation large language model generates a corresponding validation result for each inference outcome. Validation results typically include two categories: pass and fail. If the inference result meets the expected quality standards and requirements, it is marked as pass; otherwise, it is marked as fail. The validation results may also contain detailed evaluation information, such as the inference score and error type, to facilitate further analysis and processing. For example, in a text generation task, if the generated text contains grammatical errors, the validation result will indicate the type and location of the errors; in a question-answering task, if the answer is inaccurate, the validation result will explain the difference between the answer and the correct answer.

[0061] Based on the above embodiments, obtaining the inference results output by each draft model based on the number of target candidate lexical units corresponding to each draft model includes: Based on the historical verification status, current computational load, and available bandwidth information of the draft model, the number of target candidate lexical units corresponding to the draft model is determined, wherein the available bandwidth information represents the available bandwidth of the draft model when accessing the multi-level caching system. Based on the draft model, the corresponding concurrent requests are inferred to generate the target number of candidate lexical units; The inference result is constructed based on the number of candidate lexical units of the target candidate lexical unit.

[0062] In this application, the historical validation history of the draft model represents the record of the target validation large language model's acceptance of the inference results generated by the current draft model in the past. For example, in previous inference tasks, how many times the inference results output by the draft model were accepted by the target validation large language model, how many times were rejected, and the specific reasons for acceptance and rejection, etc.

[0063] If the historical validation acceptance rate of the draft model is high, it indicates that the quality of its generated inference results is relatively reliable. In this case, the number of target candidate words can be appropriately increased in the current inference task to generate richer and more accurate inference content. Conversely, if the historical validation acceptance rate is low, it may indicate that the draft model is prone to problems during the inference process. In this case, the number of target candidate words will be controlled to avoid generating too much potentially inaccurate content and wasting computing resources.

[0064] The current computational load indicates the number of requests the draft model is currently processing, the utilization of computing resources (such as CPU and GPU), and memory usage. For example, if the draft model's CPU utilization is already high, it means that it is currently processing a large number of computational tasks and is under heavy load.

[0065] When the draft model has a light computational load, it has sufficient computing power to handle more word generation tasks. Therefore, the number of target candidate words can be appropriately increased to speed up inference or generate more detailed results. However, when the computational load is heavy, the number of target candidate words will be reduced to avoid inference process lag or errors due to insufficient resources, ensuring that the inference task can proceed stably.

[0066] Available bandwidth information indicates the network bandwidth that the draft model can currently use when accessing the multi-level caching system. The multi-level caching system may be distributed across different storage devices or servers. During inference, the draft model needs to read data (such as model parameters, intermediate results, etc.) from the caching system and write data generated during inference (such as key-value pairs, etc.). Available bandwidth determines the data transmission speed. If the available bandwidth is large and the data transmission speed is fast, the draft model can obtain the required data in a timely manner and quickly write intermediate results. In this case, the number of target candidate words can be appropriately increased to fully utilize the high-speed data transmission capability. Conversely, if the available bandwidth is small and the data transmission is slow, increasing the number of target candidate words may cause the inference process to stall due to waiting for data transmission. Therefore, the number of target candidate words should be reduced to ensure the smoothness of the inference process.

[0067] Taking into account the above three factors, the draft model dynamically determines the number of target candidate lexical units to be generated in the current inference task through internally preset algorithms or rules. This number is a value that has been weighed and optimized to fully utilize system resources and improve inference efficiency while ensuring inference quality.

[0068] Furthermore, multiple independently deployed draft models pull concurrent requests assigned to them from the request queue. The request queue distributes different concurrent requests to each draft model according to certain rules (such as a global routing scheduling mechanism), ensuring that each draft model has tasks to process.

[0069] After receiving concurrent requests, the draft model begins inference. The inference process mainly consists of two stages: Prefill and Decoding. In the Prefill stage, the draft model generates an initial hidden state and partial output based on the input request information; in the Decoding stage, the draft model gradually generates subsequent lexical units based on the results of the Prefill stage.

[0070] During the inference process, the draft model generates candidate lexics strictly according to the previously determined target number of candidate lexics. Based on its own model parameters and algorithm, it selects the most likely lexics from the vocabulary, gradually constructing a sequence of candidate lexics that meets the target number. These candidate lexics represent the preliminary form of the inference results and require further verification and processing.

[0071] In this application, the candidate lexical units generated by the draft model may be a series of discrete lexical sequences, which need to be organized and arranged. For example, in text generation tasks, the generated lexical units are arranged and combined according to certain grammatical rules and semantic logic to form coherent text fragments.

[0072] After compiling the candidate lexical units, the draft model will construct a complete inference result from these lexical units according to the specific requirements of the inference task. The inference result may include various forms, such as text answers, images, audio, etc., depending on the type of inference task. For example, in a question-answering task, the inference result is the accurate answer to the user's question; in a text creation task, the inference result is a complete article or paragraph.

[0073] After constructing the inference result, the draft model will write the KV (Key-Value pairs, used to store intermediate information in the inference process) generated during the inference process and the final inference result into a multi-level cache system. This allows the subsequent target verification model to verify the inference result and to quickly read and reuse this data when needed, thereby improving the overall system efficiency.

[0074] Based on the above embodiments, the step of performing parallel verification of multiple inference results based on the target-verification large language model to obtain verification results for each inference result includes: Multiple concurrent requests and multiple inference results are concatenated to obtain a concatenated sequence; The concatenated sequence is input into the target validation large language model to obtain the probability distribution of the next word element corresponding to each position in the concatenated sequence; Based on the probability distribution and the preset sampling strategy, the inference results in the spliced ​​sequence are sampled; The selected inference results from the sampling are determined as valid inference results, and the unselected inference results are determined as invalid inference results.

[0075] Figure 2 A schematic diagram of the autonomous selection mechanism for the sampling strategy provided in this application can be found here. Figure 2 As shown in this application, in the entire model inference and verification process, multiple independently deployed draft models pull concurrent requests from the request queue and perform inference, generating their respective inference results. These inference results are stored in a multi-level caching system in the form of key-value (KV) values. The target verification large language model retrieves the speculative inference output KV values ​​of multiple draft models from the multi-level caching system, and also obtains the corresponding original concurrent request information, i.e., concurrent requests.

[0076] For each draft model, the target validation large language model will concatenate the original input Prompt (i.e., the core content of the original concurrent request, such as the task requirements and problem description proposed by the user) and the K candidate tokens (candidate nouns) generated by the draft model into a complete new sequence. The concatenation process combines the two according to certain logic and format to ensure that the new sequence can accurately reflect the relationship between the original request and the inference result.

[0077] In this way, multiple concurrent requests and their corresponding inference results are concatenated to obtain multiple concatenated sequences. Each concatenated sequence corresponds to an original concurrent request and its candidate inference result generated by the draft model.

[0078] The target validation large language model receives the concatenated long sequence as input. This long sequence contains information about the original request and candidate inference results generated by the draft model, providing the target validation large language model with comprehensive contextual information to enable it to perform accurate validation and evaluation.

[0079] In this application, the target validation large language model performs a forward propagation computation on the input concatenated sequence. During the computation, the model utilizes its internal neural network structure (such as the Transformer architecture) to process and analyze each element in the long sequence. Due to the use of parallel processing techniques, this computation processes the entire sequence in parallel, rather than processing each element sequentially, thereby greatly improving computational efficiency.

[0080] After forward propagation computation, the target validation large language model outputs the probability distribution of the next word at each position in the concatenated sequence. This probability distribution reflects the likelihood of each possible next word appearing given the current sequence.

[0081] The target validation large language model can autonomously select and adjust its sampling strategy based on user-defined expected preferences, historical usage preferences, and feedback interactions with the final output. Sampling strategies include greedy decoding, bundle search, Top-k sampling, Top-p sampling, and temperature adjustment, and also support customized sampling strategies. For example, greedy decoding selects the word with the highest probability as the next word at each step; bundle search retains several words with the highest probability (bundle width) at each step, then continues to expand these words, finally selecting the sequence with the highest overall probability as the output, which can balance accuracy and diversity to some extent; Top-k sampling considers only the top k words with the highest probability at each step, and then samples from them, adjusting the value of k can control the diversity and accuracy of the generated text.

[0082] Furthermore, depending on the selected sampling strategy, the target validation large language model samples the inference results in the concatenated sequence based on the probability distribution at each position. For example, if a greedy decoding strategy is used, the model selects the word with the highest probability at each position as the sampling result; if a Top-k sampling strategy is used, the model randomly samples from the top k words with the highest probabilities according to a certain probability distribution.

[0083] After sampling, the target validation large language model will determine the inference results in the concatenated sequence based on the sampling results. If a word in a certain inference result is selected by sampling, it means that the word is considered reasonable and meets the requirements under the given probability distribution and sampling strategy. Therefore, the entire inference result is determined as a validated inference result.

[0084] Conversely, if a word in a reasoning result is not selected for sampling, it means that the word is not suitable under the current evaluation criteria, and may have problems such as grammatical errors, semantic incoherence, or failure to meet user needs. Therefore, the entire reasoning result is determined as a reasoning result that has failed verification.

[0085] In this application, the validated inference results can be further used for subsequent task processing, such as being returned to the user as the final output; while the validated inference results can be discarded or returned to the draft model for re-inference and optimization to improve the quality of subsequent inference results.

[0086] Based on the above embodiments, the large language model decoding includes: If the current load of the target validation large language model is determined to be less than the preset large model load threshold, the inference results that failed the validation are decoded based on the target validation large language model to obtain the inference results that passed the validation. The re-inference process includes: If the current load of the target validation large language model is greater than or equal to the preset large model load threshold, the inference result output by the re-inference draft model is obtained based on the number of new candidate lexical units corresponding to the re-inference draft model. The number of new candidate lexical units is obtained by reducing the number of target candidate lexical units in the re-inference draft model. The re-inference draft model represents the draft model corresponding to the inference result that failed the validation. Based on the target validation large language model, the reasoning results output by the re-inference draft model are re-validated to obtain the valid reasoning results.

[0087] In this application, the current load of the target validation large language model is monitored in real time. The load can be measured by various metrics, such as CPU utilization, GPU utilization, memory usage, and the number of requests being processed.

[0088] The preset large model load threshold is a critical value set based on system performance and actual needs. When the load of the target validation large language model is lower than this threshold, it means that the model currently has enough computing resources to handle additional tasks.

[0089] Besides considering load conditions below a preset threshold, the decision to decode unaccepted speculative inference results from specific draft models may also be based on user-defined ranges. User-defined ranges could include the number of draft models allowed to be decoded, the time frame for decoding, etc. For example, a user might only allow decoding of unaccepted results from some important draft models, or allow decoding operations to be performed within a specific time period. The decoding process is triggered when both the load condition and the user-defined ranges are met.

[0090] Furthermore, the target validation large language model utilizes its internal Decode engine to decode inference results that fail validation. The decoding process typically involves further analysis and optimization of the lexical sequences in the inference result, attempting to correct and improve the result from multiple aspects such as semantics, syntax, and logic. For example, in text generation tasks, the decoding engine might adjust the order of lexical units or replace inappropriate lexical units to make the generated text more fluent, accurate, and in line with user needs.

[0091] After decoding, inference results that initially failed verification may be improved to meet the verification criteria. The target verification large language model will re-evaluate the decoded inference results based on certain evaluation metrics (such as semantic coherence, syntactic correctness, and matching degree with user needs). If the requirements are met, the inference results will be identified as verified and can be used for subsequent task processing or returned to the user.

[0092] When the load on the target validation large language model is greater than or equal to the preset large model load threshold, it indicates that the target validation large language model is currently under heavy load and lacks sufficient computing resources to decode the inference results that failed validation. In this case, it is necessary to determine the draft model to be re-inferred, i.e., the re-inference draft model. The re-inference draft model is the draft model that generated the inference results that failed validation. Each draft model has its unique model parameters and inference capabilities. For different validation failure results, it is necessary to return to the corresponding draft model for re-inference.

[0093] In this application, the number of target candidate lexical units in the re-inference draft model is reduced to obtain a new number of candidate lexical units. The target number of candidate lexical units is dynamically determined by the draft model during initial inference based on factors such as its historical validation, current computational load, and available bandwidth. Reducing this number during re-inference reduces the number of candidate lexical units generated by the draft model, thereby reducing the computational load and time cost of the inference process. For example, if the original target number of candidate lexical units was 100, it may be reduced to 50 after the reduction.

[0094] The re-inference draft model re-infers the inference based on the new number of candidate lexical units, generating a new inference result. During the inference process, the draft model re-analyzes the original input request, combines it with the new candidate lexical unit limit, generates a new set of candidate lexical unit sequences, and finally constructs a new inference result. Then, the new inference result output by the re-inference draft model is input again into the target validation large language model, and re-validated according to the previously set validation process and evaluation metrics. The target validation large language model comprehensively evaluates the semantics, syntax, and logic of the inference result to determine whether it meets the user's needs and set standards.

[0095] If the re-inference result passes the validation of the target large language model, meaning it meets the requirements of all evaluation metrics, it is determined to be a validated inference result. These validated inference results can be used for subsequent task processing, for example, returned to the user as the final output, or used as input to other models for further processing. If it still fails validation, it may be necessary to continue re-inference or other processing measures until a valid inference result is obtained.

[0096] Based on the above embodiments, the method further includes: If it is determined that the reasoning result output by the re-inference draft model fails the verification, the concurrent request corresponding to the re-inference draft model is switched to another draft model for reasoning, or, based on the target verification large language model, the reasoning result output by the re-inference draft model is decoded.

[0097] In this application, after the re-inference draft model re-infers the inference result that failed the verification and outputs a new inference result, the target verification large language model will re-verify the new inference result according to the established verification process and evaluation indicators.

[0098] If the reasoning result after this re-inference still fails the verification, the subsequent processing method is determined based on the user configuration. The user configuration may include various options, such as the number of times the draft model can be switched, whether the target verification large language model is allowed to decode itself, the conditions for decoding, etc., and the user can set an upper limit on the number of speculative inference attempts.

[0099] If the inference results output by the re-inference draft model fail to pass the validation of the target large language model multiple times (the number of times is configurable), it indicates that the draft model may have limitations in the current task scenario and cannot generate inference results that meet the requirements. For example, in a text generation task, a draft model may perform poorly when processing text of a specific topic or style, resulting in semantically incoherent text, numerous grammatical errors, or text that does not meet the user-defined topic requirements. In this case, a suitable model is selected as the new inference model from the available draft models according to the preset draft model list or switching rules in the user configuration. This selection process may consider factors such as the draft model's historical performance, the types of tasks it is good at handling, and the current system load. For example, if the user configuration specifies that a draft model with better performance and lower load should be selected for the current task type, the selection will be carried out according to this rule.

[0100] After determining the new draft model, the concurrent requests (including original input information, task requirements, etc.) corresponding to the re-inference draft model are switched to this new draft model. The new draft model will receive requests again and perform speculative inference, generate new inference results, and then submit them again to the target validation large language model for validation.

[0101] If the maximum number of switching attempts is reached and no valid reasoning result is still not obtained, other measures may be taken, such as returning a failure message or conducting a more in-depth investigation.

[0102] If the configuration allows the target validation large language model to perform decoding itself, and in the current situation (e.g., there are no other suitable draft models to switch to, or you want to try to improve the results through decoding), choose to decode the inference results output by the re-inference draft model based on the target validation large language model. The target validation large language model has more powerful language understanding and processing capabilities, and it is possible to correct and improve the inference results through decoding operations to make them meet the validation standards.

[0103] The target validation large language model utilizes its internal decoding engine to perform in-depth analysis of the inference results output by the re-inference draft model. The decoding engine evaluates the inference results from multiple aspects, including semantics, syntax, and logic, identifying problems and shortcomings. For example, in text generation tasks, the decoding engine might find issues such as unclear wording or disorganized sentence structure in the inference results. To address these identified problems, the decoding engine attempts to correct and optimize them, potentially adjusting the order of lexical units, replacing inappropriate units, or supplementing missing information, to make the inference results more fluent, accurate, and meet user needs.

[0104] After decoding, the target validation large language model will validate the decoded inference result again. If the decoded result meets the requirements of each evaluation metric, i.e., the validation passes, then the result can be used for subsequent task processing or returned to the user. If the decoded result still fails validation and the user-set limit for speculative inference attempts has been reached, a failure message may be returned according to the user configuration, informing the user that a valid inference result cannot be generated.

[0105] Figure 3 The overall flowchart of inference acceleration based on multi-level caching provided in this application can be found here. Figure 3 As shown, after receiving the high-concurrency original prompt, the system uses a global routing scheduling and load balancing mechanism to distribute it to each draft model for speculative inference.

[0106] Specifically, when the system receives a large number of concurrent raw Prompt inputs, it first initiates a global routing and load balancing mechanism. This mechanism monitors the load status of each draft model in real time, including computational resource utilization and the number of requests processed. Simultaneously, it considers factors such as cache affinity routing to optimize data transmission efficiency. Based on this information, the system rationally distributes the raw Prompt requests to the corresponding draft models in the request queue (e.g., draft model 1, draft model 2, ..., draft model N), ensuring a relatively balanced load across all draft models and preventing situations where one model is overloaded while others are idle.

[0107] Furthermore, each draft model employs a multi-terminal parallel adaptive inference mechanism to generate K candidate tokens (K may differ for each draft model) in parallel for each request. Specifically, upon receiving an assigned request, each draft model initiates the multi-terminal parallel adaptive inference mechanism. During inference, the draft model does not fix the number of generated tokens but dynamically adjusts and generates K candidate tokens for each request based on factors such as its historical validation adoption rate (e.g., the proportion of past inference results accepted by the target validation model), current load (e.g., CPU and GPU utilization), and available bandwidth for accessing the multi-level caching system (the value of K may differ for different draft models). This adaptive adjustment mechanism helps to fully utilize system resources and improve inference efficiency while ensuring inference quality.

[0108] Next, the system concatenates the original input Prompt with the K candidate tokens generated by the draft model into a complete new sequence. After retrieving the K candidate tokens generated by the draft model from the multi-level caching system, the system concatenates the original input Prompt with these candidate tokens to form a complete new sequence. This concatenation process follows certain logic and format to ensure that the new sequence accurately reflects the relationship between the original request and the inference result, providing complete contextual information for subsequent target validation processing of the large language model.

[0109] Then, the target validation large language model takes this concatenated long sequence as input and performs a forward propagation. This computation processes the entire sequence in parallel and outputs the probability distribution of the next token corresponding to each position in the sequence.

[0110] In this application, the target validation large model retrieves the concatenated long sequence from a multi-level caching system as input and initiates a forward propagation computation. Due to the use of parallel processing techniques, the target validation large model can process the entire sequence simultaneously, rather than processing each element sequentially, thus significantly improving computational efficiency. After the forward propagation computation is completed, the target validation large model outputs the probability distribution of the next token corresponding to each position in the sequence. This probability distribution reflects the likelihood of each possible next token appearing given the current sequence.

[0111] After obtaining the probability distribution for each position, a sampling strategy-based autonomous selection mechanism is used to decide whether to accept the draft token. Specifically, based on the probability distribution of each position in the output of the target validation model, combined with user-defined expected preferences (such as desired output style or theme), historical usage preferences (the degree of acceptance of similar outputs by past users), and feedback interactions on the final output (user feedback on previous outputs), an autonomous selection mechanism is used to decide whether to accept the draft token. Sampling strategies include greedy decoding, bundle search, and Top-k sampling; the system will select the appropriate sampling strategy based on the actual situation.

[0112] For unaccepted tokens, if the target verification model is in a light-load state and the user allows it, the target verification model will use the Decode engine to decode the unaccepted speculative reasoning results, and attempt to correct and improve the reasoning results to meet the verification standards.

[0113] If the target validation model itself is overloaded or does not meet other relevant conditions, the system will require the original draft model to reduce the number of speculative outputs M and perform speculative inference again.

[0114] Repeat the above process until all inference requests are completed. If some draft models or target validation large language models become overloaded or fail during this period, elastic scaling and hot migration can be performed based on a multi-level caching system to ensure the continuity and stability of the entire inference task until all inference requests have been processed.

[0115] Figure 4 A schematic diagram illustrating the hardware interaction between the inference node and the acceleration node provided in this application can be found in the attached diagram. Figure 4 As shown, the inference node is mainly responsible for executing the inference task of the draft model. Its hardware composition and related functions are as follows: Prefill GPU and Decode GPU: These two GPU components are responsible for the prefill and decoding tasks in the inference process, respectively. The Prefill GPU performs preliminary processing on the original input prompt to generate initial hidden states and other information; the Decode GPU, based on the results of the Prefill stage and subsequent inference requirements, gradually generates specific tokens to complete the decoding operation in the inference process.

[0116] vllm: Target Validation Large Language Model.

[0117] KV Connector: Used to connect and manage key-value (KV) cache operations during inference. During inference, a large amount of KV cache data is generated. The KV Connector is responsible for the transmission, storage, and retrieval of this data, ensuring the smooth progress of the inference process.

[0118] RDMA (Remote Direct Data Access) Network: Inference nodes use RDMA networks for high-speed data transmission with other components, such as for operations like cache lookup, cache storage, and cache prefetching. Cache lookup is used to find the required cached data in a multi-level cache system; cache storage stores intermediate results and other data generated during inference into the cache system; and cache prefetching reads potentially needed data from the cache or storage in advance to reduce data access latency and improve inference efficiency.

[0119] Acceleration nodes primarily provide additional computational and storage support to assist inference nodes in completing inference tasks. Their components and functions are as follows: CPU: As the central processing unit of the acceleration node, it is responsible for coordinating and managing various operations of the entire acceleration node, handling some computing tasks that are not suitable for GPU processing, and communicating with other system components.

[0120] RPC Server: Remote Procedure Call server, used to receive and process remote call requests from inference nodes or other components, enabling function calls and data interaction between different nodes.

[0121] DRAM (Dynamic Random Access Memory): As the main memory of acceleration nodes, it is used to temporarily store data being processed and related information during program execution, providing fast data access for the CPU and other components.

[0122] SSD (Solid State Drive): Provides persistent storage functionality for long-term storage of data that is not frequently accessed but needs to be retained, such as model parameters and historical inference results.

[0123] In this application, the inference node and the acceleration node are connected via a 100Gbps high-bandwidth link. This high-speed connection can meet the needs of rapid transmission of large amounts of data, ensuring timely data interaction between different nodes during inference and reducing data transmission bottlenecks. Furthermore, network impairment scenarios, such as network latency and packet loss, can be simulated using an impairment tester to test the system's performance and stability in complex network environments. This helps ensure reliable system operation under various network conditions during actual deployment, and allows for the early detection and resolution of potential network-related problems.

[0124] The inference acceleration device provided in the embodiments of this application is described below. The inference acceleration device described below can be referred to in correspondence with the inference acceleration method described above.

[0125] Figure 5 A schematic diagram of the inference acceleration device provided in this application is shown below. Figure 5 As shown, this application provides an inference acceleration device, including an allocation module 501, an adaptive inference module 502, a large model verification module 503, a large model processing decision module 504, and an inference output module 505. The allocation module 501 is used to allocate multiple received concurrent requests to corresponding draft models. The adaptive inference module 502 is used to obtain the inference results output by each draft model based on the number of target candidate lexical units corresponding to each draft model. The large model verification module 503 is used to perform parallel verification of multiple inference results based on a target verification large language model to obtain verification results for each inference result. The large model processing decision module 504 is used to perform re-inference processing or large language model decoding on inference results that fail verification based on the current load of the target verification large language model to obtain verified inference results. The inference output module 505 is used to complete the inference task of multiple concurrent requests after determining that the inference results of all draft models have passed verification.

[0126] The inference acceleration device provided in this application obtains inference results by distributing multiple concurrent requests to corresponding draft models, and then uses a target verification large language model for parallel verification. Based on the load of the large language model, the failed results are re-inferred or decoded. Finally, all inference results pass the verification and the inference task is completed, thereby improving inference efficiency and reducing deployment costs.

[0127] Figure 6 A schematic diagram of the structure of the electronic device provided in this application, such as... Figure 6 As shown, the electronic device may include: a processor 601, a communications interface 602, a memory 603, and a communication bus 604, wherein the processor 601, the communications interface 602, and the memory 603 communicate with each other through the communication bus 604. The processor 601 can call logical instructions in the memory 603 to execute an inference acceleration method, which includes: allocating multiple received concurrent requests to corresponding draft models; obtaining inference results output by each draft model based on the number of target candidate lexical units corresponding to each draft model; performing parallel verification of the multiple inference results based on a target verification large language model to obtain verification results for each inference result; performing re-inference processing or large language model decoding on inference results that fail verification according to the current load of the target verification large language model to obtain verification results that pass verification; and completing the inference task of the multiple concurrent requests after determining that the inference results of all draft models have passed verification.

[0128] Furthermore, the logical instructions in the aforementioned memory 603 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] On the other hand, this application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the inference acceleration method provided by the above methods. The method includes: allocating multiple received concurrent requests to corresponding draft models; obtaining inference results output by each draft model based on the number of target candidate lexical units corresponding to each draft model; performing parallel verification of the multiple inference results based on a target verification large language model to obtain verification results for each inference result; performing re-inference processing or large language model decoding on inference results that fail verification according to the current load of the target verification large language model to obtain verification results that pass verification; and completing the inference task of the multiple concurrent requests after determining that the inference results of all draft models have passed verification.

[0130] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the inference acceleration method provided in the above embodiments. The method includes: allocating multiple received concurrent requests to corresponding draft models; obtaining inference results output by each draft model based on the number of target candidate lexical units corresponding to each draft model; performing parallel verification of the multiple inference results based on a target verification large language model to obtain verification results for each inference result; performing re-inference processing or large language model decoding on inference results that fail verification according to the current load of the target verification large language model to obtain verification results that pass verification; and completing the inference task of the multiple concurrent requests after determining that the inference results of all draft models have passed verification.

[0131] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0132] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for accelerating inference, characterized in that, include: Distribute multiple received concurrent requests to the corresponding draft models; Based on the number of target candidate lexical units corresponding to each of the draft models, the inference results output by each of the draft models are obtained; Based on the target-validation large language model, multiple inference results are validated in parallel to obtain the validation results of each inference result; Based on the target verification of the current load of the large language model, perform re-inference processing or large language model decoding on the inference results that fail verification to obtain the inference results that pass verification; After verifying that the inference results of all the draft models have passed the verification, the inference task for multiple concurrent requests is completed.

2. The reasoning acceleration method according to claim 1, characterized in that, The step of allocating multiple received concurrent requests to corresponding draft models includes: Based on the global routing schedule and the current load of the draft model, the draft model corresponding to each concurrent request is determined, and each concurrent request is added to the request queue.

3. The reasoning acceleration method according to claim 1, characterized in that, The method further includes: Each of the inference results is written into a multi-level cache system; The target-based large language model performs parallel verification on multiple inference results to obtain verification results for each inference result, including: Based on the target verification large language model, the inference results in the multi-level caching system are verified in parallel to obtain the verification results of each inference result.

4. The reasoning acceleration method according to claim 3, characterized in that, The step of obtaining the inference results output by each draft model based on the number of target candidate lexical units corresponding to each draft model includes: Based on the historical verification status, current computational load, and available bandwidth information of the draft model, the number of target candidate lexical units corresponding to the draft model is determined, wherein the available bandwidth information represents the available bandwidth of the draft model when accessing the multi-level caching system. Based on the draft model, the corresponding concurrent requests are inferred to generate the target number of candidate lexical units; The inference result is constructed based on the number of candidate lexical units of the target candidate lexical unit.

5. The inference acceleration method according to claim 1, characterized in that, The target-based large language model performs parallel verification on multiple inference results to obtain verification results for each inference result, including: Multiple concurrent requests and multiple inference results are concatenated to obtain a concatenated sequence; The concatenated sequence is input into the target validation large language model to obtain the probability distribution of the next word element corresponding to each position in the concatenated sequence; Based on the probability distribution and the preset sampling strategy, the inference results in the spliced ​​sequence are sampled; The selected inference results from the sampling are determined as valid inference results, and the unselected inference results are determined as invalid inference results.

6. The inference acceleration method according to claim 1, characterized in that, The large language model decoding includes: If the current load of the target validation large language model is determined to be less than the preset large model load threshold, the inference results that failed the validation are decoded based on the target validation large language model to obtain the inference results that passed the validation. The re-inference process includes: If the current load of the target validation large language model is greater than or equal to the preset large model load threshold, the inference result output by the re-inference draft model is obtained based on the number of new candidate lexical units corresponding to the re-inference draft model. The number of new candidate lexical units is obtained by reducing the number of target candidate lexical units in the re-inference draft model. The re-inference draft model represents the draft model corresponding to the inference result that failed the validation. Based on the target validation large language model, the reasoning results output by the re-inference draft model are re-validated to obtain the valid reasoning results.

7. The inference acceleration method according to claim 6, characterized in that, The method further includes: If it is determined that the reasoning result output by the re-inference draft model fails the verification, the concurrent request corresponding to the re-inference draft model is switched to another draft model for reasoning, or, based on the target verification large language model, the reasoning result output by the re-inference draft model is decoded.

8. A reasoning acceleration device, characterized in that, include: The allocation module is used to allocate multiple received concurrent requests to the corresponding draft models; An adaptive inference module is used to obtain the inference results output by each of the draft models based on the number of target candidate lexical units corresponding to each of the draft models. The large model verification module is used to verify the large language model based on the target, and to perform parallel verification of multiple inference results to obtain the verification results of each inference result; The large model processing decision module is used to verify the current load of the large language model according to the target, perform re-inference processing or large language model decoding on the inference results that fail the verification, and obtain the inference results that pass the verification. The inference output module is used to complete the inference task for multiple concurrent requests after determining that the inference results of all the draft models have passed the verification.

9. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the inference acceleration method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the inference acceleration method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the inference acceleration method according to any one of claims 1 to 7.

Images