Large language model reasoning method and device, equipment, storage medium and program product

Abstract

The application provides a large language model reasoning method and device, equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence. When a CPU executes a draft model reasoning task of a previous round in a DCU, input memory corresponding to a main model is pre-allocated for a request batch of the current round; input data of the main model of the current round is generated and copied to the input memory corresponding to the main model; the input data indicates a reasoning result of the draft model of the previous round by using a placeholder; a first copy instruction and a main model reasoning task are sent to the DCU; the first copy instruction is used to instruct the DCU to copy the reasoning result of the draft model of the previous round to the input memory to overwrite the placeholder; in the case that the reasoning result of the main model of the current round is obtained, the reasoning result of the main model of the current round is checked, and the next round of reasoning is performed according to the checking result. The serial dependence problem of the CPU and the DCU computing task is solved, and the utilization rate of the computing resource is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, storage medium, and program product for large language model reasoning. Background Technology

[0002] With the widespread application of Large Language Models (LLMs) in fields such as intelligent customer service, content generation, multi-turn dialogue, and real-time data analysis, users' demand for model inference efficiency is increasing.

[0003] Traditional LLM inference frameworks suffer from performance bottlenecks when handling high-concurrency, real-time-critical tasks. Specifically, while Multi-Token Prediction (MTP) technology accelerates generation through parallel decoding, its scheduling logic involves significant waiting and idle states between the Central Processing Unit (CPU) and the Deep Computing Unit (DCU), limiting the full utilization of hardware performance. Summary of the Invention

[0004] This application provides a method, apparatus, device, storage medium, and program product for large language model inference, which is conducive to making full use of hardware performance.

[0005] Firstly, this application provides a large language model inference method applied to a CPU. The large language model includes a main model and a draft model. The method includes:

[0006] When the Deep Computing Unit (DCU) executes the previous round of draft model inference task, it pre-allocates the input memory corresponding to the main model for the current round of request batches.

[0007] Generate the input data for the current round of the main model and copy the input data to the corresponding input memory of the main model; the input data uses placeholders to represent the inference results of the draft model in the previous round;

[0008] Send a first copy instruction and a main model inference task to the DCU; the first copy instruction is used to instruct the DCU to copy the inference results of the draft model in the previous round to the input memory to overwrite the placeholder;

[0009] Having obtained the reasoning result of the main model in this round, we check the reasoning result of the main model in this round and proceed with the next round of reasoning based on the check result.

[0010] This application pre-allocates memory locations so that the main model input can be prepared without waiting for the inference results of the draft model. The memory locations can be reserved in advance and directly used for the main model inference task, avoiding computational interruptions due to waiting. An asynchronous copy mechanism further transfers the inference results of the draft model directly to the pre-allocated memory locations. Through these techniques, the collaboration process between the CPU and DCU is decoupled, improving the utilization of computing resources and optimizing the throughput and real-time performance of the inference task.

[0011] In one possible implementation, after sending the first copy instruction and the master model inference task to the DCU, the method further includes:

[0012] Pre-allocate input memory for the draft model for this round of requests;

[0013] The input data for the current round of the draft model is generated and copied into the input memory corresponding to the draft model; the input data uses placeholders to represent the inference results of the main model in this round.

[0014] Send a second copy instruction and a draft model inference task to the DCU; the second copy instruction is used to instruct the DCU to copy the inference result of the main model in this round to the input memory corresponding to the draft model, so as to overwrite the placeholder.

[0015] This implementation method reserves input memory in advance by using placeholders, so that the draft model input preparation stage does not need to wait for the main model inference result, breaking the waiting dependency between the CPU and DCU and reducing the CPU's idle time.

[0016] In one possible implementation, the method further includes, before checking the inference result of the main model in this round:

[0017] The inference results of the main model in the current round are read from memory; the memory stores the inference results of the main model in the current round (copied from the DCU) and the inference results of the draft model in the previous round.

[0018] In this embodiment, after the main model inference is completed, the DCU packages the inference result of the main model and the original draft data (i.e., the inference result of the draft model in the previous round) and sends them back to the CPU memory. In this way, the CPU can check the inference result of the main model in the current round while the DCU is executing the draft model inference task. Compared with the related technology, the CPU has to wait for the DCU to complete the draft model inference task before it can obtain and check the inference result of the main model in the current round. This application reduces the CPU idle waiting time.

[0019] In one possible implementation, sending the first copy instruction and the master model inference task to the DCU includes:

[0020] Simultaneously, the first copy instruction and the main model inference task are sent to the DCU;

[0021] Alternatively, send the first copy instruction to the DCU, and then send the main model inference task to the DCU after the copy instruction is sent.

[0022] This implementation supports two modes: "simultaneous issuance of instructions and tasks" and "issuing instructions first, then tasks." This allows technicians to flexibly choose the CPU-side instruction sending strategy based on specific application scenarios, hardware characteristics, or debugging needs. This design provides configurability for task scheduling timing, facilitating targeted performance tuning or compatibility adaptation in different system environments, thus improving the method's versatility and ease of use.

[0023] Secondly, this application provides a large language model inference method applied to a deep computing unit (DCU). The large language model includes a master model and a draft model. The method includes:

[0024] During the execution of the previous round of draft model inference task, the first copy instruction and main model inference task are received from the central processing unit (CPU).

[0025] After the draft model completes the previous round of reasoning, according to the first copy instruction, the reasoning result of the draft model in the previous round is copied to the input memory pre-allocated by the main model to overwrite the placeholders in the input data in the input memory, so as to obtain the updated input data.

[0026] Based on the updated input data, the main model inference task is executed.

[0027] In this application, after completing the previous round of draft computation, the DCU does not need to wait for the CPU's main model input to prepare. Instead, it directly fills the inference results of the draft model into the pre-allocated input memory according to the first copy instruction. This mechanism helps to eliminate the serial waiting delay caused by data dependency between the CPU and the DCU, and improves the overall throughput of multi-round inference.

[0028] In one possible implementation, copying the inference results of the draft model from the previous round to the pre-allocated input memory of the main model according to the first copy instruction includes:

[0029] Based on the request identifiers corresponding to each request in the first copy instruction, the inference results of the previous round of the draft model corresponding to the request identifier are copied to the memory location in the input memory corresponding to the request identifier.

[0030] In multi-user concurrent scenarios, a precise correspondence is established between the draft model inference results and the main model input memory location through request identifiers. This ensures that the main model input data of each session is not interfered with and is accurately positioned in the DCU memory. This implementation method guarantees the correctness of the mechanism under high concurrency and helps to avoid inference errors caused by memory misalignment.

[0031] In one possible implementation, the method further includes:

[0032] Receive the second copy instruction and draft model inference task sent by the CPU;

[0033] After completing the main model inference task for this round, the inference result of the main model for this round is copied to the input memory pre-allocated for the draft model according to the second copy instruction, so as to overwrite the placeholders in the input data in the input memory and obtain the updated input data.

[0034] Based on the updated input data, the draft model inference task is executed.

[0035] In this implementation, the asynchronous copy mechanism allows the inference results of the main model to be directly reused on the DCU, which helps to avoid the serial dependency between the CPU and the DCU.

[0036] In one possible implementation, the method further includes:

[0037] After completing the main model inference task, the inference results of the main model in this round and the inference results of the draft model in the previous round are copied into the CPU memory.

[0038] In this embodiment, after the main model inference is completed, the DCU packages the inference result of the main model and the original draft data (i.e., the inference result of the draft model in the previous round) and sends them back to the CPU memory. In this way, the CPU can check the inference result of the main model in the current round while the DCU is executing the draft model inference task. Compared with the related technology, the CPU has to wait for the DCU to complete the draft model inference task before it can obtain and check the inference result of the main model in the current round. This application reduces the CPU idle waiting time.

[0039] Thirdly, this application provides a large language model reasoning device integrated into a CPU. The large language model includes a main model and a draft model. The device includes:

[0040] The pre-allocation module is used to pre-allocate the input memory corresponding to the main model for the current request batch when the DCU is executing the previous round of draft model inference task;

[0041] The generation module is used to generate the input data for the main model in this round and copy the input data to the input memory corresponding to the main model; the input data uses placeholders to represent the inference results of the draft model in the previous round;

[0042] The sending module is used to send a first copy instruction and a main model inference task to the DCU; the first copy instruction is used to instruct the DCU to copy the inference results of the draft model in the previous round to the input memory to overwrite the placeholder;

[0043] The inspection module is used to inspect the inference results of the main model in the current round after obtaining the inference results of the main model in the current round, and to perform the inference in the next round based on the inspection results.

[0044] Fourthly, this application provides a large language model inference device integrated into a DCU. The large language model includes a main model and a draft model. The device includes:

[0045] The receiving module is used to receive the first copy instruction and the main model inference task sent by the central processing unit (CPU) during the execution of the previous round of draft model inference task.

[0046] The copy module is used to copy the inference results of the previous round of the draft model to the input memory pre-allocated by the main model according to the first copy instruction after the draft model has completed the previous round of inference, so as to overwrite the placeholders in the input data in the input memory and obtain the updated input data.

[0047] The execution module is used to perform the main model inference task based on the updated input data.

[0048] Fifthly, this application provides an electronic device, including: a CPU, a DCU, and a memory that is communicatively connected to the CPU and the DCU respectively;

[0049] The memory stores computer-executed instructions;

[0050] The CPU is used to execute computer execution instructions stored in the memory to implement the first aspect and / or various possible implementations of the first aspect as described above;

[0051] The DCU is used to execute computer execution instructions stored in the memory to implement the second aspect and / or various possible implementations of the second aspect as described above.

[0052] In a sixth aspect, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the first aspect and / or various possible implementations of the first aspect, or the second aspect and / or various possible implementations of the second aspect.

[0053] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed, implements the first aspect and / or various possible implementations of the first aspect, or the second aspect and / or various possible implementations of the second aspect.

[0054] The large language model inference method, apparatus, device, storage medium, and program product provided in this application, when the DCU executes the previous round of draft model inference task, the CPU pre-allocates input memory for the main model through memory pre-allocation and directly generates input data containing placeholders, which is then copied to the pre-allocated input memory. This eliminates the need to wait for the inference results of the draft model, thus avoiding computational interruptions caused by waiting and quickly completing the main model input preparation. The DCU does not need to wait for the CPU's main model input preparation; instead, after the previous round of draft model inference task is completed, it uses an asynchronous copy mechanism to directly copy the inference results of the draft model from the previous round to the pre-allocated input memory, allowing it to directly enter the main model inference stage. Through these technical means, the collaboration process between the CPU and DCU is decoupled, solving the serial dependency problem of CPU and DCU computation tasks in related technologies, improving the utilization of computing resources, and optimizing the throughput and real-time performance of the inference task. Attached Figure Description

[0055] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0056] Figure 1 This is a schematic diagram of an MTP inference process in related technologies;

[0057] Figure 2 A schematic diagram of the system architecture for a large language model reasoning method provided in this application;

[0058] Figure 3 A flowchart illustrating a large language model reasoning method provided in this application;

[0059] Figure 4 A flowchart illustrating another large language model reasoning method provided in this application;

[0060] Figure 5 A schematic diagram of the interaction flow of a large language model reasoning method provided in this application;

[0061] Figure 6 A schematic diagram of an MTP inference process provided in this application;

[0062] Figure 7 A schematic diagram of the structure of a first-class language model reasoning device provided in this application;

[0063] Figure 8 A schematic diagram of the structure of a second-largest language model reasoning device provided in this application;

[0064] Figure 9 This is a schematic diagram of the structure of an electronic device provided in this application.

[0065] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0066] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0067] The terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. The terms “at least one,” “multiple,” “each,” and “any” used in this application include “at least one,” “multiple,” “each,” and “any,” where “at least one” includes one, two, or more, “multiple” includes two or more, “each” refers to each of the corresponding multiples, and “any” refers to any one of the multiples.

[0068] It should be noted that the "and / or" mentioned in this article describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the related objects before and after it are in an "or" relationship.

[0069] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation portals for users to choose to authorize or refuse.

[0070] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0071] Transformer architecture: A deep learning model architecture based on self-attention mechanism, which is the current mainstream LLM architecture.

[0072] The large language model inference engine is a key software system connecting a trained large language model with real-world applications. Its core responsibility is to efficiently and stably perform forward propagation computation of the model in a production environment, quickly and accurately transforming user input (such as text prompts) into model output (such as generated text, decisions, etc.). Unlike model training, the inference engine focuses more on low latency, high throughput, low cost, and resource efficiency.

[0073] With the widespread application of Large Language Models (LLMs) in fields such as intelligent customer service, content generation, multi-turn dialogue, and real-time data analysis, users' demands for model inference efficiency are increasing. In practical applications, LLM inference engines need to handle massive concurrent requests from users simultaneously. Each request may involve long contexts, multi-turn interactions, or complex task logic, which places stringent requirements on the system's low latency and high throughput capabilities. For example, in intelligent customer service scenarios, users may initiate hundreds of dialogue requests simultaneously, and each dialogue needs to generate natural language responses in real time; in content generation scenarios, the system needs to complete the generation and validation of large-scale text within seconds.

[0074] On the one hand, LLMs have a massive number of parameters (ranging from billions to trillions), and the inference process involves a large number of matrix multiplications and attention calculations, placing extremely high demands on computing resources and memory (especially GPU memory). Reducing inference costs without sacrificing performance is the primary challenge. On the other hand, LLMs typically use an autoregressive approach to generate text token by token. Each time a new token is generated, the complete context sequence (including all previously generated tokens) needs to be re-inputted into the model for computation, causing the computational load to increase linearly with the generation length, creating a performance bottleneck. Furthermore, during inference, model parameters need to be repeatedly loaded from GPU memory into the computation unit. When the model is very large, parameter loading can become a more time-consuming step than the computation itself, limited by memory bandwidth. Finally, in actual deployments, the inference engine needs to handle requests from multiple users simultaneously. How to efficiently queue, batch process, allocate resources, and schedule requests to maximize hardware utilization and ensure response latency for each user is key to measuring engine performance.

[0075] To address these challenges, the industry has developed a variety of optimization technologies:

[0076] Model quantization: Compressing model parameters from high precision to low precision to reduce model size, memory usage, and computational load, thereby improving inference speed and reducing costs.

[0077] Key-Value Cache (KV Cache): To address the issue of redundant calculations in autoregressive generation, KV Cache caches and reuses the key-value matrix of historical tokens, avoiding redundant calculations in each generation step.

[0078] Speculative Decoding: This method uses a smaller draft model to quickly generate multiple candidate tokens, which are then verified in parallel by a larger target model (also known as the main model). This process handles multiple tokens in a single forward propagation, thus improving the generation speed.

[0079] Continuous batch processing and request scheduling: Dynamically package different request sequences into a batch for processing, avoiding idle computing resources caused by waiting for long sequences to complete, and maximizing the throughput of computing units.

[0080] PagedAttention is a memory management optimization technique for key-value caches. By paging the key-value cache, it solves the problems of memory fragmentation and redundancy reservation in traditional key-value cache management, allowing for more flexible and efficient batch processing and long sequence processing.

[0081] Despite the aforementioned optimization techniques, existing inference engines still have room for improvement in terms of flexibility, intelligence, and resource allocation efficiency when dealing with complex multi-turn dialogues, long contextual understanding, and specific tasks that require dynamic adjustment of inference strategies.

[0082] Traditional large language models use Next-Token Prediction (NTP) as their training objective, which is to predict the next token (t+1) at position t.

[0083] The core idea of ​​Multi-Token Prediction (MTP) is to predict not only the next token, but also multiple future tokens simultaneously. This improves inference efficiency. For example, when n=4 (predicting 4 tokens at a time), the inference speed can be accelerated by approximately 3 times.

[0084] Current LLMs are based on a transformer architecture. In the token generation stage of decoding, LLMs primarily use an autoregressive approach, meaning the prediction of the next token depends on the output of the previous token. However, this doesn't mean LLMs cannot perform multi-token inference. In the prefill stage, LLMs can take all input tokens at once, generate a key-value cache, and predict the next token.

[0085] LLM predicts the next token for each input token, but during the prefill stage, it primarily focuses on the last token, ignoring previous tokens regardless of their accuracy. MTP, or parallel decoding / speculation decoding, combines decoding and prefilling. It uses a small-parameter draft model for autoregressive decoding, while the main model takes multiple tokens predicted by the draft model as input and outputs multiple tokens simultaneously. By comparing the outputs with the draft model's outputs, it accepts or rejects the draft model's results and simultaneously predicts the next token. The main model's primary role becomes checking the draft model's results for correctness. As long as the acceptance rate remains high, LLM inference can improve the inference efficiency of large models by decoding multiple tokens in parallel without affecting the overall model accuracy. Numerous inference frameworks and practices have demonstrated that MTP is an effective way to improve inference speed.

[0086] Current MTP technology can improve the inference speed of LLM, but there are still some shortcomings in the details of the inference process, and the potential of MTP has not been fully realized.

[0087] Figure 1 This is a schematic diagram of an MTP inference process in related technologies, such as... Figure 1 As shown, the specific steps include 1-5.

[0088] 1. **Primer Input Preparation (PREPARE_INPUT):** For each of the multiple requests, the CPU generates the prime model input data for the current round of that request based on the inference results of the draft model from the previous round. Multiple requests are merged into a single batch, and the prime model input data corresponding to each request in the batch is copied from CPU memory to DCU video memory. The prime model input data for each request includes the inference results of the draft model from the previous round.

[0089] 2. Master Model Inference (TARGET_MODEL): The DCU uses the master model as input data to perform parallel forward inference, outputs the inference results, and copies the inference results to the CPU's memory.

[0090] 3. Draft model input preparation (PREPARE_INPUT): The CPU obtains the inference result of the main model in this round and prepares the draft model input data for this round. The prepared draft model input data is copied from the CPU memory to the DCU video memory.

[0091] 4. Draft Model Inference (DRAFT): The DCU uses the draft model to perform inference through autoregression based on the input data, outputs the inference results, and copies the inference results into the CPU's memory.

[0092] 5. Inference Result Check (CHECK OUT): The CPU checks whether the inference result of the main model meets the conditions. If it does, the inference task is considered complete. The request to complete the inference task is removed from the batch, and its key-value cache is released to prepare for the next inference.

[0093] It is evident that while the aforementioned MTP technology accelerates generation through parallel decoding, its scheduling logic involves numerous waiting and idle states in the collaboration between the CPU and DCU, limiting the full utilization of hardware performance. Furthermore, when deploying large language model inference services on domestically produced computing platforms (such as DCUs), it is necessary to consider both hardware characteristics and algorithm optimization to achieve efficient utilization of computing resources.

[0094] Therefore, there is an urgent need for an efficient inference scheduling method for domestically produced computing power platforms to break through the performance limitations of traditional technologies and meet the real-time, stability, and scalability requirements of large language model inference services in real-world scenarios.

[0095] The large language model inference scheme provided in this application, when the DCU executes the previous round of draft model inference task, allows the CPU to pre-allocate input memory for the main model through memory pre-allocation and directly generate input data containing placeholders, copying it to the pre-allocated input memory. This eliminates the need to wait for the draft model's inference results, avoiding computational interruptions caused by waiting and thus quickly completing the main model input preparation. The DCU, instead of waiting for the CPU's main model input preparation, uses an asynchronous copy mechanism to directly copy the previous round's inference results to the pre-allocated input memory after the previous round of draft model inference task is completed, allowing it to directly enter the main model inference stage. Through these techniques, the collaboration process between the CPU and DCU is decoupled, resolving the serial dependency problem of CPU and DCU computational tasks in related technologies, improving the utilization of computing resources, and optimizing the throughput and real-time performance of the inference task.

[0096] Figure 2 A system architecture diagram of a large language model reasoning method provided in this application is shown below. Figure 2 As shown, the system architecture includes an electronic device 20, and the electronic device 20 includes a CPU 201 and a DCU 202, which are connected together.

[0097] Understandably, the electronic device 20 integrates a CPU chip and a DCU chip. In the large language model inference scenario, the CPU 201 is used to prepare input data for the DCU 202 and issue inference tasks; the DCU 202 is used to execute the inference tasks based on the input data prepared by the CPU 201.

[0098] Optionally, the electronic device 20 is a server, specifically a server with a pre-trained large language model deployed on it.

[0099] This application applies to large language model inference scenarios on domestically developed computing platforms, especially in production environments with high concurrency, long context, or multi-turn dialogues. For example, in intelligent customer service systems, users need to handle hundreds of dialogue requests simultaneously, and each request needs to generate a natural language response in real time; in content generation scenarios, large-scale text generation and verification need to be completed within seconds. When deploying large language model inference services on DCUs, it is necessary to balance hardware characteristics (such as memory bandwidth limitations) and algorithm optimization (such as MTP parallel decoding) to achieve efficient utilization of computing resources.

[0100] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0101] This application provides a large language model reasoning method, which is applied to CPUs, for example... Figure 2 CPU201 is shown. Figure 3 A flowchart illustrating a large language model reasoning method provided in this application is shown below. Figure 3 As shown, the method includes:

[0102] S301. When the DCU executes the previous round of draft model inference task, pre-allocate the input memory corresponding to the main model for the current round of request batches.

[0103] The large language model consists of a main model and a draft model. The draft model receives the input sequence, performs forward propagation computation, and generates a sequence of candidate tokens. The main model performs forward propagation computation based on the input sequence and the candidate token sequence generated by the draft model to verify the candidate token sequence, thereby determining the subset of accepted tokens. It also resamples rejected positions to generate corrected tokens, outputting the final result. The main model has higher forward propagation accuracy, while the draft model has lower forward propagation latency. During forward propagation computation, both the main model and the draft model perform parallel computation for each request in the request batch, thus improving inference efficiency.

[0104] In practical applications, a large language model may include one or more draft models, each of which performs inference independently. This application does not limit the number of draft models, and for ease of description, it uses one draft model as an example. When there are multiple draft models, the implementation of each draft model is similar, and this application will not elaborate further.

[0105] The draft model inference task is a task that uses a draft model for forward propagation computation, and this task is executed by the DCU.

[0106] In this application, while the DCU is still executing the previous round of draft model inference task, and the draft model inference task has not yet been completed, the CPU enters the main model input preparation phase ahead of schedule. During the main model input preparation phase, the CPU merges multiple requests from the currently active request list into a single request batch and pre-allocates the input memory corresponding to the main model for this request batch. This input memory belongs to the DCU's video memory and is used by the DCU to store the inference results of the previous round of draft model. Therefore, the CPU's current memory allocation operation is a pre-allocation operation.

[0107] In this context, a single request represents an active user session. In large-scale services, hundreds or even thousands of users typically initiate conversations simultaneously. Instead of launching a separate GPU computation for each user (which would be extremely inefficient), the system collects all currently generated user sessions and packages them into a single batch.

[0108] The CPU pre-allocates a contiguous block of video memory from the DCU and maintains local video memory pool metadata to record the logical state of each memory block in that memory space, such as whether it is free or occupied. During the main model input preparation phase, the CPU selects an existing, free video memory address range from the video memory pool metadata, marks its logical state as occupied, and records the pointer value of this video memory address range. This video memory is the input memory pre-allocated for the main model.

[0109] S302. Generate the input data for the main model in this round and copy the input data to the corresponding input memory of the main model; the input data uses placeholders to represent the inference results of the draft model in the previous round.

[0110] The placeholder is a virtual marker used to reserve memory space. It does not carry actual semantics and only indicates that the token exists.

[0111] Related technologies such as Figure 1 The inference process shown occurs after the DCU completes the previous round of draft model inference and copies the inference results to the CPU's memory. Only then does the CPU enter the main model input preparation stage. During this stage, the CPU generates the main model input data based on the draft model's inference results and sends the main model inference task to the DCU. It is evident that this technology has a technical problem: while the DCU is performing draft model inference, the CPU can only wait and has no other actions; this idle state affects inference performance.

[0112] To address this technical problem, the CPU in this application enters the main model input preparation stage while the DCU is executing the previous round of draft model inference task. During this stage, it generates the main model input data and sends the main model inference task to the DCU. To achieve this, this application introduces a placeholder mechanism. As mentioned above, during the main model input preparation stage, the DCU has not yet completed the previous round of draft model inference task, causing the CPU to be unable to read the inference results from the previous round of draft model. The CPU in this application uses placeholders instead of the actual draft model inference results to generate the main model's input data for this round and copies it to the pre-allocated input memory for the main model. By reserving input memory in advance using placeholders, this application eliminates the need to wait for the draft model inference results during the main model input preparation stage, breaking the waiting dependency between the CPU and the DCU and reducing CPU idle time.

[0113] S303. Send the first copy instruction and the main model inference task to the DCU; the first copy instruction is used to instruct the DCU to copy the inference results of the draft model in the previous round to the input memory to overwrite the placeholder.

[0114] The main model inference task involves forward propagation computation using the main model, and this task is executed by the DCU. Input memory refers to the input memory pre-allocated for the main model. The input memory contains the input data of the main model, and placeholders within this input data represent the inference results of the draft model in the previous round.

[0115] The first copy instruction carries a pointer range corresponding to the input memory pre-allocated for the main model, allowing the DCU to distinguish between them. By issuing the first copy instruction to the DCU, it is instructed to copy the inference results to the pre-allocated input memory after completing the previous round of draft model inference task, thus achieving asynchronous copying on the DCU. In this way, when the DCU executes the main model inference task, it can directly read the inference results of the previous round of draft model stored in this input memory, without waiting for the CPU to prepare the main model input, thereby improving inference efficiency.

[0116] Optionally, the first copy instruction is issued by calling a device-side function. This device-side function is a kernel function, which is a piece of compiled kernel code loaded into the DCU's video memory. For example, the HIP function; HIP (Heterogeneous-computing Interface for Portability) is a DCU-related programming language.

[0117] S304. After obtaining the reasoning result of the main model in this round, check the reasoning result of the main model in this round, and perform the reasoning in the next round based on the check result.

[0118] Optionally, after completing the main model inference task, the DCU obtains the inference result of the main model in this round and copies the inference result to the CPU's memory, so that the CPU can read the inference result from the memory.

[0119] The CPU checks whether the inference result of the main model in this round meets the conditions. If it does, the inference in this round is considered complete. For example, the conditions are that the inference result of the main model in this round reaches a set length and has an end of sequence (EOS).

[0120] This application addresses the serial dependency problem between CPU and DCU computing tasks in related technologies through an asynchronous copy mechanism. Specifically, by pre-allocating memory locations, the preparation of the main model input does not require waiting for the inference results of the draft model. These memory locations can be reserved in advance and directly used for the main model's inference task, avoiding computational interruptions due to waiting. The asynchronous copy mechanism further directly transfers the inference results of the draft model to the pre-allocated memory locations. Through these technical means, the collaboration process between the CPU and DCU is decoupled, improving the utilization of computing resources and optimizing the throughput and real-time performance of the inference task.

[0121] This application also provides another large language model reasoning method, applied to DCU, for example... Figure 2 The DCU202 shown. Figure 4 A flowchart illustrating another large language model reasoning method provided in this application is shown below. Figure 4 As shown, the method includes:

[0122] S401. During the execution of the previous round of draft model inference task, receive the first copy instruction and main model inference task sent by the CPU.

[0123] In this application, while the DCU is executing the previous round of draft model inference task, the CPU simultaneously completes the main model input preparation and issues the first copy instruction and the main model inference task to the DCU, enabling the DCU to receive the main model inference task in advance. There is no need to wait for the CPU to prepare the main model input after completing the draft model inference task.

[0124] S402. After completing the previous round of draft model reasoning task, according to the first copy instruction, copy the reasoning result of the draft model in the previous round to the input memory pre-allocated to the main model, so as to overwrite the placeholders in the input data in the input memory and obtain the updated input data.

[0125] The original input data written to the input memory corresponding to the main model contains placeholders. The placeholders are used to represent the inference results of the draft model. The DCU copies the actual inference results of the draft model to the input memory through an asynchronous copy operation, thereby overwriting the placeholders and obtaining updated input data for subsequent main model inference.

[0126] S403. Execute the main model inference task based on the updated input data.

[0127] The asynchronous copy mechanism allows the inference results of the draft model to be directly reused on the DCU, which helps to avoid the serial dependency between the CPU and the DCU.

[0128] This application addresses the serial dependency problem between CPU and DCU computing tasks in related technologies through an asynchronous copy mechanism. Specifically, by pre-allocating memory locations, the preparation of the main model input does not require waiting for the inference results of the draft model. These memory locations can be reserved in advance and directly used for the main model's inference task, avoiding computational interruptions due to waiting. The asynchronous copy mechanism further directly transfers the inference results of the draft model to the pre-allocated memory locations. Through these technical means, the collaboration process between the CPU and DCU is decoupled, improving the utilization of computing resources and optimizing the throughput and real-time performance of the inference task.

[0129] In this application, a parallel pipelined operation of the CPU and DCU is achieved through a "pre-allocated memory and placeholder" mechanism combined with asynchronous copying at the DCU end. The CPU can prepare the input data for the main model and issue tasks in advance without waiting for the DCU to complete the previous round of draft calculations; after completing the previous round of draft calculations, the DCU does not need to wait for the CPU to prepare the main model input, but directly fills the inference results of the draft model into the pre-allocated input memory according to the first copy instruction. This mechanism helps to eliminate the serial waiting delay caused by data dependency between the CPU and DCU, and improves the overall throughput of multi-round inference.

[0130] Based on the above embodiments, Figure 5 A schematic diagram of the interaction flow of a large language model reasoning method provided in this application is shown below. Figure 5 As shown, the method includes:

[0131] S501. When the DCU is executing the previous round of draft model inference task, the CPU pre-allocates the input memory corresponding to the main model for the current round of request batches.

[0132] The implementation method of this step is the same as that of step S301, and will not be repeated here.

[0133] At this point, the CPU has not yet obtained the inference results from the previous draft model, so it needs to predict the required number of placeholders in order to pre-allocate input memory with an appropriate address range. Optionally, the CPU dynamically adjusts the reserved number of placeholders by monitoring the DCU memory usage and the average generation length of the request sequence. Specifically, during each main model input preparation phase, the CPU calculates the optimal number of placeholders based on the request features of the current batch and pre-allocates them accordingly. Request features include sequence length, generation rate, and other characteristic information.

[0134] By dynamically adjusting the number of token placeholders based on real-time request load, it is beneficial to avoid resource waste caused by excessive memory reservation, while ensuring memory requirements during long sequence generation and reducing performance bottlenecks caused by insufficient memory. For example, in high-concurrency scenarios, if a large fluctuation in the generation length of the token sequence is detected, the CPU can allocate more placeholders to adapt to the long sequence generation requirements; in low-load scenarios, the number of placeholders can be reduced to save video memory resources.

[0135] S502: The CPU generates the input data for the main model in this round and copies the input data to the corresponding input memory of the main model.

[0136] The implementation method of this step is the same as that of step S302, and will not be repeated here.

[0137] S503, the CPU sends the first copy instruction and the main model inference task to the DCU.

[0138] The implementation method of this step is the same as that of step S303, and will not be repeated here.

[0139] Optionally, the implementation of sending the first copy instruction and the main model inference task to the DCU includes: simultaneously sending the first copy instruction and the main model inference task to the DCU; or, sending the first copy instruction to the DCU and then sending the main model inference task to the DCU after the sending is completed.

[0140] This implementation supports two modes: "simultaneous issuance of instructions and tasks" and "issuing instructions first, then tasks." This allows technicians to flexibly choose the CPU-side instruction sending strategy based on specific application scenarios, hardware characteristics, or debugging needs. This design provides configurability for task scheduling timing, facilitating targeted performance tuning or compatibility adaptation in different system environments, thus improving the method's versatility and ease of use.

[0141] Accordingly, the DCU receives the first copy instruction and the main model inference task sent by the CPU.

[0142] It should be noted that after step S503, the CPU executes the following steps S507-S509. The DCU executes the following steps S504-S506. The CPU and DCU operate in parallel, so there is no execution order dependency between steps S507-S509 and steps S504-S506.

[0143] After completing the previous round of draft model inference task, S504 and DCU copy the inference results of the previous round of draft model to the input memory pre-allocated to the main model according to the first copy instruction, so as to overwrite the placeholders in the input data in the input memory and obtain the updated input data.

[0144] The first copy instruction has an additional dependency: the copy operation can only begin after the previous draft model inference task has completed. The main model inference task needs to be sent to the DCU after or simultaneously with the first copy instruction; otherwise, the pipeline will be interrupted. The main model inference task remains blocked, waiting for the copy operation to complete, until the draft model inference task is finished.

[0145] Optionally, the implementation of copying the inference results of the draft model in the previous round to the input memory pre-allocated to the main model according to the first copy instruction includes: according to the request ID corresponding to each request in the first copy instruction, copying the inference results of the draft model in the previous round corresponding to the request ID to the memory location in the input memory corresponding to the request ID.

[0146] In multi-user concurrent scenarios, a precise correspondence is established between the draft model inference results and the main model input memory location through request identifiers. This ensures that the main model input data of each session is not interfered with and is accurately positioned in the DCU memory. This implementation method guarantees the correctness of the mechanism under high concurrency and helps to avoid inference errors caused by memory misalignment.

[0147] The first copy instruction carries the request identifier and target memory location corresponding to each request in the current round's request batch. The target memory location is used to store the input data of the corresponding request. When the DCU performs the copy, based on the request identifier, it directly and asynchronously copies the corresponding inference result generated in the previous round of the draft model to the target memory location in the input memory that matches the request identifier, so as to complete the in-situ update of the data.

[0148] The copy operation compares the request IDs of the previous scheduling batch with the request IDs of the current scheduling result, asynchronously copying the inference results of the draft model directly on the DCU to the memory location reserved for the main model. Specifically, the DCU needs to compare the request IDs in the completed draft model inference task of the previous round (i.e., the "previous scheduling batch") with the request IDs that the main model needs to process in this round (i.e., the "requests of the current scheduling result"). By comparing, it confirms which request data is ready and needs to be copied, and then performs asynchronous copying on the successfully matched request IDs. This ensures that only valid and corresponding data is copied to the input memory of the main model, which helps to avoid data corruption or invalid copying.

[0149] S505 and DCU execute the main model inference task based on the updated input data.

[0150] The inference result of the previous draft model includes the candidate token sequence corresponding to each request in the previous request batch. The main model inference specifically verifies each token in each candidate token sequence and outputs the verification result. For the first rejected position (or all positions, depending on the specific algorithm implementation), the main model also resamples and generates a new token based on its own probability distribution. Therefore, the inference result of the main model includes the final token sequence corresponding to each request in the current request batch and the verification result of the candidate token sequences.

[0151] The final token sequence specifically includes accepted candidate tokens and tokens newly sampled by the main model. The verification result of the candidate token sequence includes the accept or reject flag corresponding to each token in the candidate token sequence.

[0152] After completing the main model inference task, S506 and DCU copy the inference results of the main model in this round and the inference results of the draft model in the previous round to the CPU memory.

[0153] Although the inference results of the previous draft model were directly copied to the input memory pre-allocated for the main model on the DCU, the inference results of the draft model still need to be copied to the CPU. However, they do not need to be copied back in real time. Instead, after the main model inference task of this round is completed, the inference results of the main model and the inference results of the previous draft model are returned to the CPU together so that they can be passed to the scheduling layer. This is because the scheduling layer may postpone the processing of some requests due to the insertion of prefill operations for other requests or insufficient key-value cache. In this way, the inference results of the draft model copied back to the CPU will be directly placed into the input memory of the main model before the next inference of the postponed request through the original scheduling logic, which helps to avoid inference errors.

[0154] The prefill operation refers to the initial stage of the model processing the user's complete input prompt.

[0155] After obtaining the inference results of the main model, the CPU executes the following steps S513.

[0156] Each round of inference includes main model inference and draft model inference. The draft model inference process is described below, specifically including the draft model input preparation stage of the CPU and the draft model inference stage of the DCU.

[0157] S507 and CPU are the input memory pre-allocated for the draft model in this round of requests.

[0158] During the draft model input preparation stage, the CPU selects an existing, free range of video memory addresses from the video memory pool metadata, marks its logical state as occupied, and records the pointer value of this video memory address range. This video memory is the input memory pre-allocated for the draft model.

[0159] S508: The CPU generates the input data for the draft model in this round and copies the input data to the corresponding input memory of the draft model; the input data uses placeholders to represent the inference results of the main model in this round.

[0160] The placeholder is a virtual marker used to reserve memory space. It does not carry actual semantics and only indicates that the token exists.

[0161] Related technologies such as Figure 1 The inference process shown involves the DCU completing the main model inference task for this round and copying the inference results to the CPU's memory. Only then does the CPU enter the draft model input preparation stage. During this stage, the CPU generates the input data for the draft model based on the main model's inference results and sends the draft model inference task for this round to the DCU. It is evident that this technology has a technical problem: while the DCU is performing main model inference, the CPU can only wait and has no other actions. This idle state affects inference performance.

[0162] To address this technical problem, the CPU in this application enters the draft model input preparation stage while the DCU is executing the main model inference task for the current round. During this stage, it generates the input data for the draft model and sends the draft model inference task to the DCU. To achieve this, this application introduces a placeholder mechanism. As mentioned above, during the draft model input preparation stage, the DCU has not yet completed the main model inference task for the current round, causing the CPU to be unable to read the inference result of the main model. The CPU in this application uses placeholders instead of the actual main model inference result to generate the input data for the draft model for the current round and copies it to the input memory pre-allocated for the draft model. By reserving input memory in advance using placeholders, this application eliminates the need to wait for the main model inference result during the draft model input preparation stage, breaking the waiting dependency between the CPU and the DCU and reducing CPU idle time.

[0163] S509, the CPU sends the second copy instruction and the draft model inference task to the DCU.

[0164] The second copy instruction is used to instruct the DCU to copy the inference results of the main model in this round to the input memory corresponding to the draft model, so as to overwrite the placeholder.

[0165] Optionally, the second copy instruction is issued by calling a device-side function.

[0166] The draft model inference task involves performing forward propagation computations using a draft model, and this task is executed by the DCU. Input memory refers to the input memory pre-allocated for the draft model. The input memory contains the input data of the draft model, and placeholders within this input data represent the inference results of the main model.

[0167] Accordingly, the DCU receives the second copy instruction and draft model inference task sent by the CPU, and performs the following steps S510-S512.

[0168] After completing the main model inference task for this round, S510 and DCU read the inference results of the main model for this round.

[0169] S511 and DCU copy the inference results of the main model in this round to the input memory pre-allocated for the draft model according to the second copy instruction, so as to overwrite the placeholders in the input data in the input memory and obtain the updated input data.

[0170] The second copy instruction has an additional dependency: the copy operation can only begin after the main model inference task of this round has been completed. The draft model inference task needs to be sent to the DCU after or simultaneously with the second copy instruction; otherwise, the pipeline will be interrupted. The draft model inference task remains blocked, waiting for the copy operation to complete, until the main model inference task is finished.

[0171] The original input data written to the input memory corresponding to the draft model contains placeholders. The placeholders are used to represent the inference results of the main model. The DCU copies the actual inference results of the main model to this input memory through an asynchronous copy operation, thereby overwriting the placeholders and obtaining updated input data for subsequent inference of the draft model.

[0172] Once the main model inference computation is complete, the inference results are temporarily stored in the DCU's on-chip cache or a specific output buffer. At this point, the DCU's control logic automatically triggers and calls a pre-loaded device-side function. This function, a lightweight program running inside the DCU, directly reads the inference results generated by the main model and, based on the memory address mapping information carried in the second copy instruction, writes the read data directly into the input memory pre-allocated for the draft model. This write operation precisely overwrites the previously reserved placeholders, thus completing the in-situ data update from "main model output" to "draft model input" without CPU intervention or consuming host bus bandwidth. This ensures the draft model's input data is immediately complete, preparing the DCU for subsequent draft model forward propagation computations.

[0173] The memory address mapping information includes the request identifier and target memory location for each request in the current request batch. The target memory location is used to store the input data for the corresponding request. When the DCU performs the copy, it asynchronously copies the corresponding inference result generated by the main model in this round to the target memory location in the input memory that matches the request identifier, based on the request identifier, to complete the in-situ update of the data.

[0174] S512 and DCU execute the draft model inference task based on the updated input data.

[0175] The asynchronous copy mechanism allows the inference results of the main model to be directly reused on the DCU, which helps to avoid the serial dependency between the CPU and the DCU.

[0176] S513. The CPU checks the inference results of the main model in this round.

[0177] The CPU reads the inference results of the main model in this round from memory, and the CPU's memory stores the inference results of the main model in this round copied by the DCU and the inference results of the draft model in the previous round.

[0178] In this application, after the main model inference is completed, the DCU packages the inference result of the main model and the original draft data (i.e., the inference result of the draft model in the previous round) and sends them back to the CPU memory. In this way, the CPU can check the inference result of the main model in the current round while the DCU is executing the draft model inference task. Compared with related technologies, where the CPU has to wait for the DCU to complete the draft model inference task before it can obtain and check the inference result of the main model in the current round, this application reduces the CPU idle waiting time.

[0179] The inference results of the main model include the final token sequence and the verification results of candidate token sequences for each request in the current request batch. The check verifies whether the final token sequence has reached the set length and whether it contains an end-of-strike symbol. If the check result for any request indicates that the final token sequence has reached the set length and contains an end-of-strike symbol, it means that the request has completed inference and can be removed from the request batch. If the check result for any request indicates that the final token sequence has not reached the set length and / or does not contain an end-of-strike symbol, it means that the request has not yet completed inference and needs to participate in the next round of inference; therefore, the request is retained in the request batch.

[0180] After step S513, the CPU performs the next round of reasoning based on the inspection results.

[0181] The CPU updates the request batch based on the check results corresponding to each request in the current request batch, and then performs the next round of inference based on the updated request batch, that is, enters the next round of main model input preparation stage. The specific implementation of the next round of inference is the same as that of the current round of inference, as detailed in the above embodiments, and will not be repeated here.

[0182] It should be noted that the reasoning process of a large language model usually involves multiple rounds of reasoning. This application only illustrates one round of reasoning as an example. In practical applications, the specific implementation of each round of reasoning is similar, and this application will not elaborate on it further.

[0183] Figure 6 A schematic diagram of an MTP inference process provided in this application is shown below. Figure 6 As shown, after the CPU completes the main model input preparation phase, it enters the draft model input preparation phase. The DCU executes the main model inference task issued by the CPU, and after completing the task, copies the inference results of the main model in this round and the inference results of the draft model in the previous round to the CPU's memory, so that the CPU can check the inference results.

[0184] It is evident that although the CPU still has waiting operations, the overall waiting time in each round of inference is shorter. The inference tasks on the DCU are dispatched in advance, eliminating waiting and improving overall computing performance.

[0185] This application improves MTP inference performance and streamlines the DCU computation process by adjusting and optimizing the scheduling logic in related technologies, reducing the interdependence and waiting between CPU logic and DCU inference. After system development and verification, the overall inference performance of MTP is improved by 5% compared to solutions in related technologies.

[0186] This application innovatively uses token placeholders in model input preparation instead of actual token values. This breaks the CPU's dependence on DCU inference results, advances the scheduling process, and allows for the early delivery of inference tasks. Since token placeholders are used for input preparation, precise token copying based on the request identifier according to the request sequence is performed before the inference task begins, ensuring the correctness of the inference. This task can also be delivered asynchronously without affecting system performance. By parallel masking the CPU scheduling task and the inference process of the draft model computed on the DCU, the overall system performance is improved. Similarly, parallel masking of the main model's input preparation process and the draft model's inference process computed on the DCU further improves the overall system performance.

[0187] This application provides a large language model reasoning device, which is integrated into a CPU, for example... Figure 2The CPU201 shown. Optionally, this large language model inference device is provided as a first large language model inference device. Figure 7 A schematic diagram of the structure of a first-class language model reasoning device provided in this application is shown below. Figure 7 As shown, the first large language model inference device 70 provided in this embodiment includes:

[0188] The pre-allocation module 701 is used to pre-allocate the input memory corresponding to the main model for the current request batch when the DCU is executing the previous round of draft model inference task;

[0189] The generation module 702 is used to generate the input data for the current round of the main model and copy the input data to the corresponding input memory of the main model; the input data uses placeholders to represent the inference results of the draft model in the previous round;

[0190] The sending module 703 is used to send a first copy instruction and a main model inference task to the DCU; the first copy instruction is used to instruct the DCU to copy the inference results of the draft model in the previous round to the input memory to overwrite the placeholder;

[0191] The inspection module 704 is used to inspect the inference result of the main model in the current round after obtaining the inference result of the main model in the current round, and to perform the inference in the next round based on the inspection result.

[0192] In one alternative implementation, the pre-allocation module 701 is further configured to pre-allocate input memory corresponding to the draft model for the request batch of this round;

[0193] The generation module 702 is also used to generate the input data for the current round of the draft model and copy the input data to the corresponding input memory of the draft model; the input data uses placeholders to represent the reasoning results of the main model in this round;

[0194] The sending module 703 is also used to send a second copy instruction and a draft model inference task to the DCU; the second copy instruction is used to instruct the DCU to copy the inference result of the main model in this round to the input memory corresponding to the draft model to overwrite the placeholder.

[0195] In one alternative implementation, before checking the inference results of the main model in the current round, the checking module 704 is further configured to: read the inference results of the main model in the current round from memory; the memory stores the inference results of the main model in the current round copied by the DCU and the inference results of the draft model in the previous round.

[0196] In one optional implementation, the sending module 703, when sending the first copy instruction and the master model inference task to the DCU, is specifically used for:

[0197] Simultaneously, the first copy instruction and the main model inference task are sent to the DCU;

[0198] Alternatively, send the first copy instruction to the DCU, and then send the main model inference task to the DCU after the copy instruction is sent.

[0199] The first large language model inference device 70 provided in this embodiment can execute the method provided in the corresponding method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0200] This application provides another large language model inference device, which is integrated into the DCU, for example... Figure 2 The DCU202 is shown. Optionally, this large language model inference device is provided as a second large language model inference device. Figure 8 A schematic diagram of the structure of a second-largest language model reasoning device provided in this application is shown below. Figure 8 As shown, the second language model inference device 80 provided in this embodiment includes:

[0201] The receiving module 801 is used to receive the first copy instruction and the main model inference task sent by the central processing unit CPU during the execution of the previous round of draft model inference task.

[0202] The copy module 802 is used to copy the inference results of the previous round of the draft model to the input memory pre-allocated by the main model according to the first copy instruction after completing the inference task of the previous round of the draft model, so as to overwrite the placeholders in the input data in the input memory and obtain the updated input data.

[0203] Execution module 803 is used to perform the main model inference task based on the updated input data.

[0204] In one optional implementation, the copy module 802, when copying the inference results of the draft model from the previous round to the input memory pre-allocated for the main model according to the first copy instruction, is specifically used for:

[0205] Based on the request identifiers corresponding to each request in the first copy instruction, the inference results of the previous round of the draft model corresponding to the request identifier are copied to the memory location in the input memory corresponding to the request identifier.

[0206] In one alternative implementation, the receiving module 801 is further configured to receive the second copy instruction and the draft model inference task sent by the CPU;

[0207] The copy module 802 is also used to copy the inference results of the main model in this round to the input memory pre-allocated for the draft model according to the second copy instruction after completing the main model inference task in this round, so as to overwrite the placeholders in the input data in the input memory and obtain the updated input data.

[0208] The execution module 803 is also used to perform the draft model inference task based on the updated input data.

[0209] In one optional implementation, the copy module 802 is further configured to copy the reasoning results of the main model in this round and the reasoning results of the draft model in the previous round to the CPU memory after completing the main model reasoning task.

[0210] The second language model inference device 80 provided in this embodiment can execute the method provided in the corresponding method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0211] This application also provides an electronic device, Figure 9 A schematic diagram of the structure of an electronic device provided in this application is shown below. Figure 9 As shown, the electronic device 20 provided in this embodiment includes: CPU 201, DCU 202, and memory 203 which is communicatively connected to CPU 201 and DCU 202 respectively.

[0212] Optionally, CPU201 and DCU202 are connected to memory 203 via a bus.

[0213] In the specific implementation process, the memory 203 stores computer execution instructions, including instructions corresponding to CPU 201 and instructions corresponding to DCU 202. Accordingly, CPU 201 executes the computer execution instructions stored in memory 203 to execute the above-mentioned large language model inference method, and DCU 202 executes the computer execution instructions stored in memory 203 to execute the above-mentioned large language model inference method.

[0214] The specific implementation process of CPU201 and DCU202 can be found in the above method embodiment. Their implementation principles and technical effects are similar, and will not be repeated here.

[0215] In the above embodiments, it should be understood that CPU 201 can also be replaced by other output units, such as other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor, or CPU 201 can be any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules within the processor.

[0216] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0217] Buses can be Industry Standard Architecture (ISA) buses, Peripheral Component Interconnect (PCI) buses, or Extended Industry Standard Architecture (EISA) buses, etc. Buses can be categorized into address buses, data buses, control buses, etc.

[0218] This application also provides a computer program product, including a computer program that, when executed, implements the above-described method.

[0219] This application also provides a computer-readable storage medium storing computer-executable instructions that, when executed, implement the above-described method.

[0220] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as read-only memory (ROM), static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0221] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0222] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0223] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0224] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the steps of the methods of the various embodiments of the present invention.

[0225] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A reasoning method for large language models, characterized in that, Applied to a central processing unit (CPU), the large language model includes a main model and a draft model, and the method includes: When the Deep Computing Unit (DCU) executes the previous round of draft model inference task, it pre-allocates the input memory corresponding to the main model for the current round of request batches. Generate the input data for the current round of the main model and copy the input data to the corresponding input memory of the main model; the input data uses placeholders to represent the inference results of the draft model in the previous round; Send a first copy instruction and a main model inference task to the DCU; the first copy instruction carries a pointer range corresponding to the input memory pre-allocated to the main model, which is used to instruct the DCU to copy the inference results of the draft model in the previous round to the memory location corresponding to the pointer range, so as to overwrite the placeholder; Having obtained the reasoning result of the main model in this round, we check the reasoning result of the main model in this round and proceed with the next round of reasoning based on the check result.

2. The method according to claim 1, characterized in that, After sending the first copy instruction and the master model inference task to the DCU, the method further includes: Pre-allocate input memory for the draft model for this round of requests; The input data for the current round of the draft model is generated and copied into the input memory corresponding to the draft model; the input data uses placeholders to represent the inference results of the main model in this round. Send a second copy instruction and a draft model inference task to the DCU; the second copy instruction is used to instruct the DCU to copy the inference result of the main model in this round to the input memory corresponding to the draft model, so as to overwrite the placeholder.

3. The method according to claim 1, characterized in that, Before checking the inference results of the main model in this round, the method further includes: The inference results of the main model in the current round are read from memory; the memory stores the inference results of the main model in the current round (copied from the DCU) and the inference results of the draft model in the previous round.

4. The method according to claim 1, characterized in that, Sending the first copy instruction and the main model inference task to the DCU includes: Simultaneously, the first copy instruction and the main model inference task are sent to the DCU; Alternatively, send the first copy instruction to the DCU, and then send the main model inference task to the DCU after the copy instruction is sent.

5. A large language model reasoning method, characterized in that, Applied to a deep computing unit (DCU), the large language model includes a master model and a draft model. The method includes: During the execution of the previous round of draft model inference task, the first copy instruction and main model inference task are received from the central processing unit (CPU). After completing the previous round of draft model inference task, according to the first copy instruction, the inference result of the draft model in the previous round is copied to the input memory pre-allocated by the main model to overwrite the placeholders in the input data in the input memory, so as to obtain the updated input data. Based on the updated input data, the main model inference task is executed.

6. The method according to claim 5, characterized in that, The step of copying the inference results of the draft model from the previous round to the input memory pre-allocated for the main model according to the first copy instruction includes: Based on the request identifiers corresponding to each request in the first copy instruction, the inference results of the previous round of the draft model corresponding to the request identifier are copied to the memory location in the input memory corresponding to the request identifier.

7. The method according to claim 5, characterized in that, The method further includes: Receive the second copy instruction and draft model inference task sent by the CPU; After completing the main model inference task for this round, the inference result of the main model for this round is copied to the input memory pre-allocated for the draft model according to the second copy instruction, so as to overwrite the placeholders in the input data in the input memory and obtain the updated input data. Based on the updated input data, the draft model inference task is executed.

8. The method according to claim 7, characterized in that, The method further includes: After completing the main model inference task, the inference results of the main model in this round and the inference results of the draft model in the previous round are copied into the CPU memory.

9. A large language model reasoning device, characterized in that, Integrated into the CPU, the large language model includes a main model and a draft model, and the device includes: The pre-allocation module is used to pre-allocate the input memory corresponding to the main model for the current request batch when the DCU is executing the previous round of draft model inference task; The generation module is used to generate the input data for the main model in this round and copy the input data to the input memory corresponding to the main model; the input data uses placeholders to represent the inference results of the draft model in the previous round; The sending module is used to send a first copy instruction and a main model inference task to the DCU; the first copy instruction carries a pointer range corresponding to the input memory pre-allocated by the main model, which is used to instruct the DCU to copy the inference result of the draft model in the previous round to the memory location corresponding to the pointer range, so as to overwrite the placeholder; The inspection module is used to inspect the inference results of the main model in the current round after obtaining the inference results of the main model in the current round, and to perform the inference in the next round based on the inspection results.

10. A large language model reasoning device, characterized in that, Integrated into the DCU, the large language model includes a main model and a draft model, and the device includes: The receiving module is used to receive the first copy instruction and the main model inference task sent by the central processing unit (CPU) during the execution of the previous round of draft model inference task. The copy module is used to copy the inference results of the previous round of the draft model to the input memory pre-allocated by the main model according to the first copy instruction after completing the previous round of the draft model inference task, so as to overwrite the placeholders in the input data in the input memory and obtain the updated input data. The execution module is used to execute the main model inference task based on the updated input data.

11. An electronic device, characterized in that, include: CPU, DCU, and memory that are communicatively connected to the CPU and DCU respectively; The memory stores computer-executed instructions; The CPU is used to execute computer execution instructions stored in the memory to implement the method as described in any one of claims 1-4; The DCU is used to execute computer execution instructions stored in the memory to implement the method as described in any one of claims 5-8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed, are used to implement the method as described in any one of claims 1-8.

13. A computer program product, characterized in that, Includes a computer program, which, when executed, is used to implement the method described in any one of claims 1-8.

Images