A high-concurrency reasoning method and system for large language models

Abstract

The present invention relates to the field of artificial intelligence technology, and discloses a high-concurrency reasoning method and system for a large language model. The method comprises the following steps: utilizing an executor to calculate the size of a memory block and allocate memory space; utilizing a scheduler to convert a request sequence and place it into a waiting queue of the scheduler; the scheduler allocates a corresponding memory block to each request sequence until each request sequence is capable of performing the next reasoning; the scheduler calculates the memory requirements of the request sequences in the waiting queue according to priority order, and transfers the request sequences in the waiting queue to a running queue; and allocating the number of memory blocks for executing pre-filled reasoning or for executing decoding reasoning based on the number of pre-filled types and the number of decoded types of the request sequence. Therefore, the present invention adopts continuous batch processing, a dynamic space allocation mechanism, and a task scheduling framework, fully utilizing the parallel reasoning capability of continuous batch processing, improving the concurrency and throughput of large model reasoning, and overcoming the limitation of traditional continuous batch processing requiring pre-allocation of space.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large language model high-concurrency reasoning method and system. Background Art

[0002] Artificial intelligence technology is a hot topic in computer science, and large language models are a key research direction in this field. As emerging large language models, they are unique in that they contain far more parameters than traditional models, typically in the hundreds of billions. This leads to higher training and inference costs. For a model with tens of billions of parameters, such as the Qwen-14B, the inference speed is approximately 25-35 tokens / second with a batch size of 1 and two 3090 GPUs. Conventional processing solutions (batch size 1, serial mode) make it difficult to support multiple users using the model inference service simultaneously.

[0003] For most traditional models (such as image processing models), simply increasing the batch size, provided sufficient video memory is available, can improve the concurrency of model inference and optimize the user experience. Unfortunately, this approach cannot be easily transferred to models based on the transformer architecture. The main reason for this is that the output of the generative model of the transformer architecture is not fixed for any input. Therefore, simply increasing the batch size usually requires allocating enough memory for each input to accommodate the maximum-length output, and waiting for the longest sample in the batch to be output before returning it all at once. This not only wastes a large amount of video memory, but sometimes waiting for a simple answer can take a long time. To address these issues, more efficient space allocation and batch scheduling methods are needed.

[0004] Therefore, how to improve resource utilization and inference request throughput, reduce inference time, and enhance the capabilities and effectiveness of large model-related services and applications during the large language model inference process is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The present invention provides a large language model high-concurrency reasoning method and system, aiming to solve at least one of the above technical problems.

[0006] To achieve the above objectives, the present invention provides a high-concurrency reasoning method for a large language model, comprising the following steps:

[0007] S1: System initialization: The executor calculates the memory block size by simulating peak data, allocates memory space based on the calculation result, and synchronizes memory resource information to the scheduler;

[0008] S2: The scheduler receives the inference HTTP request, converts the inference HTTP request into a request sequence through preprocessing, and puts the converted request sequence into the scheduler's waiting queue;

[0009] S3: When the waiting queue and the running queue are not completely empty, the scheduler checks the memory usage of the current running queue to determine whether each request sequence can proceed to the next inference. If not, the scheduler allocates the corresponding memory block to each request sequence until each request sequence can proceed to the next inference.

[0010] S4: The scheduler checks the memory usage of the current run queue, calculates the memory requirements of the request sequences in the waiting queue according to the priority order, and transfers the request sequences in the waiting queue to the run queue based on the memory requirement calculation result;

[0011] S5: defining the request sequence transferred from the waiting queue to the running queue in step S4 as a pre-fill type, defining the request sequence in the running queue in step S3 as a decoding type, and allocating the number of video memory blocks for executing pre-fill inference or for executing decoding inference based on the number of pre-fill types and the number of decoding types in the request sequence;

[0012] S6: Call the executor to perform pre-fill inference or decoding inference based on the number of allocated memory blocks;

[0013] S7: The scheduler obtains the result returned by the executor, updates the status and information of the request sequence, and changes the type of the request sequence of the pre-filled type;

[0014] S8: Traverse all request sequences in the running queue and determine whether each request sequence meets the end condition. If so, remove the request sequence from the running queue, release the video memory, and respond to the http request.

[0015] Optionally, in step S1, the memory block size is configured as the number of tokens stored in the memory block.

[0016] Optionally, in step S1, allocating video memory space according to the calculation result further includes: if the video memory is insufficient, directly exiting the program, replacing the language model with a smaller one, or reducing the maximum length.

[0017] Optionally, in step S2, the request sequence includes metadata information of the http request and a token sequence.

[0018] Optionally, in step S3, it is determined whether each request sequence can be used for the next inference. If not, a corresponding video memory block is allocated to each request sequence until each request sequence can be used for the next inference. The process further includes:

[0019] Determine whether the video memory of the current run queue can support each request sequence for the next inference. If not, allocate a new video memory block to the run queue;

[0020] When the allocated new memory blocks are insufficient, the request sequences in the run queue are removed according to their priority and the corresponding memory blocks are released until each request sequence is able to proceed to the next inference.

[0021] Optionally, in step S4, the memory requirements of the request sequences in the waiting queue are calculated in order of priority, and the request sequences in the waiting queue are transferred to the running queue according to the memory requirement calculation result, which specifically includes:

[0022] Calculate the video memory requirements of the request sequences in the waiting queue according to the priority order; wherein the priority order is configured as the arrival time order of the HTTP requests corresponding to each request sequence;

[0023] If the video memory is sufficient, the requests in the waiting queue will be placed in the running queue until the video memory is insufficient or the threshold conditions of the maximum batch character length and the maximum batch number are reached.

[0024] Optionally, in step S5, the number of video memory blocks used to perform pre-fill inference or to perform decoding inference is allocated according to the number of pre-fill types and the number of decoding types of the request sequence, specifically including:

[0025] If the number of pre-fill type request sequences is greater than or equal to the number of decoding type request sequences, pre-fill inference is performed to allocate a corresponding number of memory blocks to each pre-fill type request sequence;

[0026] If the number of pre-fill type request sequences is less than the number of decoding type request sequences, the pre-fill type request sequences are put back into the waiting queue, decoding inference is performed, and the required video memory blocks are allocated to each decoding type request sequence.

[0027] Optionally, in step S6, pre-filling reasoning or decoding reasoning is performed, specifically including:

[0028] If pre-filled inference is performed, memory resources are allocated for all pre-filled request sequences, and only pre-filled request sequences are fed into the inference execution.

[0029] If decoding inference is performed, all pre-filled request sequences are put back into the waiting queue, and the next token of video memory is allocated for each decoded request sequence, and all request sequences are sent to inference.

[0030] Optionally, in step S7, the status and information of the request sequence are updated, and the type of the request sequence of the pre-filled type is changed, specifically including:

[0031] Update the status and information of the request sequence;

[0032] If performing pre-fill inference, change all pre-fill type request sequences to decode type.

[0033] In addition, in order to achieve the above-mentioned objectives, the present invention also provides a large language model high-concurrency reasoning system, including: an executor and a scheduler, and the executor and scheduler are configured to execute the large language model high-concurrency reasoning method as described in any one of the above.

[0034] The beneficial effects of the present invention are: proposing a high-concurrency reasoning method and system for a large language model, using an executor to calculate the size of the memory block and allocate memory space; using a scheduler to convert the request sequence and put it into the waiting queue of the scheduler; the scheduler allocates corresponding memory blocks to each request sequence until each request sequence can perform the next reasoning; the scheduler calculates the memory requirements of the request sequences in the waiting queue according to the priority order, and transfers the request sequences in the waiting queue to the running queue; according to the number of pre-filled types and the number of decoding types of the request sequence, the number of memory blocks for executing pre-filled reasoning or for executing decoding reasoning is allocated; thus, the present invention adopts continuous batch processing, dynamic space allocation mechanism and task scheduling framework, fully utilizes the parallel reasoning capability of continuous batch processing, improves the concurrency and throughput of large model reasoning, and solves the limitation of traditional continuous batch processing that requires pre-allocation of space. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flowchart of a high-concurrency reasoning method for a large language model according to an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of the principle of converting traditional batch processing into continuous batch processing according to an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of a situation where video memory is insufficient in an embodiment of the present invention;

[0038] Figure 4 Schematic diagram of the structure of a large language model high-concurrency reasoning system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0040] The embodiment of the present invention provides a large language model high concurrency reasoning method, referring to Figure 1 , Figure 1 Schematic diagram of the process of a large language model high-concurrency reasoning method according to an embodiment of the present invention.

[0041] In this embodiment, the large language model high-concurrency reasoning method includes the following steps:

[0042] S1: System initialization: The executor calculates the memory block size by simulating peak data, allocates memory space based on the calculation result, and synchronizes memory resource information to the scheduler;

[0043] S2: The scheduler receives the inference HTTP request, converts the inference HTTP request into a request sequence through preprocessing, and puts the converted request sequence into the scheduler's waiting queue;

[0044] S3: When the waiting queue and the running queue are not completely empty, the scheduler checks the memory usage of the current running queue to determine whether each request sequence can proceed to the next inference. If not, the scheduler allocates the corresponding memory block to each request sequence until each request sequence can proceed to the next inference.

[0045] S4: The scheduler checks the memory usage of the current run queue, calculates the memory requirements of the request sequences in the waiting queue according to the priority order, and transfers the request sequences in the waiting queue to the run queue based on the memory requirement calculation result;

[0046] S5: defining the request sequence transferred from the waiting queue to the running queue in step S4 as a pre-fill type, defining the request sequence in the running queue in step S3 as a decoding type, and allocating the number of video memory blocks for executing pre-fill inference or for executing decoding inference based on the number of pre-fill types and the number of decoding types in the request sequence;

[0047] S6: Call the executor to perform pre-fill inference or decoding inference based on the number of allocated memory blocks;

[0048] S7: The scheduler obtains the result returned by the executor, updates the status and information of the request sequence, and changes the type of the request sequence of the pre-filled type;

[0049] S8: Traverse all request sequences in the running queue and determine whether each request sequence meets the end condition. If so, remove the request sequence from the running queue, release the video memory, and respond to the http request.

[0050] In this embodiment, an executor is used to calculate the size of a memory block and allocate memory space; a scheduler is used to convert a request sequence and place it in the scheduler's waiting queue; the scheduler allocates a corresponding memory block to each request sequence until each request sequence is able to perform the next inference; the scheduler calculates the memory requirements of the request sequences in the waiting queue according to priority order and transfers the request sequences in the waiting queue to the running queue; and the number of memory blocks used to perform pre-filled inference or for performing decoded inference is allocated based on the number of pre-filled types and the number of decoded types in the request sequence. The present invention utilizes continuous batch processing, a dynamic space allocation mechanism, and a task scheduling framework to fully utilize the parallel inference capabilities of continuous batch processing, improve the concurrency and throughput of large-model inference, and address the limitations of traditional continuous batch processing that requires pre-allocated space.

[0051] In order to explain the present invention more clearly, a specific example of the high-concurrency reasoning method for a large language model of the present invention is provided below.

[0052] In traditional deep learning models, batch processing is often used to increase the concurrency of the model. This technology utilizes the parallel computing capabilities of the GPU to merge multiple requests and process the calculation results simultaneously. Batch processing usually follows the following steps: (1) When a request arrives, the calculation is not performed immediately, but waits for a certain period of time or until the number of requests reaches a specified number; (2) Multiple request tensors are concatenated into one tensor; (3) Video memory is pre-allocated for each request, sufficient to accommodate the model inference results; (4) The inference results are calculated using the model forward propagation; and (5) The results are split and distributed back to each request.

[0053] The above batch processing method is only suitable for traditional discriminative models and is not suitable for direct migration of existing generative models based on the Transformer architecture. The reasons are as follows: First, current generative models have variable-length outputs for each model input. For example, the model may output tens of characters or thousands of characters until the maximum length is reached, with no basis for judgment. Therefore, the memory required for the maximum length must be allocated for each request to prevent insufficient memory errors, resulting in low memory utilization. Second, due to the variable output length, requests with shorter outputs within the same batch must wait for the longest request in the batch to complete before they can be returned as a batch, wasting a significant amount of time. Finally, Transformer models are typically divided into a prefill phase and a decode phase, with different computation modes. The prefill phase is a parallel computation and caches the key and value values ​​of all characters (tokens: the unit of the model) for use in the decoding phase. This step reduces the computational overhead of inference by trading space for time. The decoding phase uses the previously calculated key and value caches to iterate over a new token and append the cached token until the end condition is met. The two phases cannot be simply calculated in the same batch.

[0054] The problems caused by these reasons are the main factors that limit the model reasoning speed and concurrency. In order to solve the above problems, the present invention implements an efficient parallel reasoning method, such as Figure 2 As shown in the figure, the key steps are as follows: (1) Use continuous batch processing to replace the traditional batch processing method; (2) In continuous batch processing, only one token is generated and returned for each inference; (3) Use a virtual block cache scheme to allocate video memory; (4) Dynamically allocate video memory space, and each request only allocates the video memory required for the current inference; (5) Separate the pre-filling stage from the decoding stage, that is, the pre-filling stage request and the decoding stage request cannot exist in one execution batch at the same time.

[0055] The present invention proposes an efficient parallel reasoning method, which adopts the following specific execution process:

[0056] Step 1: Module initialization. The executor calculates the memory block size by simulating peak data. If the memory is insufficient, the program will exit directly and a smaller model or a reduced maximum length will be required. After successful calculation, memory space is allocated and relevant memory resource information is synchronized to the scheduler.

[0057] Step 2: The scheduler receives the inference HTTP request, converts it into a sequence after preprocessing, including the request metadata and token sequence, and places it in the scheduler's waiting queue.

[0058] Step 3: When the waiting queue and the running queue are not completely empty, the scheduler checks the memory usage of the current running queue. It confirms whether each sequence can proceed to the next inference. If new memory blocks need to be allocated, it determines whether there are sufficient memory blocks. If not, the scheduler removes the sequence from the running queue based on priority and releases the corresponding memory blocks until each sequence can proceed to the next inference.

[0059] Step 4: The scheduler checks the current memory usage of the running queue and calculates the memory requirements of the sequences in the waiting queue according to the priority order. If there is enough memory, the requests in the waiting queue are placed in the running queue until the memory is insufficient or the threshold conditions such as the maximum batch character length and the maximum batch number are reached.

[0060] Step 5: The sequence types transferred from the waiting queue to the running queue in step 4 are all set to prefill; the sequence types in the running queue in step 3 are all set to decode. If the number of prefill sequences is greater than or equal to the decode type, prefill inference is performed, and the corresponding number of memory blocks are allocated to each prefill sequence. If the number of prefill sequences is less than the decode type, the prefill sequences are placed back in the waiting queue, and decode inference is performed, allocating the required memory blocks to each decode sequence.

[0061] Step 6: Call the executor to perform prefill inference or decode inference;

[0062] Step 7: The scheduler obtains the result returned by the executor and updates the sequence status and information. If prefill inference is performed, all prefill type sequences must be changed to decode type.

[0063] Step 8: Traverse all sequences in the running queue. If a sequence meets the end condition, remove it from the running queue, release the video memory, and respond to the HTTP request.

[0064] The present invention provides a method for improving the concurrency of large language model inference, which can achieve the effects of shorter time consumption, higher resource utilization, and increased inference request throughput, effectively reducing the time overhead of inference waiting, and improving the capabilities and effects of large model-related services and applications.

[0065] The solution consists of two parts: the scheduler and the executor. Parameters such as the block size (block_size), the maximum number of tokens in a batch (max_token_for_batch), and the maximum batch size (max_batch_size) must be specified. The block size refers to the number of tokens that can be stored in a block; the maximum number of tokens in a batch (prefix phase) must not exceed the maximum number of tokens; and the maximum batch size refers to the maximum number of sequences that can exist in a batch.

[0066] (1) The scheduler is responsible for receiving external HTTP requests, maintaining two queues (waiting and running), maintaining the video memory block mapping table, and allocating video memory resources. The specific process is as follows:

[0067] 1. When an external request arrives, it is pre-processed and converted into a request sequence that can be processed by the module and placed in the waiting queue. One request is one sequence;

[0068] 2. After the running queue completes an inference run, ensure that all sequences in the running queue have enough space to accommodate the video memory required for the next inference generation. If the video memory is insufficient, move the sequence in the running queue to the waiting queue according to priority and release the video memory block occupied by the sequence until the video memory is sufficient;

[0069] After that, the memory requirements of the sequences in the waiting queue are calculated based on their priorities. If the memory is sufficient, the sequences in the waiting queue are moved to the running queue until the memory is insufficient or the maximum batch character length (max_token_for_batch) is reached. (Note that all sequences in the waiting queue have not yet gone through the prefill phase.)

[0070] 3. After scheduling is complete, the system selects whether to perform prefill inference or decode inference based on the number of prefill and decode stages. If the number of prefill stages is greater than or equal to the number of decode stages, the system proceeds to prefill, otherwise, it proceeds to decode.

[0071] 4. If prefill inference is being performed, video memory resources are allocated to all prefill-stage sequences, and only prefill-stage sequences are fed into the inference process. If decode inference is being performed, all prefill-stage sequences are put back into the waiting queue, and video memory for the next token is allocated to each decode-stage sequence, and all sequences are fed into the inference process.

[0072] 5. Call the executor, return the result and update the status and information of all sequences in the running queue;

[0073] 6. For sequences that meet the end conditions, remove them from the running queue, release the video memory block and return the HTTP request;

[0074] Among them, the priority is sorted by the request arrival time, and the earlier the sequence arrives, the higher the priority; the end condition is usually reaching the specified maximum length or the appearance of the end marker; the situation of insufficient video memory is as shown in the attached Figure 3 shown.

[0075] It is important to note that, based on the set block_size, a new memory block is not allocated every time the next inference is performed. Instead, it is allocated only when the space in the memory block is exhausted. For example, if block_size=4, a new memory block is allocated only after every 4 rounds of inference in a sequence.

[0076] (2) The executor is responsible for executing the model reasoning task. The main process is:

[0077] 1. Initialize and calculate the number of video memory block resources and synchronize the video memory block information to the scheduler;

[0078] 2. Determine whether to proceed to the decode phase or the prefill phase and call the model forward reasoning;

[0079] 3. In the decode phase, the batch process performs one decode iteration, generates a new token, and caches the key and value of the token. In the prefill phase, the batch process builds a prefill matrix for each sequence, performs a prefill calculation, and caches the key and value of all tokens in the sequence.

[0080] The initialization phase is an important phase, which mainly includes executing simulated inference, calculating the model inference peak, calculating the number of memory blocks and allocating the corresponding space, etc., to ensure that the system can run stably.

[0081] The purpose of each memory block is to cache the value of a token that needs to be calculated in the model, that is, the key and value of a token at each layer of the model. Its role is to reduce the calculation of the token key and value by using space in exchange for time (if the caching method is not used, the key and value of all tokens in the sequence must be recalculated at each inference).

[0082] The calculation formula of the video memory block is as follows:

[0083]

[0084] Where, Indicates the memory size of the video memory block, block_size indicates the size of the video memory block, num_layers indicates the number of decoder stacking layers of the transformer architecture, num_heads indicates the number of attention heads, head_size indicates the head dimension, Indicates the number of bytes after bit width conversion of the data type;

[0085] For example, for a 32-layer model with 32 heads and a head dimension of 128, using the fp16 format transformer architecture, when the block_size is set to 4, the memory usage of a block is:

[0086]

[0087] During initialization, simulated data is constructed using the given maximum batch character length (max_token_for_batch) and maximum batch size (max_batch_size). The peak inference memory required for inference (peak_memory) is calculated, and the remaining memory is allocated to blocks. For example, when max_token_for_batch is 8192 and max_batch_size is 128, each sentence is 64 long. Simulated data is constructed for inference peak calculation, and during subsequent runs, the total token count in each prefill batch is ensured to not exceed 8192.

[0088]

[0089] Where, Indicates the number of video memory blocks, Indicates peak memory, Indicates the size of the video memory block, / / indicates the integer division operator;

[0090] Finally, the number of available video memory blocks is obtained according to the above formula, and the block video memory space is pre-allocated.

[0091] The parallel reasoning method implemented in this invention can effectively improve the number of concurrent model reasoning. Using the Qwen-14B model on two 3090 GPUs, the generation speed can reach up to 200 tokens / s, which is more than five times the token generation speed of conventional solutions.

[0092] Reference Figure 4 , Figure 4 Schematic diagram of the structure of a large language model high-concurrency reasoning system according to an embodiment of the present invention.

[0093] like Figure 4 As shown, the large language model high-concurrency reasoning system proposed in an embodiment of the present invention includes: an executor and a scheduler, and the executor and scheduler are configured to execute the large language model high-concurrency reasoning method as described in any one of the above.

[0094] Other embodiments or specific implementations of the large language model high-concurrency reasoning system of the present invention can refer to the above-mentioned method embodiments and will not be repeated here.

[0095] It should be understood that, in the description of this specification, reference to terms such as "one embodiment," "another embodiment," "other embodiments," or "first to Nth embodiments" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples.

[0096] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0097] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A high-concurrency reasoning method for a large language model, characterized by: The following steps are involved: S1: System initialization: The executor calculates the memory block size by simulating peak data, allocates memory space based on the calculation result, and synchronizes memory resource information to the scheduler; S2: The scheduler receives the inference HTTP request, converts the inference HTTP request into a request sequence through preprocessing, and puts the converted request sequence into the scheduler's waiting queue; S3: When the waiting queue and the running queue are not completely empty, the scheduler checks the memory usage of the current running queue to determine whether each request sequence can proceed to the next inference. If not, the scheduler allocates the corresponding memory block to each request sequence until each request sequence can proceed to the next inference. S4: The scheduler checks the memory usage of the current run queue, calculates the memory requirements of the request sequences in the waiting queue according to the priority order, and transfers the request sequences in the waiting queue to the run queue based on the memory requirement calculation results. Specifically, it includes: Calculate the video memory requirements of the request sequences in the waiting queue according to the priority order; wherein the priority order is configured as the arrival time order of the HTTP requests corresponding to each request sequence; If the video memory is sufficient, the requests in the waiting queue are put into the running queue until the video memory is insufficient or the threshold conditions of the maximum batch character length and the maximum batch number are reached; S5: defining the request sequence transferred from the waiting queue to the running queue in step S4 as a pre-fill type, defining the request sequence in the running queue in step S3 as a decoding type, and allocating the number of video memory blocks for executing pre-fill inference or for executing decoding inference based on the number of pre-fill types and the number of decoding types in the request sequence; S6: Call the executor to perform pre-fill inference or decoding inference based on the number of allocated memory blocks; S7: The scheduler obtains the result returned by the executor, updates the status and information of the request sequence, and changes the type of the request sequence of the pre-filled type; S8: Traverse all request sequences in the running queue and determine whether each request sequence meets the end condition. If so, remove the request sequence from the running queue, release the video memory, and respond to the http request.

2. The high-concurrency reasoning method for a large language model according to claim 1, characterized in that: In step S1, the memory block size is configured to be the number of tokens stored in the memory block.

3. The high-concurrency reasoning method for a large language model according to claim 1, characterized in that: In step S1, video memory space is allocated according to the calculation result, and the step also includes: if the video memory is insufficient, directly exiting the program, replacing a smaller language model, or reducing the maximum length.

4. The large language model high-concurrency reasoning method according to claim 1, characterized in that: In step S2, the request sequence includes metadata information of the http request and a token sequence.

5. The large language model high concurrency reasoning method according to claim 1, characterized in that: In step S3, it is determined whether each request sequence can be used for the next inference. If not, a corresponding video memory block is allocated to each request sequence until each request sequence can be used for the next inference. The process also includes: Determine whether the video memory of the current run queue can support each request sequence for the next inference. If not, allocate a new video memory block to the run queue; When the allocated new memory blocks are insufficient, the request sequences in the run queue are removed according to their priority and the corresponding memory blocks are released until each request sequence is able to proceed to the next inference.

6. The high-concurrency reasoning method for a large language model according to claim 1, characterized in that: In step S5, the number of video memory blocks used to perform pre-fill inference or for performing decoding inference is allocated according to the number of pre-fill types and the number of decoding types of the request sequence, specifically including: If the number of pre-fill type request sequences is greater than or equal to the number of decoding type request sequences, pre-fill inference is performed to allocate a corresponding number of memory blocks to each pre-fill type request sequence; If the number of pre-fill type request sequences is less than the number of decoding type request sequences, the pre-fill type request sequences are put back into the waiting queue, decoding inference is performed, and the required video memory blocks are allocated to each decoding type request sequence.

7. The large language model high concurrency reasoning method according to claim 1, characterized in that: In step S6, pre-fill inference or decoding inference is performed, specifically including: If pre-filled inference is performed, memory resources are allocated for all pre-filled request sequences, and only pre-filled request sequences are fed into the inference execution. If decoding inference is performed, all pre-filled request sequences are put back into the waiting queue, and the next token of video memory is allocated for each decoded request sequence, and all request sequences are sent to inference.

8. The large language model high-concurrency reasoning method according to claim 1, characterized in that: In step S7, the status and information of the request sequence are updated, and the type of the request sequence of the pre-filled type is changed, specifically including: Update the status and information of the request sequence; If performing pre-fill inference, change all pre-fill type request sequences to decode type.

9. A large language model high-concurrency reasoning system, characterized by: include: An executor and a scheduler, wherein the executor and the scheduler are configured to execute the large language model high-concurrency reasoning method as described in any one of claims 1-8.

Images