Dynamic batch processing method, device, equipment and medium for large language model reasoning

Abstract

The present invention discloses a dynamic batch processing method, device, equipment and medium for large language model reasoning. By obtaining the task request queue corresponding to the current large language model; obtaining the user's selection of dynamic batch processing information; performing pre-filling processing on each current request task according to whether the pre-filling stage tasks are not batched or the pre-filling stage tasks are batched, obtaining the completion processing results of each pre-filling stage, and generating a pre-filling stage processing completion queue; performing generation stage processing operations on each pre-filling stage completion processing result in the pre-filling stage processing completion queue, and storing the obtained generation stage completion processing result in the generation stage processing completion queue and outputting it. The problems of high memory loss, low computing resource utilization and high latency, and low acceleration chip reasoning efficiency caused by static batch processing are solved, the utilization rate and reasoning efficiency of chip memory bandwidth are improved, the memory loss is reduced, and the utilization rate of computing resources is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a dynamic batch processing method, device, equipment and medium for large language model reasoning. Background Art

[0002] The reasoning of a large language model is mainly divided into the pre-filling and generation stages. The pre-filling stage processes the input task data and obtains an output word unit, and the generation stage iteratively generates the output word unit. Batch processing of a large language model means loading the model parameters once and then using these parameters to process a batch of input queues, rather than loading new model parameters every time there is an input queue. This can more efficiently utilize the chip's memory bandwidth and achieve higher throughput.

[0003] In the process of implementing the present invention, the inventors found that the prior art has the following defects: At present, static batch processing means that the output can only be obtained after the last queue in the batch is completed, which will bring extremely high memory consumption, and computing resources are not fully utilized, with disadvantages such as high latency, which greatly reduces the reasoning efficiency of the acceleration chip. Summary of the invention

[0004] The present invention provides a dynamic batch processing method, device, equipment and medium for large language model reasoning, so as to improve the utilization rate of chip memory bandwidth and reasoning efficiency, and reduce memory loss.

[0005] According to one aspect of the present invention, a dynamic batch processing method for large language model reasoning is provided, which includes:

[0006] Get the task request queue corresponding to the current large language model;

[0007] Among them, the task request queue includes multiple current request tasks; the reasoning of the current large language model includes a pre-filling stage and a generation stage; the current large language model is based on an artificial intelligence dedicated acceleration chip;

[0008] Acquire user selected dynamic batch processing information, wherein the user selected dynamic batch processing information includes whether the pre-filling stage tasks are not batch processed or whether the pre-filling stage tasks are batch processed;

[0009] According to whether the pre-filling stage tasks are not batch processed or the pre-filling stage tasks are batch processed, pre-filling processing is performed on each of the current request tasks respectively, a completion processing result of each pre-filling stage is obtained, and a pre-filling stage processing completion queue is generated;

[0010] The generation phase processing operation is performed on each pre-filling phase processing completion result in the pre-filling phase processing completion queue, and the obtained generation phase processing completion result is stored in the generation phase processing completion queue and outputted.

[0011] According to another aspect of the present invention, a dynamic batch processing device for large language model reasoning is provided, which includes:

[0012] The task request queue acquisition module is used to obtain the task request queue corresponding to the current large language model;

[0013] Among them, the task request queue includes multiple current request tasks; the reasoning of the current large language model includes a pre-filling stage and a generation stage; the current large language model is based on an artificial intelligence dedicated acceleration chip;

[0014] A user selection dynamic batch processing information acquisition module, used for acquiring user selection dynamic batch processing information, wherein the user selection dynamic batch processing information includes whether the pre-filling stage tasks are not batch processed or whether the pre-filling stage tasks are batch processed;

[0015] A pre-filling stage processing completion queue generation module is used to perform pre-filling processing on each of the current request tasks respectively according to whether the pre-filling stage tasks are not batch processed or the pre-filling stage tasks are batch processed, obtain the completion processing results of each pre-filling stage, and generate a pre-filling stage processing completion queue;

[0016] The generation phase completion processing result storage and output module is used to perform generation phase processing operations on each pre-filling phase completion processing result in the pre-filling phase processing completion queue, and store the obtained generation phase completion processing result in the generation phase processing completion queue and output it.

[0017] According to another aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the dynamic batch processing method for large language model reasoning as described in any embodiment of the present invention is implemented.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the dynamic batch processing method for large language model reasoning described in any embodiment of the present invention when executed.

[0019] The technical solution of the embodiment of the present invention is to obtain the task request queue corresponding to the current large language model; wherein the task request queue includes multiple current request tasks; the reasoning of the current large language model includes a pre-filling stage and a generation stage; the current large language model is based on an artificial intelligence dedicated acceleration chip; obtain user selection dynamic batch processing information, wherein the user selection dynamic batch processing information includes not batching the pre-filling stage tasks and batching the pre-filling stage tasks; according to not batching the pre-filling stage tasks or batching the pre-filling stage tasks, pre-fill processing is performed on each of the current request tasks respectively, and each pre-filling stage completion processing result is obtained, and a pre-filling stage processing completion queue is generated; the generation stage processing operation is performed on each pre-filling stage completion processing result in the pre-filling stage processing completion queue, and the obtained generation stage completion processing result is stored in the generation stage processing completion queue and output. The problem of high memory loss, low computing resource utilization and high latency, and low acceleration chip reasoning efficiency caused by static batch processing is solved, the utilization rate and reasoning efficiency of chip memory bandwidth are improved, the memory loss is reduced, and the computing resource utilization is improved.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of a dynamic batch processing method for large language model reasoning provided in Embodiment 1 of the present invention;

[0023] Figure 2 is a flowchart of the method according to Embodiment 2 of the present invention in which the tasks in the pre-filling phase are not batch processed;

[0024] Figure 3 is a flowchart of batch processing of pre-filling phase tasks in the method provided in Embodiment 3 of the present invention;

[0025] Figure 4 is a schematic diagram of the structure of a dynamic batch processing device for large language model reasoning provided by Embodiment 4 of the present invention;

[0026] Figure 5It is a schematic diagram of the structure of an electronic device provided according to Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "target", "current", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Embodiment 1

[0030] Figure 1 A flowchart of a dynamic batch processing method for large language model inference is provided for Embodiment 1 of the present invention. This embodiment is applicable to the case of dynamic batch processing for large language model inference. The method can be executed by a dynamic batch processing device for large language model inference, and the dynamic batch processing device for large language model inference can be implemented in the form of hardware and / or software.

[0031] Correspondingly, such as Figure 1 As shown, the method includes:

[0032] S110: Obtain a task request queue corresponding to the current large language model.

[0033] Among them, the task request queue includes multiple current request tasks; the reasoning of the current large language model includes a pre-filling stage and a generation stage; the current large language model is based on an artificial intelligence dedicated acceleration chip.

[0034] In this embodiment, the task request queue of the current large language model can be obtained in real time or according to a preset periodic time. The obtained task request queue can include multiple current request tasks, and each current request task can be processed in the pre-filling stage and the generation stage by the current large language model.

[0035] S120: Obtain dynamic batch processing information selected by the user.

[0036] The user selected dynamic batch processing information includes not batch processing the pre-filling stage tasks and batch processing the pre-filling stage tasks.

[0037] In this embodiment, batch processing can be the number of input queues that can be processed by the artificial intelligence dedicated acceleration chip in the current large language model at one time.

[0038] Specifically, the input queue may be a queue that needs to be calculated in the pre-filling stage. If the user chooses to batch process the pre-filling stage tasks, the current input queue stores a batch of queues. If the user chooses not to batch process the pre-filling stage tasks, the input queue will always have only one queue.

[0039] In detail, in the process of dynamic batch processing of the current large model language, it can include: a pre-filling stage unprocessed queue, a pre-filling stage processing completed queue, a generation stage unprocessed queue and a generation stage processing completed queue.

[0040] Furthermore, in the pre-filling stage, the unprocessed queue stores the input queue that needs to be pre-processed. The pre-filling stage processing completed queue stores the pre-filling stage processing results that have completed the pre-filling stage processing. The generation stage unprocessed queue may be the pre-filling stage processing results that have not been processed in the generation stage. The generation stage processing completed queue may be the pre-filling stage processing results that have completed the generation stage processing.

[0041] S130 , performing pre-filling processing on each of the current request tasks respectively according to whether the pre-filling stage tasks are not batch processed or the pre-filling stage tasks are batch processed, obtaining completion processing results of each pre-filling stage, and generating a pre-filling stage processing completion queue.

[0042] In this embodiment, you can choose to perform batch processing or not in the pre-filling stage. Different methods correspond to different pre-filling processes, but the corresponding pre-filling stage completion processing results can be obtained, and the obtained pre-filling stage completion processing results can be stored in the pre-filling stage processing completion queue, and a pre-filling stage processing completion queue is generated.

[0043] S140 , performing generation phase processing operations on each pre-filling phase processing completion result in the pre-filling phase processing completion queue, and storing the obtained generation phase processing completion result in the generation phase processing completion queue and outputting it.

[0044] In this embodiment, after obtaining the pre-filling stage completion processing result, it is necessary to perform the generation stage processing operation on the pre-filling stage completion processing result, so as to obtain the generation stage completion processing result and further obtain the generation stage processing completion queue.

[0045] Furthermore, after obtaining the queues that have been processed in the generation phase, it is also necessary to obtain the current free batch number and quantity, so as to perform a reprocessing operation on the pending input queues until all input queues are traversed.

[0046] The technical solution of the embodiment of the present invention is to obtain the task request queue corresponding to the current large language model; wherein the task request queue includes multiple current request tasks; the reasoning of the current large language model includes a pre-filling stage and a generation stage; the current large language model is based on an artificial intelligence dedicated acceleration chip; obtain user selection dynamic batch processing information, wherein the user selection dynamic batch processing information includes not batching the pre-filling stage tasks and batching the pre-filling stage tasks; according to not batching the pre-filling stage tasks or batching the pre-filling stage tasks, pre-fill processing is performed on each of the current request tasks respectively, and each pre-filling stage completion processing result is obtained, and a pre-filling stage processing completion queue is generated; the generation stage processing operation is performed on each pre-filling stage completion processing result in the pre-filling stage processing completion queue, and the obtained generation stage completion processing result is stored in the generation stage processing completion queue and output. The problem of high memory loss, low computing resource utilization and high latency, and low acceleration chip reasoning efficiency caused by static batch processing is solved, the utilization rate and reasoning efficiency of chip memory bandwidth are improved, the memory loss is reduced, and the computing resource utilization is improved.

[0047] Embodiment 2

[0048] Figure 2 A flowchart of not batch processing the pre-filling stage tasks in the dynamic batch processing method for large language model reasoning is provided for the second embodiment of the present invention. This embodiment is refined based on the above embodiments. In this embodiment, the pre-filling stage tasks are not batch processed, and the pre-filling processing is performed on each current request task respectively to obtain the completion processing results of each pre-filling stage, and a pre-filling stage processing completion queue is generated for refinement processing.

[0049] Correspondingly, such as Figure 2 As shown, the method includes:

[0050] S210: Obtain a task request queue corresponding to the current large language model.

[0051] S220: Obtain dynamic batch processing information selected by the user.

[0052] S230: According to the pre-filling stage tasks, batch processing is not performed, and the current request tasks are grouped into a plurality of first input queues.

[0053] Each first input queue corresponds to a first batch processing number; each first input queue includes at least one current request task.

[0054] Generally speaking, by not batching tasks in the pre-filling stage, the current request tasks with longer texts can be processed, which can better handle memory-intensive calculations and dynamically insert queues in the generation stage. There is no need to wait for all batches to be calculated before taking new queues for calculation, which reduces invalid calculations and memory usage, and greatly improves hardware resource utilization.

[0055] S240 , pre-filling each current request task in each first input queue in turn to obtain a processing result of each first pre-filling stage.

[0056] In this embodiment, since the tasks in the pre-filling stage are not batch processed, the pre-filling processing operation can only be performed on one first input queue at a time.

[0057] Further, after the first input queue is pre-filled, the processing result may be completed in the first pre-filling stage.

[0058] S250, storing the first pre-filling stage completion processing results and the first batch processing numbers corresponding to the first pre-filling stage completion processing results in a first-in-first-out order to generate a pre-filling stage processing completion queue.

[0059] In this embodiment, the first input queue can be stored in the pre-filling stage unprocessed queue; the first pre-filling stage completed processing results after the pre-filling process and the corresponding first batch processing numbers can be stored in the pre-filling stage processing completed queue.

[0060] Specifically, the order of storage is first-in-first-out, that is, the first pre-filling stage completion processing result stored in the pre-filling stage processing completion queue will be taken out first for the processing operation of the generation stage.

[0061] S260: Perform generation phase processing operations on each pre-filling phase processing completion result in the pre-filling phase processing completion queue, and store the obtained generation phase processing completion result in the generation phase processing completion queue and output it.

[0062] After obtaining the queues that have been processed in the generation phase, it is necessary to obtain the current free batch number and quantity, so as to perform a reprocessing operation on the pending input queues until all input queues are traversed.

[0063] Specifically, after performing the generation phase processing operation on each pre-filling phase completion processing result in the pre-filling phase processing completion queue, storing the obtained generation phase completion processing result in the generation phase processing completion queue and outputting it, it also includes: obtaining the current free first batch processing number corresponding to after the generation phase processing, and statistically storing the number of the current free first batch processing numbers through the scheduling strategy; obtaining the first input queue to be processed in the pre-filling phase unprocessed queue, and performing pre-filling processing on each of the current request tasks in the first input queue to be processed, and storing the second pre-filling phase completion processing result after processing in the pre-filling phase processing completion queue; filling the second pre-filling phase completion processing result into the generation phase unprocessed queue corresponding to the current free first batch processing number according to the number of the current free first batch processing numbers; performing the generation phase processing operation on the second pre-filling phase completion processing result in the generation phase unprocessed queue; returning to execute the operation of obtaining the current free first batch processing number corresponding to after the generation phase processing until all the first input queues are traversed.

[0064] Among them, the scheduling strategy can be a strategy that determines whether the artificial intelligence dedicated acceleration chip performs pre-filling stage processing or generation stage processing.

[0065] In this embodiment, a pre-filling phase processing and multiple generation phase processing operations need to be performed in the large language model. Therefore, after completing the current generation phase processing, it is necessary to obtain the current free first batch processing number in the generation phase unprocessed queue, and it is necessary to count the number of the current free first batch processing numbers. Assume that here, there are 3 current free first batch processing numbers.

[0066] Furthermore, in the unprocessed queue of the pre-filling stage, the first input queue to be processed is obtained, and each current request task in the first input queue to be processed is pre-filled, and the second pre-filling stage completion processing results after processing are stored in the pre-filling stage processing completion queue; assuming that there are 5 second pre-filling stage completion processing results in the pre-filling stage processing completion queue.

[0067] Accordingly, since there are 3 currently available first batch processing numbers, 3 second pre-filling stage completion processing results are obtained in a first-in-first-out order, and are filled into the generation stage unprocessed queue corresponding to the currently available first batch processing number. Then, the generation stage processing operation can be performed on the second pre-filling stage completion processing results in the generation stage unprocessed queue. And it is necessary to return to execute the operation of obtaining the currently available first batch processing number corresponding to the generation stage processing again, until the traversal of each of the first input queues is completed.

[0068] The technical solution of the embodiment of the present invention is to obtain the task request queue corresponding to the current large language model; obtain the dynamic batch processing information selected by the user; according to the pre-filling stage tasks not being batch processed, each of the current request tasks is formed into multiple first input queues; pre-filling processing is performed on each current request task in each first input queue in turn to obtain the first pre-filling stage completion processing results; each of the first pre-filling stage completion processing results and the first batch processing numbers corresponding to each of the first pre-filling stage completion processing results are stored in a first-in-first-out order to generate a pre-filling stage processing completion queue; each pre-filling stage completion processing result in the pre-filling stage processing completion queue is subjected to generation stage processing operations, and the obtained generation stage completion processing results are stored in the generation stage processing completion queue and output. By not batch processing the pre-filling stage tasks, the current request tasks with longer texts can be processed, which can better handle memory-intensive calculations, and dynamically insert queues in the generation stage, without waiting for all batches to be calculated and then taking new queues for calculation, reducing invalid calculations and memory usage, and greatly improving hardware resource utilization.

[0069] Embodiment 3

[0070] Figure 3 A flowchart for batch processing of pre-filling stage tasks in a dynamic batch processing method for large language model reasoning is provided for Embodiment 3 of the present invention. This embodiment is refined based on the above embodiments. In this embodiment, the pre-filling stage tasks are batch processed, and each of the current request tasks is pre-filled separately to obtain the processing results of each pre-filling stage, and a pre-filling stage processing completion queue is generated for refined processing.

[0071] Correspondingly, such as Figure 3 As shown, the method includes:

[0072] S310: Obtain a task request queue corresponding to the current large language model.

[0073] S320: Obtain dynamic batch processing information selected by the user.

[0074] S330: Perform batch processing according to the pre-filling stage tasks, and obtain the batch processing quantity and batch processing quantity threshold set by the current user.

[0075] The batch processing quantity currently set by the user may be the number of input queues that the user wants the dedicated artificial intelligence acceleration chip to process once. The batch processing quantity threshold may be the maximum number of input queues that the dedicated artificial intelligence acceleration chip can process once.

[0076] Generally speaking, batch processing of pre-filling stage tasks can process current request tasks with shorter input text or shorter generated text, which can reduce the first word delay, avoid invalid calculations in the generation stage, improve throughput and speed up inference.

[0077] S340: Determine a target batch processing quantity according to the batch processing quantity currently set by the user and the batch processing quantity threshold.

[0078] Optionally, determining the target batch quantity based on the batch quantity set by the current user and the batch quantity threshold includes: if the batch quantity set by the current user is less than the batch quantity threshold, determining the batch quantity set by the current user as the target batch quantity; if the batch quantity set by the current user is greater than or equal to the batch quantity threshold, determining the batch quantity threshold as the target batch quantity.

[0079] For example, assume that the current user sets the batch processing quantity to A and the batch processing quantity threshold to Th. , it means that the batch processing quantity currently set by the user can meet the requirements for one execution of the artificial intelligence dedicated acceleration chip, so the target batch processing quantity can be obtained as A.

[0080] Furthermore, if , it means that the batch processing quantity currently set by the user cannot meet the requirements for one execution of the artificial intelligence dedicated acceleration chip. Therefore, the maximum number of input queues that can be processed by the current artificial intelligence dedicated acceleration chip can be taken, that is, the target batch processing quantity Th can be obtained.

[0081] S350, grouping the current request tasks into a plurality of second input queues.

[0082] Each second input queue corresponds to a second batch processing number.

[0083] Optionally, after forming each of the current request tasks into multiple second input queues, the method further includes: using a pre-set attention link to perform a processing operation of filling the second input queues, mask matrices and position matrices of different lengths to the left with 0 to the same length, respectively, to obtain the filled second input queues, mask matrices and position matrices; wherein the attention link includes: a mask matrix and a position matrix.

[0084] In this embodiment, the attention link may include a mask matrix and a position matrix. The mask matrix is ​​a matrix used in the attention link to control the calculation range of the mask. The position matrix is ​​a matrix used in the attention link to perform position encoding. The position matrix is ​​often used in the rotational position encoding link after multiplication with the QK matrix (where Q is the query feature matrix and K is the key feature matrix).

[0085] In this embodiment, the second input queues, mask matrix, and position matrix of different lengths need to be padded to the left with 0 to the same length, so as to obtain a processed second input queue of the same length.

[0086] S360. According to the target batch processing quantity, pre-fill processing is performed on each current request task in the second input queue corresponding to the target batch processing quantity in sequence to obtain completion processing results of each third pre-filling stage.

[0087] Continuing with the previous example, assuming that the target batch processing quantity is A, A second input queues can be obtained, and each current request task in each second input queue is pre-filled respectively to obtain the processing results of each third pre-filling stage.

[0088] S370, storing the third pre-filling stage completion processing results and the second batch processing numbers corresponding to the third pre-filling stage completion processing results in a first-in-first-out order to generate a pre-filling stage processing completion queue.

[0089] S380, performing generation phase processing operations on each pre-filling phase processing completion result in the pre-filling phase processing completion queue, and storing the obtained generation phase processing completion result in the generation phase processing completion queue and outputting it.

[0090] After obtaining the queues that have been processed in the generation phase, it is necessary to obtain the current free batch number and quantity, so as to perform a reprocessing operation on the pending input queues until all input queues are traversed.

[0091] Specifically, after performing the generation phase processing operation on each pre-filling phase completion processing result in the pre-filling phase processing completion queue, storing the obtained generation phase completion processing result in the generation phase processing completion queue and outputting it, it also includes: obtaining the current free second batch processing number corresponding to the generation phase processing, and statistically storing the current number of free second batch processing numbers through the scheduling strategy; obtaining and judging whether the number of the remaining third pre-filling phase completion processing results in the pre-filling phase processing completion queue is greater than or equal to the statistical number of the current free second batch processing numbers; if so, obtaining the remaining third pre-filling phase completion processing results corresponding to the current number of free second batch processing numbers from the pre-filling phase processing completion queue, and performing the generation phase processing operation; if not, in the pre-filling phase In the unprocessed queue of the stage, obtain the second input queue to be processed, and pre-fill processing is performed on each of the current request tasks in the second input queue to be processed, and the fourth pre-filling stage completion processing result after processing is stored in the pre-filling stage processing completion queue, and according to the number of currently free second batch processing numbers, the remaining third pre-filling stage completion processing results and the fourth pre-filling stage completion processing results are filled into the generation stage unprocessed queue corresponding to the currently free second batch processing number, and the generation stage processing operation is performed on the remaining third pre-filling stage completion processing results and the fourth pre-filling stage completion processing results in the generation stage unprocessed queue; return to execute the operation of obtaining the corresponding current free second batch processing number after the generation stage processing until the traversal of each second input queue is completed.

[0092] In this embodiment, a pre-filling phase processing and multiple generation phase processing operations need to be performed in the large language model. Therefore, after completing the current generation phase processing, it is necessary to obtain the current free second batch processing number in the unprocessed queue of the generation phase, and it is necessary to perform statistics and store the current free second batch processing number number through the scheduling strategy. Assume that here, there are 3 current free second batch processing numbers.

[0093] Further, it is necessary to obtain and determine whether the number of remaining third pre-filling stage completion processing results in the pre-filling stage processing completion queue is greater than or equal to the number of currently available second batch processing numbers. Assume that the number of remaining third pre-filling stage completion processing results is 4.

[0094] Since the number of remaining third pre-filling stage completed processing results is greater than the number of currently available second batch processing numbers, the remaining third pre-filling stage completed processing results corresponding to the currently available second batch processing numbers are obtained from the pre-filling stage processing completion queue, and the generation stage processing operation is performed.

[0095] In addition, it is assumed that the number of remaining third pre-filling stage completed processing results is 0. Since the number of remaining third pre-filling stage completed processing results is less than the number of currently available second batch processing numbers, the pending second input queue is obtained in the pre-filling stage unprocessed queue, and each of the current request tasks in the pending second input queue is pre-filled, and the fourth pre-filling stage completed processing results after processing are stored in the pre-filling stage processing completed queue. It is assumed that there are 5 fourth pre-filling stage completed processing results in the pre-filling stage processing completed queue.

[0096] Furthermore, since there are 3 currently available second batch processing numbers, 3 fourth pre-filling stage completion processing results are obtained in a first-in-first-out order and filled into the generation stage unprocessed queue corresponding to the currently available second batch processing numbers. Then, the generation stage processing operation can be performed on the fourth pre-filling stage completion processing results in the generation stage unprocessed queue.

[0097] Correspondingly, the operation of obtaining the currently available second batch processing number corresponding to the processing in the generation phase may be returned to execution until all the second input queues are traversed.

[0098] In addition, it is assumed that the number of remaining third pre-filling stage completion processing results is 1. Since the number of remaining third pre-filling stage completion processing results is less than the number of currently available second batch processing numbers, the pending second input queue is obtained in the pre-filling stage unprocessed queue, and each of the current request tasks in the pending second input queue is pre-filled, and the fourth pre-filling stage completion processing results after processing are stored in the pre-filling stage processing completion queue. It is assumed that there are 5 fourth pre-filling stage completion processing results in the pre-filling stage processing completion queue. Further, since there are 3 currently available second batch processing numbers, 1 remaining third pre-filling stage completion processing result and 2 fourth pre-filling stage completion processing results are obtained in a first-in-first-out order, and filled into the generation stage unprocessed queue corresponding to the currently available second batch processing number. Then, the generation stage processing operation can be performed on the fourth pre-filling stage completion processing result in the generation stage unprocessed queue. Correspondingly, the operation of obtaining the currently available second batch processing number corresponding to the generation stage processing can be returned to execute until the traversal of each second input queue is completed.

[0099] Optionally, it also includes: when the generation stage processes the last second input queue, if no new queue is inserted, if the number of the second input queue does not meet the target batch processing number, the target batch processing number is reduced to the current number of the second input queue.

[0100] In this embodiment, if there is no new queue inserted, the target batch processing quantity is A. However, if the quantity of the current second input queue is B (B < A), the target batch processing quantity can be reduced and modified to B.

[0101] In addition, whether in the generation stage or the pre-filling stage, when processing the last input queue, it is necessary to determine whether the current input queue meets the target batch processing quantity, so as to perform operations of reducing or maintaining the target batch processing quantity. This can reduce ineffective calculations, thereby improving throughput and accelerating the inference speed.

[0102] The technical solution of the embodiment of the present invention includes: obtaining a task request queue corresponding to the current large language model; obtaining user-selected dynamic batch processing information; performing batch processing according to the tasks in the pre-filling stage to obtain the currently user-set batch processing quantity and the batch processing quantity threshold; determining the target batch processing quantity according to the currently user-set batch processing quantity and the batch processing quantity threshold; forming each of the current request tasks into multiple second input queues; according to the target batch processing quantity, sequentially performing pre-filling processing on each of the current request tasks in the second input queues corresponding to the number of the target batch processing quantity respectively, to obtain the processing results of each third pre-filling stage completion; storing the processing results of each third pre-filling stage completion and the second batch processing numbers respectively corresponding to the processing results of each third pre-filling stage completion in the order of first in first out to generate a pre-filling stage processing completion queue; performing generation stage processing operations on each of the processing results of the pre-filling stage completion in the pre-filling stage processing completion queue, and storing the obtained generation stage processing completion results in the generation stage processing completion queue and outputting them. Generally speaking, batch processing through the tasks in the pre-filling stage can process the current request tasks with shorter input texts or shorter generated texts, which can reduce the first word delay, avoid ineffective calculations in the generation stage, improve throughput and accelerate the inference speed.

[0103] Embodiment 4

[0104] Figure 4 FIG. is a schematic structural diagram of a dynamic batch processing device for large language model inference provided by Embodiment 4 of the present invention. The dynamic batch processing device for large language model inference provided in this embodiment can be implemented by software and / or hardware, and can be configured in a terminal device or a server to implement a dynamic batch processing method for large language model inference in the embodiment of the present invention. As Figure 4 shown, the device includes: a task request queue obtaining module 410, a user selection dynamic batch processing information obtaining module 420, a pre-filling stage processing completion queue generating module 430, and a generation stage processing completion result storing and outputting module 440.

[0105] The task request queue acquisition module 410 is used to acquire the task request queue corresponding to the current large language model;

[0106] Among them, the task request queue includes multiple current request tasks; the reasoning of the current large language model includes a pre-filling stage and a generation stage; the current large language model is based on an artificial intelligence dedicated acceleration chip;

[0107] A user selection dynamic batch processing information acquisition module 420 is used to acquire user selection dynamic batch processing information, wherein the user selection dynamic batch processing information includes whether the pre-filling stage tasks are not batch processed or whether the pre-filling stage tasks are batch processed;

[0108] The pre-filling stage processing completion queue generation module 430 is used to perform pre-filling processing on each of the current request tasks respectively according to whether the pre-filling stage tasks are not batch processed or the pre-filling stage tasks are batch processed, obtain the completion processing results of each pre-filling stage, and generate a pre-filling stage processing completion queue;

[0109] The generation phase completion processing result storage and output module 440 is used to perform generation phase processing operations on each pre-filling phase completion processing result in the pre-filling phase processing completion queue, and store the obtained generation phase completion processing result in the generation phase processing completion queue and output it.

[0110] The technical solution of the embodiment of the present invention is to obtain the task request queue corresponding to the current large language model; wherein the task request queue includes multiple current request tasks; the reasoning of the current large language model includes a pre-filling stage and a generation stage; the current large language model is based on an artificial intelligence dedicated acceleration chip; obtain user selection dynamic batch processing information, wherein the user selection dynamic batch processing information includes that the pre-filling stage tasks are not batched and the pre-filling stage tasks are batched; according to the pre-filling stage tasks not being batched or the pre-filling stage tasks being batched, each of the current request tasks is pre-filled respectively, and each pre-filling stage completion processing result is obtained, and a pre-filling stage processing completion queue is generated; each pre-filling stage completion processing result in the pre-filling stage processing completion queue is subjected to generation stage processing operation, and the obtained generation stage completion processing result is stored in the generation stage processing completion queue and output. The problem of high memory loss, low computing resource utilization and high latency, and low acceleration chip reasoning efficiency caused by static batch processing is solved, the utilization rate and reasoning efficiency of chip memory bandwidth are improved, the memory loss is reduced, and the computing resource utilization is improved.

[0111] On the basis of the above-mentioned embodiments, the pre-filling stage processing completion queue generation module 430 can be specifically used for: according to the pre-filling stage tasks not being batch processed, each of the current request tasks is organized into multiple first input queues; wherein each first input queue corresponds to a first batch processing number; each first input queue includes at least one current request task; pre-filling processing is performed on each current request task in each first input queue in turn to obtain each first pre-filling stage completion processing result; each of the first pre-filling stage completion processing results and the first batch processing numbers corresponding to each first pre-filling stage completion processing result are stored in a first-in-first-out order to generate a pre-filling stage processing completion queue.

[0112] On the basis of the above embodiments, it also includes a module for obtaining the currently available first batch processing number, which can be specifically used for: after performing the generation phase processing operation on each pre-filling phase completion processing result in the pre-filling phase processing completion queue, storing the obtained generation phase completion processing result in the generation phase processing completion queue and outputting it, obtaining the currently available first batch processing number corresponding to the generation phase processing, and statistically storing the number of currently available first batch processing numbers through the scheduling strategy; obtaining the first input queue to be processed in the pre-filling phase unprocessed queue, and performing pre-filling processing on each of the current request tasks in the first input queue to be processed, and storing the second pre-filling phase completion processing result after processing in the pre-filling phase processing completion queue; filling the second pre-filling phase completion processing result into the generation phase unprocessed queue corresponding to the currently available first batch processing number according to the number of currently available first batch processing numbers; performing the generation phase processing operation on the second pre-filling phase completion processing result in the generation phase unprocessed queue; returning to execute the operation of obtaining the currently available first batch processing number corresponding to the generation phase processing until all the first input queues are traversed.

[0113] On the basis of the above embodiments, the pre-filling stage processing completion queue generation module 430 may specifically include: a current user set batch number and batch number threshold acquisition unit, which may be specifically used to: perform batch processing according to the pre-filling stage tasks, and acquire the current user set batch number and batch number threshold; a target batch number determination unit, which may be specifically used to: determine the target batch number according to the current user set batch number and the batch number threshold; a second input queue composition unit, which may be specifically used to: compose each of the current request tasks into a plurality of second input queues; wherein each second input queue The columns correspond to a second batch processing number respectively; the third pre-filling stage completion processing result determination unit can be specifically used to: according to the target batch processing quantity, pre-fill processing is performed on each current request task in the second input queue corresponding to the target batch processing quantity in turn, to obtain each third pre-filling stage completion processing result; the pre-filling stage processing completion queue generation unit can be specifically used to: store each of the third pre-filling stage completion processing results and the second batch processing numbers corresponding to each of the third pre-filling stage completion processing results in a first-in-first-out order to generate a pre-filling stage processing completion queue.

[0114] Based on the above embodiments, the target batch quantity determination unit can be specifically used to: if the batch quantity set by the current user is less than the batch quantity threshold, the batch quantity set by the current user is determined as the target batch quantity; if the batch quantity set by the current user is greater than or equal to the batch quantity threshold, the batch quantity threshold is determined as the target batch quantity.

[0115] On the basis of the above embodiments, it also includes a left filling unit, which can be specifically used to: after the current request tasks are organized into multiple second input queues, the second input queues, mask matrices and position matrices of different lengths are respectively filled to the left with 0 to the same length through a pre-set attention link to obtain the filled second input queues, mask matrices and position matrices; wherein the attention link includes: a mask matrix and a position matrix.

[0116] On the basis of the above embodiments, it also includes a currently free second batch processing number acquisition module, which can be specifically used to: after performing the generation phase processing operation on each pre-filling phase completion processing result in the pre-filling phase processing completion queue, storing the obtained generation phase completion processing result in the generation phase processing completion queue and outputting it, obtain the current free second batch processing number corresponding to the generation phase processing, and statistically store the current number of free second batch processing numbers through the scheduling strategy; obtain and determine whether the number of remaining third pre-filling phase completion processing results in the pre-filling phase processing completion queue is greater than or equal to the statistical number of currently free second batch processing numbers; if so, obtain the remaining third pre-filling phase completion processing results corresponding to the current number of free second batch processing numbers from the pre-filling phase processing completion queue, and perform generation phase processing. If not, then in the pre-filling stage unprocessed queue, obtain the to-be-processed second input queue, and perform pre-filling processing on each of the current request tasks in the to-be-processed second input queue, and store the fourth pre-filling stage completion processing result after processing in the pre-filling stage processing completion queue, and according to the number of currently free second batch processing numbers, fill the remaining third pre-filling stage completion processing results and the fourth pre-filling stage completion processing results into the generation stage unprocessed queue corresponding to the currently free second batch processing number, and perform generation stage processing operation on the remaining third pre-filling stage completion processing results and the fourth pre-filling stage completion processing results in the generation stage unprocessed queue; return to execute the operation of obtaining the currently free second batch processing number corresponding to the generation stage processing, until all the second input queues are traversed and completed.

[0117] On the basis of the above embodiments, it also includes a target batch processing quantity reduction module, which can be specifically used for: when the generation stage processes the last second input queue, when no new queue is inserted, if the number of the second input queue does not meet the target batch processing quantity, the target batch processing quantity is reduced to the current number of the second input queue.

[0118] The dynamic batch processing device for large language model reasoning provided in the embodiment of the present invention can execute the dynamic batch processing method for large language model reasoning provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0119] Embodiment 5

[0120] Figure 5A schematic diagram of the structure of an electronic device 10 that can be used to implement the fifth embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0121] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0122] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0123] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a dynamic batch processing method for large language model reasoning.

[0124] In some embodiments, the dynamic batch processing method for large language model reasoning may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the dynamic batch processing method for large language model reasoning described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the dynamic batch processing method for large language model reasoning in any other appropriate manner (e.g., by means of firmware).

[0125] The method comprises: obtaining a task request queue corresponding to a current large language model; wherein the task request queue comprises a plurality of current request tasks; the reasoning of the current large language model comprises a pre-filling stage and a generating stage; the current large language model is based on a dedicated acceleration chip for artificial intelligence; obtaining user selected dynamic batch processing information, wherein the user selected dynamic batch processing information comprises that the pre-filling stage tasks are not batch processed and the pre-filling stage tasks are batch processed; according to whether the pre-filling stage tasks are not batch processed or the pre-filling stage tasks are batch processed, pre-filling processing is performed on each of the current request tasks respectively, and each pre-filling stage completion processing result is obtained, and a pre-filling stage processing completion queue is generated; generating stage processing operations are performed on each pre-filling stage completion processing result in the pre-filling stage processing completion queue, and the obtained generating stage completion processing result is stored in the generating stage processing completion queue and outputted.

[0126] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0128] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0130] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0131] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0132] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0133] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0134] Embodiment 6

[0135] Embodiment 6 of the present invention also provides a computer-readable storage medium, wherein the computer-readable instructions are used to execute a dynamic batch processing method for large language model reasoning when executed by a computer processor, the method comprising: obtaining a task request queue corresponding to the current large language model; wherein the task request queue includes multiple current request tasks; the reasoning of the current large language model includes a pre-filling stage and a generation stage; the current large language model is based on an artificial intelligence dedicated acceleration chip; obtaining user selection dynamic batch processing information, wherein the user selection dynamic batch processing information includes not batch processing the pre-filling stage tasks and batch processing the pre-filling stage tasks; according to whether the pre-filling stage tasks are not batch processed or the pre-filling stage tasks are batch processed, pre-filling processing is performed on each of the current request tasks respectively to obtain the completion processing results of each pre-filling stage, and a pre-filling stage processing completion queue is generated; the generation stage processing operation is performed on each pre-filling stage completion processing result in the pre-filling stage processing completion queue, and the obtained generation stage completion processing result is stored in the generation stage processing completion queue and output.

[0136] Of course, the computer-readable storage medium provided in an embodiment of the present invention has computer executable instructions that are not limited to the method operations described above, and can also execute related operations in the dynamic batch processing of large language model reasoning provided in any embodiment of the present invention.

[0137] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0138] It is worth noting that in the above-mentioned embodiment of dynamic batch processing of large language model inference, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0139] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A dynamic batch processing method for large language model reasoning, characterized in that: include: Get the task request queue corresponding to the current large language model; Among them, the task request queue includes multiple current request tasks; the reasoning of the current large language model includes a pre-filling stage and a generation stage; the current large language model is based on an artificial intelligence dedicated acceleration chip; Acquire user selected dynamic batch processing information; wherein the user selected dynamic batch processing information includes whether the pre-filling stage tasks are not batch processed or whether the pre-filling stage tasks are batch processed; According to whether the pre-filling stage tasks are not batch processed or the pre-filling stage tasks are batch processed, pre-filling processing is performed on each of the current request tasks respectively, a completion processing result of each pre-filling stage is obtained, and a pre-filling stage processing completion queue is generated; Performing generation phase processing operations on each pre-filling phase completion processing result in the pre-filling phase processing completion queue, and storing the obtained generation phase completion processing result in the generation phase processing completion queue and outputting it; The batch processing is performed according to the pre-filling stage tasks, the pre-filling processing is performed on each of the current request tasks respectively, the completion processing results of each pre-filling stage are obtained, and a pre-filling stage processing completion queue is generated, including: Perform batch processing according to the pre-filling stage tasks, and obtain the batch processing quantity and batch processing quantity threshold set by the current user; Determine a target batch processing quantity according to the batch processing quantity currently set by the user and the batch processing quantity threshold; Group the current request tasks into a plurality of second input queues; wherein each second input queue corresponds to a second batch processing number; According to the target batch processing quantity, pre-filling processing is performed on each current request task in the second input queue corresponding to the target batch processing quantity in sequence to obtain completion processing results of each third pre-filling stage; The third pre-filling stage completion processing results and the second batch processing numbers respectively corresponding to the third pre-filling stage completion processing results are stored in a first-in-first-out order to generate the pre-filling stage processing completion queue; Wherein, after batch processing is performed according to the pre-filling stage tasks, pre-filling processing is performed on each of the current request tasks respectively, each pre-filling stage completion processing result is obtained, and a pre-filling stage processing completion queue is generated; a generation stage processing operation is performed on each pre-filling stage completion processing result in the pre-filling stage processing completion queue, and the obtained generation stage completion processing result is stored in the generation stage processing completion queue and outputted, it also includes: Obtain the currently available second batch processing numbers corresponding to the generation phase processing, and perform statistics and store the number of currently available second batch processing numbers through the scheduling strategy; Obtain and determine whether the number of remaining third pre-filling stage completion processing results in the pre-filling stage processing completion queue is greater than or equal to the number of currently available second batch processing numbers; If yes, then obtaining the remaining third pre-filling stage completion processing results corresponding to the number of the currently available second batch processing numbers from the pre-filling stage processing completion queue, and performing a generation stage processing operation; If not, then in the pre-filling stage unprocessed queue, obtain the to-be-processed second input queue, perform pre-filling processing on each of the current request tasks in the to-be-processed second input queue, and store the fourth pre-filling stage completion processing result after processing in the pre-filling stage processing completion queue, and according to the number of the currently free second batch processing numbers, fill the remaining third pre-filling stage completion processing results and the fourth pre-filling stage completion processing results into the generation stage unprocessed queue corresponding to the currently free second batch processing number, and perform generation stage processing operations on the remaining third pre-filling stage completion processing results and the fourth pre-filling stage completion processing results in the generation stage unprocessed queue; Return to the operation of executing the currently available second batch processing number corresponding to the processing in the acquisition generation phase until all the second input queues are traversed.

2. The method according to claim 1, characterized in that The pre-filling stage tasks are not batch processed, and the pre-filling processing is performed on each of the current request tasks respectively to obtain the completion processing results of each pre-filling stage, and generate a pre-filling stage processing completion queue, including: According to the pre-filling stage tasks not being batch processed, each of the current request tasks is organized into a plurality of first input queues; Each first input queue corresponds to a first batch processing number; each first input queue includes at least one current request task; Pre-filling each current request task in each first input queue in turn to obtain a processing result of each first pre-filling stage; The first pre-filling stage processing completion results and the first batch processing numbers corresponding to the first pre-filling stage processing completion results are stored in a first-in-first-out order to generate the pre-filling stage processing completion queue.

3. The method according to claim 2, characterized in that After performing the generation phase processing operation on each pre-filling phase completion processing result in the pre-filling phase processing completion queue, storing the obtained generation phase completion processing result in the generation phase processing completion queue and outputting it, the method further includes: Obtain the current free first batch processing number corresponding to the generation phase processing, and count and store the current free first batch processing number number through the scheduling strategy; In the pre-filling stage unprocessed queue, obtain the first input queue to be processed, and perform pre-filling processing on each of the current request tasks in the first input queue to be processed, and store the second pre-filling stage completion processing result after processing in the pre-filling stage processing completion queue; According to the number of the currently available first batch processing numbers, filling the processing results completed in the second pre-filling stage into the unprocessed queue of the generation stage corresponding to the currently available first batch processing numbers; Performing generation phase processing operations on the second pre-filling phase completed processing results in the generation phase unprocessed queue; Return to the operation of executing the currently available first batch of processing numbers corresponding to the processing of the acquisition generation phase until all the first input queues are traversed.

4. The method according to claim 1, characterized in that: The step of determining the target batch processing quantity according to the batch processing quantity currently set by the user and the batch processing quantity threshold comprises: If the batch processing quantity currently set by the user is less than the batch processing quantity threshold, determining the batch processing quantity currently set by the user as the target batch processing quantity; If the batch processing quantity currently set by the user is greater than or equal to the batch processing quantity threshold, the batch processing quantity threshold is determined as the target batch processing quantity.

5. The method according to claim 4, characterized in that After the current request tasks are grouped into a plurality of second input queues, the method further includes: Through the pre-set attention link, the second input queue, mask matrix and position matrix of different lengths are respectively filled with 0 to the left to the same length, so as to obtain the filled second input queue, mask matrix and position matrix; Among them, the attention link includes: mask matrix and position matrix.

6. The method according to claim 1, characterized in that Also includes: When the generation phase processes the last second input queue and no new queue is inserted, if the number of the second input queue does not meet the target batch number, the target batch number is reduced to the current number of the second input queue.

7. A dynamic batch processing device for large language model reasoning, characterized in that: include: The task request queue acquisition module is used to obtain the task request queue corresponding to the current large language model; Among them, the task request queue includes multiple current request tasks; the reasoning of the current large language model includes a pre-filling stage and a generation stage; the current large language model is based on an artificial intelligence dedicated acceleration chip; A user selection dynamic batch processing information acquisition module is used to acquire user selection dynamic batch processing information; wherein the user selection dynamic batch processing information includes whether the pre-filling stage tasks are not batch processed and whether the pre-filling stage tasks are batch processed; A pre-filling stage processing completion queue generation module is used to perform pre-filling processing on each of the current request tasks respectively according to whether the pre-filling stage tasks are not batch processed or the pre-filling stage tasks are batch processed, obtain the completion processing results of each pre-filling stage, and generate a pre-filling stage processing completion queue; A generation phase completion processing result storage and output module is used to perform generation phase processing operations on each pre-filling phase completion processing result in the pre-filling phase processing completion queue, and store the obtained generation phase completion processing result in the generation phase processing completion queue and output it; The pre-filling stage processing completion queue generation module includes: A batch processing quantity and a batch processing quantity threshold value acquisition unit set by the current user is used to perform batch processing according to the pre-filling stage tasks and acquire the batch processing quantity and the batch processing quantity threshold value set by the current user; a target batch processing quantity determining unit, configured to determine a target batch processing quantity according to the current user-set batch processing quantity and the batch processing quantity threshold; A second input queue forming unit, used for forming each of the current request tasks into a plurality of second input queues; wherein each second input queue corresponds to a second batch processing number; A third pre-filling stage completion processing result determination unit is used to perform pre-filling processing on each current request task in the second input queue of a number corresponding to the target batch processing quantity in sequence according to the target batch processing quantity, to obtain each third pre-filling stage completion processing result; a pre-filling stage processing completion queue generating unit, configured to store each of the third pre-filling stage processing completion results and the second batch processing numbers respectively corresponding to each of the third pre-filling stage processing completion results in a first-in-first-out order, so as to generate the pre-filling stage processing completion queue; Currently there is a free second batch number acquisition module, which is used for: In the batch processing according to the pre-filling stage tasks, pre-filling processing is performed on each of the current request tasks respectively, and each pre-filling stage completion processing result is obtained, and a pre-filling stage processing completion queue is generated; the generation stage processing operation is performed on each pre-filling stage completion processing result in the pre-filling stage processing completion queue, and the obtained generation stage completion processing result is stored in the generation stage processing completion queue and output, and then the current free second batch processing number corresponding to the generation stage processing is obtained, and the number of the current free second batch processing numbers is statistically stored through the scheduling strategy; Obtain and determine whether the number of remaining third pre-filling stage completion processing results in the pre-filling stage processing completion queue is greater than or equal to the number of currently available second batch processing numbers; If yes, then obtaining the remaining third pre-filling stage completion processing results corresponding to the number of the currently available second batch processing numbers from the pre-filling stage processing completion queue, and performing a generation stage processing operation; If not, then in the pre-filling stage unprocessed queue, obtain the to-be-processed second input queue, perform pre-filling processing on each of the current request tasks in the to-be-processed second input queue, and store the fourth pre-filling stage completion processing result after processing in the pre-filling stage processing completion queue, and according to the number of the currently free second batch processing numbers, fill the remaining third pre-filling stage completion processing results and the fourth pre-filling stage completion processing results into the generation stage unprocessed queue corresponding to the currently free second batch processing number, and perform generation stage processing operations on the remaining third pre-filling stage completion processing results and the fourth pre-filling stage completion processing results in the generation stage unprocessed queue; Return to the operation of executing the currently available second batch processing number corresponding to the processing in the acquisition generation phase until all the second input queues are traversed.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the processor implements a dynamic batch processing method for large language model reasoning as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a dynamic batch processing method for large language model reasoning according to any one of claims 1 to 6 when executed.

Images