Model inference acceleration method and related apparatus

Abstract

A model inference acceleration method and a related apparatus, which are applied to the technical field of artificial intelligence (AI). The model inference acceleration method comprises: first acquiring a predicted processing-result length corresponding to a prompt word, so as to determine a predicted length value of a processing result obtained by an accelerator when processing each prompt word; and then dividing prompt words in a prompt word set into batches on the basis of the predicted processing-result length, such that prompt words, the processing results of which have close predicted length values, can be divided into the same batch. In this way, when an accelerator is scheduled to process prompt words divided into batches, the plurality of prompt words in the same batch can be processed at the same time point as much as possible, such that the phenomenon of computing resources of the accelerator being wasted in some time periods due to processing results of some prompt words being excessively long is avoided, thereby improving the resource utilization rate of the accelerator, and improving the overall inference efficiency for a model.

Description

Model inference acceleration method and related device

[0001] The present application claims priority from the Chinese patent application No. 202410842327.7 filed on June 26, 2024, and entitled "Model Inference Acceleration Method and Related Device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a model inference acceleration method and related device. BACKGROUND

[0003] A large language model (LLM) is a deep learning model trained based on massive text data. The large language model can not only generate natural language text, but also deeply understand the meaning of the text and process various natural language tasks, such as generating a text summary, intelligent question answering, text translation, etc.

[0004] With the rapid development and wide application of large language models, the computational demand of the inference stage of large language models is also increasing, so the inference acceleration method for large language models becomes crucial. Specifically, in most application scenarios (such as human-computer dialogue scenarios or content recommendation scenarios), it is usually required that the large language model can respond in real time. The inference speed of the large language model is usually slow, and the inference acceleration method can reduce the inference time of the large language model as much as possible to meet the real-time requirement.

[0005] At present, the large language model is usually run on an accelerator such as a graphics processing unit (GPU). In order to improve the utilization rate of the accelerator to ensure the inference efficiency of the large language model, the accelerator usually processes multiple prompt words in batches by running the large language model, thereby effectively utilizing the parallel computing capability of the accelerator. However, in some cases, there is still a time period in which the computational resources of the accelerator cannot be efficiently utilized in the related technology, thereby affecting the inference efficiency of the large language model. SUMMARY

[0006] The present application provides a model inference acceleration method, which can improve the resource utilization rate of the accelerator and further improve the overall inference efficiency of the model.

[0007] In a first aspect, a model inference acceleration method is provided, which is applied to a training and inference acceleration system. The training and inference acceleration system includes a processor and an accelerator set. The processor can be a central processing unit (CPU) or other hardware with processing capability. The accelerator set can include one or more accelerators. The accelerator can be hardware or software for artificial intelligence model training or inference, such as a GPU, a tensor processing unit (TPU), or a neural network processing unit (NPU).

[0008] In the model inference acceleration method, first, the processor obtains a processing result prediction length corresponding to each prompt in a prompt set. The processing result prediction length is used to indicate a predicted value obtained by predicting the length of the processing result of the prompt processed by the first model. The processing result prediction length corresponding to each prompt can be the length prediction value of the token output by the first model, i.e., the number prediction value of the token.

[0009] Then, based on the processing result prediction length corresponding to each prompt, the processor divides a plurality of prompts in the prompt set into a plurality of batches. Each batch includes at least two prompts, and the difference between the processing result prediction lengths corresponding to any two prompts in each batch is less than a preset threshold value. The specific value of the preset threshold value can be set or adjusted according to actual conditions, such as 10, 20, or 30, which is not limited herein. That is, when performing batch division of the prompts, the processor divides the prompts with similar processing result prediction lengths into the same batch, thereby avoiding the situation that the prompts in the same batch have a large difference in processing result length.

[0010] Finally, the processor schedules the accelerator set to process the prompts in the plurality of batches to obtain the processing result. The prompts in the same batch are processed in parallel on the same accelerator in the accelerator set.

[0011] In the scheme, before the accelerator processes the prompt word set, the processing result prediction length corresponding to the prompt word is obtained first to determine the length prediction value of the processing result obtained when the accelerator processes each prompt word, and then the prompt words in the prompt word set are batched based on the processing result prediction length, so that the prompt words with close length prediction values of processing results can be divided in the same batch. In this way, when the accelerator processes the batched prompt words, the multiple prompt words in the same batch can be processed as much as possible at the same time point, avoiding the phenomenon of waste of computing resources of the accelerator in some time periods due to the too long processing result of some prompt words, thereby improving the resource utilization rate of the accelerator and improving the overall inference efficiency of the model.

[0012] In a possible implementation, in the process of processing multiple batches of prompt words by the accelerator set, if there is a target batch in the multiple batches, the accelerator set moves the target prompt word in the target batch out of the target batch to suspend the processing of the target prompt word. The length of the actual processing result of the target prompt word in the target batch is greater than the processing result prediction length corresponding to the target prompt word. That is, the target batch actually refers to a batch containing a prompt word whose actual processing result length is greater than the processing result prediction length. The target prompt word can refer to one or more prompt words in the target batch. That is, if the actual processing result lengths of multiple prompt words in the target batch are all greater than the processing result prediction length, the multiple prompt words need to be moved out of the target batch.

[0013] In the scheme, by moving the prompt word whose actual processing result length is greater than the processing result prediction length out of the batch, it can be ensured that the accelerator set processes the prompt words in each batch strictly according to the pre-specified processing result prediction length, and it can be ensured that the time consumed by the accelerator set to process the prompt words in each batch is controllable, avoiding the influence of the too long actual processing result of some prompt words on the efficient processing of the prompt words in the current batch.

[0014] In a possible implementation, in the process of processing multiple batches of prompt words by the accelerator set, the processor monitors the length of the token generated when the accelerator set processes the multiple batches of prompt words.

[0015] When the processor monitors the target condition, the processor sends a first instruction to the accelerator set, and the first instruction is used to instruct the accelerator set to move the target prompt word out of the target batch. The target condition includes that the length of the token generated when the accelerator set processes the target prompt word is greater than the processing result prediction length corresponding to the target prompt word.

[0016] Specifically, if the processor monitors that the length of the token generated when the accelerator set processes the target prompt word is greater than the processing result prediction length corresponding to the target prompt word, it can be considered that the target prompt word is a prompt word whose prediction appears to be wrong, and therefore the first instruction is sent to the accelerator set to instruct the accelerator set to move the target prompt word out of the target batch.

[0017] In this scheme, by monitoring the length of the token generated when the accelerator set processes the prompt word by the processor, the prompt word whose actual processing result length is greater than the processing result prediction length can be found in time, and then the processor informs the accelerator set to move these prompt words out of the current batch to avoid affecting the efficient processing of the prompt words in the current batch.

[0018] In a possible implementation, after monitoring the target prompt word, the processor adds the target prompt word to the target queue, which is used to indicate the prompt word whose actual processing result length is greater than the processing result prediction length; then, the processor schedules the accelerator set to process the prompt words in the target queue.

[0019] In this scheme, by adding the target prompt word whose actual processing result length is greater than the processing result prediction length to the target queue, a new batch or batches can be formed based on the prompt words stored in the target queue, and then the target prompt word is divided into the same batch for processing, so as to realize the unified processing of the target prompt word and avoid the waste of accelerator resources caused by processing each target prompt word separately.

[0020] In a possible implementation, the accelerator set sends the target data generated when processing the target prompt word to the processor. Then, the processor caches the target data and sends the target data to the accelerator set when scheduling the accelerator set to process the prompt words in the target queue. The target data can be the feature data generated when processing the target prompt word, that is, the KV Cache. The accelerator set needs to use the generated target data when continuing to process the target prompt word.

[0021] In this scheme, by caching the feature data generated by the accelerator set in the memory space used by the processor and sending the cached feature data to the accelerator set when the processor schedules the accelerator set to continue processing the target prompt word, it can be ensured that the accelerator set continues to process the target prompt word based on the progress of the last processing of the target prompt word, avoiding the accelerator set to start processing the target prompt word from the beginning, thereby improving the efficiency of the accelerator set in processing the target prompt word.

[0022] In a possible implementation, the processing result prediction length corresponding to each prompt word is one of the candidate multiple length ranges.

[0023] In the scheme, the processing result prediction length corresponding to the prompt word is set as a length range, the processing result prediction task of the prompt word is changed to a range interval prediction task, the difficulty of length prediction is reduced, the accuracy of predicting the processing result length of the prompt word is improved, and the realizability of the scheme is ensured.

[0024] In a possible implementation, in the process of dividing the plurality of prompt words in the prompt word set into a plurality of batches, the processor divides the prompt words whose processing result prediction lengths are the same length range into the same batch. That is, the prompt words included in the same batch correspond to the same length range of processing result prediction length.

[0025] In the scheme, the batch in which the prompt word is located is determined based on the length range to which the processing result prediction length corresponding to the prompt word belongs, the prompt word can be quickly divided into batches, and the processing result prediction lengths corresponding to the prompt words in the same batch are ensured to be in the same range, which is beneficial to realize accurate allocation of accelerator storage resources.

[0026] In a possible implementation, in the process of dividing the plurality of prompt words in the prompt word set into a plurality of batches, the processor sorts the plurality of prompt words according to the processing result prediction length to obtain a sorting result; and then, based on the sorting result and the number of prompt words required to be included in one batch, the processor divides the plurality of prompt words into a plurality of batches.

[0027] In the scheme, the prompt words are sorted based on the processing result prediction length corresponding to the prompt word, and the batch in which the prompt word is located is determined based on the sorting result, which can ensure that the processing result prediction lengths corresponding to the prompt words in the same batch are as close as possible, and is beneficial to quickly and accurately divide the batches of prompt words.

[0028] In a possible implementation, the processor processes the prompt words in the prompt word set by running the second model to obtain the processing result prediction length corresponding to each prompt word. The second model is used to predict the length of the result output by the first model when processing the prompt word, and the parameter amount of the second model is less than that of the first model.

[0029] In the scheme, since the parameter amount of the second model used to predict the processing result length is relatively small, the second model can be run on the processor to process the prompt word, and then the processing result prediction length corresponding to each prompt word is obtained, so as to realize the prediction process by using the idle resources of the processor, avoid occupying the computing resources of the accelerator when running the prediction process on the accelerator, and be beneficial to free more accelerator resources to complete subsequent tasks.

[0030] In a possible implementation, the processor schedules the accelerator set to run a second model to process the prompt words in the set of prompt words, where the second model is used to predict the length of the result output by the first model when processing the prompt words, and the parameter quantity of the second model is less than that of the first model; and the processor receives the processing result prediction length corresponding to each prompt word sent by the accelerator set.

[0031] In this solution, the length of the processing result of the prompt word is predicted by scheduling the accelerator set to run the second model, which can efficiently complete the prediction process using the computing resources on the accelerator set and ensure the efficiency of predicting the length of the processing result of the prompt word in the set of prompt words. Moreover, the length of the processing result prediction of the prompt word is determined by the second model with a smaller parameter quantity, which can quickly obtain the length of the processing result prediction of the prompt word and will not affect the subsequent processing of the prompt word.

[0032] The second aspect provides a training and prediction acceleration system, which includes a processor and an accelerator set; the processor is configured to obtain a processing result prediction length corresponding to each prompt word in a set of prompt words, where the processing result prediction length is used to indicate a predicted value obtained by predicting the length of the processing result obtained by processing the prompt word by a first model; the processor is further configured to divide a plurality of prompt words in the set of prompt words into a plurality of batches based on the processing result prediction lengths, where each batch in the plurality of batches includes at least two prompt words, and the difference between the processing result prediction lengths corresponding to any two prompt words in each batch is less than a preset threshold; the processor is further configured to schedule the accelerator set to process the prompt words in the plurality of batches to obtain a processing result; and the prompt words in the same batch are processed in parallel on the same accelerator in the accelerator set.

[0033] In a possible implementation, during the process of processing the plurality of batches of prompt words, if there is a target batch in the plurality of batches, the accelerator set is configured to move a target prompt word in the target batch out of the target batch to suspend the processing of the target prompt word, where the length of the actual processing result of the target prompt word in the target batch is greater than the processing result prediction length corresponding to the target prompt word.

[0034] In a possible implementation, during the process of processing the plurality of batches of prompt words by the accelerator set, the processor is further configured to monitor the length of a token generated by the accelerator set when processing the plurality of batches of prompt words; and when a target condition is monitored, the processor is further configured to send a first instruction to the accelerator set, where the first instruction is used to instruct the accelerator set to move a target prompt word out of a target batch; and the target condition includes that the length of the token generated by the accelerator set when processing the target prompt word is greater than the processing result prediction length corresponding to the target prompt word.

[0035] In a possible implementation, the processor is further configured to add the target prompt word into a target queue, the target queue being used to indicate the prompt word whose actual processing result length is greater than the processing result predicted length; and the processor is further configured to schedule the accelerator set to process the prompt word in the target queue.

[0036] In a possible implementation, the accelerator set is further configured to send target data generated when processing the target prompt word to the processor; and the processor is further configured to cache the target data and send the target data to the accelerator set when scheduling the accelerator set to process the prompt word in the target queue.

[0037] In a possible implementation, the processing result predicted length corresponding to each prompt word is one of the candidate multiple length ranges.

[0038] In a possible implementation, the processor is specifically configured to divide the prompt words whose processing result predicted lengths are the same length range into the same batch.

[0039] In a possible implementation, the processor is further configured to: sort the multiple prompt words according to the processing result predicted lengths to obtain a sorting result; and divide the multiple prompt words into multiple batches based on the sorting result and the number of prompt words required to be included in one batch.

[0040] In a possible implementation, the processor is further configured to: process the prompt words in the prompt word set by running the second model to obtain the processing result predicted length corresponding to each prompt word; wherein the second model is used to predict the length of the result output by the first model when processing the prompt words, and the parameter amount of the second model is less than that of the first model.

[0041] In a possible implementation, the processor is further configured to: schedule the accelerator set to run the second model to process the prompt words in the prompt word set, wherein the second model is used to predict the length of the result output by the first model when processing the prompt words, and the parameter amount of the second model is less than that of the first model; and receive the processing result predicted length corresponding to each prompt word sent by the accelerator set.

[0042] In a third aspect, a training and inference acceleration system is provided, including: a processor, an accelerator set, and a memory; the memory is used to store computer instructions, when the processor executes the instructions, to make the training and inference acceleration system execute the method of any one of the above aspects.

[0043] In a fourth aspect, a computer readable storage medium is provided, the computer readable storage medium stores instructions, when the instructions run on a computer, the computer can execute the method of any one of the above aspects.

[0044] In a fifth aspect, a computer program product including instructions which, when executed on a computer, enable the computer to perform the method of any of the above aspects.

[0045] In a sixth aspect, a chip system is provided, which includes a processor and a communication interface for communicating with a module outside the chip system, the processor being configured to execute a computer program or instructions, so that a device in which the chip system is installed can perform the method of any of the above aspects.

[0046] In a seventh aspect, a computing device is provided, which includes a memory, a first accelerator set and a second accelerator set, and a processor, the memory storing code, the processor being configured to execute the code, when the code is executed, the computing device being configured to implement the operation steps of the method of any of the above aspects.

[0047] In an eighth aspect, a computing device cluster is provided, which includes at least one computing device, any of the computing devices being configured to execute a computer program or instructions, so that the computing device cluster can perform the method of any of the above aspects. Alternatively, some or all of the computing devices are collectively configured to execute a computer program or instructions, so that the computing device cluster can perform the method of any of the above aspects.

[0048] On the basis of the implementation manners of the above aspects, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0049] FIG. 1 is a schematic diagram of an accelerator running a large language model to process a plurality of batches of prompt words according to the present application;

[0050] FIG. 2 is a schematic diagram of a system architecture according to the present application;

[0051] FIG. 3 is a flowchart of a model inference acceleration method according to the present application;

[0052] FIG. 4 is a schematic diagram of determining a length range of a prompt word by a second model according to the present application;

[0053] FIG. 5 is a schematic diagram of performing batch division on a prompt word according to the present application;

[0054] FIG. 6 is a schematic diagram of processing a prompt word after length division of a prompt word batch based on a processing result according to the present application;

[0055] FIG. 7 is another flowchart of a model inference acceleration method according to the present application;

[0056] FIG. 8 is a schematic diagram of a processing flow of a prompt word set according to the present application;

[0057] Fig. 9 is a schematic diagram of a processing flow of a target prompt word provided by the present application;

[0058] Fig. 10 is a schematic diagram of a structure of a training acceleration system provided by the present application;

[0059] Fig. 11 is a schematic diagram of a structure of a computing device provided by the present application;

[0060] Fig. 12 is a schematic diagram of a structure of a computing device cluster provided by the present application;

[0061] Fig. 13 is a schematic diagram of a structure of another computing device cluster provided by the present application;

[0062] Fig. 14 is a schematic diagram of a structure of a chip provided by the present application;

[0063] Fig. 15 is a schematic diagram of a structure of a computer readable storage medium provided by the present application. DETAILED DESCRIPTION

[0064] In order to make the objects, technical solutions and advantages of the present application clearer, the embodiments of the present application are described below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Those skilled in the art can know that, as new application scenarios appear, the technical solutions provided by the present application are also applicable to similar technical problems.

[0065] The terms "first", "second", and the like in the description and in the claims of the present application and above-described drawings are used to distinguish similar objects, and are not necessarily used to describe a particular sequential or chronological order. It should be understood that the descriptions thus used can be interchanged, where appropriate, to enable the embodiments to be implemented in an order other than that illustrated or described in the present application. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a list of steps or modules as processes, methods, systems, products, or apparatuses are not necessarily limited to those steps or modules clearly listed, but can include other steps or modules not clearly listed or inherent to these processes, methods, products, or apparatuses. The naming or numbering of steps in the present application does not mean that the steps in the method flow must be performed in the order / time / logical order indicated by the naming or numbering. The flow steps that have been named or numbered can change the order of execution according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of units in the present application is a logical division, and in actual application, it can have another division method, for example, multiple units can be combined or integrated into another system, or some features can be ignored or not executed, in addition, the coupling or direct coupling or communication connection between the units shown or discussed can be through some interface, the indirect coupling or communication connection between the units can be electrical or other similar forms, which are not limited in the present application. And the units or sub-units described as separate components can or can not be physically separated, can or can not be physical units, or can be distributed to multiple circuit units, and some or all of the units can be selected according to actual needs to achieve the purpose of the present application.

[0066] For ease of understanding, some technical terms related to the present application are introduced first.

[0067] (1) Large language model (LLM)

[0068] A large language model refers to a deep learning model trained using a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, dialogue, etc., and are an important way to artificial intelligence.

[0069] Specifically, large language models are a technology that has emerged in recent years. Because large language models have undergone a fine data engineering and training process, a large amount of existing natural language processing knowledge has been learned inside their parameters. This knowledge can already replace humans for many language-related tasks, such as having large language models write code or having large language models summarize text.

[0070] (2) Prompt

[0071] Prompt originates from the input form designed by researchers for downstream tasks. Its role is to help the pre-trained model "recall" what it has "learned" during pre-training, so it can also be called prompt. For large language models, prompt is the user's input, indicating the task the large language model needs to complete. Prompt can be a simple question, a longer text, or a set of instructions, depending on the user's specific needs. Generally, prompt is usually a short text string that provides context and task-related information to help the model better understand the requirements and generate correct output. For example, in a question-answering task, the prompt may contain the question or topic description to help the large language model generate the correct answer. Moreover, prompt is usually designed by humans to help the large language model better understand specific tasks or domains.

[0072] In this way, when the large language model generates content, it first processes the prompt and then outputs based on its understanding of the prompt. The working principle of the large language model is to predict the probability of the next word appearing based on the user's input context, and then generate the next context word by word. Therefore, the difference in the user's input prompt will directly affect the quality of the large language model output. In some cases, the difference in the user's input prompt is just a few words, and the large language model may also generate significantly different content.

[0073] (3) Neural network

[0074] The neural network can be composed of neural units, and the neural unit can refer to an operation unit with xs (input data) and intercept 1 as input. The output of the operation unit can be:

[0075] where s = 1, 2, … n, n is a natural number greater than 1, Ws is the weight of xs, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce non-linear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. The neural network is a network formed by connecting multiple single neural units, i.e. the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of several neural units.

[0076] (4) Transformer network

[0077] Transformer network is a neural network model based on self-attention mechanism, which is used to process sequence data. Compared with traditional recurrent neural network model, Transformer network has better parallel performance and shorter training time, so it has been widely used in natural language processing field.

[0078] Transformer network contains multiple layers of encoder and decoder, each layer is composed of multiple attention mechanism modules and feedforward neural network modules. Among them, encoder is used to encode input sequence into a high-dimensional feature vector representation, and decoder is used to decode the vector representation into target sequence. In Transformer network, techniques such as residual connection and layer normalization are also used to accelerate model convergence and improve model performance.

[0079] Specifically, the core of Transformer network is self-attention mechanism, which assigns a weight to each position in the input sequence, and then uses these weighted position vectors as output. Generally, the calculation process of self-attention mechanism includes three steps: 1, calculate attention weight, calculate the attention weight between each position and other positions, that is, the importance of each position to other positions; 2, calculate the weighted sum, multiply each position vector with the attention weight, and then add them to get the weighted sum vector; 3, linear transformation, linear transformation of the weighted sum vector to get the final output vector.

[0080] (5) Token

[0081] Token is the smallest semantic unit in natural language processing, which can be obtained by decomposing long text such as combined words, sentences, paragraphs, articles, etc. Generally, token can be composed of letters, words or symbols, such as one or more letters, or one or more words. Simply put, token can be understood as a "fragment" of a word or phrase, that is, the smallest semantic unit that cannot be further divided. For example, "waterfall" can be divided into two tokens: water and fall. In addition, punctuation marks can also be divided into tokens, because punctuation marks also affect the semantic understanding of the whole text. For example, "I don't know." can be divided into five tokens: "I", "don", "'t", "know", ".".

[0082] (6) Key value cache (KV Cache)

[0083] When performing natural language processing tasks using a Transformer network, the Transformer network usually utilizes a self-attention mechanism to process the input sequence. In the self-attention mechanism, the Transformer network generates a corresponding Key (K) vector, Value (V) vector and Query (Q) vector for each token in the input sequence. On this basis, the Transformer network further calculates the matching degree of each Query vector and all Key vectors, which is usually realized by dot product operation. Then, the matching degree between the Query vector and the Key vector is used as the weight to calculate the weighted sum of all Value vectors, and finally the result is obtained.

[0084] When the Transformer network processes the input sequence, the Key vectors generated by the Transformer network for each token in the input sequence can be uniformly stored in a matrix, and the Value vectors generated by the Transformer network for each token in the input sequence can be stored in another matrix, so the KV Cache refers to the Key matrix and Value matrix generated by the Transformer network for the tokens in the input sequence, and these Key matrix and Value matrix are cached.

[0085] (7) Continuous batching

[0086] Continuous batching is an optimization strategy in the prompt processing process, also known as dynamic batching or iteration level scheduling batching, aiming to solve some inefficiency problems in the traditional batching strategy.

[0087] The traditional batching strategy is usually based on dynamic batching of prompts, that is, processing a batch of prompts at a time. The traditional batching strategy may cause some prompts to spend a longer time in the processing process, because some prompts in the same batch may be more time-consuming than other prompts.

[0088] In contrast, continuous batching adopts a more flexible approach. Continuous batching allows the size of a batch to be dynamically adjusted during the processing of the prompt words to adapt to the complexity of different prompt words. Specifically, continuous batching continuously adds new prompt words to the current batch during model inference without waiting for all prompt words in the current batch to be processed before processing new prompt words. That is, as soon as a prompt word in the current batch has been processed, a new prompt word can be added to the processed prompt word for processing. However, since continuous batching continuously adds prompt words for processing, a large amount of storage space in the accelerator is often required to store the data generated during the processing of the prompt words (i.e., KV Cache) to ensure that continuous batching of the prompt words can be successfully performed.

[0089] Currently, large language models are usually run on accelerators such as GPUs. In order to improve the utilization of the accelerator to ensure the inference efficiency of the large language model, the accelerator usually processes multiple prompt words in batches by running the large language model, thereby effectively utilizing the parallel computing capability of the accelerator.

[0090] However, through research by the applicant, it has been found that when multiple prompt words are processed in batches by the accelerator, the accelerator can only start processing the prompt words of the next batch when the prompt words in the same batch have all been processed. That is, the time required by the accelerator to process a batch of prompt words is actually determined by the prompt word with the longest processing result in the current batch. In this way, in the case where the lengths of the processing results corresponding to the prompt words in the same batch differ greatly, it often happens that part of the prompt words have been processed, but the processing of the prompt words in the current batch cannot be completed until the other part of the prompt words have been processed. As a result, after the accelerator has completed the processing of part of the prompt words, only part of the computing resources in the accelerator are actually utilized effectively, resulting in a low utilization rate of the accelerator in part of the time period, and further resulting in a low overall inference efficiency of the model.

[0091] Exemplarily, refer to FIG. 1, which is a schematic diagram of an accelerator running a large language model to process a plurality of batches of prompt words provided by the present application. As shown in FIG. 1, prompt words 1-10 are divided into two batches for processing. The first batch includes prompt words 1-5, and the second batch includes prompt words 6-10. Since the accelerator generates tokens one by one when processing a prompt word and converts the tokens into a final output result, the more tokens generated when processing a prompt word, the longer the accelerator needs to spend to complete the processing of the prompt word. In FIG. 1, when the accelerator processes the first batch of prompt words by running a large language model, the lengths of the processing results corresponding to prompt words 2-5 (i.e., processing result 2-processing result 5) are short, and the length of the processing result corresponding to prompt word 1 (i.e., processing result 1) is long. Therefore, after the accelerator completes the processing of prompt words 2-5, it still needs to wait for the processing of prompt word 1 to be completed before it can start processing the next batch of prompt words. Among them, the accelerator spends 6 seconds to complete the processing of prompt word 1 and obtains processing result 1. Since processing result 1 is the longest processing result in the first batch of prompt words, the processing time of the first batch of prompt words is actually the processing time of prompt word 1, i.e., 6 seconds.

[0092] Similarly, when the accelerator processes the second batch of prompt words by running a large language model, the lengths of the processing results corresponding to prompt words 6, 7, 9, and 10 are short, and the length of the processing result corresponding to prompt word 8 (i.e., processing result 8) is long. Among them, the accelerator spends 5 seconds to complete the processing of prompt word 8 and obtains processing result 8. Since processing result 8 is the longest processing result in the second batch of prompt words, the processing time of the second batch of prompt words is actually the processing time of prompt word 8, i.e., 5 seconds. That is, the total time spent by the accelerator to process the first batch of prompt words and the second batch of prompt words is 11 seconds.

[0093] As can be seen from FIG. 1, when the accelerator processes a batch of prompt words, if there are prompt words with significantly different processing result lengths in a batch of prompt words, the accelerator will continue to process the remaining prompt words with longer processing results after processing most of the prompt words with shorter processing results, resulting in higher utilization of computing resources in the first half of the time and lower utilization of computing resources in the second half of the time. That is, due to the excessively long processing results of some prompt words, the accelerator appears to waste computing resources in some time periods, which affects the overall inference efficiency of the model.

[0094] Therefore, the application provides a model inference acceleration method. Before processing the prompt word set by using an accelerator, the processing result prediction length corresponding to the prompt word is obtained to determine the length prediction value of the processing result obtained when the accelerator processes each prompt word. Then, the prompt words in the prompt word set are batch-divided based on the processing result prediction length, so that the prompt words with close length prediction values of the processing result can be divided in the same batch. In this way, when the accelerator processes the batch-divided prompt words, the prompt words in the same batch can be processed as much as possible at the same time point, so that the phenomenon of waste of computing resources of the accelerator in some time periods due to the processing result of some prompt words being too long can be avoided, and the resource utilization of the accelerator and the overall inference efficiency of the model are improved.

[0095] Please refer to FIG. 2, which is a schematic diagram of a system architecture provided by the application. As shown in FIG. 2, in the system architecture, the execution device 20 can be implemented by one physical host (computing device) or a plurality of physical hosts (computing device cluster). In the execution device 20, the accelerator set 201 and the processor 202 are included. The accelerator set 201 is used to run the first model 2011 to process the prompt word delivered by the processor 202. The processor 202 is used to obtain the task request (for example, the request of processing the prompt word) of the client and schedule the accelerator set 201 to process the prompt word based on the task request from the client.

[0096] Optionally, the second model 2012 can also be run on the processor 202, and the processor 202 can predict the length of the processing result obtained by processing the prompt word by the first model 2011 by running the second model 2012.

[0097] Optionally, the execution device 20 cooperates with other computing devices, for example, data storage devices, load balancers and the like. The execution device 20 can be arranged on one physical site or distributed on a plurality of physical sites.

[0098] In addition, the system architecture is also provided with a data storage system 21, which is used to store data such as prompt words, KV Cache or program codes.

[0099] Optionally, in order to persistently store data, the data storage system 21 can be located outside the execution device 20 and exchange data with the execution device 20 through a network. Optionally, in the case where the execution device 20 is a physical host, the data storage system 21 can also be located inside the execution device 20, such as the data storage system 21 exchanging data with the processor through a bus. At this time, the data storage system 21 behaves as a hard disk. In the case where the data storage system 21 is provided, the execution device 20 can use the data in the data storage system 21 (such as KV Cache) or call the program code in the data storage system to implement the model inference acceleration method provided in the present application.

[0100] Optionally, users can operate respective local devices to interact with the execution device 20. For example, a client 221 is deployed on the local device 22, and the user interacts with the execution device through the client 221 on the local device 22. The local device 22 can represent any computing device, such as a personal computer, a computer workstation, a smartphone, a tablet computer, a notebook computer, a smart car, etc.

[0101] The local device 22 can interact with the execution device 20 through a communication network of any communication mechanism / communication standard, which can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0102] Optionally, in the process of implementing the model inference acceleration method by the execution device 20, the local device 22 can provide the execution device 20 with a task request or a prompt word, so that the execution device 20 can process the prompt word through the model to implement the processing of the task. Moreover, after the execution device 20 implements the model inference acceleration method and obtains an output result, the output result can be fed back to the local device 22.

[0103] Optionally, the prompt word required by the execution device 20 when implementing the model inference acceleration method can also be pre-stored in the data storage system 21, and the execution device 20 obtains the prompt word to be processed by reading the prompt word data from the data storage system 21.

[0104] The model inference acceleration method provided in the present application can be applied to various scenarios of using large language models to process user task requests, such as human-computer dialogue scenarios, search recommendation scenarios, entertainment media scenarios, financial scenarios, e-commerce scenarios, social media platform scenarios, etc. Illustratively, in the human-computer dialogue scenario, a large language model is usually used to implement a chat robot or an automatic customer service system, and the model inference acceleration method provided in the present application can improve the efficiency of human-computer dialogue, thereby quickly responding to the questions raised by the user.

[0105] Referring to FIG. 3, FIG. 3 is a flow diagram of a model inference acceleration method provided by the present application. As shown in FIG. 3, the model inference acceleration method provided by the present application can be applied in a train-and-infer acceleration system, which includes a processor and an accelerator set. The processor may, for example, be a hardware with computing processing capability such as a CPU, and is configured to receive a task request from a user and schedule the accelerator set to perform the task according to the task request. The accelerator set may, for example, be a set of one or more accelerators. The accelerator may, for example, be a hardware or software for training or inferring an artificial intelligence model, such as a graphics processing unit (GPU), a tensor processing unit (TPU), or a neural network processing unit (NPU).

[0106] Specifically, the model inference acceleration method includes the following steps 301-305.

[0107] In step 301, the processor obtains a prompt word set including a plurality of prompt words.

[0108] Generally, when a user uses a model to perform a task, the user often needs to provide a corresponding prompt word so that the model can output a corresponding task result based on the input prompt word. In the present application, the plurality of prompt words included in the prompt word set may, for example, be prompt words provided by the user, or may, for example, be prompt words generated based on the task expected to be completed by the model. For example, in the case where the user sends a task request to an execution device through a client, the execution device performs a prompt word engineering according to the task request, thereby generating a corresponding prompt word set, each prompt word in the prompt word set corresponding to a task request. The present application does not make a specific limitation on the source of the prompt word set.

[0109] In addition, the prompt words in the prompt word set may, for example, be in the form of text to describe a task to be processed. For example, assuming that the task to be processed is a question-and-answer task, the prompt word may, for example, be “What is the ending of the character XX in the novel XX?” For another example, assuming that the task to be processed is a translation task, the prompt word may, for example, be “Please help translate the following sentence into English: XXX”. For another example, assuming that the task to be processed is a text generation task, the prompt word may, for example, be “Please refer to the following example XXX and write a story about XXX”. For another example, assuming that the task to be processed is a sentiment analysis task, the prompt word may, for example, be “Please analyze the following review content XXX and output whether the customer’s evaluation is positive or negative”.

[0110] In general, the prompt words included in the prompt word set can be used to describe any type of natural language processing task, and the present application does not make specific limitations thereto.

[0111] At step 302, the processor obtains a processing result prediction length corresponding to each prompt word in the prompt word set. The processing result prediction length is used to indicate a predicted value obtained by predicting the length of the processing result obtained by processing the prompt word by the first model.

[0112] In the present application, after obtaining the prompt word set, in order to facilitate batch division of the prompt words in the prompt word set, the processor can first obtain a processing result prediction length corresponding to each prompt word in the prompt word set. That is, for each prompt word in the prompt word set, the processor obtains a corresponding processing result prediction length to obtain a length prediction value of the processing result obtained by processing the prompt word by the first model.

[0113] Generally, when processing the prompt word by the first model, the obtained processing result is often the token output by the first model. Then, the processor converts the output token into a word to form the output result returned to the user client. Therefore, in the present application, the processing result prediction length corresponding to each prompt word can be a length prediction value of the token output by the first model, i.e., a token quantity prediction value. The first model may, for example, be a large language model or other model capable of natural language processing.

[0114] For each prompt word, the processing result prediction length obtained by the processor can be a specific length value, such as 50, 85, or 101. Alternatively, since it is difficult to predict an accurate length value, in order to reduce the prediction pressure of the first model and improve the prediction accuracy, the processing result prediction length obtained by the processor can be a length range, i.e., the processing result prediction length is no longer a specific length value, but a range value, such as 10-20, 30-50, 100-150, etc.

[0115] Exemplarily, the processing result prediction length corresponding to each prompt word is one of a plurality of length ranges of candidates. The plurality of length ranges can be ranges without overlap. That is, a plurality of length ranges of candidates are preset, and when determining the processing result prediction length corresponding to any prompt word, one length range of the plurality of length ranges of candidates is determined as the processing result prediction length corresponding to the prompt word. For example, in the case of 10 length ranges of candidates, each time the processing result prediction length corresponding to the prompt word is determined, one length range is selected from the 10 length ranges of candidates, so as to ensure that the processing result prediction length corresponding to the prompt word in the prompt word set falls within the 10 length ranges of candidates.

[0116] In this scheme, by setting the processing result prediction length corresponding to the prompt word as a length range, the processing result prediction task of the prompt word can be changed to a range interval prediction task, thereby reducing the difficulty of length prediction, improving the accuracy of predicting the processing result length of the prompt word, and ensuring the realizability of the scheme.

[0117] In addition, when the processing result prediction length corresponding to the prompt word is determined by the model, setting the processing result prediction length as a length range can effectively reduce the prediction difficulty, thereby ensuring that the length prediction of the processing result can be realized based on a model with a small amount of parameters, reducing the difficulty of model deployment and the computing resources required for predicting the processing result length, and ensuring the prediction efficiency of the processing result.

[0118] When the processing result prediction length corresponding to the prompt word is determined by the model, the processor can obtain the processing result prediction length corresponding to each prompt word in the prompt word set in multiple implementation manners.

[0119] In one possible implementation, the processor obtains the processing result prediction length corresponding to each prompt word in the prompt word set by running the second model to process the prompt word. The second model is used to predict the length of the processing result when the first model processes the prompt word, and the amount of parameters of the second model is less than that of the first model. The second model can be a natural language processing (NLP) model, such as a Transformer network model or a long short-term memory (LSTM) model, and the specific structure of the second model is not limited in the present application.

[0120] In addition, since the second model is only used to predict the length of the processing result of the prompt word, and does not need to give a corresponding answer based on the prompt word, compared with the first model, the second model can successfully complete the prediction task by using a network structure with a smaller number of parameters. Moreover, using the second model with a smaller number of parameters to predict the length of the processing result of the first model processing the prompt word can shorten the time required for predicting the length of the processing result, and avoid occupying too much time for predicting the length of the processing result, thereby affecting the subsequent prompt word processing process.

[0121] That is, since the second model for predicting the length of the processing result has a relatively small number of parameters, the second model can be run on the processor to process the prompt word, and then the processing result prediction length corresponding to each prompt word is obtained, so that the idle resources of the processor are used to complete the prediction process, and the calculation resources of the accelerator are not occupied when the prediction process is run on the accelerator, which is beneficial to freeing more accelerator resources to complete subsequent tasks. Moreover, the second model with a smaller number of parameters is used to determine the processing result prediction length corresponding to the prompt word, so that the processing result prediction length corresponding to the prompt word can be quickly obtained, and the subsequent processing process for the prompt word will not be affected. For example, in a specific implementation process, the time required for using the second model to determine the processing result prediction length corresponding to the prompt word can be controlled within 3 milliseconds.

[0122] In another possible implementation, the processor can schedule the accelerator set to run the second model to process the prompt words in the prompt word set, where the second model is used to predict the length of the result output by the first model processing the prompt word, and the number of parameters of the second model is smaller than that of the first model.

[0123] In the accelerator set, the processing result prediction length corresponding to each prompt word is obtained by running the second model, and the accelerator set can send the obtained processing result prediction length to the processor, so that the processor receives the processing result prediction length corresponding to each prompt word sent by the accelerator set.

[0124] In this scheme, the processing result length of the prompt word is predicted by scheduling the accelerator set to run the second model by the processor, which can efficiently complete the prediction process by using the calculation resources on the accelerator set, and ensure the efficiency of predicting the processing result length of the prompt word in the prompt word set.

[0125] Exemplarily, refer to FIG. 4, which is a schematic diagram of determining a length range of a prompt word by a second model provided in the present application. As shown in FIG. 4, after the prompt word is input into the second model running on the accelerator set, the second model can output a specific length range x to indicate the processing result prediction length corresponding to the prompt word. The length range x can be a length range determined in the candidate 10 length ranges (i.e., length range 1-length range 10). As shown in FIG. 4, the candidate 10 length ranges are respectively used to indicate different length ranges, and the length ranges indicated by the 10 length ranges do not overlap. That is, the second model can actually be understood as a classification model, which classifies the input prompt word into a certain category in the pre-set 10 categories (i.e., 10 length ranges). In this way, by converting the prediction length value task into a classification task, the difficulty of length prediction can be effectively reduced, so as to ensure that the second model implemented based on a simple network structure also has high prediction accuracy.

[0126] It should be noted that the second model can reuse the training data set of the first model when training, so as to avoid reacquiring the training data set. Specifically, since each training data in the training data set of the first model includes a prompt word and a label (i.e., a processing result of the prompt word expected by a user) corresponding to the prompt word, the label in the training data corresponding to the first model can be converted into a length range, so as to construct the training data corresponding to the second model, i.e., the training data corresponding to the second model includes a prompt word and a length range of a processing result of the prompt word. Of course, the second model can also be trained by using other training data sets, which is not limited here.

[0127] The above introduces the way of predicting the processing result prediction length of the prompt word by the model. In some embodiments, the processor can also obtain the processing result prediction length of the prompt word by other ways. For example, the processor can obtain a pre-established mapping table, which records the mapping relationship between the prompt word and the processing result prediction length. In this way, for the prompt words in the prompt word set, the processor can find the prompt word with the highest matching degree for each prompt word in the prompt word set in the mapping table, and then determine the corresponding processing result prediction length.

[0128] In step 303, based on the processing result prediction length, the processor divides the plurality of prompt words in the prompt word set into a plurality of batches, each batch in the plurality of batches includes at least two prompt words, and the difference between the processing result prediction lengths corresponding to any two prompt words in each batch is less than a pre-set threshold.

[0129] In a case where the processor obtains the processing result predicted length corresponding to each prompt word in the prompt word set, the processor can divide the prompt words in batches based on the processing result predicted length corresponding to each prompt word, so as to divide the prompt words with the same or similar processing result predicted length in the same batch, and further to avoid the situation that the prompt words in the same batch have a large difference in processing result length. The specific value of the preset threshold can be set or adjusted according to actual conditions, for example, 10, 20 or 30, which is not limited here.

[0130] Specifically, when the processor performs batch division on the prompt words in the prompt word set, the batch division on the prompt words can be performed in combination with the processing result predicted length corresponding to each prompt word in the prompt word set and the number of prompt words required to be included in one batch.

[0131] In one possible example, after obtaining the processing result predicted length corresponding to each prompt word in the prompt word set, the processor can sort the plurality of prompt words in the prompt word set in ascending or descending order of the processing result predicted length, to obtain a sorting result of the plurality of prompt words. Then, the processor combines the sorting result and the number of prompt words required to be included in one batch to sequentially determine the prompt words included in each batch, and further to implement batch division on the plurality of prompt words in the prompt word set.

[0132] Please refer to FIG. 5, which is a schematic diagram of performing batch division on prompt words provided by the present application. As shown in FIG. 5, it is assumed that the prompt word set includes prompt word 1-prompt word 10, each prompt word has a corresponding processing result predicted length, and the processing result predicted length is a length range. For example, the processing result predicted length of prompt word 1 is 6-20, the processing result predicted length of prompt word 2 is 1-5, and the processing result predicted length of prompt word 10 is 81-100. Then, the plurality of prompt words in the prompt word set can be sorted in ascending order of the processing result predicted length, and the sorted result is prompt word 2, prompt word 1, prompt word 5, prompt word 3, prompt word 4, prompt word 7, prompt word 9, prompt word 10, prompt word 6 and prompt word 8 in turn. In this way, in a case where one batch requires to include 5 prompt words, prompt word 2, prompt word 1, prompt word 5, prompt word 3 and prompt word 4 (i.e., the first 5 prompt words in the sorting result) can be divided into the first batch, and prompt word 7, prompt word 9, prompt word 10, prompt word 6 and prompt word 8 (i.e., the last 5 prompt words in the sorting result) can be divided into the second batch, so as to implement batch division on the prompt words.

[0133] In the scheme, the prompts are ranked based on the processing result prediction length corresponding to the prompts, and the batches in which the prompts are located are determined based on the ranking result, so that the processing result prediction lengths corresponding to the prompts in the same batch are as close as possible, and the batches of the prompts can be quickly and accurately divided.

[0134] In another possible example, in the case where the processing result prediction length corresponding to the prompts is in a length range, the processor can divide the prompts with the same processing result prediction length in the same length range into the same batch. That is, the prompts included in the same batch correspond to the processing result prediction lengths in the same length range.

[0135] In the scheme, the batches in which the prompts are located are determined based on the length range to which the processing result prediction length corresponding to the prompts belongs, so that the batches of the prompts can be quickly divided, and the processing result prediction lengths corresponding to the prompts in the same batch are in the same range, which is beneficial to the accurate allocation of accelerator storage resources.

[0136] In step 304, the processor schedules the accelerator set to process the multiple batches of prompts.

[0137] After determining the batches in which each prompt in the prompt set is located, the processor can send scheduling instructions and the multiple batches of prompts to the accelerator set to schedule the accelerator set to process the multiple batches of prompts.

[0138] The prompts in the same batch are processed in parallel on the same accelerator in the accelerator set. In the case where the accelerator set includes one accelerator, the processor can send the prompts to the accelerator batch by batch. That is, the accelerator can only process one batch of prompts at the same time, and after the accelerator finishes processing one batch of prompts, the processor sends the next batch of prompts to the accelerator. In the case where the accelerator set includes multiple accelerators, the processor can send different batches of prompts to different accelerators in the accelerator set, so that different accelerators can process different batches of prompts in parallel.

[0139] In step 305, the accelerator set returns the processing results of the multiple batches of prompts to the processor.

[0140] After the accelerator set completes the processing of the prompt set, the processor can obtain the processing results of the multiple batches of prompts returned by the accelerator set, so as to convert the processing results of the prompt set into output results and return the output results to the client of the user.

[0141] Specifically, when the accelerator set processes the set of prompt words, the accelerator set sends the tokens obtained by processing each prompt word to the processor, and then the processor converts the obtained tokens into words and returns them to the client of the user. Since the accelerator set continuously generates new tokens when processing the prompt words, the accelerator set continuously sends the newly generated tokens to the processor for conversion into words and return to the client of the user. During the processing of the prompt words by the accelerator set, for any one prompt word, when the output of the accelerator set is no longer a token but a terminator, it means that the processing of the prompt word is completed, and therefore the accelerator set returns the terminator to the processor, and the processor can consider that the processing of the prompt word is completed and no new word needs to be returned to the client of the user.

[0142] In this scheme, before processing the set of prompt words by using the accelerator, the processing result prediction length corresponding to the prompt word is obtained to determine the length prediction value of the processing result obtained by processing each prompt word by the accelerator, and then the prompt words in the set of prompt words are batched based on the processing result prediction length, so that the prompt words with close length prediction values of processing results can be divided into the same batch. In this way, when the accelerator processes the batched prompt words, the multiple prompt words in the same batch can be processed as much as possible at the same time point, avoiding the phenomenon that the accelerator wastes computing resources in some time periods due to the processing results of some prompt words being too long, thereby improving the resource utilization of the accelerator and improving the overall inference efficiency of the model.

[0143] Exemplarily, refer to FIG. 6, which is a prompt word processing diagram after the prompt words are batched based on the processing result prediction length provided in the present application. As shown in FIG. 6, for the prompt words 1-10 shown in FIG. 1, by determining the processing result prediction length corresponding to each prompt word, the prompt words 1-10 can be divided into corresponding batches. Specifically, the prompt words 2, 4, 5, 6 and 9 with shorter processing result prediction length are divided into the first batch of prompt words, and the prompt words 1, 3, 7, 8 and 10 with longer processing result prediction length are divided into the second batch of prompt words. In this way, when the first batch of prompt words is processed by running the second model by using the accelerator set, the processing time of the first batch of prompt words is the processing time corresponding to the prompt word with the longest processing result, i.e., the processing time of the first batch of prompt words is 2 seconds. When the second batch of prompt words is processed by running the second model by using the accelerator set, the processing time of the second batch of prompt words is the processing time corresponding to the prompt word with the longest processing result, i.e., the processing time of the second batch of prompt words is 6 seconds. Therefore, the total time consumed for processing the first batch of prompt words and the second batch of prompt words is 8 seconds, which is obviously much smaller than 11 seconds in the related technical scheme in FIG. 1.

[0144] In addition, compared with the related art, the processing result prediction length of each prompt word is obtained in the present scheme, the storage resources of the accelerator can be allocated on demand, and more accelerator storage resources can be saved, and the waste of accelerator storage resources can be avoided.

[0145] Specifically, in the process of processing the prompt word by the model, tokens are generated one by one, and each generated token is used as input to generate a new token in combination with the prompt word. Therefore, as the number of tokens generated by the model increases, the number of generated tokens required by the model when generating a new token will also increase, resulting in the need for more and more storage space in the accelerator to store the features corresponding to the generated tokens (i.e., KV Cache). Therefore, in the related art, since the processing result length corresponding to the prompt words in each batch is unknown, in order to ensure that the processing of the prompt words in each batch can be completed smoothly, it is often necessary to allocate as much storage space as possible to each batch of prompt words on the accelerator to ensure that the accelerator has enough storage space to store the KV Cache generated in the process of processing the prompt words.

[0146] However, in actual application, the actual processing result length corresponding to some batches of prompt words is relatively short, and the pre-allocated storage space in the accelerator is often not fully used in the process of processing the prompt words, resulting in a large amount of wasted storage space on the accelerator. In the present scheme, since the processing result prediction length of each prompt word is obtained in advance, the processing result prediction length corresponding to all prompt words in each batch can be determined, and then the storage space for each batch of prompt words can be allocated according to the processing result prediction length corresponding to all prompt words in each batch, so as to realize the on-demand allocation of storage space and avoid the waste of a large amount of storage space caused by the allocation of too much storage space for each batch of prompt words. That is, when the accelerator set processes the prompt words in each batch, the storage space for the prompt words in the current batch can be allocated based on the processing result prediction length corresponding to the prompt words in the current batch, and the size of the allocated storage space is related to the processing result prediction length corresponding to the prompt words in the current batch.

[0147] The above introduces the process of performing batch division on the prompt word based on the processing result prediction length of the prompt word. However, it is worth noting that since the processing result prediction length of the prompt word is actually predicted, it cannot be guaranteed that the processing result prediction length of the prompt word is completely correct in actual application, so there may be some difference between the processing result prediction length of the prompt word and the actual processing result length of the prompt word. For example, in the case where the processing result prediction length of the prompt word is a length range, the actual processing result length of the prompt word may exceed the length range.

[0148] Based on this, in order to improve the fault tolerance of the scheme, the application proposes a processing method of the prompt word, which can perform special processing on the prompt word with a difference between the processing result prediction length and the actual processing result length, to ensure the overall efficiency in the prompt word processing process.

[0149] For example, in the process of processing multiple batches of prompt words by the accelerator set, if there is a target batch in the multiple batches, the accelerator set moves the target prompt word in the target batch out of the target batch to suspend the processing of the target prompt word.

[0150] Among them, the actual processing result length of the target prompt word in the target batch is greater than the processing result prediction length corresponding to the target prompt word. That is, the target batch actually refers to a batch containing a prompt word whose actual processing result length is greater than the processing result prediction length. In addition, the target prompt word can refer to one or more prompt words in the target batch. That is, if the actual processing result length of multiple prompt words in the target batch is greater than the processing result prediction length, the multiple prompt words need to be moved out of the target batch.

[0151] Specifically, since each batch of prompt words has a corresponding processing result prediction length, if there is a prompt word in a batch whose actual processing result length is greater than the processing result prediction length during processing, it can be considered that the batch is a target batch, and the target prompt word in the target batch whose actual processing result length is greater than the processing result prediction length needs to be moved out of the target batch, that is, the processing of the target prompt word in the target batch is not continued. In this way, by moving the prompt word whose actual processing result length is greater than the processing result prediction length out of the batch, it can be ensured that the accelerator set processes the prompt words in each batch according to the pre-specified processing result prediction length, and the time consumed by the accelerator set to process the prompt words in each batch is controllable, avoiding the influence of the actual processing result of part of the prompt words on the efficient processing of the current batch of prompt words.

[0152] For example, suppose a batch contains prompts 1 through 5, and the predicted length of the processing results for each of these prompts is within the length range of 1. When an accelerator ensemble processes prompts 1 through 5 in parallel within this batch, the actual processing results for prompts 1 through 4 are within the length range of 1, while the actual processing result for prompt 5 is outside the length range of 1. Therefore, when the accelerator ensemble finishes processing prompts 1 through 4, prompt 5 has not yet been processed. That is, the length of the processing result already output by the accelerator ensemble when processing prompt 5 is already at the boundary of the length range of 1, but prompt 5 has not yet been processed. At this point, if prompt 5 continues to be processed, it will affect the processing of prompts in the next batch. Therefore, prompt 5 can be removed from the current batch, i.e., processing of prompt 5 can be paused, thereby triggering the processing of the next batch of prompts, avoiding excessive processing time for the current batch due to continuing to process prompt 5 alone.

[0153] Optionally, to confirm whether the actual processing result length of the prompt word is greater than the predicted processing result length, the processor can monitor the prompt word processing process of the accelerator set to determine whether there are prompt words in each batch whose actual processing result length is greater than the predicted processing result length.

[0154] For example, during the process of the accelerator set processing multiple batches of prompt words, the processor can monitor the length of the tokens generated by the accelerator set when processing multiple batches of prompt words. Since the accelerator set sends the tokens generated when processing each batch of prompt words to the processor, so that the processor can convert the tokens into words and return them to the user's client, the processor can monitor the length of the tokens generated by the accelerator set when processing each prompt word.

[0155] When the processor detects a target situation, it sends a first instruction to the accelerator set, which instructs the accelerator set to remove the target prompt word from the target batch. The target situation detected by the processor can be that the length of the token already generated when the accelerator set processes the target prompt word in the target batch is greater than the predicted length of the processing result corresponding to the target prompt word.

[0156] Specifically, if the processor detects that the length of the token generated when the accelerator set processes the target prompt word is greater than the predicted length of the processing result corresponding to the target prompt word, it can be considered that the target prompt word is a prompt word with a prediction error. Therefore, the processor sends a first instruction to the accelerator set to instruct the accelerator set to remove the target prompt word from the target batch.

[0157] In the scheme, the processor monitors the length of the token generated by the accelerator set when processing the prompt word, so as to find the prompt word whose actual processing result length is greater than the processing result prediction length in time, and then the processor informs the accelerator set to move the prompt word out of the current batch, so as to avoid affecting the efficient processing of the prompt word in the current batch.

[0158] Optionally, after the processor monitors the target prompt word whose actual processing result length is greater than the processing result prediction length, the processor can add the target prompt word to a target queue, which is used to indicate the prompt word whose actual processing result length is greater than the processing result prediction length. That is, the target queue is actually used to store some prompt words whose actual processing result length is greater than the processing result prediction length.

[0159] When the number of prompt words stored in the target queue reaches a certain number or the time length of adding the prompt words to the target queue reaches a certain time length, the processor can schedule the accelerator set to continue processing the prompt words in the target queue. Of course, in the case that the number of prompt words stored in the target queue is large, the prompt words in the target queue can also be divided into multiple batches for processing, so as to avoid that too many prompt words are included in the same batch, which causes the accelerator to be unable to normally process.

[0160] It should be noted that since the actual processing result length of the prompt word in the target queue is unknown, in order to ensure that the accelerator set can complete the processing of the prompt words in the target queue at one time, the processor can schedule the accelerator set to process the prompt words in the target queue when the accelerator set has sufficient free storage space.

[0161] In the scheme, by adding the target prompt word whose actual processing result length is greater than the processing result prediction length to the target queue, a new batch can be formed based on the prompt words stored in the target queue, and then the target prompt words are divided into the same batch for processing, so as to realize the unified processing of the target prompt words and avoid the waste of accelerator resources caused by processing each prompt word separately.

[0162] Optionally, in the case that the processor monitors the target prompt word, if the memory used by the processor has sufficient free memory space, the processor can simultaneously instruct the accelerator set to send the data generated when processing the target prompt word to the processor through the first instruction.

[0163] Then, after the accelerator set receives the first instruction sent by the processor, the accelerator set sends the target data generated when processing the target prompt to the processor. Then, the processor caches the target data in the memory and sends the target data to the accelerator set when scheduling the accelerator set to process the prompts in the target queue. The target data can be the feature data generated when processing the target prompt, i.e., KV Cache. The accelerator set needs to use the generated target data when continuing to process the target prompt.

[0164] That is, since the accelerator set has generated certain feature data when processing the target prompt, by caching the feature data generated by the accelerator set in the memory space used by the processor and sending the cached feature data to the accelerator set when the processor schedules the accelerator set to continue processing the target prompt, it can be ensured that the accelerator set continues to process the target prompt on the basis of the progress of the last processing of the target prompt, avoiding the accelerator set to start from the beginning to reprocess the target prompt, thereby improving the efficiency of the accelerator set in processing the target prompt.

[0165] Of course, if the memory used by the processor does not have enough free memory space, the accelerator set does not need to send the target data obtained by processing the target prompt to the processor, but deletes the target data and reprocesses the target prompt when the processor schedules the accelerator set to process the prompts in the target queue.

[0166] The above introduces the execution process of the model inference acceleration method provided by the present application. In order to facilitate understanding, the following will introduce the execution process of the model inference acceleration method in the actual application process in detail.

[0167] Please refer to FIG. 7, which is another flowchart of a model inference acceleration method provided by the present application. As shown in FIG. 7, the model inference acceleration method includes the following steps 701-709.

[0168] Step 701, the processor processes the prompt set by running the second model to obtain the processing result prediction length corresponding to each prompt.

[0169] In actual applications, each user can send a task request to the training acceleration system through a client to request the execution device to execute various tasks indicated by the user. In the case where multiple users send task requests to the training acceleration system, the prompt words carried in the task requests by each user or the prompt words converted based on the task requests can constitute a prompt word set, and the prompt word set includes multiple prompt words. After the processor in the execution device obtains the prompt word set, the processor can process each prompt word in the prompt word set by running the second model, and then obtain the processing result prediction length corresponding to each prompt word output by the second model.

[0170] Exemplarily, referring to FIG. 8, FIG. 8 is a processing flow diagram of a prompt word set provided by the present application. As shown in FIG. 8, when the processor predicts the length of the processing result of each prompt word in the prompt word set by running the second model, it is assumed that the prompt word set includes 5 prompt words, which are {1, "CEO of Google is?", 2, "The bright moonlight in front of the bed, doubt is the ground frost, look up and look at the bright moon, what is the next sentence?", 3, "Please introduce Beijing", 4, "What is good to eat in Hangzhou?", 5, "The sun is exhausted on the mountain, the Yellow River flows into the sea, and the next sentence is?"}. When the processor predicts the length of the processing result of the 5 prompt words by running the second model, it can be determined that the processing results of the 5 prompt words are located in the following 5 ranges: length range 1, length range 2, length range 9, length range 9, and length range 2. That is, the processing result prediction lengths corresponding to the above 5 prompt words are the above 5 length ranges.

[0171] 702, the processor divides the multiple prompt words in the prompt word set into multiple batches based on the processing result prediction length.

[0172] Based on the processing result prediction length corresponding to each prompt word, the processor can divide the multiple prompt words in the prompt word set into multiple batches, so that the processing result prediction lengths corresponding to the prompt words in the same batch are as same or similar as possible. The process of batch division of the processor on the multiple prompt words can refer to step 303 described above, and will not be described here.

[0173] As shown in FIG. 8, for the 5 prompt words shown in FIG. 8, the 5 prompt words can be divided into three different batches. Specifically, the first prompt word with a processing result prediction length of length range 1 can be divided into batch 1, the second prompt word and the fifth prompt word with a processing result prediction length of length range 2 can be divided into batch 2, and the third prompt word and the fourth prompt word with a processing result prediction length of length range 9 can be divided into batch 3.

[0174] 703, the processor schedules the accelerator set to process the plurality of batches of the prompt words.

[0175] After determining the batch in which each prompt word in the set of prompt words is located, the processor can then send a scheduling instruction and the plurality of batches of the prompt words to the accelerator set to schedule the accelerator set to process the plurality of batches of the prompt words and cause the prompt words in the same batch to be processed in parallel on the same accelerator in the accelerator set. Specifically, the process of the processor scheduling the accelerator set to process the plurality of batches of the prompt words is similar to the process described in step 304, and for details, please refer to step 304, which will not be described here.

[0176] 704, the accelerator set returns the token generated when processing the prompt word to the processor.

[0177] When processing the accelerator set to process the prompt word, the accelerator set will continuously generate tokens when processing each batch of the prompt word, and return the generated token to the processor, so that the processor converts the token into a word and returns it to the user's client.

[0178] For example, as shown in FIG. 8, when the token generated by the accelerator set is Chinese, a token can be a character or a word, so the processor can return the output result to the user's client word by word based on the token continuously returned by the accelerator set.

[0179] 705, the processor monitors the length of the token generated when the accelerator set processes the prompt word.

[0180] During the process of the accelerator set returning the token to the processor, the processor can continuously monitor the length of the token generated when the accelerator set processes the prompt word, so as to determine whether the length of the token generated for each prompt word in the processing of the accelerator set is greater than the predicted length of the processing result.

[0181] 706, when the processor monitors the target condition, the processor sends a first instruction to the accelerator set.

[0182] When the processor monitors that there is a prompt word in the target batch whose length of the token generated is greater than the predicted length of the processing result of the prompt word (i.e. the target condition), the processor can consider that the prompt word is the target prompt word, and then send a first instruction to the accelerator set to instruct the accelerator set to move the target prompt word out of the target batch.

[0183] 707, based on the first instruction, the accelerator set moves the target prompt word whose actual processing result length is greater than the predicted length of the processing result out of the target batch.

[0184] In the case that the accelerator set moves the target prompt word out of the target batch, the target prompt word is no longer processed in the current target batch, so that the target prompt word is no longer in the target batch.

[0185] For example, in the example shown in FIG. 8, when the accelerator set processes the fourth prompt word (i.e., “What is there to eat in Hangzhou?”), the length of the token output by the accelerator set exceeds the length range 9 corresponding to the prompt word, so the prompt word can be considered as a target prompt word, and the prompt word is moved out of batch 2.

[0186] 708, the processor adds the target prompt word to the target queue.

[0187] In the case that the length of the actual processing result corresponding to the target prompt word is greater than the processing result prediction length, the processor can add the target prompt word to the target queue. The target queue is used to record the prompt words whose actual processing result length is greater than the processing result prediction length. That is, during the process of processing the prompt word set by the accelerator set, the processor can add any prompt word to the target queue as long as the length of the actual processing result corresponding to the prompt word is greater than the processing result prediction length.

[0188] 709, the processor schedules the accelerator set to process the prompt words in the target queue.

[0189] After the processor adds the target prompt word to the target queue, if there is enough storage resource on the accelerator in the accelerator set (such as enough video memory space on the GPU), the processor can schedule the accelerator set to process the prompt words in the target queue.

[0190] As shown in FIG. 8, for the prompt word (i.e., “What is there to eat in Hangzhou?”) moved out of batch 2, the processor can schedule the accelerator set to continue processing the prompt word and other prompt words added to the target queue based on the continuous batching strategy.

[0191] Specifically, referring to FIG. 9, FIG. 9 is a flowchart of a processing process of a target prompt word provided by the present application. As shown in FIG. 9, for the target prompt word whose actual processing result length is greater than the processing result prediction length, the processor can process the target prompt word based on the flow shown in FIG. 9 to improve the processing efficiency of the target prompt word. The flow shown in FIG. 9 includes the following steps 901-906.

[0192] Step 901, for each prompt word in the processing of the accelerator set, the processor monitors the length of the token generated when the accelerator set processes the prompt word.

[0193] Since the accelerator set continuously returns the token generated when processing each prompt word to the processor, the processor can continuously monitor the length of the token generated when the accelerator set processes each prompt word.

[0194] Step 902, the processor determines whether the length of the generated token corresponding to the prompt word exceeds the predicted length range.

[0195] Since the processor has previously predicted the length range of the processing result corresponding to each prompt word, for each prompt word monitored by the processor, the processor can determine whether the length of the generated token corresponding to the prompt word has exceeded the predicted length range.

[0196] If the length of the generated token corresponding to the prompt word does not exceed the predicted length range, the processor does not intervene in the processing process of the prompt word, but processes the prompt word according to the normal inference process of the accelerator set, thereby completing the inference process of the prompt word.

[0197] Step 903, if the length of the generated token corresponding to the prompt word exceeds the predicted length range, the processor considers the prompt word as the target prompt word, and adds the target prompt word to the target queue.

[0198] Step 904, the processor schedules the accelerator set to reclaim storage resources or cache the KV Cache to the memory.

[0199] Since the accelerator set removes the target prompt word from the current batch, which means that the accelerator set no longer processes the target prompt word in the current batch, the processor can determine how the accelerator set processes the KV Cache generated when processing the target prompt word according to the current actual situation.

[0200] Specifically, when the processor has enough free storage space in the memory used by the processor, the processor can schedule the accelerator set to send the KV Cache generated when processing the target prompt word to the processor, and cache the KV Cache to the memory by the processor. When the processor does not have enough free storage space in the memory used by the processor, the processor can schedule the accelerator set to reclaim storage resources, that is, delete the KV Cache generated when processing the target prompt word to reclaim the storage resources occupied by the target prompt word on the accelerator.

[0201] Step 905, the processor determines whether the free storage resources in the accelerator set are sufficient.

[0202] After the target prompt word is added to the target queue, since the actual processing result length corresponding to the prompt word in the target queue is unknown, a large amount of storage resources on the accelerator set often need to be reserved when the prompt word in the target queue is processed. Based on this, the processor can continuously monitor the idle storage resources in the accelerator set, and determine whether the idle storage resources in the accelerator set are sufficient for processing the prompt word in the target queue.

[0203] Step 906, in the case where the idle storage resources in the accelerator set are sufficient, the processor schedules the accelerator set to continue processing the prompt word in the target queue by using the continuous batching strategy.

[0204] Specifically, for the prompt word in the target queue, if the processor caches the KV Cache corresponding to the prompt word into the memory in step 904, when the processor schedules the accelerator set to continue processing the prompt word in the target queue by using the continuous batching strategy, the processor can send the KV Cache cached in the memory to the accelerator set, so that the accelerator set can continue to process the prompt word based on the generated KV Cache, avoiding processing the prompt word from the beginning.

[0205] In addition, if the processor schedules the accelerator set to recover the storage resources of the prompt word in step 904, when the processor schedules the accelerator set to continue processing the prompt word in the target queue by using the continuous batching strategy, the accelerator set needs to process the prompt word from the beginning.

[0206] The above describes the method provided by the application in detail, and next the device provided by the application for executing the above method will be introduced.

[0207] Please refer to FIG. 10, which is a structural schematic diagram of a training and inference acceleration system provided by the application. As shown in FIG. 10, the training and inference acceleration system comprises a processor 1001 and an accelerator set 1002; the processor 1001 is configured to obtain a processing result prediction length corresponding to each prompt word in a prompt word set, the processing result prediction length being used to indicate a predicted value obtained by predicting the length of the processing result obtained by processing the prompt word by using a first model; the processor 1001 is further configured to divide a plurality of prompt words in the prompt word set into a plurality of batches based on the processing result prediction length, each batch in the plurality of batches comprising at least two prompt words, and the difference between the processing result prediction lengths corresponding to any two prompt words in each batch being less than a preset threshold value; the processor 1001 is further configured to schedule the accelerator set 1002 to process the prompt words in the plurality of batches to obtain a processing result; wherein the prompt words in the same batch are processed in parallel on the same accelerator in the accelerator set 1002.

[0208] In a possible implementation, during processing of the plurality of batches of prompt words, if a target batch exists in the plurality of batches, the accelerator set 1002 is configured to move a target prompt word in the target batch out of the target batch to suspend processing of the target prompt word, where a length of an actual processing result of the target prompt word in the target batch is greater than a processing result prediction length corresponding to the target prompt word.

[0209] In a possible implementation, during processing of the plurality of batches of prompt words by the accelerator set 1002, the processor 1001 is further configured to monitor a length of a token generated by the accelerator set 1002 when processing the plurality of batches of prompt words; and when a target condition is monitored, the processor 1001 is further configured to send a first instruction to the accelerator set 1002, where the first instruction is used to instruct the accelerator set 1002 to move a target prompt word out of a target batch, where the target condition includes that the length of the token generated by the accelerator set 1002 when processing the target prompt word is greater than a processing result prediction length corresponding to the target prompt word.

[0210] In a possible implementation, the processor 1001 is further configured to add the target prompt word into a target queue, where the target queue is used to indicate prompt words with actual processing result lengths greater than processing result prediction lengths; and the processor 1001 is further configured to schedule the accelerator set 1002 to process prompt words in the target queue.

[0211] In a possible implementation, the accelerator set 1002 is further configured to send target data generated when processing the target prompt word to the processor 1001; and the processor 1001 is further configured to cache the target data, and send the target data to the accelerator set 1002 when scheduling the accelerator set 1002 to process the prompt words in the target queue.

[0212] In a possible implementation, the processing result prediction length corresponding to each prompt word is one of a plurality of length ranges of candidates.

[0213] In a possible implementation, the processor 1001 is specifically configured to divide prompt words with the same processing result prediction length into the same batch.

[0214] In a possible implementation, the processor 1001 is further configured to: sort the plurality of prompt words according to the processing result prediction lengths to obtain a sorting result; and divide the plurality of prompt words into the plurality of batches based on the sorting result and a number of prompt words required to be included in one batch.

[0215] In a possible implementation, the processor 1001 is further configured to: process the prompt words in the prompt word set by running the second model to obtain a processing result prediction length corresponding to each prompt word; the second model is used to predict the length of the result output by the first model when processing the prompt words, and the parameter quantity of the second model is less than that of the first model.

[0216] In a possible implementation, the processor 1001 is further configured to: schedule the accelerator set 1002 to run the second model to process the prompt words in the prompt word set, wherein the second model is used to predict the length of the result output by the first model when processing the prompt words, and the parameter quantity of the second model is less than that of the first model; and receive the processing result prediction length corresponding to each prompt word sent by the accelerator set 1002.

[0217] Referring to FIG. 11, FIG. 11 is a structural schematic diagram of a computing device provided in the present application. The computing device 1100 shown in FIG. 11 can be used to execute the model inference acceleration method provided in the present embodiment. As shown in FIG. 11, the computing device 1100 includes a bus 1102, a processor 1104, an accelerator set 1105, a memory 1106, and a communication interface 1108. The processor 1104, the memory 1106, and the communication interface 1108 communicate with each other through the bus 1102. The computing device 1100 can be a server or a terminal device. It should be understood that the number of processors, accelerator sets, and memories in the computing device 1100 is not limited in the present application. For example, the computing device 1100 can include three accelerator sets, and different accelerator sets are used to perform different tasks.

[0218] The bus 1102 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is shown in FIG. 11, but it does not mean that there is only one bus or only one type of bus. The bus 1102 can include a path for transmitting information between various components (for example, the memory 1106, the processor 1104, and the communication interface 1108) of the computing device 1100.

[0219] The processor 1104 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), among other processors.

[0220] The accelerator set 1105 includes one or more accelerators, which can be, for example, a neural network processing unit (NPU), a graphics processing unit (GPU), or a tensor processing unit (TPU).

[0221] The memory 1106 can include volatile memory, such as random access memory (RAM), and non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid-state drive (SSD), among others.

[0222] The memory 1106 stores executable program code that the processor 1104 executes to respectively implement the functions of the aforementioned training acceleration system, thereby implementing the model inference acceleration method described above. That is, the memory 1106 has instructions for executing the model inference acceleration method.

[0223] The communication interface 1108 uses a transceiver module such as, but not limited to, a network interface card, a transceiver, and the like to enable communication between the computing device 1100 and other devices or communication networks.

[0224] It should be understood that the computing device 1100 according to the present application is used to perform the model inference acceleration method as shown in FIGS. 3-9, and can correspond to the respective subjects performing the method according to the present application, and for the sake of brevity, will not be described here.

[0225] The present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device, such as a desktop computer, a notebook computer, or a smart phone.

[0226] Referring to FIG. 12, FIG. 12 is a structural diagram of a computing device cluster provided by the present application. As shown in FIG. 12, the computing device cluster includes at least one computing device 1100. The memory 1106 in one or more computing devices 1100 in the computing device cluster can store the same instructions for performing the model inference acceleration method.

[0227] In some possible implementations, the memory 1106 in one or more computing devices 1100 in the computing device cluster can also respectively store partial instructions for performing the model inference acceleration method. In other words, the combination of one or more computing devices 1100 can collectively execute the instructions for performing the model inference acceleration method.

[0228] It should be noted that the memories 1106 in different computing devices 1100 in the computing device cluster can store different instructions, respectively for performing partial functions of the training acceleration system.

[0229] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc. FIG. 13 shows a possible implementation. FIG. 13 is a structural diagram of another computing device cluster provided by the present application. As shown in FIG. 13, in the computing device cluster 1300, two computing devices 1100A and 1100B are connected through a network. Specifically, the communication interface in each computing device is connected to the network. In this type of possible implementation, the memory 1106 in the computing device 1100A stores instructions for performing partial functions. At the same time, the memory 1106 in the computing device 1100B stores instructions for performing another partial function.

[0230] It should be understood that the functions of the computing device 1100A shown in FIG. 13 can also be completed by multiple computing devices 1100. Similarly, the functions of the computing device 1100B can also be completed by multiple computing devices 1100.

[0231] It should be appreciated that the computing device 1100 or the computing device cluster 1300 in the present application can correspond to the training acceleration system in FIG. 10 of the present application, and can correspond to the respective subject performing the model inference acceleration method in FIG. 3 of the present application. Moreover, the above-mentioned and other operations and / or functions of the respective modules in the computing device 1100 or the computing device cluster 1300 are respectively for implementing the respective flows of the model inference acceleration method in FIG. 3, and for brevity, will not be repeated here.

[0232] The present application also provides a chip, which comprises a processing unit, for example, a processor, and a communication unit, for example, an input / output interface, a pin, or a circuit, etc. The processing unit can execute computer execution instructions stored in a storage unit, so that the chip in the electronic device performs the method described in the above embodiments. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0233] Specifically, please refer to FIG. 14, which is a structural schematic diagram of a chip provided by the present application. The chip can be a neural processing unit (NPU), a graphics processing unit (GPU), or a tensor processing unit (TPU). Hereinafter, the chip will be taken as an example of the NPU 1400 for introduction. The NPU 1400 is mounted as a coprocessor on a host CPU (Host CPU) and is assigned tasks by the Host CPU. The core part of the NPU is an operation circuit 1403, which extracts matrix data in the memory and performs multiplication operation under the control of a controller 1404.

[0234] In some implementations, the operation circuit 1403 internally includes a plurality of processing units (PEs). In some implementations, the operation circuit 1403 is a two-dimensional systolic array. The operation circuit 1403 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1403 is a general matrix processor.

[0235] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the data of matrix B from the weight memory 1402 and caches it on each PE of the operation circuit. The operation circuit takes the data of matrix A from the input memory 1401 and performs matrix operation with matrix B to obtain a partial result or a final result of the matrix, which is stored in the accumulator 1408.

[0236] The unified memory 1406 is used to store input data and output data. The weight data is transferred to the weight memory 1402 through the DMAC 1405. The input data is also transferred to the unified memory 1406 through the DMAC.

[0237] The BIU is the Bus Interface Unit 1410, which is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer 1409.

[0238] The BIU 1410 is used for the instruction fetch buffer 1409 to obtain instructions from the external memory, and is also used for the DMAC 1405 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0239] The DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 1406, or to transfer the weight data to the weight memory 1402, or to transfer the input data to the input memory 1401.

[0240] The vector calculation unit 1407 includes a plurality of operation processing units, which further process the output of the operation circuit 1403 as needed, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / full connection layer network calculation in neural networks, such as batch normalization, pixel-level summation, upsampling of feature planes, etc.

[0241] In some implementations, the vector computation unit 1407 can store the processed output vector to the unified memory 1406. For example, the vector computation unit 1407 can apply a linear function; or, a non-linear function to the output of the arithmetic circuit 1403, such as linear interpolation on the feature planes extracted by a convolution layer, and further such as accumulating the values of the vector to generate the activation values. In some implementations, the vector computation unit 1407 generates normalized values, pixel-wise summed values, or both. In some implementations, the processed output vector can be used as an activation input to the arithmetic circuit 1403, such as for use in a subsequent layer in a neural network.

[0242] The controller 1404 is connected to an instruction fetch buffer 1409 for storing instructions used by the controller 1404;

[0243] The unified memory 1406, the input memory 1401, the weight memory 1402, and the instruction fetch buffer 1409 are all on-chip memories. Off-chip memories are private to the NPU hardware architecture.

[0244] Any of the processors mentioned above can be a general purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above programs.

[0245] It should be understood that the chip in FIG. 14 of the present application can correspond to the training and inference acceleration system in FIG. 10 of the present application, or be deployed on the computing device 1100 or the computing device cluster 1300 of the present application. Also, the chip in FIG. 14 of the present application can correspond to the subject performing the model inference acceleration method in FIG. 3 of the present application, and the above and other operations and / or functions of the modules in the chip are respectively for implementing the corresponding processes of the model inference acceleration method in FIG. 3 of the present application, and for brevity, will not be described here. In addition, in the present application, the structure of the chip is not limited to the chip structure shown in FIG. 14, and can include more or fewer hardware structures to implement the functions of the method shown in FIG. 3.

[0246] Referring to FIG. 15, FIG. 15 is a structural schematic diagram of a computer readable storage medium provided by the present application. The present application also provides a computer readable storage medium, and in some embodiments, the method disclosed in FIG. 3 above can be implemented as computer program instructions encoded in a computer readable storage medium or other non-transitory medium or article in a machine readable format.

[0247] FIG. 15 schematically shows a conceptual partial view of an example computer readable storage medium arranged in accordance with at least some embodiments presented herein, the example computer readable storage medium including a computer program for executing a computer process on a computing device.

[0248] In one embodiment, the computer-readable storage medium 1500 is provided using a signal bearing medium 1501. The signal bearing medium 1501 can include the one or more program instructions 1502 embodied in the medium 1503, which when executed by one or more processors, can provide the functionality or portions thereof described above with respect to FIG. 3.

[0249] In some examples, the signal bearing medium 1501 can encompass a computer- readable medium 1503, such as, but not limited to, a hard disk drive, a Compact Disk (CD), a Digital Video Disk (DVD), a digital tape, memory, etc.

[0250] In some embodiments, the signal bearing medium 1501 can encompass a computer- recordable medium 1504, such as, but not limited to, memory, a Read-Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a flash memory, etc. In some embodiments, the signal bearing medium 1501 can encompass a communications medium 1505, such as, but not limited to, a digital and / or an analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communication link, a wireless communication link, etc.). Thus, for example, the signal bearing medium 1501 can be delivered by a wireless form of the communication medium 1505 (e.g., wireless communication medium complying with the IEEE 1202.X standard or other transmission protocols).

[0251] The one or more program instructions 1502 can be, for example, computer- executable instructions, or configuration-dependent logic-implementing instructions. In some examples, a computing device of a computing device can be configured to provide various operations, functions, or actions in response to the program instructions 1502 being conveyed to the computing device by one or more of the computer-readable medium 1503, the computer-recordable medium 1504, and / or the communication medium 1505.

[0252] It should be noted that the apparatus embodiments described above are merely exemplary, in which the units as described can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. In addition, the apparatus embodiments provided in the drawings of the present application, the connection relationship between the modules indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0253] Those skilled in the art can clearly understand the application by the description of the above embodiments. The application can be realized by software and necessary universal hardware, or by special hardware including special integrated circuit, special CPU, special memory, special component, etc. Generally, the functions completed by computer program can be easily realized by corresponding hardware, and the specific hardware structure for realizing the same function can be various, such as analog circuit, digital circuit or special circuit, etc. However, for the application, the software program implementation is a better embodiment. Based on this understanding, the technical solutions of the application or the part of the prior art can be embodied in the form of software product, which is stored in a readable storage medium, such as a floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, etc., and includes a plurality of instructions for making a computer device (which can be a personal computer, a training device or a network device, etc.) execute the method of each embodiment of the application.

[0254] In the above embodiments, all or part of the embodiments can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part of the embodiments can be realized in the form of a computer program product.

[0255] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that the computer can store or the data storage device such as training device, data center, etc. integrated with one or more available media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as DVD) or semiconductor media (such as solid state disk (SSD)) etc.

[0256] The above is only a specific embodiment of the application. Those skilled in the art can think of changes or replacements based on the specific embodiments provided by the application, which should be covered within the protection scope of the application.

Claims

A method for accelerating model inference, characterized in that, The method is applied to a training and acceleration system, the system comprising a processor and an accelerator set, and the method includes: The processor obtains the predicted length of the processing result corresponding to each prompt word in the prompt word set. The predicted length of the processing result is used to indicate the predicted value obtained by processing the prompt word through the first prediction model. Based on the predicted length of the processing result, the processor divides multiple prompt words in the prompt word set into multiple batches. Each batch includes at least two prompt words, and the difference between the predicted lengths of the processing results corresponding to any two prompt words in each batch is less than a preset threshold. The processor schedules the accelerator set to process the multiple batches of prompt words to obtain processing results; In this context, prompts from the same batch are processed in parallel on the same accelerator within the accelerator set. The method according to claim 1, characterized in that, The method further includes: During the process of the accelerator set processing the multiple batches of prompt words, if a target batch exists among the multiple batches, the accelerator set will remove the target prompt words from the target batch to pause the processing of the target prompt words; Wherein, the actual length of the processing result of the target prompt word in the target batch is greater than the predicted length of the processing result corresponding to the target prompt word. The method according to claim 2, characterized in that, The method further includes: During the process of the accelerator set processing the multiple batches of prompt words, the processor monitors the length of the tokens generated when the accelerator set processes the multiple batches of prompt words; When the processor detects a target situation, the processor sends a first instruction to the accelerator set, the first instruction being used to instruct the accelerator set to remove the target prompt word from the target batch; The target situation includes situations where the length of the token generated when the accelerator set processes the target prompt word is greater than the predicted length of the processing result corresponding to the target prompt word. The method according to claim 2 or 3, characterized in that, The method further includes: The processor adds the target prompt word to the target queue, which is used to indicate prompt words whose actual processing result length is greater than the predicted processing result length; The processor schedules the set of accelerators to process the prompt words in the target queue. The method according to claim 4, characterized in that, The method further includes: The accelerator set sends the target data generated when processing the target prompt word to the processor; The processor caches the target data and sends the target data to the accelerator set when scheduling the accelerator set to process the prompt words in the target queue. The method according to any one of claims 1-5, characterized in that, The predicted length of the processing result corresponding to each prompt word is one of multiple candidate length ranges. The method according to claim 6, characterized in that, The processor divides multiple prompt words in the prompt word set into multiple batches, including: The processor groups prompts whose predicted lengths are within the same range into the same batch. The method according to any one of claims 1-6, characterized in that, The processor divides multiple prompt words in the prompt word set into multiple batches, including: The processor sorts the multiple prompt words according to the predicted length of the processing results to obtain a sorting result; Based on the sorting results and the number of prompt words required in a batch, the processor divides multiple prompt words into multiple batches. The method according to any one of claims 1-8, characterized in that, The processor obtains the predicted length of the processing result for each prompt word in the prompt word set, including: The processor processes the prompt words in the prompt word set by running a second model to obtain the predicted length of the processing result corresponding to each prompt word; The second model is used to predict the length of the output of the first model when it processes the prompt words, and the number of parameters of the second model is less than the number of parameters of the first model. The method according to any one of claims 1-8, characterized in that, The processor obtains the predicted length of the processing result for each prompt word in the prompt word set, including: The processor schedules the accelerator set to run a second model to process the prompt words in the prompt word set, wherein the second model is used to predict the length of the output result when the first model processes the prompt words, and the number of parameters of the second model is less than the number of parameters of the first model; The processor receives the predicted length of the processing result corresponding to each prompt word sent by the accelerator set. A training acceleration system, characterized in that, The system includes a collection of processors and accelerators; The processor is used to obtain the predicted length of the processing result corresponding to each prompt word in the prompt word set, and the predicted length of the processing result is used to indicate the predicted value obtained by processing the prompt word through the length of the processing result obtained by the first prediction model. The processor is further configured to divide multiple prompt words in the prompt word set into multiple batches based on the length predicted by the processing result, wherein each batch includes at least two prompt words, and the difference between the predicted lengths of the processing results corresponding to any two prompt words in each batch is less than a preset threshold. The processor is also configured to schedule the accelerator set to process the multiple batches of prompt words to obtain processing results; In this context, prompts from the same batch are processed in parallel on the same accelerator within the accelerator set. The system according to claim 11 is characterized in that, During the processing of the multiple batches of prompt words, if a target batch exists among the multiple batches, the accelerator set is used to remove the target prompt words from the target batch to pause the processing of the target prompt words; Wherein, the actual length of the processing result of the target prompt word in the target batch is greater than the predicted length of the processing result corresponding to the target prompt word. The system according to claim 12 is characterized in that, During the process of the accelerator set processing the multiple batches of prompt words, the processor is also used to monitor the length of the tokens generated when the accelerator set processes the multiple batches of prompt words; When a target is detected, the processor is also configured to send a first instruction to the accelerator set, the first instruction being configured to instruct the accelerator set to remove the target cue word from the target batch; The target situation includes situations where the length of the token generated when the accelerator set processes the target prompt word is greater than the predicted length of the processing result corresponding to the target prompt word. The system according to claim 12 or 13 is characterized in that, The processor is further configured to add the target prompt word to a target queue, the target queue being used to indicate prompt words whose actual processing result length is greater than the predicted processing result length; The processor is also used to schedule the accelerator set to process the prompt words in the target queue. The system according to claim 14 is characterized in that, The accelerator set is also used to send target data generated when processing the target prompt words to the processor; The processor is also configured to cache the target data and send the target data to the accelerator set when scheduling the accelerator set to process the prompts in the target queue. The system according to any one of claims 11-15 is characterized in that, The predicted length of the processing result corresponding to each prompt word is one of multiple candidate length ranges. The system according to claim 16 is characterized in that, The processor is specifically used to group prompts whose predicted lengths are within the same range into the same batch. The system according to any one of claims 11-16 is characterized in that, The processor is also used for: The multiple prompt words are sorted according to their predicted lengths based on the processing results to obtain the sorting results; Based on the sorting results and the number of prompt words required to be included in a batch, multiple prompt words are divided into multiple batches. The system according to any one of claims 11-18 is characterized in that, The processor is also used for: By running a second model to process the prompt words in the prompt word set, the predicted length of the processing result corresponding to each prompt word is obtained; The second model is used to predict the length of the output of the first model when it processes the prompt words, and the number of parameters of the second model is less than the number of parameters of the first model. The system according to any one of claims 11-18 is characterized in that, The processor is also used for: The accelerator set is scheduled to run a second model to process the prompt words in the prompt word set, wherein the second model is used to predict the length of the output result when the first model processes the prompt words, and the number of parameters of the second model is less than the number of parameters of the first model; Receive the predicted length of the processing result corresponding to each prompt word sent by the accelerator set. A computing device, characterized in that, The device includes a memory, an accelerator set, and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the computing device performs the method as described in any one of claims 1 to 10. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor, an accelerator set and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the operational steps of the method as described in any one of claims 1 to 10. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 10. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 10.

Images