Model inference acceleration method and related device

Abstract

A model inference acceleration method, used for improving the efficiency of model inference. In the method, two models having different parameter quantities are deployed separately on different accelerator sets, a first model having a smaller parameter quantity is used for processing an inputted prompt word, so as to generate tokens one by one for conversion into an output result, and the second model having a larger parameter quantity is used for verifying the accuracy of the tokens generated by the first model and correcting an erroneous token, so as to ensure the accuracy of the finally obtained output result. The time required for the first model to process the prompt word and generate the tokens one by one is short, and the time required for the second model to verify the tokens generated by the first model is far shorter than the time required for the second model to generate the tokens. Therefore, the cooperation mode of the first model and the second model can shorten the model inference time while ensuring the accuracy of an output result.

Description

A method and related apparatus for accelerating model inference

[0001] This application claims priority to Chinese Patent Application No. 202410674442.8, filed on May 27, 2024, entitled "A Model Inference Acceleration Method and Related Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method and apparatus for accelerating model inference. Background Technology

[0003] Large Language Models (LLMs) are deep learning models trained on massive amounts of text data. They can not only generate natural language text, but also deeply understand the meaning of text and handle various natural language tasks, such as text summarization, intelligent question answering, and text translation.

[0004] With the rapid development and widespread application of large language models, the computational demands of their inference phase are constantly increasing, making inference acceleration methods crucial. Currently, the importance of inference acceleration methods is primarily reflected in real-time requirements. Specifically, in most application scenarios (such as human-computer dialogue or content recommendation), large language models are typically required to respond in real time. However, the inference speed of large language models is often slow; inference acceleration methods can reduce the inference time of large language models, thus meeting real-time requirements.

[0005] However, solutions for accelerating large language model inference in related technologies typically include methods such as model compression, model quantization, and model pruning. These methods essentially reduce the model structure, which can lead to problems such as decreased model accuracy and increased complexity, affecting the model's practical application performance. Summary of the Invention

[0006] This application provides a method for accelerating model inference, which can improve the efficiency of model inference and ensure the accuracy of the output results.

[0007] Firstly, a method for accelerating model inference is provided, applied to the model inference process. This method is applied to a training and inference acceleration system, which includes a processor, a first set of accelerators, and a second set of accelerators. The processor can be, for example, a central processing unit (CPU) or other hardware with processing capabilities. The inference acceleration method includes: firstly, the processor acquiring a set of prompt words, which includes at least one prompt word, each used to describe the task to be processed. The prompt words in the set can describe the task in text form, and the task to be processed can be any type of natural language processing task.

[0008] The processor then schedules a first set of accelerators and a second set of accelerators to process the set of prompt words, wherein the first set of accelerators includes more resources than the second set of accelerators. Specifically, both the first and second sets of accelerators include computing resources for accelerating AI operations. Furthermore, the first set of accelerators includes more computing resources than the second set of accelerators.

[0009] Next, the first and second accelerator sets process the prompt word set to obtain the output result.

[0010] The first accelerator set processes the prompt words in the prompt word set and generates lexical tokens by running a first model. The tokens generated by the first accelerator set are used to convert the output results. The second accelerator set verifies the accuracy of the tokens generated by the first accelerator set and corrects erroneous tokens by running a second model. Furthermore, the first model has fewer parameters than the second model, but both the first and second models can be used to process the same task.

[0011] In summary, after the first accelerator set generates a certain number of tokens using the first model, it transmits these tokens to the second accelerator set. The second accelerator set then verifies the tokens by running the second model. Upon successful verification, the second accelerator set sends the verification results back to the first accelerator set, allowing the first accelerator set to generate new tokens based on these results. Furthermore, the new tokens generated by the first accelerator set based on the verification results are continuously transmitted to the second accelerator set for verification, until all tokens generated by the first accelerator set using the first model have been verified, and the first accelerator set no longer generates new tokens using the first model.

[0012] In this scheme, two models with different parameter sizes are deployed separately on different accelerator sets. The first model, with fewer parameters, processes the input prompts to generate tokens for the output. The second model, with more parameters, verifies the accuracy of the tokens generated by the first model and corrects erroneous words, ensuring the accuracy of the final output. Since the first model has significantly fewer parameters than the second model, it takes less time to process the prompts and generate tokens. Furthermore, the time required for the second model to verify the words generated by the first model is much shorter than the time required for the second model to generate words on its own. Therefore, this collaborative approach of using the first and second models can shorten the inference time while ensuring the accuracy of the output. In addition, since the second model requires more computational resources than the first model, deploying them separately on accelerator sets with different accelerator resources ensures that the second model can be allocated more accelerator resources within limited accelerator resources. This guarantees high utilization of accelerator resources across different accelerator sets, avoiding resource waste and further improving the efficiency of model inference.

[0013] In one possible implementation, the first accelerator set and the second accelerator set process the prompt word set, including: the first accelerator set first uses a first model to process the target prompt word in the prompt word set to generate N tokens, where N is an integer greater than 1. Then, the second accelerator set uses a second model to verify the correctness of each of the N tokens in parallel and corrects erroneous tokens to obtain a verification result. Next, based on the verification result, the first accelerator set uses the first model to continue processing the target prompt word to generate a new token. Here, the target prompt word is any prompt word in the prompt word set.

[0014] In other words, in this scheme, multiple tokens are generated one by one through the first model with higher processing efficiency, and then the multiple tokens are passed to the second model for parallel verification. There is no need for the second model to generate tokens one by one, thereby saving token generation time as much as possible and improving the processing efficiency of prompt words.

[0015] In one possible implementation, N is a preset value, and N is determined based on the difference in prompt word processing results between the first model and the second model. Specifically, for the same prompt word, if the difference between the prompt word processing results obtained by the first model and the prompt word processing results obtained by the second model is small, it means that the output results of the first model and the output results of the second model will be closer, that is, the number of tokens output by the first model that need to be corrected will be smaller. Therefore, N can be set to a larger value.

[0016] In this scheme, the number of tokens that need to be verified generated by the first model in a single round is set based on the difference in the prompt word processing results between the first model and the second model. This allows for a reasonable setting of the number of tokens, ensuring that the N tokens generated by the first model in a single round do not contain too many erroneous tokens, thereby guaranteeing the efficiency of prompt word processing.

[0017] In one possible implementation, the ratio of resources included in the second accelerator set to those included in the first accelerator set is determined based on the duration of token generation by the first model and the duration of token verification by the second model.

[0018] Specifically, in order to determine the accelerator resources included in the first and second accelerator sets, the processor first determines the first duration required for the first model to generate a single token, and the second duration required for the second model to verify a single token. The first and second durations are obtained by running the first and second models respectively on the same accelerator resources.

[0019] The processor then determines the ratio between the target duration and the second duration, where the target duration is the product of N and the first duration. That is, since the first model generates N tokens serially, multiplying the first duration of generating a single token by N gives the total duration of the first model generating N tokens serially.

[0020] In this way, based on the duration ratio, the processor can determine the ratio between the amount of resources included in the first accelerator set and the amount of resources included in the second accelerator set. For example, the calculated duration ratio can be used as the ratio between the amount of resources included in the first accelerator set and the amount of resources included in the second accelerator set. That is, the processor actually determines the ratio between the amount of resources included in the first accelerator set and the amount of resources included in the second accelerator set based on the first duration and the second duration.

[0021] In this scheme, since the second model verifies multiple tokens in parallel, the resource ratio between the first accelerator set and the second accelerator set is set based on the ratio between the target duration and the second duration. This ensures that the duration for the first accelerator set to serially generate N tokens is close to the duration for the second accelerator set to verify N tokens in parallel. This ensures that both the first and second accelerator sets are continuously in operation, thereby maximizing the processing efficiency of prompt words with limited resources.

[0022] In one possible implementation, the training and acceleration system further includes a third set of accelerators. Before the first and second accelerator sets process the cue word set, the method further includes: performing feature extraction on the cue words in the cue word set through the third accelerator set to obtain features of the cue words, the features of the cue words being used as input to the first model so that the first model generates a token.

[0023] In other words, after the third accelerator set performs feature extraction on the prompt words, it can pass the obtained prompt word features to the first model on the first accelerator set, and then the first model can generate tokens based on the prompt word features.

[0024] Because the processing resources consumed during the pre-filling phase are higher than those consumed during the decoding phase, this scheme allocates more accelerator resources to the third accelerator set used for the pre-filling phase than the sum of the accelerator resources in the first and second accelerator sets used for the decoding phase. This ensures that more accelerator resources are allocated to the pre-filling phase, maximizing accelerator resource utilization and achieving the highest possible data processing efficiency with limited resources.

[0025] In one possible implementation, at least one prompt word specifically refers to multiple prompt words. The processor performs segmentation and integration processing on the prompt words in the prompt word set and then transmits the processed prompt words to a third accelerator set in batches. The segmentation and integration processing involves splitting some prompt words so that the undisturbed prompt words and the segmented prompt words constitute a batch. This divides all prompt words in the prompt word set into multiple batches, with the number of tokens included in each batch not exceeding the target number. In other words, the total number of tokens included in each batch of prompt words transmitted by the processor to the third accelerator set is not greater than the target number. Thus, the third accelerator set processes each batch of prompt words transmitted by the processor in parallel.

[0026] In one possible implementation, the processor may first add the prompts from the prompt set to the first storage space in sequence.

[0027] When the total number of tokens included in the hints already added to the first storage space exceeds the target number, the processor performs a split on the last hint added to the first storage space to ensure that the total number of tokens included in the hints already added to the first storage space does not exceed the target number. The last hint is split into a first part and a second part. The first part remains in the first storage space, while the second part is added to the second storage space. The second storage space is the computation pool that needs to be processed after the first storage space. Furthermore, the hints in the second storage space are added one by one from the hint set.

[0028] Secondly, the processor passes the prompts in the first storage space as a batch to the third accelerator set, so that the third accelerator set performs feature extraction on the prompts in the first storage space in parallel.

[0029] The third accelerator set performs feature extraction on the prompts in the first storage space in parallel, and after the prompts in the first storage space are processed, the third accelerator set performs feature extraction on the prompts in the second storage space in parallel.

[0030] In other words, the prompts contained in the same storage space can be regarded as a batch of prompts processed in parallel. When processing prompts through the third accelerator set, each batch of prompts is processed sequentially. Furthermore, the third accelerator set processes all prompts in the same batch in parallel, thus ensuring that the number of tokens processed in parallel by the third accelerator set each time is relatively fixed. This ensures that the computing resources in the accelerator set are used as much as possible during each parallel processing of prompts, avoiding waste of computing resources and improving the efficiency of the accelerator set in processing prompts.

[0031] In one possible implementation, the target number is related to the amount of resources included in the third accelerator set; the resource utilization of the third accelerator set reaches its maximum when the total number of tokens processed in a single run by the third accelerator set is the target number. That is, the target number is actually the maximum number of tokens that the third accelerator set can process in parallel in a single run.

[0032] In one possible implementation, the accelerators included in the first accelerator set and the second accelerator set can both be neural network processing units (NPUs), graphics processing units (GPUs), or tensor processing units (TPUs), and the number of accelerators included in the second accelerator set is greater than the number of accelerators included in the first accelerator set.

[0033] Secondly, a training and push acceleration system is provided, comprising: a processor, a first set of accelerators, and a second set of accelerators; the processor is configured to acquire a set of prompt words, the set of prompt words including at least one prompt word, the at least one prompt word being used to describe the task to be processed; the processor is further configured to schedule the first set of accelerators and the second set of accelerators to process the set of prompt words, wherein the second set of accelerators includes more resources than the first set of accelerators; the first set of accelerators and the second set of accelerators are used to process the set of prompt words and obtain output results; wherein the first set of accelerators processes the set of prompt words and generates tokens by running a first model, and the tokens are used to convert into output results, and the second set of accelerators verifies the accuracy of the tokens and corrects erroneous tokens by running a second model, wherein the number of parameters of the first model is less than the number of parameters of the second model.

[0034] In one possible implementation, a first accelerator set is used to process target prompt words in the prompt word set using a first model to generate N tokens, where N is an integer greater than 1; a second accelerator set is used to verify the correctness of each of the N tokens in parallel using a second model and correct erroneous tokens to obtain a verification result; based on the verification result, the first accelerator set is used to continue processing target prompt words using the first model to generate new tokens.

[0035] In one possible implementation, N is a preset value, and N is determined based on the difference in prompt word processing results between the first model and the second model.

[0036] In one possible implementation, the processor is further configured to determine a first duration required for the first model to generate a single token and a second duration required for the second model to verify a single token, the first duration and the second duration being obtained by running the first model and the second model respectively on the same accelerator resources; the processor is further configured to determine, based on the first duration and the second duration, the ratio between the amount of resources included in the first accelerator set and the amount of resources included in the second accelerator set.

[0037] In one possible implementation, the system further includes a third set of accelerators, which, before the first and second accelerator sets process the cue word set, is used to: perform feature extraction on the cue words in the cue word set to obtain features of the cue words, the features of the cue words being used as input to the first model so that the first model generates a token.

[0038] In one possible implementation, at least one prompt word specifically refers to multiple prompt words; the processor is further configured to perform segmentation and integration on the prompt words in the prompt word set, and to pass the integrated prompt words to the third accelerator set in batches; wherein the total number of tokens included in each batch of prompt words passed by the processor to the third accelerator set is not greater than the target number.

[0039] In one possible implementation, the processor is further configured to: sequentially add the prompts from the prompt set to a first storage space; when the total number of tokens included in the prompts already added to the first storage space is greater than the target number, perform segmentation on the last prompt added to the first storage space so that the total number of tokens included in the prompts already added to the first storage space is the target number, wherein the last prompt is segmented into a first part and a second part, the first part is retained in the first storage space, and the second part is added to the second storage space; and transmit the prompts in the first storage space as a batch to a third accelerator set so that the third accelerator set performs feature extraction on the prompts in the first storage space in parallel.

[0040] In one possible implementation, the number of targets is related to the amount of resources included in the third accelerator set.

[0041] Thirdly, a training and acceleration system is provided, comprising: a processor, a first set of accelerators and a second set of accelerators, and a memory; the memory is used to store computer instructions, which, when executed by the processor, cause the training and acceleration system to perform the methods described above.

[0042] Fourthly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, cause the computer to perform the methods of any of the above aspects.

[0043] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, enable the computer to perform the methods described above.

[0044] In a sixth aspect, a chip system is provided, the chip system including a processor and a communication interface for communicating with a module other than the chip shown, the processor for running computer programs or instructions such that an apparatus on which the chip system is mounted can perform the methods of any of the above aspects.

[0045] In a seventh aspect, a computing device is provided, the computing device including a memory, a first accelerator set and a second accelerator set and a processor, the memory storing code, the processor being configured to execute the code, and when the code is executed, the computing device being used to implement the operational steps of the method of any of the above aspects.

[0046] Eighthly, a computing device cluster is provided, comprising at least one computing device, wherein any one computing device is used to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects. Alternatively, some or all of the computing devices are used together to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects.

[0047] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0048] Figure 1 is a schematic diagram of a system architecture provided in this application;

[0049] Figure 2 is a flowchart illustrating a model inference acceleration method provided in this application;

[0050] Figure 3 is a schematic diagram of the process of completing prompt word processing by the cooperation of a first model and a second model provided in this application;

[0051] Figure 4 is a schematic diagram of a first model generating a token and a second model verifying the token provided in this application;

[0052] Figure 5 is a schematic diagram illustrating the waste of computing resources caused by using a small model to generate a token and a large model to verify the token, as provided in this application.

[0053] Figure 6 is a schematic diagram illustrating the waste of computing resources caused by using a small model to generate tokens and a large model to verify tokens, as provided in the application.

[0054] Figure 7 is a schematic diagram comparing the execution of the pre-filling stage and the decoding stage provided in this application;

[0055] Figure 8 is a schematic diagram illustrating the waste of computing resources when batch processing prompts provided in this application;

[0056] Figure 9A is a flowchart illustrating a method for determining prompt words that need to be processed in parallel for an accelerator set, as provided in this application.

[0057] Figure 9B is a schematic diagram of a framework for accelerating model inference provided in this application;

[0058] Figure 10 is a schematic diagram of a training acceleration system provided in this application;

[0059] Figure 11 is a schematic diagram of the structure of a computing device provided in this application;

[0060] Figure 12 is a schematic diagram of the structure of a computing device cluster provided in this application;

[0061] Figure 13 is a schematic diagram of another computing device cluster provided in this application;

[0062] Figure 14 is a schematic diagram of the structure of a chip provided in this application;

[0063] Figure 15 is a schematic diagram of the structure of a computer-readable storage medium provided in this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will recognize that, with the emergence of new application scenarios, the technical solutions provided by this application are also applicable to similar technical problems.

[0065] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0066] To facilitate understanding, some technical terms used in this application will be introduced below.

[0067] (1) Large Language Model (LLM)

[0068] Large language models are deep learning models trained on massive amounts of text data that can generate natural language text or understand the meaning of language text. Large language models can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0069] Specifically, large language models are a technology that has emerged in recent years. Because large language models undergo meticulous data engineering and training processes, their parameters have learned a wealth of existing natural language processing knowledge. This knowledge can now replace humans in many language-related tasks, such as having large language models write code or perform text summarization.

[0070] (2) Prompt words

[0071] Cue words originated as an input format designed by researchers for downstream tasks. Their purpose is to help pre-trained models "recall" what they "learned" during pre-training, hence the name "cue words." For large language models, cue words are the user's input, instructing the model on the task to be performed. Cue words can be a simple question, a longer text, or a set of instructions, depending on the user's specific needs. Generally, cue words are short text strings that provide context and task-related information to help the model better understand the requirements and generate the correct output. For example, in question-answering tasks, cue words might contain a description of the question or topic to help the large language model generate the correct answer. Furthermore, cue words are often designed by humans to help the large language model better understand specific tasks or domains.

[0072] In this way, when generating content, the large language model first processes the prompt words and then outputs content based on its understanding of those prompt words. The working principle of the large language model is to predict the probability of the next word appearing based on the preceding context of the user input, thereby generating the following text word by word. Therefore, the different prompt words input by the user directly affect the quality of the large language model's output. In some cases, even if the difference in the user input prompt words is only a few words, the large language model may still generate significantly different content.

[0073] (3) Neural Network

[0074] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0075] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0076] (4) Transformer network

[0077] The Transformer network is a neural network model based on the self-attention mechanism, used to process sequential data. Compared to traditional recurrent neural network models, the Transformer network has better parallel performance and shorter training time, and therefore has been widely used in the field of natural language processing.

[0078] The Transformer network comprises multiple encoder and decoder layers, with each layer consisting of multiple attention mechanism modules and feedforward neural network modules. The encoder encodes the input sequence into a high-dimensional feature vector representation, while the decoder decodes this vector representation into the target sequence. The Transformer network also utilizes techniques such as residual connections and layer normalization to accelerate model convergence and improve model performance.

[0079] Specifically, the core of the Transformer network is the self-attention mechanism, which assigns a weight to each position in the input sequence and then outputs these weighted position vectors. Generally, the computation process of the self-attention mechanism includes three steps: 1. Calculate the attention weights: Calculate the attention weights between each position and other positions, i.e., the importance of each position to the other positions; 2. Calculate the weighted sum: Multiply each position vector by its attention weight and then sum them to obtain a weighted sum vector; 3. Perform a linear transformation: Apply a linear transformation to the weighted sum vector to obtain the final output vector.

[0080] (5) Token

[0081] A lexical is the smallest semantic unit in natural language processing, typically derived from the decomposition of long texts such as words, sentences, paragraphs, and articles. Generally, lexical units can be composed of letters, words, or symbols; for example, they can be composed of one or more letters or one or more words. Simply put, a lexical unit can be understood as a "fragment" of a word or phrase—the smallest indivisible semantic unit. For example, "waterfall" can be broken down into two lexical units: "water" and "fall." Additionally, punctuation marks can also be decomposed into lexical units because they also affect the semantic understanding of the entire text. For example, "I don't know." can be decomposed into five lexical units: "I," "don," "'t," "know," and "...".

[0082] .

[0083] (6) Key-Value Cache (KV Cache)

[0084] When using Transformer networks for natural language processing tasks, they typically employ a self-attention mechanism to process the input sequence. In this mechanism, the Transformer network generates a corresponding key (K) vector, value (V) vector, and query (Q) vector for each token in the input sequence. Further, the Transformer network calculates the degree of matching between each query vector and all key vectors, usually achieved through a dot product. Then, using the matching degree between query and key vectors as weights, it calculates a weighted sum of all value vectors to obtain the final result.

[0085] When the Transformer network processes the input sequence, the key vector generated by the Transformer network for each word in the input sequence can be stored in a single matrix, and the value vector generated by the Transformer network for each word in the input sequence can be stored in another matrix. Therefore, KV Cache refers to the key matrix and value matrix generated by the Transformer network for the words in the input sequence, and these key matrices and value matrices are cached.

[0086] Currently, in most application scenarios (such as human-computer dialogue or content recommendation), large language models are typically required to respond in real time. Due to the large number of parameters in large language models, their inference speed is often slow. Inference acceleration methods can reduce the inference time of large language models, thus meeting real-time requirements.

[0087] However, solutions for accelerating large language model inference in related technologies typically include methods such as model compression, model quantization, and model pruning. These methods essentially reduce the model structure, which can lead to problems such as decreased model accuracy and increased complexity, affecting the model's practical application performance.

[0088] Based on this, this application provides a method for accelerating model inference. Two models with different parameter counts are deployed separately on different accelerator sets. The first model, with fewer parameters, processes the input prompts to generate tokens for the output. The second model, with more parameters, verifies the accuracy of the tokens generated by the first model and corrects erroneous words, ensuring the accuracy of the final output. Since the first model has significantly fewer parameters than the second model, it takes less time to process the prompts and generate tokens. Furthermore, the time required for the second model to verify the words generated by the first model is much shorter than the time required for the second model to generate words itself. Therefore, this method of combining the first and second models can shorten the inference time while ensuring the accuracy of the output. In addition, since the second model requires more computational resources than the first model, deploying the first and second models separately on accelerator sets with different accelerator resources ensures that the second model can be allocated more accelerator resources within limited accelerator resources. This guarantees high utilization of accelerator resources in different accelerator sets, avoids resource waste, and further improves the efficiency of model inference.

[0089] Please refer to Figure 1, which is a schematic diagram of a system architecture provided in this application. As shown in Figure 1, in this system architecture, the execution device 10 can be implemented by a single physical host (computing device) or multiple physical hosts (computing device cluster). The execution device 10 includes an accelerator set 101, an accelerator set 102, and a processor 103. Accelerator set 101 is used to run model 1011, and accelerator set 102 is used to run model 1021. The processor 103 is used to obtain task requests from clients (e.g., requests to process prompts) and schedules accelerator sets 101 and 102 to process prompts based on the task requests from clients. Furthermore, the number of parameters in model 1011 is less than the number of parameters in model 1021. The resources included in accelerator set 101 are less than the resources included in accelerator set 102.

[0090] Optionally, the execution device 10 can be used in conjunction with other computing devices, such as data storage devices, load balancers, etc.; the execution device 10 can be deployed on a single physical site or distributed across multiple physical sites.

[0091] In addition, the system architecture also includes a data storage system 11, which is used to store data such as prompt words, KV cache, or program code.

[0092] Optionally, for persistent data storage, the data storage system 11 can be located external to the execution device 10 and exchange data with the execution device 10 via a network. Alternatively, if the execution device 10 is a physical host, the data storage system 11 can also be located internally to the execution device 10, such as exchanging data with the processor via a bus. In this case, the data storage system 11 behaves as a hard disk. With the data storage system 11, the execution device 10 can use the data in the data storage system 11 (such as a KV cache) or call the program code in the data storage system to implement the model inference acceleration method provided in this application.

[0093] Optionally, users can interact with execution device 10 using their respective local devices. For example, a client 121 is deployed on local device 12, and users interact with the execution device through client 121 on local device 12. Local device 12 can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, laptop, or smart car.

[0094] Local device 12 can interact with execution device 10 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0095] In one implementation, the execution device 10 is used to implement the model inference acceleration method provided in this application, thereby obtaining the corresponding output results.

[0096] Optionally, during the process of implementing the model inference acceleration method by the execution device 10, the local device 12 can provide the execution device 10 with task requests or prompts, so that the execution device 10 can process the prompts through the model to complete the task. Furthermore, after the execution device 10 executes the model inference acceleration method and obtains the output result, it can feed the output result back to the local device 12.

[0097] Optionally, the prompts required when the execution device 10 executes the model inference acceleration method can also be pre-stored in the data storage system 11, and the execution device 10 obtains the prompts to be processed by reading the prompt data from the data storage system 11.

[0098] Please refer to Figure 2, which is a flowchart illustrating a model inference acceleration method provided in this application. As shown in Figure 2, the model inference acceleration method provided in this application can be applied to a training and inference acceleration system, which includes a processor, a first set of accelerators, and a second set of accelerators. The processor, for example, can be hardware with computing capabilities such as a CPU, used to receive user task requests and schedule the first and second sets of accelerators to execute tasks according to the task requests. The first and second sets of accelerators can be sets including one or more accelerators. Specifically, the accelerators can be hardware or software used for training or inference of artificial intelligence models, such as graphics processing units (GPUs), tensor processing units (TPUs), or neural network processing units (NPUs).

[0099] Specifically, the model inference acceleration method includes the following steps 201-208.

[0100] Step 201: The processor obtains a set of prompt words, which includes at least one prompt word, and each prompt word is used to describe the task to be processed.

[0101] Generally, when a user uses a model to perform a task, they often need to provide corresponding prompts so that the model can output the corresponding task result based on the input prompts. In this application, at least one prompt included in the prompt set can be a prompt provided by the user, or a prompt generated based on the user's expectation that the model will complete the task. For example, when a user sends a task request to the execution device through a client, the execution device executes the prompt processing according to the task request, thereby generating a corresponding prompt set, where each prompt in the prompt set corresponds to a task request. This application does not specifically limit the source of the prompt set.

[0102] Furthermore, the prompts in the prompt set can describe the task to be processed in text form. For example, if the task is a question-and-answer task, the prompt could be: "What is the ending of character XX in the novel XX?" Another example is a translation task, where the prompt could be: "Please translate the following sentence into English: XXX." Yet another example is a text generation task, where the prompt could be: "Refer to the following example XXX and write a story about XXX." Finally, if the task is a sentiment analysis task, the prompt could be: "Analyze the following comment content XXX and output whether the customer's evaluation is approval or disapproval."

[0103] In general, the cue words included in the cue word set can be used to describe any type of natural language processing task, and this application does not make any specific limitations on this.

[0104] Step 202: The processor schedules the first accelerator set and the second accelerator set to process the cue word set.

[0105] Specifically, after acquiring the prompt word set, the processor can send instructions to the first accelerator set and the second accelerator set respectively to schedule the first accelerator set and the second accelerator set to process the prompt word set. Furthermore, during the process of the processor scheduling the first accelerator set and the second accelerator set (i.e., the process of the processor sending instructions to the first accelerator set and the second accelerator set to schedule the processing of the prompt word set), the processor can also pass the acquired prompt word set to the first accelerator set and the second accelerator set; or, the processor can only pass the prompt word set to the first accelerator set.

[0106] Step 203: Based on the prompt words in the prompt word set, the first accelerator set generates a token by running the first model.

[0107] After obtaining the set of prompt words, the first model running on the first accelerator set can process the prompt words in the set. Specifically, when processing the prompt words, the first model generates new tokens one by one based on the tokens corresponding to the input prompt words, until a terminator is generated or a sufficient number of tokens have been generated. Then, the generated tokens are converted into an output result consisting of text. That is, the tokens generated by the first model are used to convert the model into its final output result.

[0108] It's important to note that during the token generation process using the first model in the first accelerator set, each new token generated by the first model depends on previously generated tokens. That is, after generating a token, the first model uses all previously generated tokens as input, reprocesses the prompt and all existing tokens, and then generates a new token. In other words, the first model continuously adds newly generated tokens to the existing input to form new input, thus continuing to generate new tokens.

[0109] Step 204: The first accelerator set transmits the token generated by the first model to the second accelerator set.

[0110] Since the first model runs on the first accelerator set, the token generated by the first model will also be located on the first accelerator set. At this point, the token generated by the first model on the first accelerator set can be transferred to the second accelerator set so that the second model running on the second accelerator set can verify the generated token.

[0111] Step 205: The second accelerator set verifies the accuracy of the tokens generated by the first model and corrects erroneous tokens by running the second model, thus obtaining the verification result.

[0112] In this application, the number of parameters in the first model is less than that in the second model, but both the first and second models can be used to process the same task. Since the number of parameters in the second model is greater than that in the first model, the accuracy of the second model in processing prompt words is usually higher than that of the first model. Therefore, the second model can be used to verify the accuracy of the token generated by the first model.

[0113] For example, both the first and second models are large language models, but the second model has more network layers than the first model; for instance, the second model has 30 network layers, while the first model has 10. Furthermore, the first and second models can be different types of models; no specific restrictions are placed here.

[0114] Specifically, for any token generated by the first model, it is generated based on the prompt word and previously generated tokens. Therefore, when the second model verifies the token generated by the first model, it can overlay the previously generated token with the prompt word and input it into the second model to determine if the token generated by the second model is the same as the token to be verified. If the token generated by the second model is the same as the token to be verified, it means that the token generated by the first model is correct. If the token generated by the second model is different from the token to be verified, it means that the token generated by the first model is incorrect, and the token generated by the second model should be considered correct.

[0115] In other words, after the second model has verified the token generated by the first model, if the token generated by the first model is correct, the verification result can indicate that all tokens generated by the first model are correct; if there is an incorrect token generated by the first model, the verification result can indicate which token generated by the first model is incorrect and indicate the corresponding correct token.

[0116] In this application, the first model runs on a first set of accelerators, and the second model runs on a second set of accelerators, with the second set containing more resources than the first set. Since the second model has more parameters than the first model, the computational resources required for the second model are also higher than those required for the first model. Therefore, the resources in the second set running the second model can be set to be more than those in the first set running the first model.

[0117] For example, the accelerators included in the first accelerator set and the second accelerator set are all neural network processing units (NPUs), graphics processing units (GPUs), or tensor processing units (TPUs).

[0118] It is understood that the second accelerator set includes more resources than the first accelerator set. Specifically, the computing resources (computing power resources) included in the second accelerator set may be higher than those included in the first accelerator set. For example, the number of accelerators included in the second accelerator set is greater than the number of accelerators included in the first accelerator set. That is, both the first and second accelerator sets include the same type of accelerators, such as both being NPUs, both being GPUs, or both being TPUs, and these accelerators have the same model parameters (i.e., the accelerators have the same processing resources) to ensure collaborative data processing. Thus, when the individual accelerators included in the first and second accelerator sets have the same processing resources (e.g., each accelerator is an accelerator card of the same type and specifications), the number of accelerators (accelerator cards) in the second accelerator set can be set to be greater than the number of accelerators in the first accelerator set. For example, the first accelerator set includes 4 NPUs, while the second accelerator set includes 10 NPUs.

[0119] Of course, the accelerators included in the first accelerator set and the accelerators included in the second accelerator set can be different types of accelerators, as long as the computing resources included in the second accelerator set are higher than those included in the first accelerator set, so that the first accelerator set and the second accelerator set can cooperate with each other to complete the operation of the first model and the second model with higher efficiency. This embodiment does not make specific limitations in this regard.

[0120] Step 206: The second accelerator set transmits the verification results obtained based on the second model to the first accelerator set.

[0121] After the token verification process is performed based on the second model, the verification result obtained based on the second model can be passed to the first accelerator set running the first model, thereby prompting the first model to continue generating new tokens.

[0122] Step 207: Based on the verification results, the first accelerator set continues to process the prompt words through the first model to generate new tokens until all generated tokens have been verified.

[0123] Specifically, if the verification result indicates that all tokens generated by the first model are correct, then the first model can continue to generate new tokens based on the already generated tokens, and continue to transmit the newly generated tokens to the second model for verification. If the verification result indicates that the tokens generated by the first model are incorrect, then the first model can replace the previously generated incorrect tokens with the correct tokens indicated by the verification result, and continue to generate new tokens based on the replaced tokens, and continue to transmit the newly generated tokens to the second model for verification.

[0124] In summary, after the first model generates a certain number of tokens, it transmits these tokens to the second model for verification. The second model then sends the verification results back to the first model, allowing the first model to generate new tokens based on these results. Furthermore, the new tokens generated by the first model based on the verification results are continuously transmitted to the second model for verification, until all tokens generated by the first model have been verified, and the first model no longer generates new tokens (i.e., the first model generates a terminator based on the already generated tokens or the number of tokens generated by the first model reaches a certain threshold).

[0125] Step 208: Based on all verified tokens, the first accelerator set obtains the output result.

[0126] After all tokens generated by the first model have been verified, a token sequence that has passed the second model's verification is obtained. This token sequence is indistinguishable from the tokens generated by the second model based on the input prompts. Therefore, by converting the verified token sequence into a text sequence, the output result can be obtained.

[0127] The above describes the process of generating tokens using the first model and verifying them using the second model. Since the first model continuously generates tokens during the actual processing of prompt words, the following section will detail the specific workflow of the first and second models working together to complete the prompt word processing.

[0128] For example, please refer to Figure 3, which is a flowchart illustrating the process of a first model and a second model working together to complete prompt word processing according to this application. As shown in Figure 3, the process of the first model and the second model working together to complete prompt word processing includes the following steps 301-310.

[0129] Step 301: The first accelerator set uses the first model to process the target prompt words in the prompt word set and continuously generate tokens.

[0130] The target prompt word can be any word from the prompt word set. When the first accelerator set processes the target prompt word using the first model, tokens are continuously generated one by one, and the generated tokens are used as input to the first model to regenerate new tokens. Therefore, the tokens generated by the first model are generated serially; a subsequent token cannot be generated if the previous token has not been generated.

[0131] For example, if the first model generates 3 tokens, the first model first generates the first token based on the target prompt word, and then superimposes the generated first token on the target prompt word as input to generate the second token; then, superimposes the first token and the second token on the target prompt word as input to generate the third token.

[0132] Step 302: The first accelerator set determines whether the first model can stop generating tokens.

[0133] Each time the first model generates a new token, the first accelerator set can determine whether the first model can stop generating tokens. Specifically, if the token generated by the first model is a terminator, it means that the first model can stop generating tokens; or, if the number of tokens generated by the first model has reached a preset number (such as 1000), it means that the first model can also stop generating tokens.

[0134] Step 303: If the first model fails to stop generating tokens, the first accelerator set determines whether the number of newly generated unverified tokens by the first model has reached N.

[0135] If the first model has not yet stopped generating tokens, the first accelerator set will continue to determine whether the number of newly generated tokens that have not been verified by the second model has reached N. Here, N is an integer greater than 1.

[0136] Optionally, N is a preset value, and N is determined based on the difference in prompt word processing results between the first model and the second model. Specifically, for the same prompt word, if the difference between the prompt word processing results obtained by the first model and the prompt word processing results obtained by the second model is small, it means that the output results of the first model and the output results of the second model will be closer, that is, the number of tokens output by the first model that need to be corrected will be smaller. Therefore, N can be set to a larger value.

[0137] If there is a large difference between the processing results of the first model and the processing results of the second model for the same prompt word, it means that the output results of the first model and the output results of the second model are not very close. That is, the number of tokens output by the first model that need to be corrected may be larger. Therefore, N can be set to a smaller value.

[0138] Understandably, if N is set too small, the number of tokens generated by the first model in a single round will be relatively small, and the number of tokens validated in parallel by the second model will also be relatively small, resulting in low efficiency in prompt word processing. If N is set too large, the number of tokens generated by the first model in a single round will be large, and the number of incorrect tokens will often increase accordingly, causing the second model to need to regenerate a large number of tokens during the validation phase. Furthermore, the second model regenerates tokens serially during the validation phase, so a large number of incorrect tokens during a single round of validation will also lead to low efficiency in prompt word processing.

[0139] Based on this, in this solution, the number of tokens that need to be verified generated by the first model in a single round is set according to the difference in the prompt word processing results between the first model and the second model. This allows for a reasonable setting of the number of tokens, ensuring that the N tokens generated by the first model in a single round do not contain too many incorrect tokens, thereby guaranteeing the efficiency of prompt word processing.

[0140] Optionally, the difference in prompt word processing results between the first model and the second model can be determined through various implementation methods.

[0141] In one possible implementation, the same prompt word is first input into both the first and second models, and then the tokens predicted by the first and second models are obtained respectively. Specifically, for the first and second models, when generating a new token based on the input prompt word, multiple candidate tokens are first generated, each with a corresponding selection probability. Then, based on the selection probabilities of each candidate token, the token with the highest selection probability is selected as the output token, thus generating the token. Therefore, when the first and second models generate tokens for each position based on the prompt word, the differences between the tokens with the highest selection probability at each position (i.e., the output tokens) can be compared, and the differences between the output tokens can be used as the difference in prompt word processing results between the first and second models.

[0142] In another possible implementation, since the first and second models generate multiple candidate tokens when generating a new token based on the input prompt, and each candidate token has a corresponding selection probability, the probability distributions of the candidate tokens generated by the first and second models can be obtained. Then, the difference between the probability distributions of the candidate tokens generated by the first and second models can be used as the difference in the prompt processing results between the first and second models.

[0143] In another possible implementation, the difference between the output tokens of the first model and the second model can be calculated simultaneously, as well as the difference between the probability distribution of the candidate tokens generated by the first model and the probability distribution of the candidate tokens generated by the second model. The two differences are then weighted and summed to obtain the difference in prompt word processing results between the first model and the second model.

[0144] In general, in practical applications, the differences in prompt word processing results between the first model and the second model can be determined in a variety of ways, and this embodiment does not impose any specific limitations on this.

[0145] Step 304: If the first model has generated N new tokens, the first accelerator set will transmit the N newly generated tokens to the second accelerator set.

[0146] It should be noted that when transmitting the N newly generated tokens from the first accelerator set to the second accelerator set, the content used to generate these N tokens also needs to be transmitted to the second accelerator set, namely the target prompt word and other tokens generated before these N tokens, or the features corresponding to the target prompt word and the features corresponding to other tokens generated before these N tokens.

[0147] If the number of newly generated tokens in the first model does not reach N, then proceed to step 301 above, that is, continue to generate new tokens.

[0148] Step 305: The second accelerator set uses the second model to verify the correctness of each of the N tokens in parallel and corrects erroneous tokens to obtain the verification result.

[0149] Since all N tokens have already been generated, for each of the N tokens, the second accelerator set can verify the correctness of each token and correct erroneous tokens based on the content used to generate each of the N tokens by running the second model, thus obtaining the verification result.

[0150] For example, please refer to Figure 4, which is a schematic diagram of a method provided in this application where a first model generates a token and a second model verifies the token. As shown in Figure 4, compared to the related technologies where a second model with a large number of parameters generates tokens one by one, in this application, a first model with a smaller number of parameters generates tokens one by one, and a second model with a larger number of parameters verifies the multiple tokens generated by the first model in parallel.

[0151] Specifically, when the first accelerator set runs the first model, the first model first uses prompt words... <s>The first token "Several" is obtained through reasoning; then, the first model uses the prompt words... <s>And from the token "Several", the second token "famous" is derived; secondly, the first model is based on the prompt words. <s>Based on the tokens "Several" and "famous", the third token "songs" is derived; then, the first model uses the prompt words... <s>In addition to tokens "Several", "famous", and "songs", we can deduce the fourth token "are".

[0152] After the first model completes the inference of four tokens, the first accelerator set passes these four tokens and related inference data (such as KV Cache) to the second accelerator set running the second model. The second accelerator set then verifies these four tokens in parallel by running the second model. When verifying the first token, the second model provides input prompts. <s>When verifying the second token, the input to the second model is a prompt word. <s>And the token "Several"; when verifying the third token, the input of the second model is a prompt word. <s>The input to the second model is the tokens "Several" and "famous"; when validating the fourth token, the input is a prompt word. <s>And tokens "Several", "famous", and "songs".

[0153] In other words, in this application, tokens are generated serially by a first model, and multiple generated tokens are verified in parallel by a second model. Since the first model, with its smaller number of parameters, generates tokens at a speed more than ten times faster than the second model, which has a larger number of parameters, the speed of generating multiple tokens serially by the first model is significantly higher than that by the second model. Furthermore, the second model verifies multiple tokens in parallel, so the speed of verifying multiple tokens in parallel by the second model is close to the speed of generating a single token by the second model. Ultimately, this ensures that the efficiency of generating tokens serially by the first model and verifying multiple tokens by the second model is higher than the efficiency of directly generating tokens serially by the second model.

[0154] Step 306: The second accelerator set returns the verification results to the first accelerator set.

[0155] Step 307: Based on the verification results, the first accelerator set continues to process the target prompt words using the first model to generate a new token.

[0156] Steps 306-307 are similar to steps 205-206 above. Please refer to steps 205-206 above for details. They will not be repeated here.

[0157] Furthermore, after executing step 307, the process will proceed to continue executing step 302 to determine whether token generation can be stopped.

[0158] Step 308: If the first model can stop generating tokens, then the first accelerator set will transmit the tokens generated in the last round of the first model to the second accelerator set.

[0159] In this context, the tokens generated in the last round by the first model refer to newly generated tokens that have not yet been verified by the second model. It's important to note that in the preceding rounds of token generation, the first model consistently generates N tokens before passing these N newly generated, unverified tokens to the second model for verification. However, in the final round of token generation, the number of tokens generated by the first model may be less than or equal to N, depending on the actual output of the first model; no specific limit is imposed here.

[0160] Step 309: The second accelerator set returns the verification result of the token generated in the last round to the first accelerator set.

[0161] Similarly, after the first accelerator set passes the last round of generated tokens to the second accelerator set, the second accelerator set also verifies all tokens generated by the first model in the last round in parallel by running the second model, and returns the verification result of the last round of generated tokens to the first accelerator set.

[0162] At step 310, the first accelerator set obtains the output result based on all the tokens that have been verified.

[0163] After the second accelerator set returns the verification result of the last round of generated tokens, the first accelerator set can obtain all the tokens that have been verified, and then convert these tokens that have been verified into the output result by running the first model, thereby completing the processing of the target prompt.

[0164] Optionally, in the present application, the second accelerator set includes more resources than the first accelerator set, and the ratio between the resources included in the second accelerator set and the resources included in the first accelerator set is determined based on the time length for the first model to generate tokens and the time length for the second model to verify tokens.

[0165] For example, first, the first time length required for the first model to generate a single token and the second time length required for the second model to verify a single token are determined. The first time length and the second time length are obtained by respectively running the first model and the second model on the same accelerator resources.

[0166] Then, the time length ratio between the target time length and the second time length is determined, wherein the target time length is the product of N and the first time length. That is, since the first model generates N tokens in series, the first time length for the first model to generate a single token is multiplied by N, and the total time length for the first model to generate N tokens in series (i.e., the aforementioned target time length) is obtained.

[0167] In this way, based on the time length ratio, the ratio between the amount of resources included in the first accelerator set and the amount of resources included in the second accelerator set can be determined. For example, the calculated time length ratio is taken as the ratio between the amount of resources included in the first accelerator set and the amount of resources included in the second accelerator set.

[0168] It can be understood that, since the first model generates a token for a much shorter time than the second model verifies a token, the target time obtained by multiplying the time for the first model to generate a single token by N is still less than the second time for the second model to verify a single token. In addition, since the second model verifies multiple tokens in parallel, setting the proportion of the amount of resources between the first accelerator set and the second accelerator set based on the time ratio between the target time and the second time can ensure that the time for the first accelerator set to serially generate N tokens is close to the time for the second accelerator set to parallelly verify N tokens, ensuring that both the first accelerator set and the second accelerator set are continuously in working condition, thereby improving the processing efficiency of the prompt word as much as possible under limited resources.

[0169] Referring to FIG. 5, FIG. 5 is a schematic diagram of the waste of computing resources generated by using a small model to generate tokens and using a large model to verify tokens. As shown in FIG. 5, if a first model (hereinafter referred to as a small model) with a smaller amount of parameters and a second model (hereinafter referred to as a large model) with a larger amount of parameters are deployed on the same accelerator set at the same time, and the small model is first run by the accelerator set to generate multiple tokens, and then the large model is run by the accelerator set to verify the generated multiple tokens in parallel, there will be a phenomenon of idle computing resources.

[0170] Specifically, in order to ensure that the large model can verify multiple tokens in parallel, the computing resources of the accelerator set are set based on the computing resources required by the large model to verify multiple tokens in parallel, that is, to ensure that the computing resources in the accelerator set support the large model to verify multiple tokens in parallel. Then, since the small model itself consumes less computing resources when generating a token than the large model consumes when verifying a token, and the large model verifies multiple tokens in parallel, the computing resources consumed by the small model when serially generating multiple tokens are much less than the computing resources consumed by the large model when verifying multiple tokens in parallel. In this way, the computing resources actually consumed by the small model when serially generating multiple tokens are only a small part of all the computing resources in the accelerator set, and since the small model and the large model work serially, the computing resources in the entire accelerator set cannot be effectively utilized when the small model serially generates multiple tokens, resulting in a situation where most of the computing resources are idle, causing waste of computing resources.

[0171] Therefore, by assigning different accelerator sets to the small model and the large model respectively, and ensuring that the computing resources in the accelerator set running the small model are less than the computing resources in the accelerator set running the large model, the computing resources can be reasonably allocated, and the utilization of the computing resources can be improved.

[0172] Specifically, referring to FIG. 6, FIG. 6 is a diagram provided by the application to illustrate the waste of computing resources caused by generating tokens using a small model and verifying tokens using a large model. As shown in FIG. 6, the first accelerator set for running the small model includes 1 NPU, and the second accelerator set for running the large model includes 4 NPUs. In this way, the first accelerator set and the second accelerator set can independently run the small model and the large model respectively, thereby independently performing token generation and token verification, and ensuring that the computing resources in the first accelerator set and the second accelerator set can always maintain a high resource utilization rate.

[0173] The above introduces the specific process of the process of the first model and the second model cooperating to complete the processing of the prompt word. The processing of the prompt word will be further divided into multiple stages to implement the processing of the prompt word.

[0174] Specifically, the processing of the prompt word (i.e., the inference process of the model) can be divided into a pre-filling stage and a decoding stage. In the pre-filling stage, for the input prompt word, Key vectors and Value vectors are generated for each transformer layer in the model to obtain KV Cache. That is, the pre-filling stage can be understood as performing feature extraction processing on the prompt word, thereby obtaining the features of the prompt word.

[0175] The decoding stage is to use and update the KV Cache to generate tokens one by one, and the current generated token depends on the previously generated token. That is, the decoding stage uses the KV Cache generated in the pre-filling stage to generate new tokens, and the KV Cache corresponding to the newly generated token is also used together with the existing KV Cache to continue to generate new KV Cache.

[0176] Since the processing performed in the pre-filling stage and the decoding stage is different, the pre-filling stage and the decoding stage can be respectively assigned to different accelerator sets for execution in the present application, so as to improve the overall processing efficiency of the prompt word.

[0177] Exemplarily, before the prompt set is processed by the first model and the second model, feature extraction can be performed on the prompts in the prompt set by the third accelerator set to obtain the features of the prompts, and the features of the prompts are used to enable the first model to generate tokens. That is, after the third accelerator set performs feature extraction on the prompts, the obtained features of the prompts can be transmitted to the first model on the first accelerator set, and then the first model further generates tokens based on the features of the prompts.

[0178] Among them, the second model can be run on the third accelerator set, that is, the third accelerator set performs feature extraction on the prompts in the prompt set by running the second model, so as to ensure the accuracy of the extracted features of the prompts.

[0179] Specifically, please refer to FIG. 7, which is a comparison diagram of the execution of a pre-population stage and a decoding stage provided by the present application. As shown in FIG. 7, in the pre-population stage, the second model is run in the third accelerator set to perform feature extraction on the input prompt set, so as to obtain the KV Cache corresponding to the prompts and the first token inferred based on the extracted KV Cache. Then, the third accelerator set transmits the features of the prompts (i.e., the KV Cache corresponding to the prompts and the first token inferred) to the first accelerator set performing the decoding stage. In the decoding stage, the first model is run in the first accelerator set, which is used to process the features of the prompts transmitted by the third accelerator set, so as to generate tokens; and the tokens generated by the first model are transmitted to the second accelerator set, and the second model run on the second accelerator set performs verification on the tokens generated by the first model and returns the corresponding verification results. Finally, after the first model on the first accelerator set generates all the tokens and these tokens are verified, the tokens are converted into output results, so as to obtain the output results corresponding to the prompts.

[0180] In addition, the third accelerator set can also only extract the KV Cache corresponding to the prompts, without inferring the first token based on the KV Cache corresponding to the prompts; in this way, the features of the prompts transmitted by the third accelerator set to the first accelerator set only have the KV Cache corresponding to the prompts. Optionally, when the memory resources in the accelerator set are not the bottleneck of the model inference speed, the resources included in the third accelerator set are more than the sum of the resources included in the first accelerator set and the second accelerator set.

[0181] Specifically, in the pre-filling stage, the model processes the input prompt words to obtain the features of the prompt words (i.e., KV Cache), which are used to generate the first new token. Since the complete range of the input prompt words is known, the processing performed in the pre-filling stage is actually a highly parallel matrix operation, which can saturate the utilization of processing resources on the accelerator.

[0182] However, in the decoding stage, the model generates an output token at a time in an auto-regressive manner until the stop condition is met. Each output token needs to know all the tokens of the previous iterations. Therefore, compared with the pre-filling stage, the decoding stage does not actually fully utilize the processing resources on the accelerator. In simple terms, the processing resources occupied by the model during the pre-filling stage are higher than those during the decoding stage. Based on this, the third accelerator set for performing the pre-filling stage is configured to have more resources than the sum of the resources included in the first accelerator set and the second accelerator set for jointly performing the decoding stage, so as to allocate more accelerator resources to the pre-filling stage to improve the utilization of accelerator resources as much as possible, thereby ensuring the highest possible data processing efficiency based on limited resources.

[0183] In addition, when the memory resources of the accelerator set are insufficient while the computing resources of the accelerator set are relatively sufficient, the memory resources of the accelerator set will become the bottleneck of limiting the inference speed of the model. Therefore, when the memory resources in the accelerator set are the bottleneck of limiting the inference speed of the model, the third accelerator set can be configured to include fewer resources than the sum of the resources included in the first accelerator set and the second accelerator set, so that more memory resources can be allocated to the model inference stage to avoid the memory resources becoming the bottleneck of limiting the inference speed of the model as much as possible, thereby improving the overall prompt word processing efficiency.

[0184] In addition, the applicant has found through research that the current pre-filling stage performed by the model often divides the prompt words into multiple batches for processing, and each batch includes a fixed number of prompt words. When processing a batch of prompt words, multiple prompt words in the same batch are processed in parallel; after all the prompt words in the previous batch are processed, the prompt words in the next batch are processed. However, this way of dividing batches according to a fixed number of prompt words is prone to waste of computing resources.

[0185] Exemplarily, refer to FIG. 8, which is a schematic diagram of waste of computing resources when processing prompt words in batches according to the present application. As shown in FIG. 8, it is assumed that in the pre-filling stage, each 5 prompt words are divided into a batch, and therefore when the accelerator set processes a batch of prompt words at a time, 5 prompt words are processed in parallel. As shown in (a) of FIG. 8, in the scenario where the prompt words in the same batch are short, the total number of tokens of the 5 prompt words in the same batch is low, and the total number of tokens that can be processed in parallel by the accelerator set is high, so that the 5 prompt words in the same batch cannot fully use all the computing resources of the accelerator set during processing, resulting in the phenomenon of idle computing resources, i.e., waste of computing resources.

[0186] As shown in (b) of FIG. 8, in the scenario where the prompt words in the same batch are long, the total number of tokens of the 5 prompt words in the same batch is higher than the total number of tokens that can be processed in parallel by the accelerator set, so that the accelerator set will split the 5 prompt words, and first process the front part of the tokens of the 5 prompt words in parallel, and then process the back part of the tokens of the 5 prompt words in parallel. When the accelerator set processes the front part of the tokens of the 5 prompt words, since the total number of tokens in the front part is equal to the total number of tokens that can be processed in parallel by the accelerator set, the accelerator set will not have idle computing resources when processing the front part of the tokens. However, when the accelerator set processes the back part of the tokens of the 5 prompt words, since the total number of tokens in the back part is less than the total number of tokens that can be processed in parallel by the accelerator set, the accelerator set will have idle computing resources when processing the back part of the tokens.

[0187] Based on this, the present application proposes a way of determining the prompt words processed in parallel in the pre-filling stage by filling the computing pool, so as to ensure that the computing resources in the accelerator set can be used as much as possible when processing the prompt words in parallel each time, avoiding the waste of computing resources, and thereby improving the efficiency of the accelerator set in processing the prompt words.

[0188] Exemplarily, in the case where the prompt word set includes a plurality of prompt words, the third accelerator set performs feature extraction on the prompt words in the prompt word set, which can include the following processes: first, the processor adds the prompt words in the prompt word set to the first storage space one by one, i.e., the prompt words in the prompt word set are added to the first storage space one by one. The first storage space can be a memory space pre-divided for storing a batch of prompt words.

[0189] Then, when the total number of tokens included in the added prompt words in the first storage space is greater than the target number, the processor performs segmentation on the last prompt word added to the first storage space, so that the total number of tokens included in the added prompt words in the first storage space is not greater than the target number. Wherein, the last prompt word added to the first storage space is segmented into a first part and a second part, the first part is retained in the first storage space, and the second part is added to the second storage space, which can also be a memory space pre-divided for storing another batch of prompt words. That is, in the process of adding prompt words to the first storage space one by one, if the total number of tokens included in the added prompt words in the first storage space reaches a certain number after adding a certain prompt word, the last added prompt word is segmented, so that the tokens included in the first storage space are within a certain number.

[0190] It can be understood that in the pre-filling stage, when the prompt words are processed by the model, each prompt word is usually converted into multiple tokens, so that the processing process for the prompt words is converted into a processing process for multiple tokens. Therefore, after the prompt words are added to the first storage space, the total number of tokens included in the added prompt words in the first storage space can be calculated.

[0191] Secondly, the processor delivers the prompt words in the first storage space to the third accelerator set as a batch, so that the third accelerator set performs feature extraction on the prompt words in the first storage space in parallel.

[0192] Specifically, the third accelerator set performs feature extraction on the prompt words in the first storage space in parallel, and after the processing of the prompt words in the first storage space is completed, the third accelerator set performs feature extraction on the prompt words in the second storage space in parallel. Wherein, the second storage space is a calculation pool required to be processed after the first storage space. And the prompt words in the second storage space are also added one by one from the prompt word set. The way of adding prompt words in the second storage space is similar to the way of adding prompt words in the first storage space, and when the total number of tokens included in the added prompt words in the second storage space exceeds the target number, the last prompt word added to the second storage space also needs to be segmented, so that the total number of tokens included in the second storage space also does not exceed the target number.

[0193] That is, the prompt words included in the same storage space can be regarded as a batch of prompt words processed in parallel, and the prompt words included in each batch are processed in sequence by the third accelerator set. Moreover, the third accelerator set processes all the prompt words in the same batch in parallel when processing the prompt words included in each batch, so that the number of tokens processed in parallel by the third accelerator set is relatively fixed each time.

[0194] Exemplarily, please refer to FIG. 9A, which is a flowchart of determining the prompt words processed in parallel by the accelerator set provided in the present application. As shown in FIG. 9A, the flow of determining the prompt words processed in parallel by the hardware set can be executed by the processor, and the flow includes the following steps 901-906.

[0195] Step 901, obtaining a prompt word set.

[0196] In the prompt word set, a plurality of prompt words are included, and the prompt words included in the prompt word set are all prompt words that need to be processed.

[0197] Step 902, taking one prompt word from the prompt word set and adding it to the current storage space.

[0198] In the present application, a plurality of storage spaces are set, and each storage space is used to store a batch of prompt words processed in parallel by the third accelerator set. Moreover, when adding prompt words to the storage space, the prompt words are taken from the prompt word set one by one and added to the storage space.

[0199] Step 903, judging whether the total number of tokens in the current storage space reaches the target number.

[0200] Specifically, after taking one prompt word from the prompt word set and adding it to the storage space, the total number of tokens included in the prompt words added to the storage space can be judged once.

[0201] Optionally, the target number described above can be determined based on the third accelerator set, and the target number is related to the resource amount included in the third accelerator set. In the case that the total number of tokens processed by the third accelerator set at a time is the target number, the resource utilization rate of the third accelerator set reaches the maximum value. That is, the target number is actually the maximum number of tokens processed in parallel by the third accelerator set at a time.

[0202] The target number can be determined by testing. For example, for the third accelerator set, the time required by the third accelerator set to process different numbers of tokens in parallel is tested by setting different numbers of tokens, and the efficiency of the accelerator in processing tokens in different numbers is determined, and finally the number of tokens processed in parallel by the third accelerator set to achieve maximum efficiency is determined as the target number. For another example, based on the model required by the third accelerator set to process tokens, it is analyzed how many computing resources (e.g., how many floating point operations) are consumed to run a model to process a token, and then based on the computing resources provided by the third accelerator set, the maximum number of tokens that can be processed in parallel by the third accelerator set is determined, thereby determining the target number.

[0203] Step 904, if the total number of tokens in the storage space reaches the target number, the last prompt word added to the storage space is divided into a first part and a second part.

[0204] In the process of continuously adding prompt words to the storage space, if the total number of tokens included in the prompt words in the storage space reaches the target number after adding a prompt word, it means that the total number of tokens of the added prompt words has overflowed, so the last prompt word added to the storage space needs to be divided.

[0205] In the process of division, the last prompt word is divided into two parts, a first part and a second part. The first part needs to be retained in the current storage space, and the second part needs to be added to the next storage space. Therefore, when dividing the prompt word, it needs to be ensured that when the first part after division is retained in the current storage space, the total number of tokens included in the current storage space does not exceed the target number. For example, assuming that the last prompt word includes 100 tokens, the target number is 1000, and after adding the last prompt word to the current storage space, the total number of tokens included in the current storage space is 1040, so the last prompt word can be divided into a first part including 60 tokens and a second part including 40 tokens. In this way, by retaining the first part and moving the second part from the current computing pool to the next storage space, the total number of tokens included in the current storage space can be exactly 1000.

[0206] Step 905, the first part of the last prompt word is retained in the current storage space, and the second part of the prompt word is added to the next storage space.

[0207] Step 906, all prompt words in the current storage space are processed in parallel by the third accelerator set.

[0208] If the last hint added to the current storage space is split so that the total number of tokens in the current storage space equals the target number, it means that the current storage space is full of hints. Therefore, a third accelerator set can be used to process all hints in the current storage space in parallel.

[0209] It's important to note that after the current storage space is filled with hints, a similar hint addition method can be used to sequentially add hints from the hint set to the next storage space. Depending on actual needs, it's possible to split the last added hint, continuing until the number of tokens in the next storage space equals the target number. In this way, after processing the hints in the current storage space using the third accelerator set, the hints in the next storage space are processed in parallel using the third accelerator set. This ensures that the third accelerator set processes a fixed number (i.e., the target number) of tokens in parallel each time, guaranteeing that its computing resources are not wasted.

[0210] In this scheme, the prompt words are added sequentially to the pre-divided storage space, and the last prompt word added to the storage space is split. This ensures that the number of tokens included in each storage space is relatively fixed. As a result, when the tokens in the storage space are processed in parallel by the accelerator set, the computing resources of the accelerator set are used as much as possible, avoiding the waste of computing resources and improving the processing efficiency of prompt words.

[0211] To facilitate understanding, the model inference acceleration method provided by this solution will be introduced below in conjunction with the overall model inference acceleration framework.

[0212] For example, please refer to Figure 9B, which is a schematic diagram of a framework for accelerating model inference provided in this application. As shown in Figure 9B, it is assumed that the accelerator used to perform model inference includes a total of 12 NPUs, which are used to perform the pre-filling stage and the decoding stage in the model inference process.

[0213] During model inference, the pre-filling and decoding phases are deployed separately, meaning different NPUs are used to execute the pre-filling and decoding phases respectively. Furthermore, the decoding phase is further divided into a token generation phase and a token verification phase, each executed using a different NPU.

[0214] Specifically, the 12 NPUs used for model inference can be divided into three accelerator sets (accelerator set 1 to accelerator set 3). Accelerator set 1 corresponds to the third accelerator set mentioned above, and includes NPUs 1-8 for the pre-filling stage. Accelerator set 2 corresponds to the first accelerator set mentioned above, and includes NPU 9 for the token generation stage. Accelerator set 3 corresponds to the second accelerator set mentioned above, and includes NPUs 10-12 for the token verification stage.

[0215] In other words, because the pre-filling phase requires more computational resources than the decoding phase, the pre-filling phase is allocated 8 NPUs, while the decoding phase is allocated 4 NPUs. Furthermore, based on the decoding phase, the token generation phase is allocated only 1 NPU, while the token verification phase is allocated 3 NPUs.

[0216] Before the pre-filling phase, multiple storage spaces are set up for sequential processing, and prompts are taken from the prompt word set and added to the storage spaces one by one. While adding prompts from the prompt word set to the storage space, it is necessary to continuously check whether the total number of tokens included in the added prompts in the storage space reaches the target number. If the total number of tokens included in the added prompts in the storage space exceeds the target number, the last prompt word added to the storage space needs to be split. One part of the split tokens is kept in the original storage space, while the other part of the tokens is placed in the next storage space, ensuring that the number of tokens included in each storage space is as close as possible to the target number.

[0217] During the pre-filling stage, NPU 1-NPU 8 process the prompt words in each storage space in parallel to extract the features of the prompt words. The features of the prompt words are then transmitted to the decoding stage in the form of KV Cache to generate the token.

[0218] During the decoding phase, NPU 9 runs a first model to process the features of the prompt words, thereby generating tokens one by one. NPU 9 then passes the generated tokens to NPUs 10-12. NPUs 10-12 run a second model to verify the tokens generated by the first model and return the verification results to NPU 9. In this way, through the interaction between NPU 9 and NPUs 10-12, all tokens can be generated, and the output result can be obtained based on the generated tokens.

[0219] The method provided in this application has been described in detail above. Next, the device provided in this application for performing the above method will be described.

[0220] Please refer to Figure 10, which is a schematic diagram of the structure of a training and acceleration system provided in this application. As shown in Figure 10, the training and acceleration system includes: a processor 1001, used to acquire a set of prompt words, the set of prompt words including at least one prompt word, the at least one prompt word being used to describe the task to be processed; the processor 1001 is also used to schedule a first accelerator set 1002 and a second accelerator set 1003 to process the set of prompt words, wherein the second accelerator set 1003 includes more resources than the first accelerator set 1002; the first accelerator set 1002 and the second accelerator set 1003 are used to process the set of prompt words and obtain output results; wherein the first accelerator set 1002 processes the set of prompt words and generates a token by running a first model, and the token is used to convert into the output result, and the second accelerator set 1003 verifies the accuracy of the token and corrects erroneous tokens by running a second model, and the number of parameters of the first model is less than the number of parameters of the second model.

[0221] In one possible implementation, a first accelerator set 1002 is used to process target prompt words in the prompt word set using a first model to generate N tokens, where N is an integer greater than 1; a second accelerator set 1003 is used to verify the correctness of each of the N tokens in parallel using a second model and correct erroneous tokens to obtain a verification result; based on the verification result, the first accelerator set 1002 is used to continue processing target prompt words using the first model to generate new tokens.

[0222] In one possible implementation, N is a preset value, and N is determined based on the difference in prompt word processing results between the first model and the second model.

[0223] In one possible implementation, the processor 1001 is further configured to determine a first duration required for the first model to generate a single token and a second duration required for the second model to verify a single token, wherein the first duration and the second duration are obtained by running the first model and the second model on the same accelerator resources respectively; the processor 1001 is further configured to determine, based on the first duration and the second duration, the ratio between the amount of resources included in the first accelerator set 1002 and the amount of resources included in the second accelerator set 1003.

[0224] In one possible implementation, the system further includes a third accelerator set 1004, which, before the first accelerator set 1002 and the second accelerator set 1003 process the cue word set, is used to: perform feature extraction on the cue words in the cue word set to obtain features of the cue words, the features of the cue words being used as input to the first model so that the first model generates a token.

[0225] In one possible implementation, at least one prompt word specifically refers to multiple prompt words; the processor 1001 is further configured to perform segmentation and integration on the prompt words in the prompt word set, and transmit the integrated prompt words to the third accelerator set 1004 in batches; wherein, the total number of tokens included in each batch of prompt words transmitted by the processor 1001 to the third accelerator set 1004 is not greater than the target number.

[0226] In one possible implementation, the processor 1001 is further configured to: sequentially add the prompts from the prompt set to a first storage space; when the total number of tokens included in the prompts already added to the first storage space is greater than the target number, perform segmentation on the last prompt added to the first storage space so that the total number of tokens included in the prompts already added to the first storage space is the target number, wherein the last prompt is segmented into a first part and a second part, the first part is retained in the first storage space, and the second part is added to the second storage space; and transmit the prompts in the first storage space as a batch to the third accelerator set 1004 so that the third accelerator set 1004 performs feature extraction on the prompts in the first storage space in parallel.

[0227] In one possible implementation, the target number is related to the amount of resources included in the third accelerator set 1004.

[0228] Please refer to Figure 11, which is a schematic diagram of the structure of a computing device provided in this application. The computing device 1100 shown in Figure 11 can be used to execute the model inference acceleration method provided in this embodiment. As shown in Figure 11, the computing device 1100 includes: a bus 1102, a processor 1104, an accelerator set 1105, a memory 1106, and a communication interface 1108. The processor 1104, the memory 1106, and the communication interface 1108 communicate with each other through the bus 1102. The computing device 1100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors, accelerator sets, and memory in the computing device 1100. For example, the computing device 1100 may include three accelerator sets, with different accelerator sets used to perform different tasks.

[0229] Bus 1102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 11, but this does not imply that there is only one bus or one type of bus. Bus 1102 can include pathways for transmitting information between various components of computing device 1100 (e.g., memory 1106, processor 1104, communication interface 1108).

[0230] The processor 1104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0231] The accelerator set 1105 includes one or more accelerators, such as neural network processing units (NPUs), graphics processing units (GPUs), or tensor processing units (TPUs).

[0232] The memory 1106 may include volatile memory, such as random access memory (RAM). The processor 1104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0233] The memory 1106 stores executable program code, and the processor 1104 executes the executable program code to implement the functions of the aforementioned training and inference acceleration system, thereby realizing the aforementioned model inference acceleration method. That is, the memory 1106 stores instructions for executing the model inference acceleration method.

[0234] The communication interface 1108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1100 and other devices or communication networks.

[0235] It should be understood that the computing device 1100 according to this application is used to execute the model inference acceleration method as shown in Figures 2 to 9B, and can correspond to the execution of the corresponding subject in the method according to this application, which will not be described in detail here for the sake of brevity.

[0236] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0237] Please refer to Figure 12, which is a schematic diagram of a computing device cluster provided in this application. As shown in Figure 12, the computing device cluster includes at least one computing device 1100. The memory 1106 of one or more computing devices 1100 in the computing device cluster may store the same instructions for executing model inference acceleration methods.

[0238] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the computing device cluster may also store partial instructions for executing the model inference acceleration method. In other words, a combination of one or more computing devices 1100 can jointly execute the instructions for executing the model inference acceleration method.

[0239] It should be noted that the memory 1106 in different computing devices 1100 in the computing device cluster can store different instructions, which are used to execute some functions of the training and acceleration system.

[0240] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 13 illustrates one possible implementation. Figure 13 is a schematic diagram of another computing device cluster structure provided in this application. As shown in Figure 13, in computing device cluster 1300, two computing devices 1100A and 1100B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this type of possible implementation, the memory 1106 in computing device 1100A stores instructions for performing some functions. Simultaneously, the memory 1106 in computing device 1100B stores instructions for performing other functions.

[0241] It should be understood that the functions of computing device 1100A shown in Figure 13 can also be performed by multiple computing devices 1100. Similarly, the functions of computing device 1100B can also be performed by multiple computing devices 1100.

[0242] It should be understood that the computing device 1100 or computing device cluster 1300 in this application can correspond to the training and inference acceleration system in Figure 10 of this application, and can correspond to the corresponding entity executing the model inference acceleration method in Figure 2 of this application. Furthermore, the above-mentioned and other operations and / or functions of each module in the computing device 1100 or computing device cluster 1300 are respectively for implementing the corresponding process of the model inference acceleration method in Figure 2, and for the sake of brevity, will not be elaborated further here.

[0243] This application also provides a chip comprising a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip within the electronic device to perform the methods described in the above embodiments. Optionally, the storage unit may be an in-chip storage unit, such as a register or cache. Alternatively, the storage unit may be an external storage unit located within a wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0244] Specifically, please refer to Figure 14, which is a schematic diagram of the structure of a chip provided in this application. The chip can be represented as a neural processing unit (NPU), a graphics processing unit (GPU), or a tensor processing unit (TPU). The following description will use an NPU 1400 as an example. The NPU 1400 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 1403, which is controlled by a controller 1404 to retrieve matrix data from memory and perform multiplication operations.

[0245] In some implementations, the arithmetic circuit 1403 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1403 is a two-dimensional pulsating array. The arithmetic circuit 1403 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1403 is a general-purpose matrix processor.

[0246] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1402 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1401 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1408.

[0247] Unified memory 1406 is used to store input and output data. Weight data is directly transferred to weight memory 1402 via Direct Memory Access Controller (DMAC) 1405. Input data is also transferred to unified memory 1406 via DMAC.

[0248] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1409.

[0249] The Bus Interface Unit (BIU) 1410 is used by the instruction fetch memory 1409 to fetch instructions from external memory, and also by the memory access controller 1405 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0250] The DMAC is mainly used to move input data from external memory DDR to unified memory 1406, or to weight data to weight memory 1402, or to input data to input memory 1401.

[0251] The vector computation unit 1407 includes multiple arithmetic processing units that, when needed, further process the output of the computation circuit 1403, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0252] In some implementations, the vector computation unit 1407 can store the processed output vector in the unified memory 1406. For example, the vector computation unit 1407 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1403, such as performing linear interpolation on feature planes extracted from a convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1407 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 1403, for example, for use in subsequent layers of the neural network.

[0253] The instruction fetch buffer 1409 connected to the controller 1404 is used to store the instructions used by the controller 1404;

[0254] Unified memory 1406, input memory 1401, weighted memory 1402, and instruction fetch memory 1409 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0255] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0256] It should be understood that the chip in Figure 14 of this application may correspond to the training and inference acceleration system in Figure 10 of this application, or be deployed on the computing device 1100 or computing device cluster 1300 of this application. Furthermore, the chip in Figure 14 of this application may correspond to the corresponding entity executing the model inference acceleration method in Figure 2 of this application. The above-mentioned and other operations and / or functions of each module in the chip are respectively for implementing the corresponding process of the model inference acceleration method in Figure 2, which will not be elaborated here for the sake of brevity. In addition, in this application, the chip structure is not limited to the chip structure shown in Figure 14, and may include more or fewer hardware structures to implement the functions of the method shown in Figure 2.

[0257] Referring to Figure 15, which is a schematic diagram of the structure of a computer-readable storage medium provided in this application. This application also provides a computer-readable storage medium in which, in some embodiments, the method disclosed in Figure 2 above can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0258] Figure 15 schematically illustrates a conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein, the example computer-readable storage medium including a computer program for executing computer processes on a computing device.

[0259] In one embodiment, the computer-readable storage medium 1500 is provided using a signal bearer medium 1501. The signal bearer medium 1501 may include one or more program instructions 1502, which, when executed by one or more processors, can provide the functions or parts thereof described above with reference to FIG2.

[0260] In some examples, the signal carrying medium 1501 may include a computer-readable medium 1503, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video disc (DVD), a digital magnetic tape, a memory, ROM, or RAM, etc.

[0261] In some embodiments, the signal-bearing medium 1501 may comprise a computer-recordable medium 1504, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, and so on. In some embodiments, the signal-bearing medium 1501 may comprise a communication medium 1505, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, and so on). Therefore, for example, the signal-bearing medium 1501 may be transmitted by a wireless communication medium 1505 (e.g., a wireless communication medium conforming to the IEEE 1202.X standard or other transmission protocols).

[0262] One or more program instructions 1502 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1502 conveyed to the computing device via a computer-readable medium 1503, a computer-recordable medium 1504, and / or a communication medium 1505.

[0263] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0264] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0265] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0266] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions can be transferred from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0267] The above are merely specific embodiments of this application. Variations or substitutions conceived by those skilled in the art based on the specific embodiments provided in this application should all be covered within the protection scope of this application.< / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s>

Claims

A method for accelerating model inference, characterized in that, The method is applied to a training and acceleration system, the system including a processor, a first set of accelerators, and a second set of accelerators, the method including: The processor acquires a set of prompt words, the set of prompt words including at least one prompt word, the at least one prompt word being used to describe the task to be processed; The processor schedules the first set of accelerators and the second set of accelerators to process the set of prompt words, wherein the second set of accelerators includes more resources than the first set of accelerators. The first accelerator set and the second accelerator set process the prompt word set to obtain the output result; The first accelerator set processes the prompt word set and generates lexical tokens by running a first model, and the tokens are used to convert the output results. The second accelerator set verifies the accuracy of the tokens and corrects erroneous tokens by running a second model. The number of parameters in the first model is less than the number of parameters in the second model. The method according to claim 1, characterized in that, The first accelerator set and the second accelerator set process the cue word set, including: The first accelerator set uses the first model to process the target prompt words in the prompt word set to generate N tokens, where N is an integer greater than 1; The second accelerator set uses the second model to verify the correctness of each of the N tokens in parallel and corrects erroneous tokens to obtain the verification result; Based on the verification results, the first accelerator set continues to process the target prompt word using the first model to generate a new token. The method according to claim 2, characterized in that, The value N is a preset value, and the value N is determined based on the difference in prompt word processing results between the first model and the second model. The method according to claim 2 or 3, characterized in that, The method further includes: The processor determines a first duration required for the first model to generate a single token and a second duration required for the second model to verify a single token. The first duration and the second duration are obtained by running the first model and the second model on the same accelerator resources, respectively. The processor determines the ratio between the amount of resources included in the first accelerator set and the amount of resources included in the second accelerator set based on the first duration and the second duration. The method according to any one of claims 1-4, characterized in that, The system further includes a third set of accelerators, and the method further includes: [further details to be added] before the first and second sets of accelerators process the set of prompt words. The third accelerator set performs feature extraction on the prompt words in the prompt word set to obtain the features of the prompt words. The features of the prompt words are used as input to the first model so that the first model generates a token. The method according to claim 5, characterized in that, The at least one prompt word specifically refers to multiple prompt words, and the method further includes: The processor performs segmentation and integration processing on the prompt words in the prompt word set, and transmits the processed prompt words to the third accelerator set in batches; In this case, the total number of tokens included in each batch of prompts passed by the processor to the third accelerator set is not greater than the target number. The method according to claim 6, characterized in that, The processor performs segmentation and integration processing on the prompt words in the prompt word set, and transmits the processed prompt words to the third accelerator set in batches, including: The processor sequentially adds the prompt words from the prompt word set to the first storage space; When the total number of tokens included in the prompt words added to the first storage space is greater than the target number, the processor performs a segmentation on the last prompt word added to the first storage space so that the total number of tokens included in the prompt words added to the first storage space is the target number. The last prompt word is segmented into a first part and a second part. The first part is retained in the first storage space, and the second part is added to the second storage space. The processor transmits the prompt words in the first storage space as a batch to the third accelerator set, so that the third accelerator set performs feature extraction on the prompt words in the first storage space in parallel. The method according to claim 6 or 7, characterized in that, The target number is related to the amount of resources included in the third accelerator set. A training acceleration system, characterized in that, include: Processor, first accelerator set, and second accelerator set; The processor is configured to acquire a set of prompt words, the set of prompt words including at least one prompt word, the at least one prompt word being used to describe the task to be processed; The processor is further configured to schedule the first set of accelerators and the second set of accelerators to process the set of prompt words, wherein the second set of accelerators includes more resources than the first set of accelerators. The first accelerator set and the second accelerator set are used to process the prompt word set to obtain the output result; The first accelerator set processes the prompt word set and generates a token by running a first model, and the token is used to convert the output result. The second accelerator set verifies the accuracy of the token and corrects erroneous tokens by running a second model. The number of parameters in the first model is less than the number of parameters in the second model. The system according to claim 9 is characterized in that, The first accelerator set is used to process the target prompt words in the prompt word set using the first model to generate N tokens, where N is an integer greater than 1; The second set of accelerators is used to verify the correctness of each of the N tokens in parallel using the second model and to correct erroneous tokens to obtain the verification result; Based on the verification results, the first accelerator set is used to continue processing the target prompt word using the first model to generate a new token. The system according to claim 10 is characterized in that, The value N is a preset value, and the value N is determined based on the difference in prompt word processing results between the first model and the second model. The system according to claim 10 or 11 is characterized in that, The processor is further configured to determine a first duration required for the first model to generate a single token and a second duration required for the second model to verify a single token, wherein the first duration and the second duration are obtained by running the first model and the second model on the same accelerator resources respectively; The processor is further configured to determine, based on the first duration and the second duration, the ratio between the amount of resources included in the first accelerator set and the amount of resources included in the second accelerator set. The system according to any one of claims 9-12 is characterized in that, The system further includes a third set of accelerators, which, before the first and second sets of accelerators process the cue word set, are used to: Feature extraction is performed on the prompt words in the prompt word set to obtain the features of the prompt words. The features of the prompt words are used as input to the first model so that the first model generates a token. The system according to claim 13 is characterized in that, The at least one prompt word specifically refers to multiple prompt words; The processor is further configured to perform segmentation and integration on the prompt words in the prompt word set, and transmit the integrated prompt words to the third accelerator set in batches; In this case, the total number of tokens included in each batch of prompts passed by the processor to the third accelerator set is not greater than the target number. The system according to claim 14 is characterized in that, The processor is also used for: Add the prompt words from the prompt word set to the first storage space in sequence; When the total number of tokens included in the prompt words added to the first storage space is greater than the target number, the last prompt word added to the first storage space is split so that the total number of tokens included in the prompt words added to the first storage space is the target number. The last prompt word is split into a first part and a second part. The first part is kept in the first storage space, and the second part is added to the second storage space. The prompt words in the first storage space are passed to the third accelerator set as a batch, so that the third accelerator set performs feature extraction on the prompt words in the first storage space in parallel. The system according to claim 14 or 15 is characterized in that, The target number is related to the amount of resources included in the third accelerator set. A computing device, characterized in that, The device includes a memory, a first set of accelerators and a second set of accelerators, and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the computing device performs the method as described in any one of claims 1 to 8. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor, an accelerator set and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the operational steps of the method as described in any one of claims 1 to 8. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 8. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 8.

Images