Method for executing batch of requests by applying machine learning model and non-transitory computer-readable storage medium

Abstract

A method of executing a batch of requests by applying a machine learning model and a non-transitory computer-readable storage medium are provided. The method comprises the following steps: receiving a batch of two or more token sequences; accessing a machine learning model; generating one or more output representations, the generating comprising: obtaining one or more queries, keywords, and values, splitting the first and second queries from the query for the first and second token sequences, splitting the first and second keywords from the keywords, and splitting the first and second values from the values, generating a first attention output by combining at least the first query, the first keyword, and the first value, individually generating a second attention output by combining at least the second query, the second keyword, and the second value, joining at least the first attention output and the second attention output into a join tensor, and generating one or more output representations by applying at least the one or more weight tensors to the join tensor.

Description

[0001] This application is a divisional application of the invention patent application with the application date of August 18, 2022, application number 202210994036.0, and invention name “Selective batching of reasoning systems for converter-based generation tasks”. Background Art

[0002] The present invention relates generally to machine learning transformer neural network models, and more particularly to selective batching for transformer models.

[0003] The transformer neural network model is a machine learning model used in various applications such as natural language processing (NLP), image processing, or audio processing applications including sequential data. For example, the transformer model can receive an input token sequence representing a query and generate an output token sequence representing a response to the query. As another example, the transformer model can receive an input token sequence representing a German paragraph and generate an output token sequence representing a translation of the English paragraph. As yet another example, the transformer model can receive an input token sequence representing a text paragraph and generate an output token sequence representing a summarized version of the text.

[0004] Typically, a user of a client device submits requests to an inference system. The inference system executes a machine learning transformer model on the inputs of the requests (e.g., input token sequences) to generate outputs for the requests (e.g., output token sequences). The inference system can return the outputs as responses to the requesting client device. In one example, the inference system executes the requests on a dedicated hardware accelerator such as a graphics processing unit (GPU) or a tensor processing unit (TPU) to improve latency and throughput, especially when the number of parameters of the transformer model is significant.

[0005] In one instance, the inference system processes requests in batches to achieve high processor utilization of the accelerator. Specifically, the inference system can process multiple requests in a batch together to take advantage of the amount of parallel computing units in the hardware accelerator. In many cases, the inputs of the requests in the batch are variable in length. For example, the number of input tokens for each request in the batch may be variable in length. However, methods for batching for converter models typically require that the data of multiple requests in the batch be of the same length for processing. Therefore, processing a batch of requests with variable lengths may not be feasible, or workarounds to address this problem may result in using more resources than processing each request individually. Summary of the invention

[0006] An inference system that applies a machine learning transformer model to a batch of requests with variable input lengths, variable target lengths, or variable internal state lengths by selectively batching a subset of operations in the transformer model but processing the requests in the batch separately for the subset of operations in the transformer model. In one embodiment, the operations to be processed separately are attention operations of an encoder or decoder of the transformer model. By selectively batching, the inference system can enable batch operations to be performed for a batch of requests with variable input lengths, target lengths, or internal state lengths to exploit the parallel computing capabilities of hardware accelerators while preventing unnecessary computations that occur for workarounds that constrain data for a batch of requests to be of the same length.

[0007] Specifically, in one embodiment, an inference system receives a batch of requests including one or more input token sequences. The length of a first input token sequence for a first request in the batch may be different from the length of a second input token sequence for a second request. The inference system accesses a transformer model including at least one set of decoders coupled to each other. For one or more iterations, the inference system repeatedly performs the step of generating one or more output tokens for the request by applying the set of decoders to the one or more inputs for the request.

[0008] For at least one decoder in the group, the inference system generates one or more queries, one or more keywords, and one or more values ​​for the request by applying the QKV weight tensor to one or more input representations. In one instance, the queries, keywords, and values ​​are generated by batch operations. The inference system at least splits a first query for a first request from the one or more queries, splits a first keyword for the first request from the one or more keywords, and splits a first value for the first request from the one or more values. The inference system also at least splits a second query for a second request from the one or more queries, splits a second keyword for the second request from the one or more keywords, and splits a second value for the second request from the one or more values.

[0009] The inference system generates a first attention output for a first request by at least combining a first query, a first keyword, and a first value for the first request. The inference system also separately generates a second attention output for a second request by at least combining a second query, a second keyword, and a second value for the second request. The inference system at least concatenates the first attention output and the second attention output into a concatenated tensor, and generates one or more output representations by applying a weight tensor to the concatenated tensor. In one example, the one or more output representations are generated by a batch operation. The inference system sets one or more output tokens as one or more inputs of the group of decoders for a subsequent iteration, and provides the output tokens generated for at least one request to a client device as a response to the at least one request.

[0010] In one embodiment, the inference system performs iterative level dynamic batching for the converter model, which enables the inference system to dynamically modify a batch of requests being executed on the execution engine. Specifically, in existing batching methods for the converter model, it is difficult to modify a batch of requests once processing of the batch of requests begins on the execution engine. This is because some batching methods require that the length of the input or the length of the internal state is the same between all requests in the batch. Therefore, unless the newly incoming request has the same length input as the batch of requests being executed on the execution engine, it may be difficult for the inference system to modify the batch to, for example, add a new request to the batch or remove a new request.

[0011] By performing selective batching, the reasoning system can monitor and modify the batch being processed on the execution engine at the iteration level, and update the batch between iterations when requests are completed and new requests are received. Specifically, at one or more iterations, the reasoning system can modify the batch being executed on the execution engine by adding new incoming requests to the batch or removing completed requests from the batch. This is because selective batching enables processing of requests with variable lengths without restricting one or more inputs or internal states of the converter model to the same length. This enables the reasoning system to remove requests that are completed earlier than other requests in the batch, so that responses can be provided to the user more quickly, and enables the reasoning system to add new requests to a batch of requests when the execution engine is not fully utilized.

[0012] In one embodiment, a service system of an inference system receives one or more requests for execution. The service system may include a request processor and a scheduler, each coupled to one or more execution engines for executing a machine learning converter model including at least one set of decoders. The scheduler schedules a batch of requests including one or more requests for execution on the execution engine. The execution engine generates a first set of output tokens by iteratively applying the converter model to a first set of inputs for the batch of requests. In one instance, applying the converter model includes applying at least one batch operation to one or more input tensors associated with the batch of requests.

[0013] The service system may receive a new request including a sequence of input tokens from a client device. The scheduler schedules a second batch of requests for execution on the execution engine in response to determining that the execution engine has memory available to execute the second batch of requests including the one or more requests and the new request. The execution engine generates a second set of output tokens by iteratively applying the transformer model to a second set of inputs for the second batch of requests. The second set of inputs may include a sequence of input tokens for the new request. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a high-level block diagram of a system environment for an inference system according to an embodiment.

[0015] FIG. 2A to FIG. 2B A method of batching using a machine learning transformer model according to an embodiment is shown.

[0016] FIG. 3A to FIG. 3B A method for selective batching using a machine learning transformer model according to an embodiment is shown.

[0017] Figure 4 is a block diagram of an inference system according to an embodiment.

[0018] FIG. 5A to FIG. 5D A method for processing dynamic batching of requests using a machine learning transformer model is shown according to an embodiment.

[0019] FIG. 6A to FIG. 6B is a flow chart illustrating a method of selective batching using a converter model according to an embodiment.

[0020] Figure 7 is a flow chart illustrating a method for processing dynamic batching of requests using a transformer model, according to an embodiment.

[0021] Figure 8 is a diagram illustrating a computer system on which embodiments described herein may be implemented within an inference system according to embodiments.

[0022] These figures depict various embodiments of the present invention for illustrative purposes only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles of the present invention described herein. DETAILED DESCRIPTION

[0023] Overview

[0024] Figure 1 is a high-level block diagram of a system environment 100 for an inference system 130 , according to an embodiment. Figure 1 The illustrated system environment 100 includes: one or more client devices 110A, 110B; a network 12; and an inference system 130. In alternative configurations, different or additional components may be included in the system environment 100.

[0025] The reasoning system 130 receives a request from a client device 110A, 110B to perform a task using a machine learning model. In one embodiment, the machine learning model is a converter neural network model. The task may include, but is not limited to, natural language processing (NLP), image processing, and audio processing applications. Specifically, the converter model may be suitable for processing sequential data, which may be tokenized into an input token sequence for a request and an output token sequence for an expected response. The reasoning system 130 receives a request including input data (e.g., text data, image or video data, audio data), and encodes the input data into a set of input tokens. The reasoning system 130 repeatedly applies the machine learning converter model for one or more iterations to generate a set of output tokens. The reasoning system 130 decodes the set of output tokens into output data and returns the output data as a response to the request. Although for applications such as NLP applications, the input token sequence or the output token sequence is arranged along one dimension (1-D) to represent, for example, a word sequence, it should be understood that in other embodiments, the input token sequence or the output token sequence may be a multidimensional sequence. For example, for two-dimensional image data, the token sequence can be a two-dimensional (2-D) sequence arranged along a first direction (e.g., X-axis) and a second direction (e.g., Y-axis), where each token corresponds to a block of one or more pixels in the image.

[0026] In particular, NLP tasks involve using artificial intelligence and machine learning techniques to analyze language, and can include a variety of tasks, including translation, sentiment analysis, text summarization, automatic correction, etc. When processing NLP tasks, the reasoning system 130 receives a request (e.g., a query) for an input text including a sequence of words, and encodes the input text into a sequence of input tokens that each represent a corresponding word in a latent space. The reasoning system 130 repeatedly applies the transformer model for one or more iterations to generate an output token sequence (e.g., a response to the query). The output tokens are converted into output text as a response to the request.

[0027] For example, a transformer model may receive a sequence of input tokens representing a query and generate a sequence of output tokens representing a response to the query. As another example, a transformer model may receive a sequence of input tokens representing a French paragraph and generate a sequence of output tokens representing a translation of an English paragraph or sentence. As yet another example, a transformer model may receive a sequence of input tokens representing a paragraph of text and generate a sequence of output tokens representing a summarized version of the text.

[0028] In one embodiment, the reasoning system 130 includes one or more execution engines built on a dedicated hardware accelerator such as a graphics processing unit (GPU) or a tensor processing unit (CPU). Requests are executed on the execution engines. Specifically, the execution of a machine learning neural network model (e.g., a transformer model) involves a large number of operations, such as tensor multiplications between input data and high-dimensional weight tensors that may be computationally intensive. The hardware accelerator of the execution engine can be optimized to efficiently perform these operations through parallel processing, resulting in significant improvements in latency or throughput when the number of parameters in the transformer model is large.

[0029] The hardware of the inference system 130 may include one or more central processing unit (CPU) cores, CPU memory (e.g., DRAM), data storage devices, one or more execution engines (e.g., GPU devices). Each execution engine may include a set of cores (e.g., GPU cores) coupled to a local memory (e.g., GPU memory), and may be composed of one or more hardware accelerators. In addition, the inference system 130 may be composed of multiple hardware components and the following components: the components are used to configure the network to connect various components on multiple hardware components together so that these components can coordinate with each other to process requests. For example, an execution engine can communicate with multiple hardware accelerators on multiple machines. The execution engine can process data stored on its local memory. Specifically, during the training or inference of the converter model, the data required for inference or training is read by the CPU from the input file in the data storage device or read from, for example, the client device 110 through the network 120, moved to the local memory of the execution engine and processed by the execution engine. The results of the processing are retrieved by the CPU.

[0030] In one embodiment, the reasoning system 130 processes requests in batches to achieve higher processor utilization on the hardware accelerator. Specifically, the reasoning system 130 processes multiple requests in a batch together to utilize the amount of parallel computing units in the execution engine. In such an embodiment, the reasoning system 130 receives multiple requests each associated with an input token sequence. The reasoning system 130 iteratively applies the transformer model to the batch of requests to generate output tokens for the requests together. In one example, batching is made possible for the transformer model by grouping together requests with input token sequences of the same length or treating the requests in the batch as if they all have the same input token sequence length as the request with the shortest length at each iteration.

[0031] Utilizing the batched converter model

[0032] FIG. 2A to FIG. 2BA method for batching using a machine learning transformer model 200 according to an embodiment is shown. In particular, the transformer model 200 is associated with a set of parameters determined by a training process. The transformer model 200 is coupled to receive one or more inputs (e.g., an input token sequence or an output token of a previous iteration) and generate one or more outputs (e.g., output token predictions). Specifically, in one embodiment, a request is processed by applying the transformer model to the input token sequence of the request in one iteration of the "encoding phase" and applying the transformer model to the output token sequence generated at the previous iteration of one or more iterations of the "decoding phase". In the decoding phase, the output token of the previous iteration is set as the input of the next iteration, and the process is repeated for additional iterations of the decoding phase until an end token (e.g., a period "." or "<end>" indicator) is generated for each request in the batch, signaling that all predictions have been generated for the corresponding request. In other embodiments, the termination criteria for the request may be termination after a maximum number of output tokens are generated, or may be other types of conditions specified, for example, by the inference system 130 or a user of the client device 110.

[0033] Figure 2A The encoding phase for the transformer model 200 is shown, in which a set of input token sequences are processed to generate one or more output tokens. Figure 2A In the example shown, the reasoning system 130 for processing requests of the chatbot receives a first request such as the question "What is your name?", a second request such as the question "What time?", and a third request such as the question "How do I pay?". The reasoning system 130 encodes each of the requests into a corresponding set of input token sequences. The first request is encoded into an input token sequence X1, the second request is encoded into an input token sequence X2, and the third request is encoded into an input token sequence X3, each of which is shown in the figure with a different fill pattern. Figure 2A Each input token sequence in is a one-dimensional sequence, where the token sequence is arranged along a single dimension (e.g., the X direction). However, as mentioned above about Figure 1 As described, it will be appreciated that in other embodiments, the token sequence may be arranged as a multi-dimensional sequence.

[0034] like Figure 2AAs shown, since each request includes four words, each input token sequence includes four tokens, each token representing a corresponding word. For example, the input token sequence X1 of the first request is represented by four blocks representing the words "what", "is", "your", and "name". Specifically, although each word is mapped to a single block, in fact, the reasoning system 130 represents the token of the word as an embedding representing the word in a multi-dimensional latent space. Therefore, although in Figure 2A , each input token sequence is visually shown as a two-dimensional 1×4 tensor, but in fact, each input token sequence can be represented as a three-dimensional tensor 1×4×H, where H is the dimension of the embedding (e.g., the direction of entering or leaving the page). In addition, although for the rest of this specification, each token (input token or output token) is mapped to a word, this is only an example, and it is understood that in other embodiments, each token can be mapped to a different text unit, a combination of text units, etc. For example, in other embodiments, each token can be mapped to a text unit of multiple words, paragraphs, sentences, n-grams, or can be mapped to punctuation marks other than text units (e.g., "?", "!", ":").

[0035] In one embodiment, the converter model 200 includes a set of N decoders D1, D2, ..., DN. The decoders are coupled to receive a set of input representations and generate a set of output representations. For example, the first decoder D1 is coupled to receive one or more inputs of the converter model 200 as a set of input representations and generate a set of output representations. Each subsequent decoder is coupled to receive a set of output representations of the previous decoder and generate another set of output representations. For example, the second decoder D2 placed after the first decoder D1 is coupled to receive a set of output representations generated by the first decoder D1 and generate another set of output representations. Repeat this process until a set of output representations for the final decoder is generated.

[0036] The transformer model 200 may further include a LM header block 270 coupled to receive a set of output representations from the final decoder DN and generate one or more output tokens as outputs of the current iteration. Figure 2A In the example shown, the LM head 270 receives a set of output representations from the final decoder DN and generates a tensor representation of the batch request One or more output tokens in the form of . Specifically, a tensor The method includes: a first output token for a first request representing a first word prediction "my" for a response; a second output token for a second request representing a word prediction "the" for a response; and a third output token for a third request representing a word prediction "you" for a response.

[0037] Figure 2B 2 shows a decoding phase for the transformer model 200, in which a previous output token is processed as input to generate one or more additional output tokens. Unlike the encoding phase, the transformer model 200 is applied to the output token generated at the previous iteration as input to generate additional output tokens. Figure 2B As shown, the transformer model 200 receives at the next iteration the output tokens generated at the previous iteration. And generate additional output tokens for requests as output. The decoding phase is repeated until the termination criteria is met for each request in the batch.

[0038] The converter model 200, and in particular, each decoder Di in the group of decoders D1, D2, ..., DN includes one or more blocks, each block representing a corresponding operation, in particular a tensor operation. Specifically, the blocks in the converter model 200 may contain data for performing an operation on one or more input tensors of a request to generate one or more output tensors for the request. When an operation is performed on an execution engine, one or more input tensors and associated data (e.g., weight tensors) of the operation may be read from a local memory of the execution engine. The operation may be performed using one or more input tensors and associated data to generate one or more output tensors. The output tensor is provided to a CPU, another execution engine, or may be stored in a local memory of the execution engine so that it can be used for the next operation.

[0039] In one embodiment, the operation of the converter model 200 is configured as a batch operation, in which data for a batch of requests is processed together. The batch operation is coupled to receive one or more input tensors, each of which is a concatenation of input data for multiple requests in the batch. The batch operation generates one or more output tensors by applying corresponding operations to one or more input tensors, each of which is a concatenation of output data for multiple requests in the batch. For batch operations, the input tensor can be the concatenation of the input tensors of each corresponding request in the batch. Therefore, the input tensor of the batch operation is a single larger tensor that merges the input tensors of each request in the batch on an additional batch dimension.

[0040] Similarly, the output tensor from a batch operation can be represented as the concatenation of the output tensors of each corresponding request in the batch. Thus, while the output tensor of a single request can also be generated by applying the operation to the input tensors of the requests, the output tensor generated from a batch operation using concatenated input tensors is also a single larger tensor that combines the output tensors of each request in the batch across the batch dimension.

[0041] In addition, certain operations such as tensor multiplication operations involve multiplying a set of parameters (e.g., weight tensors) of the transformer model 200 with the input tensors of the batch operation. In some cases, when processing requests individually, the weight tensors are read from the local memory of the hardware accelerator each time the input tensors of the request are multiplied with the weight tensors. In contrast, when performing tensor multiplication as a batch operation, the set of parameters is read once and reused for operations on the concatenated input tensors. This can result in a significant improvement in throughput compared to processing requests individually. However, in FIG. 2A to FIG. 2B In batch processing, since the input data of a batch is processed as a single tensor, each request may be required to have the same length in the input tensor and the internal state tensor.

[0042] like Figure 2A As shown, the decoder in the converter model 200 includes a first layer normalization block 210, a QKV operation block 215, a split block 220, a self-attention block 225, an attention linear block 230, a first addition block 235, a second layer normalization block 240, a first MLP block 245, a GeLU block 250, a second MLP block 255, and a second addition block 260. In one embodiment, each of the blocks in the decoder is executed on the execution engine as a batch operation, wherein the input tensor of the batch operation is the concatenation of the input data of a batch of requests, and the output tensor is the concatenation of the output data of the batch of requests. Although the operations in the first decoder D1 are described as examples, it is understood that the remaining decoders in the group may also include operations similar to the first decoder D1.

[0043] Specifically, the inference system 130 concatenates the input token sequence X1, X2, X3 into a concatenated input tensor. The layer normalization block 210 is coupled to receive the concatenated input tensor and normalize the elements of each request to generate a normalized tensor as an output tensor. The QKV operation block 215 is coupled to receive the normalized tensor as an input tensor and generate an output tensor including a query, keyword, and value for the request. Specifically, the QKV operation block 215 generates queries, keywords, and values ​​by applying the QKV weight tensor, which is a trained parameter set of the converter model 200, to the normalized tensor including normalized data for each request to generate an output tensor including a query, keyword, and value for each request in the batch. For example, Figure 2AAs shown, the first row of the output tensor of the QKV operation 215 includes a query, a keyword, and a value (in concatenated form) for a first request, the second row includes a query, a keyword, and a value for a second request, and so on. In one embodiment, instead of applying a single QKV weight tensor, the QKV operation block 215 can be associated with separate weight tensors that are each applied to the input tensor to generate a query, a keyword, and a value, respectively. For example, the QKV operation block 215 can apply a query weight tensor to the input tensor to generate a query for a request, apply a separate keyword weight tensor to the input tensor to generate a keyword for a request, and apply a separate value weight tensor to the input tensor to generate a value for a request.

[0044] The split block 220 is coupled to: receive the output tensor from the QKV operation block 215, and split the output tensor into a query tensor, a keyword tensor, and a value tensor for the current iteration if the query tensor, the keyword tensor, and the value tensor are not generated separately. For the current iteration, the query tensor includes a query for the batch request, the keyword tensor includes keywords for the batch request, and the value tensor includes values ​​for the batch request. In one instance, the execution engine maintains an internal state for the batch request as a keyword cache tensor for caching keywords generated at a previous iteration and a current iteration, and a value cache tensor for caching values ​​generated at a previous iteration and a current iteration. The reasoning system 130 adds the keyword tensor for the current iteration to the keyword cache tensor, and adds the value tensor for the current iteration to the value cache tensor.

[0045] The self-attention block 225 is coupled to receive a query tensor, a keyword cache tensor, and a value cache tensor as input tensors, and generates an output tensor including attention outputs for the requests in the batch. In one instance, the attention output tensor is generated by multiplying the query tensor with the keyword cache tensor to generate a multiplied tensor and multiplying the multiplied tensor with the value cache tensor. In the case where the self-attention block 225 is executed as a batch operation, the query tensor can be multiplied with the keyword cache tensor on the execution engine to generate the multiplied tensor. The value cache tensor can be multiplied with the multiplied tensor to generate the attention output tensor. The attention output tensor includes attention outputs for the requests in the batch. For example, in Figure 2A , the first row of the attention output tensor Z' is the attention output for the first request, the second row of the attention output tensor Z' is the attention output for the second request, and so on.

[0046] The attention linear block 230 is coupled to receive the attention output tensor as an input tensor and generate an output tensor by applying the attention weight tensor, which is a trained parameter set of the transformer model 200, to the attention output tensor. The attention linear block 230 is configured to operate in batches. For example, in Figure 2A , the first row of the output tensor Z may be output data for the first request, and the second row of the output tensor Z may be output data for the second request, and so on. The first addition block 235 combines the concatenated input tensor having the input token sequence and the output tensor from the attention linear block 230 to generate a combined tensor. The second layer normalization block 240 is coupled to receive the combined tensor from the addition block 235 and normalizes the elements of each request to generate a normalized tensor as an output tensor.

[0047] The decoder also includes a multilayer perceptron (MLP) block 242, which includes one or more blocks including additional neural network layers. In one embodiment, the MLP block 242 includes a first MLP block 245, which is coupled to receive the normalized tensor from the second layer normalization block 240 as an input tensor, and generates an output tensor by applying the MLP weight tensor, which is a trained parameter set of the converter model 200, to the normalized tensor. The MLP block 242 includes a GeLU block 250, which is coupled to receive the output tensor from the first MLP block 245 as an input tensor, and generates an output tensor by applying the GeLU function to the input tensor. The MLP block 242 includes a second MLP block 255, which is coupled to receive the output tensor from the GeLU block 250 as an input tensor, and generates an output tensor by applying the second MLP weight tensor, which is a trained parameter set of the converter model 200.

[0048] The combined tensor from the second addition block 260 can be obtained as a set of output representations generated for the first decoder D1. Subsequently, the second decoder D2 placed after the first decoder D1 is coupled to receive a set of output representations from the first decoder D1, and repeats similar operations to generate another set of output representations. The process is repeated until a set of output representations from the final decoder DN is generated. Specifically, although each decoder may involve operations similar to the first decoder D1, the trained parameter sets associated with these operations may differ between decoders. The LM header block 270 is coupled to receive a set of output representations from the final decoder DN as an input tensor, and generates an output tensor including one or more output tokens.

[0049] As combined Figure 2B As depicted, the next iteration is the decoding phase, in which the output tokens generated in the previous iteration are input to the transformer model 200. The transformer model 200 generates another set of output tokens. The layer normalization block 210 is applied to the output tensor to generate a normalized tensor. The QKV operation block 215 is applied to the normalized tensor to generate an output tensor including the query, keyword, and value for the current iteration. The split block 220 is applied to split the output tensor into a query tensor, a keyword tensor, and a value tensor. The reasoning system 130 adds the keyword tensor for the current iteration to the keyword cache tensor, and adds the value tensor for the current iteration to the value cache tensor. Therefore, in Figure 2B During the iteration shown, the keyword cache tensor Kcache includes keywords from the previous iteration and the current keyword tensor K, and the value cache tensor Vcache includes values ​​from the previous iteration and the current value tensor V.

[0050] The self-attention block 225 is applied to the query tensor, keyword cache tensor, and value cache tensor for the current iteration to generate an attention output tensor. The attention linear block 230 is applied to the attention output tensor to generate an output tensor. The addition block 235 combines the input tensor and the output tensor from the attention linear block 230 to generate a combined tensor. The MLP block 242 is applied to the combined tensor to generate a set of output representations for the decoder D1. The process is repeated for the remaining decoders and LM head 270 to generate an output tensor including another set of output tokens. And the converter model 200 is applied iteratively until the termination criteria is met for each request in the batch.

[0051] Converter Model Using Selective Batching

[0052] In many cases, the input to a request or the internal state of a request is variable in length and may be difficult to use e.g. FIG. 2A to FIG. 2B For example, the number of input tokens between different NLP requests is likely to vary in length depending on the input text and the type of request. For example, the length of the first query request may be different from the length of the second query request due to different user synthesized languages. However, methods that batch the transformer model, such as combining FIG. 2A to FIG. 2B The described method requires that the input tensors for the batch have the same length, since the input tensors must be merged into a single concatenated tensor for the operation to work. Therefore, if the input tensors for multiple requests have different lengths, use FIG. 2A to FIG. 2B It may not be feasible to process requests as batches.

[0053] Additionally, a workaround to this problem is to treat the input token sequence as if the request has the same length as the input token sequence with the shortest length (i.e., the least number of tokens) during the encoding phase. However, this requires the inference system 130 to discard output tokens generated for requests with longer input lengths in subsequent iterations, because the original input tokens received for the request must be replaced in the corresponding elements. This results in inefficiency and unnecessary waste of resources because the inference system processes input token sequences for longer requests separately in multiple iterations rather than a single iteration. Another workaround may be to treat the input token sequence as if the request has the same length as the input token sequence with the longest length (i.e., the largest number of tokens). However, this requires the inference system to increase the size of the input tensor to match the request with the longest token sequence (and fill the remaining elements). This results in redundant computation because the inference system processes larger input tensors throughout the execution of the transformer model. For example, in some cases, using FIG. 2A to FIG. 2B The number of iterations required to process a batch of requests can be significantly greater than the number of iterations required to process each request individually.

[0054] Therefore, in one embodiment, the inference system 130 trains a machine learning transformer model using a selective batching approach and applies it to a batch of requests with variable input lengths or target lengths or internal state lengths. In the selective batching approach, the inference system 130 selectively batches a subset of operations in the transformer model, but processes requests separately for a subset of operations in the transformer model that are incompatible with batching. In one embodiment, the operations to be processed separately are attention operations of the encoder or decoder of the transformer model. Through selective batching, the inference system 130 can enable batch operations to be performed for a batch of requests with variable input lengths or target lengths or internal state lengths to exploit the parallel computing capabilities of hardware accelerators while preventing problems that occur in workarounds.

[0055] FIG. 3A to FIG. 3B A method for selective batching using the machine learning transformer model 300 is shown according to an embodiment. Figure 3A In the example shown, Figure 2A Differently, the inference system 130 receives a batch of requests with input token sequences of different lengths. Figure 3A In the example of FIG. 1 , the reasoning system 130 receives a first request such as the question “What’s up?”, a second request such as the question “Hello?”, and a third request such as the question “How are you?”, each of which has a different word length. The reasoning system 130 encodes each of the requests into a corresponding set of input token sequences. The first request is encoded into an input token sequence X1, the second request is encoded into an input token sequence X2, and the third request is encoded into an input token sequence X3.

[0056] FIG. 3A to FIG. 3B The converter model 300 shown also includes a set of N decoders D1, D2, ... DN and a LM header block 370. FIG. 2A to FIG. 2B Unlike the transformer model 200, a subset of the operations in the transformer model 300, in particular, the operations in the self-attention operation block 325, are executed individually rather than processed as a batch operation. Figure 3A As shown, the converter model 300 includes a first layer normalization block 310, a QKV operation block 315, a splitting block 320, a self-attention block 325, an attention linear block 330, a first addition block 335, a second layer normalization block 340, a first MLP block 345, a GeLU block 350, a second MLP block 355 and a second addition block 360.

[0057] Specifically, in the selective batching method, the inference system 130 may concatenate the input token sequences X1, X2, X3 into a concatenated input tensor. FIG. 2A to FIG. 2B Unlike the batching method in , the inference system 130 concatenates the input token sequence so that the input tokens of the batch request are in one dimension, such as FIG. 3A to FIG. 3B When input token sequences with different lengths are input to the transformer model 300, one way to concatenate these sequences into a concatenated tensor with the same dimensionality for the requests in a batch is to concatenate the input tokens of these requests in a single dimension. Thus, Figure 3A The concatenated tensor in is shown as Figure 3A ΣL in b ×1 tensor, where ΣL b represents the sum of the token lengths of each request b in the batch, where the length of the input token of request b is denoted by L b In practice, the concatenated input tensor can be ΣL b ×1×H tensor, where H is the dimensionality of the embedding.

[0058] The first layer normalization block 310 is applied to the concatenated input tensors to generate a normalized tensor that normalizes the elements of each request. In one example, the operation of the first layer normalization block 310 is given by:

[0059]

[0060] in, represents the concatenated input tensor, and represents a normalized tensor, ε is a constant, and

[0061] The QKV operation block 315 is applied to the normalized tensor as an input tensor to generate an output tensor including a query, a keyword, and a value for the request. The QKV operation block can be configured as one or more neural network layers. Specifically, the QKV operation block 315 generates a query, a keyword, and a value by applying a QKV weight tensor as a trained parameter set of the converter model 300 to the normalized tensor. In one example, the operation of the QKV operation block 315 is given by:

[0062]

[0063] in, represents the normalized tensor from the first layer normalization block 310, represents the output tensor containing the query, keyword, and value, is the QKV weight tensor, and (bias).

[0064] In particular, in the selective batching method, the QKV operation block 315 can be configured as a batch operation, and the QKV weight tensor is multiplied with the normalized tensor from the layer normalization block 310 to generate an output tensor including queries, keywords, and values ​​for a batch of requests. Due to the shape of the concatenated input tensor, the output tensor of the QKV operation 315 can also be arranged so that each query element of the requests is concatenated along one dimension (e.g., the vertical dimension), and each keyword element of the requests is concatenated along the one dimension, and each value element of the requests is concatenated along the one dimension. For example, as Figure 3A As shown, the first and second rows of the output tensor of the QKV operation 315 correspond to the query, keyword, and value of the first request, the third row corresponds to the query, keyword, and value of the second request, and the fourth to sixth rows correspond to the query, keyword, and value of the third request. Similarly, the QKV operation block 315 can generate queries, keywords, and values ​​separately by applying the query tensor, a separate keyword tensor, and a separate value tensor to the input tensor. In one embodiment, the QKV operation block 315 includes multiple attention heads, and generates queries, keywords, and values ​​for each attention head. In such an instance, when the QKV operation block 315 includes n attention heads, H can be equal to n×h, where h is the dimension of the latent space of each attention head.

[0065] The split block 320 is coupled to receive the output tensor from the QKV operation block 315 and split the output tensor into a query tensor, a keyword tensor, and a value tensor for the current iteration. In one example, the operation of the split block 320 is given by:

[0066] Q i，j =X i，j , Ki，j =X i，H+j , V i，j =X i，2H+j

[0067] in, represents the output tensor from the QKV operation block 315, represents the query tensor, represents a keyword tensor, When the QKV operation block 315 includes multiple attention heads, the query tensor may include n queries for each request, the keyword tensor may include n keywords for each request, and the value tensor may include n values ​​for each request.

[0068] and Figure 2A Different from the batching method of , the split block 320 also splits the query tensor Q into queries for each request, splits the keyword tensor K into keywords for each request, and splits the value tensor V into values ​​for each request. In one example, each request split operation of the split block 320 is given by:

[0069]

[0070] Among them, Q1, Q2, ..., Q B is a query for a single request, and

[0071]

[0072] Among them, K1, K2, ..., K B is a keyword for a single request, and

[0073]

[0074] Among them, V1, V2, ..., V B is a query for a single request, and

[0075] In addition, for each request b=1, 2, ..., B, the inference system 130 caches the tensor Kcache with the key b Sum value cache tensor Vcache b The internal state of the request is maintained in the form of and And L b,curris the current length of the token sequence (including both input tokens and output tokens) of request b. In other embodiments, the internal state of the request may contain different types of information than the keywords and values ​​of the request, and the embodiments are not limited thereto. Therefore, for the encoding phase, L b,curr Equal to L b The reasoning system 130 converts the keyword tensor K into b Add to the corresponding keyword cache tensor, and the value tensor V b Add to the corresponding value cache tensor. Because the set of inputs for a batch of requests (i.e., the input token sequences for X1, X2, X3) have different lengths, the length of the internal state of the requests in the batch is Figure 3A is different. FIG. 2A to FIG. 2B In batch processing of , since the self-attention operation 225 involves multiplying the query tensor, the keyword cache tensor, and the value cache tensor, it is also required that the keyword cache tensor and the value cache tensor have the same length between requests in a batch. However, since Figure 3A Selective batching is performed in , so requests with different lengths for their internal states can also be efficiently batched for selection operations, while being processed separately for self-attention operations 325.

[0076] The self-attention block 325 is coupled to receive a query tensor, a keyword cache tensor, and a value cache tensor for each request as one or more input tensors, and generates one or more output tensors as attention outputs for each request in the batch. In one embodiment, the self-attention block 325 is not batched, but at least a subset of the requests in the batch are processed separately from each other. Specifically, each request in the batch can be processed separately for the self-attention block 325.

[0077] For a request in a batch, the self-attention block 325 is coupled to receive a query for the request and a keyword cache tensor for the request to generate a multiplied tensor for the request. In one example, the operation of the self-attention block 325 for the request is given by:

[0078]

[0079] in, represents the multiplication tensor for request b, is the reshaped query tensor Q for request b b , which concatenates the queries of each attention head for request b, and is the reshaped keyword cache tensor Kcache for request b b , which concatenates the keyword cache tensors of each attention head for request b.

[0080] The self-attention block 325 is then coupled to receive the multiplied tensor and the value buffer tensor for the request and generate an attention output tensor for the request. In one example, the operation of the self-attention block 325 for the request is given by:

[0081] If k≤j

[0082] P bi，j，k =0 if k>j,

[0083] in, represents the received multiplication tensor for request b, and

[0084]

[0085] in, represents the multiplication tensor for request b, is the reshaped value tensor Vcache for request b b , which concatenates the value cache tensors of each attention head for request b. Then, the output Y is b Reshape to generate the attention output Z' b :

[0086] Z′ bj，hi+k =Y bi，j，k

[0087] in, and

[0088] In particular, the operation of the self-attention block 325 may be incompatible with batch operations for requests with different lengths because the operation of the self-attention block 325 is a multiplication between the query, keyword cache, and value cache for the request, rather than a tensor multiplication between a fixed weight tensor and one or more input tensors. Therefore, the attention output for the requests in the batch is generated separately. For example, the execution engine can use the input tensors for the first request (e.g., query tensor Q1, keyword cache tensor Kcache1, value cache tensor Vcache1) to generate a first attention output for the first request, and separately, the execution engine can use the input tensors for the second request at a separate step to generate a second attention output for the second request. In particular, the second attention output can be generated at an execution engine different from the execution engine that processes the first request, generated at a hardware accelerator different from the hardware accelerator that processes the first request in the same execution engine, generated at a GPU core different from the GPU core that processes the first request in the same hardware accelerator, or generated at the same GPU core as the GPU core that processes the first request in the same hardware accelerator, but the implementation is not limited to this.

[0089] Since the subsequent operations in the attention linear block 330 are configured as batch operations, the reasoning system 130 concatenates the attention outputs of the requests in the batch into a single attention output tensor. In one example, the concatenation operation is given by:

[0090]

[0091] in, Represents a single attention output tensor. Figure 3A As shown, the first and second rows of the concatenated attention output tensor Z' correspond to the attention output of the first request, the third row corresponds to the attention output of the second request, and the fourth to sixth rows correspond to the attention output of the third request.

[0092] The attention linear block 330 is coupled to receive the attention output tensor as an input tensor and generate an output tensor by applying the attention weight tensor as a trained parameter set of the transformer model 300 to the attention output tensor. The attention linear block 330 can be configured as one or more neural network layers. The attention linear block 330 can be executed as a batch operation. In one example, the operation of the attention linear block 330 is given by:

[0093]

[0094] in, represents the attention output tensor from the self-attention block 325, represents the output tensor, is the attention weight tensor, and (bias). As an example, in Figure 3A , the first to second rows of the output tensor Z include output data for the first request, the third row includes output data for the second request, and the fourth to sixth rows include output data for the third request.

[0095] The first addition block 335 combines the concatenated input tensor including the input token sequence and the output tensor from the attention linear block 330 to generate a combined tensor. The second layer normalization block 340 is coupled to receive the combined tensor from the addition block 335 and normalize each requested element to generate a normalized tensor as an output tensor.

[0096] and FIG. 2A to FIG. 2B Similar to the converter model 200 of the decoder, the decoder may also include an MLP block 342, which includes one or more blocks including additional neural network layers. The MLP block 342 is executed as one or more batch operations. The MLP block 342 also includes a first MLP block 345 configured as one or more neural network layers. The first MLP block 345 is coupled to receive the normalized tensor from the second layer normalization block 340, and generates an output tensor by applying the MLP weight tensor, which is a trained parameter set of the converter model 200, to the normalized tensor.

[0097] The MLP block 342 also includes a GeLU block 350, which is coupled to receive the output tensor from the first MLP block 345 as an input tensor and generates an output tensor by applying a GeLU function to the input tensor. The MLP block 342 also includes a second MLP block 355 configured as one or more neural network layers. The second MLP block 355 is coupled to receive the output tensor from the GeLU block 350 as an input tensor and generates an output tensor by applying a second MLP weight tensor that is a trained parameter set of the converter model 300. The second addition block 360 combines the output tensor from the second MLP block 355 and the output tensor from the first addition block 335 to form a combined tensor.

[0098] The combined tensor includes a set of output representations for the first decoder D1. The set of output representations is propagated to subsequent decoders and a similar process is repeated to generate a set of output representations for the final decoder DN. Specifically, the reasoning system 130 can also deploy a selective batching method for the remaining decoders, wherein the self-attention operation is performed without batching, but the remaining operations are batched. The LM header block 370 is coupled to receive a set of output representations from the final decoder DN and generate an output tensor including a set of output tokens

[0099] like Figure 3B As shown, the next iteration is the decoding phase, in which the previous output tokens generated at the previous iteration are input to the transformer model 300 to generate another set of output tokens. Specifically, the layer normalization block 310 is applied to the output tensor to generate a normalized tensor. The QKV operation block 315 is applied to the normalized tensor to generate an output tensor including the query, key, and value for the current iteration. The split block 320 is applied to split the output tensor into one or more query tensors for each request in the batch, one or more key tensors for each request in the batch, and one or more value tensors for each request in the batch. The inference system 130 adds each key tensor to the key cache tensor for the corresponding request and adds each value tensor to the value cache for the corresponding request. Figure 3B The iterations of the decoding phase for all requests in a batch are depicted, so the requests have the same length for a set of inputs fed to the transformer model 300 (i.e., a single output token for each request generated from the previous iteration). However, the internal states including the key cache tensor and the value cache tensor may still have different lengths for the requests in the batch, because Figure 3A The input token lengths for the requests in the encoding phase shown are different. However, Figure 3A Similarly, reasoning system 130 can still process a batch of requests with different internal state lengths by selectively batching operations in transformer model 300 .

[0100] The self-attention block 325 is applied separately to the query tensor, keyword cache tensor, and value cache tensor for each corresponding request to generate an attention output tensor for the request. The attention outputs for these requests are concatenated into the attention output tensor. The attention linear block 330 is applied to the attention output tensor to generate an output tensor. The addition block 335 combines the concatenated input tensor and the output tensor from the attention linear block 330 to generate a combined tensor. The MLP block 342 is applied to the combined tensor to generate a set of output representations for decoder D1. The process is repeated for the remaining decoders and the LM head block 370 to generate an output tensor including another set of output tokens. And the converter model 300 is applied iteratively until the termination criteria is met for each request in the batch.

[0101] By selectively batching selected operations of the converter model 300, the reasoning system 130 can efficiently process a batch of requests with variable input lengths. The reasoning system 130 executes requests by processing operations such as self-attention operations separately on a per-request basis while batching the remaining operations of the converter model 300. Specifically, operations involving tensor multiplication between one or more input tensors and trained weight tensors, such as QKV linear operations, attention linear operations, or MLP operations, may require high computing power during the reasoning process due to the typically large dimensionality of the weight tensor. In addition, when requests are processed separately for these operations, the weight tensor may be read from the local memory of the execution engine multiple times. However, by configuring these subsets of operations as batch operations but processing operations that do not involve operations on weight tensors, such as self-attention operations, separately, the reasoning system 130 can take advantage of the parallel computing capabilities of hardware accelerators while allowing a high degree of flexibility in processing variable-length requests.

[0102] in addition, Figure 3A shows the iteration of the encoding phase for a set of inputs as a sequence of input tokens, and Figure 3B An iteration of the decoding phase is shown for a set of inputs that are output tokens for a batch of requests generated in a previous iteration (i.e., the encoding phase). However, unlike other methods of batching the transformer model, the selective batching method described herein flexibly allows the transformer model 300 to process a set of inputs for a batch of requests with variable input lengths, target lengths, or internal state lengths, regardless of whether the requests in the batch are processed for the encoding phase or the decoding phase or how many iterations have been processed for the request.

[0103] Specifically, you may not use FIG. 2A to FIG. 2B The method batches a batch of requests: (i) when the requests are in the encoding phase and have input token sequences of different lengths; (ii) when the requests are in the decoding phase and each request processes a token with a different index from each other (i.e., the input token length plus the index of the decoding iteration), resulting in internal states of different lengths; and (iii) when each request in the batch is in a different phase (encoding or decoding), resulting in inputs of different lengths (i.e., a sequence of input tokens for the encoding phase and a single output token for the decoding phase). Although FIG. 2A to FIG. 2B The batching approach in restricts the requests in a batch to the same stage, where the number of input tokens for the encoding stage is the same and the token index for the decoding stage is the same, but this restriction greatly reduces the feasibility of batching in real workloads.

[0104] In contrast, the selective batching approach described herein enables the converter model 300 to process requests as a batch even if the requests are at different stages or different processing indices. Thus, for example, instead of Figure 3A In receiving a set of input token sequences X1, X2, X3, the converter model 300 can perform iterations using a batch of input token sequences for a first request (encoding phase), a first output token for a second request (a first iteration of a decoding phase), and a third output token for a third request (a third iteration of a decoding phase). As another example, the converter model 300 can perform iterations using a batch of second output tokens for a first request and a fourth output token for a second request, the second output token and the fourth output token each having a different length for a keyword cache tensor and a value cache tensor. As combined FIG. 5A to FIG. 5B As described, this enables the reasoning system 130 to flexibly modify and update the batch in each iteration so that the computing power of one or more execution engines is fully utilized.

[0105] In addition, although FIG. 2A to FIG. 2B and FIG. 3A to FIG. 3B A GPT (Generative Pre-Training) type transformer model including a set of decoders is shown, but the reasoning system 130 is also capable of performing selective batching on transformer models with other types of architectures as long as the transformer model includes the following attention mechanism (e.g., self-attention or encoder-decoder attention): the attention mechanism generates an attention output using the query, keywords, and values ​​generated for the request, and generates output tokens in an autoregressive manner by using the output tokens generated at the previous iteration as input for the next iteration.

[0106] In another embodiment, the converter model may have an encoder-decoder architecture and include a set of encoders coupled to a set of decoders. In such an architecture, each encoder includes at least a self-attention block coupled to an MLP block. The self-attention block of the encoder is coupled to receive a query tensor, a keyword tensor, and a value tensor obtained from processing a set of input tokens, and generates an attention output. Each decoder includes at least a self-attention block coupled to an encoder-decoder attention block coupled to the MLP block. The encoder-decoder attention block of the decoder is coupled to receive a query tensor obtained from processing the output of the self-attention block and a keyword tensor and a value tensor obtained from processing the output of the final encoder to generate another attention output.

[0107] The reasoning system 130 may apply the encoder-decoder converter model to a batch of requests. Specifically, for the encoder, requests may be processed separately for the self-attention block while being batched for the remaining operations. For the decoder, requests may be processed separately for the self-attention block and the encoder-decoder attention block while being batched for the remaining operations.

[0108] Back to Figure 1 The client devices 110A, 110B are computing devices, for example, having an operating system such as or smartphone, tablet, laptop, desktop computer or any other type of web-enabled device. Figure 1 In the illustrated system environment 100, a client device 110 enables a user to submit a request to an inference system 130 to perform a task that can be processed by a machine learning transformer model. For example, a user of the client device 100A may be affiliated with an entity that deploys one or more applications that require NLP or other types of tasks that can be processed by a transformer architecture. The request submitted by the client device 110 may be, for example, a request to generate a prediction for a chatbot application, a request to translate a sequence of words from one language to another, a request to synthesize a story or narrative given a starting sequence of words, and the like.

[0109] A typical client device 110 includes the hardware and software required to connect to the network 122 (e.g., via WiFi and / or 4G, 5G, or other wireless telecommunications standards). Specifically, the client device 110 may include an operating system and various applications running on the operating system that enable users to submit requests. For example, the client device 110 may include a browser application or a stand-alone application deployed by the reasoning system 130 that enables users of the organization to interact with the reasoning system 130 to submit requests.

[0110] The network 122 provides a communication infrastructure between the client device 110 and the online system 130. The network 122 is typically the Internet, but can be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile wired or wireless network, a private network, or a virtual private network.

[0111] Block diagram of the inference system

[0112] Figure 4 is a block diagram of the architecture of the reasoning system 130 according to an embodiment. Figure 4The illustrated reasoning system 130 includes a data management module 420, a training module 430, and a service system 435. The reasoning system 130 also includes a training corpus 460. In alternative configurations, different and / or additional components may be included in the reasoning system 130.

[0113] The data management module 420 manages a training corpus 460 of training data for training the parameters of the transformer model. In one embodiment, the training corpus 460 includes a plurality of data instances, each instance including a set of training input data and a set of training output data corresponding to known predictions of the set of training input data, the known predictions solving the task for which the transformer model should be trained. In one embodiment, the training input data and the training output data are text data, but the embodiments are not limited thereto, and the training data may include any type of data that the transformer model is trained to process in an autoregressive manner, such as audio data or image data. The training module 430 encodes the set of training input data into a set of training input tokens, and encodes the set of training output data into a set of training output tokens. Each token may represent a corresponding word in a latent space.

[0114] For example, where the transformer model is used for automatic translation from English to French, the set of training input tokens may correspond to a first sentence of English words, and the set of training output tokens may correspond to a second sentence of French words, which is a translation of the first sentence. As yet another example, where the transformer model is used to convert images to text, the set of training input tokens may correspond to pieces of image data for training images, and the set of training output tokens may correspond to converted sentences or paragraphs describing the images.

[0115] The execution engine module 425 includes one or more execution engines built on a dedicated hardware accelerator such as a graphics processing unit (GPU) or a tensor processing unit (TPU). The execution engine receives a request to execute one or more requests on the dedicated hardware. In one example, the execution engine receives a batch of requests and executes one or more iterations of the converter model via selective batch execution using the input for each request. As described above, the execution engine may include a set of cores (e.g., GPU cores) coupled to a local memory (e.g., GPU memory). Specifically, for each operation of the iteration, the execution engine is configured to read the data required for the operation from the local memory of the execution engine.

[0116] The execution engine of the execution engine module 425 can be configured as a GPU or TPU or any other dedicated hardware capable of processing neural network operations (including tensor multiplication and floating point operations) in an efficient manner. For example, the execution engine is not limited to Google TPU, Graphcore IPU, Cerebras WSE, Nvidia GPU, Intel Nervana, Qualcomm Cloud AI, Samsung Exynos, AMD Radeon, Xilinx AI acceleration card, IBM TrueNorth, AWS Trainium, etc. As described above, the execution engine can be composed of one or more hardware accelerators. In some instances, the execution engine can process workloads that require the processing power of more than one hardware accelerator. Therefore, if necessary, the execution engine can distribute workloads on multiple hardware accelerators.

[0117] In one embodiment, the execution engine is further configured to manage one or more caches on local memory required to execute one or more iterations of the converter model. FIG. 3A to FIG. 3B When the converter model 300 is executed, the execution engine maintains the keyword cache tensor and value cache tensor for the request until the request has been completed. Therefore, in response to receiving one or more new requests (requests for which the encoding phase has not yet been processed), the execution engine allocates memory to each request for maintaining the keyword cache tensor and value cache tensor for the request. For each iteration of the decoding phase, the execution engine can add the keyword tensor and value tensor for the iteration to the corresponding keyword cache and value cache allocated for the request, and retrieve the keyword cache and value cache for the request to perform the operation. In response to completing one or more requests, the execution engine can provide the output for the request to the appropriate module of the reasoning system 130, and release the allocated cache memory for the completed request so that the released memory can be used for other requests.

[0118] The training module 430 trains the parameters of the converter model by executing a training process. First, the training module 430 can generate an appropriate architecture for the converter model, which can solve the task request received by the client device 110. For example, the training module 430 can FIG. 3A to FIG. 3B The transformer model 300 shown in FIG. 4 is trained. The training module 430 can determine the number of decoders in the set of decoders to adjust the depth of the transformer model and determine, for example, how many layers are included in each neural network layer of the transformer model. In the case where the transformer model to be trained is an encoder-decoder architecture, the training module 430 can determine the number of encoders and the number of decoders.

[0119] The training module 430 may also initialize one or more weight tensors associated with the operation of the transformer model. The elements of the weight tensor correspond to the parameters of the transformer model that will be learned during the training process using the training data of the training corpus 460. The weight tensor may be initialized with the dimension based on the dimension of the input tensor with which the weight tensor is configured to operate. For example, to train FIG. 3A to FIG. 3B The training module 430 can train the QKV weight tensor W for the QKV operation block 315. QKV (e.g., dimension H×3H), the attention weight tensor W for the attention linear block 330 attn (e.g., of dimension H×H) and a first MLP weight tensor for the first MLP block 345 and a second MLP weight tensor for the second MLP block 355 are initialized.

[0120] During the training process, the training module 430 obtains a set of training data and trains the parameters of the machine learning converter model by repeatedly iterating between a forward pass step and a back-propagation step. During the forward pass step, the training module 430 generates one or more estimated output tokens by applying the estimated parameters of the converter model to a set of training input tokens in the set of training data to generate one or more estimated output tokens. The training module 430 determines a loss function indicating the difference between the one or more estimated output tokens and the set of training output tokens. During the back-propagation step, the training module 430 updates the parameters (i.e., weight tensors) of the converter model to reduce the loss function. The process is iteratively repeated for the next set of training data until convergence is reached for the parameters of the converter model or a desired level of model quality is reached.

[0121] In one embodiment, the training module 430 arranges the training data in batches and executes one or more iterations of the training process on one or more execution engines. For example, each batch of training data may include a batch of input token sequences each from a different training source. For example, a first training input token sequence may be obtained from a first text source, and a second training input token sequence may be obtained from a second text source. By performing the training process in batches, the training module 430 can take advantage of the parallel processing capabilities of the execution engines when training a highly computationally demanding transformer model.

[0122] The service system 435 receives a request from the client device 110 to perform one or more tasks that can be processed using the trained transformer model. FIG. 3A to FIG. 3BAs described, when the task is an NLP application, the request may include a sequence of words (e.g., an English sentence), which the service system 435 may parse and map into a sequence of input tokens that may be provided to the converter model. The service system 435 processes the request to generate one or more output tokens using the converter model. The service system 435 may convert the output tokens into output data such as a sequence of words (e.g., a sentence translated in French), and return the output data to the client device as a response.

[0123] In one embodiment, the service system 435 receives multiple requests from the client device 110 over time and forms multiple batches of requests. The service system 435 provides these batches to one or more execution engines. In response to the execution of the execution engine, the service system 435 receives one or more output tokens for the request and provides these outputs to the client device 110 as a response to the request. In one embodiment, the length of the internal state of the request in the batch formed by the service system 435 or the length of the input sequence of the request in the batch can be different from each other, because the execution engine can process these requests using a selective batching method. In one embodiment, the service system 435 waits until the request is completed (i.e., the termination criteria are met) and provides the output to the client device 110 as a response to the request. In another embodiment, even if the request has not been completed, the service system 435 also provides the output to the client device 110. For example, the service system 435 can provide a certain number of output tokens (e.g., 5, 10, 15) each time the output token is generated for the request. As another example, for one or more time intervals (e.g., 100ms, 200ms), the service system 435 may provide an output token generated after each time interval to the client device 110. This enables the user of the client device 110 to receive intermittent output even if the request has not yet been completed.

[0124] In one embodiment, the service system 435 performs iteration-level dynamic batching for the converter model, which enables the service system 435 to dynamically modify a batch of requests being executed on the execution engine on a per-iteration basis. Specifically, for some batching methods, it may be difficult to modify a batch of requests once processing begins on the execution engine because such methods require that the length of the input or the length of the internal state of the converter model be the same between the requests in the batch. That is, unless the newly incoming request has the same length of input and internal state as the batch of requests being executed, it is difficult for the inference system to modify the batch, for example, to add a new request to the batch.

[0125] By performing selective batching, the service system 435 can monitor and modify the batches processed on the execution engine at each iteration. In particular, at one or more iterations, the service system 435 can modify the batch by adding new incoming requests to the batch being executed on the execution engine or removing completed requests from the batch. This is because selective batching enables processing of requests with variable lengths without restricting the input or internal state of the converter model to the same length. This enables the service system 435 to provide responses to completed requests to the requesting client device 110 earlier, and enables adding new requests to the batch even when the execution engine processing the batch is not fully utilized, even when the length of the new request is different from the length of the existing request for the next iteration.

[0126] Dynamic Batching for Transformer Models

[0127] FIG. 5A to FIG. 5D A method for dynamically batching requests using a machine learning converter model to process requests according to an embodiment is shown. In one embodiment, the service system 435 includes a request processor 580 and a scheduler 585, each of which is coupled to one or more execution engines. The request processor 580 receives a request and forwards the request to the scheduler 585. The request processor 580 maintains a completion queue for storing the output of completed requests. Specifically, the request processor 580 receives the output of the completed request from the execution engine and stores these outputs in the completion queue so that they can be provided to the client device 110 of the request. The scheduler 585 receives the forwarded request from the request processor 580 and maintains an incoming request queue for storing new requests to be processed. The scheduler 585 forms a batch of requests and schedules the batch for execution on the execution engine. In one embodiment, the scheduler 585 is configured to monitor which batch of requests is assigned to each execution engine, how many iterations of the converter model have been executed for each request, and whether the request has been completed. Scheduler 585 is also configured to monitor available cache memory in each execution engine.

[0128] Specifically, Figure 5A Request processor 580 and scheduler 585 are shown coupled to execution engines 590A and 590B. Figure 5A In FIG. 5 , execution engine 590A is scheduled to execute a single request R1, and execution engine 590B is scheduled to execute a batch of requests R3, R4, and R5. Specifically, for Figure 5AIn the first iteration of , request R1 in execution engine 590A includes a single input token. On the other hand, request R3 in execution engine 590B includes a sequence of two input tokens, request R4 includes a sequence of three input tokens, and request R5 includes a sequence of two input tokens.

[0129] The request processor 580 receives a new request R2 and forwards the request R2 to the scheduler 585. The scheduler 585 stores the request R2 in the incoming request queue. The iteration of the converter model is executed on the execution engine 590A, 590B. For example, the iteration may be the encoding phase for a request in the batch currently being executed. In particular, a keyword cache tensor and a value cache tensor are allocated as an internal state cache for each request, and a portion of the cache for each request is used to store keywords and values ​​after the encoding phase. In one embodiment, the allocation of the internal state cache for the request is based on the maximum output token length that the converter model is configured to generate. For example, the converter model can be configured to generate a maximum number of 1024 output tokens, and the execution engine can allocate an internal state cache configured to store up to 1024 elements for each keyword cache tensor and value cache tensor. In another embodiment, in the case where the user of the request specifies the maximum length of the output token that can be returned as a response, the execution engine can allocate an internal state cache configured to store up to the maximum length of elements specified in the request for each tensor. In yet another embodiment, the execution engine may determine an initial number of input tokens for a request, and the execution engine may initially allocate an internal state cache that is large enough to store that number of input tokens plus some additional amount of memory. If the initially allocated memory is insufficient, the execution engine may incrementally allocate more memory to the internal state cache as the transformer model is iteratively applied to the request. The scheduler 585 monitors cache memory for execution engines 590A and 590B. In response to determining that execution engine 590A has cache memory available to process request R2, the scheduler 585 updates the batch for execution engine 590A to include the new request R2, and instructs execution engine 590A to execute the updated batch for the next iteration.

[0130] like Figure 5BAs shown, after executing the first iteration, a first output token is generated for request R1 in execution engine 590A and requests R3, R4, R5 in execution engine 590B. In addition, execution engine 590A is now scheduled to execute the updated batch of requests R1, R2 at the second iteration. Request R2 includes a single input token. In particular, since the first output token has been generated for request R1, the second iteration for request R1 can be a decoding phase, while the second iteration for request R2 can be an encoding phase. Therefore, since the execution engine is configured to perform selective batching and is capable of processing requests with different lengths, execution engine 590A can perform both the encoding phase and the decoding phase for the same batch of requests, as shown in combination with FIG. 3A to FIG. 3B This may not be feasible for other batching methods, since there is no guarantee that the length of the input token for one request will be the same as the length of the input token and output token for an existing request in the batch. Figure 5B As shown, an additional portion of the cache for each request is used to store keys and values ​​after the second iteration.

[0131] like Figure 5C As shown, after executing the second iteration, a second output token is generated for request R1 in execution engine 590A and requests R3, R4, and R5 in execution engine 590B. A first output token is generated for request R2 in execution engine 590A. Specifically, the first output token generated for request R2 is generated together with the end token, and execution engine 590A provides the output for request R2 to the completion queue of request processor 580. Execution engine 590A releases the cache memory assigned to request R2. The second output token generated for request R4 is also generated together with the end token, and execution engine 590B provides the output for request R4 to the completion queue of request processor 580. Execution engine 590B releases the cache memory assigned to request R4. Request processor 580 forwards the output of the completion queue to the client device 110 that has completed the request.

[0132] The request processor 580 also receives another new request R7 and forwards the request to the scheduler 585. The scheduler 585 stores the request R7 in the incoming request queue. In response to determining that the requests R2, R4 have been completed and the execution engine 590A has cache memory available for processing the request R7, the scheduler 585 updates the batch for the execution engine 590A to R1, R7, and updates the batch for the execution engine 590B to R3, R5. The scheduler 585 instructs the execution engines 590A, 590B to execute the updated batch. Thus, by dynamically adjusting the batch at the time of iteration, the completed request can be provided to the requesting client device 110 immediately after the processing is completed, and the scheduler 585 can schedule new requests within the batch so that the computing power of the execution engine can be utilized when the memory of the completed request is released.

[0133] like Figure 5D As shown, after executing the third iteration, a third output token is generated for request R1 in execution engine 590A and requests R3, R5 in execution engine 590B. In addition, execution engine 590A is now scheduled to execute the updated batch of requests R1, R7 at the fourth iteration, and execution engine 590B is now scheduled to execute the updated batch of requests R3, R5 at the fourth iteration. Request R7 includes a sequence of two input tokens. When service system 435 receives a request from client device 110, it can continuously execute the request with FIG. 5A to FIG. 5D A process similar to the one described in .

[0134] As described above, dynamic batching enables the service system 435 to dynamically adjust the batches processed on the execution engine so that the hardware of the execution engine can be fully utilized. This process is useful for some methods of batching for the converter model (e.g., FIG. 2A to FIG. 2B ) is unavailable because internal states such as key cache tensors and value cache tensors maintain variable lengths at each iteration in the transformer model, while other recursive machine learning models (e.g., RNNs) utilize state data with the same length between batches of requests.

[0135] Flowchart showing methods of selective batching and dynamic batching

[0136] FIG. 6A to FIG. 6B 604 , a machine learning transformer model comprising at least one set of decoders.

[0137] For one or more iterations, the reasoning system 130 repeatedly performs the step of applying the set of decoders to one or more inputs of the request. Specifically, for at least one decoder in the set, the reasoning system 130 generates 606 one or more queries, one or more keywords, and one or more values ​​for the request by applying the QKV weight tensor to the one or more input representations. The queries, keywords, and values ​​may be generated by batch operations. The reasoning system 130 splits 608 a first query for the first request from the one or more queries, splits 608 a first keyword for the first request from the one or more keywords, and splits 608 a first value for the first request from the one or more values. The reasoning system 130 also splits a second query for the second request from the one or more queries, splits a second keyword for the second request from the one or more keywords, and splits a second value for the second request from the one or more values.

[0138] The reasoning system 130 generates 610 a first attention output for a first request by combining at least a first query, a first keyword, and a first value. The reasoning system 130 separately generates 612 a second attention output for a second request by combining a second query, a second keyword, and a second value for the second request. The reasoning system concatenates 614 at least the first attention output and the second attention output into a concatenated tensor. The reasoning system 130 generates 616 one or more output representations by applying a weight tensor to the concatenated tensor. The one or more output representations may be generated by a batch operation. The reasoning system 130 sets 618 one or more output tokens as one or more inputs to the group of decoders for a subsequent iteration. The reasoning system 130 provides 620 the output tokens generated for at least one request to the client device 110 as a response to the at least one request.

[0139] Figure 7 is a flow chart illustrating a method for processing dynamic batching of requests using a transformer model according to an embodiment. The reasoning system 130 receives 710 one or more requests for execution through a service system. The service system includes a scheduler and one or more execution engines, each of which is coupled to access a machine learning transformer model including at least one set of decoders. The reasoning system 130 schedules 712 a batch of requests including one or more requests through the scheduler for execution on the execution engine. The reasoning system 130 generates 714 a first set of output tokens by the execution engine by iteratively applying the transformer model to a first set of inputs of the batch of requests. Applying the transformer model includes applying at least one batch operation to one or more input tensors associated with the batch of requests.

[0140] The reasoning system 130 receives 716 a new request from a client device via a request processor. The new request may include a sequence of input tokens. The reasoning system 130 schedules 718 a second batch of requests including one or more requests and the new request via a scheduler for execution on the execution engine in response to determining that the execution engine has memory available to execute the second batch of requests including the one or more requests and the new request. The reasoning system 130 generates 720 a second set of output tokens by the execution engine by iteratively applying the transformer model to a second set of inputs for the second batch of requests including the sequence of input tokens for the new request.

[0141] Hardware Components

[0142] Figure 8 8 is a diagram illustrating a computer system 800 on which embodiments described herein may be implemented within the inference system 130. For example, Figure 1 In the context of Figure 8 The reasoning system 130 and its subsystems may also use multiple Figure 8 The invention can be implemented by a combination of the computer systems described.

[0143] In one implementation, the computer system 800 includes processing resources 801, a main memory 803, a read-only memory (ROM) 805, a storage device 807, a communication interface 809, and a hardware accelerator 810. The computer system 800 includes at least one processor 801 and a main memory 803, the processor 801 including a CPU core for processing information, and the main memory 803, such as a random access memory (RAM) or other dynamic storage device, for storing information and instructions to be executed by the processor 801. The main memory 803 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 801. The computer system 800 may also include a ROM 805 or other static storage device for storing static information and instructions for the processor 801. A storage device 807, such as a magnetic disk or optical disk or a solid-state storage device, is provided for storing information and instructions. For example, in Figure 4 In the context of , the training corpus storage 460 of the reasoning system 130 can be stored in a read-only memory (ROM) 805 or a storage device 807. The computer system 800 also includes one or more hardware accelerators 810. As described above, while the execution engine can be configured with one or more hardware accelerators 810 in a single computer system 800, in another embodiment, the execution engine can be configured across multiple hardware accelerators 810 distributed across multiple computer systems 800.

[0144] The communication interface 809 may enable the reasoning system 130 to communicate with the client device 110 by using a communication link (wireless or wired). Using the communication link, the reasoning system 130 may communicate with different subsystems included in the client device 110 to enable receiving and processing requests. In addition, in the case where the execution engine is configured across multiple hardware accelerators distributed on multiple computer systems 800, the communication interface 809 may also enable communication between the multiple computer systems 800 so that the execution engine can process requests across multiple hardware accelerators 810.

[0145] The computer system 800 may optionally include a display device 811, such as a cathode ray tube (CRT), LCD monitor, LED monitor, TFT display, or television, for displaying graphics and information to a user, for example. An input mechanism 813, such as a keyboard including alphanumeric keys and other keys, may optionally be coupled to the computer system 800 for communicating information and command selections to the processor 801. Other non-limiting, illustrative examples of the input mechanism 813 include a mouse, a trackball, a touch-sensitive screen, or cursor direction keys for communicating direction information and command selections to the processor 801 and for controlling cursor movement on the display device 811.

[0146] The examples described herein involve using reasoning system 130 to implement the techniques described herein. According to one embodiment, these techniques are performed by reasoning system 130 in response to processor 801 executing one or more sequences of one or more instructions contained in main memory 803. Such instructions may be read into main memory 803 from another machine-readable medium, such as storage device 807. Execution of the sequences of instructions contained in main memory 803 causes processor 801 to perform the processing steps described herein. In alternative implementations, hard-wired circuitry may be used in place of or in combination with software instructions to implement the examples described herein. Thus,

[0147] The described examples are not limited to any specific combination of hardware circuitry and software.

[0148] The present invention also discloses the following technical solutions:

[0149] 1. A method for executing a batch of requests by applying a machine learning transformer model, comprising:

[0150] receiving a batch of requests including one or more token sequences, wherein a length of a first token sequence for a first request in the batch is different than a length of a second token sequence for a second request in the batch;

[0151] accessing a machine learning transformer model including at least a set of decoders;

[0152] For one or more iterations, repeatedly perform the following steps:

[0153] generating one or more output tokens for the request by applying the group of decoders to one or more inputs for the request, the generating further comprising, for at least one decoder in the group:

[0154] generating one or more queries, one or more keywords, and one or more values ​​for the request by applying the QKV weight tensor to the one or more input representations, the one or more queries, the one or more keywords, and the one or more values ​​being generated by a batch operation,

[0155] splitting a first query for the first request from the one or more queries, splitting a first keyword for the first request from the one or more keywords, and splitting a first value for the first request from the one or more values, and splitting a second query for the second request from the one or more queries, splitting a second keyword for the second request from the one or more keywords, and splitting a second value for the second request from the one or more values,

[0156] generating a first attention output for the first request by combining at least the first query, the first keyword, and the first value,

[0157] separately generating a second attention output for the second request by combining at least a second query, a second keyword, and a second value for the second request,

[0158] concatenate at least the first attention output and the second attention output into a concatenated tensor, and

[0159] generating one or more output representations by applying at least one or more weight tensors to the connection tensors, the one or more output representations being generated by a batch operation;

[0160] setting the one or more output tokens as one or more inputs of the group decoder for a following iteration; and

[0161] An output token generated for the at least one request is provided to the client device as a response to the at least one request.

[0162] 2. The method according to technical solution 1 further includes:

[0163] concatenating the tokens included in the one or more token sequences for the batch request along a single dimension to generate a concatenated input tensor, and

[0164] Wherein, for a first iteration of the one or more iterations, applying the group decoder to one or more inputs for the request comprises: applying the group decoder to the concatenated input tensor.

[0165] 3. A method according to technical solution 2, wherein connecting at least the first attention output and the second attention output into the connected tensor further includes: connecting the elements of the first attention output and the elements of the second attention output along a single dimension to generate the connected tensor.

[0166] 4. The method according to technical solution 1, wherein the first token sequence of the first request is an input token sequence for the first request, and wherein the second token sequence of the second request is an output token generated at a previous iteration.

[0167] 5. The method according to technical solution 1,

[0168] The generating a first attention output for the first request includes:

[0169] adding the first key to a first key cache, the first key cache additionally comprising a key generated for the first request at a previous iteration, and adding the first value to a first value cache, the first value cache additionally comprising a value generated for the first request at a previous iteration, and

[0170] multiplying the first query by the first keyword cache to generate a first multiplied tensor, and multiplying the first multiplied tensor by the first value cache, and

[0171] The generating a second attention output for the second request includes:

[0172] adding the second key to a second key cache, the second key cache additionally comprising the key generated for the second request at a previous iteration, and adding the second value to a second value cache, the second value cache additionally comprising the value generated for the second request at a previous iteration, and

[0173] The second query is multiplied with the second key cache to generate a second multiplied tensor, and the second multiplied tensor is multiplied with the second value cache.

[0174] 6. A method according to technical solution 1, wherein the first attention output is generated by performing a set of operations on at least the first query, the first keyword and the first value, and wherein the second attention output is generated by performing a second set of operations on at least the second query, the second keyword and the second value in a step separate from the first set of operations.

[0175] 7. A method according to technical solution 6, wherein the first attention output is generated by executing the set of operations on a first core of an execution engine, and wherein the second attention output is generated by executing the second set of operations on a second core of the execution engine or on a second execution engine.

[0176] 8. A method according to technical solution 1, wherein applying the group decoder of the converter model to one or more inputs of the request includes: executing operations of the converter model on an execution engine including a graphics processing unit (GPU) or a tensor processing unit (TPU).

[0177] 9. A method according to technical solution 1, wherein each token in the input token sequence represents a text unit.

[0178] 10. A non-transitory computer readable storage medium storing computer program instructions executable to perform operations for executing a batch of requests by applying a machine learning transformer model, the operations comprising:

[0179] receiving a batch of requests including one or more token sequences, wherein a length of a first token sequence for a first request in the batch is different than a length of a second token sequence for a second request in the batch;

[0180] accessing a machine learning transformer model including at least a set of decoders;

[0181] For one or more iterations, repeatedly perform the following steps:

[0182] generating one or more output tokens for the request by applying the group of decoders to one or more inputs for the request, the generating further comprising, for at least one decoder in the group:

[0183] generating one or more queries, one or more keywords, and one or more values ​​for the request by applying the QKV weight tensor to the one or more input representations, the one or more queries, the one or more keywords, and the one or more values ​​being generated by a batch operation,

[0184] splitting a first query for the first request from the one or more queries, splitting a first keyword for the first request from the one or more keywords, and splitting a first value for the first request from the one or more values, and splitting a second query for the second request from the one or more queries, splitting a second keyword for the second request from the one or more keywords, and splitting a second value for the second request from the one or more values,

[0185] generating a first attention output for the first request by combining at least the first query, the first keyword, and the first value,

[0186] separately generating a second attention output for the second request by combining at least a second query, a second keyword, and a second value for the second request,

[0187] concatenate at least the first attention output and the second attention output into a concatenated tensor, and

[0188] generating one or more output representations by applying at least one or more weight tensors to the connection tensor, the one or more output representations being generated by a batch operation; setting the one or more output tokens as one or more inputs of the group decoder for a subsequent iteration; and

[0189] An output token generated for the at least one request is provided to the client device as a response to the at least one request.

[0190] 11. According to the non-transitory computer-readable storage medium of technical solution 10, the operation further includes:

[0191] concatenating the tokens included in the one or more token sequences for the batch request along a single dimension to generate a concatenated input tensor, and

[0192] Wherein, for a first iteration of the one or more iterations, applying the group decoder to one or more inputs for the request comprises: applying the group decoder to the concatenated input tensor.

[0193] 12. A non-transitory computer-readable storage medium according to technical solution 11, wherein connecting at least the first attention output and the second attention output into the connected tensor further includes: connecting the elements of the first attention output and the elements of the second attention output along a single dimension to generate the connected tensor.

[0194] 13. A non-transitory computer-readable storage medium according to technical solution 10, wherein the first token sequence of the first request is an input token sequence for the first request, and wherein the second token sequence of the second request is an output token generated at a previous iteration.

[0195] 14. The non-transitory computer-readable storage medium according to technical solution 10,

[0196] The generating a first attention output for the first request includes:

[0197] adding the first key to a first key cache, the first key cache additionally comprising a key generated for the first request at a previous iteration, and adding the first value to a first value cache, the first value cache additionally comprising a value generated for the first request at a previous iteration, and

[0198] multiplying the first query by the first keyword cache to generate a first multiplied tensor, and multiplying the first multiplied tensor by the first value cache, and

[0199] The generating a second attention output for the second request includes:

[0200] adding the second key to a second key cache, the second key cache additionally comprising the key generated for the second request at a previous iteration, and adding the second value to a second value cache, the second value cache additionally comprising the value generated for the second request at a previous iteration, and

[0201] The second query is multiplied with the second key cache to generate a second multiplied tensor, and the second multiplied tensor is multiplied with the second value cache.

[0202] 15. A non-transitory computer-readable storage medium according to technical solution 10, wherein the first attention output is generated by performing a set of operations on at least the first query, the first keyword, and the first value, and wherein the second attention output is generated by performing a second set of operations on at least the second query, the second keyword, and the second value in a step separate from the first set of operations.

[0203] 16. A non-transitory computer-readable storage medium according to technical solution 15, wherein the first attention output is generated by executing the set of operations on a first core of an execution engine, and wherein the second attention output is generated by executing the second set of operations on a second core of the execution engine or on a second execution engine.

[0204] 17. A non-transitory computer-readable storage medium according to technical solution 10, wherein applying the group decoder of the converter model to one or more inputs of the request includes: executing operations of the converter model on an execution engine including a graphics processing unit (GPU) or a tensor processing unit (TPU).

[0205] 18. A non-transitory computer-readable storage medium according to technical solution 10, wherein each token in the input token sequence represents a text unit.

[0206] Summarize

[0207] The foregoing description of embodiments of the invention has been presented for illustrative purposes; it is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Those skilled in the relevant art will appreciate that many modifications and variations are possible in light of the above disclosure.

[0208] Some parts of this description describe embodiments of the present invention in terms of algorithms and symbolic representations of operations on information. Technicians in the field of data processing typically use these algorithmic descriptions and representations to effectively convey the essence of their work to other technicians in the field. Although these operations are described functionally, computationally, or logically, these operations should be understood to be implemented by computer programs or equivalent circuits, microcodes, etc. In addition, it has also been proven that it is convenient to sometimes refer to these operational arrangements as modules without loss of generality. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0209] Any of the steps, operations or processes described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product including a computer-readable medium, the computer-readable medium containing computer program code, which may be executed by a computer processor to perform any or all of the steps, operations or processes described.

[0210] Embodiments of the present invention may also relate to devices for performing the operations herein. The device may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium that may be coupled to a computer system bus or any type of medium suitable for storing electronic instructions. In addition, any computing system mentioned in the specification may include a single processor, or may be an architecture that uses multiple processor designs to increase computing power.

[0211] Embodiments of the present invention may also relate to products produced by the computing processes described herein. Such products may include information obtained from the computing processes, wherein the information is stored on a non-transitory tangible computer-readable storage medium, and may include any embodiment of the computer program product or other data combination described herein.

[0212] Finally, the language used in the specification is selected primarily for readability and instructional purposes, and may not be selected to describe or define the subject matter of the present invention. Therefore, it is intended that the scope of the present invention is not limited by this detailed description, but rather by any claims of an application based thereon. Therefore, the disclosure of embodiments of the present invention is intended to illustrate, rather than limit, the scope of the present invention set forth in the appended claims.

Claims

1. A method for executing a batch of requests by applying a machine learning model, comprising: receiving a batch of two or more token sequences, wherein a length of a first token sequence in the batch is different from a length of a second token sequence in the batch; Access machine learning models; generating one or more output representations, the generating further comprising: obtaining one or more queries, one or more keywords, and one or more values, wherein the one or more queries are generated by applying a query weight tensor to one or more input representations, wherein at least the one or more queries are generated by a batch operation, splitting a first query for the first token sequence from the one or more queries, splitting a first keyword from the one or more keywords, and splitting a first value from the one or more values, and splitting a second query for the second token sequence from the one or more queries, splitting a second keyword from the one or more keywords, and splitting a second value from the one or more values, generating a first attention output by combining at least the first query, the first keyword, and the first value, separately generating a second attention output by combining at least the second query, the second keyword, and the second value, wherein the second attention output is generated at a different execution engine, a different hardware accelerator, a different graphics processor unit (GPU) core, or the same GPU core as the first attention output, concatenate at least the first attention output and the second attention output into a concatenated tensor, and One or more output representations are generated by applying at least one or more weight tensors to the connection tensors, the one or more output representations being generated by a batch operation.

2. The method according to claim 1, wherein: The machine learning model includes a set of decoders, and obtaining the one or more queries, the one or more keywords, and the one or more values ​​for the decoders includes: Applying a QKV weight tensor comprising the query weight tensor, the keyword weight tensor, and the value weight tensor to the one or more input representations, wherein the one or more input representations are generated based on the one or more token sequences or one or more output representations from a previous decoder.

3. The method according to claim 1, wherein: The machine learning model includes a set of encoders and a set of decoders coupled to the set of encoders, and obtaining the one or more queries, the one or more keywords, and the one or more values ​​includes: obtaining the one or more keywords and the one or more values ​​from at least one encoder of the group of encoders, wherein the one or more keywords are generated by applying a keyword weight tensor to one or more output representations of the encoder and the one or more values ​​are generated by applying a value weight tensor to the one or more output representations of the encoder; and generating the one or more queries for a decoder by applying the query weight tensor to the one or more input representations, Therein, the one or more input representations are generated from the one or more token sequences or one or more output representations from a previous decoder.

4. The method according to claim 3, further comprising: Before obtaining the one or more queries, the one or more keywords, and the one or more values, obtaining one or more third queries, one or more third keywords, and one or more third values ​​for input to the self-attention layer, An attention output is generated by at least combining the one or more third queries, the one or more third keywords, and the one or more third values, and wherein the one or more input representations for generating the one or more queries are generated based on the attention output from the self-attention layer.

5. The method according to claim 3, wherein: The encoder is the final encoder in the set of encoders.

6. The method according to claim 1, wherein: The machine learning model includes a set of encoders, and obtaining the one or more queries, the one or more keywords, and the one or more values ​​for the encoders includes: Applying a QKV weight tensor comprising the query weight tensor, the keyword weight tensor, and the value weight tensor to the one or more input representations, wherein the one or more input representations are generated based on the one or more token sequences or one or more output representations from a previous encoder.

7. The method of claim 1, further comprising concatenating tokens in the one or more token sequences for the batch along a single dimension to generate a concatenated input tensor.

8. The method according to claim 1, wherein: Concatenating at least the first attention output and the second attention output into the concatenated tensor further includes concatenating elements of the first attention output and elements of the second attention output along a single dimension to generate the concatenated tensor.

9. The method according to claim 1, wherein: The first attention output is generated by performing a set of operations on at least the first query, the first keyword, and the first value, and wherein the second attention output is generated by performing a second set of operations on at least the second query, the second keyword, and the second value, and wherein the set of operations are performed on a first core of an execution engine, and wherein the second set of operations are performed on a second core of the execution engine or on a second execution engine.

10. The method according to claim 1, wherein: The first attention output is generated by performing a set of operations on at least the first query, the first keyword, and the first value, and wherein the second attention output is generated by performing a second set of operations on at least the second query, the second keyword, and the second value, and wherein the set of operations are performed on a first core of an execution engine, and wherein the second set of operations are performed on the first core of the execution engine.

11. A non-transitory computer readable storage medium storing computer program instructions executable to perform operations comprising: receiving a batch of two or more token sequences, wherein a length of a first token sequence in the batch is different from a length of a second token sequence in the batch; Access machine learning models; generating one or more output representations, the generating further comprising: obtaining one or more queries, one or more keywords, and one or more values, wherein the one or more queries are generated by applying a query weight tensor to one or more input representations, the one or more queries are generated by a batch operation, splitting a first query for the first token sequence from the one or more queries, splitting a first keyword from the one or more keywords, and splitting a first value from the one or more values, and splitting a second query for the second token sequence from the one or more queries, splitting a second keyword from the one or more keywords, and splitting a second value from the one or more values, generating a first attention output by combining at least the first query, the first keyword, and the first value, separately generating a second attention output by combining at least the second query, the second keyword, and the second value, wherein the second attention output is generated at a different execution engine, a different hardware accelerator, a different graphics processor unit (GPU) core, or the same GPU core as the first attention output, concatenate at least the first attention output and the second attention output into a concatenated tensor, and One or more output representations are generated by applying at least one or more weight tensors to the connection tensors, the one or more output representations being generated by a batch operation.

12. The non-transitory computer-readable storage medium of claim 11, wherein: The machine learning model includes a set of decoders, and obtaining the one or more queries, the one or more keywords, and the one or more values ​​for the decoders includes: Applying a QKV weight tensor comprising the query weight tensor, the keyword weight tensor, and the value weight tensor to the one or more input representations, wherein the one or more input representations are generated based on the one or more token sequences or one or more output representations from a previous decoder.

13. The non-transitory computer-readable storage medium of claim 11, wherein: The machine learning model includes a set of encoders and a set of decoders coupled to the set of encoders, and obtaining the one or more queries, the one or more keywords, and the one or more values ​​includes: obtaining the one or more keywords and the one or more values ​​from at least one encoder of the group of encoders, wherein the one or more keywords are generated by applying a keyword weight tensor to one or more output representations of the encoder and the one or more values ​​are generated by applying a value weight tensor to the one or more output representations of the encoder; and generating the one or more queries for a decoder by applying the query weight tensor to the one or more input representations, Therein, the one or more input representations are generated from the one or more token sequences or one or more output representations from a previous decoder.

14. The non-transitory computer-readable storage medium of claim 13, further comprising: Before obtaining the one or more queries, the one or more keywords, and the one or more values, obtaining one or more third queries, one or more third keywords, and one or more third values ​​for input to the self-attention layer, An attention output is generated by at least combining the one or more third queries, the one or more third keywords, and the one or more third values, and wherein the one or more input representations for generating the one or more queries are generated based on the attention output from the self-attention layer.

15. The non-transitory computer readable storage medium of claim 13, wherein: The encoder is the final encoder in the set of encoders.

16. The non-transitory computer-readable storage medium of claim 11, wherein: The machine learning model includes a set of encoders, and obtaining the one or more queries, the one or more keywords, and the one or more values ​​for the encoders includes: Applying a QKV weight tensor comprising the query weight tensor, the keyword weight tensor, and the value weight tensor to the one or more input representations, wherein the one or more input representations are generated based on the one or more token sequences or one or more output representations from a previous encoder.

17. The non-transitory computer-readable storage medium of claim 11, further comprising concatenating tokens in the one or more token sequences for the batch along a single dimension to generate a concatenated input tensor.

18. The non-transitory computer-readable storage medium of claim 11, wherein: Concatenating at least the first attention output and the second attention output into the concatenated tensor further includes concatenating elements of the first attention output and elements of the second attention output along a single dimension to generate the concatenated tensor.

19. The non-transitory computer-readable storage medium of claim 11, wherein: The first attention output is generated by performing a set of operations on at least the first query, the first keyword, and the first value, and wherein the second attention output is generated by performing a second set of operations on at least the second query, the second keyword, and the second value, and wherein the set of operations are performed on a first core of an execution engine, and wherein the second set of operations are performed on a second core of the execution engine or on a second execution engine.

20. The non-transitory computer readable storage medium of claim 11, wherein: The first attention output is generated by performing a set of operations on at least the first query, the first keyword, and the first value, and wherein the second attention output is generated by performing a second set of operations on at least the second query, the second keyword, and the second value, and wherein the set of operations are performed on a first core of an execution engine, and wherein the second set of operations are performed on the first core of the execution engine.

Images