Methods of performing a batch of requests by applying a machine learning model and non-transitory computer-readable storage media

Abstract

Methods of performing a batch of requests by applying a machine learning model and non-transitory computer-readable storage media are provided. The method includes receiving a batch of two or more token sequences, accessing a machine learning model, generating one or more output representations, the generating including obtaining one or more queries, keywords, and values, splitting a first query and a second query from the queries for first and second token sequences, splitting first and second keywords from the keywords, and splitting first and second values from the values, generating a first attention output by at least combining the first query, the first keyword, and the first value, separately generating a second attention output by at least combining the second query, the second keyword, and the second value, concatenating at least the first attention output and the second attention output into a concatenated tensor, and generating the one or more output representations by at least applying one or more weight tensors to the concatenated tensor.

Description

[0001] This application is a divisional application of the application patent application with the application date of August 18, 2022, the application number of 202210994036.0, and the invention name of “Selective Batching for Inference Systems Based on Transformer-Based Generation Tasks”. BACKGROUND

[0002] The present invention relates generally to machine learning transformer neural network models, and more specifically to selective batching for transformer models.

[0003] Transformer neural network models are machine learning models used for various applications, such as natural language processing (NLP) applications that include sequential data, image processing, or audio processing applications. For example, a transformer model can receive an input token sequence representing a query and generate an output token sequence representing a response to the query. As another example, a transformer model can receive an input token sequence representing a German paragraph and generate an output token sequence representing a translation of the paragraph into English. As yet another example, a transformer model can receive an input token sequence representing a text paragraph and generate an output token sequence representing a summarized version of the text.

[0004] Generally, users of client devices submit requests to an inference system. The inference system executes a machine learning transformer model on the input of the requests (e.g., input token sequences) to generate output (e.g., output token sequences) for the requests. The inference system can return the output as a response to the requesting client devices. In one instance, the inference system executes the requests on a dedicated hardware accelerator, such as a graphics processing unit (GPU) or a tensor processing unit (TPU), to improve latency and throughput, especially when the number of parameters of the transformer model is quite large.

[0005] In one instance, the inference system processes requests in batches to achieve high processor utilization of the accelerator. Specifically, the inference system can process multiple requests in a batch together to take advantage of the amount of parallel computing units in the hardware accelerator. In many cases, the input of the requests in a batch is variable in length. For example, the number of input tokens of each request in a batch can be variable in length. However, methods of batching for transformer models generally require the length of the data of the multiple requests in a batch to be the same for processing. As a result, processing a batch of requests with variable lengths can not be feasible, or workarounds to solve the problem can result in using more resources compared to processing each request individually. SUMMARY

[0006] An inference system applies a machine learning transducer model to a batch of requests having variable input length or variable target length or variable internal state length by selectively batching a subset of operations in the transducer model but individually processing requests in the batch for the subset of operations in the transducer model. In one implementation, the operations to be individually processed are attention operations of an encoder or a decoder of the transducer model. By selective batching, the inference system can enable batch operations to be performed for a batch of requests having variable input length or target length or internal state length to leverage parallel computing capabilities of a hardware accelerator while preventing unnecessary computation that would otherwise occur for workarounds that constrain data of a batch of requests to the same length.

[0007] In particular, in one implementation, the inference system receives a batch of requests including one or more sequences of input tokens. A length of a first sequence of input tokens for a first request in the batch can be different than a length of a second sequence of input tokens for a second request. The inference system accesses a transducer model including at least one set of decoders coupled to one another. For one or more iterations, the inference system repeatedly performs a step of generating one or more output tokens for a request by applying the set of decoders to one or more inputs for the request.

[0008] For at least one decoder in the set, the inference system generates one or more queries, one or more keys, and one or more values for the request by applying a QKV weight tensor to the one or more input representations. In one instance, the queries, keys, and values are generated by a batch operation. The inference system splits at least a first query for the first request from the one or more queries, a first key for the first request from the one or more keys, and a first value for the first request from the one or more values. The inference system also splits at least a second query for the second request from the one or more queries, a second key for the second request from the one or more keys, and a second value for the second request from the one or more values.

[0009] The inference system generates a first attention output for the first request by at least combining the first query, the first key, and the first value for the first request. The inference system also separately generates a second attention output for the second request by at least combining the second query, the second key, and the second value for the second request. The inference system concatenates at least the first attention output and the second attention output into a concatenated tensor and generates one or more output representations by applying a weight tensor to the concatenated tensor. In one instance, the one or more output representations are generated by a batch operation. The inference system sets one or more output tokens as one or more inputs to the set of decoders for a next iteration and provides the output tokens generated for at least one request as a response to the at least one request to the client device.

[0010] In one implementation, the inference system performs iteration-level dynamic batching for the transducer model, which enables the inference system to dynamically modify a batch of requests being executed on the execution engine. Specifically, in existing batching approaches for transducer models, once a batch of requests is started for processing on the execution engine, it is difficult to modify the batch. This is because certain batching approaches require that the length of the input or the length of the internal state between all requests in a batch are the same. Thus, unless a newly incoming request has the same length of input as a batch of requests being executed on the execution engine, the inference system can have difficulty modifying the batch to, for example, add or remove a new request from the batch.

[0011] By performing selective batching, the inference system can monitor and modify a batch being processed on the execution engine at an iteration level and update the batch between iterations when a request is completed and a new request is received. Specifically, at one or more iterations, the inference system can modify a batch being executed on the execution engine by adding a newly incoming request to the batch or removing a completed request from the batch. This is because selective batching enables processing of requests with variable lengths without requiring one or more inputs or internal states of the transducer model to be limited to the same length. This enables the inference system to remove a request that completes earlier than other requests in a batch, enabling a response to be provided to a user more quickly, and enables the inference system to add a new request to a batch when the execution engine is not fully utilized.

[0012] In one implementation, a service system of an inference system receives one or more requests for execution. The service system can include a request processor and a scheduler each coupled to one or more execution engines for executing a machine learning trans former model including at least one set of decoders. The scheduler schedules a batch of requests including the one or more requests for execution on the execution engines. The execution engines generate a first set of output tokens by iteratively applying the transformer model to a first set of inputs for the batch of requests. In one instance, applying the transformer model includes applying at least one batch operation to one or more input tensors associated with the batch of requests.

[0013] The service system can receive a new request including an input token sequence from a client device. The scheduler schedules a second batch of requests including the one or more requests and the new request for execution on the execution engines in response to determining that the execution engines have memory available for execution of the second batch of requests. The execution engines generate a second set of output tokens by iteratively applying the transformer model to a second set of inputs for the second batch of requests. The second set of inputs can include the input token sequence for the new request. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a high-level block diagram of a system environment for an inference system according to an implementation.

[0015] Figures 2A-2B is shown a method of batched processing using a machine learning transformer model according to an implementation.

[0016] Figures 3A-3B is shown a method of selective batch processing using a machine learning transformer model according to an implementation.

[0017] Figure 4 is a block diagram of an inference system according to an implementation.

[0018] Figures 5A-5D is shown a method of dynamic batching for processing requests using a machine learning transformer model according to an implementation.

[0019] Figures 6A-6B is a flow diagram showing a method of selective batch processing using a transformer model according to an implementation.

[0020] Figure 7 is a flow diagram showing a method of dynamic batching for processing requests using a transformer model according to an implementation.

[0021] Figure 8 is a diagram showing a computer system on which the implementations described herein can be implemented within an inference system according to an implementation.

[0022] These figures depict various embodiments of the application for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein can be employed without departing from the principles described herein. DETAILED DESCRIPTION

[0023] SUMMARY

[0024] Figure 1 is a high-level block diagram of a system environment 100 for an inference system 130 according to an embodiment. Figure 1 The illustrated system environment 100 includes one or more client devices 110A, 110B, a network 12, and an inference system 130. In alternative configurations, different or additional components can be included in the system environment 100.

[0025] The inference system 130 receives requests from the client devices 110A, 110B to perform a task using a machine learning model. In one embodiment, the machine learning model is a transducer neural network model. The task can include, but is not limited to, natural language processing (NLP), image processing, audio processing applications. Specifically, the transducer model can be adapted to process sequential data that can be tokenized into an input token sequence for a request and an output token sequence for a desired response. The inference system 130 receives a request that includes input data (e.g., textual data, image or video data, audio data) and encodes the input data into a set of input tokens. The inference system 130 repeatedly applies the machine learning transducer model for one or more iterations to generate a set of output tokens. The inference system 130 decodes the set of output tokens into output data and returns the output data as a response to the request. While for applications such as NLP applications, the input token sequence or the output token sequence is arranged along a one-dimension (1-D) to represent, for example, a sequence of words, it should be understood that in other embodiments, the input token sequence or the output token sequence can be a multi-dimensional sequence. For example, for two-dimensional image data, the token sequence can be a two-dimensional (2-D) sequence arranged along a first direction (e.g., X-axis) and a second direction (e.g., Y-axis), where each token corresponds to a block of one or more pixels in the image.

[0026] In particular, NLP tasks involve the use of artificial intelligence and machine learning techniques to analyze language, and can include various tasks including translation, sentiment analysis, text summarization, autocorrection, etc. When processing NLP tasks, the inference system 130 receives a request (e.g., a query) that includes an input text of a sequence of words, and encodes the input text into a sequence of input tokens that each represent a respective word in a latent space. The inference system 130 repeatedly applies a transformer model for one or more iterations to generate a sequence of output tokens (e.g., a response to the query). The output tokens are converted into output text as a response to the request.

[0027] For example, a transformer model can receive a sequence of input tokens representing a query and generate a sequence of output tokens representing a response to the query. As another example, a transformer model can receive a sequence of input tokens representing a passage in French and generate a sequence of output tokens representing a translation of the passage into English. As yet another example, a transformer model can receive a sequence of input tokens representing a passage of text and generate a sequence of output tokens representing a summarized version of the text.

[0028] In one implementation, the inference system 130 includes one or more execution engines built on specialized hardware accelerators, such as graphics processing units (GPUs) or tensor processing units (TPUs). Requests are executed on the execution engines. In particular, the execution of machine learning neural network models, such as transformer models, involves a large number of operations, such as tensor multiplications between input data and high-dimensional weight tensors that can be computationally intensive. The hardware accelerators of the execution engines can be optimized to efficiently perform these operations through parallel processing, resulting in a significant improvement in latency or throughput when the number of parameters in the transformer model is large.

[0029] The hardware of the inference system 130 can include one or more central processing unit (CPU) cores, CPU memory (e.g., DRAM), data storage, one or more execution engines (e.g., GPU devices). Each execution engine can include a set of cores (e.g., GPU cores) coupled to local memory (e.g., GPU memory) and can be comprised of one or more hardware accelerators. Additionally, the inference system 130 can be comprised of multiple hardware components and components described below that are used to configure the network to connect the various components on the multiple hardware components together so that the components can coordinate with each other to process requests. For example, one execution engine can communicate with multiple hardware accelerators on multiple machines. The execution engine can process data stored on its local memory. Specifically, during training or inference of the converter model, data required for inference or training is read by the CPU from input files in the data storage or from, e.g., the client device 110 over the network 120, moved to the local memory of the execution engine, and processed by the execution engine. The results of the processing are retrieved by the CPU.

[0030] In one implementation, the inference system 130 processes requests in batches to achieve higher processor utilization on the hardware accelerators. Specifically, the inference system 130 processes multiple requests in a batch together to take advantage of the amount of parallel computing units in the execution engine. In such an implementation, the inference system 130 receives multiple requests that are each associated with an input token sequence. The inference system 130 iteratively applies the converter model to the batch of requests to generate output tokens for the requests together. In one instance, batching for the converter model is made possible by grouping requests with input token sequences of the same length together or treating requests in a batch as if they all have the same input token sequence length as the request with the shortest length at each iteration.

[0031] Batched converter model

[0032] Figures 2A-2BA method of batching using a machine-learned transducer model 200 is shown. In particular, the transducer model 200 is associated with a set of parameters determined through a training process. The transducer model 200 is coupled to receive one or more inputs (e.g., a sequence of input tokens or an output token of a previous iteration) and generate one or more outputs (e.g., an output token prediction). Specifically, in one embodiment, a request is processed by applying the transducer model to a sequence of input tokens of the request in one iteration of an “encoding phase” and applying the transducer model to a sequence of output tokens generated at a previous iteration of one or more iterations of a “decoding phase.” In the decoding phase, the output tokens of the previous iteration are set as the input for the next iteration, and the process is repeated for additional iterations of the decoding phase until an end token (e.g., a period “.” or “<end>” indicator) is generated for each request in the batch, signaling that all predictions have been generated for the respective request. In other embodiments, a termination criterion for a request can be to terminate after a maximum number of output tokens is generated, or can be other types of conditions specified, for example, by the inference system 130 or a user of the client device 110.

[0033] Figure 2A An encoding phase for the transducer model 200 is shown, in which a sequence of input tokens is processed to generate one or more output tokens. In the example shown, the inference system 130 processing requests for a chatbot receives a first request such as the question “What is your name?” a second request such as the question “What time is it?” and a third request such as the question “How do I pay?” The inference system 130 encodes each of the requests as a respective sequence of input tokens. The first request is encoded into a sequence of input tokens X1, the second request is encoded into a sequence of input tokens X2, and the third request is encoded into a sequence of input tokens X3, each of which is shown in the figure with a different fill pattern. Figure 2A In the example shown, the inference system 130 processing requests for a chatbot receives a first request such as the question “What is your name?” a second request such as the question “What time is it?” and a third request such as the question “How do I pay?” The inference system 130 encodes each of the requests as a respective sequence of input tokens. The first request is encoded into a sequence of input tokens X1, the second request is encoded into a sequence of input tokens X2, and the third request is encoded into a sequence of input tokens X3, each of which is shown in the figure with a different fill pattern. Figure 2A Each of the sequences of input tokens in the example shown is a one-dimensional sequence, in which the sequence of tokens is arranged along a single dimension (e.g., the X direction). However, as described above with respect to the example of FIG. 2, it will be appreciated that in other embodiments, the sequence of tokens can be arranged as a multi-dimensional sequence. Figure 1 As described above with respect to the example of FIG. 2, it will be appreciated that in other embodiments, the sequence of tokens can be arranged as a multi-dimensional sequence.

[0034] As described above with respect to the example of FIG. 2, it will be appreciated that in other embodiments, the sequence of tokens can be arranged as a multi-dimensional sequence. Figure 2AAs shown, since each request includes four words, each input token sequence includes four tokens, each token representing a respective word. For example, the input token sequence X1 for the first request is represented by four squares representing the words “what,” “is,” “your,” and “name.” Specifically, while each word is mapped to a single square, in practice, the inference system 130 represents the tokens for the words as embeddings that represent the word in a multi-dimensional latent space. Thus, while in Figure 2A the examples, each input token sequence is visually shown as a two-dimensional 1 x 4 tensor, in practice, each input token sequence can be represented as a three-dimensional tensor 1 x 4 x H, where H is the dimension of the embedding (e.g., the direction into or out of the page). Moreover, while for the remainder of this specification, each token (input token or output token) is mapped to a single word, this is merely an example, and it can be appreciated that in other implementations, each token can be mapped to a different unit of text, a combination of units of text, and the like. For example, in other implementations, each token can be mapped to multiple words, a paragraph, a sentence, a text unit of n-grams, or can be mapped to punctuation (e.g., “?” “!” “:”) other than text units.

[0035] In one implementation, the converter model 200 includes a set of N decoders D1, D2,..., DN. The decoders are coupled to receive a set of input representations and generate a set of output representations. For example, the first decoder D1 is coupled to receive one or more inputs of the converter model 200 as a set of input representations and generate a set of output representations. Each subsequent decoder is coupled to receive a set of output representations of a previous decoder and generate another set of output representations. For example, the second decoder D2 placed after the first decoder D1 is coupled to receive a set of output representations generated by the first decoder D1 and generate another set of output representations. This process is repeated until a set of output representations for the final decoder is generated.

[0036] The converter model 200 can also include an LM head block 270 coupled to receive a set of output representations from the final decoder DN and generate one or more output tokens as output for the current iteration. In the example shown, the LM head 270 receives a set of output representations from the final decoder DN and generates one or more output tokens for the batch of requests in the form of a tensor Figure 2A In the example shown, the LM head 270 receives a set of output representations from the final decoder DN and generates one or more output tokens for the batch of requests in the form of a tensor ​includes a first output token for the first request representing a first word prediction of "my" for the response, a second output token for the second request representing a word prediction of "the" for the response, and a third output token for the third request representing a word prediction of "you" for the response.

[0037] Figure 2B A decoding phase for the converter model 200 is shown, in which a previous output token is processed as input to generate one or more additional output tokens. Unlike the encoding phase, the converter model 200 is applied to the output token generated at the previous iteration as input to generate an additional output token. As shown, the converter model 200 receives the output token generated at the previous iteration Figure 2B as input at the next iteration and generates an additional output token for the request as output. The decoding phase is repeated until a termination criterion is satisfied for each request in the batch.

[0038] The converter model 200, and in particular each decoder Di in the set of decoders D1, D2,..., DN, includes one or more blocks, each block representing a respective operation, in particular a tensor operation. Specifically, a block in the converter model 200 can contain data for performing an operation on one or more input tensors for a request to generate one or more output tensors for the request. When the operation is executed on an execution engine, the one or more input tensors and associated data (e.g., weight tensors) for the operation can be read from the local memory of the execution engine. The operation can be executed using the one or more input tensors and associated data to generate one or more output tensors. The output tensors are provided to the CPU, another execution engine, or can be stored on the local memory of the execution engine such that it can be used for a next operation.

[0039] In one implementation, the operations of the converter model 200 are configured as batch operations in which data for a batch of requests is processed together. The batch operations are coupled to receive one or more input tensors, each of which is a concatenation of the input data for multiple requests in the batch. The batch operations generate one or more output tensors, each of which is a concatenation of the output data for multiple requests in the batch, by applying a respective operation to the one or more input tensors. For the batch operations, the input tensors can be a concatenation of the input tensors for each respective request in the batch. Thus, the input tensors for the batch operations are a single larger tensor that combines the input tensors for each request in the batch over an additional batch dimension.

[0040] Similarly, the output tensor from a batch operation can be represented as a concatenation of the output tensors of each corresponding request in the batch. Therefore, while it is also possible to generate the output tensor of a single request by applying the operation to the input tensor of the request, the output tensor generated from a batch operation using concatenated input tensors is also a single, larger tensor that combines the output tensors of each request in the batch along the batch dimension.

[0041] Furthermore, certain operations, such as tensor multiplication, involve multiplying a set of parameters (e.g., weight tensors) of the converter model 200 with the input tensors of a batch operation. In some cases, when processing requests individually, the weight tensors are read from the local memory of the hardware accelerator each time the input tensor of a request is multiplied by the weight tensor. In contrast, when performing tensor multiplication as a batch operation, this set of parameters is read once and reused for operations on the chained input tensors. This can lead to a significant improvement in throughput compared to processing requests individually. However, in Figures 2A-2B In batch processing, since the batch of input data is processed as a single tensor, each request may be required to have the same length in both the input tensor and the internal state tensor.

[0042] like Figure 2A As shown, the decoder in converter model 200 includes a first-layer normalization block 210, a QKV operation block 215, a splitting block 220, a self-attention block 225, an attention linear block 230, a first addition block 235, a second-layer normalization block 240, a first MLP block 245, a GeLU block 250, a second MLP block 255, and a second addition block 260. In one embodiment, each of the blocks in the decoder is executed on the execution engine as a batch operation, wherein the input tensor of the batch operation is a concatenation of a batch of requested input data, and the output tensor is a concatenation of the batch of requested output data. Although the operations in the first decoder D1 are described as examples, it is understood that the remaining decoders in this group may also include operations similar to those of the first decoder D1.

[0043] Specifically, the inference system 130 concatenates the input token sequences X1, X2, and X3 into a concatenated input tensor. A layer normalization block 210 is coupled to receive the concatenated input tensor and normalize the elements of each request to generate a normalized tensor as the output tensor. A QKV operation block 215 is coupled to receive the normalized tensor as the input tensor and generate an output tensor including the query, keyword, and value for each request. Specifically, the QKV operation block 215 generates the query, keyword, and value by applying the QKV weight tensor, which is the trained parameter set of the converter model 200, to the normalized tensor including the normalized data for each request, to generate an output tensor including the query, keyword, and value for each request in the batch. For example, as... Figure 2AAs shown, the first row of the output tensor of the QKV operation 215 includes the queries, keys, values (in concatenated form) for the first request, the second row includes the queries, keys, values for the second request, and so on. In one implementation, instead of applying a single QKV weight tensor, the QKV operation block 215 can be associated with separate weight tensors that are each applied to the input tensor to respectively generate the queries, keys, and values. For example, the QKV operation block 215 can apply a query weight tensor to the input tensor to generate the queries for the requests, apply a separate key weight tensor to the input tensor to generate the keys for the requests, and apply a separate value weight tensor to the input tensor to generate the values for the requests.

[0044] The split block 220 is coupled to receive the output tensor from the QKV operation block 215 and, in cases where the query tensor, key tensor, value tensor are not separately generated, split the output tensor into the query tensor, key tensor, value tensor for the current iteration. For the current iteration, the query tensor includes the queries for the batch of requests, the key tensor includes the keys for the batch of requests, and the value tensor includes the values for the batch of requests. In one instance, the execution engine maintains an internal state of the batch of requests as a key cache tensor to cache the keys generated at the previous iteration and the current iteration, and a value cache tensor to cache the values generated at the previous iteration and the current iteration. The inference system 130 adds the key tensor for the current iteration to the key cache tensor and adds the value tensor for the current iteration to the value cache tensor.

[0045] The self-attention block 225 is coupled to receive the query tensor, the key cache tensor, and the value cache tensor as input tensors and generate an output tensor that includes attention outputs for the requests in the batch. In one instance, the attention output tensor is generated by multiplying the query tensor with the key cache tensor to generate a multiplied tensor and multiplying the multiplied tensor with the value cache tensor. In cases where the self-attention block 225 is executed as a batch operation, the query tensor can be multiplied with the key cache tensor to generate the multiplied tensor on the execution engine. The value cache tensor can be multiplied with the multiplied tensor to generate the attention output tensor. The attention output tensor includes the attention outputs for the requests in the batch. For example, in Figure 2A In the example shown, the first row of the attention output tensor Z’ is the attention output for the first request, the second row of the attention output tensor Z’ is the attention output for the second request, and so on.

[0046] The attention linear block 230 is coupled to receive the attention output tensor as an input tensor and generate an output tensor by applying an attention weight tensor that is a trained set of parameters of the transformer model 200 to the attention output tensor. The attention linear block 230 is configured as a batch operation. For example, inFigure 2A In this process, the first row of the output tensor Z can be the output data for the first request, and the second row of the output tensor Z can be the output data for the second request, and so on. The first summing block 235 combines the concatenated input tensor with the input token sequence and the output tensor from the attention linear block 230 to generate a combined tensor. The second-level normalization block 240 is coupled to receive the combined tensor from the summing block 235 and normalizes the elements of each request to generate a normalized tensor as the output tensor.

[0047] The decoder also includes a multilayer perceptron (MLP) block 242, which comprises one or more blocks including additional neural network layers. In one embodiment, the MLP block 242 includes a first MLP block 245, which is coupled to receive a normalized tensor from a second normalization block 240 as an input tensor and to generate an output tensor by applying an MLP weight tensor, which is a trained parameter set of the converter model 200, to the normalized tensor. The MLP block 242 includes a GeLU block 250, which is coupled to receive the output tensor from the first MLP block 245 as an input tensor and to generate an output tensor by applying a GeLU function to the input tensor. The MLP block 242 includes a second MLP block 255, which is coupled to receive the output tensor from the GeLU block 250 as an input tensor and to generate an output tensor by applying a second MLP weight tensor, which is a trained parameter set of the converter model 200.

[0048] The combined tensor from the second summing block 260 can be obtained as a set of output representations generated for the first decoder D1. Subsequently, the second decoder D2, placed after the first decoder D1, is coupled to receive a set of output representations from the first decoder D1, and a similar operation is repeated to generate another set of output representations. This process is repeated until a set of output representations from the final decoder DN is generated. Specifically, while each decoder may involve operations similar to those of the first decoder D1, the trained parameter set associated with these operations may differ between decoders. The LM head block 270 is coupled to receive a set of output representations from the final decoder DN as an input tensor and generates an output tensor including one or more output tokens.

[0049] Such as combination Figure 2B As described, the next iteration is the decoding phase, in which the output tokens generated in the previous iteration are input to converter model 200. Converter model 200 generates another set of output tokens. Layer normalization block 210 is applied to the output tensor. The QKV operation block 215 is applied to the normalized tensor to generate an output tensor comprising queries, keys, values for the current iteration. The split block 220 is applied to split the output tensor into a query tensor, a key tensor, a value tensor. The inference system 130 adds the key tensor for the current iteration to the key cache tensor and adds the value tensor for the current iteration to the value cache tensor. Thus, at the end of the iteration Figure 2B The key cache tensor Kca e includes the keys for the previous iterations and the current key tensor K, and the value cache tensor Vca e includes the values for the previous iterations and the current value tensor V, during the iteration shown.

[0050] The self-attention block 225 is applied to the query tensor for the current iteration, the key cache tensor, and the value cache tensor to generate an attention output tensor. The attention linear block 230 is applied to the attention output tensor to generate an output tensor. The add block 235 combines the input tensor and the output tensor from the attention linear block 230 to generate a combined tensor. The MLP block 242 is applied to the combined tensor to generate a set of output representations for the decoder D1. The process is repeated for the remaining decoders and the LM head 270 to generate an output tensor comprising another set of output tokens and the transformer model 200 is iteratively applied until termination criteria are met for each request in the batch.

[0051] Selective batched converter model

[0052] In many cases, the input of a request or the internal state of a request is variable in length and can be difficult to batch using, for example, the batching method in Figures 2A-2B For example, the number of input tokens between different NLP requests is likely to vary in length depending on the input text and the type of request. For example, the length of a first query request can be different from the length of a second query request due to different users synthesizing language differently. However, methods of batching a transformer model, such as the method described in conjunction with Figures 2A-2B require the length of the input tensors for the batch to be the same, as the input tensors must be merged into a single concatenated tensor for operation. Thus, if the length of the input tensors for multiple requests is different, it can not be feasible to process the requests as a batch using the batching method of Figures 2A-2B

[0053] ​Another workaround is to treat the input token sequence during the encoding phase as if the request had the same length as the input token sequence with the shortest length (i.e., the fewest number of tokens). However, this requires the inference system 130 to discard the output tokens generated for requests with longer input lengths in subsequent iterations, since the original input tokens received for that request must be replaced in the corresponding elements. This leads to inefficiency and unnecessary resource waste, as the inference system processes the input token sequences for longer requests individually over multiple iterations rather than in a single iteration. Another workaround could be to treat the input token sequence as if the request had the same length as the input token sequence with the longest length (i.e., the maximum number of tokens). However, this requires the inference system to increase the size of the input tensor to match (and fill in) the remaining elements for requests with the longest token sequence. This leads to redundant computation, as the inference system processes large input tensors throughout the execution of the converter model. For example, in some cases, using Figures 2A-2B The number of iterations required to process a batch of requests can be significantly greater than the number of iterations required to process each request individually.

[0054] Therefore, in one implementation, the inference system 130 uses a selective batching approach to train a machine learning converter model and applies it to a batch of requests with variable input lengths, target lengths, or internal state lengths. In the selective batching approach, the inference system 130 selectively batches a subset of operations in the converter model, but processes requests separately for a subset of operations in the converter model that are incompatible with batching. In one implementation, the operations to be processed separately are the attention operations of the encoder or decoder of the converter model. Through selective batching, the inference system 130 enables batch operations to be performed on a batch of requests with variable input lengths, target lengths, or internal state lengths, leveraging the parallel computing capabilities of hardware accelerators while preventing problems that may arise in workarounds.

[0055] Figures 3A-3B A method for selective batching using a machine learning converter model 300, according to an embodiment, is illustrated. Figure 3A In the example shown, with Figure 2A Unlike other systems, inference system 130 receives a batch of requests consisting of input token sequences of varying lengths. Figure 3A In the process, the inference system 130 receives a first request such as the question "What's wrong?", a second request such as the question "Hello?", and a third request such as the question "How are you?", each of which has a different word length. The inference system 130 encodes each of the requests into a corresponding set of input token sequences. The first request is encoded into input token sequence X1, the second request into input token sequence X2, and the third request into input token sequence X3.

[0056] Figures 3A-3B The illustrated transformer model 300 also includes a set of N decoders D1, D2,..., DN and an LM head block 370. However, unlike the transformer model 200, Figures 2A-2B a subset of the operations in the transformer model 300, specifically the operations in the self-attention block 325, are performed individually rather than as a batch operation. As Figure 3A illustrated, the transformer model 300 includes a first layer normalization block 310, a QKV operation block 315, a split block 320, a self-attention block 325, an attention linear block 330, a first addition block 335, a second layer normalization block 340, a first MLP block 345, a GeLU block 350, a second MLP block 355, and a second addition block 360.

[0057] In particular, in the selective batching approach, the inference system 130 can concatenate the input token sequences X1, X2, X3 into a concatenated input tensor. Unlike the batching approach in Figures 2A-2B the inference system 130 concatenates the input token sequences such that the input tokens for each request in the batch are concatenated in one dimension, e.g., the vertical dimension in Figures 3A-3B When input token sequences of different lengths are input to the transformer model 300, one way to concatenate these sequences into a concatenated tensor of the same dimensionality for the requests in the batch is to concatenate the input tokens of the requests in a single dimension. Thus, Figure 3A the concatenated input tensor in Figure 3A is illustrated as b x1 tensor, where b denotes the sum of the token lengths for each request b in the batch, where the length of the input tokens for request b is given by L b In practice, the concatenated input tensor can be a b x1 x H tensor, where H is the dimensionality of the embeddings.

[0058] The first layer normalization block 310 is applied to the concatenated input tensor to generate a normalized tensor that normalizes the elements for each request. In one instance, the operation of the first layer normalization block 310 is given by:

[0059]

[0060] where denotes the concatenated input tensor, and denotes the normalized tensor, e is a constant, and

[0061] The QKV operation block 315 is applied to the normalized tensor as an input tensor to generate an output tensor comprising queries, keys, values for the requested queries. The QKV operation block can be configured as one or more neural network layers. Specifically, the QKV operation block 315 generates queries, keys, and values by applying a QKV weight tensor as a trained parameter set of the transformer model 300 to the normalized tensor. In one instance, the operation of the QKV operation block 315 is given by:

[0062]

[0063] wherein, represents the normalized tensor from the first layer normalization block 310, represents the output tensor containing queries, keys, and values, is the QKV weight tensor, and (bias).

[0064] In particular, in the selective batching approach, the QKV operation block 315 can be configured as a batch operation, and the QKV weight tensor is multiplied with the normalized tensor from the layer normalization block 310 to generate an output tensor comprising queries, keys, and values for a batch of requests. Due to the shape of the concatenated input tensor, the output tensor of the QKV operation 315 can also be arranged such that each query element of the requests is concatenated along one dimension (e.g., a vertical dimension), and each key element of the requests is concatenated along the one dimension, and each value element of the requests is concatenated along the one dimension. For example, as shown in Figure 3A the first and second rows of the output tensor of the QKV operation 315 correspond to queries, keys, and values of a first request, the third row corresponds to queries, keys, and values of a second request, and the fourth through sixth rows correspond to queries, keys, and values of a third request. Similarly, the QKV operation block 315 can separately generate queries, keys, and values by applying a query tensor, a separate key tensor, and a separate value tensor to the input tensor. In one implementation, the QKV operation block 315 includes multiple attention heads, and queries, keys, and values are generated for each attention head. In such an instance, when the QKV operation block 315 includes n attention heads, H can equal n x h, where h is the dimensionality of the latent space of each attention head.

[0065] The split block 320 is coupled to receive the output tensor from the QKV operation block 315 and split the output tensor into a query tensor, a key tensor, a value tensor for the current iteration. In one instance, the operation of the split block 320 is given by:

[0066] Q i，j = X i，j , Ki，j = X i，H+j , V i，j = X i，2H+j

[0067] wherein, represents an output tensor from the QKV operation block 315, represents a query tensor, represents a key tensor, represents a value tensor. When the QKV operation block 315 includes multiple attention heads, the query tensor can include n queries for each request, the key tensor can include n keys for each request, and the value tensor can include n values for each request.

[0068] Unlike the batching method of Figure 2A , the split block 320 also splits the query tensor Q into queries for each request, splits the key tensor K into keys for each request, and splits the value tensor V into values for each request. In one instance, each request split operation of the split block 320 is given by:

[0069]

[0070] wherein, Q1, Q2, …, Q B is a query for a single request, and

[0071]

[0072] wherein, K1, K2, …, K B is a key for a single request, and

[0073]

[0074] wherein, V1, V2, …, V B is a value for a single request, and

[0075] Further, for each request b = 1, 2, …, B, the inference system 130 maintains the internal state of the request in the form of a key cache tensor Kcache b and a value cache tensor Vcache b wherein, and and L b,curris the current length of the token sequence (including both input tokens and output tokens) of request b. In other implementations, the internal state of a request can contain different types of information than the keys and values of the request, and implementations are not limited thereto. Thus, for the encoding phase, L b,curr is equal to L b . The inference system 130 adds the key tensor K b to the respective key cache tensor for the request, and adds the value tensor V b to the respective value cache tensor for the request. Because the set of inputs for a batch of requests (i.e., the input token sequence for X1, X2, X3) have different lengths, the length of the internal state of the requests in the batch is different in Figure 3A . In the batched processing of Figures 2A-2B , because the self-attention operation 225 involves multiplying the query tensor, the key cache tensor, and the value cache tensor, it is also required that the key cache tensor and the value cache tensor have the same length across requests in the batch. However, because selective batching is performed in Figure 3A , requests with different lengths of their internal state can also be efficiently batched for the selection operation while being processed separately for the self-attention operation 325.

[0076] The self-attention block 325 is coupled to receive, as one or more input tensors, the query tensor for each request, the key cache tensor for each request, and the value cache tensor for each request, and generate, as one or more output tensors, the attention output for each request in the batch. In one implementation, the self-attention block 325 is not batched, but at least a subset of the requests in the batch are processed separately from each other. Specifically, each request in the batch can be processed individually for the self-attention block 325.

[0077] For a request in the batch, the self-attention block 325 is coupled to receive the query for the request and the key cache tensor for the request to generate a multiplication tensor for the request. In one instance, the operation of the self-attention block 325 for a request is given by:

[0078]

[0079] wherein, denotes the multiplication tensor for request b, is a reshaped query tensor Q b for request b that concatenates the query for each attention head together for request b, and is a reshaped key cache tensor Kcache b for request b that concatenates the key cache tensor for each attention head together for request b.

[0080] The self-attention block 325 is then coupled to receive the multiply tensor for the request and the value cache tensor and generate an attention output tensor for the request. In one instance, the operation of the self-attention block 325 for the request is given by:

[0081] if k < j

[0082] P bi，j，k = 0 if k > j,

[0083] where, represents the received multiply tensor for the request b, and

[0084]

[0085] where, represents the multiply tensor for the request b, is the reshaped value tensor Vcache for the request b b that concatenates the value cache tensors for each attention head together for the request b. Subsequently, the output Y b is reshaped by: b

[0086] Z′ bj，hi+k = Y bi，j，k

[0087] where, and

[0088] ​In particular, the operation of the self-attention block 325 can not be compatible with batch operations for requests having different lengths because the operation of the self-attention block 325 is a multiplication between the query, key cache, and value cache for a request, rather than a tensor multiplication between a fixed weight tensor and one or more input tensors. Thus, the attention outputs for the requests in the batch are generated separately. For example, the execution engine can generate a first attention output for a first request using the input tensors for the first request (e.g., the query tensor Q1, the key cache tensor Kcache1, the value cache tensor Vcache1), and separately, the execution engine can generate a second attention output for a second request using the input tensors for the second request at a separate step. In particular, the second attention output can be generated at a different execution engine than the execution engine that processed the first request, at a different hardware accelerator than the hardware accelerator that processed the first request in the same execution engine, at a different GPU core than the GPU core that processed the first request in the same hardware accelerator, or at the same GPU core that processed the first request in the same hardware accelerator, although implementations are not limited thereto.

[0089] Because the subsequent operations in the attention linear block 330 are configured as batch operations, the inference system 130 concatenates the attention outputs for the requests in the batch into a single attention output tensor. In one instance, the concatenation operation is given by:

[0090]

[0091] wherein, denotes the single attention output tensor. As shown in Figure 3A the first and second rows of the concatenated attention output tensor Z’ correspond to the attention output for the first request, the third row corresponds to the attention output for the second request, and the fourth through sixth rows correspond to the attention output for the third request.

[0092] The attention linear block 330 is coupled to receive the attention output tensor as an input tensor and generate an output tensor by applying an attention weight tensor as the trained set of parameters of the transformer model 300 to the attention output tensor. The attention linear block 330 can be configured as one or more neural network layers. The attention linear block 330 can be executed as a batch operation. In one instance, the operation of the attention linear block 330 is given by:

[0093]

[0094] wherein, denotes the attention output tensor from the self-attention block 325, denotes the output tensor, is an attention weight tensor, and As an example, in Figure 3A the first and second rows of the output tensor Z include output data for the first request, the third row includes output data for the second request, and the fourth through sixth rows include output data for the third request.

[0095] The first addition block 335 combines the concatenated input tensor including the input token sequence and the output tensor from the attention linear block 330 to generate a combined tensor. The second layer normalization block 340 is coupled to receive the combined tensor from the addition block 335 and normalizes each requested element to generate a normalized tensor as an output tensor.

[0096] Similar to the converter model 200, Figures 2A-2B The decoder can also include an MLP block 342 including one or more blocks including additional neural network layers. The MLP block 342 is executed as one or more batch operations. The MLP block 342 also includes a first MLP block 345 configured as one or more neural network layers. The first MLP block 345 is coupled to receive the normalized tensor from the second layer normalization block 340 and generate an output tensor by applying an MLP weight tensor as a trained parameter set of the converter model 200 to the normalized tensor.

[0097] The MLP block 342 also includes a GeLU block 350 coupled to receive an output tensor from the first MLP block 345 as an input tensor and generate an output tensor by applying a GeLU function to the input tensor. The MLP block 342 also includes a second MLP block 355 configured as one or more neural network layers. The second MLP block 355 is coupled to receive an output tensor from the GeLU block 350 as an input tensor and generate an output tensor by applying a second MLP weight tensor as a trained parameter set of the converter model 300. A second addition block 360 combines the output tensor from the second MLP block 355 and the output tensor from the first addition block 335 to form a combined tensor.

[0098] The combined tensor includes a set of output representations for the first decoder D1. The set of output representations is propagated to subsequent decoders and the similar process is repeated to generate a set of output representations for the final decoder DN. Specifically, the inference system 130 can also deploy the selective batching approach for the remaining decoders, where the self-attention operation is performed without batching, but the remaining operations are batched. The LM head block 370 is coupled to receive the set of output representations from the final decoder DN and generate an output tensor including a set of output tokens

[0099] As Figure 3B shown, the next iteration is a decoding phase in which the previous output tokens generated at the previous iteration are input to the transformer model 300 to generate another set of output tokens. Specifically, the layer normalization block 310 is applied to the output tensor to generate a normalized tensor. The QKV operation block 315 is applied to the normalized tensor to generate an output tensor comprising queries, keys, and values for the current iteration. The split block 320 is applied to split the output tensor into one or more query tensors for each request in the batch, one or more key tensors for each request in the batch, and one or more value tensors for each request in the batch. The inference system 130 adds each key tensor to the key cache tensor for the corresponding request and adds each value tensor to the value cache for the corresponding request. Since Figure 3B the iterations of the decoding phase are depicted for all requests in the batch, these requests have the same length for the set of inputs fed to the transformer model 300 (i.e., for the single output token generated from the previous iteration for each request). However, the internal state comprising the key cache tensors and the value caches for the requests in the batch can still have different lengths because the input token lengths for the requests are different in the encoding phase shown. Figure 3A However, similar to Figure 3A the inference system 130 can still process a batch of requests with different internal state lengths by selectively batching operations in the transformer model 300.

[0100] The self-attention block 325 is applied separately to the query tensor, the key cache tensor, and the value cache tensor for each corresponding request to generate an attention output tensor for that request. The attention outputs for these requests are concatenated into an attention output tensor. The attention linear block 330 is applied to the attention output tensor to generate an output tensor. The addition block 335 combines the concatenated input tensor and the output tensor from the attention linear block 330 to generate a combined tensor. The MLP block 342 is applied to this combined tensor to generate a set of output representations for the decoder D1. This process is repeated for the remaining decoders and the LM head block 370 to generate an output tensor comprising another set of output tokens and iteratively applying the transformer model 300 until the termination criteria are met for each request in the batch.

[0101] By selectively batching the select operations of the transformer model 300, the inference system 130 can efficiently process a batch of requests with variable input lengths. The inference system 130 performs the requests by individually processing operations such as self-attention operations on a per-request basis while batching the remaining operations of the transformer model 300. Specifically, operations involving tensor multiplication between one or more input tensors and a trained weight tensor, such as the QKV linear operation, the attention linear operation, or the MLP operation, can require high computational power during the inference process due to the typically large dimensionality of the weight tensor. Moreover, when processing requests individually for these operations, the weight tensor can be read from the local memory of the execution engine multiple times. However, by configuring these subsets of operations as batch operations but individually processing operations that do not involve manipulation of the weight tensor, such as the self-attention operation, the inference system 130 can leverage the parallel computational power of the hardware accelerator while allowing for high flexibility in processing variable-length requests.

[0102] Additionally, Figure 3A iterations of the encoding stage for a set of inputs that are input token sequences are shown, and Figure 3B iterations of the decoding stage for a set of inputs that are output tokens generated for a batch of requests in a previous iteration (i.e., the encoding stage) are shown. However, unlike other methods of batching the transformer model, the selective batching method described herein flexibly allows the transformer model 300 to process a set of inputs for a batch of requests with variable input lengths, target lengths, or internal state lengths, regardless of whether the requests in the batch are being processed for the encoding stage or the decoding stage or how many iterations have already been processed for the request.

[0103] Specifically, the method of Figures 2A-2B may not be used to batch a batch of requests in the following cases: (i) when the requests are in the encoding stage and have input token sequences of different lengths; (ii) when the requests are in the decoding stage and each request processes the tokens at different indices (i.e., the input token length plus the index of the decoding iteration) from each other resulting in internal states of different lengths; and (iii) when each request in the batch is in a different stage (encoding or decoding) resulting in inputs of different lengths (i.e., input token sequences for the encoding stage and a single output token for the decoding stage). While Figures 2A-2B the batching method in limits the requests in the batch to the same stage, where the number of input tokens for the encoding stage is the same and the token indices for the decoding stage are the same, this limitation greatly reduces the possibility of batching in real-world workloads.

[0104] In contrast, the selective batching approach described herein enables the transducer model 300 to process requests as a batch even if the requests are at different stages or different processing indices. Thus, for example, instead of receiving a set of input token sequences X1, X2, X3in Figure 3A the transducer model 300 can perform an iteration using a batch of the input token sequence for the first request (encoding stage), the first output token for the second request (first iteration of the decoding stage), and the third output token for the third request (third iteration of the decoding stage). As another example, the transducer model 300 can perform an iteration using a batch of the second output token for the first request and the fourth output token for the second request, each having a different length for the key cache tensor and the value cache tensor. As described in connection with Figures 5A-5B this enables the inference system 130 to flexibly modify and update the batch in each iteration so that the computing power of the one or more execution engines is fully utilized.

[0105] Furthermore, while Figures 2A-2B and Figures 3A-3B a GPT (generative pre-training) type of transducer model including a set of decoders is shown, the inference system 130 can perform selective batching on transducer models having other types of architectures as long as the transducer model includes an attention mechanism (e.g., self-attention or encoder-decoder attention) that generates attention outputs using a query, a key, and a value generated for a request, and generates output tokens in a self-recursive manner by using output tokens generated at a previous iteration as inputs for a next iteration.

[0106] In another implementation, the transducer model can have an encoder-decoder architecture and include a set of encoders coupled to a set of decoders. In such an architecture, each encoder includes at least a self-attention block coupled to an MLP block. The self-attention block of the encoder is coupled to receive a query tensor, a key tensor, and a value tensor obtained from processing a set of input tokens and generate an attention output. Each decoder includes at least a self-attention block coupled to an encoder-decoder attention block coupled to an MLP block. The encoder-decoder attention block of the decoder is coupled to receive a query tensor obtained from processing an output of the self-attention block and a key tensor and a value tensor obtained from processing an output of a final encoder to generate another attention output.

[0107] The inference system 130 can apply the encoder-decoder converter model to a batch of requests. Specifically, for the encoder, requests can be processed individually for self-attention blocks, while the remaining operations are batched for these requests. For the decoder, requests can be processed individually for both self-attention blocks and encoder-decoder attention blocks, while the remaining operations are batched for these requests.

[0108] Return to Figure 1 Client devices 110A and 110B are computing devices, for example, devices with operating systems such as... or Smartphones, tablets, laptops, desktop computers, or any other type of network-enabled device. Figure 1 In the system environment 100 shown, client device 110 enables a user to submit requests to inference system 130 to perform tasks that can be processed by a machine learning converter model. For example, a user of client device 100A may belong to an entity that deploys one or more applications that require NLP or other types of tasks that can be processed by a converter architecture. Requests submitted by client device 110 may be, for example, requests to generate predictions for a chatbot application, requests to translate a sequence of words from one language to another, requests to synthesize a story or narrative given a starting sequence of words, etc.

[0109] A typical client device 110 includes the hardware and software required to connect to network 122 (e.g., via WiFi and / or 4G, 5G, or other wireless telecommunications standards). Specifically, client device 110 may include an operating system and various applications running on the operating system that enable users to submit requests. For example, client device 110 may include a browser application or a standalone application deployed by inference system 130 that enables users of the organization to interact with inference system 130 to submit requests.

[0110] Network 122 provides the communication infrastructure between client device 110 and online system 130. Network 122 is typically the Internet, but can be any network, including but not limited to a local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile wired or wireless network, private network or virtual private network.

[0111] Block diagram of the reasoning system

[0112] Figure 4 This is a block diagram of the architecture of the inference system 130 according to the implementation method. Figure 4The illustrated inference system 130 includes a data management module 420, a training module 430, and a serving system 435. The inference system 130 also includes a training corpus 460. In alternative configurations, different and / or additional components can be included in the inference system 130.

[0113] The data management module 420 manages the training corpus 460 of training data used to train the parameters of the transducer model. In one implementation, the training corpus 460 includes a plurality of data instances, each instance including a set of training input data and a set of training output data, the set of training output data corresponding to a known prediction of the set of training input data, the known prediction addressing the task for which the transducer model is to be trained. In one implementation, the training input data and the training output data are textual data, although implementations are not so limited, and the training data can include any type of data for which the transducer model is trained to process in an autoregressive manner, such as audio data or image data. The training module 430 encodes the set of training input data into a set of training input tokens and encodes the set of training output data into a set of training output tokens. Each token can represent a respective word in a latent space.

[0114] For example, in the case where the transducer model is used for automatic translation from English to French, the set of training input tokens can correspond to a first sentence of English words, and the set of training output tokens can correspond to a second sentence of French words, the second sentence being a translation of the first sentence. As yet another example, in the case where the transducer model is used to convert images to text, the set of training input tokens can correspond to a plurality of pieces of image data of a training image, and the set of training output tokens can correspond to a converted sentence or paragraph describing the image.

[0115] The execution engine module 425 includes one or more execution engines built on a specialized hardware accelerator, such as a graphics processing unit (GPU) or a tensor processing unit (TPU). The execution engine receives requests to execute one or more requests on the specialized hardware. In one instance, the execution engine receives a batch of requests and uses the input for each request to execute one or more iterations of the transducer model via selective batch execution. As described above, the execution engine can include a set of cores (e.g., GPU cores) coupled to local memory (e.g., GPU memory). Specifically, for each operation of an iteration, the execution engine is configured to read the data required for the operation from the local memory of the execution engine.

[0116] The execution engine of the execution engine module 425 can be configured as a GPU or a TPU or any other specialized hardware capable of processing neural network operations, including tensor multiplication and floating point operations, in an efficient manner. For example, the execution engine is not limited to Google TPU, Graphcore IPU, Cerebras WSE, Nvidia GPU, intel Nervana, Qualcomm Cloud AI, Samsung Exynos, AMD Radeon, Xilinx AI accelerator card, IBM TrueNorth, AWS Trainium, etc. As mentioned above, the execution engine can be composed of one or more hardware accelerators. In some instances, the execution engine can process workloads that require the processing power of more than one hardware accelerator. Thus, if needed, the execution engine is capable of distributing workloads across multiple hardware accelerators.

[0117] In one embodiment, the execution engine is further configured to manage one or more caches required for performing one or more iterations of the converter model on the local memory. For example, when performing the converter model 300, the execution engine maintains a key cache tensor and a value cache tensor for the request until the request has completed. Thus, in response to receiving one or more new requests for which the request for the encoding stage has not yet been processed, the execution engine allocates memory to each request for use in maintaining a key cache tensor and a value cache tensor for the request. For each iteration of the decoding stage, the execution engine can add the key tensor and the value tensor for the iteration to the respective key cache and value cache allocated for the request and retrieve the key cache and the value cache for the request to perform the operations. In response to completing one or more requests, the execution engine can provide the output for the request to the appropriate module of the inference system 130 and release the allocated cache memory for the completed request so that the released memory can be used for other requests. Figures 3A-3B

[0118] The training module 430 trains the parameters of the converter model by performing a training process. First, the training module 430 can generate an appropriate architecture for the converter model that can address the task requests received by the client device 110. For example, the training module 430 can train the converter model 300 shown in FIG. 3. The training module 430 can determine the number of decoders in the set of decoders to adjust the depth of the converter model and determine, for example, how many layers are included in each neural network layer of the converter model. In the case where the converter model to be trained is an encoder-decoder architecture, the training module 430 can determine the number of encoders as well as the number of decoders. Figures 3A-3B

[0119] ​​The training module 430 can also initialize one or more weight tensors associated with the operation of the transformer model. The elements of the weight tensors correspond to the parameters of the transformer model that will be learned during the training process using the training data of the training corpus 460. The weight tensors can be initialized with the dimensionality of the input tensors with which the weight tensors are configured to operate. For example, to train the transformer model 300 of Figures 3A-3B QKV attn

[0120] During the training process, the training module 430 obtains a set of training data and trains the parameters of the machine learning transformer model by iteratively proceeding through a forward pass step and a backward propagation step. During the forward pass step, the training module 430 generates one or more estimated output tokens by applying the estimated parameters of the transformer model to a set of training input tokens of the set of training data to generate one or more estimated output tokens. The training module 430 determines a loss function that indicates the difference between the one or more estimated output tokens and the set of training output tokens. During the backward propagation step, the training module 430 updates the parameters (i.e., the weight tensors) of the transformer model to reduce the loss function. This process is iteratively repeated for the next set of training data until convergence is reached for the parameters of the transformer model or a desired level of model quality is reached.

[0121] In one implementation, the training module 430 arranges the training data in batches and executes one or more iterations of the training process on one or more execution engines. For example, each batch of training data can include a batch of input token sequences that are each from a different training source. For example, a first training input token sequence can be obtained from a first text source and a second training input token sequence can be obtained from a second text source. By executing the training process with batching, the training module 430 can utilize the parallel processing capabilities of the execution engines when training a transformer model that requires a high level of computation.

[0122] The service system 435 receives requests from the client devices 110 to perform one or more tasks that can be processed using the trained transformer model. As discussed with respect to Figures 3A-3B ​​​As described, when the task is an NLP application, the request can include a sequence of words (e.g., an English sentence), and the service system 435 can parse and map the sequence of words into a sequence of input tokens that can be provided to the transformer model. The service system 435 processes the request to generate one or more output tokens using the transformer model. The service system 435 can convert the output tokens to output data, e.g., a sequence of words (e.g., a sentence translated into French), and return the output data as a response to the client device.

[0123] In one implementation, the service system 435 receives multiple requests from the client device 110 over time and forms multiple batches of requests. The service system 435 provides these batches to one or more execution engines. In response to execution of the execution engines, the service system 435 receives one or more output tokens for the requests and provides these outputs as responses to the requests to the client device 110. In one implementation, the length of the internal state of the requests in the batches formed by the service system 435 or the length of the input sequence of the requests in the batches can differ from one another because the execution engines are able to process the requests using a selective batching approach. In one implementation, the service system 435 waits until the requests are completed (i.e., the termination criteria are met) and provides the outputs as responses to the requests to the client device 110. In another implementation, the service system 435 provides the outputs to the client device 110 even if the requests are not completed. For example, the service system 435 can provide a number (e.g., 5, 10, 15) of output tokens each time the number of output tokens is generated for a request. As another example, for one or more time intervals (e.g., 100 ms, 200 ms), the service system 435 can provide the output tokens generated after each time interval to the client device 110. This enables the user of the client device 110 to receive intermittent outputs even if the requests are not completed.

[0124] In one implementation, the service system 435 performs iterative level dynamic batching for the transformer model, which enables the service system 435 to dynamically modify a batch of requests that is being executed on an execution engine on a per-iteration basis. Specifically, for certain batching approaches, it can be difficult to modify a batch of requests once processing has started on an execution engine because such approaches require that the length of the input to the transformer model or the length of the internal state between the requests of the batch be the same. That is, unless a newly incoming request has the same length of input and internal state as the batch of requests that is being executed, it is difficult for the inference system to modify the batch to, e.g., add a new request to the batch.

[0125] By performing selective batching, the service system 435 can monitor and modify the batches being processed on the execution engines at each iteration. In particular, at one or more iterations, the service system 435 can modify the batches by adding newly incoming requests to the batches being executed on the execution engines or removing completed requests from the batches. This is because selective batching enables processing of requests with variable lengths without needing to limit the input or internal state of the converter model to the same length. This enables the service system 435 to provide responses to completed requests to the client devices 110 of the requests earlier and enables adding new requests to a batch being processed by the execution engine for the batch even if the length of the new request is different from the length of the existing requests for the next iteration if the execution engine is not fully utilized.

[0126] Dynamic batching for converter model

[0127] Figures 5A-5D A method for dynamic batching of requests for processing using a machine learning converter model is shown in accordance with an embodiment. In one embodiment, the service system 435 includes a request processor 580 and a scheduler 585, each coupled to one or more execution engines. The request processor 580 receives requests and forwards the requests to the scheduler 585. The request processor 580 maintains a completion queue for storing outputs of completed requests. In particular, the request processor 580 receives outputs of completed requests from the execution engines and stores the outputs in the completion queue so that they can be provided to the client devices 110 of the requests. The scheduler 585 receives the forwarded requests from the request processor 580 and maintains an incoming request queue for storing new requests to be processed. The scheduler 585 forms batches of requests and schedules the batches for execution on the execution engines. In one embodiment, the scheduler 585 is configured to monitor which batches of requests are assigned to each execution engine, how many iterations of the converter model have been performed for each request, and whether the request has been completed. The scheduler 585 is also configured to monitor the available cache memory in each execution engine.

[0128] In particular, Figure 5A The request processor 580 and the scheduler 585 coupled to the execution engines 590A and 590B are shown. In this example, the execution engine 590A is scheduled to execute a single request Rl, and the execution engine 590B is scheduled to execute a batch of requests R3, R4, R5. In particular, for the execution engine 590A, the request processor 580 receives the request Rl and forwards the request to the scheduler 585. The scheduler 585 adds the request Rl to the incoming request queue and schedules the request Rl for execution on the execution engine 590A. The scheduler 585 also monitors the execution engine 590A to determine when the request Rl has been completed. When the request Rl has been completed, the scheduler 585 removes the request Rl from the incoming request queue and adds the output of the request Rl to the completion queue. The request processor 580 receives the output of the request Rl from the completion queue and provides the output to the client device 110 of the request Rl. Figure 5A Figure 5A ​The request R1 in the execution engine 590A includes a single input token for the first iteration. On the other hand, the request R3 in the execution engine 590B includes a sequence of two input tokens, the request R4 includes a sequence of three input tokens, and the request R5 includes a sequence of two input tokens.

[0129] The request processor 580 receives the new request R2 and forwards the request R2 to the scheduler 585. The scheduler 585 stores the request R2 in the incoming request queue. Iterations of the transducer model are performed on the execution engines 590A, 590B. For example, the iterations can be an encoding phase for certain requests in a batch that is currently being executed. In particular, a key buffer tensor and a value buffer tensor are allocated as internal state buffers for each request, and a portion of the buffers for each request is used to store the keys and values after the encoding phase. In one implementation, the allocation of the internal state buffers for a request is based on a maximum output token length that the transducer model is configured to generate. For example, the transducer model can be configured to generate a maximum number, 1024, of output tokens, and the execution engine can allocate an internal state buffer configured to store up to 1024 elements for each key buffer tensor and value buffer tensor. In another implementation, where a user of a request specifies a maximum length of output tokens that can be returned as a response, the execution engine can allocate an internal state buffer configured to store up to the maximum length specified in the request for each tensor. In yet another implementation, the execution engine can determine an initial number of input tokens for a request, and the execution engine can initially allocate an internal state buffer that is large enough to store that number of input tokens plus some additional amount of memory. If the initially allocated memory is insufficient, the execution engine can incrementally allocate more memory to the internal state buffers as the transducer model is iteratively applied to the request. The scheduler 585 monitors the buffer memory for the execution engines 590A and 590B. In response to determining that the execution engine 590A has buffer memory available to process the request R2, the scheduler 585 updates the batch for the execution engine 590A to include the new request R2, and instructs the execution engine 590A to execute the updated batch for the next iteration.

[0130] As Figure 5BAs shown, first output tokens are generated for request Rl in execution engine 590A and requests R3, R4, R5 in execution engine 590B after the first iteration is performed. Further, execution engine 590A is now scheduled to perform the updated batch of requests Rl, R2 at a second iteration. Request R2 includes a single input token. In particular, since a first output token has already been generated for request Rl, the second iteration for request Rl can be a decode phase, while the second iteration for request R2 can be an encode phase. Thus, since the execution engine is configured to perform selective batching and is capable of handling requests with different lengths, execution engine 590A can perform both an encode phase and a decode phase for the same batch of requests, as described in connection with Figures 3A-3B As shown, an additional portion of the cache for each request is used to store the key and value after the second iteration. Figure 5B

[0131] As shown, first output tokens are generated for request Rl in execution engine 590A and requests R3, R4, R5 in execution engine 590B after the first iteration is performed. Further, execution engine 590A is now scheduled to perform the updated batch of requests Rl, R2 at a second iteration. Request R2 includes a single input token. In particular, since a first output token has already been generated for request Rl, the second iteration for request Rl can be a decode phase, while the second iteration for request R2 can be an encode phase. Thus, since the execution engine is configured to perform selective batching and is capable of handling requests with different lengths, execution engine 590A can perform both an encode phase and a decode phase for the same batch of requests, as described in connection with Figure 5C As shown, first output tokens are generated for request Rl in execution engine 590A and requests R3, R4, R5 in execution engine 590B after the first iteration is performed. Further, execution engine 590A is now scheduled to perform the updated batch of requests Rl, R2 at a second iteration. Request R2 includes a single input token. In particular, since a first output token has already been generated for request Rl, the second iteration for request Rl can be a decode phase, while the second iteration for request R2 can be an encode phase. Thus, since the execution engine is configured to perform selective batching and is capable of handling requests with different lengths, execution engine 590A can perform both an encode phase and a decode phase for the same batch of requests, as described in connection with

[0132] ​The request processor 580 also receives another new request R7 and forwards it to the scheduler 585. The scheduler 585 stores request R7 in the incoming request queue. In response to determining that requests R2 and R4 have been completed and that execution engine 590A has cache memory available to process request R7, the scheduler 585 updates the batch for execution engine 590A to R1 and R7, and updates the batch for execution engine 590B to R3 and R5. The scheduler 585 instructs execution engines 590A and 590B to execute the updated batches. Thus, by dynamically adjusting the batches during iteration, completed requests can be provided to the requesting client device 110 immediately after processing, and the scheduler 585 can schedule new requests within the batch so that the computing power of the execution engines can be utilized when the memory for completed requests is released.

[0133] like Figure 5D As shown, after the third iteration, a third output token is generated for request R1 in execution engine 590A and requests R3 and R5 in execution engine 590B. Furthermore, execution engine 590A is now scheduled to execute the updated batch of requests R1 and R7 at the fourth iteration, and execution engine 590B is now scheduled to execute the updated batch of requests R3 and R5 at the fourth iteration. Request R7 includes a sequence of two input tokens. When service system 435 receives a request from client device 110, it can continuously execute... Figures 5A-5D The process is similar to the one described in the text.

[0134] As described above, dynamic batching allows the service system 435 to dynamically adjust the batches processed on the execution engine, ensuring full utilization of the execution engine's hardware. This process is relevant to certain batching methods for converter models (e.g., Figures 2A-2B The method is unavailable because internal states, such as key cache tensors and value cache tensors, are maintained at variable lengths in each iteration in the transformer model, while other recursive machine learning models (e.g., RNNs) utilize state data with the same length across a batch of requests.

[0135] Flowcharts showing selective batching and dynamic batching methods are provided.

[0136] Figures 6A-6B This is a flowchart illustrating a method for selective batching using a converter model according to an embodiment. Inference system 130 receives 602 a batch of requests comprising one or more sequences of input tokens. In one instance, the length of the first token sequence for a first request in the batch may differ from the length of the second token sequence for a second request in the batch. Inference system 130 accesses 604 a machine learning converter model comprising at least one set of decoders.

[0137] For one or more iterations, the inference system 130 repeatedly performs the step of applying the set of decoders to the one or more inputs of the request. Specifically, for at least one decoder in the set, the inference system 130 generates 606 one or more queries, one or more keys, and one or more values for the request by applying the QKV weight tensor to the one or more input representations. The queries, keys, and values can be generated by a batch operation. The inference system 130 splits 608 a first query for a first request from the one or more queries, splits 608 a first key for the first request from the one or more keys, and splits 608 a first value for the first request from the one or more values. The inference system 130 also splits a second query for a second request from the one or more queries, splits a second key for the second request from the one or more keys, and splits a second value for the second request from the one or more values.

[0138] The inference system 130 generates 610 a first attention output for the first request by combining at least the first query, the first key, and the first value. The inference system 130 separately generates 612 a second attention output for the second request by combining the second query, the second key, and the second value for the second request. The inference system concatenates 614 at least the first attention output and the second attention output into a concatenated tensor. The inference system 130 generates 616 one or more output representations by applying the weight tensor to the concatenated tensor. The one or more output representations can be generated by a batch operation. The inference system 130 sets 618 one or more output tokens as one or more inputs of the set of decoders for a next iteration. The inference system 130 provides 620 the output tokens generated for at least one request to the client device 110 as a response to the at least one request.

[0139] Figure 7 is a flow diagram illustrating a method for dynamic batching of requests using a transformer model, in accordance with an embodiment. The inference system 130 receives 710 one or more requests for execution by a serving system. The serving system includes a dispatcher and one or more execution engines, each coupled to access a machine learning transformer model including at least a set of decoders. The inference system 130 schedules 712 a batch of requests including the one or more requests for execution on the execution engines by the dispatcher. The inference system 130 generates 714 a first set of output tokens by the execution engines by iteratively applying the transformer model to a first set of inputs of the batch of requests. Applying the transformer model includes applying at least one batch operation to one or more input tensors associated with the batch of requests.

[0140] The inference system 130 receives 716 a new request from a client device through the request handler. The new request can include an input token sequence. The inference system 130 schedules 718 the second batch of requests for execution on the execution engine through the dispatcher in response to determining that the execution engine has memory available to execute the second batch of requests including the one or more requests and the new request. The inference system 130 generates 720 a second set of output tokens from the execution engine by iteratively applying the converter model to a second set of inputs for the second batch of requests including the input token sequence for the new request.

[0141] Hardware components

[0142] Figure 8 is a diagram illustrating a computer system 800 on which the implementations described herein can be implemented within the inference system 130. For example, in the context of Figure 1 the inference system 130 and its subsystems can be implemented using a computer system such as Figure 8 described. The inference system 130 and its subsystems can also be implemented using a combination of multiple computer systems such as Figure 8 described.

[0143] In one implementation, the computer system 800 includes a processing resource 801, a main memory 803, a read-only memory (ROM) 805, a storage device 807, a communication interface 809, and a hardware accelerator 810. The computer system 800 includes at least one processor 801 including CPU cores for processing information, and a main memory 803, such as a random access memory (RAM) or other dynamic storage device, for storing information and instructions to be executed by the processor 801. The main memory 803 can also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by the processor 801. The computer system 800 can further include a ROM 805 or other static storage device for storing static information and instructions for the processor 801. A storage device 807, such as a magnetic disk or optical disk, or solid-state storage, is provided for storing information and instructions. For example, in the context of Figure 4 the training corpus memory 460 of the inference system 130 can be stored in the read-only memory (ROM) 805 or the storage device 807. The computer system 800 also includes one or more hardware accelerators 810. As described above, while the execution engine can be configured with one or more hardware accelerators 810 in a single computer system 800, in another implementation, the execution engine can be configured across multiple hardware accelerators 810 distributed across multiple computer systems 800.

[0144] The communication interface 809 can enable the inference system 130 to communicate with the client device 110 using a communication link (wireless or wired). Using the communication link, the inference system 130 can communicate with different subsystems included in the client device 110 to enable receiving and processing requests. Additionally, in case of an execution engine being configured across multiple hardware accelerators distributed across multiple computer systems 800, the communication interface 809 can also enable communication between the multiple computer systems 800 so that the execution engine can process requests across the multiple hardware accelerators 810.

[0145] The computer system 800 can optionally include a display device 811, such as a cathode ray tube (CRT), an LCD monitor, a LED monitor, a TFT display, or a television set, for example, for displaying graphics and information in, for instance, a graphical user interface. An input mechanism 813, such as a keyboard including alphanumeric and other keys, can be coupled to the computer system 800 for inputting information and command selections to the processor 801. Other, non-limiting, illustrative examples of input mechanism 813 include a mouse, a trackball, a touch-sensitive screen, or a cursor direction key for communicating direction information and command selections to the processor 801 and for

[0146] Examples described herein relate to implementing the techniques described herein using the inference system 130. According to one implementation, the techniques are performed by the inference system 130 in response to the processor 801 executing one or more sequences of one or more instructions contained in the main memory 803. Such instructions can be read into the main memory 803 from another machine-readable medium, such as the storage device 807. Execution of the sequences of instructions contained in the main memory 803 causes the processor 801 to perform the process steps described herein. In alternative implementations, the examples described herein can be implemented using hard-wired circuitry in place of software instructions, or in combination with software instructions.

[0147] The described examples are not limited to any specific combination of hardware circuitry and software.

[0148] The present invention also discloses the following technical solutions:

[0149] 1. A method of executing a batch of requests by applying a machine-learned transducer model, comprising:

[0150] receiving a batch of requests comprising one or more sequences of tokens, wherein a length of a first sequence of tokens for a first request in the batch is different from a length of a second sequence of tokens for a second request in the batch;

[0151] accessing a machine-learned transducer model comprising at least one set of decoders;

[0152] repeatedly performing the following steps for one or more iterations:

[0153] generating one or more output tokens for the request by applying the set of decoders to one or more inputs for the request, the generating further comprising, for at least one decoder in the set:

[0154] generating one or more queries, one or more keys, and one or more values for the request by applying a QKV weight tensor to the one or more input representations, the one or more queries, the one or more keys, and the one or more values generated by a batch operation,

[0155] splitting a first query for the first request from the one or more queries, a first key for the first request from the one or more keys, and a first value for the first request from the one or more values, and splitting a second query for the second request from the one or more queries, a second key for the second request from the one or more keys, and a second value for the second request from the one or more values,

[0156] generating a first attention output for the first request by combining at least the first query, the first key, and the first value,

[0157] generating a second attention output for the second request separately by combining at least a second query, a second key, and a second value for the second request,

[0158] concatenating at least the first attention output and the second attention output into a concatenated tensor, and

[0159] generating one or more output representations by applying at least one or more weight tensors to the concatenated tensor, the one or more output representations generated by a batch operation;

[0160] setting the one or more output tokens as one or more inputs to the set of decoders for a next iteration; and

[0161] providing the output tokens generated for at least one request to a client device as a response to the at least one request.

[0162] 2. The method of claim 1, further comprising:

[0163] concatenating tokens included in the one or more token sequences for the batch of requests along a single dimension to generate a concatenated input tensor, and

[0164] wherein, for a first iteration of the one or more iterations, applying the set of decoders to one or more inputs for the request comprises applying the set of decoders to the concatenated input tensor.

[0165] 3. The method of solution 2, wherein concatenating at least the first attention output and the second attention output into the concatenated tensor further comprises concatenating elements of the first attention output and elements of the second attention output along a single dimension to generate the concatenated tensor.

[0166] 4. The method of solution 1, wherein the first token sequence for the first request is an input token sequence for the first request, and wherein the second token sequence for the second request is an output token generated at a previous iteration.

[0167] 5. The method of solution 1,

[0168] wherein generating the first attention output for the first request comprises:

[0169] adding the first key to a first key cache that additionally includes keys generated for the first request at previous iterations, and adding the first value to a first value cache that additionally includes values generated for the first request at previous iterations, and

[0170] multiplying the first query by the first key cache to generate a first multiplication tensor, and multiplying the first multiplication tensor by the first value cache, and

[0171] wherein generating the second attention output for the second request comprises:

[0172] adding the second key to a second key cache that additionally includes keys generated for the second request at previous iterations, and adding the second value to a second value cache that additionally includes values generated for the second request at previous iterations, and

[0173] multiplying the second query by the second key cache to generate a second multiplication tensor, and multiplying the second multiplication tensor by the second value cache.

[0174] 6. The method of clause 1, wherein the first attention output is generated by performing a set of operations on at least the first query, the first key, and the first value, and wherein the second attention output is generated by performing a second set of operations on at least the second query, the second key, and the second value in a step separate from the first set of operations.

[0175] 7. The method of clause 6, wherein the first attention output is generated by performing the set of operations on a first kernel of an execution engine, and wherein the second attention output is generated by performing the second set of operations on a second kernel of the execution engine or on a second execution engine.

[0176] 8. The method of clause 1, wherein applying the set of decoders of the converter model to one or more inputs of the request comprises performing operations of the converter model on an execution engine comprising a graphics processing unit (GPU) or a tensor processing unit (TPU).

[0177] 9. The method of clause 1, wherein each token in a sequence of input tokens represents a unit of text.

[0178] 10. A non-transitory computer-readable storage medium storing computer program instructions executable to perform operations for performing a batch of requests by applying a machine learning converter model, the operations comprising:

[0179] receiving a batch of requests comprising one or more sequences of tokens, wherein a length of a first sequence of tokens for a first request in the batch is different from a length of a second sequence of tokens for a second request in the batch;

[0180] accessing a machine learning converter model comprising at least a set of decoders;

[0181] for one or more iterations, repeatedly performing the following steps:

[0182] generating one or more output tokens for the request by applying the set of decoders to one or more inputs for the request, the generating further comprising, for at least one decoder in the set:

[0183] generating one or more queries, one or more keys, and one or more values for the request by applying a QKV weight tensor to one or more input representations, the one or more queries, the one or more keys, and the one or more values generated by a batch operation,

[0184] splitting a first query for the first request from the one or more queries, splitting a first keyword for the first request from the one or more keywords, and splitting a first value for the first request from the one or more values, and splitting a second query for the second request from the one or more queries, splitting a second keyword for the second request from the one or more keywords, and splitting a second value for the second request from the one or more values,

[0185] generating a first attention output for the first request by at least combining the first query, the first keyword, and the first value,

[0186] generating a second attention output for the second request separately by at least combining a second query, a second keyword, and a second value for the second request,

[0187] concatenating at least the first attention output and the second attention output into a concatenated tensor, and

[0188] generating one or more output representations by at least applying one or more weight tensors to the concatenated tensor, the one or more output representations generated by a batch operation; setting the one or more output tokens as one or more inputs to the group decoder for a next iteration; and

[0189] providing output tokens generated for at least one request to a client device as a response to the at least one request.

[0190] 11. The non-transitory computer-readable storage medium of technical solution 10, the operations further comprising:

[0191] concatenating tokens included in the one or more sequences of tokens for the batch of requests along a single dimension to generate a concatenated input tensor, and

[0192] wherein, for a first iteration of the one or more iterations, applying the group decoder to one or more inputs for the request comprises applying the group decoder to the concatenated input tensor.

[0193] 12. The non-transitory computer-readable storage medium of technical solution 11, wherein concatenating at least the first attention output and the second attention output into the concatenated tensor further comprises concatenating elements of the first attention output and elements of the second attention output along a single dimension to generate the concatenated tensor.

[0194] 13. The non-transitory computer-readable storage medium of claim 10, wherein the first sequence of tokens for the first request is an input sequence of tokens for the first request, and wherein the second sequence of tokens for the second request is an output token generated at a previous iteration.

[0195] 14. The non-transitory computer-readable storage medium of claim 10,

[0196] wherein generating the first attention output for the first request comprises:

[0197] adding the first key to a first key cache that additionally includes keys generated for the first request at previous iterations, and adding the first value to a first value cache that additionally includes values generated for the first request at previous iterations, and

[0198] multiplying the first query by the first key cache to generate a first multiplied tensor, and multiplying the first multiplied tensor by the first value cache, and

[0199] wherein generating the second attention output for the second request comprises:

[0200] adding the second key to a second key cache that additionally includes keys generated for the second request at previous iterations, and adding the second value to a second value cache that additionally includes values generated for the second request at previous iterations, and

[0201] multiplying the second query by the second key cache to generate a second multiplied tensor, and multiplying the second multiplied tensor by the second value cache.

[0202] 15. The non-transitory computer-readable storage medium of claim 10, wherein the first attention output is generated by performing a set of operations on at least the first query, the first key, and the first value, and wherein the second attention output is generated by performing a second set of operations on at least the second query, the second key, and the second value in a step separate from the first set of operations.

[0203] 16. The non-transitory computer-readable storage medium of claim 15, wherein the first attention output is generated by performing the set of operations on a first kernel of an execution engine, and wherein the second attention output is generated by performing the second set of operations on a second kernel of the execution engine or on a second execution engine.

[0204] 17. The non-transitory computer-readable storage medium of claim 10, wherein applying the set of decoder models of the converter model to the one or more inputs of the request comprises performing operations of the converter model on an execution engine comprising a graphics processing unit (GPU) or a tensor processing unit (TPU).

[0205] 18. The non-transitory computer-readable storage medium of claim 10, wherein each token in the input token sequence represents a unit of text.

[0206] SUMMARY

[0207] The foregoing description of implementations of the application has been presented for the purposes of illustration and description; it is not intended to be exhaustive or to limit the application to the precise form disclosed. Many modifications and variations are possible in light of the above teaching.

[0208] Some portions of this description are presented in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combinations thereof.

[0209] Any of the steps, operations, or processes described herein can be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented with a computer program product comprising a computer-readable medium containing computer program code, which can be executed by a computer processor for performing any or all of the steps, operations, or processes described herein.

[0210] Embodiments of the application can also relate to an apparatus for performing the operations herein. This apparatus can be specially constructed for the required purposes, and / or it can comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program can be stored in a non-transitory, tangible computer readable storage medium or any type of media suitable for storing electronic instructions, which can be coupled to a computer system bus. Furthermore, any computing systems referred to in the specification can include a single processor or can be architectures employing multiple processor designs for increased computing capability.

[0211] Embodiments of the application can also relate to a product that is produced by a computing process described herein. Such products can include information resulting from computing processes, where the information is stored on a non-transitory, tangible computer-readable storage medium and can include any embodiment of a computer program product or other data combination described herein.

[0212] Finally, the language used in the specification has been principally selected for readability and instructional purposes and can not have been selected to delineate or circumscribe the subject application. Accordingly, the scope of the application is intended to be limited only by the claims submitted herewith and it is specifically intended to be limited only in accordance with the appended claims, rather than being construed in a literal and / or limiting sense.

Claims

1. A method of executing a batch of requests by applying a machine learning model, comprising: receiving a batch of two or more token sequences, wherein a length of a first token sequence in the batch is different from a length of a second token sequence in the batch; accessing a machine learning model; generating one or more output representations, the generating further comprising: obtaining one or more queries, one or more keys, and one or more values, wherein the one or more queries are generated by applying a query weight tensor to one or more input representations, wherein at least the one or more queries are generated by a batch operation, splitting a first query for the first token sequence from the one or more queries, a first key from the one or more keys, and a first value from the one or more values, and splitting a second query for the second token sequence from the one or more queries, a second key from the one or more keys, and a second value from the one or more values, generating a first attention output by multiplying at least the first query, the first key, and the first value, generating a second attention output separately by multiplying at least the second query, the second key, and the second value, wherein the second attention output is generated at a different execution engine, a different hardware accelerator, a different graphics processing unit (GPU) core, or a same GPU core than the first attention output, concatenating at least the first attention output and the second attention output into a concatenated tensor, and generating one or more output representations by applying at least one or more weight tensors to the concatenated tensor, the one or more output representations generated by a batch operation, wherein one or more tokens in a token sequence input to the machine learning model represent text units.

2. The method of claim 1, wherein, the machine learning model comprises a set of decoders, and obtaining the one or more queries, the one or more keys, and the one or more values for a decoder comprises: applying a QKV weight tensor comprising the query weight tensor, the key weight tensor, and the value weight tensor to the one or more input representations, wherein the one or more input representations are generated from the one or more token sequences or one or more output representations from a previous decoder.

3. The method of claim 1, wherein, the machine learning model comprises a set of encoders and a set of decoders coupled to the set of encoders, and obtaining the one or more queries, the one or more keys, and the one or more values comprises: obtaining the one or more keys and the one or more values from at least one encoder in the set of encoders, wherein the one or more keys are generated by applying a key weight tensor to one or more output representations of the encoder, and the one or more values are generated by applying a value weight tensor to the one or more output representations of the encoder; and generating the one or more queries for a decoder by applying the query weight tensor to the one or more input representations, wherein the one or more input representations are generated from the one or more sequences of tokens or one or more output representations from a previous decoder.

4. The method of claim 3, further comprising: obtaining one or more third queries, one or more third keys, and one or more third values for input to a self-attention layer prior to obtaining the one or more queries, the one or more keys, and the one or more values, generating an attention output by at least multiplying the one or more third queries, the one or more third keys, and the one or more third values, and wherein the one or more input representations for generating the one or more queries are generated from the attention output from the self-attention layer.

5. The method of claim 3, wherein, the encoder is a final encoder of the set of encoders.

6. The method of claim 1, wherein, the machine learning model comprises a set of encoders, and obtaining the one or more queries, the one or more keys, and the one or more values for an encoder comprises: applying a QKV weight tensor comprising the query weight tensor, the key weight tensor, and the value weight tensor to the one or more input representations, wherein the one or more input representations are generated from the one or more sequences of tokens or one or more output representations from a previous encoder.

7. The method of claim 1, further comprising concatenating tokens in the one or more sequences of tokens for the batch along a single dimension to generate a concatenated input tensor.

8. The method of claim 1, wherein, concatenating at least the first attention output and the second attention output into the concatenated tensor further comprises concatenating elements of the first attention output and elements of the second attention output along a single dimension to generate the concatenated tensor.

9. The method of claim 1, wherein, the first attention output is generated by performing a set of operations on at least the first query, the first key, and the first value, and wherein the second attention output is generated by performing a second set of operations on at least the second query, the second key, and the second value, and wherein the set of operations is performed on a first core of an execution engine, and wherein the second set of operations is performed on a second core of the execution engine or on a second execution engine.

10. The method of claim 1, wherein, the first attention output is generated by performing a set of operations on at least the first query, the first key, and the first value, and wherein the second attention output is generated by performing a second set of operations on at least the second query, the second key, and the second value, and wherein the set of operations is performed on a first core of an execution engine, and wherein the second set of operations is performed on the first core of the execution engine.

11. A non-transitory computer-readable storage medium storing computer program instructions executable to perform operations comprising: receiving a batch of two or more token sequences, wherein a length of a first token sequence in the batch is different from a length of a second token sequence in the batch; accessing a machine learning model; generating one or more output representations, the generating further comprising: obtaining one or more queries, one or more keys, and one or more values, wherein the one or more queries are generated by applying a query weight tensor to one or more input representations, the one or more queries are generated by a batch operation, a first query for the first token sequence is split from the one or more queries, a first key is split from the one or more keys, and a first value is split from the one or more values, and a second query for the second token sequence is split from the one or more queries, a second key is split from the one or more keys, and a second value is split from the one or more values, generating a first attention output by multiplying at least the first query, the first key, and the first value, generating a second attention output separately by multiplying at least the second query, the second key, and the second value, wherein the second attention output is generated at a different execution engine, a different hardware accelerator, a different graphics processing unit (GPU) core, or the same GPU core than the first attention output, concatenating at least the first attention output and the second attention output into a concatenated tensor, and generating one or more output representations by applying at least one or more weight tensors to the concatenated tensor, the one or more output representations generated by a batch operation, wherein one or more tokens in a token sequence input to the machine learning model represent text units.

12. The non-transitory computer-readable storage medium of claim 11, wherein, the machine learning model comprises a set of decoders, and obtaining the one or more queries, the one or more keys, and the one or more values for a decoder comprises: applying a QKV weight tensor comprising the query weight tensor, the key weight tensor, and the value weight tensor to the one or more input representations, wherein the one or more input representations are generated from the one or more token sequences or one or more output representations from a previous decoder.

13. The non-transitory computer-readable storage medium of claim 11, wherein, the machine learning model comprises a set of encoders and a set of decoders coupled to the set of encoders, and obtaining the one or more queries, the one or more keys, and the one or more values comprises: applying a QKV weight tensor comprising the query weight tensor, the key weight tensor, and the value weight tensor to the one or more input representations, wherein the one or more input representations are generated from the one or more token sequences or one or more output representations from a previous decoder. obtaining the one or more queries, the one or more keys, and the one or more values from at least one encoder of a set of encoders, wherein the one or more keys are generated by applying a key weight tensor to one or more output representations of the encoder and the one or more values are generated by applying a value weight tensor to the one or more output representations of the encoder; and generating the one or more queries for a decoder by applying the query weight tensor to the one or more input representations, wherein the one or more input representations are generated from the one or more sequences of tokens or one or more output representations from a previous decoder.

14. The non-transitory computer-readable storage medium of claim 13, further comprising: obtaining one or more third queries, one or more third keys, and one or more third values for input to a self-attention layer prior to obtaining the one or more queries, the one or more keys, and the one or more values, generating an attention output by multiplying at least the one or more third queries, the one or more third keys, and the one or more third values, and wherein the one or more input representations for generating the one or more queries are generated from the attention output from the self-attention layer.

15. The non-transitory computer-readable storage medium of claim 13, wherein, the encoder is a final encoder of the set of encoders.

16. The non-transitory computer-readable storage medium of claim 11, wherein, the machine learning model comprises a set of encoders, and obtaining the one or more queries, the one or more keys, and the one or more values for an encoder comprises: applying a QKV weight tensor comprising the query weight tensor, the key weight tensor, and the value weight tensor to the one or more input representations, wherein the one or more input representations are generated from the one or more sequences of tokens or one or more output representations from a previous encoder.

17. The non-transitory computer-readable storage medium of claim 11, further comprising concatenating tokens in the one or more sequences of tokens for the batch along a single dimension to generate a concatenated input tensor.

18. The non-transitory computer-readable storage medium of claim 11, wherein, concatenating at least the first attention output and the second attention output into the concatenated tensor further comprises concatenating elements of the first attention output and elements of the second attention output along a single dimension to generate the concatenated tensor.

19. The non-transitory computer-readable storage medium of claim 11, wherein, the first attention output is generated by performing a set of operations on at least the first query, the first key, and the first value, and wherein the second attention output is generated by performing a second set of operations on at least the second query, the second key, and the second value, and wherein the set of operations is performed on a first core of an execution engine, and wherein the second set of operations is performed on a second core of the execution engine or on a second execution engine.

20. The non-transitory computer-readable storage medium of claim 11, wherein, The first attention output is generated by executing a set of operations on at least the first query, the first key, and the first value, and wherein the second attention output is generated by executing a second set of operations on at least the second query, the second key, and the second value, and wherein the set of operations is executed on a first core of an execution engine, and wherein the second set of operations is executed on the first core of the execution engine.

Images