Elastic converter service system self-adaptive through lexical elements

Abstract

The present disclosure relates to methods and systems based on a transformer model, in which an elastic transformer service system, referred to herein as an online lexical adaptation system (OTAS), is involved that accommodates diversified user requests with fluctuating query loads, while optimizing output accuracy and runtime delay. The OTAS uses lexical adaptation techniques that involve adding cue lexical elements to improve accuracy and removing redundant lexical elements to accelerate reasoning. To handle fluctuating query loads and diversified user requests, the OTAS also applies a perceptual selective batch processing algorithm for online lexical adaptive design. In a system service process, OTAS firstly processes incoming queries with similar requirements in batches to improve ingress throughput. Then, in order to balance between the overhead of lexical element growth and the potential of improving accuracy, OTAS adaptively adjusts lexical element execution settings by solving an optimization problem.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 591,112, filed on October 17, 2023, entitled “ELASTIC TRANSFORMER SERVICE SYSTEM VIA LETTERS ADAPTATION” The entire contents of this prior application are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a cloud-based transformer model service for artificial intelligence (AI) applications, and more particularly, to a flexible transformer service system implemented through token adaptation. Background Art

[0004] Being able to effectively and efficiently deliver machine learning models for artificial intelligence (AI) applications through cloud-based architectures is important for many industries, which can greatly affect the quality of user experience and the resulting economic profits. For example, Facebook TM With billions of daily active users issuing tens of trillions of model inference queries every day, this requires a fundamental redesign to facilitate model optimization and serving efficiency.

[0005] Recent advances in self-supervised pre-training techniques have facilitated the development of large Transformer models for many AI applications. These pre-trained Transformer models have dramatically changed our lives and have great potential for our society. For example, large generative pre-trained Transformer (GPT) models like GPT-2 and GPT-3 have spawned a large number of AI applications, such as Copilot TM and ChatGPT TM etc. Specifically, ChatGPT TM It has over 100 million active users and received 176 million visits in April 2023.

[0006] Despite the surge in such diverse applications, the resource-intensive nature of transformer models, coupled with dynamic query loads and heterogeneous user requirements, has exacerbated the challenges associated with cloud-based transformer model services, making it extremely challenging to efficiently adapt to various service requirements. For example, existing techniques for improving service quality and taking into account different user needs and service requirements involve model adaptation. Model adaptation involves training multiple model variants of a base transformer model of different sizes to adapt to changing service requirements. These multiple model variants are stored by the service system and are dynamically selected and applied to adapt to changes in query loads. Unfortunately, this technique is not suitable for large transformer models because training these models is prohibitive in terms of high monetary costs and time overhead. For example, training a GPT-3 model can consume thousands of petaflops and days of computing power and cost millions of dollars. In addition, loading this type of model into a service system processing unit (e.g., a graphics processing unit (GPU), etc.) for execution may take several seconds, which adds huge latency costs because a large number (hundreds, thousands, millions, etc.) of user queries are served even in a relatively short service period (e.g., about 30 minutes). Furthermore, the sizes of different Transformer models vary significantly, making it difficult to prepare different fine-grained model versions tailored for different inference tasks.

[0007] Therefore, there is a need for techniques to efficiently facilitate provisioning cloud-based transformer models with reduced training costs and computing resources that adapt to dynamic query loads and heterogeneous user needs. Summary of the invention

[0008] An overview is presented below to provide a basic understanding of one or more embodiments of the present disclosure. This overview is not intended to identify key or important elements or to delineate any scope of the different embodiments or any scope of the claims. Its sole purpose is to present the concepts in a simplified form as a prelude to a more detailed description presented later. In one or more embodiments, a system, computer-implemented method, apparatus, and / or computer program product that implements an elastic transformer service system that is adaptive through word units is described.

[0009] According to an embodiment, a computing system including at least one processor is provided. The computing system is configured to receive a query for processing by a transformer model, wherein the processing of the query is associated with a tokenized representation of each input data sample for processing the query, and the tokenized representation includes each initial token number of the execution token. The computing system is also configured to determine a token execution setting for processing the query using the transformer model based on one or more characteristics of the query, wherein at least some of the token execution settings include token adaptation settings, and the token adaptation settings include using the transformer model to adjust at least some initial token numbers through at least one of a token prompting process and a token merging process. After instructing the transformer model to adopt the token execution setting, the computing system applies the query as an input to the transformer model to generate an inference result for the query.

[0010] To this end, using the transformer model to adjust the number of each initial word (at least some of them) through the word-gram hinting process leads to an improvement in the accuracy of the inference result. In addition, using the transformer model to adjust the number of each initial word (at least some of them) through the word-gram merging process leads to a reduction in the latency level of the transformer model associated with the processing of the query.

[0011] In one or more embodiments, the one or more characteristics are selected from the group consisting of: a first characteristic applicable to the content of the input data sample, a second characteristic applicable to the query task, a third characteristic applicable to the utility value, and a fourth characteristic applicable to the latency requirement. According to the disclosed technology, based on the different characteristics of the query, the word unit execution settings can vary between queries and be dynamically customized to optimize the balance between the accuracy and latency of the inference results.

[0012] In another embodiment, a computing system is provided that includes at least one processor. The at least one processor is configured to receive queries for processing by a transformer model, wherein processing of the queries is associated with processing tokenized representations of respective input data samples of the queries, the tokenized representations including respective initial token numbers for executing tokens. The computing system is further configured to distribute the queries into different batches that respectively include different subsets of the queries.

[0013] For each of the different batches, the computing system also determines an adaptive number of word units for corresponding execution word units of each query included in the batch based on one or more characteristics of the queries included in the batch. The one or more characteristics may include a first characteristic related to the content of the input data sample, a second characteristic related to the query task, a third characteristic related to the utility value, and a fourth characteristic related to the delay requirement. The computing system is also configured to instruct the transformer model to apply the adaptive number of word units when applying the batch as input to the transformer model to adjust or maintain the corresponding initial number of word units of each execution word unit. The computing system is also configured to apply the batch as input to the transformer model to generate inference results for the queries included in the batch, wherein the transformer model is configured to apply the adaptive number of word units when applying the batch as input to the transformer model to adjust or maintain the corresponding initial number of word units of each execution word unit.

[0014] At this point, based on the word-gram adaptive number being greater than the threshold number, the transformer model configured according to the word-gram adaptive number adds one or more hint word-grams to the corresponding execution word-gram. Based on the word-gram adaptive number being less than the threshold number, the transformer model configured according to the word-gram adaptive number uses a word-gram merging process to reduce the corresponding initial word-gram number. Based on the word-gram adaptive number being equal to the threshold number, the transformer model configured according to the word-gram adaptive number maintains the corresponding initial word-gram number. To this end, based on the addition of one or more hint word-grams, the accuracy of the inference results of the batch is increased, and based on the reduction of the initial word-gram number, the delay level of the transformer model associated with the processing of the batch is reduced.

[0015] In one or more embodiments, allocating involves allocating queries into different batches by grouping similar queries having similar characteristics into the same batch according to a defined similarity criterion or a defined similarity metric. The similar characteristics are selected from the group consisting of: a first characteristic related to an arrival time of the query, a second characteristic related to the query task, a third characteristic related to a utility value, and a fourth characteristic related to a delay requirement. The computing system may also determine an execution order of the different batches based on the arrival times of different queries included in the different batches, store the different batches in a batch queue, and sequentially apply the transformer model to the different batches according to the execution order.

[0016] In some implementations, the computing system may also determine, based on the query load of the batch queue, an adaptive number of word-grams for corresponding execution word-grams of each query included in the batch.

[0017] In some implementations, elements described in conjunction with the disclosed system may be embodied in a different form, such as a computer-implemented method, a computer program product, or another form. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A block diagram of an exemplary, non-limiting computing system implementing an elastic transformer service through word-gram adaptation in accordance with one or more implementations of the disclosed subject matter is shown.

[0019] Figure 2 An exemplary transformer model including word-gram hinting and word-gram merging is shown in accordance with one or more implementations of the disclosed subject matter.

[0020] Figure 3A A high level converter servicing process according to the model adaptation technique is shown.

[0021] Figure 3B A high-level transformer service process according to the disclosed word-gram adaptation technique is shown in accordance with one or more implementations of the disclosed subject matter.

[0022] Figure 4 Prompted learning of a transformer model in accordance with one or more implementations of the disclosed subject matter is shown.

[0023] Figure 5 An exemplary word-gram merging mechanism in accordance with one or more implementations of the disclosed subject matter is shown.

[0024] Figure 6 An exemplary process for registering a task and generating task parameters and task analytics data in accordance with one or more implementations of the disclosed subject matter is shown.

[0025] Figure 7 An exemplary elastic transformer service process through word-gram adaptation is shown in accordance with one or more implementations of the disclosed subject matter.

[0026] Figure 8 An exemplary batch processing process in accordance with one or more implementations of the disclosed subject matter is shown.

[0027] Fig. 9 An exemplary word-gram adaptation process in accordance with one or more implementations of the disclosed subject matter is shown.

[0028] Fig.10 Another exemplary word-gram adaptation process in accordance with one or more implementations of the disclosed subject matter is shown.

[0029] Fig.11 Another exemplary elastic transformer service process through word-gram adaptation in accordance with one or more implementations of the disclosed subject matter is shown.

[0030] Fig.12An exemplary computer-implemented method for performing elastic transformer services in accordance with one or more implementations of the disclosed subject matter is shown.

[0031] Fig.13 Another exemplary computer-implemented method for performing elastic transformer services in accordance with one or more implementations of the disclosed subject matter is shown.

[0032] Fig.14 A table of a projection function from arrival rate to number of token adaptations according to an exemplary implementation of the disclosed online token adaptation system (OTAS) is presented.

[0033] Fig.15 A table of latency and utility of queries according to an exemplary implementation of the disclosed OTAS is presented.

[0034] Fig.16A and Fig. 16B Query trace graphs for different data sets according to an exemplary implementation of the disclosed OTAS are presented.

[0035] Fig.17A and Fig. 17B A comparison of the utility of different transformer service system designs on different datasets is presented.

[0036] Fig.18A and Fig.18B A comparison of the effectiveness of different transformer service methods on different datasets is presented.

[0037] Fig.19A and Fig.19B The cumulative distribution function (CDF) plots of the accuracy of different elastic transformer service methods on different datasets are presented.

[0038] Fig. 20A and Fig. 20B A pie chart showing the distribution of different word-gram adaptation numbers used on different datasets according to an exemplary implementation of the disclosed OTAS is presented.

[0039] Fig.21A and Fig.21B A graph comparing the ratio of execution information of different transformer service methods on different data sets is presented.

[0040] Fig. 22 An exemplary, non-limiting operating environment is shown that may facilitate one or more embodiments described herein.

[0041] Fig.23is a block diagram illustrating an exemplary computing environment with which the disclosed subject matter can interact according to implementations. DETAILED DESCRIPTION

[0042] The following detailed description is illustrative only and is not intended to limit the application or use of the embodiments and / or embodiments. In addition, it is not intended to be bound by any expressed or implied information presented in the previous background technology section, the invention summary section or the detailed description section.

[0043] In one or more embodiments, systems, computer-implemented methods, apparatus, and / or computer program products are described that implement an elastic transformer service system (referred to herein as an online token adaptation system (OTAS)) that is adaptive through tokens. A token is a basic unit of an input data sample to be processed by a transformer model, such as a text unit, a code unit, or a patch of an image. The transformer model segments the input data sample of an input query into input tokens (e.g., corresponding to different patches of an input image, different words in an input text prompt, etc.) and processes these tokens using an attention mechanism that calculates the similarity (i.e., attention weights) between the input query and a key, and projects the attention values ​​using the similarity weights to obtain a new representation. A key feature of the transformer model's attention mechanism is its ability to support token sequences of varying lengths.

[0044] Driven by this feature, the disclosed OTAS adopts a novel word-element adaptation technique, which involves dynamically adjusting the execution word-element of the transformer model to improve the quality of service with negligible training cost. More specifically, the word-element adaptation technique involves adding hint words to improve the inference accuracy of the transformer model, and removing redundant words to accelerate the inference speed of the transformer model. OTAS is designed to handle incoming request queries with varying arrival times, inputs, tasks, utilities, and latency requirements to be processed by the transformer model. In various embodiments, OTAS customizes the word-element adaptation settings applied by the transformer model for the corresponding request query based on the corresponding characteristics of the query (e.g., arrival time, input task, utility, and / or latency requirements) and the fluctuating query load. In other embodiments, in order to further improve the quality of service, OTAS designs an application-aware selective batching algorithm for online word-element adaptation in view of the fluctuating query load and diverse user requests. In an exemplary embodiment, OTAS first batches incoming queries with similar service level characteristics to improve the inbound throughput. Then, to balance the overhead of token growth with the potential for improved inference result accuracy, OTAS adaptively adjusts the token execution settings for the corresponding batch by solving an optimization problem fundamentally balancing increased accuracy and reduced transformer model processing latency.

[0045] One or more embodiments are now described with reference to the accompanying drawings, wherein the same figure words are used to refer to the same elements throughout. In the following description, for the purpose of explanation, many specific details are set forth to provide a more thorough understanding of one or more embodiments. However, it is apparent that in various circumstances, one or more embodiments may be practiced without these specific details.

[0046] Now turning to the attached figure, Figure 1 A block diagram of an exemplary, non-limiting computing system that implements an elastic transformer service through word-gram adaptation according to one or more embodiments of the disclosed subject matter is illustrated. The computing system 100 may include or correspond to one or more computing devices, machines, virtual machines, computer-executable components, data stores, etc., which may be communicatively coupled to each other directly or via one or more wired or wireless communication frameworks.

[0047] The computing system 100 may include computer-executable (i.e., machine-executable) components or instructions implemented within one or more machines (e.g., implemented in one or more computer-readable storage media associated with the one or more machines) that may perform one or more operations described with respect to the corresponding components. For example, the computing system 100 may include (or be operatively coupled to) at least one memory 142 storing computer-executable components 108 and at least one processor (e.g., at least one processing unit 144) executing the computer-executable components 108 stored in the at least one memory 142. These computer-executable components may include (but are not limited to) a receiving component 110, a query interface component 112, a scheduling component 114, a word-gram setting component 116, a performance analysis component 118, an execution component 122, a transformer model 122, a task registration component 124, a task analyzer component 126, and a training component 128. The memory 142 may also store data 130 received, used, and / or generated by the computer executable components 108 to facilitate the operations described with respect to the data 130 (e.g., task registration data 132, task analysis data 134, hint repository data 136, and batch queue 138). In various embodiments, the processing unit 144 includes or corresponds to a graphics processing unit (GPU) or a tensor processing unit (TPU). Additional examples of the memory 142 and processing unit 144, as well as other suitable computer or computing-based elements, may be referenced. Fig. 22 (e.g., system memory 2206 and processing unit 2204, respectively) and may be implemented in conjunction with Figure 1 Or one or more components shown and described in other figures disclosed herein.

[0048] The computing system 100 may also include one or more input / output devices 146 to facilitate receiving user input and providing data to the user associated with performing various operations with respect to the machine-executable components 108 and / or processes described herein. Fig. 22 Suitable examples of input / output devices 146 are described.Computing system 100 may also include a system bus 140 that couples memory 142, processing unit 144, and input / output devices 146 to one another.

[0049] According to various embodiments, the computing system 100 may be configured to receive (e.g., via the receiving component 110) a query 102 to be processed by the transformer model 122, and to execute the transformer model 122 for the query 102 (e.g., via the execution component 120 and the processing unit 144) to generate an inference result corresponding to the corresponding query. In various embodiments, the computing system 100 may include or correspond to a cloud-based server that hosts (e.g., stored in the memory 142) and executes (e.g., via the processing unit 144 and the execution component 120) the transformer model 122. To this end, the computing system 100 may receive the query 102 from a plurality of user devices via any suitable wired or wireless communication network, and return the corresponding inference result 106 to the corresponding user device via the wired or wireless communication network.

[0050] In some implementations, query interface component 112 can facilitate computing system 100 receiving query 102. For example, query interface component 112 can provide a user interface (e.g., a graphical user interface) that can be presented to a user (e.g., provided via their respective user device) to enable the user to enter and submit a request query to be processed by transformer model 122.

[0051] In various embodiments, the transformer model 122 may include pre-trained transformer models (e.g., GPT-2, GPT-3, GPT-4, and various other non-proprietary and proprietary pre-trained GPTs) configured to perform a variety of different reasoning tasks. For example, in some embodiments, the transformer model 122 may include or correspond to a large language model (LLM). A large language model (LLM) is a type of GPT designed to understand, generate, and manipulate human language. These models are built using advanced machine learning techniques, particularly deep learning, and are trained on large amounts of text data. LLMS (such as ChatGPT, etc.) have shown extraordinary success in various natural language processing tasks (such as text generation and question answering). For example, LLMs can generate consistent and contextually relevant text based on given input requests (such as prompts or questions provided in natural human language), making them useful for writing papers, articles, stories, etc. LLMS can also answer questions by extracting and synthesizing information from texts that have been trained. For example, when applied to a use case in the medical field, the large amount of clinical text data that LLM is trained on can be used to analyze patient electronic health records (EHRs) to answer questions related to the patient's medical history by extracting relevant information from the unstructured text in the patient's EHR using natural language processing (NLP).

[0052] In other embodiments, the transformer model 122 may include or correspond to a pre-trained GPT that has been fine-tuned or specifically designed to generate computer code. These models can understand natural language prompts and generate corresponding code in various programming languages ​​such as Python, JavaScript, C++, etc. These models can be used to perform a variety of different reasoning tasks, such as converting natural language to code, code completion (e.g., automatically completing lines of code given an input code snippet), code interpretation (e.g., explaining the function of a piece of input code), and other tasks.

[0053] In other embodiments, the transformer model 122 may include or correspond to a pre-trained GPT that is adapted or extended to handle tasks related to image generation, manipulation, or understanding. These types of GPTs are generally referred to as visual transformers. For example, tasks may include text-to-image generation (e.g., generating an image from a text description input), image captioning or image classification (e.g., taking an image as input and generating a text description of what the image depicts), image style transfer or adaptation (e.g., changing an input image to appear to have a particular style, image quality, etc.), and various other applications or tasks.

[0054] The transformer model 122 may also include or correspond to a pre-trained GPT that is suitable for processing tasks related to processing other forms of input data (e.g., audio data, video data, sensor data, etc.) and generating output data in various formats (e.g., text, images, audio, code, etc.). In other embodiments, the transformer model may include or correspond to a multi-model GPT that can process and generate data in multiple modalities (such as text, images, audio, etc.). Unlike traditional GPT models that are primarily text-based, multi-model GPTs are designed to handle inputs and outputs across different types of data, allowing more complex and general interactions and tasks.

[0055] To this end, regardless of the type of transformer model 122 and the type of input data (e.g., corresponding to the input data included in the query 102) and output data (e.g., corresponding to the inference result 106) that the transformer model 122 has been pre-trained to process and generate, the transformer model 122 can be configured to perform a variety of different tasks associated with different queries. According to the disclosed technology, the computing system 100 is designed to optimize the accuracy of the inference results 106 generated by the transformer model 122 in the use scenario of the query 102 with respect to the characteristics of the change, including but not limited to: the specific task requested to be performed by the transformer model 122 represented by the corresponding query 102, the specific input data content of the corresponding query 102 (e.g., with respect to the amount and type of input data), the utility reward associated with the query (e.g., a measure corresponding to a certain gain attributed to the service corresponding request, such as a monetary gain, etc., which can vary for different queries), and the latency requirements associated with the corresponding query (e.g., a latency constraint corresponding to the corresponding query, which can vary for different queries). At the same time, the computing system 100 is also designed to optimize the inference speed or latency of the computing system 100 associated with processing the query 102 to generate the corresponding inference result 106, especially in the scenario where the transformer model 122 is used to process a large number (e.g., hundreds, thousands, etc.) of queries received simultaneously or substantially simultaneously at a high inflow rate (e.g., hundreds to thousands of queries 102 per second, or another inflow rate), thereby increasing the total online service throughput of the computing system 100 in any given service period. The computing system 100 is also designed to handle query loads that vary over time. In this context, the computing system 100 is also designed to adapt to the changing query load associated with balancing improving or maintaining the accuracy of the inference results and minimizing the latency, while taking into account the changing service characteristics of the query 102.

[0056] To facilitate this purpose, the computing system 100 employs a novel technique called word-gram adaptation that is motivated by the key characteristics of the transformation model (i.e., the employed attention mechanism and its ability to support word-gram sequences of varying lengths). In this regard, in general, all transformer models (e.g., including the transformer model 122 employed by the computing system 100) work by initially segmenting the input data of the received query into smaller blocks and converting these blocks into word-grams (a process known as word-gramming). For example, when applied to input text, word-grams include smaller units of the input text, typically words, subwords, or characters, depending on the tokenizer used. For example, the sentence "The transformer is powerful" may be tokenized into ["transformer", "is", "powerful"] or subword word-grams such as ["transformer", "device", "is", "powerful", "of"]. In another example, when applied to an input image, a word-gram includes a fixed-size patch of the input image. In other words, a word-gram is a basic unit of an input data sample to be processed by the transformer model 122, such as a text unit, a code unit, or a patch of an image. The number of input tokens generated for a given input data sample included in a query may vary based on the content of the input data sample (e.g., number of words, size of an image, etc.). Thus, the attention mechanism of the transformer model 122 may handle input token sequences of varying lengths.

[0057] In some implementations, the tokenization process can be integrated into the transformer model 122. In other implementations, the tokenization process can correspond to a pre-processing step performed by the receiving component 110 on the query 102 received by the computing system 100. In this regard, the transformer model 122 is configured to process a plurality of tokenized representations of the corresponding input data samples of the query 102, the plurality of tokenized representations respectively including a number of tokens, wherein the number varies according to the content of the input data sample and one or more data types included in the input data sample (e.g., text data, image data, code data, audio data, sensor data, etc.).

[0058] For example, Figure 2 An example of a transformer model 122 according to one or more implementations of the disclosed subject matter is illustrated. In this example, the transformer model 122 is configured to process a tokenized representation of an image to generate a corresponding inference result (e.g., corresponding to output 236). The specific inference task performed by the transformer model 122 on the input image can vary. For example, the inference result can correspond to a classification of an object in the input image, a textual description of what the input image depicts, a modified version of the input image, etc. It should be understood that although in Figure 2The transformer model 122 is illustrated as a visual transformer in FIG. 1 , but the transformer model 122 is not limited to the visual transformer and may correspond to a GPT configured to process various types of input data and generate various types of output data.

[0059] Reference Figure 1 and Figure 2 ,like Figure 2 As shown, the input image is segmented into patches (e.g., input patches 202), and the patches are transformed into a sequence of tokens 202 by the transformer model 122 through a linear projection performed by a linear projection layer 204. More specifically, after the input data samples of the query have been segmented into tokens (e.g., tokens 705), the transformer model 122 maps each token to a dense vector representation called an embedding. These embeddings are learned during training and capture the semantic meaning of the tokens in a continuous vector space. Since the transformer lacks inherent sequence information (unlike a recurrent neural network (RNN)), positional encodings (e.g., through a positional embedding layer 206) are added to the token embeddings to introduce information about the token position within the sequence. This allows the model to understand the order of tokens.

[0060] The core of the transformer model is the self-attention mechanism. It allows the model to weight the importance of each word relative to other words in the sequence, regardless of their distance from each other. Typically, the attention mechanism is performed by the encoder neural network of the transformer (i.e., the transformer encoder 208), which calculates the similarity (i.e., attention weight) between the input query and the key (represented by the word), and uses the similarity weight to project the attention value to obtain a new representation. More specifically, for each word, the transformer model calculates three vectors: query (Q), key (K), and value (V). These vectors are obtained by multiplying the word embedding with the learned weight matrix. The query vector of the word is compared with the key vectors of all words in the sequence to calculate the attention score (dot product). These scores determine the degree of attention each word should pay to other words. The attention scores are passed through the softmax function to normalize them to probabilities. The final representation of each word is the weighted sum of the value vectors, where the weight is the attention probability.

[0061] In this regard, the Transformer model uses multiple attention heads in order to capture different types of relationships and patterns in the data. Figure 2 As shown, the transformer encoder 208 is stacked by multiple attention blocks, which may include a normalization layer (e.g., normalization layer 220 and normalization 224), a multi-head attention layer 222, and a multi-layer perception (MLP) layer 232. Figure 2As shown, in addition to these layers, the transformer encoder 208 of the transformer model 122 also includes a hint layer 210 before the multi-head attention layer 222 and a merge layer 226 inserted between the multi-head attention layer 222 and the MLP layer 232. These two additional layers will be described in more detail below.

[0062] The multi-head attention layer 222 allows the model to extract features from different representation spaces, and each space is called an attention head i. Note that there is a query Q, a key K, and a value V, and Q, K, Where n is the length of the sequence of tokens, and d model is the feature dimension. As shown in the following formula (1), Q, K, and V are first mapped to a low-dimensional space with projection parameters. Then, the model uses the attention mechanism to model the interaction between tokens and extracts semantic features according to formula (2).

[0063]

[0064] According to formula (2), we first calculate the query Q i and key K i The attention weights between them are then applied to V i to obtain a new representation. i , K i , And d k d v is a feature representation. In self-attention, Q, K, and V are equal to the input x of each layer. Each attention head of the multi-head attention layer 222 independently performs the self-attention process using its own learned weights. The outputs of all attention heads are cascaded by the multi-head attention layer 222 to generate cascaded attention heads. The cascaded attention heads are further processed by a linear module (not shown), and the output of the multi-head attention layer is forwarded to the subsequent block.

[0065] In this regard, according to a conventional transformer model, after the multi-head attention layer 222, the representation of each word unit is standardized (e.g., by a normalization layer 224) and passed through a feed-forward neural network (FFN) referred to as an MLP layer 232. The MLP layer typically consists of two linear transformations with nonlinear activations in between. This step introduces nonlinearity and further refines the word unit representation. In order to improve the flow of gradients and stabilize learning during training, the transformer model uses residual connections. The input of each sub-layer (such as self-attention or FFN) is added to its output. After the residual connection, layer normalization is applied to standardize the output, ensuring that the model remains stable during training.

[0066] The output of the transformer encoder 208 is a collection of context-enriched word-gram representations represented by the head 234, where the vector of each word-gram has been adjusted to include information from all other words-grams in the sequence. Inference output 236 is generated by the transformer model 122 by processing the final contextualized word-gram representations, which vary based on the specific task involved. For example, in tasks such as machine translation or text generation, the final word-gram representation is passed to the decoder (in an encoder-decoder model) or used directly to predict the next word-gram in the sequence.

[0067] As described in the background section, the prior art relates to model adaptation for improving the quality of service provided by a cloud-based transformer model hosting and executing a system while taking into account different query and service requirements. Figure 3A An overview of this level of the model adaptation process is shown in .

[0068] according to Figure 1 And refer to Figure 3A , Figure 3A A high-level transformer service process 300A according to a model adaptation technique is illustrated. The process 300A is illustrated as being applied to a visual transformer model where the service is configured to process an input image 302 to perform a specific requested reasoning task on the input image, such as object classification or another task indicated in a query. As described above, at 304 the input image is first segmented into patches, and at 306 the patches are tokenized. At 308A, the input tokens are further processed by an attention head mechanism of a transformer model hosted by the service system. Figure 3A As shown, the model adaptation technique used by the hosting system at 308A involves training multiple model variants of base (i.e., pre-trained) transformer models (denoted by letters T, S, B, and L) of different sizes to adapt to changing service requirements. These multiple model variants are stored by the service system and are dynamically selected and applied to adapt to changes in query load during runtime. For example, at 308A in process 300A, the B variant of the transformer model has been selected to process the input token.

[0069] Unfortunately, this technique is not applicable to large transformer models because training these models can be prohibitive in terms of high monetary cost and time overhead. In addition, loading this type of model into a serving system processing unit (e.g., a graphics processing unit (GPU), etc.) for execution can take several seconds, which adds a huge latency cost due to input / output latency (I / O) delays, since a large number (hundreds, thousands, millions, etc.) of user queries must be served even within a relatively short serving period (e.g., 30 minutes or so). In addition, the sizes of different transformer models vary significantly, making it difficult to prepare different fine-grained model versions customized for different inference tasks.

[0070] Figure 3B A high-level transformer service process 300B according to the disclosed token adaptation technique is illustrated in accordance with one or more implementations of the disclosed subject matter. For ease of comparison with process 300A, process 300B is also illustrated as being applied to a visual transformer model that is configured to process an input image 302 to perform a specific requested reasoning task (such as object classification or another task indicated in a query) on the input image. As with process 300A, the input image is segmented into patches at 304 and tokenized at 306. However, at 308B, a hosting system (e.g., corresponding to computing system 100) employs a unified transformer model (e.g., transformer model 122) to process the input tokens. In this regard, instead of training multiple different models from scratch and selecting to apply a specific model customized for the input query, the hosting system can employ a large pre-trained model that has been specifically configured to perform the novel token adaptation technique in order to minimize latency, I / O latency, and serve queries with a wide variety of characteristics (e.g., regarding input content, tasks, utility rewards, and latency requirements) and fluctuating query loads at runtime (i.e., online). To this end, the hosting / service system corresponding to computing system 100 is referred to herein as an online word-gram adaptation system (OTAS). In some implementations, the characteristics of the query are selected from the group consisting of: a first characteristic applicable to the content of the input data sample, a second characteristic applicable to the query task, a third characteristic applicable to the utility value, and a fourth characteristic applicable to the latency requirement.

[0071] like Figure 3B As shown, the word-gram adaptation technique involves a word-gram prompting process and / or a word-gram reduction process (also referred to herein as a word-gram merging process). Figure 2As shown, in addition to the conventional transformer layers and processing mechanisms discussed above for the general transformer model, the transformer model 122 used by the computing system 100 also includes an additional hinting layer 210 configured to perform a word-gram hinting process (e.g., process 212 in some embodiments) and an additional merging layer 226 configured to perform a word-gram merging process (process 228 in some embodiments).

[0072] Reference Figure 1 , Figure 2 and Figure 3B Typically, the word-gram hinting process 212 performed by the hinting layer 210 involves adding one or more hint words to the initial word-grams generated for the input data sample through the word-gramming fed as input to the transformer encoder 208. The added hint words can be used to improve the accuracy of the corresponding reasoning results generated by the transformer model 122 customized for the requested reasoning task. However, the added hint words can increase the processing delay (or reduce the processing speed) of the transformer model 122. On the other hand, the word-gram merging process 228 performed by the merging layer 226 removes unnecessary words from the initial input words, which have little or no effect on the accuracy of the reasoning results. The reduced number of words is equal to the reduction in the processing delay (or increase in processing speed) of the transformer model 122.

[0073] Figure 4 The concept of word-gram hints for a transformer model is illustrated in accordance with one or more implementations of the disclosed subject matter. Figures 1 to 4 , Figure 4 A simplified view of a transformer encoder 400 representing a general transformer model. Token hinting involves adding a pre-trained (and pre-defined) hint token 404 to an initial input token 402 (e.g., an initial token generated by tokenization of the input data). The combined hint token 404 and the initial input token 402 form a runtime execution token, which are cascaded and forwarded to the transformer encoder layer for multi-head attention. Hint tokens correspond to task-specific clues that are used to add context to the query and model input to guide the output response. In this regard, hint tokens are carefully designed to elicit specific types of responses or outputs through the transformer model. For example, assume that the transformer model corresponds to an LLM, and a user wants to use the LLM to generate a query for product descriptions of an online store. Token hint input can be: Input: "Product: Wireless Bluetooth Headphones. Description: These headphones are [good / bad] for music lovers because...", where the hint token corresponds to the added token of the word "good" or "bad". At this point, depending on the word / cue token inserted (e.g., "good"), the model will proceed with a positive description, while "bad" may lead to a negative description.

[0074] According to the disclosed technique, hint tokens are randomly initialized and trained using stochastic gradient descent (e.g., by training component 128). During inference, as discussed in more detail below, well-trained hint tokens can be added by transformer encoder 208 directly before the initial input tokens defined for the specific reasoning task (and the model output response to be elicited) via an added hint layer 210.

[0075] Figure 5 The concept of word-gram merging for a transformer model according to one or more embodiments of the disclosed subject matter is illustrated. Generally, word-gram merging involves removing redundant or unnecessary word-grams, which speeds up inference. Figures 1 to 5 In various implementations, the word-gram merging process 228 performed by the merging layer 226 involves merging similar word-grams together using a word-gram merging method. Figure 5 An exemplary word-gram merging method 500 that can be used by the merging layer 226 to reduce the number of word-grams is shown. According to the word-gram merging method 500, at 501, word-grams are split into two sets, referred to as set A and set B, and the similarity between the two sets is calculated using a defined similarity metric. At 502, assuming that a constraint indicating the number of word-grams to be removed from the initial input word-grams has been defined (e.g., by the word-gram setup component 116), the second step at 502 is to keep only the γ edges with the highest similarity values. Finally, at 503, similar word-grams are merged using a weighted average and concatenated into a new sequence.

[0076] Although word-gram merging reduces the number of word-grams and thus reduces the processing latency of the transformer model 122, the accuracy of the model may also decrease as the number of word-grams decreases. The amount of accuracy reduction may vary based on the input data sample content and the reasoning task involved, which may vary significantly between received queries 102. At this point, it is difficult to determine an appropriate merging ratio to be applied to balance the individual request inputs as well as the query load and computing resources of the computing system 100. Similarly, the disclosed word-gram adaptation technique may be used to evaluate how different numbers of prompt word-grams affect the output accuracy and throughput or latency of the transformer model, for example, as a result, it is found that the accuracy gain for different numbers of prompt word-grams varies for different tasks. It is understandable that as the number of prompt word-grams increases, the service throughput tends to decrease.

[0077] Therefore, to fully exploit the benefits of adding hint tokens and merging tokens, the disclosed token adaptation techniques (e.g., via the token setting component 116 and the performance analysis component 118) dynamically determine optimal token execution settings regarding the number of hint tokens to be added by the hint layer 210 and / or the number of tokens to be removed by the merging layer 226 for each query 102 that strikes a balance between improving or maintaining a desired accuracy and minimizing processing latency of a transformer model for an incoming query 102 that matches the request burden, the type of task, and service characteristics associated with the query 102, and the hardware resources of the computing system 100 (e.g., in terms of memory 142 constraints and processing unit 144 constraints). In other words, the word-gram adaptation technique employed by the computing system 100 can adapt the initial number of word-grams generated for the corresponding input data samples of the query through word-gram morphing based on word-gram hints and / or word-gram merging, wherein the word-gram hints and / or word-gram merging are customized to the corresponding characteristics of the query (which can vary), the fluctuating query load, and the resource constraints of the computing system 100. As described above, the characteristics of the query requests considered in this dynamic evaluation include, but are not limited to: the specific tasks requested to be performed by the transformer model 122 represented by each query 102, the specific input data content of each query 102 (e.g., regarding the amount and type of input data), the utility reward associated with the query (e.g., a measure corresponding to some benefit (such as monetary benefit, etc.) brought about by satisfying the corresponding request, which can vary for different queries), and the latency requirements or constraints associated with each query.

[0078] To facilitate this purpose, the computing system 100 employs an initial process to first understand and define how different word-gram adaptation settings involving word-gram hints and word-gram merging (where different numbers of word-grams are added and removed, such as for a request query corresponding to the request query 102) affect the accuracy of the inference results of the transformer model 122 and the latency of the transformer model for different defined task types. Figure 6 Described in more detail, this involves registering different task types and generating task registration data 132 (e.g., by task registration component 124), generating task analysis data 134 (e.g., by task analysis component 126), and generating prompt repository data 136 (e.g., by training component 128).

[0079] At this point, Figure 6 An exemplary process 600 for generating task registration data 132, task analysis data 134, and prompt repository data 136 is shown in accordance with one or more implementations of the disclosed subject matter. Figures 1 to 6In various implementations, process 600 corresponds to an offline process performed before transformer model 122 is served online by computing system 100 .

[0080] The process 600 begins at 602, where a task corresponding to a received task hint 104 is registered with the computing system 100. This involves receiving user input (e.g., from a model developer, etc.) through a corresponding task registration user interface provided by a task registration component 124. The received user input includes individual task hints 104 corresponding to different types of reasoning tasks that can be performed by the transformer model 122, and a unique task identifier (ID) is assigned to each task hint. The user input received at 602 when registering each task may also include task-specific parameters defined by the developer, including the number of tokens required or preferred for the task, delay constraints or requirements for the task, and utility values ​​associated with the task. In some implementations, the user input may also define one or more task hint tokens that may be used for task types that are sorted in a preferred order of use by the hint layer 210. All of this information is stored in the task registration data 132. In this regard, the task registration data 132 may include or correspond to an index or table that identifies a plurality of different task types. Each defined task type in the task registration data 132 includes a unique ID and defined parameters, including but not limited to: a required or preferred number of tokens, a delay value associated with the task, a utility value associated with the task, and one or more prompt tokens sorted by the prompt layer 210 in order of use.

[0081] At 604, the task analysis component 126 performs task-specific analysis to generate and store task-specific analysis data 134 for each registered task. This may involve determining, by the task analysis component 126, for each registered task in the task registration data 132, how word-gram merging and word-gram hinting using different numbers of removed word-grams and / or added hint word-grams, respectively, affect the inference output accuracy and processing latency of the transformer model 122 executed by the computing system 100. In various embodiments, this may be achieved by applying the transformer model 122 to training queries corresponding to the task type, while directing the transformer model 122 to use different word-gram settings corresponding to different combinations of adding different numbers of hint word-grams and / or removing different numbers of initial execution word-grams, and evaluating how different word-gram adaptation settings affect the inference result accuracy and latency. At this point, the task analysis data 134 may include information for each task indicating how different word-gram adaptation settings affect the inference accuracy and latency.

[0082] In some implementations of these embodiments, the word-gram adaptation setting may allow word-gram merging or word-gram hinting (but not both). With these implementations, the word-gram setting component 116 may indicate whether to remove word-grams or add hint word-grams and the corresponding word-gram amount for removal or addition based on the word-gram adaptation number (represented herein as gamma γ). The word-gram setting component 116 may define the word-gram adaptation number γ such that if γ is less than a threshold number such as zero (e.g., γ<0), this corresponds to reducing the number of word-grams; if γ is greater than the threshold number (e.g., γ>0), this indicates adding one or more hint word-grams, and wherein if γ is equal to the threshold number (e.g., γ=0), this indicates that no word-gram adjustment is performed (i.e., reasoning is performed using the initial number of execution word-grams generated from the lemma process). In other words, the value of the word-gram adaptation number γ may be a discrete number corresponding to the number of word-grams to be removed from the initial execution word-gram when the discrete number is negative, so that when the discrete number is positive, the discrete number corresponds to the number of hint word-grams to be added. In some implementations, the word-gram adaptation number γ can be a discrete value that can be selected from a predefined list of possible word-gram adaptation numbers that can be used. For example, in one or more implementations, the predefined list of possible word-gram adaptation numbers can include {-20, -15, -10, -5, 0, 2, 4, 8}. At this point, a word-gram adaptation number of γ=-20 corresponds to removing 20 initial execution word-grams, a word-gram adaptation number of γ=-15 corresponds to removing 15 initial execution word-grams, and so on. With these embodiments, the task analysis data 134 can include information for each task that indicates how different word-gram adaptation values ​​(i.e., γ values) affect the inference accuracy and processing latency of the transformer model 122 associated with processing batches of queries with different batch numbers.

[0083] With these embodiments, the task analysis data 134 may include information indicating how different word-gram adaptation numbers (i.e., γ values) affect the inference accuracy and processing latency of the transformer model 122 when processing request queries with varying characteristics (e.g., varying task types, varying latency constraints, input data sample content, etc.). In various embodiments, this may be achieved by applying the transformer model 122 to training request queries with varying characteristics, while directing the transformer model 122 to use different word-gram adaptation numbers, and evaluating how different word-gram numbers and query characteristics affect the inference accuracy and processing latency of the transformer model 122 executed by the computing device 102 (e.g., at the processing unit 144 and memory 142 capacity of the computing system 100).

[0084] Additionally, or optionally, the task analysis data 134 may include information indicating how different word-unit adaptation numbers (i.e., γ values) affect the inference accuracy and processing latency of the transformer model 122 when processing grouped batches of request queries with similar and varying characteristics (e.g., task type, latency constraints, input data sample content, etc.) and different batch sizes. In this regard, in one or more embodiments, the transformer model 122 may be configured to perform batch processing as opposed to processing a single request query at a time. In the context of a service transformer model, a "batch" refers to a group of queries that are processed together in a single forward pass of the transformer model 122. Batch processing is often used in machine learning including transformer models to optimize computational efficiency and speed during inference or training. The batch size corresponds to the number of queries 102 processed together. Batch processing allows the use of parallel processing on a GPU or other hardware accelerator, which is designed to process multiple calculations simultaneously. This increases throughput (the number of inputs processed per unit time) compared to processing each input individually. Larger batch sizes increase the memory usage of the computing system 100 because the model needs to store all intermediate calculations for each input in the batch. However, GPUs and TPUs are generally optimized to handle these parallel calculations efficiently. In this context, latency refers to the time it takes to process all queries included in the same batch from the beginning to the end. According to the batching of the transformer model, all request queries included in the same batch use the same word unit adaptation number γ.

[0085] To this end, in some embodiments, at runtime, the scheduling component 114 can be configured to distribute the queries 102 into different batches that include different groups or subsets of queries, respectively. This involves distributing queries with similar characteristics (e.g., similar task types, similar input data, similar latency requirements, and / or similar utilities) and similar arrival times together in the same batch. All batches are stored in a chronological order in the batch queue 138 and are executed in sequence accordingly. As described in reference Figure 7 As described in more detail, the scheduling component 114 can employ an adaptive batching mechanism to adjust the size of each batch as customized to produce a desired impact on latency and throughput.

[0086] At this point, referring again to process 600, in accordance with a batch processing implementation, at 604, the task analysis component 126 can determine how different numbers of word-gram adaptations (i.e., γ values) for the batched input query groups affect the accuracy and latency of the transformer model 122 (and / or the latency and throughput of the computing system 100 associated with serving the queries overall), and this information can be stored in the analysis data 134.

[0087] The process 600 also involves performing prompt learning at 606 to define task-specific prompt tokens stored in the prompt repository data 136. At this point, prompt tokens are trained offline by the training component 128 and stored in the prompt repository data 136. Token pairs are associated with a task ID and a prompt number, which serves as an index for the token pair. During training, the prompt repository data 136 is first randomly initialized, and each token pair is trained individually. At each training epoch in cascade with the initial input token, a token pair is obtained from the prompt repository data 136.

[0088] When applied to batches, if the batch size is n b , and the input word length is n i After the prompt, the word shape becomes n b ×(n i +γ). The concatenated tokens can be forwarded to the next module. During the online execution of the transformer model 122 inference on each batch of queries 102, the hint layer 210 model directly uses the well-trained hint parameters in the hint repository data 136 to add hint tokens corresponding to each task type included in the batch.

[0089] Figure 7 An exemplary elastic transformer service process 700 implemented by word-gram adaptation according to one or more implementations of the disclosed subject matter is illustrated. Process 700 corresponds to a run-time or online process that can be performed by computing device 100 after generating task registration data 132, task analysis data 134, and hint repository data 136. Figures 1 to 6 And refer to Figure 7 According to process 700, at 702, receiving component 110 receives query 102. It should be understood that in the envisioned usage scenario, the receiving component receives a fluctuating stream of queries 102 at varying inflow rates (e.g., starting from an arbitrary rate to one query per second (or larger time window) to hundreds or thousands of requests per second). At this point, the received queries 102 have varying arrival times and characteristics (e.g., varying task types, varying input data sample content, varying latency requirements, and varying utility values). In some embodiments, the queries 102 can be received using metadata, wherein the metadata indicates the corresponding characteristics of the queries 102. For example, each query can be received using metadata indicating its corresponding task identifier, utility value, and latency requirement. In other embodiments, the receiving component 110 can indicate these characteristics based on its task type (as indicated or inferred by the received task ID) and using task analysis data 134.

[0090] At 704, the scheduling component 114 allocates or groups the queries into chronologically ordered batches (received over time) based on the arrival time (or reception time) and similar characteristics of the queries, and stores them in the batch queue 138. With these embodiments, the execution component 120 sequentially executes the transformer model 122 on each batch in the batch queue (e.g., according to the chronological order).

[0091] Although batching has obvious benefits in terms of increasing throughput and reducing latency, one challenge is to design an adaptive batching strategy that effectively groups similar queries together. In various embodiments, to achieve this, at 704, the scheduling component 114 can be configured to allocate incoming queries into batches based on their similar arrival patterns and similar service level objectives (e.g., similar delay constraints, utility values, processing completion deadlines, etc.). For example, the allocation at 704 can include allocating the received queries 102 into different batches by grouping similar queries with similar arrival times together under a maximum batch size constraint and / or a time constraint defined based on arrival time and completion deadline. The scheduling component 114 can also allocate incoming queries to batches by grouping queries with one or more similar characteristics (e.g., the same or similar task type, similar arrival time, similar delay requirements, and / or similar utility values) in the same batch according to a defined similarity criterion or a defined similarity metric. In some implementations, the similar characteristics are selected from the group consisting of: a first characteristic related to arrival time, a second characteristic related to query task, a third characteristic related to utility value, and a fourth characteristic related to delay requirement.

[0092] In one or more embodiments, to facilitate this purpose, the scheduling component 114 may Figure 8 The process 1 shown is used to batch incoming queries 102. Figures 1 to 7 And refer to Figure 8 According to these embodiments, the symbol r is used to represent a query, where s r , l r d r and u r denote the arrival time, delay requirement, completion deadline and utility value of the request, respectively, so that d r =s r +l r The scheduling component 114 stores the grouped queries in the batch queue 138, which process 1 designates as B, and executes them in chronological order based on the earliest arrival time of the queries included in each batch as required. Process 1 designates the bth batch as B b According to process 1, the arrival time of batch b is s bcan be defined as the earliest arrival time s of the queries included in the batch r , that is, s b =min{sr}, r∈B b Similarly, the completion deadline of batch b is d b can be defined as the earliest completion deadline d among its requests r , that is, d b =min{d r}, r∈B b .

[0093] According to process 1, the scheduling component 114 assigns the query to one of the current batches in the batch queue B, or initializes / creates a new batch. The key idea of ​​process 1 is based on constructing batches with constraints on batch size, arrival time, utility, and deadline. More specifically, process 1 ensures that the waiting time of the first request in the batch is less than a threshold duration δ, the batch size is less than a predefined threshold size ∈, and the deadline difference between the batch and the query r is not greater than a deadline threshold η. Process 1 uses u b to denote the utility of the first arriving query in batch b and constrain the utility values ​​of subsequent incoming queries r to be close to u with a threshold μ b These constraints ensure that queries with similar arrival patterns and service level targets are processed together, which is beneficial for word-unit adaptation. If a batch that satisfies the constraints of the incoming query is found, the query is added to the batch b (lines 1 to 9 of process 1). Otherwise, the scheduling component 114 creates a new batch for the query and adds the new batch to batch queue B.

[0094] After one or more completed batches have been established in the batch queue 138, the next step is to assign word-meta adaptation settings to the respective batches in the batch queue 138. Although grouped based on similar characteristics, the resulting batches in the batch queue 138 may contain query requests corresponding to different tasks with varying utilities and latency requirements. At this point, continuing with the process 700, at 706, the word-meta setting component 116 further determines word-meta execution settings for the respective batches in the batch queue 138 characteristics (e.g., respective task types, utilities, latency requirements, completion deadlines, etc.) of the respective queries included in the respective batches. The scheduling component 114 may also determine word-meta execution settings for the respective batches in the batch queue 138 based on the query load in the batch queue (e.g., based on the number of batches, the corresponding sizes of the batches, and / or the respective latency constraints associated with the batches) and / or the overall load on the computing system 100 (e.g., also considering the influx rate of incoming queries).

[0095] In various embodiments, determining, by the word-gram setting component 116, word-gram settings for respective batches in the batch queue 138 at 706 involves, for each of the different batches in the batch queue 138, determining, based on one or more characteristics of the queries comprising the batch, a word-gram adaptation number γ for adjusting or maintaining the respective (e.g., generated by word-gramming) initial execution word-grams of the respective queries included in the batch. As described above, when applied to batch processing, the word-gram setting component 116 assigns the same word-gram adaptation number γ to all queries included in the same batch. In this regard, the one or more characteristics may include, but are not limited to, input data sample content of the respective queries, respective tasks of the queries, respective utility values ​​of the queries, and respective latency requirements of the queries.

[0096] In general, to facilitate this purpose, based on the corresponding characteristics of the queries in the batch and the task analysis data 134, the task analysis component 118 can estimate how the corresponding characteristics of the queries in the batch affect the accuracy of the inference results of the batch and the latency or inference time of the batch under different word-gram adaptation settings (e.g., considering only the use of word-gram hints, considering only word-gram merging, and / or a combination of word-gram hints and word-gram merging). Additionally, or alternatively, based on the corresponding characteristics of the queries in the batch and the task analysis data 134, the task analysis component 118 can estimate how the corresponding characteristics of the queries in the batch affect the accuracy of the inference results of the batch and the latency or inference time of the batch under different word-gram adaptation numbers γ. For example, a word-gram adaptation number of γ>0 can be used to indicate that the hint layer 210 will add a corresponding number of hint word-grams, and a word-gram adaptation number of γ<0 can be used to indicate that the hint layer merging layer 226 will merge the corresponding number of word-grams to reach the γ number of word-grams after the MLP layer 232.

[0097] Based on how different word-gram adaptation settings or word-gram adaptation numbers for a batch affect the inference result accuracy and / or processing latency of the batch, the word-gram setting component 116 can also select the optimal word-gram adaptation setting and / or word-gram adaptation number γ for the batch to strike a balance between increasing or maintaining the inference output accuracy of the batch and reducing latency. In this context, the word-gram setting component 116 can also consider how the word-gram adaptation number will affect the throughput of the transformer model 122 in association with processing all batches in the batch queue 138, thereby determining the word-gram adaptation number γ for the batch. In other words, the word-gram setting component 116 can customize the word-gram adaptation number γ for each batch in the batch queue 138 based on the query load of the batch queue, select a smaller word-gram adaptation number when the load is high to increase throughput, and select a larger word-gram number when the load is low to increase accuracy.

[0098] In one or more embodiments, the token setting component 116 can formulate the task of determining the optimal token adaptation settings for each batch and / or the optimal number of token adaptations for each batch as an optimization problem. In some implementations of these embodiments, the token setting component 116 can use a dynamic programming process to obtain a solution to the optimization problem (i.e., for token adaptation settings for each batch and / or for the number of runtime executions for each batch).

[0099] In some implementations of these embodiments, the token setting component 116 can define the token adaptation setting for batch b as γb, where γb < 0 indicates reducing the number of tokens, γb > 0 indicates adding one or more prompt tokens, and γb = 0 indicates reasoning using the initial number of execution tokens generated from the tokenization process. With these embodiments, the token adaptation number γ can be a discrete value that can be selected by the token execution setting 116 for a batch from a predefined list of possible token adaptation numbers (e.g., as defined in the task analysis data 134). For example, in the offline process 600, at 604, the possible values can include the same distribution of different γ values evaluated by the task analysis component 126 for each training request query and / or training batch. For example, in one or more implementations, the predefined list of possible token adaptation numbers can include {-20, -15, -10, -5, 0, 2, 4, 8}.

[0100] The optimization problem is defined in Equation (3) below, where the goal is to assign the token adaptation number γ to the corresponding batches to maximize the total utility of all queries in all batches in the batch queue B.

[0101]

[0102] Constraint (3a)

[0103]

[0104] Constraint (3b)

[0105] And

[0106] M < M GPU Constraint (3c)

[0107] At this point, Equation (3) is based on the following assumption: If the transformer model 122 successfully provides an accurate result for query r under the latency requirements for the query, the computing system 100 can be rewarded with a utility u weighted by the utility value associated with the query r, such as reward points, monetary rewards, or other forms of rewards. The symbol αr∈{0,1} can be used to indicate whether the transformer model 122 successfully serves the query r executed by the computing system 100. The required memory of the batch b and the available memory (e.g., of the memory 142) to the processing unit (e.g., the processing unit 144) are represented as M b and M GPU Constraint (3a) ensures that all requests can be completed within their respective completion deadlines, where t(q)r and t(p)r are the queuing time and processing time. Constraint (3b) ensures that batches in the batch queue (arranged according to their time order) are executed sequentially. Constraint (3c) imposes a memory limit, since larger batch sizes and number of hints can increase the memory requirements.

[0108] Formula (3) takes into account both query load and request characteristics. If the batch queue has a large number of queries, the word unit setting component 116 should select a smaller γ to reduce queuing and processing time and serve more requests. Conversely, the word unit setting component 116 can increase the value of γ to obtain more accurate inference results and obtain more utility.

[0109] In some implementations, Equation 3 can be considered a non-deterministic polynomial time difficult problem (i.e., an NP-hard problem) because it can be simplified to another NP-hard problem called the Weighted Interval Scheduling Problem (WISP). To this end, given a set of intervals with weights, the goal of the WISP is to select some intervals that can maximize the sum of the weights when the selected intervals are pairwise disjoint. Therefore, Equation (3) can be considered as a WISP. To this end, each batch can be considered as an interval with a weight equal to its utility, and the purpose is to efficiently process the batches in the queue so that the sum of the utilities is maximized. However, Equation (3) is more difficult than WISP because the word unit setting component 116 also needs to adjust the running time for picking intervals with different γ values.

[0110] Due to the NP-hardness of equation (3), in some implementations, the word-unit setting component 116 can use an efficient dynamic programming process to derive a solution. The dynamic programming process formulates equation (3) according to process 2. Fig. 9 Process 2 is shown in FIG. 1 . In some implementations where the computing system 100 initially begins servicing queries, the number of batches in the batch queue ready to execute may be less than a threshold number that makes process 2 suitable for determining the adaptive number of word units for the batches. In these scenarios, the word unit setting component 116 may be configured to employ a less complex process (referred to herein as process 3) to assign the adaptive number of word units. Fig.10 Process 3 is shown in FIG.

[0111] based on Figures 1 to 8 And refer to Fig. 9 and Fig.10 , process 2 and process 3 respectively set the batch queue B, the current time T, and the predefined available γ list L (γ) and the estimated incoming request rate q as input, and output an updated batch queue with an assigned adaptive number of tokens γ for each batch in the batch queue. In various embodiments, the performance analysis component 118 can calculate the incoming request rate q based on the sampled request rate for the previous time window before the current time T. Based on determining the adaptive number of tokens γ for each batch in the batch queue using process 2 and process 3 at 706 of process 700, the token setting component 116 begins by sorting the batches according to their required deadlines.

[0112] Referring to process 3, if the size of batch queue B is less than threshold β, or computing system 100 has just started accepting queries 102 and is therefore in the initial service phase, then word unit setting component 116 can be configured to determine the adaptive number of word units for each batch based on the query load of process 3. This is because the dynamic programming process works well when there are enough batches for long-term scheduling, which does not yet exist in the initial service phase. Process 3 allocates the number of word units γ by comparing the incoming request rate q and the throughput of different γ values. Word unit setting component 116 can also apply a projection function To map q to the appropriate γ value (line 1). At this point, the task analysis component 126 can analyze the throughput of different γ values ​​associated with performing task-specific analysis at 604 of process 600. Offline. The word-gram setting component 116 can then adjust the selection of γ according to the query characteristics based on the task analysis data 134. The performance analysis component 118 can predict the execution time of batch b based on the task analysis data 134 and the corresponding query tasks (and their corresponding waiting time constraints) of the queries included in the batch. If the estimated completion time exceeds the deadline, the word-gram setting component 116 can set the number of word-grams to the minimum value to meet the delay constraint (lines 3-5). If the average utility If the word-gram setting component 116 is greater than the threshold κ, the word-gram adaptation number γ can be set to the maximum value to prioritize the key queries. Finally, the performance analysis component 118 estimates the execution time and updates the current time T.

[0113] Referring to process 2, on the other hand, if the size of the batch queue 138 is greater than the threshold β, the word unit setting component 116 can be configured to use process 2 to determine the number of word units for each batch. Process 2 corresponds to an autonomous word unit adaptation algorithm. The key idea of ​​process 2 is to find the word units with γ by iterative traversal. b The maximum utility value of batch b.

[0114] Process 2 uses a size of (N B +1)×(N γ +1) to realize dynamic programming, where N B and N γ are the size of the batch queue and the number of available γ values, respectively. Specifically, dp records the accumulated utility, S records the previous γ selection plan, and C records the clock time after executing batch b with γ. The matrix J represents whether executing b with γ meets the deadline requirement. For each batch in the batch queue, the word unit setting component 116 uses the index l b Iteratively add from list L (γ) The γ value is assigned to batch b (lines 9-11). If batch b-1 cannot be used by l b -1 index, the word-unit setup component 116 continues to the next iteration of the loop (lines 12-13). b When the value of is 0, it indicates that batch b has not been executed, and the word-gram setting component 116 directly finds the larger utility value from batch b-1 and assigns it to batch b (lines 14-19).

[0115] When γ is executed on batch b b , the performance analysis component 118 first estimates the inference time and utility by using the analysis of the task analysis data 134 (line 22). If the inference time is less than the required deadline, the task setup component 116 calculates the total utility and sets the execution plan to 1 (lines 23-25). If the utility is greater than the previous value, the task setup component 116 updates the matrix. If for a task with γ b If there is no feasible execution plan for batch b, the task setting component 116 sets the dp value to -∞ and the clock time to +∞ (lines 30 to 32).

[0116] Once the task setting component 116 has calculated the utility values ​​and their corresponding choices, the task setting component 116 can derive a solution to process 2 by backtracking. In this regard, the task setting component 116 first determines the Nth process based on the highest dp value. B-thThe γ value of the batch. For each batch, the task setting component 116 obtains the index of γ according to the value of S[b+1,γ]. Finally, the task setting component returns the updated batch queue B, which has the word-unit adaptation number γ determined for each batch in the batch queue.

[0117] Based on process 2 and process 3, the performance analysis component 118 (or the task setting component 116) estimates the execution time and utility of batch b using the task analysis data 134. For example, as shown in FIG. Figure 6 As described in process 600 , before the transformer model 122 is served online, the task analysis component 126 analyzes the accuracy and sample-level (ie, specific task and specific input data content) inference latency of all tasks and stores them in the task analysis data 134 .

[0118] To estimate the inference time of the current batch during online service, the performance analysis component 118 (or task setting component 116) first counts the number of samples for each task, and then multiplies the number of samples by the corresponding analyzed inference time to obtain the execution time of the task. The performance analysis component 118 (or task setting component 116) then sums the calculation results of all tasks to obtain the predicted inference time of the batch. To calculate the total utility, the performance analysis component 118 (or task setting component 116) calculates the product of accuracy and the selected γ and the utility of each query in the batch. The performance analysis component 118 (or task setting component 116) then adds the product results of all queries to obtain the total utility of the batch. During analysis, the performance analysis component 118 (or task setting component 116) ensures that all running processes comply with equation (3c) and the memory constraints of the computing system 100.

[0119] Refer again Figure 7 and process 700, once the task setting component 116 has determined the corresponding word-gram execution settings at 706, the process 700 continues to 709. At this point, the word-gram execution settings correspond to the corresponding word-gram adaptation numbers γ for each batch according to the use of process 2 and / or process 3. In other embodiments, the word-gram execution settings may include or correspond to word-gram adaptation settings customized for each batch, which combine a combination of word-gram prompts and word-gram merging, and the corresponding number of word-grams added and removed by the corresponding process. At 709, the execution component 120 then applies the transformer model 122 to the batches in the batch queue in sequence, while directing the transformer model 122 to adopt the corresponding word-gram execution settings for each batch to generate an inference result for the query.

[0120] At this point, as referenced Figure 2As described, the transformer model 122 is configured to perform token hinting (e.g., the token hinting process 212) and / or token merging (e.g., the token merging process 228) in association with processing each batch according to the assigned token execution settings for each batch to generate inference results 106 for the corresponding request queries included in each batch. For example, in various embodiments, to facilitate this purpose, in association with providing the batch to the transformer model 122, the execution component 120 can instruct the transformer model about applicable token execution settings, and the transformer model 122 can be configured to apply them. At this point, at 709, in association with applying the transformer model 122 to the batch, the execution component 120 can forward the batch of input data samples for the batch to the transformer model 122, and include information identifying the gamma values ​​of the corresponding tasks for each data sample in the batch (e.g., via the corresponding task ID).

[0121] Based on the word-gram adaptation number γ being a positive number, the transformer model 122 can be configured to execute the word-gram hinting process 212 and add / concatenate the corresponding number of hint word-grams represented by γ to the initial execution word-gram for the input data sample. For example, a word-gram adaptation number of γ=2 indicates that the transformer model 122 (e.g., through the hinting layer 210 and the hinting process 212) adds 2 hint word-grams to the initial word-gram (e.g., generated during tokenization) for each input data sample of the batch, a word-gram adaptation number of γ=4 indicates that the transformer model 122 adds 4 hint word-grams to the initial word-gram for each input data sample of the batch, and so on. In association with adding hint word-grams, the hinting layer 210 is configured to select task-specific hint word-grams for each query included in the batch as provided in the hinting repository data 136. At this point, using the task ID for each data sample, the hinting layer 210 can extract one or more hint word-grams and / or hint parameters defined for each task ID from the hinting repository data 136. According to the word-gram hinting, if the batch size is n b And the input word length is n i , then after being prompted by the prompt layer 210, the word shape becomes n b ×(n i +γ). The added hint tokens can motivate multi-head attention to generate better (i.e., more accurate) results.

[0122] Based on the word-gram adaptation number γ being a negative number, the transformer model 122 can be configured to perform a word-gram merging process 228 through the merging layer 226 to reduce the number of initial executed word-grams by the corresponding number γ. For example, a word-gram adaptation number of γ=-20 indicates that the transformer model 122 (e.g., through the merging layer 226 and the merging process 228) removes 20 initial word-grams (e.g., generated during lemma morphing) for each input data sample of the batch, a word-gram adaptation number of γ=-15 indicates that the transformer model 122 removes 15 initial word-grams, and so on. According to the word-gram merging process 228, the transformer model 122 directly processes the input word-grams. Considering the word-gram similarity obtained from the multi-head attention and the defined merging rule 230 (e.g., corresponding to the merging rule 230 of the process 500, etc.), the merging layer 226 can change the word-gram shape from n to n. b ×n i Reduce to n b ×(n i -|γ|).

[0123] Fig.11 Another exemplary elastic transformer service process 1100 through word-gram adaptation according to one or more embodiments of the disclosed subject matter is illustrated. Process 1100 is similar to process 700, but is simplified to be customized for processing individual queries, as opposed to batch processing. At this point, it should be understood that the disclosed word-gram adaptation technology can be extended to improve inference accuracy and inference speed in other service scenarios, where individual queries are processed by a computing system (e.g., corresponding to computing system 100) with lightweight memory and processing hardware and service requirements.

[0124] At this point, according to the process 1100, at 1102, the computing system (e.g., via the receiving component 110) receives the query 102. At 1104, the scheduling component 114 can store and arrange the queries into a time-ordered arrangement based on the arrival time of the queries. At 1106, the word-gram setting component 116 can determine the word-gram execution settings for the queries in the query queue 1138 based on the query load in the query queue 1138 and the corresponding characteristics of the queries. For example, similar to the mechanism employed for batches, the word-gram setting component 116 can determine the word-gram adaptive number γ for each query in the query queue based on the corresponding latency requirements, corresponding task types, corresponding utilities, corresponding input data sample contents, corresponding arrival times, corresponding completion deadlines, etc. of the queries in a manner that optimizes the overall utility of the query queue while meeting or exceeding (e.g., meaning processing even faster than required) their corresponding latency constraints. At 1108 , the execution component 120 can apply the transformer model 122 to the queries in the query queue 1138 in sequence, while directing the transformer model 122 to adopt corresponding word-element execution settings for each query to generate the inference result 106 of the corresponding query.

[0125] Reference Figures 1 to 11 In various embodiments, the techniques provided by computing system 100, process 600, process 700, and / or process 100 can be summarized for language and multimodal tasks, various transformer models, different execution environments, and different hint learning mechanisms and token reduction processes. At this point, the disclosed techniques can be summarized as follows: Simplicity. (1) Select a pre-trained transformer model as the base model. (2) Study the hint learning method of the pre-trained model and train the hint pool. (3) Design a token reduction process and establish a unified transformer model that flexibly adds and removes tokens. (4) Analyze accuracy and inference latency for different γ values, batch sizes, and sequence lengths. (5) Based on the analysis results, determine a γ list for online adaptation. (6) Apply the analysis data to process 2 for adaptively selecting γ values ​​for corresponding batches and / or individual queries during the online service period.

[0126] Fig.12An example computer-implemented method 1200 for performing an elastic transformer service in accordance with one or more embodiments of the disclosed subject matter is illustrated. The method 1200 includes, at 1202, receiving, by a computing system (e.g., computing system 100) including at least one processor, a query (e.g., query 102) to be processed using a transformer model (e.g., transformer model 122), wherein processing of the query is associated with processing tokenized representations of respective input data samples of the query, the tokenized representations including corresponding (e.g., generated by tokenization) initial numbers of tokens for executing tokens. At 1204, the method 1200 includes determining, by the system (e.g., by token setting component 116 and performance profiler component 118), token adaptation settings suitable for processing the query using the transformer model based on one or more characteristics of the query (e.g., input data sample content, task type, latency constraints, utility, arrival time, etc.), wherein at least some of the token adaptation settings include adjusting at least some of the initial token numbers using the transformer model through at least one of a token hinting process or a token merging process. At 1206 , method 1200 includes applying, by the system (eg, via execution component 120 ), the query as input to the transformer model with the word-gram adaptation setting to generate an inference result (eg, inference result 106 ) corresponding to the query.

[0127] At this point, according to method 1200, adjusting at least some of the initial number of word units using the transformer model through a word unit hinting process results in an improved accuracy level of the inference results, and wherein adjusting at least some of the initial number of word units using the transformer model through a word unit merging process results in a reduced latency level of the transformer model associated with the processing of the query.

[0128] Fig.13Another example computer-implemented method 1300 for performing an elastic transformer service in accordance with one or more embodiments of the disclosed subject matter is illustrated. The method 1300 includes, at 1302, receiving, by a computing system (e.g., computing system 100) including at least one processor, a query for processing by a transformer model (e.g., transformer model 122) in association with tokenized representations of respective input data samples of the query, wherein the tokenized representations include respective initial numbers of execution tokens. At 1304, the method 1300 includes distributing, by the computing system (e.g., via scheduling component 114), the query into different batches that respectively include different subsets of the query. The method 1300 also includes a subprocess 1306 performed by the computing system 100 for each of the different batches. In this regard, the subprocess 1300 includes determining, by the system at 1308, an adaptive number of tokens for respective execution tokens of respective queries included in the batch based on one or more characteristics of the queries included in the batch (e.g., via token setting component 116). At 1310, the transformer model is directed by the system (e.g., by the execution component 120) to apply a word-gram adaptation number when applying the batch as input to the transformer model to adjust (e.g., increase by adding one or more hint words and / or decrease by removing one or more execution words) or maintain a corresponding initial word-gram number of corresponding execution words. At 1312, the method 1300 includes applying, by the system (e.g., by the execution component 120), the batch as input to the transformer model to generate an inference result for a query included in the batch, wherein the transformer model is configured to apply a word-gram adaptation number to adjust or maintain a corresponding initial word-gram number of each execution word-gram when applying the batch as input to the transformer model.

[0129] According to method 1300, based on the word-gram adaptive number being greater than a threshold number (e.g., zero), the transformer model configured according to the word-gram adaptive number adds one or more hint word-grams to each execution word-gram. Based on the word-gram adaptive number being less than the threshold number, the transformer model configured according to the word-gram adaptive number uses a word-gram merging process to reduce each initial word-gram number. Based on the word-gram adaptive number being equal to the threshold number, the transformer model configured according to the word-gram adaptive number maintains each initial word-gram number.

[0130] Experimental Implementation of OTAS

[0131] The OTAS prototype can be based on process 700 and Figure 7 to be implemented (eg, computing system 100), the process 700 and Figure 7Supports dynamically allocating the number of tokens for batches and running transformer models in a flexible manner. Results show that OTAS can improve the utility by at least 18.2% on both simulation and real production tracking on Azure compared to other state-of-the-art methods. The observed performance improvement is achieved because OTAS can identify the optimal balance between the overhead of token growth and the benefits of improved accuracy based on real-time query load and user requirements. Figures 14 to 21B The implementation prototype, experiments and results of OTAS are given.

[0132] I. Implementation

[0133] OTA implementation description. Four data structures and corresponding interfaces can be provided to implement the OTAS prototype. TransformerModel is a transformer model class that includes lemma prompts and lemma reduction modules. This model is loaded with pre-trained weights. TaskModel stores all parameters of a task, such as prompts and classification heads. ServeModel acts as a basic model at the front-end level. Its forward method accepts a batch of inputs, the corresponding input tasks, a parameter list of the task model, and a gamma value as input, and returns the inference result. The batch class is responsible for adding queries to a batch, providing analysis results, and returning a batch of queries for inference within the latency constraints.

[0134] Implementation tools. The OTAS prototype can be implemented based on PetS. Python can be used, for example, to process incoming queries and implement batch processing and token adaptation processes. For example, PyTorch can be used to define a neural network, including TransformerModel, TaskModel, and ServeModel. The transformer model can be built with the timm library, and two modules can be inserted to add and remove tokens at each layer. For example, the hint learning and token reduction processes can be implemented based on VPT and ToMe.

[0135] User Interfaces. The system enables users to make queries and register tasks using two interfaces. The Make Query interface processes queries, which include image samples and various attributes such as task ID, latency requirements, and utility. The queries can then be assigned to batches utilizing Process 1. The Register Task interface saves the task parameters in the Task Model list and the corresponding latency and utility values ​​in the Task Data list.

[0136] II. Experiment

[0137] Settings. The ViT-Base model can be used, which can be pre-trained on ImageNet 21K as the base model and includes 12 transformer layers. The number of attention heads is 12 and the feature size is 768. The patch size of the image is 16×16. Three datasets can be used, including CIFAR10, CIFAR100, and EuroSAT, and 1 / 5 of the training data is randomly selected as the analysis set. The γ selection list can be defined as {-20, -15, -10, -5, 0, 2, 4, 8}, and the γ selection list can be adjusted according to the query rate. The values ​​of δ, ∈, η, and μ in process 1 are set to 0.5s, 64, 0.5s, and 0.8, respectively. The value of β can be set to 5, and the initial stage can be defined as the first 2 seconds of the service. The value of κ in process 3 is 0.8, and the projection function exist Fig.14 The training data is defined in Table I. Offline training tips for the tasks. The training batch size, epochs and learning rate are set to 32, 50 and 0.002. OTAS can be evaluated on a machine such as NVIDIA Geforce RTX 4080 (12th Generation Intel Core(TM) i9-12900K CPU).

[0138] Baselines. OTAS can be compared with PetS and INFaaS. PetS is a unified framework for serving transformers in a parameter-efficient way and optimizing task-specific and task-shared operators. PetS keeps tokens unchanged and performs inference with a shared base model and task-specific heads. INFaaS is a model adaptation method that selects an appropriate model based on the query load. The list of candidate models is set to ViT-Small, Vit-Base, and ViT-Large. OTAS can also be compared with ToMe and VPT using a fixed number of merges or hints.

[0139] Workloads. The process can be evaluated using both synthetic query traces and real-world production traces. For synthetic workloads, query traces with fluctuating loads are generated. Arrival times for queries are randomly generated according to a Poisson distribution. Fig.15 Table II shown in Figure 1 provides the six different query types used and the delay constraints and utility values ​​for each query type. For each query, a query type is randomly selected from Table II. Based on experiments within a 30-minute service period, when more than 63k queries are processed, Fig.16A The number of queries per second during the first 200 seconds is presented. The query rate varies between 200 requests per second (200Req / s) and 700Req / s.

[0140] For real-world workloads, you can use publicly available traces (MAF) collected by Microsoft from Azure Functions in 2021. For example, you can use 120 hours of traces to experiment. Requests collected in two-minute intervals can be aggregated into one-second intervals to create a challenging trace. Fig. 16B The number of queries per second in the first 1000 seconds is presented in Figure . During more than 60% of the service period, the query rate remained below 300Req / s. There were still some cases where the number of requests per second exceeded 600Req / s.

[0141] III. Results

[0142] Overall utility. If the system can return accurate results for a query under latency constraints, the utility of the query can be rewarded. Fig.17A The cumulative utilities of the three system designs on the synthetic dataset are shown in Fig. 2. OTAS obtains about 1.46×105 utilities and leads to 18.2% and 72.5% utility improvements. INFaaS performs the worst because it has long I / O latency to switch models. Fig. 17B The overall utility of the MAF dataset is shown in . OTAS can improve the utility by up to 90.1%. Fig.18A A comparison of the effectiveness of different methods on a synthetic dataset with a fixed number of tokens is shown in . Fig.18B The effectiveness comparison of different methods on the MAF dataset with a fixed number of tokens is shown in . OTAS outperforms both ToMe and VPT because it can adjust the token strategy according to the query load.

[0143] Batch accuracy. Fig.19A CDF plots of accuracy for serving batches using five methods on a synthetic dataset of , and higher accuracy values ​​generally indicate a more efficient process. The VPT method with the number of hints of 2 achieves the highest accuracy due to the incorporation of well-trained hint tokens. The ToMe method exhibits relatively low accuracy due to the reduction in tokens. OTAS can dynamically select an appropriate execution scheme to achieve a balance between accuracy and latency. The average accuracy of the method is greater than 90%, indicating that our method is able to provide accurate results for served queries.

[0144] like Fig.19B As shown in Figure 2, the batch accuracy on the MAF dataset is similar to that observed on the synthetic dataset. The accuracy curve shows a sudden increase when approaching 1, which is mainly due to the large number of batches with a perfect accuracy score of 1.

[0145] Choice of γ. OTAS can vary the number of tokens γ based on the incoming load and query characteristics. Fig. 20AThe γ selection ratio of OTAS on the synthetic dataset is presented. Fig. 20B The MAF dataset is presented. On the synthetic traces, OTAS chooses γ=8 at most, given the smooth request load during most of the service periods. Another major choice is γ=-15, as it can reduce the inference time while keeping the accuracy almost unchanged. On the MAF traces, more batches are executed with a hint number of 8 due to the light query load. During the busy period, OTAS chooses a γ value of -15 to serve more queries.

[0146] Execution type of the query. Queries have different processing results, which can be classified into the following categories. Type 1 - obtain accurate results and meet latency constraints; Type 2 - obtain incorrect results while still meeting latency constraints; Type 3 - obtain inference results while failing to meet latency deadlines; and Type 4 - queries that fail to meet latency constraints and are evicted before actual execution.

[0147] The execution ratio of different query types on the synthetic dataset is Fig.21A It can be seen that OTAS can successfully serve 85.54% of the queries (type 1) and all queries can meet the latency requirements. On the other hand, ToMe can serve fewer queries because it has low prediction accuracy. VPT and INFaaS have longer inference times, which leads to higher eviction ratios. Therefore, our approach can achieve an excellent trade-off between accuracy and latency, which leads to a higher service rate.

[0148] Fig.21B The execution ratios of different query types on the MAF dataset are presented. Because there are some high bursty loads in the MAF dataset, the ratio of evicted queries (type 4) increases due to limited computing resources. Compared with other methods, OTAS serves the largest number of requests with a success rate of 75.58%. For the ToMe method, 16.85% of the requests are mispredicted, which still consumes computing resources. The success rate of VPT is only 64.14%, which is due to the high inference latency caused by word-unit hints. As a result, 32.16% of the requests cannot meet the specified deadline and are evicted. Our method is more flexible in dealing with bursty query loads.

[0149] Example operating environment

[0150] One or more embodiments may be systems, methods, and / or computer program products at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or medium) having computer-readable program instructions thereon for causing a processor to perform various aspects of the present disclosure. Fig. 22and Fig.23 , provides a link to this article Figures 1 to 13 A detailed description of additional context for one or more embodiments described.

[0151] In order to provide additional context for the various embodiments described herein, Fig. 22 The following discussion is intended to provide a brief, general description of a suitable computing environment 2200 in which the various embodiments described herein may be implemented. Although the embodiments are described above in the general context of computer-executable instructions that may be executed on one or more computers, those skilled in the art will recognize that the embodiments may also be implemented in conjunction with other program modules and / or as a combination of hardware and software.

[0152] Typically, program modules include routines, programs, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In addition, those skilled in the art will appreciate that the method can be practiced with other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, IoT devices, distributed computing systems, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronic devices, etc., each of which can be operably connected to one or more associated devices.

[0153] The embodiments described herein may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0154] Computing devices typically include various media, which may include computer-readable storage media, machine-readable storage media, and / or communication media, where the two terms used herein as described below are different from each other. Computer-readable storage media or machine-readable storage media can be any available storage media that can be accessed by a computer, and include both volatile and non-volatile media, removable and non-removable media. As an example and not limitation, computer-readable storage media or machine-readable storage media can be implemented in conjunction with any method or technology for storing information such as computer-readable or machine-readable instructions, program modules, structured data, or unstructured data.

[0155] Computer-readable storage media may include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD), Blu-ray disk (BD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, solid-state drives or other solid-state storage devices, or other tangible and / or non-transitory media that can be used to store the desired information. In this regard, the terms "tangible" or "non-transitory" as applied herein to storage, memory, or computer-readable media should be understood as excluding only the propagation of transient signals themselves as a modifier, without disclaiming the right to all standard storage, memory, or computer-readable media that propagate more than just transient signals themselves.

[0156] Computer-readable storage media may be accessed (eg, through access requests, queries, or other data retrieval protocols) by one or more local or remote computing devices for various operations regarding the information stored on the media.

[0157] Communication media typically embodies computer readable instructions, data structures, program modules, or other structured or unstructured data in a data signal (such as a modulated data signal, such as a carrier wave or other transport mechanism), and includes any information transfer or transmission media. The term "modulated data signal" or signal refers to a signal whose characteristics are set or changed in such a manner as to encode information in one or more signals. By way of example and not limitation, communication media include wired media such as a wired network or direct wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.

[0158] Refer again Fig. 22 , an example environment 2200 for implementing various embodiments of various aspects described herein includes a computer 2202, which includes a processing unit 2204, a system memory 2206, and a system bus 2208. The system bus 2208 couples system components including, but not limited to, the system memory 2206 to the processing unit 2204. The processing unit 2204 can be any of a variety of commercially available processors and can include a cache memory. Dual microprocessors and other multi-processor architectures can also be used as the processing unit 2204.

[0159] The system bus 2208 may be any of several types of bus structures that may further interconnect to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. The system memory 2206 includes ROM 2210 and RAM 2212. A basic input / output system (BIOS) containing the basic routines that help transfer information between elements within the computer 2202, such as during startup, may be stored in a nonvolatile memory such as ROM, an erasable programmable read-only memory (EPROM), an EEPROM, or the like. The RAM 2212 may also include high-speed RAM, such as static RAM for caching data.

[0160] The computer 2202 also includes an internal hard disk drive (HDD) 2214 (e.g., EIDE, SATA), one or more external storage devices 2216 (e.g., a magnetic floppy disk drive (FDD) 2216, a memory stick or flash drive reader, a memory card reader, etc.), and an optical drive 2220 (e.g., which can read or write from a CD-ROM disk, DVD, BD, etc.). Although the internal HDD 2214 is shown as being located within the computer 2202, the internal HDD 2214 can also be configured for external use in a suitable chassis (not shown). Additionally, although not shown in the environment 2200, a solid-state drive (SSD) other than the HDD 2214 can also be used, or a solid-state drive (SSD) can be used instead of the HDD 2214. The HDD 2214, the external storage device 2216, and the optical drive 2222 can be connected to the system bus 2208 via a HDD interface 2224, an external storage interface 2226, and an optical drive interface 2228, respectively. The interface 2224 for external drive implementations may include at least one or both of Universal Serial Bus (USB) and Institute of Electrical and Electronics Engineers (IEEE) 10224 interface technologies. Other external drive connection technologies are also contemplated by the embodiments described herein.

[0161] The drives and their associated computer-readable storage media provide non-volatile storage of data, data structures, computer-executable instructions, etc. For the computer 2202, the drives and storage media accommodate the storage of any data in an appropriate digital format. Although the above description of computer-readable storage media relates to corresponding types of storage devices, it should be understood by those skilled in the art that other types of computer-readable storage media (whether currently existing or developed in the future) may also be used in the example operating environment, and further, any such storage media may contain computer-executable instructions for performing the methods described herein.

[0162] A number of program modules may be stored in the drives and RAM 2212, including an operating system 2230, one or more application programs 2232, other program modules 2234, and program data 2236. All or portions of the operating system, applications, modules, and / or data may also be cached in RAM 2212. The systems and methods described herein may be implemented using various commercially available operating systems or combinations of operating systems.

[0163] Computer 2202 may optionally include emulation technology. For example, a hypervisor (not shown) or other medium may emulate the hardware environment of operating system 2230, and the emulated hardware may optionally be different from the hardware of operating system 2230. Fig. 22 The hardware shown. In such an embodiment, the operating system 2230 may include one of a plurality of virtual machines (VMs) residing at the computer 2202. In addition, the operating system 2230 may provide a runtime environment, such as a Java runtime environment or a .NET framework, for the application 2232. The runtime environment is a consistent execution environment that allows the application 2232 to run on any operating system that includes a runtime environment. Similarly, the operating system 2230 may support containers, and the application 2232 may be in the form of containers, which are lightweight, independent, executable software packages that include, for example, code, runtime, system tools, system libraries, and settings for the application.

[0164] In addition, the computer 2202 may include a security module, such as a trusted processing module (TPM). For example, using a TPM, a boot component hashes the next boot component in time and waits for the result to match a security value before loading the next boot component. This process can occur at any layer in the code execution stack of the computer 2202, for example, at the application execution level or the operating system (OS) kernel level, thereby achieving security at any code execution level.

[0165] A user may enter commands and information into the computer 2202 through one or more wired / wireless input devices, such as a keyboard 2238, a touch screen 2240, and a pointing device such as a mouse 2242. Other input devices (not shown) may include a microphone, an infrared (IR) remote control, a radio frequency (RF) remote control, or other remote control, a joystick, a virtual reality controller and / or a virtual reality headset, a game pad, a stylus, an image input device such as a camera, a gesture sensor input device, a visual motion sensor input device, an emotion or facial detection device, a biometric input device such as a fingerprint or iris scanner, and the like. These and other input devices are typically connected to the processing unit 2204 through an input device interface 2244, which may be coupled to the system bus 2208, but may be connected through other interfaces such as a parallel port, an IEEE 10224 serial port, a game port, a USB port, an IR port, a fingerprint scanner, or a fingerprint scanner. interface, etc.) connection.

[0166] A monitor 2246 or other type of display device may also be connected to the system bus 2208 via an interface, such as a video adapter 2248. In addition to the monitor 2246, computers typically include other peripheral output devices (not shown), such as speakers, printers, and the like.

[0167] The computer 2202 can operate in a networked environment using logical connections to one or more remote computers (e.g., remote computer 2250) via wired and / or wireless communications. The remote computer 2250 can be a workstation, server computer, router, personal computer, portable computer, microprocessor-based entertainment device, peer device, or other public network node, and typically includes many or all of the elements described relative to the computer 2202, although only a memory / storage device 2252 is shown for simplicity. The logical connections described include wired / wireless connections to a local area network (LAN) 2254 and / or a larger network (e.g., a wide area network (WAN) 2256). Such LAN and WAN network environments are common in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a global communication network, such as the Internet.

[0168] When used in a LAN networking environment, the computer 2202 can be connected to the local network 2254 through a wired and / or wireless communication network interface or adapter 2258. The adapter 2258 can facilitate wired or wireless communication to the LAN 2254, which can also include a wireless access point (AP) disposed thereon for communicating with the adapter 2258 in a wireless mode.

[0169] When used in a WAN networking environment, the computer 2202 may include a modem 2260, or may be connected to a communication server on the WAN 2256 via other means for establishing communications over the WAN 2256 (e.g., through the Internet). The modem 2260 may be internal or external and may be a wired or wireless device, and the modem 2260 may be connected to the system bus 2208 via the input device interface 2244. In a networked environment, program modules described relative to the computer 2202, or portions thereof, may be stored in the remote memory / storage device 2252. It should be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

[0170] When used in a LAN or WAN networking environment, the computer 2202 can access a cloud storage system or other network-based storage system in addition to or in place of the external storage device 2216 described above. Typically, the connection between the computer 2202 and the cloud storage system can be established through the LAN 2254 or WAN 2256 (e.g., through an adapter 2258 or a modem 2260, respectively). When the computer 2202 is connected to the associated cloud storage system, the external storage interface 2226 can manage the storage provided by the cloud storage system with the help of the adapter 2258 and / or the modem 2260, just as it would be other types of external storage. For example, the external storage interface 2226 can be configured to provide access to cloud storage sources as if these sources were physically connected to the computer 2202.

[0171] The computer 2202 is operable to communicate with any wireless device or entity that is operable to be arranged in wireless communication, such as printers, scanners, desktop and / or portable computers, portable data assistants, communication satellites, any device or location associated with a wirelessly detectable tag (e.g., information kiosks, news stations, store shelves, etc.), and telephones. This may include Wireless Fidelity (Wi-Fi) and Wireless technology. Thus, the communication can be a predetermined structure like a conventional network, or simply an ad hoc communication between at least two devices.

[0172] The above description includes non-limiting examples of various embodiments. Of course, for the purpose of describing the disclosed subject matter, it is not possible to describe every conceivable combination of components or methods, and those skilled in the art will recognize that further combinations and permutations of the various embodiments are possible. The disclosed subject matter is intended to include all such changes, modifications, and variations that fall within the spirit and scope of the appended claims.

[0173] Now refer to Fig.23Details of one or more elements shown depict illustrative cloud computing environment 2300 . Fig.23 23 is a schematic block diagram of a computing environment 2300 with which the disclosed subject matter can interact. System 2300 includes one or more remote components 2310. Remote components 2310 can be hardware and / or software (e.g., threads, processes, computing devices). In some embodiments, remote components 2310 can be distributed computer systems that are connected to local autoscaling components and / or programs that use resources of the distributed computer systems via communication framework 2340. Communication framework 2340 can include wired network devices, wireless network devices, mobile devices, wearable devices, radio access network devices, gateway devices, femtocell devices, servers, etc.

[0174] The system 2300 also includes one or more local components 2320. The local component 2320 can be hardware and / or software (e.g., thread, process, computing device). In some embodiments, the local component 2320 can include an automatic scaling component and / or a program that communicates / uses remote resources 2310 and 2320, etc., which is connected to a remotely located distributed computing system via a communication framework 2340.

[0175] One possible communication between the remote component 2310 and the local component 2320 may be in the form of data packets suitable for sending between two or more computer processes. Another possible communication between the remote component 2310 and the local component 2320 may be in the form of circuit-switched data suitable for sending between two or more computer processes in a radio time slot. The system 2300 includes a communication framework 2340, which can be adopted to facilitate communication between the remote component 2310 and the local component 2320, and can include an air interface, such as the Uu interface of a UMTS network, via a long-term evolution (LTE) network, etc. The remote component 2310 can be operably connected to one or more remote data storages 2350, such as a hard drive, a solid-state drive, a SIM card, a device memory, etc., which can be used to store information on the remote component 2310 side of the communication framework 2340. Similarly, the local component 2320 is operably connected to one or more local data storages 2330, which can be adopted to store information on the local component 2320 side of the communication framework 2340.

[0176] With respect to the various functions performed by the above-described components, devices, circuits, systems, etc., unless otherwise specified, terms used to describe such components (including references to "means") are also intended to include any structure (e.g., functional equivalent) that performs the designated functions of the described components, even if not structurally equivalent to the disclosed structures. In addition, while specific features of the disclosed subject matter may be disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desirable and advantageous for any given or specific application.

[0177] The terms "exemplary" and / or "illustrative" as used herein are intended to mean used as examples, instances, or illustrations. For the avoidance of doubt, the subject matter disclosed herein is not limited to these examples. Additionally, any aspect or design described herein as "exemplary" and / or "illustrative" is not necessarily to be construed as being preferred or advantageous over other aspects or designs, nor is it meant to exclude equivalent structures and techniques known to those skilled in the art. In addition, where the terms "include," "have," "comprises," and other similar words are used in the detailed description or claims, these terms are intended to include a manner similar to the term "includes" of an open transitional term, without excluding any additional or other elements.

[0178] As used herein, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." For example, the phrase "A or B" is intended to include instances of A, B, and both A and B. Additionally, the articles "a" and "an" used in this disclosure and the appended claims should generally be construed to mean "one or more" unless otherwise specified or clear from context to be directed to a singular form.

[0179] The term "set" as used herein excludes an empty set, i.e., a set with no elements therein. Thus, a "set" in the present disclosure includes one or more elements or entities. Likewise, the term "group" as used herein refers to a set of one or more entities.

[0180] The terms "first", "second", "third", etc. used in the claims are for clarity only and do not refer to or imply any temporal order unless the context indicates otherwise. For example, "a first determination", "a second determination", and "a third determination" do not mean or imply that the first determination is made before the second determination, or vice versa, etc.

[0181] As used in the present disclosure, in some embodiments, the terms "component", "system", etc. are intended to mean or include a computer-related entity or an entity related to an operating device having one or more specified functions, where the entity can be hardware, a combination of hardware and software, software, or software in execution. As an example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable program, an execution thread, a computer-executable instruction, a program, and / or a computer. As an example and not limitation, an application running on a server and a server can both be a component.

[0182] One or more components may reside in a process and / or a thread of execution, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may communicate via local and / or remote processes, such as according to signals with one or more data packets (e.g., data from a component interacting with another component in a local system, a distributed system, and / or data from another component interacting with other systems via a signal across a network (such as the Internet)). As another example, a component may be a device having a specific function provided by a mechanical component operated by an electrical or electronic circuit, which is operated by a firmware application executed by a software application or a processor, wherein the processor may be inside or outside the device, and execute at least a portion of the software or firmware application. As another example, a component may be a device providing a specific function by an electronic component without a mechanical component, and the electronic component may include a processor therein to execute software or firmware that at least partially assigns the function of the electronic component. Although various components have been described as separate components, it will be appreciated that multiple components may be implemented as a single component, or a single component may be implemented as multiple components without departing from the exemplary embodiment.

[0183] As used herein, the term "facilitates" "facilitates" one or more actions or operations in the context of a system, device, or component, in terms of the nature of a complex computing environment in which multiple components and / or multiple devices may be involved in certain computing operations. Non-limiting examples of actions that may or may not involve multiple components and / or multiple devices include sending or receiving data, establishing a connection between devices, determining intermediate results to obtain a result, and the like. In this regard, a computing device or component may facilitate an operation by playing any portion when the operation is completed. Therefore, when the operation of a component is described herein, it should be understood that, where the component helps describe the operation, the operation may optionally be completed through the cooperation of one or more other computing devices or components, such as, but not limited to, sensors, antennas, audio and / or video output devices, other devices, and the like.

[0184] In addition, various embodiments can be implemented as methods, devices or products to produce software, firmware, hardware or any combination thereof using standard programming and / or engineering techniques to control a computer to implement the disclosed subject matter. The term "manufacture" used herein is intended to include a computer program that can be accessed from any computer-readable (or machine-readable) device or computer-readable (or machine-readable) storage / communication medium. For example, a computer-readable storage medium may include, but is not limited to, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic strip), an optical disk (e.g., an optical disk (CD), a digital versatile disk (DVD)), a smart card, and a flash memory device (e.g., a card, a stick, a key drive). Of course, those skilled in the art will recognize that many modifications may be made to this configuration without departing from the scope or spirit of the various embodiments.

[0185] In addition, terms such as "mobile device equipment", "mobile station", "mobile", "subscriber station", "access terminal", "terminal", "handset", "communication equipment", "mobile device" (and / or terms representing similar terms) may refer to wireless devices used by subscribers or mobile devices of wireless communication services to receive or transmit data, control, voice, video, sound, games, or substantially any data stream or signaling stream. The above terms are used interchangeably herein and with reference to the relevant drawings. Similarly, the terms "access point (AP)", "base station (BS)", "BS transceiver", "BS equipment", "cell site", "cell site equipment", "gNodeB (gNB)", "evolved Node B (eNode B, eNB)", "home Node B (HNB)", etc. refer to wireless network components or devices that send and / or receive data, control, voice, video, sound, games, or substantially any data stream or signaling stream from one or more subscriber stations. Data and signaling streams may be packetized or frame-based streams.

[0186] Furthermore, the terms "device," "communication device," "mobile device," "subscriber," "client entity," "consumer," "client entity," "entity," and the like are used interchangeably throughout the text unless the context warrants a specific distinction between the terms. It should be understood that such terms may refer to a human entity or an automated component supported by artificial intelligence (e.g., the ability to reason based on complex mathematical formulas) that may provide simulated vision, voice recognition, and the like.

[0187] It should be noted that although various aspects and embodiments are described herein in the context of 5G or other next generation networks, the disclosed aspects are not limited to 5G implementations and may be applied to other network next generation implementations, such as the sixth generation (6G) or other wireless systems. In this regard, aspects or features of the disclosed embodiments may be utilized in substantially any wireless communication technology. Such wireless communication technologies may include Universal Mobile Telecommunications System (UMTS), Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Wideband CDMA (WCMDA), CDMA2000, Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Multi-Carrier CDMA (MC-CDMA), Single Carrier CDMA (SC-CDMA), Single Carrier FDMA (SC-FDMA), Orthogonal Frequency Division Multiplexing (OFDM), Discrete Fourier Transform Spread OFDM (DFT-spread OFDM), Filter Bank Based Multi-Carrier (FBMC), Zero Tail DFT Spread OFDM (ZT-s-OFDM), Generalized Frequency Division Multiplexing (GFDM), Fixed Mobile Convergence (FMC), Universal Fixed Mobile Convergence (UFMC), Unique Word OFDM (UW-OFDM), Unique Word DFT Spread OFDM (UW-OFDM), DFT-Spread-OFDM), cyclic prefix OFDM (CP-OFDM), resource block filtered OFDM, Wireless Fidelity (Wi-Fi), Worldwide Interoperability for Microwave Access (WiMAX), Wireless Local Area Network (WLAN), General Packet Radio Service (GPRS), Enhanced GPRS, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), 5G, 3rd Generation Partnership Project 2 (3GPP2), Ultra Mobile Broadband (UMB), High Speed ​​Packet Access (HSPA), Evolved High Speed ​​Packet Access (HSPA+), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Zigbee, or another Institute of Electrical and Electronics Engineers (IEEE) 802.12 technology.

[0188] It should be understood that when an element is referred to as being "coupled" to another element, it may describe one or more different types of coupling, including but not limited to a chemical coupling, a communicative coupling, an electrical coupling, an electromagnetic coupling, an operational coupling, an optical coupling, a physical coupling, a thermal coupling, and / or another type of coupling. Likewise, it should be understood that when an element is referred to as being "connected" to another element, it may describe one or more different types of coupling, including but not limited to an electrical coupling, an electromagnetic coupling, an operational coupling, an optical coupling, a physical coupling, a thermal coupling, and / or another type of coupling.

[0189] The description of the illustrated embodiments of the present disclosure provided herein (including what is described in the abstract) is not intended to be exhaustive or to limit the disclosed embodiments to the precise form disclosed. Although specific embodiments and implementations are described herein for illustrative purposes, various modifications may be considered within the scope of such embodiments and implementations as will be appreciated by those skilled in the art. In this regard, although the subject matter has been described herein in conjunction with various implementations and corresponding drawings, it will be understood that other similar implementations may be used, or the described implementations may be modified and added, where applicable, to perform the same, similar, alternative or alternative functions of the disclosed subject matter without departing from the present disclosure. Therefore, the disclosed subject matter should not be limited to any single implementation described herein, but should be interpreted in breadth and scope according to the appended claims.

Claims

1. A method comprising: Receiving, by a computing system including at least one processor, a query to be processed using a transformer model, wherein processing of the query is associated with processing tokenized representations of respective input data samples of the query, the tokenized representations including respective initial token numbers for executing tokens; determining, by the computer system, word-gram adaptation settings suitable for processing the query using the transformer model based on one or more characteristics of the query, wherein at least some of the word-gram adaptation settings include adjusting at least some initial word-gram numbers using the transformer model through at least one of a word-gram hinting process and a word-gram merging process; and The query is applied by the computer system as input to the transformer model with the word-gram adaptation setting to generate an inference result corresponding to the query.

2. The method according to claim 1, wherein: The one or more characteristics are selected from the group consisting of: a first characteristic applicable to input data sample content, a second characteristic applicable to query tasks, a third characteristic applicable to utility values, and a fourth characteristic applicable to delay requirements.

3. The method according to claim 3, wherein: Based on different characteristics of the queries, the word-gram adaptation setting varies between the queries.

4. The method according to any one of claims 1 to 3, wherein: Determining the word-unit adaptation setting includes: Based on the one or more characteristics, changes to at least some of the respective numbers of initial word-grams for use by the transformer model are determined, wherein the changes are associated with processing the query using the transformer model.

5. The method according to claim 1, wherein: The one or more characteristics include: various task types of the query, wherein determining the word-unit adaptation setting includes: The word-gram adaptation setting is determined based on task analysis data for the respective task types, wherein the task analysis data indicates how different adjustments to the respective initial word-gram numbers affect the accuracy of the inference results associated with the processing of the query and the latency level of the transformer model.

6. The method according to claim 5, wherein: The different adjustments include: Predefined prompt tokens are added to the execution tokens customized for the respective task types.

7. The method according to claim 1, wherein: Determining the word-unit adaptation setting includes: The word-gram adaptation setting is determined based on a current query load of the query.

8. The method according to claim 1, wherein: The use of the converter model includes: An encoder is employed, the encoder being configured to perform at least one of the word-gram prompting process and the word-gram merging process.

9. A method comprising: receiving, by a computing system including at least one processor, a query for processing by a transformer model, wherein processing of the query is associated with processing tokenized representations of respective input data samples of the query, the tokenized representations including respective initial token numbers for executing tokens; distributing, by the computing system, the queries into different batches respectively including different subsets of the queries; For each of the different batches: Determining, by the computer system, an adaptive number of word-grams for corresponding execution word-grams of each query included in the batch based on one or more characteristics of the queries included in the batch; Instructing the transformer model, by the computer system, to apply the adaptive number of word-grams when applying the batch as input to the transformer model to adjust or maintain the number of word-grams of the corresponding execution word-grams; and The batch is applied as input to the transformer model by the computer system to generate inference results for the queries included in the batch, wherein the transformer model is configured to apply the adaptive number of word-grams when applying the batch as input to the transformer model.

10. The method according to claim 9, wherein: In response to the word-gram adaptation number being greater than a threshold number, the transformer model configured according to the word-gram adaptation number adds one or more hint word-grams to the corresponding execution word-gram, In response to the word-gram adaptation number being less than the threshold number, the transformer model configured according to the word-gram adaptation number uses a word-gram merging process to reduce the corresponding initial word-gram number, or In response to the word-gram adaptation number being equal to the threshold number, the transformer model configured according to the word-gram adaptation number maintains the corresponding initial word-gram number.

11. The method according to claim 10, wherein: Based on adding the one or more hint tokens, the accuracy of the inference result increases for the batch, and wherein based on reducing the initial token number, a latency level of the transformer model decreases in association with the processing of the batch.

12. The method according to claim 9, wherein: The allocation includes: The queries are assigned to the different batches by grouping similar queries with similar characteristics in the same batch according to a defined similarity criterion or a defined similarity measure, wherein the similar characteristics are selected from the group consisting of: a first characteristic related to arrival time, a second characteristic related to query task, a third characteristic related to utility value, and a fourth characteristic related to delay requirement.

13. The method according to claim 12, further comprising: Determining, by the computer system, an execution order for the different batches based on arrival times of different queries included in the different batches; storing, by the computer system, the different batches in a batch queue; as well as The transformer model is sequentially applied to the different batches by the computer system according to the execution order.

14. The method according to claim 13, wherein: Determining the adaptive number of word units includes: Based on the query load of the batch queue, the adaptive number of word-grams of corresponding execution word-grams of each query included in the batch is determined.

15. The method according to claim 9, wherein: The one or more characteristics are selected from the group consisting of: a first characteristic applicable to input data sample content, a second characteristic applicable to query tasks, a third characteristic applicable to utility values, and a fourth characteristic applicable to delay requirements.

16. The method according to claim 9, wherein: The one or more characteristics include: a task type of each query included in the batch, wherein determining the adaptive number of word-units includes: The adaptive number of word-grams is determined based on task analysis data for the task type, wherein the task analysis data indicates how different adjustments to one or more of the corresponding initial numbers of word-grams affect the accuracy of the inference results associated with the processing of the batch and the latency level of the transformer model.

17. A computing system comprising at least one processor, wherein the at least one processor is configured to: A query is received for processing by a transformer model, wherein The processing of the query is associated with processing a tokenized representation of each input data sample of the query, the tokenized representation including a respective initial token number of execution tokens; determining, based on one or more characteristics of the query, word-gram adaptation settings for processing the query using the transformer model, wherein at least some of the word-gram adaptation settings include adjusting at least some initial word-gram numbers using the transformer model through at least one of a word-gram hinting process or a word-gram merging process; as well as After instructing the transformer model to adopt the word-gram adaptation setting, the query is applied as input to the transformer model to generate an inference result for the query.

18. The computing system of claim 17, wherein: The one or more characteristics are selected from the group consisting of: a first characteristic applicable to input data sample content, a second characteristic applicable to query tasks, a third characteristic applicable to utility values, and a fourth characteristic applicable to delay requirements.

Images