Self-balancing hybrid expert

Abstract

Methods for identifying and redistributing MOE machine learning model experts distributed on different accelerators are provided. The system identifies a set of input markers to be routed to a plurality of experts and identifies routing assignments of the set of input markers to the plurality of experts. After identifying a current distribution of the plurality of experts over the plurality of accelerators, the system determines a new distribution of the plurality of experts over the plurality of accelerators, and determines a new distribution of the plurality of experts over the plurality of accelerators based on a route of the set of marks to the plurality of experts compared to the current distribution of the plurality of experts over the plurality of accelerators. The new distribution will result in an improved processing efficiency of the set of input flags by multiple accelerator assignments. A new distribution of multiple experts is also applied to achieve an expected improvement in processing efficiency.

Description

Background Art

[0001] Machine learning models using mixture of experts (MOE) techniques typically consist of N layers, which can be roughly categorized as MOE layers and non-MOE layers. Various distribution strategies are used to distribute large MOE machine learning models to computing system hardware.

[0002] When distributing models using a traditional MOE distribution strategy, a single accelerator or graphics processing unit (GPU) is assigned some or all layers of the model, including both MOE and non-MOE layers. However, there are many problems associated with this distribution. For example, some components will remain idle while others are still processing input data. Furthermore, due to the limitations of current hardware devices in existing computing systems, such models are not scalable. Furthermore, training MOE models with multiple distributed layers and experts can be computationally intensive and time-consuming.

[0003] In view of the foregoing, there is a continuing need for improved systems and methods for MOE machine learning models that can be distributed across different types of hardware configurations.

[0004] The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is provided merely to illustrate one exemplary technology area where some embodiments described herein may be practiced. Summary of the Invention

[0005] The disclosed embodiments include systems and methods for distributing MOE models across different computing systems. Specifically, systems and methods are provided for determining new expert distributions based on potentially improving the processing efficiency of a computing system by self-balancing expert components of MOE models across multiple accelerators.

[0006] The disclosed system accesses a computing system having multiple experts distributed across different accelerators. The system also identifies a routing assignment (e.g., a data set or index table correlating the relationships between input tags, experts, and / or accelerators of the computing system) that specifies which input tags will be routed to one or more of the experts. After identifying the current distribution of the experts across the accelerators, the system determines a new distribution of the experts across the accelerators based on comparing the routing assignments of the tags with the current distribution of the experts, the new distribution resulting in improved processing efficiency of the tags by the different accelerators, thereby efficiently processing the expected load associated with the accelerators. Finally, the system applies the new distribution of experts across the accelerators to achieve the expected improvement in processing efficiency. In some cases, this new distribution is applied before the machine learning model actually receives and / or processes the input tags that are routed to or by the MOE model that is instantiated and distributed across the accelerators.

[0007] Also provided are systems and methods for determining new distributions during runtime processing of input tokens. For example, the system can identify a real-time or near-real-time processing imbalance of the input tokens based on the current distribution of experts across accelerators. In response to identifying the processing imbalance, the system can determine a new distribution that will improve the processing imbalance and apply the new distribution to the computing system.

[0008] Also provided are systems and methods for determining a new distribution of experts after an initial processing iteration has completed. For example, the system identifies a historical record of processing input tokens by a machine learning model and identifies a current distribution of experts. Then, based on the historical record of processing the input tokens and the current distribution of experts, the system determines a new distribution that will result in improved processing efficiency for the computing system and applies the new distribution of experts to the accelerator.

[0009] This summary is provided to introduce some concepts in a simplified form that are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] To illustrate how the advantages and features of the systems and methods described herein can be obtained, the embodiments briefly described above will be described in more detail with reference to specific embodiments illustrated in the accompanying drawings. It should be understood that these drawings depict only typical embodiments of the systems and methods described herein and, therefore, should not be considered limiting of their scope. Certain systems and methods will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0011] Figure 1A-Figure 1D Various example embodiments of existing MOE systems are illustrated.

[0012] Figure 2 Illustrated is an example diagram of an MOE machine learning model distributed on a computing system according to the disclosed embodiments.

[0013] Figure 3 Illustrated is a computing environment in which a computing system incorporates and / or is utilized to perform the disclosed aspects of the disclosed embodiments.

[0014] Figures 4A-4E Various embodiments for distributing experts across multiple accelerators are illustrated.

[0015] Figure 5A-5B Various embodiments of label routing assignments are illustrated.

[0016] Figure 6A-Figure 6B Various embodiments of accelerator capacity and expert position tracking are illustrated.

[0017] Figure 7A-7B Various embodiments of distributing expert slices across multiple accelerators are illustrated.

[0018] Figures 8-10 Various embodiments are illustrated with a flow diagram having multiple actions for distributing an expert on a computing system. DETAILED DESCRIPTION

[0019] The disclosed embodiments are directed to systems and methods for distributing and / or redistributing expert components of MOE machine learning model instances across accelerators of a computing system. For example, the disclosed embodiments include identifying existing distributions of experts for MOE machine learning model instances that have been loaded or instantiated on different accelerators of a computing system to improve processing efficiency and overall input labeling balance of the computing system.

[0020] It should be understood that some of the disclosed embodiments are particularly directed to improved systems and methods for determining the distribution of machine learning model instances based on the separation of sparse layers and dense layers on custom hardware devices. The distribution of machine learning model instances referred to herein refers to the process of identifying different layers of a machine learning model and assigning these different layers to different components of a computing system, where certain layers (or sets of layers) are stored and processed independently of each other by different accelerators.

[0021] For example, a hybrid expert machine learning model includes multiple experts, each of which is trained for a specific task. Each expert includes one or more machine learning model layers, where each expert can be independently loaded and stored on an accelerator independent of other experts and used to process inputs separately from other experts. Therefore, the hybrid expert machine learning model can be distributed or redistributed to a computing system in a variety of different configurations, where different accelerators store one or more experts to process inputs, as will be described in more detail below. During the mentioned distribution and redistribution, experts can be moved to or removed from corresponding accelerators. This can also include migrating experts from one accelerator to another in whole or in part, as will be described in more detail below.

[0022] In light of the above, references to determining the distribution of a machine learning model or machine learning model instance refer to assigning, planning, or organizing different layers to different components (e.g., accelerators) of a computing system. References to distributed applications herein refer to the process of separating layers or experts of a machine learning model according to a distribution (e.g., a distribution scheme) and storing and processing them on different components of the computing system.

[0023] The disclosed embodiments provide numerous technical advantages over existing systems. For example, some accelerators (e.g., GPUs or other processors) may be overloaded based on how many tokens the expert component of the model placed on the corresponding accelerator is assigned to process. Notably, if an accelerator is overloaded, it may drop input tokens, which can significantly reduce the overall throughput and quality of the output processed by the machine learning model.

[0024] Additionally or alternatively, even if an accelerator is not overloaded, it may be out of balance compared to other accelerators that may be underutilized. In this case, migrating or swapping experts to underutilized accelerators may still be beneficial in order to improve the processing efficiency of the computing system. Therefore, the disclosed embodiments are directed to distributing and redistributing experts across the computing system at one or more different points in time during a processing step. By implementing the system in this manner, the system is able to automatically self-balance the distribution of experts across different accelerators in the system.

[0025] In addition, traditional transformer-based machine learning models are constructed using stacked transformer layers that process input data sequentially. For example, the output from the previous transformer layer is used as the input to the next transformer layer. All neurons from a typical transformer layer participate in processing each input. Transformer layers that use all or most of the neurons in the layer are identified as dense layers, while transformer layers that use one or a limited number of neurons in the layer are identified as sparse layers. Dense layers require a large number of floating-point operations (FLOPS) and a large amount of GPU memory to process the input. Machine learning models configured with dense layers in this way are difficult to scale.

[0026] Some data scientists have begun using a variant of the traditional transformer layer, which is called a mixture of experts (MOE) layer, as a way to expand machine learning models. In some cases, the MOE layer is a sparse layer that is built using a group of experts. For example, if a model is trained to perform a specific task, the specific task (e.g., a predictive modeling task) can be decomposed into two or more subtasks. Each expert is then trained for one of the subtasks. Although in some cases, experts are configured as models, such as neural networks with their own node or neuron sets, when a group of experts within a specific machine learning model layer forms a neural network, experts can also be referred to as nodes or neurons. Therefore, in the case of an MOE layer (i.e., a sparse layer), each input can be processed by a limited subset of experts (i.e., neurons) from the MOE layer.

[0027] This is in contrast to dense layers, where all or most neurons participate in data processing, rather than just a few neurons as in sparse layers. In some existing systems, the entire machine learning model, including dense and sparse layers, is distributed on a single piece of hardware, referred to herein as an accelerator (e.g., GPU 1), such as Figure 1A As shown in . For example, as shown, GPU 1 includes multiple layers (e.g., layer N-1, layer N, and layer N+1). Layer N also includes a residual connection & normalization (Add&Norm) layer, one or more feedforward network expert layers (e.g., FFN1, FFN2, FFNe, etc.), a gating layer, an additional residual connection & normalization layer, and a multi-head attention layer. In this way, layer N-1 and layer N+1 are dense layers, while the sparse layers within layer N include different experts.

[0028] With respect to the foregoing and the remainder of this disclosure, the term "accelerator" is mentioned several times. Such an accelerator is a part of a computing system and is a hardware device or processing unit (i.e., a microprocessor) that includes memory and processing capabilities that enhance the performance of the computing system. In some cases, these components are referred to as accelerators because they can increase the speed at which a computing system can process data and perform various functions for which it is programmed. For example, by utilizing an accelerator, a computing system can be enabled to process in parallel with other processing units (such as a CPU) in the computing system.

[0029] It should be understood that there are many different types of accelerators, including but not limited to hardware accelerators, multi-core central processing units (CPUs), graphics accelerators (e.g., graphics processing units (GPUs)), cryptographic accelerators, web accelerators, PHP accelerators, or other types of accelerators, all of which are collectively referred to herein as accelerators, and each accelerator includes one or more dense layers or sparse layers of a corresponding machine learning model.

[0030] It is also noted that the terms "MOE model," "MOE machine learning model," "MOE machine learning model instance," and "model" are all sometimes used interchangeably in this disclosure. Each term generally refers to a MOE-based transformer machine learning model architecture and a corresponding specific instance of the model, where the model is composed of components for processing data (e.g., labeled), and the different layers of the model are used to determine probabilities and / or for generating output predictions or determinations based on inputs and output probabilities corresponding to the inputs, where the probabilities are determined by algorithms, weights, and attention mechanisms applied at each of the different layers.

[0031] Different layers of the MOE-based transformer machine learning model can be configured in various configurations. In some existing systems, different layers of the machine learning model are distributed to multiple accelerators (e.g., GPU 1 and GPU N), where each accelerator has a single expert in its sparse layer, such as Figure 1B For example, although each of GPU 1 and GPU N has Figure 1A However, GPU 1 and GPU N each include only a single expert (e.g., FFN1 in GPU 1 and FFNn in GPU N) in its sparse layer (i.e., layer N).

[0032] In some configurations, dense layers and sparse layers are interleaved. For example, if a machine learning model is constructed using two dense layers (e.g., dense layer 1, dense layer 2) and two sparse layers (e.g., sparse layer 1, sparse layer 2), the machine learning model can be constructed as Figure 1C As shown, the input data is first processed by dense layer 1, then by sparse layer 1, then by dense layer 2, and finally by sparse layer 2 to generate the final output.

[0033] exist Figure 1D , each sparse layer consists of at least two experts. For example, sparse layer 1 includes a first plurality of experts (e.g., S1E1 and S1E2), and sparse layer 2 includes a second plurality of experts (e.g., S2E1 and S2E2). In order to accommodate such a large machine learning model (i.e., the entire model will not fit into a single accelerator), the layers of the machine learning model are distributed to multiple accelerators. For example, dense layer 1, sparse layer 1, and dense layer 2 are distributed on GPU 1, while dense layer 3, sparse layer 2, and dense layer 4 are distributed on GPU 2. This scheme for distributing or configuring model layers and experts on accelerators is called model parallelism. Model parallelism is extremely inefficient because when GPU 1 is processing input, GPU 2 is idle; and when GPU 2 is processing input, GPU 1 is idle.

[0034] Some work has focused on alleviating this inefficiency by introducing processing pipelining, such that GPU 1 starts processing the second input while GPU 2 is processing the first (after GPU 1 has finished processing the first input). However, this configuration still has drawbacks because any experts in one or more sparse layers that are not involved in processing a given input still occupy a large amount of GPU memory, so GPU utilization remains low.

[0035] An additional improvement, called expert parallelism, was explored, which provides a model configuration in which the experts are evenly distributed across the GPUs. In this configuration, the system can process up to N inputs simultaneously across N GPUs. In one example, with four GPUs and four experts, each GPU is assigned only a single expert for each sparse layer. In this configuration, the system can process up to four inputs simultaneously.

[0036] The sparse layers swap inputs so that each input is sent to the GPU that stores the expert selected to process that input. However, even with this improvement, there are still capacity limitations. For example, each GPU processes dense layers in addition to the sparse layers. In some cases, this is inefficient because the sparse layers take up a lot of memory, while the sparse layers do require less processing than the dense layers. This reduces the overall computational efficiency of the system. Additionally or alternatively, the model on each GPU consumes the same amount of memory and computing resources. Therefore, scalability is limited to the GPU with the lowest computing power and memory, which leads to inefficiency for GPUs in the system with larger memory storage and / or computing power.

[0037] Turn your attention to Figure 2 , which illustrates an example embodiment of a special expert machine learning model in which sparse layers are distributed to sparse hardware and dense layers are distributed to dense hardware, and in which the sparse layers are interleaved with the dense layers so that the sparse hardware can process multiple outputs from multiple dense hardware devices.

[0038] For example, computing system 200 is shown as having multiple accelerators (e.g., accelerator 202, accelerator 204, accelerator 206, accelerator 208, and one or more other accelerators not shown). The machine learning model is distributed to each accelerator. For example, a first plurality of model layers (e.g., layer 210, layer 211, and layer 222) are shown as being distributed to accelerator 202. Each layer also includes one or more layers (i.e., sublayers). For example, layer 211 includes layer 212 (e.g., residual connection & normalization), layer 214 (e.g., sparse layer) (which also includes gating layer 216), layer 218 (e.g., residual connection & normalization), and layer 220 (e.g., multi-head attention).

[0039] Similarly, a second plurality of model layers (e.g., layer 224, layer 215, and layer 234) are shown as being distributed to accelerator 204. Each layer further includes one or more layers (i.e., sub-layers). For example, layer 215 includes layer 226 (e.g., residual connection & normalization), layer 224 (e.g., sparse layer) (which also includes gating layer 228), layer 230 (e.g., residual connection & normalization), and layer 232 (e.g., multi-head attention).

[0040] like Figure 2 As shown in , experts from different sparse layers have been distributed to different accelerators. For example, experts 236 and 238 from the sparse layer associated with layer 210, and experts 240 and 242 from layer 214 associated with layer 211, are distributed to accelerator 206. Furthermore, experts 244, 246, 248, and 250 are distributed to accelerator 208. As shown, the experts are configured to receive different inputs from various layers of the machine learning model, which are distributed across multiple accelerators. This prevents certain components from being idle while other components are actively processing data.

[0041] Now turn your attention to Figure 3 , which illustrates components of a computing system 310 that may include and / or be used to implement aspects of the disclosed invention. As shown, the computing system includes multiple machine learning (ML) engines, models, and data types associated with the inputs and outputs of the machine learning engines and models. For example, Figure 3 The diagram shows a computing system 310 as part of a computing environment 300 that also includes a remote / third-party system 320 in communication with the computing system 310 (via a network 330). The computing system communicates with the remote / third-party system(s) 320, which include one or more processors 322 and one or more computer-executable instructions 324. In some cases, it is contemplated that the remote / third-party system(s) 320 also include a database for containing data that can be used as training data, such as external speaker data. Additionally or alternatively, the remote / third-party system(s) 320 include a machine learning system external to the computing system 310. In some embodiments, the remote / third-party system 320 is a software program or application.

[0042] For example, computing system 310 includes one or more processors 312 (such as one or more hardware processors) and a storage device (i.e., hardware storage device(s) 340) for storing computer-executable instructions 318. One or more of hardware storage device(s) 340 can store any number of data types and any number of computer-executable instructions 318. When computer-executable instructions 318 are executed by one or more processors 312, computing system 310 is configured to implement one or more aspects of the disclosed embodiments through these instructions.

[0043] The computing system 310 also includes multiple accelerators (e.g., a dense accelerator 313 and a sparse accelerator 314). In some configurations, the dense accelerator 313 is configured to process input data using dense layers, wherein the dense accelerator 313 is custom hardware optimized for processing power. In such configurations, the sparse accelerator 314 is configured to process input data using sparse layers, wherein the sparse accelerator 314 is custom hardware optimized for memory storage. The sparse accelerator 314 is more efficient than the dense accelerator in processing sparse data (e.g., sparse tensors, sparse layers). Each accelerator may include a dedicated processor or other hardware capable of storing and / or executing corresponding dense layers and sparse layers (respectively 344 and 346).

[0044] In some cases, the sparse accelerator has at least 10% greater memory or storage capacity than the dense accelerator, or even greater (e.g., 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, or even greater than 100% greater memory or storage capacity than the dense accelerator). Additionally or alternatively, the sparse accelerator is at least 10% more efficient than the dense accelerator in processing sparse data, or even greater (e.g., 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, or even greater than 100% more efficient than the dense accelerator in processing sparse data).

[0045] In some cases, a dense accelerator has at least 10% greater processing power than a sparse accelerator, or even greater (e.g., 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, or even greater than 100% greater processing power than a sparse accelerator). For example, a dense accelerator is more efficient than a sparse accelerator at processing dense data (e.g., dense layers, dense tensors).

[0046] In some cases, a sparse accelerator is distinguished from a dense accelerator based at least on its greater efficiency in processing sparse data. Additionally or alternatively, a sparse accelerator is distinguished from a dense accelerator in that the sparse accelerator has a larger memory capacity and / or a lower raw FLOP count than the dense accelerator. Computing system 310 is also shown as including user interface(s) 315 and input / output (I / O) device(s) 316.

[0047] like Figure 3, hardware storage device(s) 340 are shown as a single storage unit. However, it should be understood that in some embodiments, hardware storage device(s) 340 are distributed storage devices that are distributed to multiple separate, sometimes remote and / or third-party systems 320 (e.g., hardware storage device 324). In some embodiments, computing system 310 may also include a distributed system in which one or more of the components of computing system 310 are maintained / operated by different discrete systems that are remote from each other and each perform different tasks. In some cases, multiple distributed systems perform similar and / or shared tasks to implement the disclosed functionality, such as in a distributed cloud environment.

[0048] In this manner, different layers of a machine learning model may be distributed onto computing system 310 and / or distributed within a distributed computing system 300 including computing system 310 and one or more third-party systems 320. Hardware storage device(s) 340 are configured to store different data (e.g., input labels 348), including various models, such as a machine learning model 342 that includes multiple experts (e.g., expert 343).

[0049] The storage device (e.g., hardware storage device(s) 340) includes computer executable instructions 318 for instantiating or executing one or more of the models and / or engines shown in the computing system 310. The models are configured as machine learning models or machine learning models, such as deep learning models and / or algorithms. In some cases, one or more models are configured as engines or processing systems (e.g., computing systems integrated within the computing system 310), where each engine (i.e., model) includes one or more processors (e.g., hardware processor(s) 312) and computer executable instructions 318 corresponding to the computing system 310.

[0050] Figure 3 3. The present invention also shows an additional storage unit for storing (multiple) machine learning (ML) engines 350, which store multiple machine learning models and / or engines. For example, the computing system 310 includes one or more of the following components: a data retrieval engine 351, a distribution engine 352, and an implementation engine 353, which are individually and / or collectively configured to implement the various functions described herein.

[0051] For example, the data retrieval engine 351 is configured to locate and access data sources, databases, and / or storage devices comprising one or more data types, from which the data retrieval engine 351 can extract a data set or subset to be used as input data. The data retrieval engine 351 receives data from the database and / or hardware storage device, wherein the data retrieval engine 351 is configured to reformat or otherwise expand the received data to be used as training data or input data. Additionally or alternatively, the data retrieval engine 351 communicates with one or more remote / third-party systems (e.g., remote / third-party system(s) 320) comprising remote / third-party data sets and / or data sources. In some cases, these data sources include audiovisual services that record voice, text, images, and / or video.

[0052] The data retrieval engine 351 accesses electronic content including acoustic data, text data, and / or other types of audiovisual data (including video data, image data, holographic data, 3D image data, etc.). The data retrieval engine 351 is an intelligent engine that is capable of learning the optimal dataset extraction process to provide sufficient data volume in a timely manner and retrieve data that is most suitable for the desired application of the machine learning model / engine. For example, the data retrieval engine 351 can learn which databases and / or datasets will generate training data that will be used to train the model (e.g., for a specific query or a specific task), thereby improving the accuracy, efficiency, and efficacy of the model in the desired tier distribution configuration.

[0053] In some embodiments, the computing system 310 includes a distribution engine 352 that is configured to determine the distribution of different layers of the machine learning model 342 across different accelerators. The distribution engine 352 is further configured to apply the distribution before instantiation of the model, or to redistribute the machine learning model after instantiation of the model. In some cases, the redistribution is based on identifying potential improvements in one or more properties of the computing system (e.g., model throughput, computational efficiency), and / or based on scaling up or down the machine learning model. In some embodiments, the redistribution is achieved by migrating one or more experts from one accelerator to another accelerator.

[0054] The distribution engine 352 is configured to dynamically identify the total number of accelerators comprising the computing system 310 and to identify which accelerators are hardware devices specifically designed or optimized for dense layers and sparse layers. Furthermore, the distribution engine 352 is further configured to identify which accelerators are full (i.e., have no storage capacity to store another expert's accelerator) and / or which accelerators have available or expected processing capacity and / or memory space to store and execute one or more additional experts.

[0055] The distribution engine 352 is also configured to identify accelerators that are underutilized, full, or overloaded, terms that refer to the accelerator's ability to process input tokens. An underutilized accelerator (e.g., Figure 6A An accelerator 412 in (e.g., 412) refers to an accelerator that stores a set of experts that are assigned to collectively process a number of inputs that is less than its processing capacity, meaning that the accelerator may be able to store additional experts and / or other experts that are assigned to process more labels than the current experts. A full capacity accelerator (e.g., Figure 6A An accelerator 422 in the example is an accelerator that stores a set of experts that are assigned to collectively process a number of tokens equal to the number of tokens that the accelerator is configured to process. An overloaded accelerator is an accelerator that stores a set of experts that are assigned to collectively process a number of inputs greater than the number of inputs that the accelerator is capable of processing. For example, in Figure 6A In the example, accelerator 402 has an accelerator capacity for processing a maximum of 4 tags. However, expert A (which is assigned to process 4 tags) and expert B (which is assigned to process 2 tags) are currently stored in accelerator 402. This means that based on the stored set of experts, accelerator 402 is assigned to process a total of 6 tags, which means that accelerator 402 is overloaded and will likely lose one or more tags during processing.

[0056] The distribution engine 352 is further configured to identify how many layers the machine learning model consists of, and which layers are dense layers and which layers are sparse layers. The distribution engine 352 is further configured to identify how many experts there are in each different sparse layer, and to identify the attributes of the experts (e.g., the specialized tasks associated with the experts).

[0057] Thus, based on the number of dense layers and the number of sparse layers (or the number of experts across one or more sparse layers), the distribution engine 352 is configured to dynamically and automatically distribute different layers or different experts to one or more accelerators of the computing system 310. In some cases, the distribution engine 352 is configured to distribute the dense layer 344 to the dense accelerator 313 (e.g., an accelerator determined to have the ability to store and / or execute dense layers) and distribute the sparse layer 346 to the sparse accelerator 314 (e.g., an accelerator with more limited capacity and capabilities than the dense accelerator). The distribution engine 352 can also distribute or redistribute different slices of an expert to different accelerators.

[0058] In some cases, the distribution engine 342 is further configured to separate the sparse layers (which include one or more experts) from the dense layers, and then distribute the sparse layers to an accelerator configured to store / process the sparse layers (e.g., an accelerator with greater processing power than other accelerators), and distribute the dense layers to an accelerator configured to store / process the dense layers (e.g., an accelerator with greater memory power than other accelerators). In this way, the sparse layers and dense layers are divided into different groups, and each group is assigned its own set of accelerators. By distributing machine learning models according to this configuration, the computing system can achieve the following technical advantages. First, the system can selectively apply specific performance optimizations applicable to dense and sparse computations. Additionally, MOE layers (such as sparse layers including one or more experts) contain a large amount of communication overhead. Therefore, the configuration of distributing the sparse layers and dense layers to different accelerators enables the system to utilize the higher communication bandwidth available within a subset of the cluster (e.g., on a single or multiple nodes in a single rack).

[0059] Furthermore, by distributing machine learning models in this manner, a single set of MOE layers can be scheduled to process multiple interleaved inputs, effectively increasing model throughput (for training, inference, and / or implementation). The disclosed embodiments are also directed to systems and methods that can use heterogeneous clusters to reduce computational costs and improve the performance of computing systems / machine learning models. Specifically, such a configuration facilitates a significant reduction in the time required to train machine learning, thereby allowing users to deploy machine learning models to implementation tasks more quickly.

[0060] In some embodiments, computing system 310 includes an implementation engine 353 that communicates with any one of the models and / or ML engines 350 (or all of the models / engines) included in computing system 310, such that implementation engine 353 is configured to implement, initiate, or execute one or more functions of the plurality of machine learning engines 350. In one example, implementation engine 353 is configured to operate data retrieval engine 351, such that data retrieval engine 351 retrieves data at the appropriate time to enable processing of input tokens 348. Implementation engine 353 facilitates process communication and communication timing between one or more of the machine learning engines 350.

[0061] In another example, implementation engine 353 is configured to implement one or more functions of machine learning model 342 (i.e., processing input tokens 348) that are distributed on computing system 310 or across computing system 310 and third-party system(s) 320. Implementation engine 353 is also configured to implement distribution engine 352 to identify distribution or redistribution of different layers of the machine learning model.

[0062] Furthermore, implementation engine 353 is configured to select which experts distributed across the various accelerators will be used to process input tokens 348. By implementing the systems and methods according to these disclosed embodiments, computing system 310 can achieve technical advantages, such as customizability and scalability. Specifically, different experts can be used to process input tokens 348 at different times. Thus, the system can be configured to select a limited number of experts to use for processing input data based on the type of input data, the format of the input data, the context of the input data, and / or the downstream application of the processed input data.

[0063] When one or more experts are identified as being used to process input data, the system can distribute or redistribute one or more sparse layers including these identified experts to different accelerators to improve model throughput and increase the computational efficiency of the system. Additionally, the system can distribute or redistribute one or more specific experts to accelerators to further improve the computational efficiency of the system.

[0064] Now turn your attention to Figures 4A-4E , which illustrate various embodiments of distributing and redistributing machine learning models (e.g., a mixture of experts) across different accelerators of a computing system (e.g., computing system 310). It should be understood that the following systems and methods related to redistributing experts (e.g., self-balancing of experts) to achieve improved processing efficiency, in Figure 2 Further improvements are achieved on the systems and methods illustrated in FIG.

[0065] First turn your attention to Figure 4A , which illustrates the initial distribution of experts across multiple accelerators. For example, accelerator 402 includes a gating function 406 that receives input 404 (e.g., one or more input labels), experts A and B, and a residual connection and normalization layer (e.g., residual connection + normalization 408) that generates output 410. The gating function is used to dynamically or according to a predetermined routing to route the input labels to the corresponding experts located on the accelerator. Accelerator 412 includes a gating function 416 that receives input 414, experts C and D, and a residual connection + normalization 418 that generates output 420.

[0066] In some cases, each tag is pre-assigned to a specific expert, as indicated by a routing assignment (e.g., input tag routing 401). Figure 4AAs shown in , tag 1 will be routed to expert A, tag 2 will be routed to expert C, tag 3 will be routed to expert B, tag 4 will be routed to expert D, tag 5 will be routed to expert A, tag 6 will be routed to expert B, tag 7 will be routed to expert B, and tag 8 will be routed to expert A. In this case, expert A will process three tags, expert B will process three tags, expert C will process one tag, and expert D will process one tag. Since experts A and B are currently distributed on accelerator 402 (e.g., GPU 1), accelerator 402 will process six tags according to the routing assignment (i.e., the input tag routing assignment). Since experts C and D are currently distributed on accelerator 412 (e.g., GPU 2), accelerator 412 will process two tags according to the input tag routing assignment. Therefore, even if the two accelerators have the same number of experts, accelerator 402 will process more tags than accelerator 412.

[0067] In some cases, this results in an imbalance in the distribution of input processing for machine learning models on the current computing system. For example, accelerator 402 may be overloaded with an excessive number of labels compared to underutilized accelerator 412. Therefore, embodiments of the present disclosure are directed to systems and methods for improving the processing efficiency of a computing system and / or improving the balance of processed labels across multiple accelerators by redistributing experts among available accelerators.

[0068] Now turn your attention to Figure 4B ,The figure illustrates the redistribution of experts on different accelerators, whose input label routing is Figure 4A For example, Expert B and Expert C have swapped positions, such that (i) Accelerator 402 now stores Expert A and Expert C, and (ii) Accelerator 412 stores Expert B and Expert D. Thus, Accelerator 402 now processes four tokens, and Accelerator 412 also processes four tokens.

[0069] This improves the processing efficiency of the computing system because both accelerators are now being utilized equally. Furthermore, because output 410 and output 420 will be generated at approximately the same time (or closer), processing efficiency is also improved because both accelerators are processing the same number of tokens. This also reduces computation time because if a subsequent layer of the machine learning model requires output 410 and output 420 before it can generate a subsequent output, the subsequent layer does not need to process both outputs 410 and 420 as often. Figure 4A Wait for accelerator 402 to complete processing of more tags as in .

[0070] Now turn your attention to Figure 4C, which illustrates an alternative redistribution of experts across multiple accelerators. In some cases, to improve the processing efficiency of a computing system, one or more experts may be relocated (e.g., where an expert is moved from being stored on one accelerator to being stored on a different accelerator) to a newly available accelerator, rather than Figure 4B Interchange (i.e., swap, replace) between current accelerators as shown in . In this case, the computing system can identify other accelerators that can store one or more experts. Figure 4C As shown in FIG, accelerator 422 is identified as an additional accelerator that can be used for expert distribution, in addition to accelerator 402 and accelerator 412. In some cases, if the system determines that the accelerator that initially stores the expert(s) will exceed its capacity (e.g., it will be overloaded, it is currently overloaded, or is marked as overloaded by input), the computing system initiates a search for additional accelerators and / or initiates the relocation of one or more experts to other accelerators.

[0071] like Figure 4C As shown in FIG, accelerator 422 includes a gating function 426 that receives input 424, experts relocated from other accelerators, and a residual connection + normalization 428 that generates output 430. For example, expert C has been relocated from accelerator 402, which now has empty space (i.e., empty space or memory storage available for the experts), to accelerator 422, while expert D has been relocated from accelerator 412 (which now also has empty space) to accelerator 422.

[0072] In this new distribution (or redistribution) of the expert, there is Figure 4A The same token input routing, accelerator 402 (e.g., GPU 1) processes three tokens, accelerator 412 (e.g., GPU 2) processes three tokens, and accelerator 422 (e.g., GPU 3) processes two tokens. Figure 4A Initial distribution in or Figure 4B Compared to the redistribution performed by utilizing additional accelerators in FIG4 , this redistribution reduces the number of tokens processed by accelerator 402 and accelerator 412. This, in turn, improves the processing efficiency of the computing system and reduces the computation time for processing input tokens because accelerator 402, accelerator 412, and accelerator 422 can process tokens simultaneously.

[0073] Now turn your attention to Figure 4D, which illustrates an alternative embodiment of redistributing experts across multiple accelerators according to input tag routing 403 (which is a new input tag routing). In the new input tag routing, tag 1 is assigned to expert A, tag 2 is assigned to expert A, tag 3 is assigned to expert C, tag 4 is assigned to expert C, tag 5 is assigned to expert A, tag 6 is assigned to expert D, tag 7 is assigned to expert D, and tag 8 is assigned to expert 8. Therefore, expert A will process four tags, expert B will process zero tags, expert C will process two tags, and expert D will process two tags.

[0074] Based on the new input tag routing, if accelerator 402 initially stores expert A and expert B, and accelerator 412 initially stores expert C and expert D (see, for example, Figure 4A ), then accelerator 402 will process four tokens and accelerator 412 will also process four tokens. However, the computing system is able to identify the redistribution, which improves the processing efficiency of the computing system and reduces the size of the machine learning model. For example, Figure 4D As shown in , the computing system determines that expert B will not be used to process any tokens from the set of input tokens associated with input token routing 403. Therefore, expert B is removed from accelerator 402, which now has empty space. In some cases, expert B can be cached in a hold storage memory until it is needed again to process input tokens.

[0075] Now turn your attention to Figure 4E , which illustrates another example embodiment of redistributing experts across multiple accelerators. For example, when an accelerator has one or more empty spaces (i.e., it has available capacity to store one or more experts), a new expert can be stored at that location on the accelerator or relocated to that location from a different accelerator.

[0076] Additionally or alternatively, a particular expert with a large number of tags to process can be replicated so that multiple identical experts can process the assigned tags in parallel. As shown in diagram 402, a copy of expert A is stored on accelerator 402 so that accelerator 402 now has two identical experts (i.e., two experts A). Thus, although accelerator 402 still processes four tags and accelerator 412 still processes four tags according to input tag routing 403, accelerator 402 will be able to process more tags than accelerator 412. Figure 4A or Figure 4D The initial distribution of experts shown in processes the tags assigned to expert A more quickly because two experts A can process the assigned tags in parallel. This improves the processing efficiency of the computing system and reduces computing time, particularly by facilitating parallel processing of tags assigned to the same expert.

[0077] Note that during model training, the system is also configured in some cases to synchronize the two replicas of expert A after processing the input tokens. Each expert's weights are updated during training as part of the input token processing. Therefore, synchronizing the replicated experts ensures that the updated weights remain replicas of each other, even if processing the input tokens modifies one or more of either expert's weights.

[0078] Now turn your attention to Figure 5A-5B , which illustrate the current and new distribution of experts. Since the computing system can implement the redistribution of experts before input tag processing, during runtime of input tag processing, or after initial input tag processing, it is beneficial to keep track of which tags are assigned to which experts and which accelerators store which experts.

[0079] The computing system is advantageously configured to track markers, experts, and accelerators. For example, Figure 5A As shown in , under the current or initial distribution of experts, label 1 is assigned to expert A stored on accelerator 402; label 2 is assigned to expert A stored on accelerator 402; label 3 is assigned to expert B also stored on accelerator 402, and so on. After the computing system determines that the new distribution will result in an improvement in the processing efficiency of the computing system, the system will redistribute the experts according to the new distribution. For example, Figure 5B It is shown that, although label 3 is still assigned to expert B, expert B has been relocated to accelerator 412 .

[0080] In some cases, such as Figure 5B As shown in , the user interface displays a labeled routing assignment chart with a visual indication or formatting (e.g., underlining, bolding, font color change, etc.) that indicates whether the expert location has changed (e.g., "Accelerator 412"). Some of the disclosed embodiments include using a data structure (e.g., a tracking chart, table, or other data structure) to manage routing assignments. For example, Figure 5A-5B The diagram shows a tag routing assignment chart that can be used to track the current and correct status of accelerator positions. A similar chart with modified data fields can also be used to track the current and / or historical load and / or expected load for different accelerators and experts. This advantageously allows the user to visually check the current and any subsequent new distribution of tag assignments and expert positions, as well as the expected utilization / load of different experts and accelerators in a convenient manner.

[0081] Now turn your attention to Figure 6A-Figure 6B , which illustrate an unbalanced system and a subsequent balanced system in terms of accelerator capacity and expert tracking. Specifically, Figure 6AThe current distribution of experts across multiple accelerators, the number of tags expected to be processed by the accelerators, and the analysis between the accelerator's capacity and the number of tags are shown. For example, accelerator 402 stores Expert A and Expert B, resulting in a total of six tags expected to be processed.

[0082] In this case, because the accelerator's processing capacity is four tags, the system determines that accelerator 402 is overloaded. Accelerator 412 initially stores Expert C and Expert D and is expected to process a total of two tags. Therefore, accelerator 412 is determined to be underutilized. Accelerator 422 is assumed to accommodate Expert E and Expert F, each processing two tags, so accelerator 422 is expected to process a total of four tags. Since accelerator 422 has a capacity of four tags per processing iteration, accelerator 422 is determined to be at capacity. Overall, the system is determined to be unbalanced.

[0083] Now turn your attention to Figure 6B , which illustrates the new distribution of experts to achieve a balanced system state. In this example, the system identifies that expert D should be removed from accelerator 412 because no tags are assigned to accelerator 412. This frees up expert space on accelerator 412. Additionally, because accelerator 402 is overloaded, the system determines that relocating expert B to accelerator 412 will achieve a balanced system. Figure 6B As shown in , each accelerator is now processing at a capacity that achieves an equilibrium system state.

[0084] It should be understood that although Figure 6A-Figure 6B Accelerators are shown having the same capacity for storing experts (e.g., two experts per accelerator) and the same capacity for processing tags (e.g., four tags per accelerator), but each accelerator may include a different capacity for storing one or more experts and may include a different capacity for processing one or more tags. The computing system is advantageously configured to identify a combination of available accelerators that will produce a more balanced system than the initial combination if there is an opportunity to redistribute the experts to improve the processing efficiency of the computing system.

[0085] Now turn your attention to Figure 7A-7B , which illustrate alternative embodiments for redistributing experts across multiple accelerators by relocating an expert's slice from one accelerator to another, removing an expert's slice from one accelerator, and / or swapping slices of different experts between accelerators. For example, Figure 7AAs shown in , each expert is split into multiple shards. Each shard consists of a fully functional executable module that can be separated and run independently from the other modules of the expert. Similar to how experts are trained to perform specific subtasks of a more complex task corresponding to the overall machine learning model, shards are trained to perform sub-subtasks of the subtask for which the expert was trained.

[0086] For example, if a particular subtask associated with Expert A can be divided into three separate parts or functions, then Expert A can be divided into three different shards accordingly, with each shard associated with a different function (i.e., each function / part has one or more shards). Figure 7A As shown in , expert A includes slices A1, A2, and A3. Expert B includes slices B1, B2, and B3. Expert C includes slices C1, C2, and C3. Expert D includes slices D1, D2, and D3. In this case, input tags can be assigned to specific experts and / or specific slices corresponding to the experts.

[0087] Figure 7A Also shown is an accelerator 702, which includes a gating function 706 that receives an input 704 (e.g., an input tag), expert A and its corresponding shards, expert B and its corresponding shards, and residual connections + normalization 708. An accelerator 710 is also shown with a gating function 714, expert C and its corresponding shards, expert D and its corresponding shards, and residual connections + normalization 716. In this example, the computation determines that the current distribution of experts and their corresponding shards can be changed to result in an improvement in computational processing efficiency. For example, it is determined that shard A3 and shard C1 should be swapped. The new distribution is as follows: Figure 7B As shown in , accelerator 702 now stores slice A1, slice A2, and slice C1, while accelerator 710 now stores slice A3, slice C2, and slice C3.

[0088] The following discussion will involve several methods and method actions. Although method actions may be discussed in a particular order or illustrated in a flowchart as occurring in a particular order, no particular order is required unless otherwise indicated or because one action depends on another action being completed before the action is performed.

[0089] Now turn your attention to Figure 8 , which illustrates a flowchart 800 including various actions (action 810, action 820, action 830, action 840, action 850, action 860, and action 870) associated with an exemplary method for redistributing a hybrid expert machine learning model onto a computing system that may be implemented by computing system 310. The following actions will also be described in conjunction with the components illustrated in the figure.

[0090] The first action shown includes an action for accessing a computing system that includes multiple accelerators (e.g., dense accelerator 313, sparse accelerator 314) (action 810). The computing system also accesses a machine learning model (e.g., machine learning model 342) that includes multiple experts (e.g., expert 343) distributed across the multiple accelerators (action 820). By utilizing a hybrid expert machine learning model, the system is able to customize the machine learning model by using a specific set of experts in each iteration or process.

[0091] The system identifies a set of input tags (e.g., input tags 348) to be routed to a plurality of experts (act 830), and identifies routing assignments of the set of input tags to the plurality of experts (e.g., input tag routing 401) (act 840). For example, the system accesses a data set or index table that stores information about the relationships between input tags and experts (and / or accelerators). Based on this routing assignment information, the system is able to identify and track which tags have been processed, are being processed, or will be processed by different experts. By identifying the routing assignments for the set of tags, the system is able to determine the number of tags assigned to each expert. In addition, the system identifies the current distribution of the plurality of experts across the plurality of accelerators (act 850) (see Figure 4A By identifying the current distribution, the system can determine the number of tags that each accelerator will process. The system can then determine whether there is an imbalance in the number of tags processed across multiple accelerators and / or whether an accelerator is overloaded, at capacity, or underutilized.

[0092] Then, based on the routing assignments of the set of labels to the plurality of experts, the system determines a new distribution of the plurality of experts across the plurality of accelerators that will result in improved processing efficiency of the plurality of accelerators for the set of input labels compared to the current distribution of the plurality of experts across the plurality of accelerators (act 860) (e.g., see Figure 4B Finally, the system applies the new distribution of experts to the multiple accelerators. By applying the new distribution to the system, one or more experts will be relocated to different accelerators, removed from an accelerator, and / or swapped between accelerators so that the new input is processed according to the new distribution of experts across the accelerators.

[0093] In some cases, before applying the machine learning model to a set of labels, a new distribution of multiple experts is first applied to multiple accelerators. This is beneficial because it allows the computing system to improve computational processing efficiency in advance.

[0094] Additionally, in some cases, the computing system also determines a specific number of iterations to process before determining whether a new distribution of the multiple experts should be applied. The system then performs the specific number of iterations before applying the new distribution of the multiple experts to the multiple accelerators. In some cases, an iteration refers to an instance of updating model parameters based on information / feedback learned by processing a specific set of input tags. Sometimes, the model parameters are updated at specific time intervals or based on the number of tags processed. Additionally or alternatively, an iteration refers to processing a discrete set of input tags. For example, the first iteration processes the first set of input tags, the second iteration processes the second set of input tags, and so on. Therefore, in some cases, it is beneficial to utilize computing system resources (e.g., time and / or processing power) to analyze the current distribution of the experts, identify inefficiencies, and generate a new distribution after each iteration to have the latest distribution based on the latest iteration. Alternatively, in some cases, it is beneficial to wait a predetermined number of iterations before changing any distribution of the multiple experts. By implementing the method in this way, the system is able to tune and control the timing of determining and applying new distributions during multiple iterative processing.

[0095] It should be appreciated that the new distribution of experts can occur according to a number of different techniques, including relocating experts to existing or new accelerators, removing experts from a particular accelerator, duplicating experts, exchanging experts between accelerators, and / or a combination of the foregoing techniques. For example, in some cases, applying the new distribution of experts across multiple accelerators includes relocating experts from an accelerator that has been determined to be overloaded to an underutilized accelerator.

[0096] In some cases, applying the new distribution of experts across the plurality of accelerators includes replicating overloaded experts to particular accelerators so that input tokens are processed in parallel by the set of replicated experts. Additionally or alternatively, the system can (i) identify one or more experts that will not be used to process the input tokens based on the routing of the set of tokens to the plurality of experts; and (ii) remove the one or more experts that will not be used from the one or more corresponding accelerators.

[0097] In some cases, an expert is temporarily removed from an accelerator for a period of time and then relocated back to that accelerator. For example, in this case, the system processes a first set of input tags according to a first input tag routing assignment, which causes a particular accelerator to be overloaded, wherein the system determines to remove the expert from the particular accelerator. Subsequently, the system processes a second set of input tags according to a second input tag routing assignment, wherein the system determines to move the expert back to the accelerator because the second input tag routing assignment would not cause the particular accelerator to be overloaded if the expert was stored on that accelerator.

[0098] Different events can trigger a new distribution or redistribution of experts, including identifying an overloaded accelerator or an underutilized accelerator, determining an overall unbalanced system state, predicting and / or identifying discarded tags, and / or identifying a potential improvement in processing efficiency. For example, in some cases, determining a new distribution of multiple experts is performed in response to an active prediction that an overloaded accelerator will likely discard input tags that are currently directed to the overloaded accelerator, and identifying experts that need to be relocated is based on determining that the experts are currently located on the overloaded accelerator. Triggers can also be based on actively predicting a new, currently unapplied processing load (which is expected to be applied to the overloaded accelerator or the underutilized accelerator that will cause it to be overloaded), such as based on detecting that the system has instantiated a new program associated with a particular processing load history for the referenced accelerator(s) (e.g., based on tracked historical data).

[0099] In some systems, experts are further divided into one or more shards corresponding to the experts, e.g. Figure 7A-7B As previously noted, each shard comprises a fully functional executable module that can be separated from the other modules of the expert and run independently of the other modules of the same expert. Similar to how experts are trained to perform specific subtasks of a more complex task corresponding to an overall machine learning model, shards are trained to perform sub-subtasks of the subtask for which the corresponding experts are trained. In such a system, the system splits one or more experts into multiple shards. This is beneficial because the distribution of experts can now become a distribution of shards across multiple accelerators, thereby fine-tuning and further improving the processing efficiency of the computing system because the system is able to distribute or redistribute the machine learning model at a finer granularity using individual shards (rather than entire experts). Therefore, in some cases, applying the new distribution of multiple experts to multiple accelerators includes relocating at least one shard from a first accelerator to a second accelerator and retaining at least one shard located on the first accelerator.

[0100] As described above, in some systems, at least some of the multiple accelerators have a larger memory capacity than other accelerators, and at least some of the accelerators have a larger processing power than other accelerators, which allows for customized computing systems. In addition, the machine learning model includes multiple dense layers and multiple sparse layers, each sparse layer also including one or more experts from a plurality of experts. Therefore, the machine learning model can be beneficially distributed across multiple accelerators, such that the multiple dense layers of the machine learning model are distributed across one or more accelerators with greater processing power, while the multiple sparse layers of the machine learning model are distributed across one or more accelerators with greater memory power.

[0101] In this way, the sparse layers and dense layers are divided into different groups, and each group is assigned its own set of accelerators. By distributing machine learning models according to this configuration, the computing system is able to achieve the following technical advantages, including the ability to apply specific performance optimizations applicable to dense and sparse computations in a selective manner. In addition, MOE layers (such as sparse layers including one or more experts) contain a large amount of communication overhead. Therefore, the configuration of distributing sparse layers and dense layers to different accelerators enables the system to utilize the higher communication bandwidth available within a subset of the cluster (for example, on a single or multiple nodes in a single rack). In addition, by distributing machine learning models in this manner, a single group of MOE layers can be scheduled to process multiple interleaved inputs, thereby effectively improving model throughput (for training, inference and / or implementation).

[0102] It should be understood that each accelerator can store any number of experts. For example, in some cases, multiple experts are evenly distributed across multiple accelerators in the new distribution. Alternatively, each accelerator includes a different number of experts in the new distribution.

[0103] Now turn your attention to Figure 9 , which illustrates a flowchart 900 that includes various actions (action 910, action 920, action 930, action 940, action 950, action 960, and action 970) associated with an exemplary method for redistributing a hybrid expert machine learning model onto a computing system that can be implemented by the computing system 310.

[0104] The first action illustrated includes accessing a computing system (e.g., computing system 310) that includes a plurality of accelerators (action 910). The system also accesses a machine learning model (e.g., machine learning model 342) that includes a plurality of experts (e.g., expert 343) distributed across the plurality of accelerators (action 920) and identifies a set of input tags (e.g., input tags 348) to be routed to the plurality of experts (action 930). By utilizing a hybrid expert machine learning model, the system is able to customize the machine learning model by using a specific set of experts in each iteration or processing. Furthermore, by identifying the set of input tags, the system is able to track where each input tag will be processed in different machine learning layers. Next, the system executes one or more computer-executable instructions (e.g., computer-executable instructions 318) that are configured to cause the computing system to apply the machine learning model to the set of input tags based on the routing assignments of the set of tags to the plurality of experts (e.g., input tag routing 401) (action 940). The system then identifies a real-time processing imbalance of the set of input tags based on the current distribution of the plurality of experts across the plurality of accelerators (action 950) (e.g., see Figure 4ABy identifying real-time processing imbalances, the system is able to dynamically respond and redistribute experts to quickly fix the imbalance.

[0105] Subsequently, based on the routing assignments of the set of labels to the plurality of experts, the system determines a new distribution of the plurality of experts across the plurality of accelerators that will result in improved balanced processing of the set of input labels by the plurality of accelerators compared to the current distribution of the plurality of experts across the plurality of accelerators (act 960), and applies the new distribution of the plurality of experts to the plurality of accelerators (act 970) (e.g., see Figure 4B ).

[0106] It is noted that there are many techniques or triggers for identifying real-time or near real-time processing imbalances. For example, in some cases, identifying a set of marked real-time processing imbalances includes identifying an accelerator that is determined to be overloaded.

[0107] In some cases, identifying an imbalance in real-time processing of a set of tags includes identifying an underutilized accelerator. In some cases, identifying an imbalance in real-time processing of a set of tags is based on determining that an accelerator has discarded input tags. Additionally or alternatively, identifying an imbalance in real-time processing of a set of tags is based on determining that a first accelerator is processing more tags than a second accelerator. In this case, applying a new distribution includes relocating a particular expert from the first accelerator to the second accelerator.

[0108] Now turn your attention to Figure 10 , which illustrates a flowchart 1000 that includes various actions (action 1010, action 1020, action 1030, action 1040, action 1050, action 1060, and action 1070) associated with an exemplary method for redistributing a hybrid expert machine learning model onto a computing system that can be implemented by a computing system 310.

[0109] The first action illustrated includes accessing a computing system (e.g., computing system 310) including a plurality of accelerators (e.g., dense accelerator 313 and sparse accelerator 314) (action 1010); and accessing a machine learning model including a plurality of experts (e.g., expert 343) distributed across the plurality of accelerators (action 1020). The system also identifies a historical record of processing input tokens by the machine learning model (action 1030). The system identifies a current distribution of the plurality of experts across the plurality of accelerators (action 1040) (e.g., see Figure 4A). The system can then determine a new distribution of the experts across the plurality of accelerators based on the historical processing records of the experts on the set of labels, the new distribution resulting in improved and / or balanced processing of the set of input labels by the plurality of accelerators compared to the current distribution of the experts across the plurality of accelerators (action 1050), and apply the new distribution of the experts to the plurality of accelerators (e.g., see Figure 4B ).

[0110] In some cases, the system also identifies accelerators that are determined to be overloaded based on historical processing records and relocates specific experts from overloaded accelerators to underutilized accelerators (e.g., see Figure 6A-Figure 6B Additionally or alternatively, the system further identifies an accelerator determined to be overloaded based on historical processing records, and swaps a first expert in the overloaded accelerator with a second expert in an underutilized accelerator to balance future processing of a set of input tokens by multiple experts on multiple accelerators.

[0111] Example computing system

[0112] Embodiments of the present disclosure may include or utilize a special-purpose or general-purpose computer system (e.g., computer system 310) that includes computer hardware, such as, for example, a processor system (e.g., hardware processor(s) 312) and system memory (e.g., hardware storage device(s) 340), as discussed in more detail below. Embodiments within the scope of the present disclosure also include physical media and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media can be any available media that can be accessed by a general-purpose or special-purpose computer system. Computer-readable media that store computer-executable instructions and / or data structures are computer storage media. Computer-readable media that carry computer-executable instructions and / or data structures are transmission media. Thus, for example, embodiments of the present disclosure may include at least two distinct types of computer-readable media: computer storage media and transmission media.

[0113] Computer storage media are physical storage media that store computer-executable instructions and / or data structures. Physical storage media include computer hardware, such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), solid-state drives (SSDs), flash memory, phase-change memory (PCM), optical disk storage, magnetic disk storage or other magnetic storage devices, or any other hardware storage device(s), which can be used to store program code in the form of computer-executable instructions or data structures that can be accessed and executed by general-purpose or special-purpose computer systems to implement the disclosed functionality.

[0114] Transmission medium can comprise network and / or data link, and they can be used to carry the program code of computer executable instruction or data structure form, and they can be accessed by general or special computer system." network " is defined as one or more data links that can transmit electronic data between computer system and / or module and / or other electronic equipment.When information is transferred or provided to computer system by network or other communication connection (wired, wireless or wired and wireless combination), computer system can be considered as transmission medium by this connection.The combination of above-mentioned connection also should be included in the scope of computer readable medium.

[0115] Furthermore, program code, in the form of computer-executable instructions or data structures, can be automatically transferred from a transmission medium to a computer storage medium (or vice versa) upon reaching various computer system components. For example, computer-executable instructions or data structures received over a network or data link can be cached in RAM within a network interface module and then ultimately transferred to the computer system's RAM and / or less volatile computer storage medium. Therefore, it should be understood that computer storage media can be included in computer system components that also (or even primarily) utilize transmission media.

[0116] Computer-executable instructions include, for example, instructions and data that, when executed on one or more processors, cause a general-purpose computer system, a special-purpose computer system, or a special-purpose processing device to perform a specific function or group of functions. Computer-executable instructions can be, for example, binary files, intermediate format instructions (such as assembly language), or even source code.

[0117] Should be understood that disclosed system and method can be put into practice in the network computing environment with multiple computer system configurations, including personal computers, desktop computers, notebook computers, message processors, handheld devices, multiprocessor systems, based on microprocessor or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablet computers, pagers, routers, switches, etc. The embodiments of the present disclosure can also be put into practice in a distributed system environment, wherein local and remote computer systems are linked (by hard-wired data link, wireless data link or by the combination of hard-wired and wireless data link) through a network, and all perform tasks. Like this, in a distributed system environment, a computer system can include multiple component computer systems. In a distributed system environment, a program module can be located in a local and remote memory storage device.

[0118] It should also be understood that embodiments of the present disclosure can be practiced in a cloud computing environment. A cloud computing environment can be distributed, but this is not required. When distributed, a cloud computing environment can be distributed internationally within an organization and / or have components owned across multiple organizations. In this specification and the claims that follow, "cloud computing" is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services). A cloud computing model can consist of various features, such as on-demand self-service, broad network access, resource pooling, rapid elasticity, measurable services, and the like. A cloud computing model can also take the form of various service models, such as, for example, Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). A cloud computing model can also be deployed using different deployment models, such as private clouds, community clouds, public clouds, hybrid clouds, and the like.

[0119] Some embodiments, such as cloud computing environments, may include a system that includes one or more hosts, each capable of running one or more virtual machines. During operation, the virtual machines emulate an operational computing system, thereby supporting an operating system and possibly one or more other applications. In some embodiments, each host includes a hypervisor that emulates virtual resources for the virtual machines using physical resources abstracted from the virtual machine's perspective. The hypervisor also provides appropriate isolation between virtual machines. Thus, from the perspective of any given virtual machine, the hypervisor provides the illusion that the virtual machine is interacting with physical resources, even though the virtual machine is only interacting with the appearance of physical resources (e.g., virtual resources). Examples of physical resources include processing capacity, memory, disk space, network bandwidth, media drives, and the like.

Claims

1. A method for distributing machine learning model experts on a computing system, the method comprising: accessing a computing system including a plurality of accelerators; accessing a machine learning model comprising a plurality of experts distributed across the plurality of accelerators; identifying a set of input tokens to be routed to the plurality of experts; identifying routing assignments of the set of input tokens to the plurality of experts; identifying a current distribution of the plurality of experts across the plurality of accelerators; determining a new distribution of the plurality of experts across the plurality of accelerators, the new distribution resulting in increased processing efficiency of the set of input tags by the plurality of accelerators when using the routing assignments of the set of tags to the plurality of experts, compared to the current distribution of the plurality of experts across the plurality of accelerators; as well as The new distribution of the plurality of experts is applied to the plurality of accelerators.

2. The method of claim 1 , wherein applying the new distribution of the plurality of experts to the plurality of accelerators occurs before applying the machine learning model to the set of labels.

3. The method according to claim 1, further comprising: determining a particular number of iterations for processing between determining whether the new distribution of the plurality of experts is applied; as well as The particular number of iterations is performed before applying the new distribution of the plurality of experts to the plurality of accelerators.

4. The method of claim 1 , wherein determining the new distribution comprises: Overloaded accelerators are identified and underutilized accelerators are identified, and rules are used to form the new distribution to relocate experts from the accelerators determined to be overloaded to the underutilized accelerators.

5. The method of claim 4 , wherein determining a new distribution of the plurality of experts is performed in response to predicting that an overloaded accelerator will likely discard input tags in the current distribution that are directed to the overloaded accelerator and causing the expert to be relocated to be the expert currently located on the overloaded accelerator.

6. The method of claim 1 , wherein determining the new distribution comprises identifying overloaded experts and forming the new distribution using rules such that the overloaded experts are replicated; and wherein applying the new distribution of the plurality of experts to the plurality of accelerators comprises: The overloaded experts are replicated onto specific accelerators to perform parallel processing of input tokens by a set of replicated experts.

7. The method according to claim 1, further comprising: identifying one or more experts that are not to be used for processing input tokens based on the routing assignments of the set of tokens to the plurality of experts; as well as The one or more experts are temporarily removed from the one or more corresponding accelerators.

8. The method according to claim 1, further comprising: Split one or more experts into multiple shards, Wherein applying the new distribution of the plurality of experts to the plurality of accelerators comprises: relocating at least one shard from a first accelerator to a second accelerator and retaining at least one shard located on the first accelerator. 9 . The method of claim 1 , wherein at least some of the plurality of accelerators have a larger memory capacity than other accelerators, and at least some of the accelerators have a larger processing capability than other accelerators.

10. The method according to claim 9, wherein the machine learning model comprises a plurality of dense layers and a plurality of sparse layers, each sparse layer further comprising one or more experts from the plurality of experts; and The machine learning model is distributed across the plurality of accelerators such that the plurality of dense layers of the machine learning model are distributed across one or more accelerators having the greater processing capability, and the plurality of sparse layers of the machine learning model are distributed across one or more accelerators having the greater memory capability. 11 . The method of claim 1 , wherein determining the new distribution comprises using a rule such that the plurality of experts are evenly distributed across the plurality of accelerators in the new distribution.

12. A method for distributing machine learning model experts on a computing system, the method comprising: accessing a computing system including a plurality of accelerators; accessing a machine learning model comprising a plurality of experts distributed across the plurality of accelerators; identifying a set of input tokens to be routed to the plurality of experts; executing one or more computer-executable instructions configured to cause the computing system to apply the machine learning model to the set of input tokens based on the routing assignment; identifying a real-time processing imbalance of the set of input tokens based on a current distribution of the plurality of experts across the plurality of accelerators; determining a new distribution of the plurality of experts across the plurality of accelerators, the new distribution resulting in improved balanced processing of the set of input tags by the plurality of accelerators using the routing assignments of the set of tags to the plurality of experts compared to the current distribution of the plurality of experts across the plurality of accelerators; as well as The new distribution of the plurality of experts is applied to the plurality of accelerators. 13 . The method of claim 12 , wherein identifying the set of marked real-time processing imbalances comprises identifying an accelerator determined to be overloaded. The method of claim 12 , wherein identifying a real-time processing imbalance of the set of tags comprises identifying an underutilized accelerator.

15. The method of claim 12, wherein identifying a real-time processing imbalance of the set of tokens comprises determining that an accelerator has discarded input tokens.

Images