System and method for distributing layers of dedicated hybrid expert machine learning model

Abstract

Some disclosed embodiments relate to a computing system having different accelerators such that a first set of accelerators has greater memory capability than a second set of accelerators and the second set of accelerators has greater processing capability than the first set of accelerators. Machine learning models having different dense and sparse layers are distributed over different accelerators such that the dense layers are distributed over one or more accelerators selected from the first set of accelerators and the sparse layers are distributed over one or more accelerators in the second set of accelerators.

Description

Background Art

[0001] Machine learning models using mixture of experts (MOE) techniques are typically composed of N number of layers, which are broadly classified into MOE layers and non-MOE layers. Various distribution strategies are used to distribute large MOE machine learning models to computing system hardware.

[0002] When the model is distributed according to the conventional MOE distribution strategy, a single accelerator or graphics processing unit (GPU) will be assigned some or all of the layers of the model, including MOE layers and non-MOE layers. However, there are many problems associated with this allocation. For example, some components will remain idle while other components are still processing input data. In addition, due to the limitations of current hardware devices of existing computing systems, this model is not scalable. Additionally, training a MOE model with many distributed layers and experts may be computationally heavy and time-consuming.

[0003] In view of the foregoing, there is a continuing need for improved systems and methods for MOE machine learning models that can be distributed on different types of hardware configurations. The subject matter claimed herein is not limited to embodiments that solve any disadvantages or operate only in environments such as those described above. Instead, this background is provided only to illustrate an exemplary technical field in which some embodiments described herein can be practiced. Summary of the invention

[0004] The disclosed embodiments include systems, methods, and apparatus for distributing MOE models into various system configurations.

[0005] In some embodiments, the MOE model is distributed in a system including multiple accelerators, the multiple accelerators including a first accelerator set and a second accelerator set, wherein according to these embodiments, the accelerators in the second accelerator set have greater memory capabilities than the accelerators in the first accelerator set, and the accelerators in the first accelerator set have greater processing capabilities than the accelerators in the second accelerator set.

[0006] In some embodiments, the MOE model is configured with multiple dense layers and multiple sparse layers distributed on multiple system accelerators (e.g., the first accelerator set and the second accelerator set mentioned above), so that the multiple dense layers are distributed on one or more accelerators selected from the first accelerator set, and the multiple sparse layers are distributed on one or more accelerators in the second accelerator set. In some cases, the machine learning model is also distributed in this way: at least one accelerator storing one or more sparse layers of the machine learning model is configured to receive multiple inputs from one or more accelerators storing dense layers of the machine learning model.

[0007] As described herein, the disclosed embodiments also relate to systems and methods for distributing different dense layers and sparse layers of a machine learning model onto a custom computing system. For example, the system is configured to access a machine learning model comprising multiple dense layers and multiple sparse layers, and identify several dense layers and several experts included in the machine learning model. The system is also configured to access a computing system comprising multiple accelerators, at least some of which have greater memory capabilities than other accelerators and at least some of which have greater processing capabilities than other accelerators, identify which accelerators are available, and determine which of the available accelerators are configured to support dense layers and which accelerators are configured to support experts.

[0008] The disclosed system is also configured to distribute multiple dense layers to one or more available accelerators configured to support dense layers, and to distribute several experts to one or more available accelerators configured to support experts, such that the one or more accelerators configured to support experts are configured to receive multiple inputs from the one or more accelerators configured to support dense layers.

[0009] This summary is provided to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0010] Additional features and advantages will be set forth in the following description, and in part will be apparent from the description, or may be learned by practicing the teachings herein. The features and advantages of the present invention may be realized and obtained by means and combinations specifically indicated in the appended claims. The features of the present invention will become more fully apparent from the following description and the appended claims, or may be learned by practicing the present invention as set forth below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to describe the manner in which the above-recited and other advantages and features can be obtained, a more particular description of the subject matter briefly described above will be rendered by reference to specific embodiments that are illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments and are not therefore to be considered limiting of scope, the embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0012] Figures 1A to 1D Various example embodiments of existing MOE systems are shown.

[0013] Figure 2An example diagram of a MOE machine learning model distributed across computing systems in accordance with the disclosed embodiments is shown.

[0014] Figure 3 A computing environment is shown in which a computing system incorporates and / or is used to perform the disclosed aspects of the disclosed embodiments.

[0015] FIG. 4A to FIG. 4D Various example embodiments of a distributed special MOE machine learning model are shown.

[0016] Figure 5 One embodiment of a flow diagram having multiple acts for determining a distribution of a machine learning model including a MOE is shown.

[0017] Figure 6 One embodiment of a flow diagram having multiple actions for migrating one or more experts of a machine learning model to one or more different accelerators of a computing system is shown. DETAILED DESCRIPTION

[0018] The disclosed embodiments relate to systems and methods for distributing machine learning models including MOEs onto computing systems. In this regard, it will be appreciated that some of the disclosed embodiments relate specifically to improved systems and methods for determining distribution of machine learning based on separating sparse layers from dense layers on customized hardware devices. The disclosed embodiments provide numerous technical advantages over existing systems.

[0019] For example, traditional transformer-based machine learning models are constructed using a stack of transformer layers that process input data sequentially. For example, the output from the previous transformer layer is used as the input to the next transformer layer. All neurons from a typical transformer layer participate in processing each input. Transformer layers that employ all or most of the neurons in the layer are identified as dense layers, while transformer layers that employ one or a limited number of neurons in the layer are identified as sparse layers. Dense layers require a large number of floating point operations (FLOPS) and a large amount of GPU memory to process the input. Machine learning models configured with dense layers in this way are difficult to scale.

[0020] Some data scientists have begun using a variation of the traditional transformer layer, which has been called a mixture of experts (MOE) layer, as a way to scale machine learning models. In some cases, a MOE layer, which is a type of sparse layer, is constructed using a collection of experts. For example, if a model is being trained to perform a specific task, the specific task (e.g., a predictive modeling task) can be decomposed into two or more subtasks. Each expert is then trained on one of the subtasks. Although in some cases, experts are configured as models, such as neural networks with their own sets of nodes or neurons, when a set of experts within a particular machine learning model layer forms a neural network, the experts may also be referred to as nodes or neurons. Therefore, in the case of a MOE layer (i.e., a sparse layer), each input can be processed by a limited subset of experts (i.e., neurons) from the MOE layer.

[0021] This is in contrast to dense layers where all or most neurons participate in data processing, rather than a select few as is the case for sparse layers. In some existing systems, the entire machine learning model including dense and sparse layers is distributed on a single piece of hardware, which is referred to herein as an accelerator (e.g., GPU 1), such as Figure 1A As shown in . For example, as shown, GPU 1 includes multiple layers (e.g., layer N-1, layer N, and layer N+1). Layer N also includes residual connections and layer normalization layers, one or more experts on feedforward network layers (e.g., FFN1, FFN2, FFNe, etc.), gating layers, additional residual connections and layer normalization layers, and multi-head attention layers. In this way, layer N-1 and layer N+1 are dense layers, while the sparse layers within layer N include different experts.

[0022] With respect to the foregoing, it will be understood that there are many different types of accelerators, including but not limited to hardware accelerators, graphics accelerators, cryptographic accelerators, web accelerators, PHP accelerators, or other types of accelerators, all of which are collectively referred to herein as accelerators, and each of which includes one or more dense layers or sparse layers of a corresponding machine learning model.

[0023] Different layers of the MOE-based transformer machine learning model are configurable in various configurations. In some existing systems, different layers of the machine learning model are distributed across multiple accelerators (e.g., GPU 1 and GPU N), where each accelerator has a single expert in its sparse layer, such as Figure 1B For example, although each of GPU 1 and GPU N has Figure 1A, but both GPU 1 and GPU N each include only a single expert in its sparse layer (layer N) (e.g., FFN1 in GPU 1 and FFNn in GPU N).

[0024] In some configurations, dense layers and sparse layers are interleaved. For example, if a machine learning model is constructed using two dense layers (e.g., dense layer 1, dense layer 2) and two sparse layers (e.g., sparse layer 1, sparse layer 2), the machine learning model can be constructed based on Figure 1C As shown, the input data is first processed by dense layer 1, then by sparse layer 1, then by dense layer 2, and then by sparse layer 2 to generate the final output.

[0025] exist Figure 1D , each sparse layer consists of at least two experts. For example, sparse layer 1 includes a first plurality of experts (e.g., S1E1, S1E2) and sparse layer 2 includes a second plurality of experts (e.g., S2E1, S2E2). In order to accommodate such large machine learning models (i.e., the entire model will not fit on a single accelerator), the layers of the machine learning model are distributed to more than one accelerator. For example, dense layer 1, sparse layer 1, and dense layer 2 are distributed on GPU1, while dense layer 3, sparse layer 2, and dense layer 4 are distributed on GPU2. However, this distribution scheme, which is often referred to as model parallelism, is very inefficient because when GPU1 is processing input, GPU2 remains idle, and when GPU2 is processing input, GPU1 is idle.

[0026] Some work has focused on alleviating this inefficiency by introducing processing pipelining, so that GPU 1 starts processing the second input while GPU 2 is processing the first input (after the first input has been processed by GPU 1). However, this configuration still has a disadvantage in that GPU utilization remains low because any experts in one or more sparse layers that are not involved in processing a given input still occupy a large amount of GPU memory.

[0027] Additional improvements have been explored, which are called expert parallelism, which provides a model configuration in which experts are evenly distributed across GPUs. In this configuration, the system can process up to N inputs simultaneously based on N GPUs. In one example, there are four GPUs and four experts, and each GPU is assigned only a single expert from each sparse layer. In this configuration, the system can process up to four inputs simultaneously. The sparse layer will exchange inputs so that each input is sent to the GPU, where the expert that has been selected to process the input is stored. However, even this improvement still has limited capabilities. For example, in addition to the sparse layer, each GPU processes a dense layer. In some cases, because a large amount of memory is occupied by the sparse layer, which does require less processing than the dense layer, this is inefficient. This reduces the overall computational efficiency of the system. Additionally or alternatively, the model on each GPU consumes the same amount of memory and computing resources. Therefore, scalability is constrained by the GPU with the smallest computing power and memory, which leads to the inefficiency of the GPU with larger memory storage and / or computing power in the system.

[0028] Attention will be focused on Figure 2 , which shows an example embodiment of a special expert machine learning model, where sparse layers are distributed to sparse hardware and dense layers are distributed to dense hardware, and where the sparse layers are interleaved with the dense layers so that the sparse hardware can process multiple outputs from multiple dense hardware devices. The disclosed embodiments provide many technical advantages over existing systems.

[0029] For example, computing system 200 is shown having multiple accelerators (e.g., accelerator 202, accelerator 204, accelerator 206, accelerator 208, and one or more other accelerators not shown). The machine learning model is distributed to various accelerators. For example, a first plurality of model layers (e.g., layer 210, layer 211, and layer 222) are shown to be distributed to accelerator 202. Each layer also includes one or more layers (i.e., sublayers). For example, layer 211 includes layer 212 (e.g., residual connection and layer normalization), layer 214 (e.g., sparse layer) (which also includes gating layer 216), layer 218 (e.g., residual connection and layer normalization), and layer 220 (e.g., multi-head attention).

[0030] Similarly, a second plurality of model layers (e.g., layer 224, layer 215, and layer 234) distributed to accelerator 204 is shown. Each layer further includes one or more layers (i.e., sub-layers). For example, layer 215 includes layer 226 (e.g., residual connection and layer normalization), layer 224 (e.g., sparse layer) (which also includes gating layer 228), layer 230 (e.g., residual connection and layer normalization), and layer 232 (e.g., multi-head attention).

[0031] As described above, in a conventional system, a sparse layer would include one or more experts. However, in the illustrated embodiment, which is representative of some of the embodiments disclosed herein, experts from different sparse layers have been distributed to separate accelerators. For example, experts 236 and 238 from the sparse layer associated with layer 210 and experts 240 and 242 from layer 214 associated with layer 211 are distributed to accelerator 206. Additionally, experts 244, 246, 248, and 250 are distributed to accelerator 208. As shown, the experts are configured to receive different inputs from various layers of a machine learning model such as those distributed across multiple accelerators. This prevents certain components from being idle while other components are actively processing data.

[0032] In this way, the MOE layer and the non-MOE layer are separated into separate groups, and each group is assigned its own set of accelerators. By distributing machine learning models according to this configuration, the computing system is able to achieve the following technical advantages. First, the system is able to apply specific performance optimizations suitable for dense and sparse computations in a selective manner. Additionally, the MOE layer incorporates heavy communication overhead. Therefore, compared with Figure 2 A configuration similar to the one shown in enables the system to take advantage of higher communication bandwidth available within a subset of the cluster (eg, on a single node or on nodes in a single rack).

[0033] Furthermore, in accordance with the disclosed embodiments, a single set of MOE layers may be scheduled to process multiple interleaved inputs to effectively increase model throughput (either for training, inference, and / or implementation).

[0034] The disclosed embodiments also relate to systems and methods that can use heterogeneous clusters to reduce computing costs and improve the performance of computing systems / machine learning models. Specifically, this configuration helps to significantly reduce the time required for machine learning training, thereby allowing users to deploy machine learning models to implement tasks more quickly.

[0035] Now focus on Figure 3 , which illustrates components of a computing system 310 that may include and / or be used to implement various aspects of the disclosed invention. As shown, the computing system includes multiple machine learning (ML) engines, models, and data types associated with inputs and outputs of the machine learning engines and models. For example, Figure 3A computing system 310 is shown as part of a computing environment 300 that also includes remote / third-party system(s) 320 in communication with the computing system 310 (via a network 330). The computing system communicates with the remote / third-party system(s) 320 that include one or more processors 322 and one or more computer-executable instructions 324. It is contemplated that in some cases, the remote / third-party system(s) 320 also includes a database containing data that can be used as training data, such as external speaker data. Additionally or alternatively, the remote / third-party system(s) 320 include a machine learning system external to the computing system 310. In some embodiments, the remote / third-party system(s) 320 are software programs or applications.

[0036] For example, computing system 310 includes one or more processors 312 (e.g., one or more hardware processors) and storage (i.e., hardware storage device(s) 340) storing computer-executable instructions 318. One or more of hardware storage devices (multiple) 340 can accommodate any number of data types and any number of computer-executable instructions 318, and when computer-executable instructions 318 are executed by one or more processors 312, computing system 310 is configured by the computer-executable instructions 318 to implement one or more aspects of the disclosed embodiments.

[0037] The computing system 310 also includes multiple accelerators (e.g., a dense accelerator 313 and a sparse accelerator 314). In some configurations, the dense accelerator 313 is configured to process input data using dense layers, wherein the dense accelerator 313 is custom hardware optimized for processing power. In this configuration, the sparse accelerator 314 is configured to process input data using sparse layers, wherein the sparse accelerator 314 is custom hardware optimized for memory storage. The sparse accelerator 314 is more efficient than the dense accelerator in processing sparse data (e.g., sparse tensors, sparse layers). Each of the accelerators may include a dedicated processor or other hardware capable of storing and / or executing corresponding dense layers and sparse layers (344 and 346, respectively).

[0038] In some cases, the sparse accelerator has at least 10% greater memory or storage capacity than the dense accelerator, or even more (e.g., 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, or even more than 100% greater memory storage capacity than the dense accelerator). Additionally or alternatively, the sparse accelerator is at least 10% more efficient than the dense accelerator in processing sparse data, or even more (e.g., 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, or even more than 100% more efficient than the dense accelerator in processing sparse data).

[0039] In some cases, a dense accelerator has at least 10% greater processing power than a sparse accelerator, or even more (e.g., 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, or even 100% greater processing power than a sparse accelerator). For example, a dense accelerator is more efficient at processing dense data (e.g., dense layers, dense tensors) than a sparse accelerator.

[0040] In some cases, sparse accelerators are distinguished from dense accelerators based at least on their higher efficiency in processing sparse data. Additionally or alternatively, sparse accelerators are distinguished from dense accelerators based on their increased memory capacity and / or reduced number of raw FLOPs compared to dense accelerators.

[0041] Computing system 310 is also shown to include user interface(s) 315 and input / output (I / O) device(s) 316 .

[0042] like Figure 3 , hardware storage device(s) 340 are shown as a single storage unit. However, it will be understood that in some embodiments, hardware storage device(s) 340 are distributed storage devices that are distributed to several separate and sometimes (multiple) remote and / or third-party systems 320 (e.g., hardware storage device 324). In some embodiments, computing system 310 may also include a distributed system in which one or more of the components of computing system 310 are maintained / operated by different discrete systems that are remote from each other and each performs different tasks. In some cases, such as in a distributed cloud environment, multiple distributed systems perform similar and / or shared tasks for implementing the disclosed functionality.

[0043] In this way, different layers of the machine learning model may be distributed across computing system 310 and / or across a distributed computing system 300 including computing system 310 and one or more third-party systems 320.

[0044] The hardware storage device(s) 340 are configured to store different data (e.g., input data 348) including various models such as a machine learning model 342 that includes dense layers 344 and sparse layers 346 used to process the input data 348.

[0045] The storage device (e.g., hardware storage device(s) 340) includes computer executable instructions 318 for instantiating or executing one or more of the models and / or engines shown in the computing system 310. The model is configured as a machine learning model or a machine learned model, such as a deep learning model and / or algorithm. In some cases, one or more models are configured as an engine or processing system (e.g., a computing system integrated within the computing system 310), where each engine (i.e., model) includes one or more processors (e.g., hardware processor(s) 312) and computer executable instructions 318 corresponding to the computing system 310.

[0046] Additional storage units for storing (multiple) machine learning (ML) engines 350 are currently in Figure 3 310 is shown as storing multiple machine learning models and / or engines. For example, computing system 310 includes one or more of the following: data retrieval engine 351, distribution engine 352, and implementation engine 353, which are configured individually and / or collectively to implement the different functions described herein.

[0047] For example, the data retrieval engine 351 is configured to locate and access data sources, databases, and / or storage devices including one or more data types, from which the data retrieval engine 351 can extract sets or subsets used as input data. The data retrieval engine 351 receives data from a database and / or hardware storage device, wherein the data retrieval engine 351 is configured to reformat or otherwise expand the received data to be used as training data or input data. Additionally or alternatively, the data retrieval engine 351 communicates with one or more remote / third-party systems (e.g., (multiple) remote / third-party systems 320) including remote / third-party data sets and / or data sources. In some cases, these data sources include audiovisual services that record voice, text, images, and / or video.

[0048] The data retrieval engine 351 accesses electronic content including sound data, text data, and / or other types of audiovisual data (including video data, image data, holographic data, 3-D image data, etc.). The data retrieval engine 351 is an intelligent engine that is capable of learning the optimal data set extraction process to provide a sufficient amount of data in a timely manner and retrieve the data most suitable for the desired application, for which the machine learning model / engine will be used. For example, the data retrieval engine 351 can learn which databases and / or data sets will generate training data that will train the model (e.g., for a specific query or a specific task) to improve the accuracy, efficiency, and efficacy of the model in the desired layer distribution configuration.

[0049] In some embodiments, computing system 310 includes a distribution engine 352 configured to determine the distribution of different layers of machine learning model 342 across different accelerators. Distribution engine 352 is also configured to facilitate distribution before instantiation of the model or redistribution of the machine learning model after instantiation of the model. In some cases, the redistribution is based on identifying potential improvements in one or more attributes of the computing system (e.g., model throughput, computational efficiency) and / or based on scaling up or down of the machine learning model. In some embodiments, the redistribution is achieved by migrating one or more experts from one accelerator to a different accelerator.

[0050] The distribution engine 352 is configured to dynamically identify the total number of accelerators that make up the computing system 310, and to identify which accelerators are specialized or optimized hardware devices for dense layers compared to sparse layers. Additionally, the distribution engine 352 is configured to identify which accelerators are full and / or which accelerators have available or expected processing power and / or memory space for storing and executing one or more additional layers.

[0051] The distribution engine 352 is also configured to identify how many layers make up the machine learning model, and which layers are dense layers and which layers are sparse layers. The distribution engine 352 is also configured to identify how many experts there are in each layer in the different sparse layers, and to identify the attributes of the experts (e.g., what specialized tasks are associated with the experts).

[0052] Thus, based on the number of dense layers and the number of sparse layers (or the number of experts across one or more sparse layers), distribution engine 352 is configured to dynamically and automatically distribute the different layers to one or more accelerators of computing system 310. In some cases, distribution engine 352 is configured to distribute dense layers 344 to dense accelerators 313 (e.g., accelerators determined to have the capability to store and / or execute dense layers) and to distribute sparse layers 346 to sparse accelerators 314 (e.g., accelerators having more limited capacity and capabilities than dense accelerators).

[0053] In some embodiments, computing system 310 includes an implementation engine 353 in communication with any of the models and / or ML engines (or all of the models / engines) 350 included in computing system 310, such that implementation engine 353 is configured to implement, launch, or run one or more functions of multiple ML engines 350. In one example, implementation engine 353 is configured to operate data retrieval engine 351, such that data retrieval engine 351 retrieves data at an appropriate time to enable processing of input data 348. Implementation engine 353 facilitates process communication and communication timing between one or more of ML engines (multiple) ML engine 350.

[0054] In other examples, implementation engine 353 is configured to implement one or more functions of machine learning model 342 (i.e., processing input data 348) to be distributed onto computing system 310 or across computing system 310 and third-party system(s) 320. Implementation engine 353 is also configured to implement distribution engine 352 in order to identify distribution or redistribution of different layers of the machine learning model.

[0055] In addition, the implementation engine 353 is configured to select which experts distributed across the various accelerators will be used in processing the input data 348. By implementing systems and methods according to these disclosed embodiments, the computing system 310 is able to achieve technical benefits such as customizability and scalability. Specifically, different experts can be used at different times in processing the input data 348. Thus, the system can be configured to select a limited number of experts to use in processing the input data based on the type of input data, the formatting of the input data, the context of the input data, and / or the downstream application of the processed input data. Thus, when one or more experts are identified as experts to be used to process the input data, the system can distribute or redistribute one or more sparse layers including those identified experts to different accelerators in order to increase model throughput and increase the computational efficiency of the system.

[0056] Now focus on FIG. 4A to FIG. 4D , which illustrates various embodiments of machine learning models (e.g., machine learning model 342) distributed to a computing system (e.g., computing system 310). First, attention is focused on Figure 4A , which illustrates an example embodiment of a novel configuration. As shown, computing system 400 includes multiple accelerators (e.g., accelerator 402, accelerator 404, accelerator 406, accelerator 408, and accelerator 410). In some cases, such as Figure 4A As shown in , the accelerators are configured as GPUs (e.g., GPU1, GPU2, GPU3, GPU4, and GPU5). Accelerators 402 and accelerators 410 are configured to process inputs using dense layers. For example, multiple dense layers (e.g., D1, D2, D3, and D4) are distributed to accelerators 402 and accelerators 410. Accelerators 404, accelerators 406, and accelerators 408 are configured to process inputs from previous layers using sparse layers.

[0057] For example, a first sparse layer including a first plurality of experts (e.g., S1E1, S1E2, S1E3, and S1E4) is distributed to accelerator 404, a second sparse layer including a second plurality of experts (e.g., S2E1, S2E2, S2E3, and S2E4) is distributed to accelerator 406, and a third sparse layer including a third plurality of experts (e.g., S3E1, S3E2, S3E3, and S3E4) is distributed to accelerator 408. In the experts described above, the notation is as follows: "S" refers to a sparse layer with the following number indicating the number of sparse layers, and "E" refers to an expert with the following number indicating the number of experts. Although each accelerator is shown as having at least four layers, it should be understood that any number of layers, including no layers, may be distributed to any of the accelerators or across multiple accelerators.

[0058] like Figure 4A As shown in FIG. 4 , accelerators 404, 406, and 408 are respectively configured to simultaneously receive multiple inputs from accelerators 402 and 410 through interleaving. In this way, multiple accelerators can process inputs simultaneously, thereby increasing the computational efficiency and processing power of the computing system.

[0059] By distributing the different layers in this manner, further technical advantages are achieved. For example, a system can be equipped with a first set of customized hardware that has been optimized for processing dense layers and a second set of customized hardware that has been optimized for processing sparse layers. This was not possible in a previous configuration where the accelerator was configured to process inputs with both dense and sparse layers.

[0060] In some cases, a hardware device optimized for dense layers is characterized by a high processing power attribute. Additionally, such a hardware device is also characterized by a smaller memory storage device. In some cases, a hardware device optimized for sparse layers is characterized by a high memory storage attribute. Additionally, such a hardware device is also characterized by a lower processing power attribute than a hardware device optimized for dense layers.

[0061] Now focus on Figure 4B , which shows a machine learning model distributed to a computing system 401 including custom hardware. For example, Figure 4B Included with Figure 4A Similar components, where accelerator 402 and accelerator 410 are configured for dense layers (ie, dense hardware), and where accelerator 404, accelerator 406, and accelerator 408 are configured for sparse layers. By customizing the different hardware components of computing system 400, the computing efficiency and processing power of the computing system are increased.

[0062] In some cases, efficiency is characterized by how long a layer is used in processing compared to the memory it occupies on a particular accelerator. Thus, an efficient system is one in which layers that do little processing take up a smaller amount of memory than layers that participate in a large amount of processing that take up a larger amount of memory. Alternatively, layers that do little processing but require a large amount of memory (e.g., sparse layers) can be distributed onto dedicated hardware (e.g., dense hardware) that is optimized for memory storage that exceeds processing capacity, while layers (e.g., dense layers) are distributed onto dedicated hardware (e.g., sparse hardware) that is optimized for processing capacity that exceeds storage capacity.

[0063] The disclosed embodiments advantageously achieve additional technical benefits, including high scalability. Now focusing on Figure 4C , which shows a computing system 403, which is representative of computing system 401 that also includes an additional accelerator (e.g., accelerator 412) configured as a sparse hardware device. A fourth sparse layer including a fourth plurality of experts is distributed to accelerator 412. As shown, the fourth plurality of experts includes two experts (e.g., S4E1 and S3E4). It should be understood that any plurality of experts on any accelerator in different accelerators may include one or more experts.

[0064] Because the layers of a machine learning model are distributed in a selective manner (i.e., dense layers on dense hardware and sparse layers on sparse hardware), the machine learning model is highly scalable, such as Figure 4C . For example, to increase the throughput of the system, additional accelerators (e.g., GPU 6) can be added in smaller increments. In this way, smaller or larger accelerators can be added to the system to accommodate one or more additional sparse layers including one or more additional experts. This also provides additional flexibility and customizability in how different layers of the machine learning model are distributed across the computing system.

[0065] Additionally or alternatively, one or more experts from another accelerator may be migrated to the additional accelerator. Figure 4D As shown in FIG. 4 , a computing system 405 representing a computing system 403 having a different distribution similar to hardware devices other than a machine learning model layer is shown as having initially distributed to an accelerator 408 (see FIG. Figure 4C) that are now distributed to accelerator 412 (e.g., S3E3 and S3E4). In the event that additional accelerators with available processing and memory storage capabilities are identified, the system can redistribute (i.e., migrate) one or more experts to the additional accelerators. For example, if the system selects expert S3E1 and expert S3E3 as the experts to be used to process new input data, the system determines that the redistribution of layers will increase model throughput and promote improvements in computational processing.

[0066] In contrast, if S3E1 and S3E3 had been kept on the same sparse hardware (e.g., accelerator 408), the input data would have to be processed serially, first using S3E1 and then using S3E3. However, now through redistribution, the input data can be processed in parallel using both accelerator 408 and accelerator 412. In this way, latency is reduced (one layer does not wait for another layer to complete processing) and model throughput is increased.

[0067] It will be appreciated that redistribution and migration of experts may occur during downtime and / or during processing being performed by the model, based on a dynamic assessment of capabilities and constraints associated with executing the model on different data sets.

[0068] Because each expert is trained on a specific task or data application, the machine learning model is able to process input data using a variety of different training tasks or applications. This special MOE configuration allows the machine learning model to easily and efficiently incorporate new or updated experts without having to retrain the entire model and / or reconfigure other layers distributed across the computing system.

[0069] Additionally, in some embodiments, new experts are added to available accelerators or to new accelerators that are easily integrated into the system. This is an efficient configuration because the hardware is not idle (i.e., multiple hardware devices are processing multiple inputs simultaneously). This is a significant improvement over traditional hybrid expert systems, which are not easily adaptable to new tasks, scalable, and have limited throughput because some hardware devices are idle while other hardware devices are processing inputs.

[0070] Now focus on Figure 5 , which shows a flowchart 500 including various actions (action 510, action 520, action 530, action 540, action 550, action 560, and action 570) associated with an exemplary method that can be implemented by a computing system 310 for distributing a hybrid expert machine learning model to a customized computing system.

[0071] The first action shown includes an action for accessing a machine learning model including a plurality of dense layers and a plurality of sparse layers (action 510). Because the model has dense layers and sparse layers, the machine learning model is able to implement a plurality of different experts within the sparse layers, which provides additional flexibility and customizability of machine learning for different training tasks, domains, and applications. Additionally, the plurality of sparse layers are interleaved with the plurality of dense layers.

[0072] Then, the system identifies a number of dense layers and a number of experts included in the machine learning model (act 520). Additionally, the system accesses a computing system including a plurality of accelerators, wherein at least some of the accelerators have greater memory capabilities than other accelerators, and at least some of the accelerators have greater processing capabilities than other accelerators (act 530). By identifying a specific number of elements in each of the computing system and the machine learning model, the system is able to generate an accurate and efficient distribution strategy. Furthermore, the accelerator is a hardware accelerator. For example, one or more hardware accelerators may be configured as a graphics unit processor.

[0073] The system then identifies which accelerators are available (act 540) and determines which of the available accelerators are configured to support dense layers and which are configured to support experts (act 550).

[0074] After accessing the machine learning model and the computing system, the system is configured to distribute the plurality of dense layers to one or more available accelerators configured to support dense layers (act 560), and to distribute the plurality of experts to one or more available accelerators configured to support experts, such that the one or more accelerators configured to support experts are configured to receive a plurality of inputs from the one or more accelerators configured to support dense layers (act 570).

[0075] By generating a distribution strategy in this manner, efficiency is improved for the system and model combination. This is because layers that perform little processing take up a smaller amount of memory than layers that participate in a larger amount of processing that takes up a larger amount of memory. Alternatively, layers that perform little processing (e.g., sparse layers) but require a large amount of memory can be distributed onto specialized hardware (e.g., dense hardware) that is optimized for memory storage that exceeds processing capacity, while layers (e.g., dense layers) are distributed onto specialized hardware (e.g., sparse hardware) that is optimized for processing capacity that exceeds memory capacity.

[0076] It should be understood that each sparse layer included in the multiple sparse layers also includes one or more experts, so that the machine learning model is configured as a MOE machine learning model.

[0077] In some embodiments, the method also includes an act of accessing an additional accelerator configured to support the one or more experts and an act of integrating the additional accelerator into the computing system such that the computing system is an extended computing system.

[0078] In such an embodiment, the system is able to access one or more additional experts associated with the machine learning model and distribute the one or more additional experts to additional accelerators.

[0079] Additionally or alternatively, the system is configured to select a limited number of experts distributed across one or more accelerators such that the limited number of experts being selected to process the input data access the set of input data; and process the input data using at least the limited number of experts.

[0080] Now focus on Figure 6 , which shows a flowchart 600 including various actions (action 610, action 620, action 630, action 640, action 650, action 660, and action 670) associated with an exemplary method that can be implemented by the computing system 310 to determine the redistribution of one or more layers of the MOE machine learning model.

[0081] The first act shown includes an act of accessing a computing system that includes a plurality of accelerators, wherein at least some of the accelerators have greater memory capabilities than other accelerators and at least some of the accelerators have greater processing capabilities than other accelerators (act 610).

[0082] The method also includes an action of accessing a machine learning model including a plurality of dense layers and a plurality of sparse layers, each sparse layer also including one or more experts, the machine learning model being distributed on one or more accelerators included in the plurality of accelerators, such that the plurality of dense layers of the machine learning model are distributed on one or more accelerators having greater processing power, and the plurality of sparse layers of the machine learning model are distributed on one or more accelerators having greater memory capacity (action 620). The one or more accelerators having greater processing power are configured as dense accelerators that are optimized to process dense layers. The one or more accelerators having greater memory capacity are configured as sparse accelerators that are optimized to process sparse layers. Thus, the dense accelerator has greater processing power than the sparse accelerator, and the sparse accelerator has greater memory capacity than the dense accelerator. This is because the sparse layers require more memory than the dense layers, but require less computational density, while the dense layers require less memory than the sparse layers, but require greater computational density.

[0083] By distributing the different layers in this manner, further technical advantages are achieved. For example, a system can be equipped with a first set of customized hardware that has been optimized for processing dense layers and a second set of customized hardware that has been optimized for processing sparse layers. This was not possible in previous configurations where the accelerator was configured to process inputs with both dense and sparse layers.

[0084] After accessing the computing system and the machine learning model, the system is configured to execute one or more computer-executable instructions configured to cause the computing system to apply the machine learning model to the input data (act 630).

[0085] After executing the one or more computer-executable instructions, the system then identifies a number of available accelerators included in the plurality of accelerators (act 640), and determines a processing efficiency of the computing system based on a first distribution of the machine learning model on the one or more accelerators (act 650).

[0086] Based at least on the processing efficiency, the system identifies a new distribution of the machine learning model across the plurality of accelerators, the new distribution being configured to increase the processing efficiency of the computing system (act 660). Finally, the system migrates at least one of the one or more experts to at least one of the plurality of available accelerators having greater memory capabilities to facilitate an improvement in the processing efficiency of the computing system (act 670). By implementing the systems and methods in this manner, the machine learning model can be optimized based on increasing model throughput and facilitating improvements in hardware distribution efficiency.

[0087] In addition, such disclosed embodiments provide the ability to scale the machine learning model to a larger or smaller number of layers, including integrating one or more different experts as part of the MOE or removing the expert as part of the MOE. This facilitates reducing the amount of computation and time used for training and for processing new input data. For example, in some cases, the method also includes the actions of selecting a limited number of experts distributed across one or more accelerators, the limited number of experts being selected to process the input data, access the input data set; and using at least the limited number of experts to process the input data.

[0088] Additionally, some methods also include an action of determining an additional distribution of the limited number of experts across the plurality of accelerators based on the selected limited number of experts. In this way, the system can migrate one or more of the limited number of experts to different accelerators according to the additional distribution.

[0089] Example computing system

[0090] Embodiments of the present invention may include or utilize a special-purpose or general-purpose computer (e.g., computing system 310) including computer hardware, as discussed in more detail below. Embodiments within the scope of the present invention also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media that can be accessed by a general-purpose or special-purpose computer system.

[0091] A computer-readable medium (eg, a computer-executable medium) storing computer-executable instructions (eg, computer-executable instructions 318 of FIG. 1 ) Figure 3 The hardware storage device(s) 340) are physical hardware storage media / devices excluding transmission media. Computer-readable media that carry computer-executable instructions or computer-readable instructions (e.g., computer-executable instructions 318) in one or more carrier waves or signals are transmission media. Thus, by way of example and not limitation, embodiments of the present invention may include at least two distinct types of computer-readable media: physical computer-readable storage media / devices and transmission computer-readable media.

[0092] Physical computer-readable storage media / devices are hardware and include RAM, ROM, EEPROM, CD-ROM or other optical disk storage (e.g., CD, DVD, etc.), magnetic disk storage or other magnetic storage devices, or any other hardware that can be used to store desired program code means in the form of computer-executable instructions or data structures and can be accessed by a general or special purpose computer.

[0093] "Network" (e.g. Figure 3 A network 330) is defined as one or more data links that enable the transmission of electronic data between computer systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computer via a network or another communication connection (or hardwired, wireless, or a combination of hardwired or wireless), the computer appropriately regards the connection as a transmission medium. The transmission medium may include a network and / or a data link (which may be used to carry), or a desired program code device in the form of a computer executable instruction or data structure, and which may be accessed by a general or special purpose computer. Combinations of the above are also included within the scope of computer readable media.

[0094] In addition, upon reaching various computer system components, program code means in the form of computer executable instructions or data structures may be automatically transferred from a transmission computer readable medium to a physical computer readable storage medium (or vice versa). For example, computer executable instructions or data structures received over a network or data link may be cached in RAM within a network interface module (e.g., a "NIC"), and then ultimately transferred to computer system RAM and / or a less volatile computer readable physical storage medium at the computer system. Thus, a computer readable physical storage medium may be included in a computer system component that also (or even primarily) utilizes a transmission medium.

[0095] For example, computer executable instructions include instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a specific function or group of functions. For example, computer executable instructions can be binary code, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in a language dedicated to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. Instead, the described features and actions are disclosed as example forms of implementing the claims.

[0096] It will be appreciated by those skilled in the art that the present invention can be practiced in a network computing environment with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, pagers, routers, switches, etc. The present invention can also be practiced in a distributed system environment, where local and remote computer systems that are linked (or by a hardwired data link, a wireless data link, or a combination of a hardwired and wireless data link) through a network all perform tasks. In a distributed system environment, program modules can be located in local and remote memory storage devices.

[0097] Alternatively or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. Illustrative types of hardware logic components that may be used include, for example but not limited to, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), etc.

[0098] The present invention may be implemented in other specific forms without departing from the basic characteristics of the present invention. The described embodiments will be considered in all respects as illustrative only and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than by the preceding description. All changes within the meaning and scope of the equivalents of the claims are included within their scope.

Claims

1. A computing system comprising: one or more hardware processors; one or more hardware storage devices configured to store computer executable instructions executable by the one or more hardware processors for managing distribution of sparse layers and dense layers of a mixture of experts (MOE) model across a plurality of accelerators; a plurality of accelerators, the plurality of accelerators comprising a first set of accelerators and a second set of accelerators, the accelerators in the second set of accelerators having a greater memory capacity than the accelerators in the first set of accelerators, and the accelerators in the first set of accelerators having a greater processing capability than the accelerators in the second set of accelerators; as well as A machine learning model comprising a plurality of dense layers and a plurality of sparse layers, wherein the machine learning model is distributed across the plurality of accelerators such that (i) the plurality of dense layers are distributed across one or more accelerators selected from the first set of accelerators, (ii) the plurality of sparse layers are distributed across one or more accelerators selected from the second set of accelerators, and (iii) at least one accelerator storing the one or more sparse layers of the machine learning model is configured to receive a plurality of inputs from the one or more accelerators storing the dense layers of the machine learning model.

2. A computing system according to claim 1, wherein each sparse layer included in the multiple sparse layers also includes one or more experts, so that the machine learning model is configured as a hybrid expert machine learning model.

3. The computing system of claim 1, wherein each expert is trained on a unique task. The computing system of claim 1 , wherein the one or more accelerators are hardware accelerators. 5 . The computing system of claim 4 , wherein the one or more accelerators are configured as a graphics unit processor. 6 . The computing system of claim 1 , wherein the first set of accelerators is configured as dense accelerators having a capability for processing dense layers that is at least 10% greater than the second set of accelerators configured as sparse accelerators. 7 . The computing system of claim 1 , wherein the second set of accelerators are configured as sparse accelerators that are at least 10% more efficient at processing sparse data than the first set of accelerators that are configured as dense accelerators.

8. The computing system of claim 1, wherein the plurality of sparse layers are interleaved with the plurality of dense layers.

9. A method for managing the distribution of a machine learning model comprising a mixture of experts (MOE) on a computing system, the method comprising: Access machine learning models that include multiple dense layers and multiple sparse layers; identifying a number of dense layers and a number of experts included in the machine learning model; accessing a computing system including a plurality of accelerators, wherein at least some of the accelerators have a greater memory capacity than other accelerators and at least some of the accelerators have a greater processing capability than other accelerators; Identify which accelerators are available; determining which of the available accelerators are configured to support dense layers and which accelerators are configured to support experts; Distributing the plurality of dense layers to one or more available accelerators configured to support dense layers; as well as The number of experts are distributed to one or more available accelerators configured to support experts such that the one or more accelerators configured to support experts are configured to receive a plurality of inputs from the one or more accelerators configured to support dense layers.

10. The method according to claim 9, wherein each sparse layer included in the multiple sparse layers further includes one or more experts, so that the machine learning model is configured as a hybrid expert machine learning model.

11. The method according to claim 9, further comprising: access to additional accelerators configured to support one or more experts; as well as The additional accelerator is integrated into the computing system such that the computing system is an extended computing system.

12. The method according to claim 11, further comprising: accessing one or more additional experts associated with the machine learning model; as well as The one or more additional experts are distributed to the additional accelerators.

13. The method according to claim 9, further comprising: selecting a finite number of experts distributed across one or more accelerators, the finite number of experts selected to process the input data; Access input data sets; as well as The input data is processed using at least the limited number of experts. The method of claim 9 , wherein the plurality of sparse layers are interleaved with the plurality of dense layers.

15. The method of claim 9, wherein the plurality of accelerators are hardware accelerators.

Images