System and method for distributing layers of a special hybrid expert machine learning model

Abstract

Some of the disclosed embodiments are directed to computing systems having various accelerators, whereby a first set of accelerators has a higher memory capacity than a second set of accelerators, and the second set of accelerators has a higher processing capacity than the first set of accelerators. A machine learning model having various dense and sparse layers is distributed across the various accelerators, whereby the dense layers are distributed across one or more accelerators selected from the first set of accelerators, and the sparse layers are distributed across one or more accelerators within the second set of accelerators.

Description

Background Art

[0001] Machine learning models that use the Mixture of Experts (MOE) technique typically consist of N layers that are generally classified into an MOE layer and non-MOE layers. To distribute a large-scale MOE machine learning model within computing system hardware, various distribution strategies are used.

[0002] When a model is distributed according to conventional MOE distribution strategies, a single accelerator, or Graphics Processing Unit (GPU), is assigned a portion or all of the layers of the model, including the MOE layer as well as non-MOE layers. Further, there are many problems associated with such distribution. For example, one component remains idle while other components are still processing input data. Further, due to the limitations of current hardware devices in existing computing systems, such models are not scalable. Further, training an MOE model with many distributed layers and experts can be computationally heavy and time-consuming.

[0003] In view of the above, there continues to be a need for improved systems and methods for MOE machine learning models that can be distributed across various types of hardware configurations. The subject matter claimed herein solves any disadvantages and is not limited to embodiments that operate only in the environments described above. Rather, this background is provided only to show one exemplary technical area in which some of the embodiments described herein may be implemented.

Summary of the Invention

[0004] The disclosed embodiments include systems, methods, and devices for distributing an MOE model within various system configurations. In some embodiments, the MOE model is distributed within a system comprising a plurality of accelerators including a first set of accelerators and a second set of accelerators. According to such embodiments, the accelerators within the second set of accelerators have a higher memory capacity than the accelerators within the first set of accelerators, and the accelerators within the first set of accelerators have a higher processing capacity than the accelerators within the second set of accelerators.

[0005] In some embodiments, the MOE model is composed of a plurality of dense layers and a plurality of sparse layers that are distributed across a plurality of system accelerators, such as the aforementioned first and second sets of accelerators. Thereby, the plurality of dense layers are distributed across one or more accelerators selected from the first set of accelerators, and the plurality of sparse layers are distributed across one or more accelerators within the second set of accelerators. In some examples, the machine learning model is further distributed such that at least one accelerator storing one or more sparse layers of the machine learning model is configured to receive a plurality of inputs from one or more accelerators storing the dense layers of the machine learning model.

[0006] As described herein, the disclosed embodiments also relate to systems and methods for distributing various dense and sparse layers of a machine learning model across a customized computing system. For example, the system is configured to access a machine learning model that includes a plurality of dense layers and a plurality of sparse layers, and to identify some of the dense layers and some of the experts included within the machine learning model. The system is also configured to access a computing system comprising a plurality of accelerators, wherein at least some of the accelerators have a higher memory capacity than other accelerators, and at least some of the accelerators have a higher processing capacity than other accelerators, to identify which accelerators are available, and to determine which of the available accelerators are configured to support dense layers and which are configured to support experts.

[0007] The disclosed system is also configured to distribute a plurality of dense layers across one or more available accelerators configured to support dense layers, and to distribute some of the experts across one or more available accelerators configured to support experts, such that one or more accelerators configured to support experts are configured to receive a plurality of inputs from one or more accelerators configured to support dense layers.

[0008] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0009] Additional features and advantages will be described in the following description, will be apparent in part from the description, or may be learned by practice of the teachings herein. The features and advantages of the invention may be realized and obtained by means of the instrumentalities and combinations particularly pointed out in the appended claims. The features of the invention will be more fully apparent from the following description and appended claims, and may be learned by practice of the invention as set forth hereinafter.

[0010] To explain the manner in which the foregoing and other advantages and features can be obtained, a more particular description of the subject briefly described above will be made by reference to the specific embodiments shown in the accompanying drawings. It is to be understood that these drawings illustrate only typical embodiments and are not to be considered limiting of the scope, and that the embodiments will be described in greater specificity and clarity by using the accompanying drawings.

Brief Description of the Drawings

[0011]

FIG. 1A

FIG. 1B

FIG. 1C

FIG. 1D

FIG. 2

FIG. 3

FIG. 4A

FIG. 4B

FIG. 4C

FIG. 4D

FIG. 5

FIG. 6

[0012] The disclosed embodiments are directed to systems and methods for distributing a machine learning model including MOE on a computing system. In this regard, some of the disclosed embodiments are specifically directed to improved systems and methods for determining the distribution of machine learning based on separating a dense layer from a sparse layer on a customized hardware device. The disclosed embodiments provide many technical advantages over existing systems.

[0013] For example, conventional transformer-based machine learning models are constructed using a stack of transformer layers that process input data sequentially. For example, the output from a previous transformer layer is used as the input to the next transformer layer. All neurons from a typical transformer layer are involved in processing each input. A transformer layer that utilizes all or most of the neurons within the layer is identified as a dense layer, and a transformer layer that utilizes one or a limited number of the neurons within the layer is identified as a sparse layer. Dense layers require a large number of floating-point operations (FLOPS) and a large amount of GPU memory to process the input. A machine learning model configured with dense layers in this way is difficult to scale.

[0014] Some data scientists began using a variant of the conventional transformer layer, known as the mixture of experts (MOE) layer, as a way to scale machine learning models. In some examples, the MOE layer, which is a type of sparse layer, is constructed using a set of experts. For example, if a model is trained to perform a specific task, that specific task (e.g., a prediction modeling task) can be decomposed into two or more subtasks. Each expert is then trained with respect to one of the subtasks. In some examples, an expert is configured as a model such as a neural network that has its own set of nodes or neurons, but when the set of experts within a particular machine learning model layer forms a neural network, an expert may also be referred to as a node or a neuron. Thus, in the case of an MOE layer (i.e., a sparse layer), each input can be processed by a limited subset of the experts (i.e., neurons) from the MOE layer.

[0015] This is in contrast to a dense layer where all or most neurons are involved in data processing, rather than a few carefully selected neurons as in the case of a sparse layer. In some existing systems, the entire machine learning model, including dense and sparse layers, is distributed over a single hardware, herein called an accelerator (e.g., GPU1) as shown in FIG. 1A. For example, as shown, GPU1 includes multiple layers (e.g., layer N-1, layer N, and layer N+1). Layer N further includes one or more experts, a gating layer, additional Add & Norm layers, and a multi-head attention layer on an Add & Norm layer, a feed-forward network layer (e.g., FFN1, FFN2, FFNe, etc.). In this way, layer N-1 and layer N+1 are dense layers, and the sparse layer within layer N includes different experts.

[0016] In connection with the above, without limitation, there are many different types of accelerators, including hardware accelerators, graphics accelerators, cryptographic accelerators, web accelerators, PHP accelerators, or other types of accelerators, all of which are collectively called accelerators herein, and each of which is understood to include one or more dense or sparse layers of a corresponding machine learning model.

[0017] The various layers of an MOE-based transformer machine learning model can be configured in various configurations. In some existing systems, as shown in FIG. 1B, the various layers of the machine learning model are distributed over multiple accelerators (e.g., GPU1 and GPU N), and each accelerator has a single expert within its sparse layer. For example, each of GPU1 and GPU N has a layer configuration similar to GPU1 in FIG. 1A, but both GPU1 and GPU N include only a single expert (e.g., FFN1 in GPU1 and FFNn in GPU N) within layer N, which is its sparse layer.

[0018] In some configurations, a dense layer and a sparse layer are interleaved. For example, if a machine learning model is constructed using two dense layers (e.g., Dense Layer 1, Dense Layer 2) and two sparse layers (e.g., Sparse Layer 1, Sparse Layer 2), the machine learning model can be configured according to FIG. 1C. As shown, to generate the final output, the input data is first processed by Dense Layer 1, then by Sparse Layer 1, then by Dense Layer 2, and then by Sparse Layer 2.

[0019] In FIG. 1D, each sparse layer is composed of at least two experts. For example, Sparse Layer 1 includes a first plurality of experts (e.g., S1E1, S1E2), and Sparse Layer 2 includes a second plurality of experts (e.g., S2E1, S2E2). To accommodate such a large-scale machine learning model (i.e., the entire model does not fit on a single accelerator), the layers of the machine learning model are distributed across multiple accelerators. For example, Dense Layer 1, Sparse Layer 1, and Dense Layer 2 are distributed on GPU1, and Dense Layer 3, Sparse Layer 2, and Dense Layer 4 are distributed on GPU2. However, this distribution method, often called Model Parallelism, is very inefficient. This is because while GPU1 is processing an input, GPU2 remains idle, and while GPU2 is processing an input, GPU1 is idle.

[0020] One study has focused on this inefficiency by introducing a processing pipeline. Thus, when GPU2 is processing the first input (after the first input has been processed by GPU1), GPU1 starts processing the second input. However, this configuration still has drawbacks in that the GPU utilization remains low because any expert within one or more sparse layers not involved in processing a given input still occupies significant GPU memory.

[0021] An additional improvement called Expert Parallelism was explored to achieve a model configuration when experts are evenly distributed across GPUs. In such a configuration, the system can process up to N inputs simultaneously, based on N GPUs. As an example, with 4 GPUs and 4 experts, each GPU is allocated for only a single expert from each sparse layer. In this configuration, the system can process up to 4 inputs simultaneously. The sparse layer exchanges the inputs so that each input is sent to the GPU that stores the expert selected to process it. However, even with this improvement, the capabilities are still limited. For example, each GPU processes the dense layer in addition to the sparse layer. In some cases, this is inefficient because the sparse layer, which requires less processing than the dense layer, occupies a large amount of memory, reducing the overall computational efficiency of the system. Additionally or alternatively, the models on each GPU consume the same amount of memory and computational resources. Thus, scalability is limited by the GPU with the least computational power and memory, resulting in inefficiency of the GPUs within a system with larger memory storage and / or computational power.

[0022] Referring to FIG. 2, FIG. 2 shows an exemplary embodiment of a special expert machine learning model, where the sparse layer is distributed on sparse hardware, the dense layer is distributed on dense hardware, and the sparse layer is interleaved with the dense layer such that the sparse hardware can process multiple outputs from multiple dense hardware devices. The disclosed embodiments provide many technical advantages over existing systems.

[0023] For example, a computing system 200 having a plurality of accelerators (e.g., accelerator 202, accelerator 204, accelerator 206, accelerator 208, and one or more other accelerators not shown) is shown. A machine learning model is distributed across the various accelerators. For example, a first plurality of model layers (e.g., layer 210, layer 211, and layer 222) distributed on accelerator 202 are shown. Each layer further includes one or more layers (i.e., sublayers). For example, layer 211 includes layer 212 (e.g., Add & Norm), layer 214 (e.g., a sparse layer) further including gating layer 216, layer 218 (e.g., Add & Norm), and layer 220 (e.g., multi-head attention).

[0024] Similarly, a second plurality of model layers (e.g., layer 224, layer 215, and layer 234) distributed on accelerator 204 are shown. Each layer further includes one or more layers (i.e., sublayers). For example, layer 215 includes layer 226 (e.g., Add & Norm), layer 224 (e.g., a sparse layer) further including gating layer 228, layer 230 (e.g., Add & Norm), and layer 232 (e.g., multi-head attention).

[0025] As described above, in conventional systems, the sparse layer includes one or more experts. However, in the illustrated embodiments, which represent some of the disclosed embodiments herein, experts from various sparse layers are distributed across separate accelerators. For example, expert 236 and expert 238 from the sparse layer associated with layer 210, and expert 240 and expert 242 from layer 214 associated with layer 211, are distributed across accelerator 206. Further, expert 244, expert 246, expert 248, and expert 250 are distributed across accelerator 208. As illustrated, experts are configured to receive different inputs from different layers of the machine learning model when distributed across multiple accelerators. This prevents a component from being idle while other components are actively processing data.

[0026] In this way, the MOE layers and non-MOE layers are separated into separate groups, and each group is assigned its own set of accelerators. By distributing the machine learning model according to this configuration, the computing system can achieve the following technical advantages. First, the system can selectively apply specific performance optimizations suitable for dense and sparse computations. Further, MOE layers incorporate heavy communication overhead. Thus, the configuration as shown in FIG. 2 enables the system to utilize the higher communication bandwidth available within a subset of the cluster (e.g., on a single node or nodes within a single rack).

[0027] Furthermore, according to the disclosed embodiments, a single set of MOE layers can be scheduled to process multiple interleaving inputs to efficiently improve model throughput (for training, inference, and / or implementation).

[0028] The disclosed embodiments also relate to systems and methods that can use a heterogynous cluster to reduce the computational cost and improve the performance of a computing system / machine learning model. Specifically, such a configuration facilitates a significant reduction in the time required to train machine learning, thus enabling the user to more quickly deploy the machine learning model within the implementation task.

[0029] Turning now to FIG. 3, FIG. 3 shows the components of a computing system 310 that includes and / or can be used to implement aspects of the disclosed invention. As shown, the computing system includes a plurality of machine learning (ML) engines, models, and data types related to the input and output of the machine learning engines and models. For example, FIG. 3 shows the computing system 310 as part of a computing environment 300 that also includes a remote / third party system 320 communicating with the computing system 310 (via network 330). The computing system is communicating with a remote / third party system 320 that includes one or more processors 322 and one or more computer-executable instructions 324. In some examples, it is contemplated that the remote / third party system 320 further includes a database that can house training data, e.g., data that can be used as external speaker data. Additionally or alternatively, the remote / third party system 320 includes a machine learning system external to the computing system 310. In some embodiments, the remote / third party system 320 is a software program or application.

[0030] Computing system 310 includes, for example, one or more processors 312 (such as one or more hardware processors), and storage (i.e., hardware storage device 340) that stores computer-executable instructions 318. One or more of the hardware storage devices 340 can accommodate any number of data types and any number of computer-executable instructions 318, and when the computer-executable instructions 318 are executed by one or more processors 312, the computing system 310 is configured to implement one or more aspects of the disclosed embodiments by the computer-executable instructions 318.

[0031] Computing system 310 further includes a plurality of accelerators (such as dense accelerator 313 and sparse accelerator 314). In some configurations, dense accelerator 313 is configured to process input data using a dense layer, and dense accelerator 313 is customized hardware optimized for processing power. In such a configuration, sparse accelerator 314 is configured to process input data using a sparse layer, and sparse accelerator 314 is customized hardware optimized for memory storage. Sparse accelerator 314 is more efficient in processing sparse data (such as sparse tensors, sparse layers) than dense accelerators. Each of the accelerators can include a dedicated processor or other hardware capable of storing and / or executing the corresponding dense layer and sparse layer (344 and 346, respectively).

[0032] In some examples, the sparse accelerator has at least 10% more memory or storage capacity than the dense accelerator, or even more (e.g., 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, or even more than 100% more memory storage capacity than the dense accelerator). Additionally or alternatively, the sparse accelerator is at least 10% more efficient, or even more, than the dense accelerator when processing sparse data (e.g., 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, or even more than 100% more efficient than the dense accelerator when processing sparse data).

[0033] In some examples, the dense accelerator has at least 10% more, or even more, processing power than the sparse accelerator (e.g., 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, or even more than 100% more processing power than the sparse accelerator). For example, the dense accelerator is more efficient than the sparse accelerator when processing dense data (e.g., a dense layer, a dense tensor).

[0034] In some examples, the sparse accelerator is distinguished from the dense accelerator based at least on its high efficiency when processing sparse data. Additionally or alternatively, the sparse accelerator is distinguished from the dense accelerator based on its improved memory capacity and / or reduced raw FLOP count compared to the dense accelerator.

[0035] Also shown is a computing system 310 that includes a user interface 315 and an input / output (I / O) device 316. As shown in FIG. 3, the hardware memory device 340 is shown as a single storage device. However, it will be understood that in some embodiments, the hardware memory device 340 is distributed storage that is distributed across several separate, sometimes remote and / or third-party systems 320 (e.g., hardware memory device 324). The computing system 310 may also include a distributed system, and in some embodiments, one or more of the components of the computing system 310 are remote from each other and are maintained / executed by different separate systems that perform different tasks. In some examples, multiple distributed systems perform similar and / or shared tasks to implement the functionality disclosed in, for example, a distributed cloud environment.

[0036] In this way, the various layers of the machine learning model can be distributed across the computing system 310 and / or across the distributed computing system 300 that includes the computing system 310 and one or more third-party systems 320.

[0037] The hardware memory device 340 is configured to store various data (e.g., input data 348) including various models such as a machine learning model 342 that includes a dense layer 344 and a sparse layer 346, which are used to process the input data 348.

[0038] A storage (e.g., hardware storage device 340) includes computer-executable instructions 318 that instantiate or execute one or more of the models and / or engines shown in computing system 310. The model is configured as a machine learning model or a trained machine learning model, such as a deep learning model and / or algorithm. In some examples, one or more models are configured as an engine or processing system (e.g., a computing system integrated within computing system 310), and each engine (i.e., model) includes one or more processors (e.g., hardware processor 312) corresponding to computing system 310 and computer-executable instructions 318.

[0039] An additional storage device for storing a machine learning (ML) engine 350 is currently shown in FIG. 3 as storing a plurality of machine learning models and / or engines. For example, computing system 310 includes one or more of a data search engine 351, a distributed engine 352, and an implementation engine 353, configured individually and / or collectively to implement the various functions described herein.

[0040] For example, the data search engine 351 is configured to identify and access the locations of data sources, databases, and / or storage devices that include one or more data types from which the data search engine 351 can extract a set or subset of data to be used as input data. The data search engine 351 receives data from a database and / or a hardware storage device, and the data search engine 351 is configured to reformat or enhance the received data for use as training data or input data. Additionally or alternatively, the data search engine 351 communicates with one or more remote / third-party systems (such as the remote / third-party system 320) that include remote / third-party data sets and / or data sources. In some examples, such data sources include audiovisual services that record audio, text, images, and / or video.

[0041] The data search engine 351 accesses electronic content that includes other types of audiovisual data such as acoustic data, text data, and / or video data, image data, holographic data, 3D image data, etc. The data search engine 351 is a smart engine that can learn an optimal data set extraction process that timely provides an efficient amount of data and searches for the data most applicable to the desired application in which the machine learning model / engine is used. For example, the data search engine 351 can learn which databases and / or data sets generate training data that trains a model (for example, for a particular query or particular task) to improve the accuracy, efficiency, and effectiveness of that model within a desired layer distribution configuration.

[0042] In some embodiments, computing system 310 includes a dispersion engine 352 configured to determine the dispersion of the various layers of machine learning model 342 across the various accelerators. Dispersion engine 352 is also configured to facilitate pre-instantiation dispersion of the model or re-dispersion of the machine learning model after instantiation. In some examples, re-dispersion is based on identifying potential improvements in one or more attributes of the computing system (e.g., model throughput, computing efficiency), and / or based on scaling up or down the machine learning model. In some embodiments, re-dispersion is implemented by migrating one or more experts from one accelerator to a different accelerator.

[0043] Dispersion engine 352 is configured to dynamically identify the total number of accelerators that make up computing system 310 and to identify which accelerators are dedicated or optimized hardware devices for dense and sparse layers. Further, dispersion engine 352 is configured to identify which accelerators are full and / or which accelerators have available or expected processing power and / or memory space for storing and executing one or more additional layers.

[0044] Dispersion engine 352 is also configured to identify how many layers make up the machine learning model and to identify which layers are dense layers and which layers are sparse layers. Dispersion engine 352 is further configured to identify how many experts are in each of the various sparse layers and to identify the attributes of the experts (e.g., which dedicated tasks are associated with the experts).

[0045] Accordingly, based on the number of dense layers and the number of sparse layers (or the number of experts across one or more sparse layers), the distribution engine 352 is configured to dynamically and automatically distribute the various layers of the computing system 310 onto one or more accelerators. In some examples, the distribution engine 352 distributes the dense layer 344 onto a dense accelerator 313 (e.g., an accelerator determined to have the ability to store and / or execute the dense layer) and distributes the sparse layer 346 onto a sparse accelerator 314 (e.g., an accelerator having a more limited capacity and capabilities than the dense accelerator).

[0046] In some embodiments, the computing system 310 includes an implementation engine 353 and / or an ML engine 350 (all of the models / engines) communicating with any of the models included within the computing system 310, whereby the implementation engine 353 is configured to implement, initiate, or execute one or more functions of the plurality of ML engines 350. As an example, the implementation engine 353 is configured to operate the data search engine 351, and thus the data search engine 351 can search for data at an appropriate time to process the input data 348. The implementation engine 353 facilitates process communication and communication timing between one or more of the ML engines 350.

[0047] In another example, the implementation engine 353 is configured to implement one or more functions of the machine learning model 342 (i.e., processing the input data 348) as being distributed across the computing system 310 or across the computing system 310 and the third-party system 320. The implementation engine 353 is also configured to implement the distribution engine 352 to identify the distribution or redistribution of the various layers of the machine learning model.

[0048] Furthermore, implementation engine 353 is configured to select which experts, which are distributed across various accelerators, are used when processing input data 348. By implementing the system and method according to the disclosed embodiments, computing system 310 can achieve technical advantages such as being customizable and scalable. Specifically, various experts can be used at various times when processing input data 348. Thus, the system can be configured to select a limited number of experts for use when processing input data based on the type of input data, the formatting of the input data, the context of the input data, and / or the downstream application of the processed input data. Thus, when one or more experts are identified as the experts to be used when processing input data, the system can distribute or redistribute one or more sparse layers including those identified experts across various accelerators to improve the model throughput and improve the computational efficiency of the system.

[0049] Next, looking at FIGS. 4A - 4D, FIGS. 4A - 4D show various embodiments of a machine learning model (e.g., machine learning model 342) distributed over a computing system (e.g., computing system 310). First, looking at FIG. 4A, FIG. 4A shows an exemplary embodiment of a novel configuration. As shown, computing system 400 includes a plurality of accelerators (e.g., accelerator 402, accelerator 404, accelerator 406, accelerator 408, and accelerator 410). In some examples, as shown in FIG. 4A, the accelerators are configured as GPUs (e.g., GPU1, GPU2, GPU3, GPU4, and GPU5). Accelerator 402 and accelerator 410 are configured to process inputs using dense layers. For example, a plurality of dense layers (e.g., D1, D2, D3, and D4) are distributed over accelerator 402 and accelerator 410. Accelerator 404, accelerator 406, and accelerator 408 are configured to process inputs from previous layers using sparse layers.

[0050] For example, a first sparse layer including a first plurality of experts (e.g., S1E1, S1E2, S1E3, and S1E4) is distributed over accelerator 404, a second sparse layer including a second plurality of experts (e.g., S2E1, S2E2, S2E3, and S2E4) is distributed over accelerator 406, and a third sparse layer including a third plurality of experts (e.g., S3E1, S3E2, S3E3, and S3E4) is distributed over accelerator 408. In the foregoing experts, the notation is as follows: "S" refers to the sparse layer, the following number indicates the number of the sparse layer, "E" refers to the expert, and the following number indicates the number of the expert. Although each accelerator is shown as having at least four layers, it should be understood that any number of layers, including no layers, can be distributed over any one of the accelerators or across multiple accelerators.

[0051] As shown in FIG. 4A, accelerators 404, 406, and 408 are each configured to simultaneously receive a plurality of inputs from accelerators 402 and 410 through interleaving. In this way, multiple accelerators can process the inputs simultaneously, thus improving the computing efficiency and processing power of the computing system.

[0052] By dispersing the various layers in this way, another technical advantage is achieved. For example, the system can comprise a first set of customized hardware optimized to process dense layers and a second set of customized hardware optimized to process sparse layers. This was not possible in previous configurations where the accelerators were configured to process inputs in both dense and sparse layers.

[0053] In some examples, the hardware devices optimized for dense layers are characterized by high processing power attributes. Further, such hardware devices are also characterized by less memory storage. In some examples, the hardware devices optimized for sparse layers are characterized by high memory storage attributes. Further, such hardware devices are also characterized by lower processing power attributes than the hardware devices optimized for dense layers.

[0054] Next, turning to FIG. 4B, FIG. 4B shows a machine learning model distributed over a computing system 401 with customized hardware. For example, FIG. 4B includes components similar to those of FIG. 4A, where accelerators 402 and 410 are configured for dense layers (i.e., dense hardware), and accelerators 404, 406, and 408 are configured for sparse layers. By customizing the various hardware components of computing system 400, the computing efficiency and processing power of the computing system are improved.

[0055] In some examples, efficiency is characterized by how long a layer uses the memory occupied by the layer on a particular accelerator. Thus, an efficient system is one in which layers that perform little processing occupy less memory than layers that perform a large amount of processing that occupy a larger amount of memory. Alternatively, layers that perform little processing but require a large amount of memory (e.g., sparse layers) can be distributed over dedicated hardware (e.g., dense hardware) optimized for memory storage rather than processing power, and layers (e.g., dense layers) can be distributed over dedicated hardware (e.g., sparse hardware) optimized for processing power rather than storage capacity.

[0056] The disclosed embodiments achieve additional technical advantages, including, beneficially, being highly scalable. Next, turning to FIG. 4C, FIG. 4C shows a computing system 403, which represents a computing system 401 further including an additional accelerator (e.g., accelerator 412) configured as a sparse hardware device. A fourth sparse layer including a fourth plurality of experts is distributed over accelerator 412. As shown, the fourth plurality of experts includes two experts (e.g., S4E1 and S3E4). It should be understood that any plurality of experts on any of the various accelerators can include one or more experts.

[0057] As shown in FIG. 4C, since the layers of the machine learning model are selectively (i.e., dense layers on dense hardware and sparse layers on sparse hardware) distributed, the machine learning model is highly scalable. For example, to improve the throughput of the system, additional accelerators (e.g., GPU 6) can be added in smaller increments. In this way, smaller or larger accelerators can be added to the system to accommodate one or more additional sparse layers that include one or more additional experts. This also realizes additional flexibility and customizability in how the various layers of the machine learning model are distributed over the computing system.

[0058] Additionally or alternatively, one or more experts from another accelerator can be migrated onto an additional accelerator. For example, as shown in FIG. 4D, a computing system 405 representing a computing system 403 having similar hardware devices but a different distribution of machine learning model layers, which was initially distributed over accelerator 408 (see FIG. 4C), is shown here to have some experts (e.g., S3E3 and S3E4) that are now distributed over accelerator 412. When an additional accelerator with available processing and memory storage capacity is identified, the system can redistribute (i.e., migrate) one or more experts onto the additional accelerator. For example, if the system selects expert S3E1 and expert S3E3 as the experts to be used when processing new input data, the system determines that redistributing the layers will improve the model throughput and facilitate improved computational processing.

[0059] On the one hand, if S3E1 and S3E3 stay on the same sparse hardware (e.g., accelerator 408), the input data would have to be processed serially, first with S3E1 and then with S3E3. However, here, using reallocation, the input data can be processed in parallel using both accelerator 408 and accelerator 412. In this way, latency is reduced (one layer does not wait for another layer to finish processing), and the model throughput is improved.

[0060] It should be understood that expert reallocation and migration can be performed during downtime and / or during processing performed by the model, based on a dynamic assessment of the capabilities and constraints associated with running the model on various data sets.

[0061] Since each expert is trained for a specific task or data application, the machine learning model can process input data using a variety of different training tasks or applications. This special MOE configuration enables the machine learning model to easily and efficiently incorporate new or updated experts without the need to retrain the entire model and / or without the need to reconfigure other layers distributed across the computing system.

[0062] Furthermore, in some embodiments, new experts are added onto an available accelerator or onto a new accelerator that is easily integrated within the system. This is a very efficient configuration since the hardware is not left idle (i.e., multiple hardware devices are processing multiple inputs simultaneously). This is a significant improvement over conventional mixture-of-experts systems that are not easily adaptable to new tasks, not scalable, and have limited throughput because a hardware device is idle while other hardware devices are processing inputs.

[0063] Next, turning to FIG. 5, FIG. 5 shows a flowchart 500 that includes various operations (operation 510, operation 520, operation 530, operation 540, operation 550, operation 560, and operation 570) related to an exemplary method that can be implemented by a computing system 310 to distribute a mixture of expert machine learning models over a customized computing system.

[0064] The first illustrated operation includes an operation to access a machine learning model that includes a plurality of dense layers and a plurality of sparse layers (operation 510). Since the model has both dense layers and sparse layers, the machine learning model can implement a plurality of different experts within the sparse layers, which realizes additional flexibility and customizability of machine learning for various training tasks, domains, and applications. Further, a plurality of sparse layers are interleaved with the plurality of dense layers.

[0065] Next, the system identifies some of the dense layers and some of the experts included in the machine learning model (operation 520). Further, the system accesses a computing system having a plurality of accelerators, and at least some of the accelerators have a higher memory capacity than other accelerators, and at least some of the accelerators have a higher processing capacity than other accelerators (operation 530). By identifying a specific number of elements in each of the computing system and the machine learning model, the system can generate an accurate and efficient distribution strategy. Further, the accelerators are hardware accelerators. For example, one or more hardware accelerators can be configured as a graphics unit processor.

[0066] Thereafter, the system identifies which accelerators are available (operation 540) and determines which of the available accelerators are configured to support dense layers and which are configured to support experts (operation 550).

[0067] After accessing the machine learning model and the computing system, the system distributes a plurality of dense layers to one or more available accelerators configured to support dense layers (operation 560), and is configured to distribute some experts to one or more available accelerators configured to support experts, whereby one or more accelerators configured to support experts are configured to receive a plurality of inputs from one or more accelerators configured to support dense layers (operation 570).

[0068] By generating a distribution policy in this way, the efficiency of the system and model combination is improved. This is because layers that perform little processing occupy less memory than layers that perform a large amount of processing that occupy a large amount of memory. Alternatively, layers that perform little processing but require a large amount of memory (e.g., sparse layers) can be distributed on dedicated hardware (e.g., dense hardware) optimized for memory storage rather than processing power, and layers (e.g., dense layers) can be distributed on dedicated hardware (e.g., sparse hardware) optimized for processing power rather than storage capacity.

[0069] It should be understood that each sparse layer included within a plurality of sparse layers further includes one or more experts, whereby the machine learning model is configured as a mixture-of-experts (MOE) machine learning model.

[0070] In some embodiments, the method further includes accessing an additional accelerator configured to support one or more experts, and integrating the additional accelerator into the computing system such that the computing system becomes an extended computing system.

[0071] In such an embodiment, the system can access one or more additional experts related to the machine learning model and distribute one or more additional experts onto additional accelerators.

[0072] Additionally or alternatively, the system selects a limited number of experts that are distributed across one or more accelerators, such that the limited number of experts are selected to process input data, access a set of input data, and process the input data using at least the limited number of experts.

[0073] Turning now to FIG. 6, FIG. 6 shows a flowchart 600 that includes various operations (operation 610, operation 620, operation 630, operation 640, operation 650, operation 660, and operation 670) related to an exemplary method that may be implemented by a computing system 310 for determining re-distribution of one or more layers of a MOE machine learning model.

[0074] The first illustrated operation includes accessing a computing system with a plurality of accelerators, where at least some of the accelerators have a higher memory capacity than other accelerators and at least some of the accelerators have a higher processing capacity than other accelerators (operation 610).

[0075] The method also includes an operation of accessing a machine learning model that includes a plurality of dense layers and a plurality of sparse layers, each sparse layer further including one or more experts, and the machine learning model is distributed across one or more accelerators included within a plurality of accelerators, whereby the plurality of dense layers of the machine learning model are distributed across one or more accelerators having higher processing capabilities, and the plurality of sparse layers of the machine learning model are distributed across one or more accelerators having higher memory capabilities (operation 620). The one or more accelerators having higher processing capabilities are configured as dense accelerators optimized to process dense layers. The one or more accelerators having higher memory capabilities are configured as sparse accelerators optimized to process sparse layers. Thus, the dense accelerator has higher processing capabilities than the sparse accelerator, and the sparse accelerator has a larger memory capacity than the dense accelerator. This is because the sparse layer requires more memory than the dense layer but is less computationally intensive than the dense layer, and the dense layer requires less memory than the sparse layer but is more computationally intensive than the sparse layer.

[0076] By dispersing the various layers in this way, another technical advantage is achieved. For example, the system can include a first set of customized hardware optimized to process dense layers and a second set of customized hardware optimized to process sparse layers. This was not possible in previous configurations where the accelerator was configured to process inputs with both dense and sparse layers.

[0077] After accessing the computing system and the machine learning model, the system is configured to execute one or more computer-executable instructions configured to cause the computing system to apply the machine learning model to input data (operation 630).

[0078] After executing one or more computer-executable instructions, the system then identifies some available accelerators included within a plurality of accelerators (operation 640) and determines the processing efficiency of the computing system according to an initial distribution of a machine learning model over one or more accelerators (operation 650).

[0079] Based at least on the processing efficiency, the system identifies a new distribution of the machine learning model over the plurality of accelerators, and the new distribution is configured to improve the processing efficiency of the computing system (operation 660). Finally, the system migrates at least one of one or more experts to at least one of some available accelerators having higher memory capabilities to facilitate an improvement in the processing efficiency of the computing system (operation 670). By implementing the system and method in this way, the machine learning model can be optimized based on improving the model throughput and promoting the improvement of the hardware distribution efficiency.

[0080] Furthermore, such disclosed embodiments provide the ability to scale a machine learning model to a greater or fewer number of layers, including integrating or removing one or more different experts as parts of the MOE. This facilitates a reduction in computation and time for training and for processing new input data. For example, in some instances, the method comprises selecting a limited number of experts distributed across one or more accelerators, where the limited number of experts are selected to process input data, accessing a set of input data, and processing the input data using at least the limited number of experts.

[0081] Furthermore, some methods also include the operation of determining an additional distribution of a limited number of experts across the plurality of accelerators based on a selected limited number of experts. In this way, the system can migrate one or more of the limited number of experts to different accelerators according to the additional distribution. Exemplary computing system Embodiments of the present invention may comprise, or utilize, a special purpose or general purpose computer including computer hardware (e.g., computing system 310), as will be discussed in more detail below. Embodiments within the scope of the present invention may also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer system.

[0082] A computer-readable medium (e.g., hardware storage device 340 of FIG. 3) that stores computer-executable instructions (e.g., computer-executable instructions 318 of FIG. 1) is a physical hardware storage medium / device, excluding transmission media. A computer-readable medium that conveys computer-executable instructions or computer-readable instructions (e.g., computer-executable instructions 318) in one or more carrier waves or signals is a transmission medium. Thus, by way of example and not limitation, embodiments of the present invention may include at least two distinctly different types of computer-readable media: physical computer-readable storage media / devices and transmission computer-readable media.

[0083] The physical computer-readable storage media / devices are hardware and include RAM, ROM, EEPROM, CD-ROM, or other optical disk storage (CD, DVD, etc.), magnetic disk storage, or other magnetic storage devices, or any other hardware that can be used to store desired program code means in the form of computer-executable instructions or data structures and that can be accessed by a general purpose or special purpose computer.

[0084] "Network" (for example, network 330 in FIG. 3) is defined as one or more data links that enable the transfer of electronic data between computer systems and / or modules and / or other electronic devices. When information is transferred or provided to a computer via a network or another communication connection (hardwired, wireless, or a combination of hardwired or wireless), the computer appropriately considers the connection as a transmission medium. The transmission medium can include a network and / or data link that can be used for carrying, or desired program code means in the form of computer-executable instructions or data structures that can be accessed by a general-purpose or special-purpose computer. The above combinations are also included within the scope of computer-readable media.

[0085] Furthermore, when reaching various computer system components, program code means in the form of computer-executable instructions or data structures can be automatically transferred from a transmission computer-readable medium to a physical computer-readable storage medium (and vice versa). For example, computer-executable instructions or data structures received via a network or data link are buffered in RAM within a network interface module (for example, "NIC") and then can ultimately be transferred to the computer system RAM and / or a less volatile computer-readable physical storage medium of the computer system. Thus, the computer-readable physical storage medium can be included within computer system components that also (and even primarily) utilize the transmission medium.

[0086] Computer-executable instructions can include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a certain function or group of functions. Computer-executable instructions can be, for example, binary, intermediate format instructions such as assembly language, and even source code. Although the subject matter has been described in terms of structural features and / or methodological acts peculiar to it, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or the aforementioned acts. Rather, the described features and acts are disclosed as exemplary forms for implementing the claims.

[0087] Those skilled in the art will understand that the present invention can be implemented in a network computing environment having many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based, or programmable consumer electronics, network PCs, minicomputers, mainframe computers, cellular phones, PDAs, pagers, routers, switches, and the like. The present invention can also be implemented in a distributed system environment where local and remote computer systems linked through a network (by hardwired data links, wireless data links, or a combination of hardwired and wireless data links) together perform tasks. In a distributed system environment, program modules can be located in both local and remote memory storage devices.

[0088] Alternatively, or in addition, the functionality described herein may be performed, at least in part, by one or more hardware logic components. By way of example, and without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), and the like.

[0089] The present invention may be embodied in other specific forms without departing from its essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. Thus, the scope of the invention is indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.

Claims

1. One or more hardware processors, One or more hardware storage devices configured to store computer executable instructions, wherein the computer executable instructions are executable by one or more hardware processors to manage the distribution of sparse and dense layers of a mixed expert (MOE) model across multiple accelerators, A plurality of accelerators comprising a first set of accelerators and a second set of accelerators, wherein the first set of accelerators is configured to support multiple sparse layers by having dense hardware with a larger memory capacity and smaller processing power, and the second set of accelerators is configured to support multiple dense layers by having sparse hardware with a higher processing power and smaller memory capacity, A machine learning model comprising the plurality of dense layers and the plurality of sparse layers, wherein the machine learning model is distributed on the plurality of accelerators, the plurality of dense layers are distributed on the second set of accelerators, the plurality of sparse layers are distributed on the first set of accelerators, and at least one accelerator storing one or more sparse layers of the machine learning model is configured to receive a plurality of inputs from one or more accelerators storing the dense layers of the machine learning model. A computing system that includes this.

2. A computing system according to claim 1, wherein each sparse layer contained within the plurality of sparse layers further includes one or more experts, thereby configuring the machine learning model as a mixed expert machine learning model.

3. A computing system according to claim 1, wherein each expert is trained for a specific task.

4. A computing system according to claim 1, wherein one or more accelerators are hardware accelerators.

5. A computing system according to claim 4, wherein one or more accelerators are configured as graphics unit processors.

6. A computing system according to claim 1, wherein the second set of accelerators is configured as a dense accelerator having at least 10% greater processing capacity for the dense layer than the first set of accelerators configured as sparse accelerators.

7. A computing system according to claim 1, wherein the first set of accelerators is configured as a sparse accelerator that is at least 10% more efficient in processing sparse data than the second set of accelerators configured as a dense accelerator.

8. A computing system according to claim 1, wherein the plurality of sparse layers are interleaved with the plurality of dense layers.

9. A method for managing the distribution of machine learning models, including mixed experts (MOEs), on a computing system, Accessing machine learning models that include multiple dense and multiple sparse layers, Identifying several dense layers and several experts included in the aforementioned machine learning model, Accessing a computing system including multiple accelerators, wherein a first set of accelerators is configured to support multiple sparse layers by having dense hardware with greater memory capacity and less processing power, and a second set of accelerators is configured to support multiple dense layers by having sparse hardware with higher processing power and less memory capacity. Identifying which accelerators are available, Determining which of the available accelerators is configured to support the multiple dense layers and which accelerator is configured to support the expert, Distributing the multiple dense layers in one or more available accelerators configured to support the dense layers, Distributing the several experts across one or more available accelerators configured to support the experts, such that the one or more accelerators configured to support the experts are configured to receive multiple inputs from the one or more accelerators configured to support the dense layer. A method that includes this.

10. A method according to claim 9, wherein each sparse layer contained within the plurality of sparse layers further comprises one or more experts, thereby configuring the machine learning model as a mixed expert machine learning model.

11. The method according to claim 9, Access to additional accelerators configured to support one or more experts, To integrate the additional accelerator into the computing system so that the computing system becomes an extended computing system, A method that further includes this.

12. The method according to claim 11, Access to one or more additional experts related to the aforementioned machine learning model, Distributing the one or more additional experts on the additional accelerator A method that further includes this.

13. The method according to claim 9, Selecting a limited number of experts distributed across one or more accelerators, wherein the limited number of experts are selected to process input data, Accessing the set of input data, Processing the input data using at least the limited number of experts A method that further includes this.

14. A method according to claim 9, wherein the plurality of sparse layers are interleaved with the plurality of dense layers.

15. A method according to claim 9, wherein the plurality of accelerators are hardware accelerators.

16. The method according to claim 9, wherein the plurality of accelerators are configured as graphics unit processors.

17. A method for determining the redistribution of a machine learning model including mixed experts (MOEs) on a computing system, Accessing a computing system that includes multiple accelerators, where the multiple accelerators are The first set of accelerators is configured to support multiple sparse layers by having dense hardware with a larger memory capacity and smaller processing power. The second set of accelerators is configured to support multiple dense layers by having sparse hardware with higher processing power and smaller memory capacity. It is structured in such a way, This involves accessing a machine learning model that includes multiple dense layers and multiple sparse layers, Each sparse layer further contains one territorial expert, The machine learning model is distributed across the multiple accelerators, The aforementioned multiple dense layers are distributed on the second set of accelerators having relatively high processing power. The plurality of sparse layers are distributed on the first set of accelerators having a relatively large memory capacity, The computing system executes one or more computer-executable instructions configured to apply the machine learning model to the input data. After executing one or more computer-executable instructions, identify some of the available accelerators included in the plurality of accelerators, The processing efficiency of the computing system is determined according to the first distribution of the machine learning model on the one or more accelerators, Identifying a new distribution of the machine learning model on the plurality of accelerators, wherein the new distribution is configured to improve the processing efficiency of the computing system. A method that includes this.

18. The method according to claim 17, In accordance with the new distribution, at least one of the one or more experts is migrated to at least one of the several available accelerators having a larger memory capacity. A method that further includes this.

19. The method according to claim 17, Selecting a limited number of experts distributed across one or more accelerators, wherein the limited number of experts are selected to process input data, Accessing the set of input data, Processing the input data using at least the limited number of experts A method that further includes this.

20. The method according to claim 19, Based on the selected limited number of experts, determine the additional distribution of the limited number of experts across the multiple accelerators, In accordance with the aforementioned additional distribution, one or more of the limited number of experts are moved to a different accelerator. A method that further includes this.

Images