Machine learning model processing based on perplexity

Abstract

A method for operating a machine learning model is presented. The machine learning model includes a plurality of sequential transformer blocks. The method comprises receiving input data at a transformer block and processing the input data via a mixture of experts layer. At an auxiliary classifier, a measure of perplexity of the processed input data is determined. Based on the determined measure of perplexity, one or more experts in a downstream transformer block that will subsequently process the input data are indicated. Weight matrices are then fetched for the indicated one or more experts.

Description

[Technical Field]

[0001] This disclosure relates to perplexity-based machine learning model processing. [Previous Technology]

[0002] Neural networks can be trained as components of machine learning models to perform tasks on input data. Such tasks may include image recognition, speech recognition, natural language processing, and other advanced functions. Expert hybrid techniques break down large tasks into subtasks, assign each subtask to an expert, and then use a gated model to determine which expert to use to process a given input. [Summary of the Invention]

[0003] This summary is provided to introduce, in a simplified form, a selection of concepts that will be further described in the embodiments described below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to embodiments that address any or all of the shortcomings mentioned in any part of this disclosure.

[0004] A method for operating a machine learning model is provided. The machine learning model includes a plurality of successive transformer blocks. The method includes receiving input data at a transformer block and processing the input data via an expert mixing layer. At an auxiliary classifier, a measure of the perplexity of the processed input data is determined. Based on the determined measure of perplexity, one or more experts in downstream transformer blocks that will subsequently process the input data are indicated. A weight matrix for the indicated one or more experts is then extracted.

Implementation Method

[0016] The quality of models in multimodal scenarios and natural language processing models is closely related to the model's size. Model size involves three main factors: the number of model parameters (excluding embeddings), the complexity and size of the underlying dataset, and the amount of computational resources required. Increasing the number of model parameters while keeping the data complexity and required computation constant has been shown to be effective in improving model quality to some extent. For a given data complexity, a technique called sparsely-activated expert (MoE) hybrid can be used to increase the number of model parameters at a fixed computational cost. However, although MoE models can be successfully applied to make training more efficient at scale, due to the load imbalance problem, it produces severely low resource utilization (e.g., <5%) during inference. Therefore, powerful computational clusters cannot utilize their full capacity.

[0017] MoE can be used to divide the problem space into smaller homogeneous regions, each of which is assigned to one or more trained experts. For example, an image detection model can use one neural network expert for human detection and another neural network expert for pose estimation.

[0018] Not every input fragment is passed to every expert within the expert mix. Instead, the MoE model uses a local routing algorithm to pass each input data fragment (e.g., a patch of an image or a symbol in a sentence) to a specific expert. Therefore, not all experts can be active simultaneously. In this way, by performing conditional computation based on the complexity of the input data, MoE inherently saves resources by using coarse-grained sparsity at the software level. However, the gating process inherently introduces redundancy, which may result in only moderate sparsity.

[0019] Machine learning models can employ several layers of MoE. For a face recognition model, the first few layers may include general filters, such as edge detectors. The last few layers may include more specialized experts for recognizing facial parts (such as lips, nose, ears, etc.). Therefore, in the early stages, deciding which expert receives the image patch may not be as important as in later stages. As another example, natural language processing (NLP) systems can break down text into tokens. Some tokens may be replaceable or do not have inherent meaning. If all tokens are treated with equal importance, the tokens can be accurately routed, but the computational cost is high.

[0020] Therefore, multiple input data fragments can be processed in batches by a single expert or a group of experts to concentrate hardware utilization in those layers. Conversely, for later layers, the accuracy of the routing mechanism can be highly important. Therefore, depending on the nature of the input data and the layers of the model, it is important to predict where noise from more random patches can be tolerated and where it cannot. Analyzing the importance or complexity of each input data fragment in earlier layers can allow more general experts to process less important fragments in batches, thereby increasing utilization and saving resources.

[0021] Transformer-based models are commonly used to implement MoE models. A transformer model can be considered as a collection of shallow networks, so that some fragments do not need to be resolved through all MoE layers to produce conclusions. More important or complex data fragments may need to be passed through more layers and can be passed to more precisely trained experts. Properly guiding these fragments allows for the intelligent allocation of these resources.

[0022] This paper provides systems and methods that enable more efficient operation of MoE models with a smaller footprint. In one instance, the individual nodes of the MoE are sparsified. In another instance, the perplexity of the input data is used to determine which experts to load for subsequent processing. In yet another instance, input data fragmentation is achieved through modal batching, thereby allowing the loading of the most relevant experts.

[0023] Figure 1 schematically illustrates an example machine learning system 100 configured to evaluate input data 102 using a series of transformer blocks 105, each transformer block containing one or more MoE layers 107.

[0024] MoE can be considered an ensemble learning technique that implements training the opinions of neural network experts (e.g., first expert 110, second expert 111, third expert 112) on sub-tasks of a prediction modeling problem. The task (e.g., image classification) can be divided into multiple sub-tasks. For each sub-task, one or more neural network experts (110, 111, 112) can be trained to produce predictions, such as numerical values ​​in the case of a regression model or class labels in the case of a classification model. A routing gate network 115 may contain one or more gated models trained to determine which neural network expert uses a given input data fragment 120. For MoE layer 107, the outputs of the routing gate network 115 and any selected experts are pooled to produce a uniform prediction at output gate 122. Generally, in contrast to ensemble techniques that combine results from all available models, only a few or even just one expert model are run at a time.

[0025] The machine learning system 100 can be built on a converter architecture and may contain several repeating converter blocks (105, 136), each block consisting of a self-attention sublayer 130, a first addition and normalization layer 132, a MoE layer 107 containing a fully connected feedforward network, and a second addition and normalization layer 134. When the system supports a MoE model on a converter architecture, the properties of the self-attention layer 130 are largely fixed.

[0026] Input data is provided to the self-attention layer 130 and the addition and normalization layer 132. Normalized data is then provided to the MoE layer 107 and the second addition and normalization layer 134. The output of the second addition and normalization layer 134 can be provided to the next converter block 136. Depending on the configuration of the system 100 and the next converter block 136, the output of the second addition and normalization layer 134 can be received by the self-attention layer, the next MoE layer, convolutional layers, etc.

[0027] Input data 102 can be divided into input data fragments 120 (e.g., dividing an image into patches, dividing a sentence into symbols, etc.). For example, an image can be divided into separate elements, such as background, foreground, objects, colors, lines, etc. However, when the input data elements are not obvious, the feature space can be divided more generally, or the overlap of professional domains can be increased. The lower layer of system 100 can be configured to extract features, and these features are subsequently evaluated by experts.

[0028] The gate control model (typically a neural network model itself) is trained alongside each expert to interpret the predictions made by each expert and to help determine which expert to trust to make an accurate prediction for a given input. The routing gate control network 115 may contain a single shared gate or may have multiple task-specific gates. For example, each routing gate may deploy a top-k function to route input fragments to the k most relevant experts. For example, if there are 10 potential experts and a top-2 function that selects 2 experts, then 80% coarse-grained sparsity (e.g., 1-20% density) will be established.

[0029] The routing gate network 115 can treat the input data fragment 120 as input and output the contribution that each expert should have in making predictions for each input data fragment. The input data fragment 120 is then passed to the appropriate expert. In this example, a second expert 111 is selected to receive the input data fragment 120, but in other examples, the input data fragment 120 can be routed to one or more different or additional experts.

[0030] When more than one expert is selected to process the input data fragment, its output must be pooled to determine the output value. This may involve selecting the expert with the maximum output or confidence level provided by the routing gate network 115. The output of the routing gate network 115 is also supplied to the output gate 122, thus modulating the output of the selected expert. For example, weighted sum predictions may be performed, which explicitly combine the predictions made by each expert with the confidence level evaluated by the routing gate network 115. The routing gate network 115 may be configured to select a sparse combination of experts to process each input data fragment 120, for example, to make the system 100 sparse in order to maintain floating point operations per second (FLOP).

[0031] Experts 110, 111, and 112 may include feedforward neural networks, each containing its own set of learnable expert weights and parameters 140. These expert weights and parameters 140 can be programmed on storage device 142. When a particular expert is selected to evaluate a fragment of input data, the expert weights and parameters 140 can be maintained in storage device 142 and selectively retrieved by MoE layer 107. In this way, instead of loading all possible expert weights and parameters 140 even when the corresponding expert is not in use, the area occupied by transformer block 105 can be minimized.

[0032] The self-attention layer 130 may apply a filter and / or a vector of importance weights that limits the amount of input data 102 received at one time in the transformer block 105. By moving attention around, the system 100 can process large amounts of input. For example, images can be parsed line by line, and long paragraphs of text can be evaluated across sentences or paragraphs. By evaluating the input data fragments 120 in this way, the MoE layer 107 can extract rich relationships between the input data elements. When specific features are identified, these features can be passed on to experts who specialize in parsing those features. Using attention vectors, the correlation with other input features can be approximated.

[0033] As examples of experts 110, 111, and 112, Figure 2 schematically illustrates a multilayer neural network 200, including an input layer (x0) 202, two hidden layers (x1) 204 and (x2) 206, and an output layer (x3) 208. In this example, the input layer 202 includes 5 neurons (210, 211, 212, 213, 214), the first hidden layer 204 includes 3 neurons (220, 221, 222), the second hidden layer 206 includes 4 neurons (230, 231, 232, 233), and the output layer 208 includes 3 neurons (241, 242, 243).

[0034] The neural network 200 includes corrected linear units as a start-up function. The forward pass includes a series of matrix-vector products f(x0;w), where x0 is the input or feature vector. The network function f(x0;w) is parameterized by weight matrices w1 250, w2 251, and w3 252 and a bias vector. Each weight matrix includes the weights for each connection between two adjacent layers.

[0035] The size of deep neural networks such as network 200 rapidly exceeds the hardware's ability to quickly store and train these deep neural networks. Sparsity can be applied to reduce the number of network parameters at initialization, during training, and during inference by pruning edges from the underlying topology. Figure 3 illustrates a sparse version 300 of network 200, comprising a hidden input layer (x0') 302, hidden layers (x1') 304 and (x2') 306, and an output layer (x3') 308. In this example, the third input feature 212 and all its neighboring weights are removed from the input layer (x0') 302 (dashed lines). Furthermore, hidden neurons 222 and 232 and their weights are removed from hidden layers (x1') 304 and (x2') 306, respectively. The other weights have been removed from sparse version 300, resulting in weight matrices (w1')350, (w2')351, and (w3')352. Removing neurons or input features in this manner corresponds to removing columns or rows in the layer weight matrices. Removing independent weights corresponds to removing independent elements of the weight matrix. Sparsity can be naturally introduced or produced, and can be applied to other tensors and matrices, such as matrices for startup, error, bias, etc. For startup, startup of a node essentially produces zero output. The same applies to sparsity applied to startup; for example, higher-amplitude startups are of higher value to the network and are preserved. In some instances, startup naturally approximates sparsity, thus sparsity also adds a modest effect.

[0036] Sparsity can be unstructured or balanced. For unstructured sparsity, the mask has several constraints and can be configured in virtually any random pattern. In Figure 4, mask 410 is an example of unstructured sparsity. Each black square masks the underlying value to 0. Each white square allows the underlying value to be non-zero. The numerical series or row count on the grid axis, for example, how many non-zero values ​​exist in that dimension. For example, the top column of mask 410 has one white square (non-zero value) and the second row from the left of mask 410 has two white squares (non-zero values). This convention is used throughout this disclosure.

[0037] Unstructured sparsity is usually applied after training the network, but in some cases it can also be applied during training. Unstructured sparsity is the lowest constrained form of sparsity, but its inherent randomness makes it difficult to speed up at the hardware level.

[0038] The most common constraint for balanced sparsity is N out of M constraints. For a row or column with a value of M, only N (N < M) can be non-zero. For example, Mask 420 is an instance of balanced sparsity with a value N = 1. Each column of Mask 420 has a white square (non-zero value). The rows of Mask 420 vary from 0 to 2 non-zero values.

[0039] Therefore, balanced sparsity is more constrained than unstructured sparsity, but easier to accelerate with hardware because the hardware can predict the expectation of each column or row for each constraint. Known constraints can be preloaded into the hardware.

[0040] In this paper, most instances can be considered as balanced stochastic fine-grained sparsity. In this context, fine-grained means that only a portion of the tensor is sparse, while balanced means that all blocks (e.g., columns, rows) have the same sparsity level, but the pattern within each block is random. This is the opposite of unstructured sparsity, in which the size of each sparse block is equal to the size of the tensor. As the block size increases, the fidelity also increases because different configurations can be represented more flexibly. However, as the block size increases beyond a threshold, the gains decrease. Therefore, balanced stochastic fine-grained sparsity represents an opportunity to strike a good balance between hardware implementation cost and accuracy. In addition, balanced sparsity can be scaled to different sparsity levels and can be applied to weights, initiation, error, and gradients, and can also have a scalable impact on training by selecting which tensors to sparsify.

[0041] Based on the selection of the routing gate network for the MoE layer, the MoE model can increase parameterization by selectively loading only a few experts at a time. The level of coarse-grained sparsity in the MoE model can be determined by taking the ratio (n_topk / n_expert), where n_topk indicates the number of active experts specified by the routing gate network to evaluate each input data fragment, and n_expert indicates the total number of experts in the layer. For a fixed n_topk and hidden dimension (H=n_expert*d_mlp, where d_mlp is the dimension of each expert), the gain decreases when increasing the number of experts, thereby setting an upper limit on the level of coarse-grained sparsity. In addition, this ratio mainly represents the perspective of a single input data fragment. If different related fragments enter a layer and are distributed among each expert in the layer, then there is actually no coarse-grained sparsity at all, because all weights of the layer are always loaded in the end.

[0042] In the current state-of-the-art MoE model, each expert is locally dense. To further improve efficiency, we propose a balanced configuration of locally fine-grained sparse experts on top of the previously mentioned coarse-grained sparsity (reducing the overhead of executing each expert). The effective global sparsity level is then achieved by Sfine_grained*Scoarse_grained, which can increase sparsity by up to 99%.

[0043] Figure 5 illustrates an example method 500 for operating a machine learning model, such as a machine learning system 100, comprising one or more expert hybrid layers. When implemented, method 500 may allow a mixture of balanced, balanced, fine-grained sparsity and coarse-grained sparsity inherent in the MoE model.

[0044] At 510, method 500 includes receiving one or more input data fragments at a routing gate network comprising an expert hybrid layer containing a plurality of neural network experts. For example, the input data fragments may contain image patches, sentence symbols, etc., depending on the configuration of the machine learning model.

[0045] At 520, method 500 includes specifying one or more neural network experts in a hybrid layer of experts to evaluate each piece of input data. For example, the number of specified neural network experts among a plurality of neural network experts may be based on a top-K function.

[0046] At 530, method 500 includes, for each specified neural network expert, extracting a weight matrix having a predetermined sparsity to generate sparse specified neural network experts. The number of specified neural network experts among the plurality of neural network experts may be based on instances of a top-K function, and the predetermined sparsity may be based at least on the ratio of the top-K function to the total number of neural network experts.

[0047] In some instances, the predetermined sparsity can be configured as unstructured sparsity. Unstructured sparsity can produce the highest sparsity level for a fixed accuracy. In some instances, the predetermined sparsity is configured as a balanced N sparsity among M sparsities (e.g., N elements in every M subsequent elements are non-zero, as shown in Figure 4). In such instances, each neural network expert can implement it using hardware configured to implement balanced sparsity for a given block size.

[0048] As an example, the balanced N sparsity among M sparsity can have block sizes M ≥ 8, M ≥ 16, etc., depending on the hardware configuration. As M increases, the balanced N:M pattern can progressively approach the efficiency of unstructured sparsity.

[0049] In some instances, the predetermined sparsity is based at least on the expertise levels of specified neural network experts. For example, Figure 6 schematically illustrates an example machine learning system 600 comprising several consecutive MoE layers (602, 604, 606). The first MoE layer 602 includes a routing gate network 610, three experts (611, 612, 613), and an output gate 615. The output of the output gate 615 can be routed to the second MoE layer 604 via one or more normalization and / or attention layers 617. The second MoE layer 604 includes a routing gate network 620, three experts (621, 622, 623), and an output gate 625. The output of the output gate 625 can be routed to the third MoE layer 606 via one or more normalization and / or attention layers 627. The third MoE layer 606 includes a routing gate network 630, three experts (631, 632, 633), and an output gate 635. Storage device 640 is configured to store weights and parameters 642 for at least experts 611, 612, 613, 621, 622, 623, 631, 632, and 633. Although primarily described in terms of weight matrices, expert weights and parameters may include priming matrices, gradients, errors, and / or other parameters to which sparsity can be applied. MoE layers 602, 604, and 606 can be considered experts with increasingly more specialized knowledge. In other words, experts 611, 612, and 613 may be the most general, while experts 631, 632, and 633 may be the most specialized. Therefore, experts in MoE layer 602 can allow for more noise and sparsity compared to experts in MoE layers 604 and 606.

[0050] Input data fragment 650 is received at the routing gateway network 610, and the routing gateway network assigns an expert 613 to evaluate the input data fragment. Subsequently, the MoE layer 602 can retrieve a sparse weight matrix 652 from a storage device 640, which can generate a sparse weight matrix 652 from a dense weight matrix. The expert 613 can then sparsify and thus evaluate the input data fragment 650. In this example, the sparse weight matrix 652 has 75% sparsity.

[0051] Subsequently, expert 613 can output data to output gate 615, which guides the data to routing gate network 620 via normalization and attention layer 617. Routing gate network 620 can assign expert 623 to evaluate the input data. Subsequently, MoE layer 604 can retrieve sparse weight matrix 654 from storage device 640. Sparse weight matrix 654 is less sparsity (62.5%) compared to sparse weight matrix 652, which is commensurate with a higher level of expertise.

[0052] Subsequently, expert 623 can output data to output gate 625, which directs the data to routing gate network 630 via normalization and attention layer 627. Routing gate network 630 can assign expert 633 to evaluate the input data. Subsequently, MoE layer 606 can retrieve sparse weight matrix 656 from storage device 640. Sparse weight matrix 656 is less sparsity (50%) compared to sparse weight matrices 652 or 654, and is commensurate with even higher levels of expertise.

[0053] Returning to Figure 5, at 540, method 500 includes evaluating each input data fragment with a corresponding sparse specified neural network expert. In some instances, evaluating each input data fragment includes performing inference on the input data fragment. Alternatively or additionally, each input data fragment may be evaluated as part of training the corresponding neural network expert.

[0054] Once the sparse matrices are loaded, they will be used to perform vector-matrix multiplications, which can be performed faster when sparsified, thereby reducing the computational cost required to evaluate each piece of input data. Additionally, by sparsifying the weights and / or parameters for each expert, experts can be loaded more quickly, thus reducing end-to-end latency. Specifically, when performing inference, loading the weights and parameters contributes more to latency than the multiplications themselves.

[0055] As described in Figure 1, the transformer block is a common topology used in several different fields (e.g., computer vision, NLP). A transformer-based model can be considered as a collection of shallow networks. In other words, input samples can take different routes based on their perplexity. Less perplexed data fragments may be able to skip multiple layers and / or leave the entire network earlier. In this way, redundant or irrelevant computations can be avoided for samples that do not need to be evaluated at every and every block of the model.

[0056] Figure 7 schematically illustrates an example machine learning system 700 configured to preload expert weights based on the perplexity of input data. The machine learning system 700 represents three consecutive transformer blocks: transformer block 702, transformer block 704, and transformer block 706. Transformer block 702 includes a MoE layer 710, comprising three experts 711, 712, and 713. Transformer block 704 includes a MoE layer 720, comprising three experts 721, 722, and 723. Transformer block 706 includes a MoE layer 730, comprising three experts 731, 732, and 733. A storage device 735 is configured to store weights and parameters 737 for each expert in the machine learning system 700.

[0057] Transformer block 702 is fed into auxiliary classifier 740, while transformer block 704 is fed into auxiliary classifier 742. In other instances, such as systems where transformer blocks are aligned with the parallelism of the sorting buffer, each set of parallel blocks may be fed into an auxiliary classifier. Each auxiliary classifier may analyze data fragments evaluated by previous blocks and instruct one or more experts at subsequent layers to evaluate one or more probabilities of the data fragments based on their perplexity at the current block.

[0058] In the example of Figure 7, input data fragment 750 can be considered a higher perplexity data fragment, while input data fragment 752 can be considered a lower perplexity data fragment. Input data fragment 750 is received at transformer block 702 and processed by expert 711 in MoE layer 710. The processed input data fragment is then evaluated by auxiliary classifier 740. Auxiliary classifier 740 determines that input data fragment 750 has high perplexity and assigns it to expert 721 in MoE layer 720 in subsequent transformer block 704. Transformer block 704 can therefore retrieve the weights and parameters of expert 721 from storage device 735. Subsequently, auxiliary classifier 742 can assign input data fragment 750 to expert 732 in MoE layer 730 in subsequent transformer block 706. Transformer block 706 can therefore retrieve the weights and parameters of expert 732 from storage device 735.

[0059] In this way, relevant weights can be pre-selected for use by experts in subsequent layers. This can significantly reduce end-to-end latency in inference model generation. When applied to training and fine-tuning models, this methodology can reduce the amount of time spent in the forward and backward paths of each iteration.

[0060] Input data fragment 752 can be received at transformer block 702 and processed by expert 713 in MoE layer 710. The processed input data fragment is then evaluated by auxiliary classifier 740. Auxiliary classifier 740 determines that input data fragment 752 has low perplexity and is a candidate for early exit from machine learning system 700. Input data fragment can therefore bypass transformer block 704 and be assigned to be processed by expert 733 in MoE layer 730 in transformer block 706. Transformer block 706 can therefore retrieve the weights and parameters of expert 733 from storage device 735.

[0061] In the example of Figure 7, input data fragment 750 can be considered a higher perplexity data fragment, while input data fragment 752 can be considered a lower perplexity data fragment. Input data fragment 750 is received at transformer block 702 and processed by expert 711 in MoE layer 710. The processed input data fragment is then evaluated by auxiliary classifier 740. Auxiliary classifier 740 determines that input data fragment 750 has high perplexity and assigns it to expert 721 in MoE layer 720 in subsequent transformer block 704. Transformer block 704 can therefore retrieve the weights and parameters of expert 721 from storage device 735. For example, storage device 740 can receive a request for a sparse weight matrix with predetermined sparsity, retrieve the requested stored weight matrix, sparsify the retrieved stored matrix, and provide a sparse stored weight matrix for expert 721. Subsequently, the auxiliary classifier 742 can assign the input data fragments 750 to the expert 732 in the MoE layer 730 of the subsequent transformer block 706. The transformer block 706 can then retrieve the weights and parameters of the expert 732 from the storage device 735.

[0062] In this way, relevant weights can be pre-selected for use by experts in potential subsequent layers. This can significantly reduce end-to-end latency in inference model generation. When applied to training and fine-tuning models, this methodology can reduce the amount of time spent in the forward and backward paths of each iteration.

[0063] Input data fragment 752 can be received at transformer block 702 and processed by expert 713 in MoE layer 710. The processed input data fragment is then evaluated by auxiliary classifier 740. Auxiliary classifier 740 determines that input data fragment 752 has low perplexity and is a candidate for leaving the machine learning system 700 early. The input data fragment can therefore bypass transformer block 704 and be assigned to be processed by expert 733 in MoE layer 730 in transformer block 706. Transformer block 706 can therefore retrieve the weights and parameters of expert 733 from storage device 735. In this way, low perplexity data can avoid redundant or unnecessary processing and leave the transformer block earlier, thereby allowing resources to be reallocated to higher perplexity data.

[0064] Figure 8 illustrates an example method 800 for operating a machine learning model (such as system 700) comprising a plurality of successive transformer blocks. When implemented, method 800 allows the machine learning model to predict, given the perplexity of the input at the current layer, which experts (if any) in subsequent transformer blocks will be active to process the specific input.

[0065] At 810, method 800 includes receiving input data at a transformer block. For example, the input data may include image data, NLP data, etc., depending on the configuration of the machine learning model. Continuing at 820, method 800 includes processing the input data via an expert hybrid layer. In some instances, processing the input data includes performing inference on the input data. Additionally or alternatively, the input data may be evaluated as part of training and / or fine-tuning the machine learning model.

[0066] At 830, method 800 includes, at an auxiliary classifier, determining a measure of the perplexity of the processed input data. For example, the measure of perplexity is a loss function, such as a cross-entropy loss function. In some instances, the measure of perplexity is determined for a single piece of input data (such as a symbol). Alternatively or additionally, the measure of perplexity may be determined for a set of input data pieces, such as a sentence or a set of words or sentences.

[0067] At 840, method 800 includes indicating one or more experts in a downstream transformer block that will subsequently process the input data, based on a determined measure of perplexity. In some instances, the measure of perplexity will be used to indicate whether a specialized or more general expert will subsequently process the input data. For example, an input data fragment with lower perplexity may provide additional noise and be routed to whichever expert is available or already loaded, while an input data fragment with higher perplexity will benefit from highly accurate routing.

[0068] In some instances, the downstream converter block is the next converter block. However, in some instances, a metric based on the determined perplexity may indicate that the input data is likely to bypass one or more converter blocks among a plurality of consecutive converter blocks. In some instances, a top-k selection may be performed based on the determined perplexity metric to select one or more experts to subsequently process the input data. However, in many instances, the determined perplexity metric will allow bypassing or eliminating local routing mechanisms, so that local top-k selection does not need to be performed at each layer. In this way, hardware utilization can be improved by improving scheduling based on perplexity.

[0069] At 850, method 800 includes extracting a weight matrix of one or more indicated experts. Extracting the weight matrix of one or more indicated experts may include extracting sparse weights based on a determined metric of perplexity. For example, input data with lower perplexity may allow for extracting sparser weights, while input data with higher perplexity may benefit from extracting a denser weight matrix.

[0070] Another problem associated with the MoE model is the batch processing during inference time. Inference is limited by memory, not computation. Especially for multimodal neural networks, fragments within each batch of input data require access from different experts. For example, input data may include image data along with associated text, or video data including audio data and subtitles.

[0071] The MoE model may include some experts extended to different modalities, but will also include specialists for each modality. For example, not every expert in image recognition will be an expert in NLP processing. Even when using a text model, different experts can be deployed for each possible language, and not all experts will be experts in all languages.

[0072] For this type of dataset, loading experts in all possible modalities may be inefficient. Additionally, loading experts for individual input data fragments and then unloading them, only to reload them when the next relevant input data fragment is listed, may also be inefficient. Therefore, routing input data to the most relevant experts is challenging and requires high-level quasi-scheduling.

[0073] Figure 9 illustrates a method 900 for operating a machine learning model. When implemented, method 900 allows the machine learning model to process input fragments in batches via modality, thereby enabling selective loading and scheduling expertise.

[0074] At 910, method 900 includes, at the expert hybrid layer, receiving input data comprising a plurality of fragments of input data. For example, the machine learning model may be configured to receive input data from different machine learning domains (e.g., modalities) corresponding to different machine learning tasks, such as speech recognition, image classification, machine translation, or parsing.

[0075] At 920, method 900 includes classifying input data fragments into multiple batches based on a common modality. For example, input data fragments may be classified into batches such as speech, image, language, text, etc., or batches such as English, Spanish, Chinese, Arabic, etc.

[0076] At 930, method 900 includes extracting weights of neural network experts trained in modalities represented in multiple batches. For example, for input data including video of a speaker, weights of one or more experts in speech recognition and image classification may be extracted.

[0077] At 940, method 900 includes scheduling each batch for processing by neural network experts trained in the relevant modality. Scheduling each batch for processing by neural network experts trained in the relevant modality can be performed by a reinforcement learning agent and / or any suitable online or offline learning algorithm. In some instances, the reinforcement learning agent and / or other learning algorithm are trained in load balancing. In this way, constraints are imposed on the processing path. Some experts are eliminated from the process, and their potential inputs are preferentially directed to other experts.

[0078] In some instances, method 900 may include maintaining the extracted weights of each neural network expert at the node based on the relevant batch processing schedule. For example, the expert may be maintained at the node until the batch is complete. Additionally or alternatively, method 900 may include unloading the extracted weights from the node after processing a batch. In some instances, method 900 may subsequently include extracting weights for different neural network experts to be loaded onto the node. In this way, the total number of nodes to be maintained in the machine learning model may be less than the total number of experts available for the machine learning model.

[0079] By routing input data in this manner, the overlap between selected experts increases, and the total number of experts and / or the diversity of experts loaded to process the input dataset decreases. As described with respect to Figures 5 and 6, weight extraction may include extracting a sparse weight matrix. Therefore, loading it from memory reduces memory footprint and loading time, thereby further improving end-to-end performance. When the model can predict what will happen, the loading process can overlap with other computational processes, and thus overlap communication between the storage machine and the computing node.

[0080] As an example, Figure 10 schematically illustrates an example machine learning system 1000 configured to process input data fragments by modality batching and scheduling. Input data 1002, containing multiple input data fragments of various modalities, is evaluated by a reinforcement learning agent 1005. The reinforcement learning agent 1005 divides the input data fragments into multiple batches. In this example, there are three batches: the first batch 1010 (squares), the second batch 1012 (circles), and the third batch 1014 (triangles).

[0081] The reinforcement learning agent 1005 can provide instructions on which experts should be loaded into nodes 1022, 1024, and 1026 of the MoE layer 1020. The MoE layer 1020 retrieves weights from a storage device 1030 configured to store expert weights and parameters 1032. In this way, node 1022 is configured to process the first batch 1010, node 1024 is configured to process the second batch 1012, and node 1026 is configured to process the third batch 1014.

[0082] In some embodiments, the methods and processes described herein may relate to a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as computer applications or services, application-programming interfaces (APIs), libraries, and / or other computer program products.

[0083] Figure 11 schematically illustrates a non-limiting embodiment of a computing system 1100 that can perform one or more of the methods and processes described herein. The computing system 1100 is illustrated in a simplified form. The computing system 1100 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), and / or other computing devices. Systems 100, 200, 300, 600, 700, and 1000 may be examples of the computing system 1100.

[0084] The computing system 1100 includes a logic machine 1110 and a storage machine 1120. The computing system 1100 may include a display subsystem 1130, an input subsystem 1140, a communication subsystem 1150, and / or other components not shown in Figure 11.

[0085] The logic machine 1110 includes one or more physical means configured to execute instructions. For example, the logic machine may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, change the state of one or more components, achieve technical effects, or otherwise achieve desired results.

[0086] The logic machine may include one or more processors configured to execute software instructions. Alternatively, the logic machine may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processor of the logic machine may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The independent components of the logic machine may be distributed across two or more separate devices, which may be remotely located and / or configured for coordinated processing. The state of the logic machine may be virtualized and executed via remotely accessible networked computing devices configured with cloud computing capabilities.

[0087] In addition to one or more GPUs 1154, the logic subsystem may include one or more CPUs 1152, and the one or more CPUs 1152 may be configured to send executable instructions and / or data to one or more GPUs 1154. In response to processing instructions and / or data by one or more GPUs 1154, the CPU 1152 may receive result data from one or more GPUs 1154. In this way, the logic subsystem can perform a large number of computations in parallel via GPUs. Specifically, the logic subsystem can efficiently execute method 500 of Figure 5, method 800 of Figure 8, and method 900 of Figure 9.

[0088] This disclosure relates to GPUs as computing devices well-suited for distributed learning processes because GPUs are configured to execute a very large number of repeated instances of the same program (e.g., GPU cores) in parallel, where each instance of the program receives different input data and works on different input data. However, it will be understood that other configurations of the logic subsystem can be configured to provide the same or similar benefits. Therefore, it will be understood that any discussion of GPUs also applies to other suitable computing devices, and this disclosure does not in any way limit itself to performing methods 500, 800, 900, or any other configuration of training machine learning models on GPUs to exclude other suitable computing devices.

[0089] Storage machine 1120 includes one or more physical means configured to store instructions executable by a logic machine to implement the methods and processes described herein. When such methods and processes are implemented, the state of storage machine 1120 can be changed, for example, for storing different data. Storage devices 142, 640, 735, and 1030 may be examples of storage machine 1120.

[0090] Storage device 1120 may include removable and / or built-in devices. Storage device 1120 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk drive, magnetic tape drive, MRAM, etc.). Storage device 1120 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location-addressable, file-addressable, and / or content-addressable devices.

[0091] It will be understood that the storage machine 1120 includes one or more physical devices. However, the manner of the instructions described herein may alternately be transmitted by communication media (e.g., electromagnetic signals, optical signals, etc.) which are not stored by physical devices for a limited time.

[0092] The states of logic machine 1110 and storage machine 1120 can be integrated together into one or more hardware logic components. For example, such hardware logic components may include field-programmable gate arrays (FPGAs), program-and-specific standard products (PASIC / ASIC), program-and-application-specific standard products (PSSP / ASSP), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs).

[0093] The terms "module," "program," and "engine" can be used to describe the state of a computing system 1100 implemented to perform a specific function. In some cases, a module, program, or engine can be instantiated via a logic machine 1110 that executes instructions stored by a storage machine 1120. It will be understood that different modules, programs, and / or engines can be instantiated from the same application, service, code block, target, library, routine, API, function, etc. Similarly, the same module, program, and / or engine can be instantiated via different applications, services, code blocks, targets, routines, APIs, functions, etc. The terms "module," "program," and "engine" can encompass standalone or grouped executable files, data files, libraries, drivers, scripts, database records, etc.

[0094] It will be understood that, as used herein, a "service" is an application that can be executed across multiple user communication periods. A service may be applied to one or more system components, programs, and / or other services. In some implementations, a service may run on one or more server-computing devices.

[0095] When included, the display subsystem 1130 can be used to represent a visual representation of data stored by the storage machine 1120. This visual representation may take the form of a graphical user interface (GUI). As the methods and processes described herein change the data stored by the storage machine, and thus change the state of the storage machine, the state of the display subsystem 1130 may similarly change to visually represent the changes in the underlying data. The display subsystem 1130 may include one or more display devices virtually utilizing any type of technology. Such display devices may be integrated with the logic machine 1110 and / or the storage machine 1120 in a common housing, or such display devices may be peripheral display devices.

[0096] When included, the input subsystem 1140 may include or interface with one or more user input devices, such as a keyboard, mouse, touchscreen, or game controller. In some embodiments, the input subsystem may include or interface with a selected natural user input (NUI) component. Such a component may be integrated or peripheral, and the conversion and / or processing of input actions may be handled on-board or off-board. Example NUI components may include: a microphone for voice and / or audio recognition; an infrared, color, stereo, and / or depth camera for machine vision and / or gesture recognition; a head tracker, eye tracker, accelerometer, and / or gyroscope for motion detection and / or intent recognition; and an electric field sensing component for assessing brain activity.

[0097] When included, the communication subsystem 1150 may be configured to communicatively couple the computing system 1100 to one or more other computing devices. The communication subsystem 1150 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem may be configured for communication via a wireless telephone network, or a wired or wireless local or wide area network. In some embodiments, the communication subsystem may allow the computing system 1100 to send messages to and / or receive messages from other devices via a network (such as the Internet).

[0098] In one instance, a method for operating a machine learning model comprising a plurality of successive transformer blocks includes receiving input data at a transformer block; processing the input data via an expert mixing layer; determining a measure of perplexity of the processed input data at an auxiliary classifier; indicating one or more experts in a downstream transformer block to subsequently process the input data based on the determined measure of perplexity; and extracting a weight matrix for the indicated one or more experts. In this instance, or any other instance, extracting the weight matrix for the indicated one or more experts additionally or alternatively includes extracting sparse weights based on the determined measure of perplexity. In any of the previous instance, or any other instance, the downstream transformer block additionally or alternatively is a next transformer block. In any of the previous instance, or any other instance, the method additionally or alternatively includes indicating that the input data is likely to bypass one or more transformer blocks in the plurality of successive transformer blocks. In any of the previous instance, or any other instance, the method additionally or alternatively includes performing a top K selection to select one of the one or more experts to subsequently process the input data. In the previous instance, or any of the other instances, the perplexity metric is determined additionally or alternatively for a single piece of input data. In the previous instance, or any of the other instances, the perplexity metric is determined additionally or alternatively for a set of input data pieces. In the previous instance, or any of the other instances, the perplexity metric is additionally or alternatively a loss function. In the previous instance, or any of the other instances, the loss function is additionally or alternatively a cross-entropy loss function. The technical effect of implementing this method is an improvement in the use of computational resources.

[0099] In another instance, a method for operating a machine learning model includes, at an expert hybrid layer, receiving input data comprising a plurality of input data fragments; classifying the input data fragments into batches based on a common modality; extracting weights of neural network experts trained in the modality represented by the batches; and scheduling each batch for processing by neural network experts trained in the relevant modality. In this instance, or any other instance, the method additionally or alternatively includes maintaining the extracted weights for each neural network expert at a node based on the relevant batch processing schedule. In any of the previous instance, or any other instance, the method additionally or alternatively includes unloading the extracted weights from the node after processing a batch; and extracting weights for different neural network experts to be loaded onto the node. In any of the previous instance, or any other instance, extracting weights additionally or alternatively includes extracting a sparse weight matrix. In any of the previous instance, or any other instance, scheduling each batch for processing by neural network experts trained in the relevant modality is additionally or alternatively performed by a reinforcement learning agent. In the previous instance, or any other instance, the reinforcement learning agent is additionally or alternatively trained in load balancing. The technical effect of implementing this method is improved utilization of computational resources.

[0100] In yet another example, a machine learning model includes a plurality of successive transformer blocks configured to receive input data. Each transformer block includes: an expert mixing layer configured to process the input data; an auxiliary classifier configured to determine a measure of perplexity of the processed input data; one or more processors configured to, based on the determined measure of perplexity, indicate one or more experts in downstream transformer blocks that will subsequently process the input data; and extract a weight matrix for the indicated one or more experts. In this example, or any other example, extracting the weight matrix for the indicated one or more experts additionally or alternatively includes extracting sparse weights based on the determined measure of perplexity. In any of the previous examples, or any other examples, one or more processors additionally or alternatively are configured to indicate that the input data is likely to bypass one or more transformer blocks among the plurality of successive transformer blocks. In any of the previous examples, or any other examples, the measure of perplexity is additionally or alternatively determined for a set of input data fragments. In the previous instance, or any other instance, the measure of perplexity is additionally or alternatively incorporated into the loss function. The technical effect of implementing this machine learning model is an improvement in the use of computational resources.

[0101] It will be understood that the configurations and / or approaches described herein are exemplary in nature, and such specific embodiments or examples are not intended to be limiting, as several variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Thus, the various actions shown and / or described may be shown and / or described in sequence, in another sequence, in parallel, or omitted. Similarly, the order of the processes described above may be changed.

[0102] The subject matter of this disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations disclosed herein, as well as other features, functions, actions and / or properties, and any and all equivalents thereof. [Simplified Explanation of the Diagram]

[0005] Figure 1 schematically illustrates an example machine learning model that includes one or more expert hybrid layers.

[0006] Figure 2 schematically illustrates an example of a dense neural network.

[0007] Figure 3 schematically illustrates an example of a sparse neural network.

[0008] Figure 4 schematically illustrates an unstructured and unbalanced sparse mask.

[0009] Figure 5 is a flowchart of an example method for operating a machine learning model that includes one or more expert hybrid layers.

[0010] Figure 6 schematically illustrates the sparsity level of the weight matrix associated with the expert experience level of neural networks.

[0011] Figure 7 schematically illustrates an example machine learning system configured to preload expert weights based on the perplexity of the input data.

[0012] Figure 8 is a flowchart of an example method for preloading expert weights based on the perplexity of input data.

[0013] Flowchart of an example method for operating a machine learning system in Figure 9.

[0014] Figure 10 schematically illustrates an example machine learning system configured to process input data fragments in batches based on modality.

[0015] Figure 11 illustrates an example computing system. [Biomaterial Storage]

[0104] Domestic storage information (please note in order of storage institution, date, and number): None. International storage information (please note in order of storage country, institution, date, and number): None.

Claims

1. A method for operating a machine learning model comprising a plurality of successive transformer blocks, comprising the steps of: receiving input data at a transformer block; processing the input data via an expert mixing layer; determining a metric of perplexity of the processed input data at an auxiliary classifier; instructing one or more experts in a downstream transformer block to subsequently process the input data based on the determined metric of perplexity; and extracting a weight matrix for the instructed one or more experts.

2. The method of claim 1, wherein the step of extracting a weight matrix for the indicated one or more experts includes the step of extracting sparse weights based on the determined metric of perplexity.

3. The method as described in claim 1, wherein the downstream converter block is a next converter block.

4. The method of claim 1 further includes the step of: indicating that the input data is likely to bypass one or more converter blocks among the plurality of consecutive converter blocks.

5. The method as described in claim 1 further includes the following steps: performing a top k selection to select one or more experts who will subsequently process the input data.

6. The method as described in claim 1, wherein the measure of perplexity is determined for a single piece of input data.

7. The method as described in request 1, wherein the measure of perplexity is determined for a set of input data fragments.

8. The method as described in request 1, wherein the measure of perplexity is a loss function.

9. The method as described in claim 2, wherein the loss function is a cross-entropy loss function.

10. A method for operating a machine learning model, comprising the steps of: receiving input data comprising a plurality of input data fragments at an expert mixing layer; classifying the input data fragments into multiple batches based on a common modality; extracting weights for neural network experts trained in the modalities represented by the batches; and scheduling each batch for processing by a neural network expert trained in a relevant modality.

11. The method of claim 10 further comprises the step of: maintaining the weights of such extractions for each neural network expert at a node based on a relevant batch processing schedule.

12. The method of claim 11 further comprises the steps of: unloading the extracted weights from the node after processing a batch; and extracting weights of a different neural network expert to be loaded onto the node.

13. The method of claim 10, wherein the step of extracting weights includes the step of extracting a sparse weight matrix.

14. The method of claim 10, wherein the step of scheduling each batch for processing by a neural network expert trained in a relevant modality is performed by a reinforcement learning agent.

15. The method as described in request item 14, wherein the reinforcement learning agent is trained in load balancing.

16. A machine learning model comprising a plurality of successive transformer blocks configured to receive input data, each transformer block comprising: an expert mixing layer configured to process the input data; an auxiliary classifier configured to determine a metric of perplexity of the processed input data; and one or more processors configured to: instruct one or more experts in a downstream transformer block to subsequently process the input data based on the determined metric of perplexity; and extract a weight matrix for the instructed one or more experts.

17. The machine learning model as described in claim 16, wherein the weight matrix extracted for the indicated one or more experts includes sparse weights extracted based on the determined metric of perplexity.

18. The machine learning model as described in claim 16, wherein the one or more processors are further configured to: indicate that the input data is likely to bypass one or more of the plurality of successive transformer blocks.

19. The machine learning model as described in claim 16, wherein the measure of perplexity is determined for a set of fragments of input data.

20. The machine learning model as described in claim 16, wherein the measure of perplexity is a loss function.

Images