Embedding decision trees in neural networks

Abstract

Methods, systems, and apparatus including a computer program encoded on a computer storage medium for scheduling operations represented in a computation graph, one of the methods includes receiving data representing a neural network comprising a plurality of layers arranged in a sequence, selecting one or more groups of layers, each group of layers having one or more layers adjacent to one another in the sequence, and generating, for each group of layers, a new machine learning model comprising a respective decision tree that replaces the group of layers, each decision tree receiving as input a quantized version of an input to a respective first layer in the group and producing as output a quantized version of an output of a respective last layer in the group, a tree depth of each decision tree based at least in part on the number of layers in the group.

Description

[Technical field]

[0001] Technical Field This document relates to incorporating decision trees in large-scale neural networks. [Background technology]

[0002] background A neural network is a machine learning model that applies one or more layers of nonlinear units to predict an output for a received input. Some neural networks contain one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a set of respective parameters.

[0003] More specifically, each neural network layer includes multiple nodes, and each layer represents a set of operations defined by the neural network. In general, these operations are arithmetic operations that can include linear operations, such as addition and multiplication, and nonlinear operations, such as nonlinear activation functions, such as the "Relu" or "Sigmoid" functions. The linear operations combine the layer inputs and the layer weights. The linear operations of each layer can be implemented using tensor operations, where the layer weights are represented in the form of matrices or tensors, and the layer inputs of the layer are represented in the form of vectors.

[0004] Large neural networks, i.e., neural networks with many layers and a large number of parameters, have shown good performance in various machine learning tasks. However, these large neural networks can have high latency and consume large amounts of computational resources, e.g., they require large amounts of memory and consume a significant number of processor cycles to make predictions. One conventional technique for increasing the computational efficiency of neural networks is to manipulate parts of the weight matrix to make it a sparse matrix. A sparse matrix is ​​a matrix in which many terms are zero. Summary of the Invention

[0005] overview This document describes techniques for incorporating decision trees into large-scale neural networks to generate new machine learning models.

[0006] In general, one innovative aspect of the subject matter described in this specification can be embodied in a method that includes the following acts: receiving data representing a neural network having a plurality of layers arranged in a sequence; and selecting one or more groups of layers from the plurality of layers, each group of layers comprising one or more layers that are adjacent to one another in the sequence; and the method further includes generating a new machine learning model corresponding to the neural network.

[0007] Generating the new machine learning model includes selecting, for each group of layers, a respective decision tree to replace the group of layers, where each decision tree receives as input a quantized version of the input to a respective first layer in the group and produces as output a quantized version of the output of a respective last layer in the group, with the tree depth of each decision tree being based at least in part on the number of layers in the group.

[0008] Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs stored on one or more computer storage devices, each configured to perform the operations of the methods.

[0009] Each of the above and other embodiments can optionally include one or more of the following features, either alone or in combination. In particular, one embodiment includes all of the following features in combination.

[0010] The method may further include an act of training a new machine learning model by training at least a portion of the layers in the neural network that were not replaced by the respective decision trees based on the training data of the neural network.

[0011] As discussed above, the act of selecting each of the one or more groups of layers may further include the acts of selecting a respective initial layer in the neural network; generating a respective plurality of candidate groups, each having the respective initial layer as a first layer in the candidate group; for each of the respective plurality of candidate groups, determining a respective performance measure of the candidate group that measures the performance of a corresponding new machine learning model having a layer in the candidate group replaced by the respective decision tree; and, for each of the respective plurality of candidate groups, selecting one of the candidate groups as a group based on the respective performance measure.

[0012] The selection of each initial layer can be performed by a random process or based on a sequence of neural networks. The quantized version of the input to each first layer in the group and the quantized version of the output of each last layer in the group can be generated using binary or ternary quantization. Each decision tree layer that replaces a group of layers can include a GradientBoost decision tree or an AdaBoost decision tree.

[0013] The method may further include outputting the new machine learning model to a system configured to implement the new machine learning model, the system comprising one or more computing units for implementing a decision tree with one or more functions selected from an add function, a select function, or a switch function. That is, the new machine learning model (e.g., having been trained) may be output to a system including one or more computing units (such as a multiplexer or arithmetic logic unit) for implementing a decision tree without requiring a more expensive computing unit, such as a multiply-accumulator unit (MAC), to perform multiplications.

[0014] The subject matter described herein can be implemented in particular embodiments to realize one or more of the following advantages.

[0015] The described systems, which implement the techniques described below, can reduce the computational cost and improve the efficiency of performing inference calculations for large neural networks.

[0016] First, the described technique for replacing one or more network layers of a large-scale neural network with a decision tree can reduce the amount of calculations when the system performs inference calculations for the neural network. For example, the decision tree that replaces one or more layers of the neural network can have only one or a few layers (e.g., tree stumps or shallow trees). The computational cost of performing operations on the tree stumps and shallow trees is much less than the computational cost required by computing the neural network layers of the large-scale neural network. As another example, when the decision tree is an Adaboost tree, since there is no need to perform multiplication operations for the Adaboost tree, the system can perform fewer operations than computing a traditional neural network layer, which usually requires both multiplications and additions, improving efficiency.

[0017] Second, the described technique can reduce the total size of the neural network by quantizing at least the inputs and outputs of the network layers that are replaced by one or more decision trees. Quantizing these inputs and outputs to reduce the number of significant digits involved for the calculation can reduce the computational cost, particularly of the inserted decision tree and at least the neural network layers next to the decision tree (i.e., the layers preceding or following the decision tree). Since the size of the neural network has been reduced by quantization, quantization can also reduce the total memory / storage requirements of the computing system. In light of this, the described technique can enable devices (e.g., smartphones, tablets) with less memory and computing power to efficiently perform the inference calculations of the modified neural network. In some circumstances, one or more hardware accelerators in the device can be customized to perform the inference calculations of the particular modified neural network (i.e., one or more layers are replaced by one or more decision trees and the inputs and outputs of one or more layers are quantized), thereby reducing the memory usage of the device, reducing power consumption, and performing the inference calculations more efficiently and faster.

[0018] Additionally, in many cases, the high precision provided by unquantized inputs and outputs is not needed to accurately detect and represent the presence or absence of important features when performing inference computations for neural networks: the error introduced by quantizing the inputs and outputs to layers replaced by decision trees is minimal compared to the efficiency gains.

[0019] Although quantized data is actually used to train and compute the neural network, quantizing the inputs and outputs to be suitable for each decision tree and the remaining neural network layers is important for the techniques described below. More specifically, the system can quantize floating-point numbers to reduce the number of digits to represent the floating-point number, so that the system can use fewer digits for the sign, exponent, and mantissa of the floating-point number. In the case of binary and ternary quantization, the system can map the floating point to integers such as {1, -1} or {1, 0, -1}, to name a few.

[0020] Also, the described techniques can efficiently train the modified neural network (i.e., a new neural network in which layers are replaced by decision trees). The system simply fine-tunes the parameters in the modified neural network using at least some of the same training examples to train the original neural network. Thus, the time period required to train the modified neural network can be significantly shorter than training the original neural network.

[0021] Furthermore, the described technique can reduce costs by using low-cost programmable hardware to perform operations of the decision tree. For example, the computing system can use a multiplexer (MUX) unit instead of a multiply-accumulator unit (MAC) to compute operations in the decision tree. Those skilled in the art know that a MUX unit consumes less power and space than a MAC unit. Thus, the computing system can only include a programmable hardware unit suitable for the modified neural network, rather than an expensive hardware accelerator such as a GPU or TPU. Thus, the total cost of building a hardware system to perform inference calculations of the modified neural network is much less than that of the original neural network.

[0022] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 illustrates an example neural network deployment system including an example neural network modification engine. [Figure 2A] FIG. 1 illustrates a portion of an example new machine learning model having a decision tree with binary quantized output. [Figure 2B] FIG. 1 illustrates a portion of another example new machine learning model having a decision tree with ternary quantized outputs. [Diagram 3] FIG. 1 is a flow diagram of an example process for generating a new machine learning model that is trained. [Figure 4] FIG. 1 is a flow diagram of an example process for selecting one or more groups of neural network layers. [Diagram 5] FIG. 2 illustrates an example of a decision tree. [Figure 6] FIG. 2 illustrates an exemplary implementation of a decision tree that uses fixed functionality hardware. [Figure 7] FIG. 2 illustrates an example programmable core for performing inference calculations for decision trees. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Detailed Description Like reference numbers and designations in the various drawings indicate like elements.

[0025] One conventional approach to increase efficiency in training and computing large neural networks is to build a respective sparse matrix for each layer's activation input and weight. However, the approach of building a sparse matrix can cause new problems. For example, the use of a sparse matrix can prevent a computing system from accessing the same memory address during a period of time, also known as lack of inference locality. More specifically, when building a sparse matrix, the computing system stores the non-zero terms of the original matrix in different memory addresses that may be physically located far away from each other. In some situations, the memory address of each term in the sparse matrix may change dynamically during the computation. For example, a zero term may become non-zero after being added with another non-zero term. Thus, the computing system may suffer from memory latency, cache trashing, and even cache pollution when accessing data stored in memory, ultimately reducing the computational efficiency of training and input processing with large neural networks.

[0026] Alternatively, another prior art approach is to fuse the weighting matrix into the layer logic. For example, a system employing this fused weighting technique can determine the zero terms in the weighting matrix and not perform multiplication and accumulation operations on this term. With this in mind, the fused weighting technique can reduce the computational cost of large neural networks by not performing calculations where the output is zero because one of the inputs is zero. However, fused weighting comes at a price. First, the fused weighting technique must not be changed once the neural network is deployed in the hardware accelerator, i.e., the zero terms in the weighting matrix must remain zero during the calculation. However, in practice, the weighting matrix of the deployed neural network may need to be fine-tuned using new training data. Second, if the zero terms in the sparse weighting matrix are shared by multiple operations, such as, for example, a multiplication followed by an addition followed by another multiplication, the system still needs to perform operations for the zero terms.

[0027] The techniques described hereinbelow can address the above problems. More specifically, the described techniques can use quantization and decision trees to efficiently perform one or more inference calculations for large-scale neural networks. In general, the described techniques involve replacing one or more layers of a neural network with respective decision trees, where the inputs to and outputs from the decision trees are quantized versions of the inputs and outputs of the corresponding layers.

[0028] This specification describes a system, implemented as a computer program in one or more computers at one or more locations, that generates new machine learning models by replacing one or more groups of network layers of a neural network with one or more decision trees to reduce the computational cost and improve the efficiency of performing inference calculations using economical hardware.

[0029] FIG. 1 illustrates an example neural network evolution system 100 that includes an example neural network modification engine 120 .

[0030] In general, the neural network deployment system 100 receives as input data 110 representing a neural network and outputs a new machine learning model 180 that has been trained. The neural network deployment system 100 includes a neural network modification engine 120 for generating a new machine learning model 130 for the input neural network model. The new machine learning model 130 is a hybrid of the original input neural network model and one or more neural network layers that are replaced by one or more decision trees. Details of the generation of the new machine learning model 130 are described below. The neural network deployment system 100 also includes a training engine 140 configured to train the new machine learning model using the training data 150, and a memory 160 configured to store and provide data for the training engine 140 (e.g., training and output data for the machine learning model, and data defining the machine learning model).

[0031] More specifically, the data 110 representing the neural network received by the deployment system 100 may include information that defines the neural network, such as the operations of each layer of the neural network and the weights of each network layer.

[0032] Data 110 may also represent other aspects of the neural network. For example, data 110 may include data representing the number of network layers in the neural network, the number of respective nodes in each layer, one or more types of inter-layer connections, e.g., element-wise connections or total connections, and data representing the type of each layer in the neural network, e.g., a pooling layer, a fully connected layer, or a SoftMax layer, etc.

[0033] In general, the data 110 may represent a trained neural network that has been trained on a plurality of training data 150, or a neural network that has not yet been trained.

[0034] The neural network deployment system 100 can provide the received data 110 to a neural network modification engine 120. The modification engine 120 can select one or more groups of layers of the neural network and replace each group of layers with a respective decision tree to output a new machine learning model 130. The selection of one or more groups of layers is described in more detail below.

[0035] Each decision tree that replaces a respective group of layers may be stored in memory 160 and accessible for modification engine 120. More specifically, data representing the decision trees and stored in memory 160 may include data specifying a total number of nodes (e.g., a root and multiple leaves), connectivity between the nodes (e.g., how a leaf is connected to one or more other leaves), and one or more node operations (e.g., a logical comparison for one or more nodes).

[0036] The modification engine 120 can automatically determine the respective decision trees for replacing the groups of layers. Alternatively, the type of decision tree for replacing each group of layers can be predetermined by a user or by a computer program implemented by one or more computers external to the modification engine 120. The decision trees can be GradientBoost trees or AdaBoost trees.

[0037] A GradientBoost (or gradient boost) tree can be obtained via gradient boosting, which is a machine learning method that produces a model in the form of a combination (e.g., weighted sum) of one or more simple predictive models (e.g., decision trees). AdaBoost (or Adaptive Boosting) adaptively combines one or more simple predictive models (e.g., decision trees) such that during training, greater weights are assigned to poorly performing simple predictive models (e.g., misclassification measures), and the trained model produced by AdaBoost is more likely to produce accurate predictions given a particular input.

[0038] The modification engine 120 may also quantize at least a portion of the neural network represented by the data 110. For example, the modification engine 120 may quantize the inputs to the first layer of the group of layers to be replaced by the decision tree and the output from the last layer of the group of layers. Alternatively, the modification engine 120 may quantize the entire neural network such that the inputs and outputs to each layer are quantized. Additionally, the modification engine 120 may quantize at least a portion of the training data 150 and use the quantized training data to train the respective decision trees and / or at least a portion of the new machine learning model 130.

[0039] Quantization is the process of mapping a large set of input values ​​to a smaller set of output values, typically using rounding and truncation. More specifically, quantization can be used to reduce the precision of a number. For example, quantization can reduce the precision of a floating-point number from 32 bits to 8 bits. In the context of neural networks, the modification engine 120 can quantize the activation tensors, weight tensors, or layer outputs of a neural network layer from 8 bits to 4 bits, or even 1 bit (e.g., 1 bit for the mantissa) precision. Details of quantization (e.g., binary quantization and ternary quantization) are described in connection with Figures 2A and 2B.

[0040] The modification engine 120 can provide the new machine learning model 130 to the training engine 140. The training engine 140 can then train at least a portion of the new machine learning model 130 based on the training data 150 used to train the original neural network 110, and output the new trained machine learning model 180 as an output of the system 100. More specifically, the training engine 140 can use the training data 150 to train the remaining layers of the neural network that have not been replaced by decision trees during the time period. Alternatively, or in addition, the training engine 140 can train the entire new machine learning model 130 based on the quantized training data, assuming respective gradients of each decision tree.

[0041] The time period for training the new machine learning model 130 may be minutes, hours, or days. Alternatively, or in addition, the time period may be based on the size of the training data 150 that is being used to train at least a portion of the new machine learning model 130. For example, the time period may be determined by the time required to train (e.g., fine-tune) the new machine learning model 130 using 100 mini-batches or 1000 mini-batches of the training data 150.

[0042] The training engine 140 can train a decision tree that replaces one or more layers of the original neural network.

[0043] More specifically, the training engine 140 trains the decision tree before replacing the network layers of the original neural network with the decision tree. The training engine 140 can also fine-tune the decision tree in the new machine learning model 130, as described above.

[0044] To train the decision tree, the training engine 140 may use as training data the corresponding portion of the same, but quantized, training data 150 for training the original neural network. Specifically, the training engine 140 trains the decision tree using each training example. Each training example includes a quantized version of the layer input to the first layer of the group of layers and a quantized version of the layer output from the last layer of the group of layers. The group of layers is replaced by the decision tree. The quantized versions of the layer inputs and layer outputs are associated with the layer inputs and layer outputs of the corresponding training data 150 used to train the group of layers in the original neural network.

[0045] When training a decision tree, the training engine 140 can define a loss and adjust the node operations to reduce the loss until it is below a certain criterion. The loss can be a hinge loss, which indicates mislabeling, or a logarithmic loss, which represents information gain based on entropy theory, or any other loss suitable for training a decision tree. The training engine 140 can adjust the node operations, such as the critical values ​​of each of the logical comparison operations in one or more nodes. In some circumstances, the training engine 140 can reduce overfitting by pruning the decision tree during training, i.e., removing one or more leaves and all branches and child leaves associated with one or more leaves. The training engine 140 can keep a portion of the original training dataset as a validation set to detect and improve overfitting.

[0046] The training engine 140 used to train at least a portion of the new machine learning model 130 may include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), or any other computational unit suitable for performing operations of a neural network. In particular, the new machine learning model may still have layers that have not been replaced that include linear operations (e.g., primarily tensor operations), and thus the training engine 140 may include more TPUs than CPUs or GPUs to facilitate the training process.

[0047] The new machine learning model 180 being trained can be used to efficiently generate inferences given an input. The new machine learning model 180 being trained can be used to generate inferences using less computational power. For example, the new machine learning model 180 being trained requires less memory size to store than the original neural network because one or more layers of the neural network layer have been replaced by a shallow decision tree with fewer nodes. As another example, the new machine learning model being trained is compatible with quantized inputs and outputs that are represented using fewer bits, thus reducing system memory bandwidth during calculations. Furthermore, because computational units such as MUX units or arithmetic logic (ALC) units can be used to perform operations in the decision tree, the neural network inference engine or system can replace large and expensive computational units such as TPUs or GPUs with smaller and cheaper programmable cores with one or more MUX units or ALC units to perform inference calculations of the decision tree of the new machine learning model 180. Thus, the total cost and size of the device for generating inferences of the new machine learning model can be reduced.

[0048] FIG. 2A illustrates a portion of an example new machine learning model 295 having a decision tree with a binary quantized output 225.

[0049] 2A, the portion of the original neural network 200 represented by the input data 110 includes multiple network layers. The multiple network layers may include a group of network layers 290 that has been determined by the system 100 to be replaced by a corresponding decision tree, a first network layer 210 that precedes the first layer of the group of network layers 290, and a second network layer 230 that follows the last layer of the group of network layers.

[0050] Each of the layers in the portion of neural network 200 has a number of nodes each representing a linear operation and a nonlinear operation. For example, network layer 210 includes nodes 210a-f. As another example, network layer 230 includes nodes 230a-f. Also, each layer in network layer group 290 includes a respective number of nodes (not shown).

[0051] Some of the network layers of neural network 200 are arranged in a sequence such that for each input to the neural network, the preceding layer generates a layer output and provides that output as a layer activation input to the subsequent layer. For example, the first layer of the group of network layers 290 receives from network layer 210 as layer activation input 213. As another example, the last layer of the group of network layers 290 provides layer output 217 to the subsequent layer 230.

[0052] The inputs and outputs can have different precisions according to the computational requirements. For example, the inputs and outputs can have a floating-point format with 32-bit precision. As another example, the inputs of the first few layers and the outputs from the last few layers of a neural network can have higher precision than the intermediate layers.

[0053] In some implementations, the system 100 can quantize the inputs and outputs of one or more layers of the neural network using different quantization methods to reduce precision. For example, the activation inputs 213 and layer outputs 217 can be quantized using binary or ternary quantization and can have 8-bit or even 1-bit precision. Alternatively, the system 100 can quantize only a portion of the weight inputs or activation inputs for a network layer.

[0054] As explained above, quantization is the process of reducing the precision of a numeric value (e.g., reducing the number of bits that represent the sign, exponent, and mantissa of the number). Binary quantization and ternary quantization are branches of the quantization process.

[0055] With respect to binary quantization, and with reference to the neural network deployment system 100 and FIG. 2A, the system 100 can quantize floating-point numbers into a binary set {1, -1}. Binary quantization can be considered as a particular quantization process in which the system 100 uses only one digit for the sign, zero digits for the exponent power, and one digit for the significand of the floating-point number. More specifically, using binary quantization, a weight of 0.744 in floating-point format can be quantized as 1, and an activation input of -0.21 in floating-point format can be quantized as -1, just to name a few examples.

[0056] Similarly, ternary quantization is an alternative to binary quantization that can improve accuracy at the expense of larger model size. With reference to FIG. 2B, the system 100 can quantize floating-point numbers into a ternary set {1, 0, -1}. More specifically, a normalization weight in floating-point format greater than 0.66 can be quantized as 1, another normalization weight less than 0.66 and greater than -0.66 can be quantized as 0, and another normalization weight less than -0.66 can be quantized as -1.

[0057] The system 100 may apply a respective scaling factor to each quantized input and output by multiplying each quantized input or output by the respective scaling factor. Given a particular input, the system 100 may obtain the respective scaling factor based on a measure of closeness (or similarity) between the neural network being quantized and the original unquantized neural network. The system 100 may determine the scaling factor after training the modified neural network and set the scaling factor as a constant when performing inference calculations on the modified neural network.

[0058] Returning to reference to Figure 2A, the neural network modification engine 120 can generate a new machine learning model 295 (or a new portion thereof) by replacing a group 290 of network layers of an original portion of the neural network 200 with a decision tree 220 having a tree depth based at least on the number of layers in the group 290 of network layers. The decision tree can be of any type as long as it is suitable for replacing the group of network layers. For example, the decision tree can be a GradientBoost decision tree or an AdaBoost decision tree.

[0059] The system 100 may perform binary quantization on at least the input 213 to the first layer of the group of network layers 290 and the output 217 from the last layer of the group of network layers 290. Alternatively, the system 100 performs binary quantization on the entire neural network.

[0060] The decision tree 200 may receive binary quantized inputs 215 from a previous layer 210 and output quantized outputs 225 to a subsequent layer 230. More specifically, the quantized inputs 215 include binary inputs (either 1 or -1) from nodes 210a-f of the previous layer 210. Similarly, for purposes of example only, the quantized output 225 may be either an output 225a representing {1} or an output 225b representing {-1}. Each output from the decision tree is quantized as either 1 or -1 and provided to the subsequent layer 230.

[0061] Before replacing the group of network layers 290 with the decision tree 220, the system 100 may train the decision tree 220 using quantized versions of the corresponding portions of the training examples. More specifically, the system 100 may obtain a quantized version of the input to the first layer of the group of layers 290 and a quantized version of the output from the last layer of the group of layers 290 given the training examples of the original neural network. The system 100 may set the quantized version of the input as the input to the decision tree 220, set the quantized version of the output as the output from the decision tree 220, and use the quantized versions of the input and output to train the decision tree 220. tree You can train the decision 220.

[0062] FIG. 2B illustrates a portion of another example new machine learning model 255 having a decision tree with ternary quantized outputs 275.

[0063] 2A, the system 100 may replace a different group 285 of layers in a portion of the neural network 250 with a different decision tree 270 and quantize the inputs and outputs of the decision tree using ternary quantization. The quantized input 265 and the quantized output 275 include one of the value set {-1, 0, 1}. For illustrative purposes only, the quantized output 275 may be one of output 275a representing {1}, output 275b representing {0}, and output 275c representing {-1}.

[0064] For ease of illustration, in Figure 2A the number of network layer groups is three and in Figure 2B the number is five, however, the number of network layer groups replaced by decision trees may be any suitable value as determined by system 100. For example, the number may be 1, 10, or 50.

[0065] 3 is a flow diagram of an exemplary process 300 for generating a new machine learning model that is trained. For convenience, the process 300 is described as being performed by one or more computer systems located at one or more locations. For example, a neural network deployment system 100, such as the system 100 shown in FIG. 1 and suitably programmed in accordance with this specification, can perform the process 300.

[0066] The system receives 310 data representative of a neural network with multiple layers arranged in a sequence. The received data can include operations being performed by each of the network layers of the neural network and the weights of each network layer, to name just a few. The network layers are arranged in sequence such that layer outputs from a preceding layer are provided as layer inputs to a subsequent layer.

[0067] The system selects one or more groups of layers from the plurality of layers, each group of layers comprising one or more layers adjacent to one another in sequence 320. For example, the system may select three groups of layers, with a first group of layers including only one layer, a second group of layers including three layers, and a last group of layers including five layers with the penultimate layer of the neural network.

[0068] The system generates (330) a new machine learning model corresponding to the neural network by replacing each of the one or more groups of the selected layers with a respective decision tree.

[0069] Each decision tree for replacing a respective group of layers can have a tree depth based at least on the number of layers in the layer group. For example, the tree depth of a decision tree is equal to the number of layers in the layer group. As another example, the tree depth of a decision tree is 3 and the layer group replaced by the decision tree has 5 network layers.

[0070] The decision tree may include any suitable tree type, for example, the decision tree may be a GradientBoost tree or an AdaBoost tree.

[0071] The system may quantize at least the inputs to a first layer of a group of layers and the outputs from a last layer of the group of layers, and provide the quantized inputs and outputs to a respective decision tree. More specifically, for each of one or more groups of layers, a respective decision tree receives as input a quantized version of the input to a respective first layer in the group and produces as output a quantized version of the output of a respective last layer in the group.

[0072] In some implementations, the system may obtain quantized versions of the inputs and outputs to each decision tree based on the respective scaling factors. More specifically, the system may generate quantized versions of the inputs to the decision trees by multiplying the quantized inputs by the respective scaling factors. As explained above, the scaling factors are obtained based at least on a measure of similarity between the layer being quantized and the original layer.

[0073] The system trains a new machine learning model based on the training data of the original neural network (340). The system trains at least some of the layers in the original neural network that were not replaced by the respective decision trees. In some implementations, the system trains layers of the neural network that follow one or more groups of layers of the neural network in the sequence.

[0074] In some situations, the system can train the entire new machine learning model using the same, but quantized, training samples as for the original neural network. In some implementations, the system can use quantized inputs and outputs for forward propagation during training, and floating-point inputs and outputs to update the weights during backward propagation. Alternatively, the system can calculate data representing the respective gradients for each decision tree in the new machine learning model.

[0075] The system may select any suitable algorithm for selecting tier groups and replacing the tier groups with their respective decision trees. In some implementations, the system may iteratively select tier groups. More specifically, one exemplary algorithm for selecting tier groups is described below.

[0076] Assuming that the neural network includes N network layers, the system indexes each layer of the neural network according to sequence as layer i, where i ∈ [0, 1, 2, . . . , N-1].

[0077] The system sets the total number of layers (all layers) from which to select a group of layers equal to the size of the neural network (N layers).

[0078] The system randomly selects layer L from among all layers as the first layer of the group of layers. In some implementations, the system may select the first layer of the group of layers according to a layer sequence. For example, the system may start with layer 0 as the first layer of the group of layers.

[0079] For each layer in the sequence, starting from layer L up to the final layer N, the system first checks whether the current layer has already been replaced by or belongs to a decision tree.

[0080] If the system determines that the current layer is not replaced by or belongs to a decision tree, it adds the current layer to the group of layers.

[0081] The system performs quantization on the inputs to an initial layer L of the group of layers and the outputs from the current layer, and provisionally replaces layers L through the current layer with their respective decision trees. As explained above, the system can perform binary or ternary quantization on the inputs and outputs to the neural network layers. In some implementations, the system quantizes all outputs of each layer in the neural network, allowing the size of the accumulator to be limited to a specific number of decision trees.

[0082] The system updates the layer information for the layer groups and measures the performance (e.g., inference accuracy) of the modified network. Details of the layer group determination and performance measurement are described in further detail in connection with FIG.

[0083] After determining the layer groups, the system replaces the layer groups with their respective decision trees.

[0084] The system trains the remaining layers starting from the last layer of the group up to the last layer of the neural network. More specifically, the system fine-tunes the weights of the remaining layers using the corresponding portions of the same training examples used to train the original neural network.

[0085] 4 is a flow diagram of an exemplary process 400 for selecting one or more groups of neural network layers. For convenience, the process 400 is described as being performed by one or more computer systems located at one or more locations. For example, a neural network deployment system 100, such as the system 100 shown in FIG. 1, suitably programmed in accordance with this specification, may perform the process 400.

[0086] The system selects each initial layer in the neural network 410. As described above, the system can select the initial layer of the group of layers randomly or based on the sequence of the layers.

[0087] The system generates (420) a plurality of respective candidate groups, each having a respective initial layer as the first layer in the candidate group. More specifically, the system provisionally sets the respective number of layers for each candidate group having the same initial layer. For example, the system generates a first candidate group having only two layers, a second candidate group having four layers, and a third candidate group having six layers. In some implementations, the candidate groups can have consecutive numbers of layers. For example, the first candidate group includes a single layer, the second candidate group includes two layers, and the third candidate group includes three layers.

[0088] For each of the respective plurality of candidate groups, the system determines (430) a respective performance measure of the candidate group that measures the performance of a corresponding new machine learning model in which layers in the candidate group have been replaced by the respective decision tree. More specifically, the system generates a plurality of new machine learning models, each of the plurality of new machine learning models including a respective candidate group of layers replaced by the respective decision tree. The system performs an inference calculation for each of the new machine learning models using the same input data, and obtains a respective performance score for each of the new machine learning models. vinegar Core (performances s cores) can be obtained based on the inference accuracy.

[0089] The system selects 440 one of the candidate groups of layers as the group of layers to be replaced by the decision tree based on the respective performance measure for each of the plurality of candidate groups. In some implementations, the system determines a maximum performance measure among the respective performance measures and selects as the group from each of the plurality of candidate groups the candidate group associated with the maximum performance measure. Alternatively, the system selects the candidate group that has a relatively high performance score but is the fastest to perform the inference calculation.

[0090] FIG. 5 illustrates an example of a decision tree 500. As shown in Figure 5, the decision tree 500 may include multiple nodes having a particular tree depth. The tree depth is determined based on the total number of tree layers. tree Each layer of the decision Tree ) may include nodes that represent respective node operations, such as logical comparisons or other suitable criteria.

[0091] For example, as shown in Figure 5, a decision tree 500 may have multiple nodes, including a root node 510 and respective non-root nodes 520, 530 in different tree layers. The root node 510 is the starting point of the decision tree 500 and has no parent node, while the non-root nodes 520 and 530 each have a parent node and are also called child nodes. In general, each node except the leaf nodes in a tree layer may have one or more branches (arrows in Figure 5) connecting the node to respective child nodes in the next tree layer. Nodes that have child nodes are also called non-leaf nodes, e.g., nodes 510, 520a, and 520b.

[0092] For the leaf nodes in the deepest or final tree layer (e.g., nodes 530a, 530b, 530c, and 530d), each may represent an inference output of decision tree 500 and have no branches connecting to other child nodes. The inference output may be a prediction, e.g., the probability P1 of leaf node 530a.

[0093] Referring to FIG. 5, a system (e.g., system 100 in FIG. 1) may include a coefficient n f1 , n f2 , and n f3 When performing an inference operation on a decision tree using input data representing , each non-leaf node except the leaf node in the deepest tree layer has a respective node operation (e.g., a logical comparison as shown in FIG. 5). The system can perform the node operation on the root node 510, obtain a logical result (e.g., true or false) from the node operation, and approach corresponding child nodes along each branch based on the result. For example, if the result is false, the system approaches node 520a. In some implementations, the system can use the integers 0 and 1 to represent false and true. Finally, the system approaches a particular leaf node in the deepest layer of the decision tree and returns the inference represented by the particular leaf node.

[0094] FIG. 6 illustrates an exemplary implementation of a decision tree using fixed functionality hardware 600.

[0095] As shown in Figure 6, a system (e.g., system 100 shown in Figure 1) can perform inference computations for a decision tree using fixed functional hardware 600. More specifically, the system can receive input x 610 and return a final inference output 620 for the example decision tree shown in Figure 5. Decision tree 600 can include multiple computation units 630 and 640 to obtain inference results from different nodes and generate the final inference output 620, which is represented by a particular leaf node in the decision tree.

[0096] A system using fixed function hardware 600 can assign each input data to a corresponding tree node and perform a respective function at each node using respective comparators and multiplexers. As shown in FIG. 6, the system can assign each input data x ∈ [h 1, l1] to the top node (corresponding to the root node 510 in FIG. 5), and the input data x ∈ [h 2, l2] to the left node 630a (corresponding to the left non-leaf node 520a in the second tree layer), and the input data x ∈ [h 3, l3] to right node 630b (corresponding to right non-leaf node 520b in the second tree layer). The system can perform the operations at each node using respective comparators 640 and multiplexers 630. For example, the system can assign comparators 640c and multiplexer 630c to perform the operation at the top node. More specifically, the system receives an input x, which may be a vector of real values, and a range x ∈ [h 1,l1] to provide to the top root node. The system uses a comparator 640c to compare the portion of the input with a criterion a1, which may represent a real-valued scalar. If the input data x is greater than or equal to a1, the system uses a multiplexer 630c assigned to the node to output a result representing "true", e.g., the multiplexer may output an integer 1 representing "true". The system may continue to perform operations on child nodes to which the current node links along the corresponding tree branch. After processing operations on non-leaf nodes in the penultimate tree layer, the system may output a final result 620 based on the values ​​(e.g., probabilities) represented by the corresponding leaf nodes in the decision tree.

[0097] FIG. 7 illustrates an example programmable core 700 for performing decision tree inference calculations.

[0098] As shown in FIG. 7, a system (e.g., system 100 shown in FIG. 1) can use a programmable core 700 to perform inference computations for a decision tree. The programmable core 700 can receive tree inputs 710 and generate inference outputs 720 for the tree inputs. The programmable core 700 can include multiple computation components, e.g., a MUX unit, an ALU unit, and a static random access memory (SRAM). The programmable core 700 can perform node operations by simply adding, selecting, and switching functions configured in the computation components. Thus, the programmable core 700 does not need to include a MAC unit to perform multiplication and addition as node operations for generating inference outputs of a decision tree.

[0099] Referring to FIG. 7, to perform an operation at a root node (e.g., root node 510 in FIG. 5), the system receives an input 710 and combines the received input 710 with an array (x, a, l, h, i) that is specific to the root node and previously stored in the queue by a combine unit. 0 , i 1 ) can be used to modify the node criterion. As explained above, a represents the node criterion, l and h represent the range of values ​​of the input x, and the index i 0 , i 1 Each of the n may represent a tree branch connecting the current node to a respective child node, and may be assigned a respective integer value to represent the result of performing an operation on the node operation.

[0100] The system can use any unit to select the combined inputs and perform the node operation at the root node. For example, the system can use each comparator to perform a comparison between the input x∈[h,l] and the criterion a1. In response, the system can select the inputs with index i 0 , i 1 We can assign,i,to each integer value to represent the result of the comparison and direct,the computation of the decision tree to the corresponding child node.,For example, if the system determines that x,∈[h,l] is,greater than the criterion a,, the system assigns,i, 0 =0, i 1 = 1, the system can assign i 1 An operation can be performed on the corresponding next child node along the tree branch represented by

[0101] To perform an operation on the next child node, the system can first apply a switching unit to determine whether the next child node is a non-leaf node or a leaf node based on a numbering criterion N. To determine, the system can number each node with a respective tag K (e.g., an integer) and determine the type of the next child node based on a predetermined numbering criterion. For example, with reference to FIG. 5, the system can tag nodes 510, 520a, and 520b as node 0, node 1, and node 2, and tag leaf nodes 530a, 530b, 530c, and 530d as node 3, node 4, node 5, and node 6. The system can set the numbering criterion N=3, such that a node with a tag K<3 is a non-leaf node, and a node with a tag K>=3 is a leaf node.

[0102] In response to determining that the next child node is a non-leaf node, the system selects from the data stored in the non-leaf SRAM the respective arrays (x, a, l, h, i 0 , i 1 ) may be updated. Similarly, in response to determining that the next child node is a leaf node, the system may provide a final output associated with the leaf node that is stored in the leaf SRAM.

[0103] For parallel computation, the system can employ a fork function to store a portion of the input data of a node in a queue and a respective array identifying the operation and type of the node in SRAM. During parallel computation, the system can automatically determine when to modify the input data with the respective array using one or more coupling units based on the respective latencies observed for each computation unit.

[0104] The embodiments referred to herein provide an improved method for training a neural network. A neural network may be configured to receive any kind of digital data input and generate any kind of score, classification, or regression output based on the input. The input data items may comprise image data (including here video data), audio data, or text data, such as words or word parts (or representations, e.g., embeddings) of a natural language. The input data items may comprise continuous data, such as a sequence of data samples representing digitized audio, or an image represented as a sequence of pixels, or a video represented by a sequence of images, or a sequence representing a sequence of words in a natural language. As used herein, "image" includes, for example, LIDAR images.

[0105] In some implementations, the neural network output may comprise feature representations, which may then be further processed to generate a system output. For example, the system output may comprise a classification output for classifying the input data items into one of a number of categories, e.g., image, video, or audio categories (e.g., data representing an estimated likelihood that the input data items or objects / elements of the input data items belong to a category), or a segmentation output for segmenting regions of the input data items into objects or actions, e.g., represented in an image or video. Or, the system output may be an action selection output in a reinforcement learning system.

[0106] In some other implementations, the network output may comprise another data item of the same or different type. For example, the input data item may be an image, audio, or text, and the output data item may be a modified version of the image, audio, or text, e.g., modifying the style, content, properties, pose, etc. of the input data item, or one or more objects or elements within the input data item, or filling in (missing) parts of the input data item, or predicting another version of the data item, or an extension of a video or audio data item, or providing an upsampled (or downsampled) version of the input data item. For example, the input data item may be a representation of a text in a first language, and the output data item may be a translation of the text into another language, or a score for the translation of the text into another language. In another example, an input image may be converted into a video, or a wireframe model, or a CAD model, or a 2D input image may be converted into 3D or vice versa. Or, the input data items may comprise features derived from or derived from verbal utterances or sequences of verbal utterances, and the network system output may comprise a score for each of a series of text portions, each score representing an estimated likelihood that the portion of text is a correct replacement based on the features. In another example, the input data items may be images, audio, or text, and the output data items may be representations of the input data items in different formats. For example, a neural network may convert text to speech or vice versa (for speech recognition), or convert images (or videos) to text (e.g., for captioning). When generating an output comprising continuous data, the neural network may include one or more convolutional, e.g., dilated convolutional layers.

[0107] In some other implementations, the network output may comprise an output for selecting an action to be performed by an agent, such as a robot or other mechanical agent, in an environment, such as a real-world environment or a simulation of a real-world environment.

[0108] In some implementations, the neural network is configured to receive an input data item and process the input data item to generate a feature representation of the input data item according to the network parameters. In general, the feature representation of a data item is an ordered collection of numerical values, e.g., a vector, that represents the data item as a point in a multidimensional feature space. In other words, each feature representation may include a numerical value for each of a number of features of the input data item. As previously mentioned, the neural network may be configured to receive any type of digital data input as an input and generate a feature representation from the input. For example, the input data item, which may be referred to as a network input, may be an image, a portion of a document, a text sequence, audio data, medical data, etc.

[0109] Once trained, the feature representations can provide input to another system, e.g., for use in performing machine learning tasks on the network input. Example tasks can include feature-based search, clustering, near-duplicate detection, validation, feature matching, domain adaptation, video-based weakly supervised learning, and in the case of video, e.g., object tracking across video frames, gesture recognition of gestures being performed by entities depicted in the video.

[0110] If the input to the neural network is an image or features extracted from the image, the output generated by the neural network for a given image may be a score for each of a set of object categories, each score representing an estimated likelihood that the image contains an image of an object belonging to the category. More specifically, each of the input image or features extracted from the input image may include one or more pixels, each having a respective intensity value. The neural network is configured to process each intensity value of the input image or features extracted from the image to generate a prediction, such as, for example, image classification, image recognition, or image segmentation.

[0111] As another example, if the inputs to a neural network are Internet resources (e.g., web pages), documents, or portions of documents, or features extracted from Internet resources, documents, or portions of documents, the output generated by the neural network for a given Internet resource, document, or portion of a document may be a score for each of a set of topics, each score representing an estimated likelihood that the Internet resource, document, or portion of a document is related to that topic.

[0112] As another example, if the input to a neural network is features of the impression context of a particular ad, the output generated by the neural network may be a score representing the estimated likelihood that a particular ad will be clicked.

[0113] As another example, if the inputs to the neural network are features of a recommendation personalized for a user, e.g., features characterizing the context of the recommendation, e.g., features characterizing previous actions taken by the user, the output generated by the neural network can be a score for each of a set of content items, each score representing an estimated likelihood that the user will respond positively to the content item being recommended.

[0114] As another example, if the input to a neural network is a sequence of text in one language, the output generated by the neural network may be a score for each portion of the sequence of text in another language, each score representing an estimated likelihood that the portion of text in the other language is an appropriate translation of the input text into the other language.

[0115] As another example, if the input to a neural network is a sequence representing a verbal utterance, the output generated by the neural network may be a score for each of the series of text portions, each score representing an estimated likelihood that the portion of text is a correct transcription of the utterance.

[0116] The subject matter and functional operations described herein may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. The subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded in a tangible non-transitory storage medium for execution by or control of the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random access memory device or a serial access memory device, or one or more combinations thereof. Alternatively, or in addition, the program instructions may be encoded in an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiving device for execution by the data processing device.

[0117] The term "data processing apparatus" refers to data processing hardware and encompasses any type of apparatus, device, and machine for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. The apparatus may also be or even include special purpose logic circuitry, such as, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0118] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, such as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may correspond to a file in a file system, but need not. A program can be stored in a single file dedicated to the program in question, in part of a file holding other programs or data, for example one or more scripts stored in a markup language document, or in multiple coordinated files, for example a file storing one or more modules, subprograms, or code portions. A computer program can be deployed to be executed in one computer or in multiple computers located at one site or distributed across multiple sites and interconnected by a data communication network.

[0119] As used herein, the term "database" is used broadly to refer to any collection of data, which need not be structured in any particular manner, or at all, and which may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different manner.

[0120] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components and installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine, and in other cases, multiple engines may be installed and executed on the same computer or computers.

[0121] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs that perform functions by performing operations on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as, for example, an FPGA or ASIC, or a combination of special purpose logic circuitry and one or more programmed computers.

[0122] A computer suitable for executing a computer program may be based on a general-purpose or dedicated microprocessor or both, or any other type of central processing unit. In general, the central processing unit receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for executing or implementing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated in dedicated logic circuitry. In general, a computer also includes one or more mass storage devices for storing data, e.g., magnetic disks, magneto-optical disks, or optical disks, or is operatively coupled to receive data from them, transfer data to them, or both. However, a computer need not have such devices. Furthermore, a computer may also be incorporated in another device, such as, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name a few.

[0123] Computer readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, CD ROM disks, and DVD-ROM disks.

[0124] To provide interaction with a user, embodiments of the subject matter described herein can be implemented in a computer having a display device, such as, for example, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and a pointing device, such as, for example, a mouse or trackball, for allowing the user to provide input to the computer. Other types of devices can also be used to provide interaction with the user, for example, the feedback provided to the user can be any form of sensory feedback, such as, for example, visual feedback, auditory feedback, or tactile feedback, and the input from the user can be received in any form, including acoustic, voice, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device being used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, such as, for example, a smartphone running a messaging application, and receiving a response message from the user in return.

[0125] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units for handling typical, computationally intensive parts of, for example, machine learning training or production, i.e., inference, workloads.

[0126] The machine learning models can be implemented and deployed using machine learning frameworks such as, for example, the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0127] An embodiment of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., as a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an embodiment of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0128] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, the server transmits data, e.g., HTML pages, to a user device, e.g., for the purpose of displaying data to and receiving user input from a user interacting with the device acting as a client. Data being generated at the user device, e.g., results of user interaction, may be received at the server from the device.

[0129] Although the present specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, although features may be described above as operating in a particular combination and may initially be claimed as such, in some cases one or more features from the claimed combination may be excluded from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0130] Similarly, although operations are shown in the figures and claimed in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown, or in any sequential order, or that all of the illustrated operations be performed, to achieve desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0131] Specific embodiments of the subject matter are described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. 1. A method implemented by one or more computers, the method comprising: receiving data representative of a neural network comprising a plurality of layers arranged in a sequence; selecting one or more groups of layers from the plurality of layers; each group of layers comprises one or more layers adjacent to one another in said sequence; The method comprises: and generating a new machine learning model corresponding to the neural network, the generating the new machine learning model comprising:

1. A method for computing a quantized decision tree for each group of layers, comprising: selecting, for each group of layers, a respective decision tree to replace the group of layers, each of the decision trees receiving as input a quantized version of an input to a respective first layer in the group and producing as output a quantized version of an output of a respective last layer in the group, a tree depth of each of the decision trees being based at least in part on a number of layers in the group.

2. 2. The method of claim 1, further comprising training the new machine learning model by training at least a portion of the layers in the neural network that were not replaced by a respective decision tree based on training data of the neural network.

3. Training at least a portion of the layers in the neural network that were not replaced by a respective decision tree includes: The method of claim 2 , further comprising training the layer of the neural network subsequent to the one or more groups of layers of the neural network in accordance with the sequence.

4. Selecting each of the one or more groups of layers comprises: selecting a respective initial layer in the neural network; generating respective plurality of said candidate groups, each having a respective said initial layer as a first layer in said candidate group; determining, for each of the respective plurality of candidate groups, a respective performance measure of the candidate group that measures the performance of a corresponding new machine learning model having the layer in the candidate group replaced by a respective decision tree; and for each of said plurality of candidate groups, selecting one of said candidate groups as said group based on a respective performance measure.

5. Selecting each of the initial layers in the neural network to generate a respective group of layers includes: The method of claim 4 , comprising selecting each of the initial layers by a random process or based on the sequence of the neural networks.

6. Selecting one of the candidate groups as the group based on a respective performance measure includes: determining a maximum performance measure from each of said performance measures; and selecting as said group from each of said plurality of candidate groups the candidate group associated with said maximum performance measure.

7. 7. The method of claim 1, wherein the quantized version of the input to each first layer in the group and the quantized version of the output of each last layer in the group are generated using binary or ternary quantization.

8. The method of any one of claims 1 to 7, wherein each decision tree layer replacing the group of layers comprises a GradientBoost decision tree or an AdaBoost decision tree.

9. Each layer has a respective set of weights, and the method comprises:

9. The method of claim 1, further comprising, for each layer in the neural network that is not in the one or more groups of layers, quantizing at least a portion of the weights associated with the layer.

10. The method of any one of claims 1 to 9, wherein the tree depth of each of the decision trees is equal to the number of layers in the group.

11. The method of any one of claims 1 to 10, wherein the quantized version of the input to each first layer in the group or the quantized version of the output of each last layer in the group is generated by one or more scaling factors.

12. the neural network represented by the received data is a neural network that has been initially trained with a training data set; each decision tree replacing a respective group of layers is trained based on a portion of a respective quantized version of the training dataset; 12. The method of claim 1, wherein each training sample of the respective portion of the quantized version of the training dataset comprises (i) a quantized version of a layer input to the first layer of the group, and (ii) a quantized version of a layer output from the last layer of the group.

13. The method according to any one of claims 1 to 12, further comprising outputting the new machine learning model to a system configured to implement the new machine learning model, the system comprising one or more computing units for implementing the decision tree with one or more functions selected from an add function, a select function, or a switch function.

14. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 13.

15. A program for causing one or more computers to execute the method according to any one of claims 1 to 13.

Images