Integrating decision trees into neural networks

Abstract

To provide methods, systems, and apparatus, including computer programs encoded on computer storage media, for scheduling operations represented on a computation graph.SOLUTION: One of methods comprises: receiving data representing a neural network comprising a plurality of layers arranged in a sequence; selecting one or more groups of layers each comprising one or more layers adjacent to each other in the sequence; and, for each group of layers, using respective decision trees, replacing the group of layers. Each of the decision trees receives, as input, a quantized version of the inputs to respective first layers in the group and generates, as output, a quantized version of the outputs of respective last layers in the group. A tree depth of each decision tree is based at least in part on the number of layers of the group.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Technical field This specification relates to the incorporation of decision trees into large-scale neural networks. [Background technology]

[0002] background A neural network is a machine learning model that applies layers of one or more nonlinear units to predict an output for a given input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from a given input according to the current values ​​of its respective set of parameters.

[0003] More specifically, each neural network layer contains multiple nodes, and each layer represents a set of operations defined by the neural network. Generally, these operations are arithmetic operations that can include linear operations such as addition and multiplication, and nonlinear operations such as nonlinear activation functions such as the "Relu" or "Sigmoid" function. Linear operations connect the layer inputs to the layer weights. Each layer's linear operation can be performed using tensor operations, where the layer weights are represented in matrix or tensor form and the layer inputs are represented in vector form.

[0004] Large-scale neural networks, i.e., neural networks with many layers and numerous parameters, demonstrate excellent performance in various machine learning tasks. However, these large-scale neural networks can have high latency and consume a large amount of computational resources; for example, they require a large amount of memory to make predictions and consume a considerable number of processor cycles. One conventional technique to improve the computational efficiency of neural networks is to manipulate a portion of the weight matrix to make it a sparse matrix. A sparse matrix is ​​a matrix in which many terms are zero. [Overview of the project]

[0005] overview This specification describes a technique for generating new machine learning models by incorporating decision trees into large-scale neural networks.

[0006] In general, one innovative aspect of the subject matter described herein can be embodied in a manner that includes the following operations: the operation of receiving data representing a neural network having a plurality of layers arranged in a sequence; the operation of selecting one or more groups of layers from the plurality of layers, each group of layers having one or more layers adjacent to each other in this sequence; and the operation of generating a new machine learning model corresponding to the neural network.

[0007] Generating a new machine learning model involves selecting a decision tree for each group of layers that replaces the existing group of layers. Each decision tree takes a quantized version of the input to each first layer in the group as its input and produces a quantized version of the output to each last layer in the group as its output. The tree depth of each decision tree is at least partially based on the number of layers in the group.

[0008] Other embodiments of this aspect include a corresponding computer system, an apparatus, and a computer program recorded in one or more computer storage devices, each configured to perform the operation of the method.

[0009] Each of the embodiments described above and other embodiments may optionally include one or more of the following features, either individually or in combination. In particular, one embodiment includes all of the following features in combination.

[0010] This method can further include training a new machine learning model by training at least some of the layers in the neural network that were not replaced by each decision tree, based on the neural network's training data.

[0011] As discussed above, the operation of selecting each of one or more groups of layers may further include the operation of selecting each initial layer in the neural network, generating each of several candidate groups, each having its initial layer as the first layer in the candidate group, determining each performance metric for each of the several candidate groups, which measures the performance of the corresponding new machine learning model having layers in the candidate group replaced by its decision tree, and selecting one of the candidate groups as a group based on its performance metric for each of the several candidate groups.

[0012] The selection of each initial layer can be carried out by a random process or based on the sequence of the neural network. The quantized versions of the inputs to each first layer in the group, and the quantized versions of the outputs to each last layer in the group, can be generated using binary or ternary quantization. Each decision tree layer replacing a group of layers can include a GradientBoost decision tree or an AdaBoost decision tree.

[0013] The method may further include outputting a new machine learning model to a system configured to implement the new machine learning model, which comprises one or more computing units for implementing the decision tree by one or more functions selected from additional, selection, or switching functions. In other words, a new machine learning model (e.g., being trained) may be output to a system that includes one or more computing units (such as multiplexers or arithmetic logic units) for implementing the decision tree, without requiring more expensive computing units, such as multiplication accumulator units (MACs), to perform multiplication.

[0014] The subject matter described herein can be implemented in particular embodiments to achieve one or more of the following advantages:

[0015] The system described, which implements the techniques described below, can reduce computational costs and improve the efficiency of performing inference computations for large-scale neural networks.

[0016] Firstly, the described technique for replacing one or more network layers of a large neural network with a decision tree can reduce the computational load when the system performs inference computations for the neural network. A decision tree that replaces one or more layers of a network can have only one or a few layers (e.g., treestamps or shallow trees). The computational cost of performing operations on treestamps and shallow trees is far less than the computational cost required to compute the neural network layers of a large neural network. As another example, if the decision tree is an Adaboost tree, the system can be more efficient by performing fewer operations than a conventional neural network layer, which typically requires both multiplication and addition, because it does not need to perform multiplication operations for Adaboost trees.

[0017] Secondly, the techniques described can reduce the total size of a neural network by quantizing at least the inputs and outputs of network layers that are replaced by one or more decision trees. Quantizing these inputs and outputs reduces the number of significant digits involved in the computation, thereby reducing the computational cost, in particular, of the inserted decision trees and at least the neural network layers adjacent to the decision trees (i.e., layers preceding or following the decision trees). Because the size of the neural network is reduced by quantization, the total memory / storage requirements of the computing system can also be reduced. With this in mind, the techniques described enable devices with less memory and computing power (e.g., smartphones, tablets) to efficiently perform inference computations of modified neural networks. In some scenarios, one or more hardware accelerators in a device can be customized to perform inference computations of a particular modified neural network (i.e., one or more layers are replaced by one or more decision trees, and the inputs and outputs of one or more layers are quantized), thereby reducing the device's memory usage, lowering power consumption, and enabling more efficient and faster inference computations.

[0018] In addition, often when performing inference calculations of a neural network, the high precision provided by non-quantized inputs and outputs is not necessary in order to accurately detect and represent the presence or absence of important features. That is, the error introduced by quantizing the inputs and outputs to the layers replaced by decision trees is minimal compared to the improvement in efficiency.

[0019] In fact, quantized data is being used to train and calculate neural networks, but it is important for the techniques described below to quantize the inputs and outputs to be suitable for each decision tree and the remaining neural network layers. More specifically, since the system can quantize floating-point numbers and reduce the number of digits representing the floating-point numbers, the system can use fewer digits for the sign, exponent, and mantissa of the floating-point numbers. In the case of binary quantization and ternary quantization, for some examples, the system can map the floating-point number to an integer such as {1, -1} or {1, 0, -1}.

[0020] Also, the techniques described can efficiently train the modified neural network (i.e., the new neural network where layers are replaced by decision trees). The system can simply fine-tune the parameters in the modified neural network using at least some of the same training examples used to train the original neural network. Therefore, the time period required to train the modified neural network can be significantly shortened compared to training the original neural network.

[0021] Furthermore, the techniques described can reduce costs by using low-cost programmable hardware to perform the operations of the decision trees. For example, the computing system The tem can use a multiplexer (MUX) unit to calculate the operations in the decision tree instead of a multiply-accumulator unit (MAC). It is known to those skilled in the art that the MUX unit consumes less power and space than the MAC unit. Therefore, the computing system can include only programmable hardware units suitable for the modified neural network, rather than expensive hardware accelerators such as GPUs or TPUs. Therefore, the total cost of constructing a hardware system for executing the inference calculation of the modified neural network is much less than that of the original neural network.

[0022] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

Brief Description of the Drawings

[0023] [Figure 1] FIG. shows an exemplary neural network deployment system including an exemplary neural network modification engine. [Figure 2A] FIG. shows a part of an exemplary new machine learning model having a decision tree with a binary quantization output. [Figure 2B] FIG. shows a part of another exemplary new machine learning model having a decision tree with a ternary quantization output. [Figure 3] FIG. is a flowchart of an exemplary process for generating a trained new machine learning model. [Figure 4] FIG. is a flowchart of an exemplary process for selecting one or more groups of neural network layers. [Figure 5] FIG. shows an example of a decision tree. [Figure 6] FIG. shows an exemplary embodiment of a decision tree using fixed functional hardware. [Figure 7] This figure shows an example programmable core for performing inference computations for a decision tree. [Modes for carrying out the invention]

[0024] Detailed explanation Similar reference numbers and designations in various drawings refer to the same elements.

[0025] One conventional approach to improving efficiency in training and computation of large neural networks is to construct a sparse matrix for each layer's activation inputs and weights. However, the approach of constructing sparse matrices can introduce new problems. For example, using sparse matrices can cause the computing system to lose access to the same memory address over a period of time, a phenomenon also known as a lack of inference locality. More specifically, when constructing sparse matrices, the computing system stores the non-zero terms of the original matrix at different memory addresses that may be physically far apart from each other. In some situations, the memory address of each term in a sparse matrix may change dynamically during computation. For example, a zero term may become non-zero after being added to another non-zero term. Consequently, when the computing system accesses data stored in memory, it may suffer from memory latency, cache tracking, and even cache contamination, ultimately reducing the computational efficiency of training and input processing with large neural networks.

[0026] Alternatively, another conventional approach involves integrating the weight matrix into the layer logic. For example, a system employing this fused weighting technique would have the following characteristics in the weight matrix: Since the zero term can be determined, multiplication and accumulation operations are not performed on this term. Considering this, fused weighting techniques can reduce the computational cost of large neural networks by not performing calculations where the output is zero because one of the inputs is zero. However, fused weighting comes at a cost. Firstly, fused weighting techniques must not be modified once the neural network is deployed on a hardware accelerator; that is, the zero term in the weight matrix must remain zero during computation. However, in practice, the weight matrix of the deployed neural network may need to be fine-tuned with new training data. Secondly, if the zero term in the sparse weight matrix is ​​shared by multiple operations, such as multiplication followed by addition and then another multiplication, the system still needs to perform operations for the zero term.

[0027] The techniques described below can address the above problems. More specifically, the techniques described can efficiently perform one or more inference computations for large neural networks using quantization and decision trees. In general, the techniques described relate to replacing one or more layers of a neural network with their respective decision trees, where the inputs to and outputs from the decision trees are quantized versions of the inputs and outputs of the corresponding layers.

[0028] This specification describes a system, implemented as a computer program on one or more computers in one or more locations, that generates a new machine learning model by replacing one or more groups of network layers of a neural network with one or more decision trees, in order to reduce the computational cost and improve efficiency for performing inference calculations using economical hardware.

[0029] Figure 1 shows an exemplary neural network deployment system 100, which includes an exemplary neural network modification engine 120.

[0030] Generally, the neural network deployment system 100 takes data 110 representing a neural network as input and outputs a new machine learning model 180 being trained. The neural network deployment system 100 includes a neural network modification engine 120 for generating a new machine learning model 130 for the input neural network model. The new machine learning model 130 is a hybrid of the original input neural network model and one or more neural network layers that are replaced by one or more decision trees. Details of the generation of the new machine learning model 130 are described below. The neural network deployment system 100 also includes a training engine 140 configured to train the new machine learning model using training data 150, and a memory 160 configured to store and provide data for the training engine 140 (e.g., training and output data for the machine learning model, as well as data defining the machine learning model).

[0031] More specifically, the data 110 representing the neural network received by the deployment system 100 may include information that defines the neural network, such as the operations of each layer of the neural network and the weights of each network layer.

[0032] Data 110 can also represent other embodiments of a neural network. For example, data 110 may include the number of network layers in the neural network, the number of nodes in each layer, data representing one or more types of inter-layer connections, such as element-wise connections or full connections, and data representing the type of each layer in the neural network, such as a pooling layer, a fully connected layer, or a SoftMax layer. It is possible.

[0033] Generally, data 110 can represent a trained neural network that has been trained on multiple training data 150, or a neural network that has not yet been trained.

[0034] The neural network deployment system 100 can provide the received data 110 to the neural network modification engine 120. The modification engine 120 can select one or more groups of layers in the neural network, replace each group of layers with its respective decision tree, and output a new machine learning model 130. The selection of one or more groups of layers is described in more detail below.

[0035] Each decision tree that replaces each group in the layer may be stored in memory 160 and accessible for the change engine 120. More specifically, the data representing the decision tree and stored in memory 160 may include data specifying the total number of nodes (e.g., the root and multiple leaves), the connectivity between nodes (e.g., how a leaf is connected to one or more other leaves), and one or more node operations (e.g., logical comparisons for one or more nodes).

[0036] The change engine 120 can automatically determine the decision tree for replacing each group of layers. Alternatively, the type of decision tree for replacing each group of layers can be predetermined by the user or by a computer program running on one or more computers outside the change engine 120. The decision tree can be a GradientBoost tree or an AdaBoost tree.

[0037] A GradientBoost (or gradient boost) tree can be obtained through gradient boosting, a machine learning method that creates a model in the form of a combination (e.g., a weighted sum) of one or more simple predictive models (e.g., decision trees). AdaBoost (or Adaptive Boosting) adaptively combines one or more simple predictive models (e.g., decision trees) so that during training, a larger weight is assigned to the less performing simple predictive models (e.g., measures of misclassification), making the model being trained by AdaBoost more likely to produce accurate predictions given a particular input.

[0038] The modification engine 120 can also quantize at least a portion of the neural network represented by the data 110. For example, the modification engine 120 can quantize the input to the first layer of a group of layers that will be replaced by a decision tree, and the output from the last layer of the group of layers. Alternatively, the modification engine 120 can quantize the entire neural network so that the inputs and outputs to each layer are quantized. Furthermore, the modification engine 120 can quantize at least a portion of the training data 150 and use the quantized training data to train each decision tree, or at least a portion of a new machine learning model 130, or both.

[0039] Quantization is the process of mapping a large set of input values ​​to a smaller set of output values, and is typically used for rounding and truncation. More specifically, quantization can be used to reduce the precision of numbers. For example, quantization can reduce the precision of a floating-point number from 32 bits to 8 bits. In the context of neural networks, this can be used to modify... The 120 can quantize the activation tensor, weight tensor, or layer output of a neural network layer with precision ranging from 8 bits to 4 bits, and even 1 bit (e.g., 1 bit in the mantissa). Details of the quantization (e.g., binary and ternary quantization) are described in relation to Figures 2A and 2B.

[0040] The modification engine 120 can provide the new machine learning model 130 to the training engine 140. The training engine 140 can then train at least a portion of the new machine learning model 130 based on the training data 150 used to train the original neural network 110, and output the newly trained machine learning model 180 as the output of the system 100. More specifically, the training engine 140 can use the training data 150 to train the remaining layers of the neural network that have not been replaced by decision trees over the time period. Alternatively, or in addition, the training engine 140 can train the entire new machine learning model 130 based on the quantized training data, assuming the respective gradients of each decision tree.

[0041] The time required to train a new machine learning model 130 can range from minutes, hours, or days. Alternatively, the time period can also be based on the size of the training data 150 used to train at least a portion of the new machine learning model 130. For example, the time period can be determined by the time required to train (e.g., fine-tune) the new machine learning model 130 using 100 minibatches or 1000 minibatches of the training data 150.

[0042] The training engine 140 can train a decision tree that replaces one or more layers of the original neural network.

[0043] More specifically, the training engine 140 trains the decision tree before replacing the network layers of the original neural network with the decision tree. The training engine 140 can also fine-tune the decision tree in the new machine learning model 130, as described above.

[0044] To train the decision tree, the training engine 140 can use the corresponding portion of the same but quantized training data 150 used to train the original neural network as training data. Specifically, the training engine 140 trains the decision tree using each training sample. Each training example includes a quantized version of the layer input to the first layer of a group of layers and a quantized version of the layer output from the last layer of the group of layers. The groups of layers are replaced by the decision tree. The quantized versions of the layer inputs and layer outputs are associated with the layer inputs and layer outputs of the corresponding training data 150 used to train the groups of layers in the original neural network.

[0045] When training a decision tree, the training engine 140 can define a loss and adjust node operations to reduce the loss until it falls below a certain threshold. The loss could be a hinge loss indicating label errors, a log loss representing information acquisition based on entropy theory, or any other loss suitable for training a decision tree. The training engine 140 can adjust node operations such as the critical values ​​for each logical comparison operation in one or more nodes. In some situations... The training engine 140 can reduce overfitting by pruning the decision tree during training, i.e., by removing one or more leaves and all branches and child leaves associated with those leaves. The training engine 140 may retain a portion of the original training dataset as a validation set for detecting and improving overfitting.

[0046] The training engine 140 used to train at least a portion of the new machine learning model 130 may include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), or any other computing unit suitable for performing neural network operations. In particular, the new machine learning model may still have unreplaced layers, including linear operations (e.g., primarily tensor operations), and therefore the training engine 140 may include more TPUs than CPUs or GPUs to facilitate the training process.

[0047] The newly trained machine learning model 180 can be used to efficiently generate inferences given an input. The newly trained machine learning model 180 can be used to generate inferences using less computational power. For example, because one or more layers of the neural network are replaced by shallower decision trees with fewer nodes, the newly trained machine learning model 180 requires less memory to store than the original neural network. As another example, the newly trained machine learning model is compatible with quantized inputs and outputs represented using fewer bits, thus reducing system memory bandwidth during computation. Furthermore, because computational units such as MUX units or arithmetic logic (ALC) units can be used to perform operations on the decision tree, the neural network inference engine or system can replace large and expensive computational units such as TPUs or GPUs with smaller and less expensive programmable cores having one or more MUX units or ALC units to perform the decision tree inference computations of the newly trained machine learning model 180. Thus, the total cost and size of the device for generating inferences of the new machine learning model can be reduced.

[0048] Figure 2A illustrates a portion of an exemplary new machine learning model 295 having a decision tree with a binary quantized output 225.

[0049] As shown in Figure 2A, a portion of the original neural network 200 represented by the input data 110 includes multiple network layers. The multiple network layers may include a group of network layers 290 determined by the system 100 to be replaced by a corresponding decision tree, a first network layer 210 preceding the first layer of the group of network layers 290, and a second network layer 230 following the last layer of the group of network layers.

[0050] Each layer within a portion of the neural network 200 has multiple nodes, each representing linear and nonlinear operations. For example, network layer 210 contains nodes 210a to 210f. Another example is network layer 230 containing nodes 230a to 230f. Also, each layer in group 290 of network layers contains a certain number of nodes (not shown).

[0051] Some network layers of the neural network 200 use the output of the preceding layer, which generates a layer output for each input to the neural network, as the layer activation input. They are arranged in sequence so that they can be provided to subsequent layers. For example, the first layer of group 290 of network layers receives from network layer 210 as a layer activation input 213. As another example, the last layer of group 290 of network layers provides a layer output 217 to the subsequent layer 230.

[0052] The inputs and outputs can have their respective precisions according to the computational requirements. For example, the inputs and outputs can be in 32-bit precision floating-point format. As another example, the inputs of the first few layers of a neural network and the outputs from the last few layers may have higher precision than the hidden layers.

[0053] In some embodiments, system 100 can quantize the inputs and outputs of one or more layers of the neural network using different quantization methods to reduce precision. For example, the activation input 213 and layer output 217 can be quantized using binary or ternary quantization, and may have a precision of 8 bits or even 1 bit. Alternatively, system 100 can quantize only a portion of the weight inputs or activation inputs for the network layers.

[0054] As explained above, quantization is the process of reducing the precision of a number (for example, by reducing the number of bits that represent the sign, exponent, and mantissa of a number). Binary quantization and ternary quantization are branches of the quantization process.

[0055] With respect to binary quantization, in relation to the neural network deployment system 100 and Figure 2A, system 100 can quantize floating-point numbers into a binary set {1, -1}. Binary quantization can be thought of as a specific quantization process in which system 100 uses only one digit for the sign, zero digits for the exponent, and one digit for the significant digits of the floating-point number. More specifically, to give just a few examples, using binary quantization, the floating-point weight 0.744 can be quantized as 1, and the floating-point activation input -0.21 can be quantized as -1.

[0056] Similarly, ternary quantization is an alternative to binary quantization, and although the model size is larger, it can improve accuracy. In relation to Figure 2B, system 100 can quantize floating-point numbers into a ternary set {1, 0, -1}. More specifically, normalized weights in floating-point form greater than 0.66 can be quantized as 1, another normalized weight less than 0.66 and greater than -0.66 can be quantized as 0, and another normalized weight less than -0.66 can be quantized as -1.

[0057] System 100 can apply each scaling factor to each quantized input and output by multiplying each quantized input or output by its respective scaling factor. Given a specific input, System 100 can obtain each scaling factor based on a measure of approximation (or similarity) between the quantized neural network and the original, unquantized neural network. System 100 can set the scaling factor as a constant if it has trained the modified neural network, determined the scaling factor, and is performing inference calculations on the modified neural network.

[0058] Returning to the reference in Figure 2A, the neural network modification engine 120 can generate a new machine learning model 295 (or a new part thereof) by replacing the group of network layers 290 of the original part of the neural network 200 with a decision tree 220 having a tree depth based on at least the number of layers in the group of network layers 290. The decision tree replaces the group of network layers. It can be any type, as long as it is appropriate. For example, a decision tree can be a GradientBoost decision tree or an AdaBoost decision tree.

[0059] System 100 can perform binary quantization on at least the input 213 to the first layer of group 290 of the network layer, and the output 217 from the last layer of group 290 of the network layer. Alternatively, system 100 can perform binary quantization on the entire neural network.

[0060] The decision tree 200 can receive a binary quantization input 215 from the preceding layer 210 and output a quantized output 225 to the subsequent layer 230. More specifically, the quantization input 215 contains binary inputs (either 1 or -1) from nodes 210a to f of the preceding layer 210. Similarly, for illustrative purposes only, the quantized output 225 can be either output 225a representing {1} or output 225b representing {-1}. Each output from the decision tree is quantized as either 1 or -1 and provided to the subsequent layer 230.

[0061] Before replacing group 290 of the network layer with the decision tree 220, system 100 can train the decision tree 220 using quantized versions of the corresponding parts of the training example. More specifically, system 100 can obtain quantized versions of the inputs to the first layer of group 290 and quantized versions of the outputs from the last layer of group 290, given the training example of the original neural network. System 100 can set the quantized input versions as inputs to the decision tree 220 and the quantized output versions as outputs from the decision tree 220, and train the decision 220 using the input and output quantized versions.

[0062] Figure 2B shows a portion of another exemplary novel machine learning model 255, which has a decision tree with a tri-quantized output 275.

[0063] Similar to Figure 2A, System 100 can replace different groups 285 of layers in a portion of the neural network 250 with different decision trees 270, and quantize the inputs and outputs of the decision trees using tri-level quantization. The quantized input 265 and quantized output 275 contain one of the value sets {-1, 0, 1}. For illustrative purposes only, the quantized output 275 can be one of the outputs 275a representing {1}, output 275b representing {0}, and output 275c representing {-1}.

[0064] For the sake of illustration, Figure 2A shows three groups in the network layer, and Figure 2B shows five groups; however, the number of network layer groups replaced by the decision tree can be any appropriate value determined by system 100. For example, the number could be 1, 10, or 50.

[0065] Figure 3 is a flowchart of an exemplary process 300 for generating a new machine learning model to be trained. For convenience, the process 300 is described as being performed by one or more computer systems located in one or more locations. For example, a neural network deployment system 100, for example shown in Figure 1 and appropriately programmed according to this specification, can perform the process 300.

[0066] The system receives data representing a neural network with multiple layers arranged in a sequence (310). The data received is, to name just a few examples, a neural network. This can include the operations performed by each network layer of the network network, as well as the weights of each network layer. The network layers are arranged in sequence such that the layer output from a preceding layer is provided as a layer input to a subsequent layer.

[0067] The system selects one or more groups of layers from a plurality of layers, each group of layers comprising one or more layers adjacent to each other in sequence (320). For example, the system can select three groups of layers, the first group of layers comprising only one layer, the second group of layers comprising three layers, and the last group of layers comprising five layers having the second-to-last layer of the neural network.

[0068] The system generates a new machine learning model corresponding to the neural network by replacing each of one or more groups of selected layers with its own decision tree (330).

[0069] Each decision tree used to replace each group of layers can have a tree depth based on at least the number of layers in the group of layers. For example, the tree depth of a decision tree is equal to the number of layers in the group of layers. Another example is a decision tree with a tree depth of 3, where the group of layers replaced by the decision tree has 5 network layers.

[0070] A decision tree can contain any appropriate tree type. For example, a decision tree can be a GradientBoost tree or an AdaBoost tree.

[0071] The system can quantize at least the input to the first layer of a group of layers and the output from the last layer of the group of layers, and provide the quantized input and output to the respective decision tree. More specifically, for each of one or more groups of layers, the respective decision tree takes the quantized version of the input to the first layer of each in the group as its input and produces the quantized version of the output to the last layer of each in the group as its output.

[0072] In some embodiments, the system can obtain quantized versions of the inputs and outputs to each decision tree based on their respective scaling factors. More specifically, the system can generate quantized versions of the inputs to the decision tree by multiplying the quantized inputs by their respective scaling factors. As described above, the scaling factors are obtained based at least on a measure of similarity between the quantized layer and the original layer.

[0073] The system trains a new machine learning model based on the training data of the original neural network (340). The system trains at least some of the layers in the original neural network that were not replaced by each decision tree. In some embodiments, the system trains layers of the neural network following one or more groups of layers of the neural network in a sequence.

[0074] In some situations, the system can train an entire new machine learning model using the same but quantized training samples as the original neural network. In some embodiments, the system can use quantized inputs and outputs for forward propagation during training and floating-point inputs and outputs to update weights during backward propagation. Alternatively, the system can train a new machine learning model using the same but quantized training samples as the original neural network. For each decision tree in the learning model, it is also possible to compute data representing the respective gradients.

[0075] The system can select a group of layers and choose any appropriate algorithm to replace the group of layers with their respective decision trees. In some embodiments, the system can repeatedly select groups of layers. More specifically, one exemplary algorithm for selecting groups of layers is described below.

[0076] Assuming the neural network contains N network layers, the system indexes each layer of the neural network according to the sequence i ∈ [0, 1, 2, ..., N-1] as layer i.

[0077] The system sets the total number of layers (all layers) and selects a group of layers from which the size of the neural network (N layers) is equal.

[0078] The system randomly selects layer L from all layers as the first layer of the group of layers. In some embodiments, the system can select the first layer of the group of layers according to a layer sequence. For example, the system can start with layer 0 as the first layer of the group of layers.

[0079] For each layer in the sequence, starting from layer L and ending at layer N, the system first checks whether the current layer has already been replaced by a decision tree or belongs to a decision tree.

[0080] If the system determines that the current layer has not been replaced by a decision tree, or belongs to a decision tree, it adds the current layer to the group of layers.

[0081] The system performs quantization on the inputs to the initial layer L of a group of layers and on the outputs from the current layer, provisionally replacing the layers from layer L to the current layer with their respective decision trees. As described above, the system can perform binary or ternary quantization on the inputs and outputs to the neural network layers. In some embodiments, the system quantizes all outputs of each layer in the neural network, allowing the size of the accumulator to be limited to a specific number of decision trees.

[0082] The system updates the layer information for each layer group and measures the performance of the modified network (e.g., inference accuracy). Details of layer grouping and performance measurement are described in more detail in relation to Figure 4.

[0083] After determining the groups of layers, the system replaces each group of layers with its respective decision tree.

[0084] The system trains the remaining layers of the neural network, starting from the last layer of the group and continuing up to the last layer. More specifically, the system fine-tunes the weights of the remaining layers using the corresponding parts of the same training examples that were used to train the original neural network.

[0085] Figure 4 is a flowchart of an exemplary process 400 for selecting one or more groups of neural network layers. For convenience, process 400 is described as being executed by one or more computer systems located in one or more locations, for example, appropriately programmed according to this specification, even The neural network deployment system 100, shown in Figure 1, can execute process 400.

[0086] The system selects each initial layer in the neural network (410). As described above, the system can select the initial layers of a group of layers randomly or based on the sequence of layers.

[0087] The system generates several candidate groups, each having its own initial layer as the first layer in the candidate group (420). More specifically, the system provisionally sets the number of layers for each candidate group having the same initial layer. For example, the system generates a first candidate group having only two layers, a second candidate group having four layers, and a third candidate group having six layers. In some embodiments, the candidate groups may have a consecutive number of layers. For example, the first candidate group may have a single layer, the second candidate group may have two layers, and the third candidate group may have three layers.

[0088] For each of the multiple candidate groups, the system determines a performance metric for each candidate group that measures the performance of the corresponding new machine learning model in which the layers in the candidate group are replaced by the respective decision tree (430). More specifically, the system generates multiple new machine learning models, each of which contains each candidate group of layers that are replaced by the respective decision tree. Using the same input data, the system performs inference computations for each of the new machine learning models and obtains a performance score for each of the new machine learning models. Each performance core can be obtained based on inference accuracy.

[0089] The system selects one of the candidate groups for each of the multiple candidate groups to be replaced by the decision tree, based on each performance metric (440). In some embodiments, the system determines the highest performance metric among the multiple candidate groups and selects the candidate group associated with the highest performance metric from each of the multiple candidate groups. Alternatively, the system selects a candidate group that has a relatively high performance score but performs the inference computation fastest.

[0090] Figure 5 shows an example of decision tree 500. As shown in Figure 5, the decision tree 500 can contain multiple nodes with a specific tree depth. The tree depth is determined based on the total number of tree layers. Each layer of the decision can contain nodes representing their respective node operations, such as logical comparison or other appropriate criteria.

[0091] For example, as shown in Figure 5, the decision tree 500 may have multiple nodes, including the root node 510 and the respective non-root nodes 520, 530 in different tree layers. The root node 510 is the starting point of the decision tree 500 and has no parent node, while the non-root nodes 520 and 530 each have a parent node and are also called child nodes. In general, each node in a tree layer, excluding leaf nodes, may have one or more branches (arrows in Figure 5) that connect that node to its respective child node in the next tree layer. Nodes that have child nodes are also called non-leaf nodes, for example, nodes 510, 520a, and 520b.

[0092] For each leaf node in the deepest, i.e., last, tree layer (e.g., nodes 530a, 530b, 530c, and 530d), each can represent an inference output of decision tree 500 and has no branches connecting to other child nodes. The inference output can be a prediction, for example, the probability P1 of leaf node 530a.

[0093] Referring to Figure 5, the system (for example, system 100 in Figure 1) has coefficient n f1 , n f2 , and n f3 When performing a decision tree inference operation using input data representing the following, each non-leaf node in the deepest tree layer, excluding the leaf nodes, has its own node operation (for example, a logical comparison as shown in Figure 5). The system performs a node operation on the root node 510, obtains a logical result (for example, true or false) from the node operation, and based on that result, can approach the corresponding child node along each branch. For example, if the result is false, the system approaches node 520a. In some embodiments, the system can use integers 0 and 1 to represent false and true. Finally, the system approaches a specific leaf node in the deepest layer of the decision tree and returns the inference represented by that specific leaf node.

[0094] Figure 6 shows an exemplary embodiment of a decision tree using fixed functional hardware 600.

[0095] As shown in Figure 6, a system (for example, system 100 shown in Figure 1) can perform decision tree inference computations using fixed functional hardware 600. More specifically, for the exemplary decision tree shown in Figure 5, the system can take input x610 and return a final inference output 620. The decision tree 600 may include multiple computation units 630 and 640 to take inference results from different nodes and generate the final inference output 620, which is represented by a particular leaf node in the decision tree.

[0096] A system using fixed functional hardware 600 can assign each input data to a corresponding tree node and perform its respective function at each node using its respective comparator and multiplexer. As shown in Figure 6, the system can assign input data x ∈ [h 1, Assign l1] to the top-level node (corresponding to root node 510 in Figure 5), and input data x∈[h 2, Assign l2] to the left node 630a (corresponding to the left non-leaf node 520a in the second tree layer), and input data x∈[h 3, Assign l3] to the right-hand node 630b (corresponding to the right-hand non-leaf node 520b in the second tree layer). The system can perform operations at each node using its respective comparator 640 and multiplexer 630. For example, the system performs operations at the top-level node using comparator 630c and multiplexer 630c. More specifically, the system receives an input x which may be a vector of real numbers, with range x ∈ [h 1,Select a portion of the input having [l1] and provide it to the top-level root node. The system compares the portion of the input with a criterion a1 using comparator 640c, where a1 can represent a real-valued scalar. If the input data x is greater than or equal to a1, the system outputs a result representing "true" using multiplexer 630c assigned to the node; for example, the multiplexer may output the integer 1, which represents "true". The system can continue to perform operations on the child nodes linked along the corresponding branch of the tree where the current node is located. After processing operations on non-leaf nodes in the second-to-last tree layer, the system can output a final result 620 based on the value (e.g., probability) represented by the corresponding leaf node in the decision tree.

[0097] Figure 7 shows an exemplary programmable core 700 for performing decision tree inference computations. This is an example.

[0098] As shown in Figure 7, a system (for example, system 100 shown in Figure 1) can perform inference computations for a decision tree using a programmable core 700. The programmable core 700 can receive a tree input 710 and produce an inference output 720 for the tree input. The programmable core 700 can include multiple computational components, such as a MUX unit, an ALU unit, and static random access memory (SRAM). The programmable core 700 can perform node operations simply by adding, selecting, and switching the functions configured in the computational components. Therefore, the programmable core 700 does not need to include MAC units for performing multiplication and addition as node operations for generating the inference output of the decision tree.

[0099] Referring to FIG. 7, to perform an operation at the root node (e.g., the root node 510 in FIG. 5), the system receives an input 710 and passes the received input 710 to a combining unit to be modified using an array (x, a, l, h, i 0 , i 1 ) that is specific to the root node and has been previously stored in a queue. As described above, a represents a node reference, l and h represent the numerical range of the input x, and the indices i 0 , i 1 can each represent a tree branch connecting the current node to its respective child nodes and can be assigned to respective integer values to represent the result of performing an operation on the node operation.

[0100] The system can select the inputs combined using any unit and perform a node operation at the root node. For example, the system can use each comparator to perform a comparison between the input x ∈ [h, l] and a reference a1. Accordingly, the system can assign the indices i 0 , i 1 to respective integer values to represent the result of the comparison and instruct the calculation of the decision tree to the corresponding child nodes. For example, if the system determines that x ∈ [h, l] is greater than the reference a1, the system can assign i 0 = 0, i 1 = 1, so that the system can execute an operation on the corresponding next child node along the tree branch represented by i 1 .

[0101] To perform calculations on the next child node, the system can first apply a switching unit to determine, based on the numbering criterion N, whether the next child node is a non-leaf node or a leaf node. To make this determination, the system can number each node using its respective tag K (for example, an integer) and determine the type of the next child node based on a predetermined numbering criterion. For example, relating to Figure 5, the system can tag nodes 510, 520a, and 520b as nodes 0, 1, and 2, and leaf nodes 530a, 530b, 530c, and 530d as nodes 3, 4, 5, and 6. The system can set the numbering criterion N=3 such that nodes with tag K<3 are non-leaf nodes and nodes with tag K>=3 are leaf nodes.

[0102] Upon determining that the next child node is a non-leaf node, the system retrieves the respective arrays (x, a, l, h, i) of the next child node from the data stored in the non-leaf SRAM. 0 i 1 The system can update the following: Similarly, depending on whether the next child node is a leaf node, the system can provide the final output associated with the leaf node stored in the leaf SRAM.

[0103] For parallel computing, the system employs a fork function to identify a portion of the node's input data in the queue and the operation and type of the node in SRAM. The system can store columns and arrays. During parallel computing, the system can automatically determine when to modify the input data using one or more combined units and their respective arrays, based on the latency observed for each computing unit.

[0104] The embodiments referred to herein provide an improved method for training a neural network. The neural network may be configured to accept any kind of digital data input and generate any kind of score, classification, or regression output based on that input. Input data items may comprise image data (including video data), audio data, or text data, such as words or word parts of a natural language (or their representations, e.g., embeddings). Input data items may comprise sequential data, such as a sequence of data samples representing digitized audio, or an image represented as a sequence of pixels, or a video represented by a sequence of images, or a sequence representing a sequence of words in a natural language. In this specification, “image” includes, for example, a LIDAR image.

[0105] In some embodiments, the neural network output may include feature representations, which may then be further processed to generate a system output. For example, the system output may include a classification output for classifying input data items into one of several categories, such as images, videos, or audio (e.g., data representing the estimated likelihood that an input data item or an object / element of an input data item belongs to a category), or a segmentation output for segmenting a region of an input data item into objects or actions represented, for example, in an image or video. Alternatively, the system output may be an action selection output in a reinforcement learning system.

[0106] In several other embodiments, the network output may comprise other data items of the same or different types. For example, the input data item may be an image, audio, or text, and the output data item may be a modified version of the image, audio, or text, for example, modifying the style, content, properties, pose, etc., of the input data item, or one or more objects or elements within the input data item, or filling in (missing) parts of the input data item, or predicting another version of the data item, or the extension of a video or audio data item, or providing an upsampled (or downsampled) version of the input data item. For example, the input data item may be a representation of text in a first language, and the output data item may be a translation of the text into another language, or a score for a translation of the text into another language. In another example, the input image may be converted to a video, or a wireframe model, or a CAD model, or a 2D input image may be converted to 3D, or vice versa. Alternatively, the input data items may consist of features derived from or from oral utterances or sequences of oral utterances, and the network system output may consist of scores for each of the text segments, each score representing the estimated likelihood that the text segment is a correct feature-based replacement. In another example, the input data items may be images, audio, or text, and the output data items may be representations of the input data items in different formats. For example, a neural network may convert text to speech or vice versa (in the case of speech recognition), or convert images (or videos) to text (for example, for captions). When generating an output consisting of sequential data, the neural network may include one or more convolutions, such as an extended convolutional layer.

[0107] In some other embodiments, the network output may include an output for selecting actions to be performed by an agent, such as a robot or other mechanical agent, in an environment such as a real-world environment or a simulation of a real-world environment.

[0108] In some embodiments, a neural network is configured to receive an input data item and process it to generate a feature representation of the input data item according to network parameters. Generally, a feature representation of a data item is an ordered set of numbers, such as a vector, that represents the data item as a point in a multidimensional feature space. In other words, each feature representation may contain numbers for each of the multiple features of the input data item. As mentioned above, a neural network can be configured to receive any kind of digital data input and generate a feature representation from that input. For example, an input data item, which may also be called a network input, can be an image, a portion of a document, a text sequence, audio data, medical data, etc.

[0109] Once trained, feature representations can provide input to another system for use in performing machine learning tasks on network inputs, for example. Exemplary tasks may include feature-based search, clustering, semi-duplicate detection, validation, feature matching, region adaptation, and video-based weak supervision, and in the case of video, may include, for example, object tracking across video frames and gesture recognition of gestures performed by entities depicted in the video.

[0110] When the input to a neural network is an image or features extracted from an image, the output produced by the neural network for a given image may be a score for each of a set of object categories, each score representing the estimated likelihood that the image contains images of objects belonging to that category. More specifically, each of the input image or features extracted from the input image may contain one or more pixels, each having its own intensity value. The neural network is configured to process the respective intensity values ​​of the input image or the features extracted from the image and generate predictions, such as image classification, image recognition, or image segmentation.

[0111] As another example, if the input to a neural network is an internet resource (e.g., a web page), a document, or a part of a document, or features extracted from an internet resource, document, or part of a document, then for a given internet resource, document, or part of a document, the output generated by the neural network could be a score for each of a set of topics, each score representing the estimated likelihood that the internet resource, document, or part of a document is related to that topic.

[0112] As another example, if the input to a neural network is the characteristics of the impression context of a particular advertisement, the output generated by the neural network could be a score representing the estimated likelihood that the particular advertisement will be clicked.

[0113] As another example, if the input to the neural network is personalized recommendation features for the user, such as features that characterize the context of the recommendation, or features that characterize previous actions taken by the user, the output generated by the neural network could be a score for each of a set of content items, each score representing the estimated likelihood that the user will respond positively to the recommended content item.

[0114] As another example, if the input to a neural network is a sequence of text in one language, the output produced by the neural network could be a score for each of the parts of text in another language, each score representing the estimated likelihood that the part of text in the other language is a good translation of the input text into the other language.

[0115] As another example, if the input to a neural network is a sequence representing spoken utterances, the output generated by the neural network could be a score for each part of the text, each score representing the estimated likelihood that the part of the text is a correct transcription of the utterance.

[0116] Embodiments of subject matter and functional operations described herein can be implemented in digital electronic circuit configurations, tangibly embodied computer software or firmware, computer hardware including structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions executed by a data processing device or encoded in a tangible non-temporary storage medium to control operations of the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random-access memory device or a serial-access memory device, or one or more combinations thereof. Alternatively, or in addition, the program instructions may be encoded in artificially generated propagating signals, such as machine-generated electrical signals, optical signals or electromagnetic signals, which are generated to encode information for transmission to a suitable receiving device for execution by the data processing device.

[0117] The term "data processing device" refers to data processing hardware and includes any type of device, machine, or apparatus for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. This device may also be, or further include, a dedicated logic circuit configuration such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit). In addition to hardware, this device may optionally include code that creates an execution environment for computer programs, such as processor firmware, a protocol stack, a database management system, an operating system, or code that constitutes one or more of these.

[0118] Computer programs, also called, or described as, programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, such as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not have to, correspond to a file in a file system. A program can be stored, for example, in one or more scripts stored in a markup language document, as part of a file that holds other programs or data, in a single file dedicated to the program in question, or in multiple coordinated files, for example, files that hold one or more modules, subprograms, or parts of code. A computer program can be deployed to run on one computer, or on multiple computers located in one site, or distributed across multiple sites and interconnected by a data communication network.

[0119] In this specification, the term “database” is used broadly to refer to any collection of data, which does not need to be structured in any particular way, or does not need to be structured at all, and can be stored in one or more storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different manner.

[0120] Similarly, in this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components and installed on one or more computers in one or more locations. In some cases, one or more computers may be dedicated to a particular engine, while in other cases, multiple engines may be installed and run on the same one or more computers.

[0121] The processes and logic flows described herein can be executed by one or more programmable computers that run one or more computer programs that perform functions by performing operations on input data and generating outputs. The processes and logic flows can also be executed, for example, by dedicated logic circuit configurations such as FPGAs or ASICs, or by a combination of dedicated logic circuit configurations and one or more programmed computers.

[0122] A computer suitable for running computer programs can be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are a central processing unit for executing or enforcing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be complemented by or incorporated into a dedicated logic circuit configuration. Generally, a computer is also operationally coupled to one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or for receiving data from them, transferring data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can also be incorporated into another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, Global Positioning System (GPS) receiver, or a portable storage device such as a Universal Serial Bus (USB) flash drive.

[0123] Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and all forms of non-volatile memory, media, and memory devices, including CD-ROM disks and DVD-ROM disks.

[0124] To provide user interaction, embodiments of the subject matter described herein can be implemented in a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and a pointing device, such as a mouse or trackball, to which the user can provide input to the computer. Other types of devices can also be used to provide user interaction; for example, the feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback, and input from the user can be received in any form, including acoustic, voice, or tactile input. In addition, the computer can provide input to the devices used by the user. A computer can interact with a user by sending documents and receiving documents from a device, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. A computer can also interact with a user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user in return.

[0125] Data processing devices for implementing machine learning models may also include, for example, dedicated hardware accelerator units for handling the general and computationally intensive parts of machine learning training or production, i.e., inference and workload.

[0126] Machine learning models can be implemented and deployed using machine learning frameworks such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0127] Embodiments of the subject matter described herein can be implemented in a computing system that includes, for example, a backend component as a data server, or a middleware component which is, for example, an application server, or a frontend component which is, for example, a graphical user interface, a web browser, or a client computer which has an application through which a user can interact with the embodiment of the subject matter described herein, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), such as the Internet.

[0128] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via a communication network. The relationship between the client and server arises from computer programs running on each computer that have a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from the user. Data generated on the user device, such as the results of user interactions, may be received from the device by the server.

[0129] This specification includes details of many specific embodiments, which should not be construed as limitations on the scope of any invention or the scope of claims, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable partial combination in multiple embodiments. Furthermore, features described above as operating in a particular combination, and initially claimed as such, may in some cases be excluded from the claimed combination, and the claimed combination may be directed towards a partial combination or a variation of a partial combination.

[0130] Similarly, while the operations are shown in the drawings and described in the claims in a specific order, this should not be understood as requiring that such operations be performed in a specific or sequential order shown, or that all the exemplified operations be performed, in order to achieve the desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the described program components and systems should generally be understood as being able to be integrated together in a single software product or packaged in multiple software products.

[0131] Specific embodiments of the subject matter are described. Other embodiments are within the scope of the claims below. For example, the operations described in the claims can still achieve the desired results even if they are performed in a different order. As one example, the processes shown in the accompanying drawings do not necessarily require the specific order or sequence shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A system for performing inference computations based on a decision tree using a programmable computing core, The system comprises a coupling unit capable of receiving a first data value from a plurality of input data values ​​and a first array of node data values ​​from a data queue, wherein the first array of node data values ​​comprises an output index data value and a reference data value, and the coupling unit is further capable of combining the first data value and the first array of node data values ​​to generate a first array of data values, wherein the first array of data values ​​is associated with a first node of the decision tree, and the system further comprises The system includes a comparator capable of receiving a first array of data values ​​from the coupling unit and generating a first array of modified data values, wherein the output index data value is modified based on the comparison result between the first data value and the reference data value, and the system further includes The system includes a switching unit that receives a first array of the modified data values ​​and can determine whether the first data value is associated with the node of the last row of the decision tree, If the switching unit determines that the first data value is associated with the last row of the decision tree, it can access the final output from the first memory device. If the system determines that the first data value is not associated with the last row of the decision tree, the switching unit can generate a second array of data values ​​by modifying the first array of modified data values ​​with data values ​​stored in a second memory device, the second array of data values ​​comprising data values ​​associated with subsequent nodes in the decision tree, the subsequent node being determined by the output index data value of the first array of modified data values.

2. The system according to claim 1, wherein the plurality of input data values ​​include output values ​​from layers of a preceding neural network.

3. The system according to claim 1 or 2, wherein the reference data value of the first array of node data values ​​is learned during the training process.

4. The system according to any one of claims 1 to 3, wherein the first memory device and the second memory device are SRAM (static random access memory) memory devices implemented on the programmable computing core.

5. The system according to any one of claims 1 to 4, wherein each of the aforementioned decision trees comprises a GradientBoost decision tree or an AdaBoost decision tree.

6. The system according to any one of claims 1 to 5, wherein the decision tree replaces the selection groups of the neural network layers of the neural network.

7. The system according to claim 6, wherein the tree depth of the decision tree is equal to the number of layers in the selection group.

8. Selecting each of the aforementioned groups of layers means Selecting each initial layer in the aforementioned neural network, Each of these generates multiple candidate groups, each having the aforementioned initial layer as the first layer in the candidate group. For each of the aforementioned multiple candidate groups, the performance metric for each candidate group is determined, which measures the performance of the corresponding new machine learning model having the aforementioned layer in the candidate group that is replaced by the respective decision tree. The system according to claim 6, further comprising selecting one of the candidate groups as the group based on the respective performance metrics for each of the aforementioned multiple candidate groups.

9. To generate each group of layers, selecting each initial layer in the neural network is: The system according to claim 8, comprising selecting each of the initial layers by a random process or based on the sequence of the neural network.

10. Selecting one of the candidate groups as the group based on each performance measure is: The process involves determining the highest performance measure from among the aforementioned performance measures, The system according to claim 8, further comprising selecting from each of the aforementioned multiple candidate groups the candidate group associated with the greatest performance measure as the group.

11. A method for performing inference computations based on a decision tree using a programmable computing core, The coupling unit of the programmable computing core includes receiving a first data value from a plurality of input data values ​​and a first array of node data values ​​from a data queue, wherein the first array of node data values ​​comprises an output index data value and a reference data value, and the method further includes: The combining unit includes combining the first data value and the first array of node data values ​​to generate a first array of data values, the first array of data values ​​being associated with the first node of the decision tree, and the method further includes, The programmable computing core comparator includes generating a first array of modified data values ​​based on a first array of data values ​​received from the coupling unit, wherein the output index data value is modified based on the result of comparing the first data value with the reference data value, and the method further includes: The switching unit of the programmable computing core determines, based on a first array of modified data values ​​received from the comparator, whether the first data value is associated with a node in the last row of the decision tree, and the method further includes: If it is determined that the first data value is associated with the last row of the decision tree, then the final output from the first memory device is accessed, A method comprising: determining that the first data value is not associated with the last row of the decision tree; modifying the first array of modified data values ​​with data values ​​stored in a second memory device to generate a second array of data values, wherein the second array of data values ​​comprises data values ​​associated with subsequent nodes of the decision tree, and the subsequent node is determined by the output index data value of the first array of modified data values.

12. The method according to claim 11, wherein the plurality of input data values ​​include output values ​​from a preceding neural network layer.

13. The method according to claim 11 or 12, wherein the reference data value of the first array of node data values ​​is learned during the training process.

14. The method according to any one of claims 11 to 13, wherein the first memory device and the second memory device are SRAM (static random access memory) memory devices implemented on the programmable computing core.

15. The method according to any one of claims 11 to 14, wherein each of the aforementioned decision trees comprises a GradientBoost decision tree or an AdaBoost decision tree.

16. The method according to any one of claims 11 to 15, wherein the decision tree replaces the selection groups of the neural network layers of the neural network.

17. The method according to claim 16, wherein the tree depth of the decision tree is equal to the number of layers in the selection group.

18. Selecting each of the aforementioned groups of layers means Selecting each initial layer in the aforementioned neural network, Each of these generates multiple candidate groups, each having the aforementioned initial layer as the first layer in the candidate group. For each of the aforementioned multiple candidate groups, the performance metric for each candidate group is determined, which measures the performance of the corresponding new machine learning model having the aforementioned layer in the candidate group that is replaced by the respective decision tree. The method according to claim 16, further comprising selecting one of the candidate groups as the group based on the respective performance metrics for each of the aforementioned multiple candidate groups.

19. To generate each group of layers, selecting each initial layer in the neural network is: The method according to claim 18, comprising selecting each of the initial layers by a random process or based on the sequence of the neural network.

20. Selecting one of the candidate groups as the group based on each performance measure is: The process involves determining the highest performance measure from among the aforementioned performance measures, The method according to claim 18, further comprising selecting a candidate group associated with the greatest performance measure from each of the aforementioned group of candidates as the group.

Images