Training neural networks to perform machine learning tasks

Abstract

To provide a method, program, and system for training a neural network to perform machine learning tasks. The method includes receiving input data for a neural network. The method further includes determining values ​​for a plurality of hyperparameters of the neural network. The method further includes constructing a neural network according to the hyperparameter values, the neural network comprising a plurality of neurons. Each neuron includes a probability distribution for the logical operators, such that the neuron includes a corresponding probability for each of the plurality of logical operators. The method further includes training the neural network according to the hyperparameter values ​​and the input data by learning the probability distribution for each neuron. The method further includes determining a logical operator of the plurality of logical operators for each neuron by selecting a value in the probability distribution.

Description

[Technical field]

[0001] The technical field is artificial intelligence, in particular neural networks. Aspects relate to training a neural network to perform a machine learning task. The machine learning task may be a classification task or a regression task. The classification task may include one or more of pattern and / or sequence recognition, novelty detection (i.e., outlier or anomaly detection), sequential decision making. The regression task may include compression or recovery. Additionally, aspects may relate to using a trained neural network to perform a machine learning task. [Background technology]

[0002] A neural network may include artificial neurons (i.e., neurons, nodes, computational units, or units) and may be used to solve artificial intelligence problems. The connections between neurons may be modeled as weights in the neural network. A neural network may include parameters, which are components or elements of a neural network that can be trained. For example, weights may be parameters in some neural networks. However, other parameters are possible.

[0003] Activation functions may be used to control the magnitude of the output of a neural network. Thus, activation functions may be used to limit the values ​​of the output of a neural network and to add nonlinearity to the neural network.

[0004] The neural network may be trained using an optimization algorithm. Training may include adjusting the parameters of the neural network until the neural network is sufficiently accurate. The optimization algorithm may be based on (or dependent on) gradient descent. Examples of optimization algorithms based on gradient descent include stochastic gradient descent (SGD) and Adam (an adaptive learning rate optimization algorithm - realized as a combination of root-mean-square propagation and SGD with momentum). More information on Adam can be found in "Adam: A method for stochastic optimization", Kingma et al., 2014.

[0005] Gradient descent may be based on backpropagation (i.e., backprop or backward propagation). Gradient descent may include a cost function that measures the accuracy of the parameters and determines in which direction a change in the parameters can improve the network, i.e., thereby improving the parameters. Sufficient accuracy can be obtained when the output of the cost function is zero, at a minimum, or within a specified range of zero. The specified range may depend on the machine learning task. In some cases, gradient descent (or algorithms based on gradient descent) can only be used to train neural networks with differentiable parameters, so binary or Boolean parameters cannot be trained via gradient descent. Boolean parameters (represented in neural networks as 0 or 1) can lead to discontinuities (e.g., jumps from 0 to 1), in which case the gradient is ill-defined and gradient descent is not possible.

[0006] The accuracy of a neural network may be calculated in a variety of ways. For example, in the context of classification tasks, the training error (or loss) may be obtained by measuring how accurately a neural network can classify the data used to train it. Classification loss (also called generalization error or expected loss) may be obtained by measuring how accurately a neural network can perform a classification task on previously unseen data.

[0007] In some cases, a neural network may include multiple layers, i.e., at least one hidden layer and an output layer. In such cases, the neural network may be called a deep learning neural network, and there may be an infinite number of hidden layers. Thus, multiple layers may be used to incrementally extract higher level features from the input. A neural network may be realized as a feedforward network, where data flows from the previous hidden layer to the output layer without back-looping. Initialization of a neural network may include assigning random weights to the connections between neurons.

[0008] Traditionally, if a neural network does not perform a machine learning task accurately, the weights are adjusted, thus calibrating the influence of different neurons in the network until the proper mechanical manipulation of the inputs to perform the machine learning task is determined.

[0009] Once trained to perform a machine learning task, a neural network may be used to perform the machine learning task, which may also be referred to as inference or performing inference.

[0010] Neural networks face a variety of problems. For example, overfitting can occur when the training error is small but the classification loss is relatively large. In other words, the neural network learns an approach that is specific to the training data used to train the neural network that does not generalize well to other data other than the training data. Underfitting can occur when the neural network cannot accurately capture the relationship between its inputs and outputs, resulting in high error rates on both the training data and other data.

[0011] Although neural networks can be effective at performing classification tasks, they can also be inefficient. Therefore, various approaches have been developed to increase the efficiency of neural networks.

[0012] For example, a traditional neural network may be a deep learning neural network that includes at least one dense layer, such that some or all of the neural network is fully connected, i.e., each neuron in the dense layer receives inputs for every neuron in the previous layer. In this example, the neural network may be referred to as a densely connected neural network or a fully connected neural network.

[0013] In addition, traditional neural networks may use full-precision (e.g., double-precision, i.e., 64-bit) floating-point computations for weights and activations, which generally require expensive matrix multiplications. Even with lower-precision floating-point weights and activations (e.g., single-precision or half-precision), the computations can still be expensive.

[0014] Binary neural networks may differ from conventional neural networks in that 1-bit activations (and / or weights) replace the floating-point activations (and / or weights) of conventional neural networks. Thus, elements propagated through the network are binary in the sense that they can have only one of two values, e.g., 0 or 1. Thus, expensive matrix multiplications compared to conventional neural networks may be simplified to faster XNOR and bit-counting operations, which significantly improves binary neural network inference performance compared to conventional neural networks. In some cases, XNOR may be faster than bit-counting operations, and both may be faster than floating-point operations.

[0015] Floating-point operations required to train a neural network may be supported in computer hardware. Specifically, a float32 adder and / or a float32 multiplier may require at least 1000 logic gates (or look-up tables) and may have delays of tens of logic levels. The float32 adder and / or the float32 multiplier may be implemented directly in a central processing unit (CPU) or a graphics processing unit (GPU). In any case, the float32 adder and the float32 multiplier may require significantly more computational resources (e.g., processing cycles) than bitwise logical operations on an int64 data type, even in a GPU optimized for float32 and int32 data types. If a GPU is optimized for int32, int64 may not be supported efficiently. Thus, it may be more efficient to use the int32 data type for such a GPU. In some hardware implementations, the logical operations may be faster than the addition and / or multiplication operations.

[0016] A CPU may be able to perform about 3 to about 10 int64 bitwise operations per clock cycle, whereas floating-point operations typically require an entire clock cycle (i.e., only one floating-point operation may be performed per clock cycle). Thus, the XNOR and bit counting operations performed when training binary neural networks may have significant performance advantages compared to floating-point operations performed for traditional neural networks, even if the floating-point operations (e.g., float32 adders or float32 multipliers) have dedicated hardware support in the GPU or CPU. Considerations regarding floating-point and integer operations may also be applicable to training neural networks for machine learning tasks.

[0017] A convolutional neural network (CNN) may be a deep learning neural network in which at least one hidden layer has structured connections, i.e., is not fully connected. Thus, a CNN may perform convolutions, e.g., a hidden layer performs a dot product of a convolution kernel with the input matrix of the hidden layer. Thus, only neurons in the hidden layer that are covered by a convolution kernel may be connected to respective neurons in another layer of the CNN.

[0018] In a classic example, a binary neural network may be based on a very deep convolutional neural network (e.g., VGG) architecture that increases the depth via very small (3 × 3) convolutional filters, includes three fully connected layers, and uses rectified linear units (ReLU) to reduce training time. Alternatively, a binary neural network may be based on a residual neural network (ResNet) architecture that uses skip connections to skip some layers. Both VGG and ResNet architectures can result in very large neural networks, which can lead to inefficiencies.

[0019] A neural network may be sparse (in this example, a sparse neural network). Thus, a sparse neural network has a proper subset of the connections of a fully connected neural network. In other words, instead of a fully connected layer, not all neurons in a layer of a sparse neural network are connected to each other. Sparseness may refer to the connectivity of a neural network, such that each neuron receives input from only a limited number of other neurons, and / or it may refer to the state of a neural network, describing the level of activity of all neurons in the neural network, such that not all neurons in a sparse neural network are active at any given time.

[0020] A sparse neural network may have less than 90% of the connections of a fully connected neural network, i.e., a sparse neural network may have at least 10% sparsity (10%-70% sparsity is sometimes called low sparsity, 70%-90% sparsity is sometimes called medium sparsity, 90%-99.9% sparsity is sometimes called moderate sparsity, and greater than 99.9% sparsity is sometimes called high sparsity). Neural networks with less than 10% sparsity are sometimes called dense.

[0021] A sparse neural network may be derived from a dense or fully connected neural network using pruning or compression techniques, e.g., regularization techniques. For example, one or more analytical methods or heuristics may be used to determine which weights should be retained and which weights should be removed from the neural network (removing a weight may involve setting the weight to zero). More specifically, L1 or L2 regularization may be used to update the general cost function by adding a regularization term. As yet another example, dropout is a regularization technique in which random neurons of a neural network are cancelled or removed. Dropout may be variational and may be limited by bounds (e.g., limits on how many neurons are cancelled). In some cases, rather than removing weights, the neurons themselves may be removed from the neural network. Pruning techniques, as discussed above, may help avoid overfitting and thereby reduce generalization error. Other techniques may involve removing entire layers of a neural network or adding additional costs for relatively large weights of the neural network. Thus, a sparse neural network may provide improved accuracy (e.g., less overfitting) compared to a dense neural network. A sparse neural network may also require less storage space compared to a dense neural network. However, if a sparse neural network is derived by removing weights from a dense neural network, the sparse neural network may require more storage space to store the weight positions in addition to the actual weights.

[0022] In two examples of conventional implementations, the connectivity of the neural network may be relaxed:

[0023] In a first example, the neurons of a neural network may be realized as logic modules, half of which are defined by fuzzy (i.e., relaxed) conjunctions and the other half by fuzzy (i.e., relaxed) disjunctions. In this example, the parameters of the neural network include weights, but the operators (i.e., fuzzy conjunctions and disjunctions) are fixed. Thus, the weights of neurons involved in fuzzy conjunctions and disjunctions may be relaxed from being only 0 or 1 to values ​​in the range 0 to 1. In this way, the parameters of the logic modules can be trained via gradient descent, unlike Boolean parameters.

[0024] In the second example, a set of arithmetic operations is predefined for each layer and implemented for the neurons of that layer. A form of softmax is used for training, where the output of the previous layer is used as the input for the arithmetic operation corresponding to the current layer. In this way, a symbolic representation can be learned from the data.

[0025] In another conventional example, fully connected layers of an artificial neural network are replaced with sparse layers before training to quadratically reduce the number of parameters without significantly decreasing accuracy (e.g., by more than 5% or 10%). The sparse topology has two successive layers and evolves during training into a scale-free topology. The scale-free topology has a power law frequency distribution P(d)~d -y where a fraction P(d) of all neurons in the network have d connections to other neurons, and the parameter γ typically remains in the range γ ∈ (2,3).

[0026] In yet another conventional example, a single shared weight parameter is assigned to all connections in a neural network. Instead of optimizing weights, the neural network is optimized for a network topology that performs well over a wide range of weights. Neurons and connections are added incrementally, and activation functions (e.g., sine, cosine, Gaussian, hyperbolic tangent, sigmoid, inverse, absolute value, ReLU) are randomly assigned to new neurons. Activation functions evolve as the neural network grows. The neural network uses floating-point operators.

[0027] According to another prior art example, a method is provided for memorizing a binary classification data set using a network of binary (two-input) look-up tables. The network of look-up tables is constructed by counting the conditional frequencies in the training data (e.g., how many times a pattern p is associated with a particular output 0 in the training data). As with neural networks, the look-up tables are arranged in successive layers. Unlike neural networks, training is done via memorization and does not involve backpropagation or gradient descent. Memorization involves remembering the outputs most commonly associated with the inputs in the training data.

[0028] In a further conventional example, a neural network is first trained for a machine learning task, then the neural network is translated into a Random Forest, which is then translated into a network of AND-Inverter logic gates, i.e. the network is based on "AND" and "NOT" logic gates. Summary of the Invention [Problem to be solved by the invention]

[0029] It may be desirable to provide a neural network for performing machine learning tasks that is more efficient than the prior art examples described above while maintaining comparable accuracy.

[0030] For example, a logic gate network may be used. The logic gate network may be a neural network in which each neuron includes a logical operator (i.e., a logical gate such as "AND" or "NAND"). Each logical operator may have 0 to 2 inputs. The logic gate network is sparse compared to a fully connected neural network, in which all neurons of the previous layer are provided as inputs to each neuron of the current layer. In other words, the logic gate network is sparse because each neuron of the logic gate network receives only two inputs, rather than n inputs (where n is the number of neurons per layer). Thus, machine learning tasks may be performed efficiently, since the logical operations can be calculated very quickly (e.g., because the logical operations are often built into the architecture of the processor). In addition, since each neuron has at most two inputs, the logic gate network may be very sparse, e.g., at least intermediate sparsity, which may lead to improved accuracy compared to denser networks.

[0031] Optionally, the logic gate network may include at least one operator having three or more inputs.

[0032] Advantageously, logic gate networks may not require activation functions because they operate on binary values ​​(e.g., after training) and are nonlinear.

[0033] However, implementing a neural network as a logic gate network can be problematic because a conventional logic gate network cannot be trained using gradient descent based optimization algorithms such as SGD or Adam (mentioned above). This is because the values ​​in a conventional logic gate network are always either 0 or 1. Therefore, it is not possible to compute the gradient in a conventional logic gate network because the loss function with respect to the parameters of the layers of the network has no derivatives. Therefore, it may be desirable to provide a logic gate network that can be trained using a gradient descent based optimization algorithm.

[0034] Moreover, a conventional logic gate network may provide a single binary output (e.g., for regression tasks) or k binary outputs for k classes (e.g., for classification tasks). It is not capable of step-by-step prediction, i.e., ranking of classes relative to each other. In other words, it is not capable of classification according to a "maximum activation" classification scheme ("How a Neural Network Works", Arkin Gupta, May 27, 2018). [Means for solving the problem]

[0035] According to one aspect, a computer-implemented method for training a neural network to perform a machine learning task is provided. The method includes receiving input data for the neural network. The method further includes determining values ​​for a plurality of hyperparameters of the neural network. The method further includes constructing a neural network according to the hyperparameter values, the neural network comprising a plurality of neurons. Each neuron includes a probability distribution for the logical operators, such that the neuron includes a corresponding probability for each of the plurality of logical operators. The method further includes training the neural network according to the hyperparameter values ​​and the input data by learning the probability distribution for each neuron. The method further includes determining a logical operator of the plurality of logical operators for each neuron by selecting a value in the probability distribution.

[0036] Neural networks are sometimes called binary neural networks (because when a neural network is trained, elements propagating through the network can have only one of two values), logic gate networks (because neurons implement logic operators), or binary logic gate networks. Logical operators are sometimes called logic gates.

[0037] In some cases, there are at most two inputs per neuron. If there are exactly two inputs per neuron and n neurons per layer, then the neural network can have at most 1-(2 / n) sparsity.

[0038] The values ​​of the multiple hyperparameters may be determined from received hyperparameters (e.g., used for performing a similar machine learning task or used by a similar neural network to perform the same machine learning task) or may be defined without receiving initial values.

[0039] In some cases, the probability distribution of each neuron may be a categorical probability distribution derived from multiple floating point values, such that all entries of the probability distribution sum to 1 and are non-negative. The probability distribution may be derived from the floating point values ​​via softmax, such that the floating point values ​​are located in a probability simplex, thereby resulting in a categorical probability distribution. Since the probability distributions are trained, the probability distributions of a neural network are sometimes referred to as the parameters of the neural network.

[0040] The probabilities of the probability distribution may be ≧0 and ≦1.

[0041] Softmax may be a conventional function used to convert values ​​into probabilities. Softmax may include a temperature (i.e., a temperature parameter), which may increase the sensitivity to low probabilities. In other words, temperature may be a means of controlling the entropy of a probability distribution (high temperatures increase the entropy, making the probability distribution more uniform, while low temperatures decrease the entropy, accentuating high probabilities).

[0042] Selecting a value for a probability distribution may be accomplished by selecting a most likely value or by selecting a random value. More specifically, selecting a value may involve iteratively selecting random values ​​until a suitable value is found. In some cases, the selection may involve replacing the probability distribution with a mode of the probability distribution. Thus, in a trained neural network, the inputs and outputs of each neuron may be Boolean (0 or 1) and / or the operator of each neuron may be fixed (i.e., a single operator rather than a probability distribution).

[0043] Thus, the logic gate network can be trained by learning the probability distribution of each neuron, since the use of the probability distribution makes it possible to calculate a gradient that can be used to minimize the loss function. Thus, in comparison with the conventional logic gate network described above, the derivative of the loss function can be calculated, and a gradient vector can be obtained. The derivative of the loss function is the direction in which the neural network improves. Thus, being able to calculate the derivative of the loss function makes it possible to improve the neural network during training.

[0044] The input data may include inputs and corresponding desired outputs. The input data may include training data, validation data, and test data. The training data, validation data, and test data may be mutually exclusive or disjoint. In other words, none of the samples in the training data appear in the validation data or the test data, and none of the samples in the test data appear in the validation data or the training data. In this way, overfitting can be determined when using the validation data to evaluate hyperparameters and when using the test data to test the accuracy of the neural network. In some cases, it may be desirable to avoid overfitting. In other cases (sometimes called benign overfitting), even if approximately 100% accuracy is achieved on the training data, the neural network may still be relatively accurate on the validation and test data. The samples may also be referred to as data points.

[0045] In some cases, the training data may be used to train the neural network according to the determined hyperparameter values. The validation data may be used to compare one set of determined hyperparameter values ​​with another set of determined hyperparameter values. The test data may be used to determine the accuracy of the neural network when performing a machine learning task once the final (e.g., most accurate) values ​​of the hyperparameters have been determined.

[0046] The desired outputs are sometimes called labels. Input data may be provided as a set of tuples, where each tuple has the form (input or sample, corresponding output). The samples may be provided as fixed-size vectors (e.g., binary vectors) or may have a known topological structure (e.g., images).

[0047] In some cases, hyperparameter values ​​may be determined to minimize a validation error of the neural network under one or more specified computational resource constraints, which may include processor and / or memory constraints.

[0048] There may be at least five hyperparameters or at least ten hyperparameters. The hyperparameter values ​​may determine the structure of the neural network and how the neural network is trained. The hyperparameters may affect the computation time and memory cost of inference performed through the neural network. In addition, the hyperparameters may affect the accuracy of the neural network. For example, too many neurons per layer may lead to overfitting, while too few layers or too few neurons per layer may lead to underfitting.

[0049] In some cases, multiple sets of hyperparameter values ​​may be determined and a neural network may be trained according to each set of hyperparameter values, and the most accurate of the trained neural network may be used for inference if sufficiently accurate.

[0050] The hyperparameter values ​​(values ​​for multiple hyperparameters) may be chosen randomly, via an optimization algorithm (e.g., grid search or automated machine learning (AutoML)), or via random search. The optimization algorithm may be implemented using constraints provided by a user. The constraints may depend on the machine learning task. In the context of grid search, the constraints are sometimes referred to as a grid. The grid search may be implemented via a traditional grid search algorithm.

[0051] In some cases, the input data may include training data and validation data. Thus, for each set in the multiple sets of hyperparameter values, a neural network may be trained using the training data and the set of hyperparameter values, and then the neural network may be validated for accuracy using the set of hyperparameter values ​​and the validation data. The set of hyperparameter values ​​with the least error with the validation data (i.e., the lowest validation error and most accurate hyperparameter values) may be the determined hyperparameter values.

[0052] The hyperparameters may include one or more of the following: neurons per layer, number of layers, number of epochs, batch size, and learning rate. In the context of classification tasks, the hyperparameters may further include a softmax temperature for training the probability distribution. The softmax temperature may be used to increase as the number of neurons per layer increases. Thus, the softmax temperature may depend on the number of outputs per class (e.g., the softmax temperature may increase as the number of outputs per class increases).

[0053] The number of neurons per layer in the neural network may be determined (e.g., selected from a grid) according to the constraints {100, 1000, 10,000, 100,000, 1,000,000, 10,000,000}. The number of layers in the neural network may be determined (e.g., selected from a grid) according to the constraints {4,5,6,7,8}. Alternatively, a grid of {2,3,4,5,6,7,8,9} may be used. Neural networks with fewer neurons may perform better for certain machine learning tasks, e.g., classification tasks involving tabular data. In such cases, fewer neurons may help avoid overfitting. In contrast, more neurons (e.g., 64,000) may be better for image classification.

[0054] The softmax temperature may be determined (e.g., selected from a grid) according to the constraints {1, 3, 10, 30, 100}. Alternatively, a grid of {1, 1 / 0.3, 1 / 0.1, 1 / 0.03, 1 / 0.01} may be used. Increasing the softmax temperature as the number of neurons per layer increases may improve performance. For example, if the number of neurons per layer is between 2000 and 10,000, a softmax temperature of about 10 may lead to an accurate neural network (low validation error) compared to other softmax temperature values. If the number of neurons per layer is between 12,000 and 100,000, a softmax temperature of about 30 may lead to an accurate neural network. If the number of neurons per layer is greater than 100,000, a softmax temperature of about 100 may lead to an accurate neural network. If the number of neurons per layer is 100, a softmax temperature of less than 3 (e.g., 1 or 3) may lead to an accurate neural network.

[0055] Training the neural network according to the hyperparameter values ​​may include training at a learning rate for a number of epochs at a batch size.

[0056] The number of epochs may be the number of times during training that all training data is presented to the neural network (e.g., provided as input to the neural network). For example, when determining hyperparameters, the number of epochs may be increased until the validation error begins to increase (overfitting) even when the training error is decreasing. A suitable value for the number of epochs may be around 200.

[0057] Batch size may refer to the number of samples (i.e., instances, observations, input vectors, or feature vectors) from the input data that are propagated through the neural network per training iteration. For example, in the context of a classification task of image classification, each sample of input data may include an image and a label for the image. Possible values ​​for batch size may be 32, 64, 128, 256, and other powers of 2. In some cases, a batch size of around 100 may be used.

[0058] The learning rate may specify how much to modify the neural network in response to the validation error for each epoch. When setting the learning rate, there may be a trade-off between the rate of convergence to a low validation error and the increase in training error during training. The learning rate can be set to {0.1, 0.01, 10 -3 ,10 -4 ,10 -5}. For example, a learning rate of about 0.01 may be used.

[0059] Training a neural network may include successively parameterizing the probability distributions. Thus, the probability distribution of each neuron may be parameterized during training (e.g., using softmax to independently select elements from a standard normal distribution). For example, an outcome may be calculated from each probability distribution in the neural network, e.g., during each epoch (i.e., each learning stage). The outcome may be the mean, weighted mean, or another statistical function of the probability distributions.

[0060] The neural network may include multiple layers, including an output layer. The neural network may include at least one hidden layer. At least one layer (e.g., a hidden layer) may precede the output layer. An input to the neural network may be accessed by a first layer, e.g., a first of multiple hidden layers.

[0061] In the case of multiple hidden layers, each hidden layer may have the same number of neurons.

[0062] A neural network may have the same number of neurons in the hidden and output layers. Additionally or alternatively, a neural network may have between 4 and 8 layers.

[0063] When the machine learning task is a classification task, the classification task may have at least two classes. In some cases, the number of neurons per layer may be selected such that each class of the classification task is associated with multiple neurons in the output layer. Thus, by using multiple neurons per class and aggregating them by summation, a hierarchical classification with as many grades (levels) as neurons per class can be performed. Each neuron of a class can capture a different evidence (or aspect) of the class, allowing for finer-grained classification compared to traditional logic gate networks.

[0064] The method may further include performing a machine learning task using the trained neural network after determining the logical operator for each neuron. For a classification task, the output layer may include n neurons and the classification task may have k classes. Thus, performing a classification task using the trained neural network may include: Outputting a classification score for each class by the trained neural network; and grouping the output into k groups of size n / k, where the number of ones in each group corresponds to the classification score of the class corresponding to the group; determining a classification, including determining a maximum of the classification scores.

[0065] Determining the maximum of the classification scores may include using an argmax operation to determine the class with the maximum probability (i.e., the maximum predicted probability). Thus, the maximum of the classification scores may be determined using an argmax operation.

[0066] In some cases, the output bits for each class (i.e., each classification score) may be summed into a binary number to reduce the memory bandwidth required to return the prediction (i.e., classification score). This may be represented via a fixed logic gate network. Specifically, adders may be implemented that can add 1-bits into a binary number with logic gates. Thus, an adder may be implemented for each classification score such that the number of adders corresponds to the number of classification scores. The adders may be modified to be suitable for summing the output bits of the corresponding classes into an integer. In this way, the summation is very efficient, specifically, the summation is faster than storing the unsummed results in VRAM.

[0067] After determining the logical operator for each neuron and before outputting the classification score, the method may further include converting the trained neural network into a binary executable on a central processing unit (CPU) or a graphics processing unit (GPU). In some cases, the executable binary is callable via a program capable of handling shared object binaries, e.g., Python. The binary executable on the CPU may be compiled from C code. The binary executable on the GPU may be compiled from CUDA (formerly Compute Unified Device Architecture).

[0068] The inputs and outputs of a neuron may be real-valued.

[0069] In some cases, during training, the output of all neurons may be a value in the range 0 to 1 (i.e., greater than or equal to 0 and less than or equal to 1). Similarly, the inputs to the neurons may be greater than or equal to 0 and less than or equal to 1. Additionally or alternatively, each neuron may receive between 0 and 3 inputs. More specifically, each neuron may receive between 0 and 2 inputs. For example, a "0" operator may receive 0 inputs, an "A" operator may receive only one input, and an "A-AB" operator may receive two inputs.

[0070] Building the neural network may further include pseudo-randomly initializing the connections between neurons (i.e., the weights of the neural network). Pseudo-random initialization may have the advantage that the connections do not need to be stored and can be generated as needed. In particular, the connections may be generated from a seed (i.e., a random seed used to initialize a pseudo-random number generator) and reproduced from the same seed. The seed may be generated from the state of the computer (e.g., time) or from a hardware-based dedicated random number generator. Building the neural network may also include using other heuristics to pseudo-randomly initialize the connections, for example within groups, or to initialize structures that allow faster computation for certain hardware.

[0071] Thus, once a neural network is trained, all that is required to store the neural network is to store 4 bits of information that specify which logical operator is used for each neuron. In other words, storing a neural network may require n × 4 bits (where n is the number of neurons in the neural network) plus a relatively small constant value. The constant values ​​may include (e.g., be limited to) the seed, and the size of the layers (the number of inputs and the number of outputs). If the layer sizes are the same (i.e., all layers have the same size), then only one size value is required. In this way, the memory requirements of a neural network may be significantly less than traditional neural networks (e.g., dense neural networks, convolutional neural networks, sparse neural networks, or even binary neural networks).

[0072] During training, the connections between neurons may remain fixed, in other words, the neural network does not have to be trained by adjusting the weights of neurons.

[0073] The classification task may include one or more of binary classification, pattern recognition (e.g., feature classification), image classification, object identification or recognition, character recognition, gesture or face recognition, voice detection or speech recognition, and text classification.

[0074] For example, input data for a binary classification task may include binary vectors of size 17. Each of the binary vectors may be classified into class 0 or class 1. In another example, input data for a pattern recognition (specifically, image classification) task may include images of digits 0 through 9, as well as corresponding labels for each of the images. Each of the images may be classified by identifying which digit the image corresponds to.

[0075] Binary classification may include one of medical testing (determining whether a patient has a disease), quality control (determining whether technical specifications have been met), and information retrieval (determining whether a page should be in a search result set).

[0076] The pattern recognition may include image processing (e.g., image classification). The pattern recognition may be used for one of identification and / or authentication, medical diagnosis, defense, and mobility. The identification and / or authentication may include license plate recognition, fingerprint analysis, facial recognition, voice-based authentication. The medical diagnosis may include screening for cervical cancer, breast tumors, or heart sounds. The defense may include navigation and guidance systems, target recognition systems, shape recognition technology. The mobility may include driver assistance systems (to assist with driving and parking) and / or autonomous vehicle technology (in the case of ground vehicles that can navigate safely with little or no human input).

[0077] The machine learning task may be a regression task, and the regression task may include one or more of image generation, text generation, video generation, speech generation, 3D reconstruction, compression, encoding. In particular, the regression task may include video, audio, or image compression. The encoding may include speech encoding.

[0078] Training the neural network may further include determining whether the neural network is sufficiently accurate by comparing the accuracy of the neural network to a specified accuracy. Training is performed in a differentiable manner, i.e., the probability distributions of the logical operators for the neurons of the neural network are learned during training. Thus, training includes determining the loss or error of the neuron probability distributions. In contrast to training, inference (i.e., using the trained neural network) may not necessarily be differentiable and may be performed using fixed logical operators and hard values ​​(0 or 1) rather than relaxed values ​​(probabilities between 0 and 1).

[0079] The accuracy of a neural network may be determined traditionally using a conventional loss function. For example, the softmax cross-entropy classification loss function may be used. In this context, cross-entropy is a divergence measure of the difference between two probability distributions. Alternatively, a mean squared error loss may be used for regression.

[0080] The specified accuracy may be determined according to the machine learning task, for example, the specified accuracy for binary classification may be about 100%, while the specified accuracy for image classification may be at least 95% or at least 97%.

[0081] Training the neural network may further include determining new values ​​for the hyperparameters when the neural network is not sufficiently accurate. The new values ​​for the hyperparameters may be determined iteratively, for example using Bayesian optimization. The neural network may then be reconstructed according to the new values, and the reconstructed neural network may be trained according to the new values. After performing a specified number of iterations (or upon reaching a specified accuracy), a most accurate neural network may be determined from the neural networks and the reconstructed neural networks. The most accurate neural network may be provided as the trained neural network.

[0082] There may be a trade-off between accuracy and efficiency. In particular, neural networks that contain more neurons may be less efficient at inference. Therefore, it may be desirable to reduce accuracy given the efficiency of interest. In addition, new hyperparameter values ​​for subsequent iterations may be determined using knowledge from previous iterations. For example, constraints for determining the hyperparameter values ​​may be modified.

[0083] In some examples, the plurality of logical operators may include at least two operators, at least eight operators, or exactly sixteen operators.

[0084] In some examples, the logical operators may be real-valued (i.e., the relaxed counterparts of logic gates).

[0085] The logical operators may conform to one of the following interpretations: probabilistic, Hamacher t-norm, relativistic Einstein sum t-conorm, Lukasiewicz t-norm and t-conorm.

[0086] Real-valued logical operators may be derived from fuzzy logic.

[0087] A t-norm may be a binary operation used in fuzzy logic to represent conjunctions. Similarly, a t-conorm may be used to represent disjunctions in fuzzy logic.

[0088] The use of real-valued logic operators and / or real-valued inputs may contribute to making a neural network differentiable. Traditional logic gate networks are not differentiable because their parameters are limited to 0 and 1. The use of real values ​​may allow the gradient to be calculated as the derivative (e.g., multi-dimensional derivative) of the loss function with respect to the network parameters, since the use of real values ​​eliminates discontinuities, e.g., jumps from 0 to 1. Thus, a neural network may be referred to as a differentiable logic gate network. Making a neural network differentiable may facilitate training using algorithms based on gradient descent.

[0089] Furthermore, the inputs and outputs of neurons, which are real-valued during training, may be Boolean (fixed) in the trained neural network. Similarly, the logical operators of the trained neural network may be discretized by taking their modes, such that each probability distribution is replaced by the logical operator with the highest value in the probability distribution.

[0090] Real-valued logical operators may include: 0,A·B,A-AB,A, B-AB, B, A+B-2AB, A+B-AB, 1-(A+B-AB),1-(A+B-2AB),1-B,1-B+AB, 1-A, 1-A+AB, 1-AB, 1.

[0091] For the above real-valued logical operators, A and B may be inputs from corresponding neurons. In other words, the real-valued logical operator of a given neuron may perform an operation on an input A from a first neuron and an input B from a second neuron. For 0 and 1, these values ​​may be output regardless of the input. The above real-valued logical operators may correspond to probabilistic interpretations (i.e., probabilistic product T-norm and probabilistic sum T-conorm). However, other interpretations (as described above) may be used.

[0092] In some cases, the input data may include test data having a plurality of samples. The method may further include performing the machine learning task by accessing the total amount of test data by the first layer of the neural network during inference (i.e., after training), where the accessing includes assigning elements of each sample of the test data to successive bits of an integer. Thus, during inference (i.e., after training), the two inputs of each neuron have a numeric data type that corresponds to a hardware implementation data type of a processor of the computer on which the machine learning task is to be performed.

[0093] Thus, the test data aggregate may have an aggregate size, i.e., the aggregate size of the batch being inferred. The aggregate size corresponds to the hardware implementation data type (i.e., assembly language data type) of the processor and is therefore efficient. For example, an aggregate size of 64 can be used for an int64 hardware implementation data type, and an aggregate size of 32 can be used for an int32 hardware implementation data type.

[0094] Specifically, the test data samples may be images and the elements of the samples may be pixels of the images. The numeric data type may be an integer data type, e.g., int64, rather than the Boolean data type traditionally used with binary neural networks. Thus, each neuron input may be a portion of an int64 integer such that the whole integer consists of 64 different neuron inputs. The total size may be 64. Thus, each batch of images provided to a machine learning task may include 64 images, each having 784 binary pixels (i.e., a 28x28 bitmap image), so that the images may be batched (i.e., spread across) into an array of 784 integer variables, each having a hardware-implemented data type of int64. The processor of the computer on which the machine learning task is to be performed may be a CPU or a GPU.

[0095] Accessing the test data by the first layer neurons may correspond to spreading the test data across the first layer neurons. The accessing may include assigning (all) values ​​of image 1 to bit 1 of an integer and (all) values ​​of image 2 to bit 2 of the integer such that each successive image is spread across all integers in the first layer. In other words, Boolean value i of image k may be assigned to bit k of int64 value i.

[0096] Therefore, all logical operations are performed on integers since the operations are the same for each image, so you can usually apply a bitwise operator to all 64 bits of an int64 simultaneously with the same (or comparable) processing cost as applying a bitwise operator to a single value with a Boolean data type.

[0097] In this manner, performance can be significantly improved as compared to the performance of operations on values ​​having a Boolean data type. In particular, this performance improvement relies on the fact that many processors (e.g., many if not most CPUs) that perform bitwise operations on values ​​having a Boolean data type and on values ​​having an integer data type that corresponds to the processor's hardware implemented data type (e.g., an int64 data type) efficiently take the same amount of time, i.e., one instruction. Further efficiencies may be realized by using data types larger than int64, for example, using Advanced Vector Extensions (e.g., AVX, AVX2, AVX-512) instruction sets.

[0098] As an alternative to a CPU or GPU, dedicated hardware, e.g. an ASIC or FPGA, may be used, in which case Boolean data types may be used efficiently.

[0099] According to another aspect, a computer program is provided. The computer program includes instructions that, when the program is executed by a computer, cause the computer to perform the method as described above. The computer program may be part of (or be included in) a computer program product. The computer program may be tangibly embodied in a computer-readable medium. The computer program may be implemented in hardware or software. In particular, the computer program may be implemented using an FPGA or an ASIC. Additionally or alternatively, the computer program may be implemented in a processor (CPU or GPU) component that includes hardware coding (e.g., FPGA or ASIC) that implements a neural network to perform a specific task.

[0100] According to yet another aspect, a computer system for training a neural network to perform a machine learning task is provided. The computer system includes at least one processor. The processor is configured to receive input data for the neural network, determine values ​​for a plurality of hyperparameters of the neural network, and construct the neural network according to the hyperparameter values. The neural network includes a plurality of neurons. Each neuron includes a probability distribution for the logical operators, such that the neuron includes a corresponding probability for each of the plurality of logical operators. The processor is further configured to train the neural network according to the hyperparameter values ​​and the input data by learning the probability distribution for each neuron. The processor is further configured to determine a logical operator of the plurality of logical operators for each neuron by selecting a value in the probability distribution.

[0101] The processor may be a GPU that includes features that can be leveraged to accelerate the training of a neural network for performing a machine learning task (using the techniques of this disclosure) or to accelerate the performance of a machine learning task.

[0102] Additionally or alternatively, the computer system may be implemented using at least one FPGA and / or ASIC.

[0103] Additionally or alternatively, the computer system may include hardware acceleration (e.g., in an ASIC) for the logic gate network.

[0104] The subject matter described in this disclosure may, in some cases, be embodied as a method or on a device in the form of one or more computer programs (e.g., computer program products), which may cause a data processing apparatus to perform one or more operations described in this disclosure.

[0105] The subject matter described in this disclosure may be embodied in a data signal or on a machine-readable medium, the medium being embodied in one or more information carriers, such as a tape, a CD-ROM, a DVD-ROM, a semiconductor memory, or a hard disk. In particular, the subject matter disclosed may be tangibly embodied in a machine (computer) readable medium.

[0106] In addition, the subject matter described in this disclosure can be implemented as a system including at least one processor and a memory coupled to the processor. The processor may be a central processing unit (CPU) or a graphics processing unit (GPU). The memory may encode one or more programs to cause the processor to perform one or more of the methods described in this application. The further subject matter described in this disclosure can be implemented using various machines. The CPU and / or GPU may include an integrated circuit with hardware acceleration, for example hardware acceleration for floating point operations. Alternatively, the processor may be general-purpose hardware without hardware acceleration (e.g., floating point acceleration).

[0107] Furthermore, the subject matter of the present disclosure may be implemented in at least one field programmable gate array (FPGA). The FPGA may be specially designed for artificial intelligence and / or for implementing neural networks in particular. The FPGA may be a configurable hardware accelerated processor capable of efficiently implementing a predefined task (or set of tasks) when the predefined task is represented through logic gates. The FPGA may be particularly suitable for tasks where high speed is required but complexity is limited, such as mining cryptocurrencies (e.g., Bitcoin) or implementing an oscilloscope. Thus, for operations on neural networks, an FPGA specially designed for processing neural networks may be 10 to 100 times faster than a conventional CPU.

[0108] Additionally or alternatively, the subject matter of this disclosure may be implemented using application specific integrated circuits (ASICs) customized for a particular application. For example, ASICs may be developed to support artificial intelligence. In particular, Google's Tensor Processing Unit or Fujitsu's Deep Learning Unit may be used.

[0109] Additionally or alternatively, a hardware implementation of the logic gate network may be used, in particular an FPGA or ASIC may be used to realize the logic gate network, or a GPU may include features that can be exploited to accelerate the training of a neural network for performing a machine learning task (using the techniques of this disclosure) or to accelerate the performance of the machine learning task.

[0110] The details of one or more implementations are set forth in the exemplary drawings and the description below. Other features will be apparent from the description, drawings, and claims. [Brief description of the drawings]

[0111] [Figure 1] A simplified neural network including an input, two hidden layers, and an output layer. [Diagram 2] Figure 2 shows another simplified neural network. [Diagram 3] FIG. 1 illustrates method steps for training a neural network to perform a machine learning task. [Figure 4] Figure 1 shows the distribution of logical operators in a four-layer neural network after training. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0112] In the following document, a detailed description of examples is provided with reference to the drawings. Various modifications to the examples may be made. In particular, one or more elements of one example may be combined and used in another example to form a new example.

[0113] FIG. 1 illustrates an exemplary neural network for performing a machine learning task. More specifically, the neural network is for performing a classification task. The neural network may be constructed according to determined hyperparameter values. Thus, the hyperparameter values ​​may define a structure of the neural network. The hyperparameter values ​​may include multiple layers and multiple neurons per layer. Thus, the neural network may be constructed according to the hyperparameters such that the neural network includes an input 101, a first layer 103, a second layer 105, and an output layer 107. Further, the output layer 107 includes four neurons. Additionally, layers 103 and 105 each include four neurons.

[0114] Thus, layers 103 and 105 of the neural network and output layer 107 may each include the same number of neurons. In other words, the layers of the neural network may each include the same number of neurons.

[0115] The classification task of the simplified neural network has two classes, class 0 and class 1. Thus, the classification task involves binary classification. For example, neurons 3.1 and 3.2 may each output "1", and neurons 3.3 and 3.4 may each output "0". In this case, the number of classes k is 2, and the number of neurons n in the outer layer 107 is 4, so the outputs can be grouped into two groups of size 2. Thus, the classification score for class 0 is 2, and the classification score for class 1 is 0. Thus, by determining the classification according to the maximum classification score, class 0 is determined as the classification. In other examples, more classes may be used.

[0116] Each neuron 1.1-3.4 in the neural network receives two inputs from two different neurons, for example neuron 1.4 in the first layer 103 receives inputs 0.5 and 0.6 from input 101.

[0117] Each neuron may contain a corresponding probability for each of multiple logical operators. The neuron's probability may be part of the neuron's probability distribution, which may be learned as the neural network is trained. The logical operators may be real-valued (i.e., relaxed). The real-valued logical operators may be based on T-norms (relaxed "and") and T-conorms (relaxed "or"). The real-valued logical operators may be differentiable and / or continuous. Thus, neither the extreme T-norms nor the minimum T-norm may provide an adequate basis for the real-valued logical operators.

[0118] Real-valued logical operators are sometimes seen as an extension of traditional Boolean logical operators, in that real-valued logical operators are defined not only for inputs of 0 and 1, but also for inputs between 0 and 1.

[0119] An exemplary probability distribution for neuron 1.1 of layer 103 may include the following probabilities as shown in Table 1:

[0120] [Table 1]

[0121] Table 2 shows the probabilistic interpretations of real-valued logical operators (probabilistic logical operators) and their corresponding Boolean interpretations.

[0122] [Table 2]

[0123] In Table 2, the ID column identifies each row. The Operator column indicates a Boolean operator. A row with a Real Value column indicates a real valued operator corresponding to the Boolean operator in that row. A row with columns "00", "01", "10", "11" indicates the output of the operator (Boolean or real valued) corresponding to that row, given the column header value (e.g., "00") as input. Testing has shown that reducing the number of operators in Table 2 can lead to reduced performance. In other words, using 16 logical operators as opposed to fewer than 16 may improve the efficiency of a method for training a neural network to perform machine learning tasks.

[0124] The values ​​in Table 2 show the probabilistic logical operators. Other interpretations for the real-valued logical operators may be used. For example, the Hamacher product T-norm and its dual, the Einstein sum T-conorm, may be differentiable and provide a suitable basis for the real-valued logical operators. The real-valued logical operators based on the Hamacher product T-norm and the Einstein sum T-conorm are shown in Table 3 below.

[0125] [Table 3]

[0126] Some operators in Table 3 can be derived from other operators in Table 3. In rows 2, 4, and 9 of Table 3, the respective Boolean operators do not have corresponding real-valued operators. For implications (rows 2 and 4 of Table 3), the R-implications (or residuals) corresponding to the T-norms may be used (see "Continuous R-implications," B. Jayaram et al., July 20-24, 2009). The probabilistic operators in Table 2 have been found to perform better than the Hamacher T-norm and Einstein sum T-conorm of Table 3 in testing.

[0127] Other implementations of real-valued operators may be based on the Frank T-norm, the Yager T-norm, the Aczel-Alsina T-norm, the Dombi T-norm, and the Sugeno-Weber T-norm, in addition to the corresponding T-conorms. Further information on logical operators can be found in "Analyzing Differentiable Fuzzy Logic Operators", van Krieken et al., August 24, 2021.

[0128] Further T-norms and T-conorms that may be used to implement real-valued operators are shown in Tables 4 and 5 below.

[0129] [Table 4]

[0130] [Table 5]

[0131] Exemplary values ​​for the neurons of FIG. 1 during training are provided in the following table. Logical operator probabilities during mid-training of neural networks JPEG2025505431000007.jpg72170Logic operator probability in the later stages of neural network training, i.e. after convergence JPEG2025505431000008.jpg78170

[0132] The neuron in FIG. 1 that corresponds to each row of logical operator probabilities shown above is provided at the beginning of the row. For example, the row beginning with "1.1" shows the values ​​for neuron 1.1 in layer 103. Each column in a of the logical operator probabilities rows corresponds to a successive ID in Table 2. For example, row 1 of the logical operator probabilities corresponds to ID 0 in Table 2, row 2 of the logical operator probabilities corresponds to ID 1, etc. Thus, at the midpoint of training, the highest probability for neuron 3.2 is 0.678, which corresponds to ID 10 in Table 2, i.e., real-valued operator 1-B.

[0133] Mid-stage training may refer to the number of epochs that the network is trained for. Thus, mid-stage training may refer to approximately half the number of epochs. Similarly, late-stage training may refer to epochs within 10% of the final epoch. For example, if a neural network is trained for 200 epochs, mid-stage training may be epochs 90 to 110. Similarly, late-stage training may be epochs 190 to epoch 200. Convergence may refer to the stage of training where additional training no longer improves the neural network.

[0134] As can be seen from the example values ​​above, in the mid-training stage, for each neuron there is a number of probabilities of logical operators between 0 and 1. However, after convergence, typically only one or two of the probabilities have a value other than 0, and at least one value is close to 1.

[0135] Figure 2 shows a neural network for performing a regression task. In this case, instead of "class 0" or "class 1", the output of the neural network is a value in the output layer 107. More specifically, the output may be an array of Boolean values ​​(i.e., a vector of values ​​where each value may be either 0 or 1). Alternatively, there may be a scalar output.

[0136] More specifically, there may be n neurons in the output layer 107 to generate (i.e., predict) k-dimensional outputs. The outputs may be grouped into k groups (e.g., of size n / k). For each dimension i of the output (prediction), α i A scalar parameter α such that n / k is valid and provides a determined (desired) range for the output (prediction). i There may be.

[0137] For example, if the regression task is to predict rainfall, there may be one dimension, if the regression task is to predict rainfall and wind speed, there may be two dimensions. The determined range of rainfall may be 0-200 mm. In another example, if the regression task is image generation, k may be the number of pixels, so to generate an image of 784 pixels, k may be 784.

[0138] The final output may be determined by counting the 1 values ​​of the output neurons in the output layer 107.

[0139] A bias β is used to shift the determined range (i.e., output space) to generate both positive and negative outputs. i Therefore, the final output may be determined by counting the output neurons and applying an affine transformation as shown in the following equation:

number

[0140] An affine transformation may be used to transform (or shift) the determined range from 0 to n / k to a more suitable application specific range.

[0141] In some cases, α = α for all i i and β = β i All dimensions of the output have the same range, such that

[0142] In some cases, it may be desirable for the determined range to cover all real numbers, which can be achieved using a logit transform:

number

number

[0143] Using the outputs, we can formulate the mean squared error (MSE) loss as follows:

number

[0144] In another respect, the neural network of FIG. 2 corresponds to the neural network of FIG.

[0145] Figure 3 illustrates a method for training a neural network to perform a machine learning task. The neural network may correspond to the neural networks of Figures 1 and 2 described above.

[0146] The neural network may be referred to as a deep differentiable logic gate network, which after training, implements all calculations as binary operations on Boolean values, rather than floating-point calculations typically implemented in conventional neural networks. The operators for each neuron of the neural network may be realized exclusively using logic gates. This may lead to a very sparse network and increased efficiency when performing machine learning tasks with the trained neural network. Efficiency may be further increased by spreading data for the machine learning task (e.g., test data) across the neurons of the first layer 103.

[0147] In step S201, input data for the neural network is received. The input data may be received as binary valued inputs 101 such that each neuron in layers 103-105 receives two inputs. The input data may include training data, validation data, and test data. The training data may be used to learn probability distributions for each neuron. The validation data may be used to determine whether the neural network is sufficiently accurate, i.e., whether training is complete. The test data may be used during inference (after training is complete), for example, to evaluate the neural network against other or conventional neural networks.

[0148] In step S203, values ​​for a number of hyperparameters of the neural network are determined. The hyperparameters may include the number of layers (e.g., from about 2 to about 32), the number of neurons in each layer (e.g., from about 12 to about 1,024,000), and a learning rate.

[0149] For example, each layer may have the same number of neurons. Additionally or alternatively, there may be about 4 to about 8 layers. This may lead to the advantage that the architecture / structure of the neural network does not need to be fine-tuned. This may therefore simplify and speed up the determination of hyperparameters.

[0150] The learning rate may have a constant value of approximately 0.01.

[0151] In step S205, a neural network may be constructed according to the hyperparameter values. This may include generating the number of layers (e.g., 4) and the number of neurons per layer (e.g., 8000) according to the hyperparameter values. The constructing may include pseudo-randomly initializing the connections between the neurons, i.e., the weights of the network. Other means of initializing the connections (i.e., the weights) are also possible. For example, a trained neural network with weights determined according to a conventional approach may be used as a basis.

[0152] Each neuron may contain a probability distribution for multiple logical operators, each of which may have the following signature: f:{0,1}x{0,1}→{0,1}

[0153] Instead of hard binary values ​​p∈{0,1}, the probabilities in a probability distribution are p∈[0,1] 16 This may be the step to make the neural network differentiable.

[0154] In a neural network, a single neuron may be defined as follows: The two inputs to the neuron may be defined as a, b∈[0,1]. Thus, p is a probability simplex Δ 15 is a probability distribution over logical operators. p is

number

number

[0155] The output o may be ∈[0,1].

[0156] Since neurons of a neural network can be distinguished from one another even with binary inputs, i.e., the network values ​​are no longer constrained to ∈{0.1}, training can be performed (e.g., using an algorithm based on gradient descent) by representing the choice of which logical operators (i.e., logic gates) are present at each neuron by a probability distribution (e.g., a categorical probability distribution).

[0157] Advantageously, the memory requirements for storing the neural network may be significantly less than for neural networks that perform floating-point operations instead of using logical operators. For example, the plurality of logical operators may consist of 16 logical operators. Thus, after the neural network is trained, it takes only 4 bits to represent the logical operator for a given neuron. In other words, it takes only 4 bits to store information that specifies the operation that a given neuron performs. This may be significantly less memory than is required for neural networks whose neurons perform more complex operations.

[0158] In a traditional sparse neural network, the connections between the neurons (weights) of the sparse neural network may be trained.

[0159] In step S207, the neural network may be trained according to the hyper-parameter values. In contrast to conventional sparse neural networks, training the neural network may involve learning which logical operators (i.e., binary functions) to realize at each neuron while keeping the connections between neurons fixed after initialization. Thus, the learning objective may be to determine which of the logical operators should be present at each neuron. The network may thus be parameterized (successively) by learning probability distributions for the logical operators at each neuron.

[0160] Neural networks may be relaxed. In other words, instead of fixed logical operators, probability distributions of logical operators may be used and the neural network may work on probabilities during training. Relaxing the logical operators may be another step in making the neural network differentiable. Learning the probability distributions for each neuron may be implemented via a (relaxed) softmax parameterization.

[0161] Learning the probability distribution may be accomplished by parameterizing each neuron with 16 floating point values ​​corresponding to 16 logical operators. A softmax may be used to map the 16 floating point values ​​to a probability simplex (i.e., a categorical probability distribution where all entries sum to 1 and only non-negative values ​​exist). With reference to the discussion of Equation 4 above, parameterizing the neurons during initialization of the neural network may include selecting (depicting) the elements of q independently from a standard normal distribution.

[0162] Training may involve evaluating all 16 logical operators for each neuron and computing their weighted average using categorical probability distributions. Thus, during training, the output of every neuron may be ∈[0,1].

[0163] More specifically, for a classification task with k classes and n neurons in the output layer, the outputs may be grouped into k groups of size n / k. In this way, the classification score for each of the classes may be determined by counting the number of 1s in each group. Thus, in the context of a classification task, the output of the neural network may be determined by taking the argmax of the classification scores.

[0164] To determine whether a neural network is sufficiently accurate, the probability of the output in each group may be added together, rather than counting the number of 1s. Thus, a measure of accuracy may be determined by calculating the classification loss. For example, the softmax cross entropy classification loss may be calculated as follows:

number

[0165] After training is complete, the two inputs to the neuron may be defined as a, b ∈ {0,1}. Thus, in contrast to a neuron during training (see Equation 4), where the inputs to the neuron may be floating-point values ​​≧0 and ≦1, the inputs to the neuron after training may be Boolean values, i.e., either 0 or 1. Similarly, the output o of the neuron may be ∈ {0,1}. Thus, the output o of the neuron after training may be defined as o=0p γ (a,b) (Equation 6) In the formula, γ=argmax in formula (6) i p i where i refers to the ID in Table 2.

[0166] In step S209, a logical operator may be determined for each neuron. The logical operator may be determined after training is completed, i.e. during inference. The determined logical operator may be the logical operator of maximum likelihood, e.g., the logical operator of the probability distribution with the highest probability. In other words, the probability distributions may be discretized by taking their modes. Thus, the machine learning task may be performed by computing Boolean values ​​rather than floating point values, which makes classification efficient compared to neural networks that rely on floating point arithmetic.

[0167] Prior to using the trained neural network to perform a machine learning task, the neural network may be compiled into at least one binary executable. The binary executable may be processor-dependent. For example, two binary executables may be compiled, one for the CPU (e.g., from C code) and one for the GPU (e.g., from CUDA).

[0168] Additionally, logical expressions and / or sub-representations may be simplified, for example, instead of Boolean data types, hardware implemented data types corresponding to the processor on which the machine learning task is to be performed may be used.

[0169] For example, for a 64-bit CPU, the hardware implementation data type may be int64. Additionally, an aggregate size of 64 may be used, i.e., 64 samples (e.g., images) are processed through the neural network at a given iteration (i.e., epoch). In a GPU, output neurons may be directly aggregated using the logic gates that make up each adder, since any write to the GPU memory may result in a bottleneck (i.e., reduced performance). In general, the bottleneck (point in the system that may cause a reduction in speed) may be the data loader and / or transmission speed.

[0170] Thus, bitwise operations may be performed on large batches, which can have a significant impact on the speed at which machine learning tasks can be performed.

[0171] The output layer 107 may generate multiple outputs for each class. The outputs may be aggregated via bit counting, i.e., by counting the ones resulting in a score for each class. Thus, if the machine learning task is a classification task, the classification task may be completed by providing the class with the highest score as the output of the neural network.

[0172] When performing a machine learning task with a binary vector given as input, pairs of Boolean values ​​may be selected from the binary vector, logical operators (i.e., binary logic gates) in one of the layers (e.g., layer 103) may be applied to the Boolean values, and those outputs may then be used in subsequent layers of the neural network (e.g., layers 105 or 107).

[0173] After training, the computational cost of performing machine learning tasks may be reduced by at least an order of magnitude compared to traditional binary and sparse neural networks, and possibly even compared to other types of traditional neural networks.

[0174] Exemplary machine learning tasks are the Monk problems, as discussed in "The monk's problems: A performance comparison of different learning algorithms", Thrun et al., 1991. The Monk problems, MONK-1, MONK-2, and MONK-3, are three machine learning tasks that have been used to benchmark machine learning algorithms. These tasks consist of three binary classification tasks on a data set with six attributes and two to four possible values ​​each. Thus, the data points (samples) can be encoded as binary vectors of size 17.

[0175] Tests show that the above neural network performs more accurate classification than logistic regression for all three MONK data sets. In addition, for MONK-3, the above neural network (i.e., Fig. 1 and Fig. 2) is more accurate than a much larger convolutional neural network. In addition, the above neural network is more than three times faster than logistic regression and more than seven times faster than a larger convolutional neural network. In addition, the above neural network requires significantly less storage space than either logistic regression or a convolutional neural network.

[0176] As another exemplary machine learning task, the adult census ("Uci machine learning repository: Adult data set", Kohavi et al., 1996) and breast cancer ("Uci machine learning repository breast cancer dataset", Zwitter et al., 1988) data sets may be considered. For the adult data set, the machine learning task is to predict whether a given adult will earn more than $50,000 per year based on attributes such as education and hours worked per week. For breast cancer, the machine learning task involves binary classification, determining whether the cancer diagnosis is benign or malignant based on properties of the cell nucleus, including perimeter, area, and smoothness. For these tasks, the classification accuracy achieved by the above-mentioned neural networks is comparable to traditional (convolutional) neural networks and logistic regression. In addition, the classification speed is more than 10 times faster than logistic regression and more than 40 times faster than traditional neural networks. In addition, the neural network described above requires approximately 20% less memory space than logistic regression and approximately 75% less than traditional neural networks.

[0177] Furthermore, very high frame rates may be achieved for image classification. For example, in the case of the neural network mentioned above, frame rates of over 1 million images per second may be achieved for images from the Modified National Institute of Standards and Technology (MNIST) dataset (http: / / yann.lecun.com / exdb / mnist / ) and the Canadian Institute For Advanced Research (CIFAR-10) dataset ("Learning Multiple Layers of Features from Tiny Images", Alex Krizhevsky, April 8, 2009). In other words, classification rates of over 1 million images per second may be achieved using a single CPU core. This may exceed the efficiency of any conventional approach.

[0178] More specifically, the above-described neural network may have image classification accuracy on the MNIST dataset comparable to the fastest conventional binary neural network, while requiring less than 10% of the number of binary operations. On a standard GPU (e.g., NVIDIA A6000), the above-described neural network requires only 7% of the GPU utilization, while the above-described neural network may be implemented 12 times faster than a conventional binary neural network on dedicated FPGA hardware. Compared to another conventional binary neural network, the above-described neural network may be three orders of magnitude faster. Compared to an evolutionarily trained sparse function network, the above-described neural network is more accurate.

[0179] For image classification using the CIFAR-10 dataset, the accuracy of the above neural networks can be comparable to traditional convolutional neural networks while requiring less than 0.1% of the memory in some cases and less than 1% of the memory in others. A dedicated fully connected network is slightly more accurate (less than 4%) but requires 64% more memory.

[0180] When performing image classification using the CIFAR-10 data set, a traditional fully connected neural network that relies on floating-point arithmetic may require 2,000,000 floating-point operations to perform a machine learning task, whereas the neural network described above requires 5,000,000 bitwise logical operations before pruning or optimization. In floating-point hardware accelerated integrated circuits (e.g., GPUs and many CPUs today), 2,000,000 floating-point operations are approximately 100 times slower than 5,000,000 bitwise logical operations. Without floating-point acceleration, the speed difference would be an order of magnitude larger, i.e., a difference of three orders of magnitude.

[0181] Even though a traditional sparse neural network is faster than a traditional fully connected neural network, it is still at least an order of magnitude slower than the neural networks mentioned above. A sparse neural network requires at least twice as much storage space as the neural networks mentioned above.

[0182] A possible exemplary architecture for the neural network described above (related to Figures 1 and 2) is shown in Table 6 below.

[0183] [Table 6]

[0184] Table 7 shows the configuration of a fully connected ReLU network that was used as a basis for comparison with the neural network configurations in Table 6.

[0185] [Table 7]

[0186] In addition, in each clock cycle (e.g., per Hertz) of a CPU in a typical general-purpose desktop or laptop computer, it may be possible to calculate an average of about 250 binary logic gates for each core of the CPU. This is possible because a typical CPU executes many instructions per clock cycle, even with a single core. This may be significantly faster than would be possible if relying on floating-point arithmetic to execute the neural network. In addition, the CPU may be able to spread data for a machine learning task across the neurons of the first layer 103 by grouping bits from multiple samples of data (e.g., images) into a single integer (e.g., an integer having a data type of int64). Grouping bits from multiple samples into a single integer is sometimes referred to as Single Instruction Multiple Data (SIMD). Additional efficiency gains may be possible using Advanced Vector Extensions (AVX).

[0187] The computational cost of some of the machine learning tasks performed by a layer having n neurons (e.g., layers 103, 105, or 107) may be 0(n), including fine fixed costs, since it requires only Boolean logic gates (i.e., the operations are performed exclusively through logical operators and can therefore be performed very efficiently). In comparison, a fully connected layer with m input neurons has a computational cost of O(n m), with significantly higher fixed costs, especially since fully connected layers generally require floating point operations. Overall, performing machine learning tasks using a neural network trained according to the method described with reference to Figures 1 and 2 may lead to inference speeds that are two orders of magnitude faster than fully connected ReLU neural networks (neural networks that use a ReLU activation function). In addition, a neural network trained according to the method of Figures 1 and 2 may be more than 13 times faster than a conventional binary neural network and two to three orders of magnitude faster than the theoretical speed of a conventional sparse neural network.

[0188] Figure 4 shows the distribution of logic gates for a neural network trained to perform a machine learning task. In this case, the machine learning task involves image classification. The neural network has four layers with 12,000 neurons per layer. For layers 1 and 2, the values ​​on the x-axis correspond to the IDs in Table 2, as well as the values ​​on the x-axis for layers 3 and 4. The values ​​on the y-axis provide a count of the number of neurons for which there are logic operators on the x-axis.

[0189] From Figure 4, we can see that logical operators for constants “0” and “1” (corresponding to IDs 0 and 15 in Table 2) are used very infrequently and do not occur at all in tier 4. In the first tier, “and”, “nand”, “or”, and “nor” have relatively strong probabilities.

[0190] In the second and third layers, one of the two inputs is ignored and the other input is passed forward (possibly in a modified form) to the next layer, resulting in more occurrences of "A", "B", "┐A", and "┐B", which can be seen as residual direct connections. This may allow the neural network to model lower-order dependencies more efficiently by representing them in fewer than a predefined number of layers.

[0191] In layer 4, the most frequent operations are "xor" and "xnor", which can create conditional dependencies on activations of previous layers. As shown, implication is used less frequently. However, tests show that using only a proper subset of logical operators (rather than all 16 operators of the probabilistic interpretation in Table 2) can lead to reduced accuracy.

Claims

1. 1. A computer-implemented method for training a neural network to perform a machine learning task, such as image classification, comprising: Receiving input data for the neural network (S201); Determining values ​​for a plurality of hyperparameters of the neural network (S203); constructing (S205) the neural network comprising a plurality of neurons (1.1-3.4) according to the hyper-parameter values, constructing said neural network, each neuron (1.1-3.4) including a probability distribution for each of a plurality of logical operators, such that said neuron (1.3-3.4) includes a corresponding probability for each of said logical operators; training (S207) the neural network according to the hyperparameter values ​​and the input data by learning the probability distribution of each neuron (1.3-3.4); determining (S209) a logical operator of the plurality of logical operators for each neuron (1.3-3.4) by selecting a value in the probability distribution; 4. A computer-implemented method comprising:

2. the input data includes an input and a corresponding desired output; the input data includes one or more of training data, validation data, and test data; The method of claim 1.

3. the hyper-parameter values ​​are determined to minimize a validation error of the neural network under one or more specified computational resource constraints; The specified computational resource constraints may include processor and / or memory constraints; The method according to claim 1 or 2.

4. the hyperparameters include one or more of: neurons per layer, number of layers, number of epochs, batch size, and learning rate; 4. The method according to any one of claims 1 to 3.

5. The method of any one of claims 1 to 4, wherein the neural network comprises a number of layers including an output layer (107).

6. the output layer (107) includes n neurons, the machine learning task is a classification task with k classes, The method further includes performing the classification task using the trained neural network, the performing the classification task comprising: outputting a classification score for each of the classes by the trained neural network; and Grouping the output into k groups of size n / k, where the number of ones in each group corresponds to the classification score of the class to which the group corresponds; determining a classification, including determining a maximum of the classification scores; The method of claim 5 , comprising:

7. After training the neural network and before performing the machine learning task, the method further comprises: converting the trained neural network into a binary executable on a central processing unit or a graphics processing unit; The executable binary is callable via a program that can handle shared object binaries, which program may be in Python; 7. The method according to any one of claims 1 to 6.

8. the output of each neuron is a value ranging from 0 to 1; and / or Each neuron receives 0 to 3 inputs, 8. The method according to any one of claims 1 to 7.

9. constructing the neural network further comprises pseudo-randomly initializing connections between the neurons; During the training, connections between neurons may remain fixed.

9. The method according to any one of claims 1 to 8.

10. The machine learning task is: Binary classification, pattern recognition, image classification, object identification or recognition, character recognition, gesture or face recognition, voice detection or speech recognition, text classification The method of claim 1 , wherein the classification task comprises one or more of:

11. The machine learning task is a regression task, the regression task comprising: Image generation, text generation, video generation, speech generation, 3D reconstruction, compression, encoding The method of any one of claims 1 to 10, comprising one or more of the following:

12. the plurality of logical operators includes at least two operators or exactly 16 operators; The logical operators may be real-valued operators; The real-valued operator may conform to one of the following interpretations: probabilistic, Hamacher t-norm, relative Einstein sum t-conorm, Lukasiewicz t-norm, and t-conorm; 12. The method according to any one of claims 1 to 11.

13. the input data includes test data having a plurality of samples; performing the machine learning task by accessing, during inference, a total of the test data by a first layer (103) of the neural network, said accessing including assigning elements of each sample of the test data to consecutive bits of an integer number; During inference, the two inputs of each neuron have a numeric data type that corresponds to a hardware implementation data type of a processor of a computer on which the machine learning task is to be performed.

13. The method according to any one of claims 1 to 12.

14. A computer program comprising instructions which, when said program is executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 13.

15. 1. A computer system for training a neural network to perform a machine learning task, such as image classification, comprising: At least one processor, the processor comprising: receiving input data for the neural network; determining values ​​for a plurality of hyperparameters of the neural network; constructing said neural network comprising a plurality of neurons (1.3-3.4) according to said hyper-parameter values; constructing said neural network, each neuron (1.3-3.4) including a probability distribution for each of a plurality of logical operators, said neuron (1.3-3.4) including a corresponding probability for each of said logical operators; training the neural network according to the hyperparameter values ​​and the input data by learning the probability distributions for each neuron (1.3-3.4); Determine the logical operator of the plurality of logical operators for each neuron (1.3-3.4) by selecting a value in the probability distribution. The computer system is configured to:

Images