Training neural networks to perform machine learning tasks

Abstract

To provide a method, program, and system for training a neural network to perform machine learning tasks. The method includes receiving input data for a neural network. The method further includes determining values ​​for a plurality of hyperparameters of the neural network. The method further includes constructing a neural network according to the hyperparameter values, the neural network comprising a plurality of neurons. Each neuron includes a probability distribution for the logical operators, such that the neuron includes a corresponding probability for each of the plurality of logical operators. The method further includes training the neural network according to the hyperparameter values ​​and the input data by learning the probability distribution for each neuron. The method further includes determining a logical operator of the plurality of logical operators for each neuron by selecting a value in the probability distribution.

Description

[Technical Field]

[0001] The technical field is artificial intelligence, particularly neural networks. Aspects relate to training a neural network to perform a machine learning task. The machine learning task may be a classification task or a regression task. The classification task may include one or more of pattern and / or sequence recognition, novelty detection (i.e., outlier or anomaly detection), and sequential decision making. The regression task may include compression or recovery. Additionally, aspects may relate to performing a machine learning task using a trained neural network. [Background technology]

[0002] A neural network may include artificial neurons (i.e., neurons, nodes, computational units, or units) and may be used to solve artificial intelligence problems. The connections between neurons may be modeled as weights in the neural network. A neural network may include parameters, which are components or elements of the neural network that can be trained. For example, weights may be parameters in some neural networks. However, other parameters are possible.

[0003] Activation functions may be used to control the magnitude of the neural network's output. Thus, activation functions may be used to limit the value of the neural network's output and add nonlinearity to the neural network.

[0004] Neural networks may be trained using an optimization algorithm. Training may involve adjusting the parameters of the neural network until the neural network is sufficiently accurate. The optimization algorithm may be based on (or rely on) gradient descent. Examples of optimization algorithms based on gradient descent include stochastic gradient descent (SGD) and Adam (an adaptive learning rate optimization algorithm—realized as a combination of root-mean-square propagation and SGD with momentum). More information on Adam can be found in "Adam: A method for stochastic optimization," Kingma et al., 2014.

[0005] Gradient descent may be based on backpropagation (i.e., backprop or backward propagation). Gradient descent may include a cost function that measures the accuracy of the parameters and determines the direction in which parameter changes can improve the network, thereby improving the parameters. Sufficient accuracy is achieved when the output of the cost function is zero, at a minimum, or within a specified range of zero. The specified range may depend on the machine learning task. In some cases, gradient descent (or algorithms based on gradient descent) can only be used to train neural networks with differentiable parameters, so binary or Boolean parameters cannot be trained via gradient descent. Boolean parameters (represented as 0 or 1 in neural networks) can lead to discontinuities (e.g., jumps from 0 to 1), in which case the gradient is ill-defined and gradient descent is impossible.

[0006] The accuracy of a neural network may be calculated in a variety of ways. For example, in the context of classification tasks, training error (or loss) may be obtained by measuring how accurately a neural network can classify the data used to train it. Classification loss (also known as generalization error or expected loss) may be obtained by measuring how accurately a neural network can perform a classification task on previously unseen data.

[0007] In some cases, a neural network may include multiple layers, i.e., at least one hidden layer and an output layer. In such cases, the neural network may be called a deep learning neural network, and may have an infinite number of hidden layers. Thus, multiple layers may be used to incrementally extract higher-level features from the input. A neural network may be implemented as a feedforward network, where data flows from the previous hidden layer to the output layer without a back-loop. Initialization of a neural network may involve assigning random weights to the connections between neurons.

[0008] Traditionally, if a neural network does not perform a machine learning task accurately, the weights are adjusted, thus calibrating the influence of different neurons within the network until the proper mechanical manipulation of the inputs to perform the machine learning task is determined.

[0009] Once trained to perform a machine learning task, a neural network may be used to perform the machine learning task, which is sometimes called inference or performing inference.

[0010] Neural networks face a variety of problems. For example, overfitting can occur when the training error is small but the classification loss is relatively large. In other words, the neural network learns an approach specific to the training data used to train the neural network that does not generalize well to other data beyond the training data. Underfitting can occur when a neural network cannot accurately capture the relationship between its input and output, resulting in high error rates on both the training data and other data.

[0011] Although neural networks can be effective at performing classification tasks, they can also be inefficient. Therefore, various approaches have been developed to increase the efficiency of neural networks.

[0012] For example, a traditional neural network may be a deep learning neural network that includes at least one dense layer, such that part or all of the neural network is fully connected, meaning that each neuron in the dense layer receives input for every neuron in the previous layer. In this example, the neural network may be called a densely connected neural network or a fully connected neural network.

[0013] Additionally, traditional neural networks may use full-precision (e.g., double-precision, i.e., 64-bit) floating-point calculations for weights and activations, which generally require expensive matrix multiplications. Even with lower-precision floating-point weights and activations (e.g., single-precision or half-precision), the calculations can still be expensive.

[0014] Binary neural networks may differ from conventional neural networks in that single-bit activations (and / or weights) replace the floating-point activations (and / or weights) of conventional neural networks. Therefore, elements propagated through the network are binary in the sense that they can have only one of two values, e.g., 0 or 1. Therefore, expensive matrix multiplications compared to conventional neural networks may be simplified to faster XNOR and bit-counting operations, which significantly improves binary neural network inference performance compared to conventional neural networks. In some cases, XNOR may be faster than bit-counting operations, and both may be faster than floating-point operations.

[0015] The floating-point operations required to train neural networks may be supported by computer hardware. Specifically, a float32 adder and / or a float32 multiplier may require at least 1,000 logic gates (or look-up tables) and may have delays of tens of logic levels. The float32 adder and / or multiplier may be implemented directly in a central processing unit (CPU) or a graphics processing unit (GPU). In either case, float32 adders and multipliers may require significantly more computational resources (e.g., processing cycles) than bitwise logical operations on the int64 data type, even on GPUs optimized for float32 and int32 data types. If a GPU is optimized for int32, int64 may not be supported efficiently. Therefore, it may be more efficient to use the int32 data type for such GPUs. In some hardware implementations, logical operations may be faster than addition and / or multiplication operations.

[0016] A CPU may be able to perform approximately 3 to 10 int64 bitwise operations per clock cycle, whereas floating-point operations typically require an entire clock cycle (i.e., only one floating-point operation may be performed per clock cycle). Therefore, XNOR and bit-counting operations performed when training binary neural networks may have significant performance advantages compared to floating-point operations performed for traditional neural networks, even if the floating-point operations (e.g., float32 adders or float32 multipliers) have dedicated hardware support in the GPU or CPU. Considerations regarding floating-point and integer operations may also be applicable to training neural networks for machine learning tasks.

[0017] A convolutional neural network (CNN) may be a deep learning neural network in which at least one hidden layer has structured connections, i.e., is not fully connected. Thus, a CNN may perform convolutions, e.g., a hidden layer performs a dot product of a convolution kernel with the hidden layer's input matrix. Thus, only neurons in the hidden layer that are covered by a convolution kernel may be connected to respective neurons in another layer of the CNN.

[0018] In a traditional example, a binary neural network may be based on a very deep convolutional neural network (e.g., VGG) architecture, which increases depth through very small (3x3) convolutional filters, includes three fully connected layers, and uses rectified linear units (ReLU) to reduce training time. Alternatively, a binary neural network may be based on a residual neural network (ResNet) architecture, which uses skip connections to skip some layers. Both VGG and ResNet architectures can result in very large neural networks, which can lead to inefficiencies.

[0019] A neural network may be sparse (in this example, a sparse neural network). Thus, a sparse neural network has a proper subset of the connections of a fully connected neural network. In other words, instead of fully connected layers, not all neurons in a layer of a sparse neural network are connected to each other. Sparseness may refer to the connectivity of a neural network, such that each neuron receives input from only a limited number of other neurons, and / or it may refer to the state of a neural network, describing the level of activity of all neurons in a neural network, such that not all neurons in a sparse neural network are active at any given time.

[0020] A sparse neural network may have fewer than 90% of the connections of a fully connected neural network; that is, a sparse neural network may have at least 10% sparsity. (10%-70% sparsity is sometimes called low sparsity, 70%-90% sparsity is sometimes called medium sparsity, 90%-99.9% sparsity is sometimes called moderate sparsity, and greater than 99.9% sparsity is sometimes called high sparsity.) Neural networks with less than 10% sparsity are sometimes called dense.

[0021] Sparse neural networks may be derived from dense or fully connected neural networks using pruning or compression techniques, such as regularization techniques. For example, one or more analytical methods or heuristics may be used to determine which weights should be retained and which should be removed from the neural network (removing a weight may involve setting it to zero). More specifically, L1 or L2 regularization may be used to update the general cost function by adding a regularization term. As yet another example, dropout is a regularization technique in which random neurons in a neural network are cancelled or removed. Dropout may be variational and may be limited by bounds (e.g., a limit on how many neurons are cancelled). In some cases, rather than removing weights, neurons themselves may be removed from the neural network. Pruning techniques, as discussed above, can help avoid overfitting and thereby reduce generalization error. Other techniques may involve removing entire layers of a neural network or adding additional cost to relatively large weights in a neural network. Therefore, sparse neural networks may provide improved accuracy (e.g., less overfitting) compared to dense neural networks. Sparse neural networks may also require less storage space compared to dense neural networks. However, if a sparse neural network is derived by removing weights from a dense neural network, the sparse neural network may require more storage space to store the weight positions in addition to the actual weights.

[0022] In two examples of conventional implementations, the connectivity of the neural network may be relaxed:

[0023] In a first example, the neurons of a neural network may be implemented as logic modules, half of which are defined by fuzzy (i.e., relaxed) conjunctions and the other half by fuzzy (i.e., relaxed) disjunctions. In this example, the parameters of the neural network include weights, but the operators (i.e., fuzzy conjunctions and disjunctions) are fixed. Thus, the weights of neurons involved in fuzzy conjunctions and disjunctions may be relaxed from being exclusively 0 or 1 to values ​​in the range 0 to 1. In this way, the parameters of the logic modules, unlike Boolean parameters, can be trained via gradient descent.

[0024] In the second example, a set of arithmetic operations is predefined for each layer and implemented for the neurons in that layer. A form of softmax is used for training, with the output of the previous layer being used as the input for the arithmetic operation corresponding to the current layer. In this way, a symbolic representation can be learned from the data.

[0025] In another conventional example, fully connected layers of an artificial neural network are replaced with sparse layers before training to quadratically reduce the number of parameters without significantly reducing accuracy (e.g., by more than 5% or 10%). The sparse topology has two successive layers and evolves into a scale-free topology during training. The scale-free topology has a power-law frequency distribution P(d)~d -y where a proportion P(d) of all neurons in the network have d connections to other neurons, and the parameter γ typically remains in the range γ∈(2,3).

[0026] In yet another conventional example, a single shared weight parameter is assigned to all connections in a neural network. Instead of optimizing weights, the neural network is optimized for a network topology that performs well over a wide range of weights. Neurons and connections are added incrementally, and activation functions (e.g., sine, cosine, Gaussian, hyperbolic tangent, sigmoid, inverse, absolute value, ReLU) are randomly assigned to new neurons. The activation functions evolve as the neural network grows. The neural network uses floating-point operators.

[0027] Another prior art example provides a method for memorizing a binary classification data set using a network of binary (two-input) lookup tables. The network of lookup tables is constructed by counting conditional frequencies in the training data (e.g., how many times a pattern p is associated with a particular output 0 in the training data). Like neural networks, the lookup tables are arranged in successive layers. Unlike neural networks, training is via memorization and does not involve backpropagation or gradient descent. Memorization involves remembering the outputs most commonly associated with the inputs in the training data.

[0028] In a further conventional example, a neural network is first trained for a machine learning task, then the neural network is translated into a random forest, and the random forest is then translated into a network of AND-Inverter logic gates, i.e., the network is based on "AND" and "NOT" logic gates. Summary of the Invention [Problem to be solved by the invention]

[0029] It may be desirable to provide a neural network for performing machine learning tasks that is more efficient than the prior art examples described above while maintaining comparable accuracy.

[0030] For example, logic gate networks may be used. A logic gate network may be a neural network in which each neuron contains a logical operator (i.e., a logic gate such as "AND" or "NAND"). Each logical operator may have zero to two inputs. Logic gate networks are sparse compared to fully connected neural networks, in that all neurons in the previous layer provide inputs to each neuron in the current layer. In other words, logic gate networks are sparse because each neuron in a logic gate network receives only two inputs, rather than n inputs (where n is the number of neurons per layer). Therefore, machine learning tasks may be performed efficiently because logical operations can be calculated very quickly (e.g., because logical operations are often built into processor architectures). Additionally, because each neuron has a maximum of two inputs, logic gate networks can be very sparse, e.g., at least intermediate sparsity, which can lead to improved accuracy compared to denser networks.

[0031] Optionally, the logic gate network may include at least one operator with three or more inputs.

[0032] Advantageously, logic gate networks may not require activation functions because they operate on binary values ​​(e.g., after training) and are nonlinear.

[0033] However, implementing neural networks as logic gate networks can be problematic because conventional logic gate networks cannot be trained using gradient descent-based optimization algorithms such as SGD or Adam (discussed above). This is because the values ​​of conventional logic gate networks are always either 0 or 1. Therefore, it is impossible to calculate the gradient in conventional logic gate networks because the loss function with respect to the parameters of the network's layers has no derivative. Therefore, it may be desirable to provide logic gate networks that can be trained using gradient descent-based optimization algorithms.

[0034] Furthermore, traditional logic gate networks may provide a single binary output (e.g., for regression tasks) or k binary outputs for k classes (e.g., for classification tasks). This makes hierarchical prediction, i.e., ranking classes relative to each other, impossible. In other words, it makes classification according to a "maximum activation" classification scheme ("How a Neural Network Works," Arkin Gupta, May 27, 2018). [Means for solving the problem]

[0035] According to one aspect, a computer-implemented method for training a neural network to perform a machine learning task is provided. The method includes receiving input data for the neural network. The method further includes determining values ​​for multiple hyperparameters of the neural network. The method further includes constructing a neural network according to the hyperparameter values, the neural network comprising multiple neurons. Each neuron includes a probability distribution for a plurality of logical operators, the neuron including a corresponding probability for each of the multiple logical operators. The method further includes training the neural network according to the hyperparameter values ​​and the input data by learning the probability distribution for each neuron. The method further includes determining a logical operator of the multiple logical operators for each neuron by selecting a value in the probability distribution.

[0036] Neural networks are sometimes called binary neural networks (because when a neural network is trained, elements propagating through the network can have only one of two values), logic gate networks (because neurons implement logical operators), or binary logic gate networks. Logical operators are sometimes called logic gates.

[0037] In some cases, there are at most two inputs per neuron. If there are exactly two inputs per neuron and n neurons per layer, the neural network can have a sparsity of at most 1-(2 / n).

[0038] The values ​​of the multiple hyperparameters may be determined from received hyperparameters (e.g., used for performing a similar machine learning task, or used by a similar neural network to perform the same machine learning task), or may be defined without receiving initial values.

[0039] In some cases, the probability distribution for each neuron may be a categorical probability distribution derived from multiple floating-point values, such that all entries of the probability distribution sum to 1 and are non-negative. The probability distribution may also be derived from the floating-point values ​​via softmax, such that the floating-point values ​​are mapped onto a probability simplex, thereby resulting in a categorical probability distribution. Since the probability distributions are trained, the probability distributions of a neural network are sometimes referred to as the parameters of the neural network.

[0040] The probabilities of the probability distribution may be ≧0 and ≦1.

[0041] Softmax may be a conventional function used to convert values ​​to probabilities. Softmax may include a temperature (i.e., a temperature parameter), which may increase sensitivity to low probabilities. In other words, temperature may be a means of controlling the entropy of a probability distribution (high temperatures increase entropy, making the probability distribution more uniform, while low temperatures decrease entropy, accentuating high probabilities).

[0042] Selecting a value for a probability distribution may be accomplished by selecting the most likely value or by selecting a random value. More specifically, selecting a value may involve iteratively selecting random values ​​until a suitable value is found. In some cases, the selection may involve replacing the probability distribution with the mode of the probability distribution. Thus, in a trained neural network, the input and output of each neuron may be Boolean (0 or 1) and / or the operator of each neuron may be fixed (i.e., a single operator rather than a probability distribution).

[0043] Therefore, the use of probability distributions allows us to calculate gradients that can be used to minimize the loss function, so the logic gate network can be trained by learning the probability distributions of each neuron. Therefore, compared to the traditional logic gate network described above, we can calculate the derivative of the loss function and obtain the gradient vector. The derivative of the loss function is the direction in which the neural network improves. Therefore, being able to calculate the derivative of the loss function makes it possible to improve the neural network during training.

[0044] The input data may include inputs and corresponding desired outputs. The input data may include training data, validation data, and test data. The training data, validation data, and test data may be mutually exclusive or disjoint. In other words, none of the samples in the training data appear in the validation or test data, and none of the samples in the test data appear in the validation or training data. In this way, overfitting can be determined when evaluating hyperparameters using the validation data and when testing the accuracy of the neural network using the test data. In some cases, it may be desirable to avoid overfitting. In other cases (sometimes called benign overfitting), even if approximately 100% accuracy is achieved on the training data, the neural network may still be relatively accurate on the validation and test data. Samples may also be referred to as data points.

[0045] In some cases, the training data may be used to train the neural network according to the determined hyperparameter values. The validation data may be used to compare one set of determined hyperparameter values ​​with another set of determined hyperparameter values. The test data may be used to determine the accuracy of the neural network when performing a machine learning task once the final (e.g., most accurate) values ​​of the hyperparameters have been determined.

[0046] The desired output is sometimes called a label. Input data may be provided as a set of tuples, each of which has the form (input or sample, corresponding output). Samples may be provided as fixed-size vectors (e.g., binary vectors) or may have a known topological structure (e.g., images).

[0047] In some cases, hyperparameter values ​​may be determined to minimize the validation error of the neural network under one or more specified computational resource constraints, which may include processor and / or memory constraints.

[0048] There may be at least five hyperparameters or at least ten hyperparameters. The hyperparameter values ​​may determine the structure of the neural network and how it is trained. Hyperparameters may affect the computational time and memory cost of inference performed through the neural network. In addition, hyperparameters may affect the accuracy of the neural network. For example, too many neurons per layer can lead to overfitting, while too few layers or too few neurons per layer can lead to underfitting.

[0049] In some cases, multiple sets of hyperparameter values ​​may be determined, and a neural network may be trained according to each set of hyperparameter values, with the most accurate trained neural network being used for inference if it is sufficiently accurate.

[0050] Hyperparameter values ​​(values ​​for multiple hyperparameters) may be chosen randomly, via an optimization algorithm (e.g., grid search or automated machine learning (AutoML)), or via random search. The optimization algorithm may be implemented using constraints provided by the user. The constraints may depend on the machine learning task. In the context of grid search, the constraints are sometimes referred to as a grid. Grid search may be implemented via a traditional grid search algorithm.

[0051] In some cases, the input data may include training data and validation data. Thus, for each set in the plurality of sets of hyperparameter values, a neural network may be trained using the training data and the set of hyperparameter values, and then the neural network may be validated for accuracy using the set of hyperparameter values ​​and the validation data. The set of hyperparameter values ​​with the lowest error from the validation data (i.e., the lowest validation error and most accurate hyperparameter values) may be the determined hyperparameter values.

[0052] The hyperparameters may include one or more of the following: neurons per layer, number of layers, number of epochs, batch size, and learning rate. In relation to classification tasks, the hyperparameters may further include a softmax temperature for training the probability distribution. The softmax temperature may be used to increase as the number of neurons per layer increases. Thus, the softmax temperature may depend on the number of outputs per class (e.g., the softmax temperature may increase as the number of outputs per class increases).

[0053] The number of neurons per layer in a neural network may be determined (e.g., selected from a grid) according to the constraints {100, 1000, 10,000, 100,000, 1,000,000, 10,000,000}. The number of layers in a neural network may be determined (e.g., selected from a grid) according to the constraints {4, 5, 6, 7, 8}. Alternatively, a grid of {2, 3, 4, 5, 6, 7, 8, 9} may be used. Neural networks with fewer neurons may perform better for certain machine learning tasks, such as classification tasks involving tabular data. In such cases, fewer neurons may help avoid overfitting. In contrast, a larger number of neurons (e.g., 64,000) may be better for image classification.

[0054] The softmax temperature may be determined (e.g., selected from a grid) according to the constraints {1, 3, 10, 30, 100}. Alternatively, a grid of {1, 1 / 0.3, 1 / 0.1, 1 / 0.03, 1 / 0.01} may be used. Increasing the softmax temperature as the number of neurons per layer increases may improve performance. For example, when the number of neurons per layer is between 2000 and 10,000, a softmax temperature of approximately 10 may lead to an accurate neural network (low validation error) compared to other softmax temperature values. When the number of neurons per layer is between 12,000 and 100,000, a softmax temperature of approximately 30 may lead to an accurate neural network. When the number of neurons per layer is greater than 100,000, a softmax temperature of approximately 100 may lead to an accurate neural network. When the number of neurons per layer is 100, a softmax temperature of less than 3 (e.g., 1 or 3) may lead to an accurate neural network.

[0055] Training the neural network according to the hyperparameter values ​​may include training at a learning rate for a number of epochs with a batch size.

[0056] The number of epochs may be the number of times during training that all training data is presented to the neural network (e.g., provided as input to the neural network). For example, when determining hyperparameters, the number of epochs may be increased until the validation error begins to increase (overfitting) even when the training error is decreasing. A suitable value for the number of epochs may be around 200.

[0057] Batch size may refer to the number of samples (i.e., instances, observations, input vectors, or feature vectors) from the input data that are propagated through a neural network per training iteration. For example, in the context of an image classification classification task, each sample of input data may include an image and a label for the image. Possible values ​​for batch size may be 32, 64, 128, 256, and other powers of 2. In some cases, a batch size of around 100 may be used.

[0058] The learning rate may specify how much to modify the neural network in response to the validation error for each epoch. When setting the learning rate, there may be a trade-off between the rate of convergence to a low validation error and the increase in training error during training. The learning rate can be set to {0.1, 0.01, 10 -3 ,10 -4 ,10 -5}. For example, a learning rate of about 0.01 may be used.

[0059] Training a neural network may involve sequentially parameterizing probability distributions. Thus, the probability distribution for each neuron may be parameterized during training (e.g., by independently selecting elements from a standard normal distribution using softmax). For example, an outcome may be calculated from each probability distribution in the neural network, e.g., during each epoch (i.e., each learning stage). The outcome may be the mean, weighted mean, or another statistical function of the probability distribution.

[0060] A neural network may include multiple layers, including an output layer. A neural network may include at least one hidden layer. At least one layer (e.g., a hidden layer) may precede the output layer. An input to a neural network may be accessed by a first layer, e.g., the first of multiple hidden layers.

[0061] In the case of multiple hidden layers, each hidden layer may have the same number of neurons.

[0062] A neural network may have the same number of neurons in the hidden and output layers. Additionally or alternatively, a neural network may have 4 to 8 layers.

[0063] When the machine learning task is a classification task, the classification task may have at least two classes. In some cases, the number of neurons per layer may be selected so that each class in the classification task is associated with multiple neurons in the output layer. Therefore, by using multiple neurons per class and aggregating them through summation, a hierarchical classification with as many levels as there are neurons per class can be performed. Each neuron in a class can capture different evidence (or aspects) of the class, allowing for finer-grained classification compared to traditional logic gate networks.

[0064] The method may further include, after determining the logical operators for each neuron, performing a machine learning task using the trained neural network. For a classification task, the output layer may include n neurons, and the classification task may have k classes. Thus, performing a classification task using the trained neural network may include: outputting a classification score for each class by the trained neural network; Grouping the output into k groups of size n / k, where the number of ones in each group corresponds to the classification score of the class corresponding to the group; and determining a classification, including determining a maximum classification score.

[0065] Determining the maximum classification score may involve using an argmax operation to determine the class with the maximum probability (i.e., the maximum predicted probability). Thus, the maximum classification score may be determined using an argmax operation.

[0066] In some cases, the output bits for each class (i.e., each classification score) may be summed to a binary number to reduce the memory bandwidth required to return the prediction (i.e., classification score). This may be represented via a fixed logic gate network. Specifically, adders may be implemented that can add single bits to a binary number with logic gates. Thus, an adder may be implemented for each classification score, with the number of adders corresponding to the number of classification scores. The adders may be modified to be suitable for summing the output bits of corresponding classes to form an integer. In this way, summing is very efficient; specifically, summing is faster than storing unsummed results in VRAM.

[0067] After determining the logical operators for each neuron and before outputting the classification score, the method may further include converting the trained neural network into a binary executable on a central processing unit (CPU) or a graphics processing unit (GPU). In some cases, the executable binary is callable via a shared object binary, e.g., a program capable of working with Python. A binary executable on a CPU may be compiled from C code. A binary executable on a GPU may be compiled from CUDA (formerly Compute Unified Device Architecture).

[0068] The inputs and outputs of a neuron may be real-valued.

[0069] In some cases, during training, the output of all neurons may range from 0 to 1 (i.e., greater than or equal to 0 and less than or equal to 1). Similarly, the inputs to neurons may range from 0 to 1. Additionally or alternatively, each neuron may receive between 0 and 3 inputs. More specifically, each neuron may receive between 0 and 2 inputs. For example, a "0" operator may receive 0 inputs, an "A" operator may receive only one input, and an "A-AB" operator may receive two inputs.

[0070] Building a neural network may further include pseudorandomly initializing connections between neurons (i.e., neural network weights). Pseudorandom initialization may have the advantage that connections do not need to be stored and can be generated as needed. In particular, connections may be generated from a seed (i.e., a random seed used to initialize a pseudorandom number generator) and reproduced from the same seed. The seed may be generated from computer state (e.g., time) or from a dedicated hardware-based random number generator. Building a neural network may also include using other heuristics to pseudorandomly initialize connections, for example, within groups, or to initialize structures that enable faster computation for particular hardware.

[0071] Therefore, once a neural network is trained, storing it only requires storing 4 bits of information specifying which logical operator is used for each neuron. In other words, storing a neural network may require n × 4 bits (where n is the number of neurons in the neural network) plus a relatively small constant value. The constant value may include (e.g., be limited to) the seed and the layer size (number of inputs and number of outputs). If the layer sizes are the same (i.e., all layers have the same size), only one size value is required. In this way, the memory requirements of a neural network may be significantly less than those of traditional neural networks (e.g., dense neural networks, convolutional neural networks, sparse neural networks, or even binary neural networks).

[0072] During training, the connections between neurons may remain fixed, i.e., the neural network may not be trained by adjusting the weights of neurons.

[0073] The classification task may include one or more of binary classification, pattern recognition (e.g., feature classification), image classification, object identification or recognition, character recognition, gesture or face recognition, voice detection or speech recognition, and text classification.

[0074] For example, input data for a binary classification task may include binary vectors of size 17. Each binary vector may be classified as class 0 or class 1. In another example, input data for a pattern recognition (specifically, image classification) task may include images of digits 0 through 9, along with corresponding labels for each image. Each image may be classified by identifying which digit the image corresponds to.

[0075] Binary classification may include one of medical testing (determining whether a patient has a disease), quality control (determining whether technical specifications are met), and information retrieval (determining whether a page should be in a search result set).

[0076] The pattern recognition may include image processing (e.g., image classification). The pattern recognition may be used for one of identification and / or authentication, medical diagnosis, defense, and mobility. The identification and / or authentication may include license plate recognition, fingerprint analysis, facial recognition, and voice-based authentication. The medical diagnosis may include screening for cervical cancer, breast tumors, or heart sounds. The defense may include navigation and guidance systems, target recognition systems, and shape recognition technology. The mobility may include driver assistance systems (assisting with driving and parking) and / or autonomous vehicle technology (in the case of ground vehicles that can navigate safely with little or no human input).

[0077] The machine learning task may be a regression task, and the regression task may include one or more of image generation, text generation, video generation, speech generation, 3D reconstruction, compression, and encoding. In particular, the regression task may include video, audio, or image compression. The encoding may include speech encoding.

[0078] Training the neural network may further include determining whether the neural network is sufficiently accurate by comparing the accuracy of the neural network to a specified accuracy. Training is performed in a differentiable manner, i.e., the probability distributions of the logical operators for the neurons of the neural network are learned during training. Thus, training includes determining the loss or error of the neuron probability distributions. In contrast to training, inference (i.e., using the trained neural network) does not necessarily need to be differentiable and can be performed using fixed logical operators and hard values ​​(0 or 1) rather than relaxed values ​​(probabilities between 0 and 1).

[0079] The accuracy of a neural network may be determined traditionally using a conventional loss function. For example, the softmax cross-entropy classification loss function may be used. In this context, cross-entropy is a divergence measure of the difference between two probability distributions. Alternatively, the mean squared error loss may be used for regression.

[0080] The specified accuracy may be determined according to the machine learning task, for example, the specified accuracy for binary classification may be about 100%, and the specified accuracy for image classification may be at least 95% or at least 97%.

[0081] Training the neural network may further include determining new values ​​for the hyperparameters when the neural network is not sufficiently accurate. The new values ​​for the hyperparameters may be determined iteratively, for example, using Bayesian optimization. The neural network may then be reconstructed according to the new values, and the reconstructed neural network may be trained according to the new values. After performing a specified number of iterations (or once a specified accuracy is reached), a most accurate neural network may be determined from the neural networks and the reconstructed neural networks. The most accurate neural network may be provided as the trained neural network.

[0082] There may be a trade-off between accuracy and efficiency. In particular, neural networks with more neurons may be less efficient at inference. Therefore, it may be desirable to reduce accuracy for efficiency reasons. In addition, knowledge from previous iterations may be used to determine new hyperparameter values ​​for subsequent iterations. For example, the constraints used to determine the hyperparameter values ​​may be modified.

[0083] In some examples, the plurality of logical operators may include at least two operators, at least eight operators, or exactly 16 operators.

[0084] In some examples, multiple logical operators may be real-valued (i.e., relaxed counterparts of logic gates).

[0085] The logical operators may conform to one of the following interpretations: probabilistic, Hamacher t-norm, relativistic Einstein sum t-conorm, Lukasiewicz t-norm and t-conorm.

[0086] Real-valued logical operators may be derived from fuzzy logic.

[0087] A t-norm may be a binary operation used in fuzzy logic to represent conjunctions. Similarly, a t-conorm may be used to represent disjunctions in fuzzy logic.

[0088] The use of real-valued logical operators and / or real-valued inputs may contribute to making a neural network differentiable. Traditional logic gate networks are not differentiable because their parameters are limited to 0 and 1. The use of real values ​​may allow the gradient to be calculated as the derivative (e.g., multidimensional derivative) of the loss function with respect to the network parameters, since the use of real values ​​eliminates discontinuities, e.g., jumps from 0 to 1. Thus, a neural network may be referred to as a differentiable logic gate network. Making a neural network differentiable may facilitate training using algorithms based on gradient descent.

[0089] Furthermore, neuron inputs and outputs, which are real-valued during training, may be Boolean (fixed) in the trained neural network. Similarly, logical operators in a trained neural network may be discretized by taking their modes, such that each probability distribution is replaced by the logical operator with the highest value in the probability distribution.

[0090] Real-valued logical operators may include: 0,A·B,A-AB,A, B-AB, B, A+B-2AB, A+B-AB, 1-(A+B-AB),1-(A+B-2AB),1-B,1-B+AB, 1-A,1-A+AB,1-AB,1.

[0091] In the above-mentioned real-valued logical operators, A and B may be inputs from corresponding neurons. In other words, the real-valued logical operator of a given neuron may perform an operation on input A from a first neuron and input B from a second neuron. For 0 and 1, these values ​​may be output regardless of the input. The above-mentioned real-valued logical operators may correspond to probabilistic interpretations (i.e., probabilistic product T-norm and probabilistic sum T-conorm). However, other interpretations (as described above) may be used.

[0092] In some cases, the input data may include test data having multiple samples. The method may further include, during inference (i.e., after training), performing the machine learning task by accessing the sum of the test data by the first layer of the neural network, where the accessing includes assigning elements of each sample of the test data to consecutive bits of an integer. Thus, during inference (i.e., after training), the two inputs of each neuron have numeric data types that correspond to hardware implementation data types of the processor of the computer on which the machine learning task is to be performed.

[0093] Therefore, the aggregate of test data may have an aggregate size, i.e., the aggregate size of the batch during inference. The aggregate size corresponds to the processor's hardware implementation data type (i.e., assembly language data type) and is therefore efficient. For example, an aggregate size of 64 can be used for an int64 hardware implementation data type, and an aggregate size of 32 can be used for an int32 hardware implementation data type.

[0094] Specifically, the test data samples may be images, and the elements of the samples may be pixels of the images. The numeric data type may be an integer data type, e.g., int64, rather than the Boolean data type traditionally used with binary neural networks. Thus, each neuron input may be a portion of an int64 integer, such that the whole integer consists of 64 different neuron inputs. The total size may be 64. Thus, each batch of images provided to a machine learning task may contain 64 images, each with 784 binary pixels (i.e., a 28x28 bitmap image), so the images can be batched (i.e., spread across) into an array of 784 integer variables, each with a hardware-implemented data type of int64. The processor of the computer on which the machine learning task is to be performed may be a CPU or a GPU.

[0095] Accessing the test data by the first layer neurons may correspond to spreading the test data across the first layer neurons. The accessing may include assigning (all) values ​​of image 1 to bit 1 of an integer and (all) values ​​of image 2 to bit 2 of the integer, such that each successive image is spread across all integers in the first layer. In other words, Boolean value i of image k may be assigned to bit k of int64 value i.

[0096] Therefore, all logical operations are performed on integers, since the operations are the same for each image. Therefore, you can usually apply a bitwise operator to all 64 bits of an int64 simultaneously with the same (or comparable) processing cost as applying a bitwise operator to a single value with a Boolean data type.

[0097] In this way, performance can be significantly improved compared to the performance of operations on values ​​having a Boolean data type. In particular, this performance improvement relies on the fact that many processors (e.g., many, if not most, CPUs) that perform bitwise operations on values ​​having a Boolean data type and values ​​having an integer data type that corresponds to the processor's hardware-implemented data type (e.g., the int64 data type) efficiently take the same amount of time, i.e., one instruction. Further efficiencies may be achieved by using data types larger than int64, for example, using Advanced Vector Extensions (e.g., AVX, AVX2, AVX-512) instruction sets.

[0098] As an alternative to a CPU or GPU, dedicated hardware, such as an ASIC or FPGA, may be used, in which case Boolean data types may be used efficiently.

[0099] According to another aspect, a computer program is provided. The computer program includes instructions that, when executed by a computer, cause the computer to perform a method as described above. The computer program may be part of (or included in) a computer program product. The computer program may be tangibly embodied on a computer-readable medium. The computer program may be implemented in hardware or software. In particular, the computer program may be implemented using an FPGA or an ASIC. Additionally or alternatively, the computer program may be implemented in a processor (CPU or GPU) component that includes hardware coding (e.g., an FPGA or ASIC) that implements a neural network to perform specific tasks.

[0100] According to yet another aspect, a computer system for training a neural network to perform a machine learning task is provided. The computer system includes at least one processor. The processor is configured to receive input data for the neural network, determine values ​​for multiple hyperparameters of the neural network, and construct the neural network according to the hyperparameter values. The neural network includes multiple neurons. Each neuron includes a probability distribution for each of the multiple logical operators, the neuron including a corresponding probability for each of the multiple logical operators. The processor is further configured to train the neural network according to the hyperparameter values ​​and the input data by learning the probability distribution for each neuron. The processor is further configured to determine a logical operator for each of the multiple logical operators by selecting a value in the probability distribution.

[0101] The processor may be a GPU that includes features that can be leveraged to accelerate the training of a neural network for performing a machine learning task (using the techniques of this disclosure) or to accelerate the performance of a machine learning task.

[0102] Additionally or alternatively, the computer system may be implemented using at least one FPGA and / or ASIC.

[0103] Additionally or alternatively, the computer system may include hardware acceleration (e.g., in an ASIC) for the logic gate network.

[0104] The subject matter described in this disclosure may, in some cases, be embodied as a method or on a device in the form of one or more computer programs (e.g., computer program products), which may cause a data processing apparatus to perform one or more operations described in this disclosure.

[0105] The subject matter described in this disclosure may be embodied in a data signal or on a machine-readable medium, which may be embodied in one or more information carriers, such as a tape, a CD-ROM, a DVD-ROM, a semiconductor memory, or a hard disk. In particular, the disclosed subject matter may be tangibly embodied in a machine (computer) readable medium.

[0106] Additionally, the subject matter described in this disclosure can be implemented as a system including at least one processor and a memory coupled to the processor. The processor may be a central processing unit (CPU) or a graphics processing unit (GPU). The memory may encode one or more programs to cause the processor to perform one or more of the methods described herein. The subject matter described in this disclosure can be implemented using various machines. The CPU and / or GPU may include integrated circuits with hardware acceleration, such as hardware acceleration for floating-point operations. Alternatively, the processor may be general-purpose hardware without hardware acceleration (e.g., floating-point acceleration).

[0107] Additionally, the subject matter of this disclosure may be implemented in at least one field programmable gate array (FPGA). The FPGA may be specially designed for artificial intelligence and / or specifically for implementing neural networks. The FPGA may be a configurable, hardware-accelerated processor capable of efficiently performing a predefined task (or set of tasks) when the predefined task is represented via logic gates. FPGAs may be particularly well-suited for tasks requiring limited complexity but high speed, such as mining cryptocurrencies (e.g., Bitcoin) or implementing oscilloscopes. Thus, for operations on neural networks, FPGAs specially designed for processing neural networks may be 10 to 100 times faster than conventional CPUs.

[0108] Additionally or alternatively, the subject matter of this disclosure may be implemented using an application-specific integrated circuit (ASIC) customized for a particular application. For example, an ASIC may be developed to support artificial intelligence. In particular, Google's Tensor Processing Unit or Fujitsu's Deep Learning Unit may be used.

[0109] Additionally or alternatively, a hardware implementation of the logic gate network may be used. In particular, an FPGA or ASIC may be used to realize the logic gate network, or a GPU may include features that can be exploited to accelerate the training of a neural network for performing a machine learning task (using the techniques of this disclosure) or to accelerate the performance of the machine learning task.

[0110] The details of one or more implementations are set forth in the exemplary drawings and the description below. Other features will be apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0111] [Figure 1] Figure 1 shows a simplified neural network containing an input, two hidden layers, and an output layer. [Figure 2] Figure 1 shows another simplified neural network. [Figure 3] FIG. 1 illustrates the steps of a method for training a neural network to perform a machine learning task. [Figure 4] Figure 1 shows the distribution of logical operators in a four-layer neural network after training. DETAILED DESCRIPTION OF THE INVENTION

[0112] The following document provides a detailed description of examples with reference to the drawings. Various modifications to the examples may be made. In particular, one or more elements of one example may be combined and used in another example to form a new example.

[0113] FIG. 1 illustrates an exemplary neural network for performing a machine learning task. More specifically, the neural network is for performing a classification task. The neural network may be constructed according to determined hyperparameter values. Thus, the hyperparameter values ​​may define the structure of the neural network. The hyperparameter values ​​may include multiple layers and multiple neurons per layer. Thus, the neural network may be constructed according to the hyperparameters such that the neural network includes an input 101, a first layer 103, a second layer 105, and an output layer 107. Furthermore, the output layer 107 includes four neurons. Additionally, layers 103 and 105 each include four neurons.

[0114] Thus, layers 103 and 105 of the neural network and output layer 107 may each include the same number of neurons. In other words, the layers of the neural network may each include the same number of neurons.

[0115] The classification task of a simplified neural network has two classes, class 0 and class 1. Therefore, the classification task involves binary classification. For example, neurons 3.1 and 3.2 may each output "1," and neurons 3.3 and 3.4 may each output "0." In this case, the number of classes k is 2, and the number of neurons n in the outer layer 107 is 4, so the outputs can be grouped into two groups of size 2. Therefore, the classification score for class 0 is 2, and the classification score for class 1 is 0. Therefore, by determining the classification according to the maximum classification score, class 0 is determined as the classification. In other examples, more classes may be used.

[0116] Each neuron 1.1-3.4 in the neural network receives two inputs from two different neurons. For example, neuron 1.4 in the first layer 103 receives inputs 0.5 and 0.6 from input 101.

[0117] Each neuron may contain a corresponding probability for each of multiple logical operators. The neuron's probability may be part of the neuron's probability distribution, which may be learned as the neural network is trained. Logical operators may be real-valued (i.e., relaxed). Real-valued logical operators may be based on T-norms (relaxed "and") and T-conorms (relaxed "or"). Real-valued logical operators may be differentiable and / or continuous. Therefore, neither extreme T-norms nor minimum T-norms may provide an adequate basis for real-valued logical operators.

[0118] Real-valued logical operators are sometimes seen as an extension of traditional Boolean logical operators, in that real-valued logical operators are defined not only for inputs of 0 and 1, but also for inputs between 0 and 1.

[0119] An exemplary probability distribution for neuron 1.1 of layer 103 may include the following probabilities as shown in Table 1:

[0120] [Table 1]

[0121] Table 2 shows the probabilistic interpretations of real-valued logical operators (probabilistic logical operators) and their corresponding Boolean interpretations.

[0122] [Table 2]

[0123] In Table 2, the ID column identifies each row. The Operator column indicates a Boolean operator. Rows in the Real Value column indicate the real-valued operator corresponding to the Boolean operator in that row. Rows in columns "00," "01," "10," and "11" indicate the output of the operator (Boolean or real-valued) corresponding to that row, given the column header value (e.g., "00") as input. Testing has shown that reducing the number of operators in Table 2 can lead to decreased performance. In other words, using 16 logical operators as opposed to fewer than 16 may improve the efficiency of methods for training neural networks to perform machine learning tasks.

[0124] The values ​​in Table 2 indicate probabilistic logical operators. Other interpretations for real-valued logical operators may be used. For example, the Hamacher product T-norm and its dual, the Einstein sum T-conorm, may be differentiable and provide a suitable basis for real-valued logical operators. Real-valued logical operators based on the Hamacher product T-norm and the Einstein sum T-conorm are shown in Table 3, below.

[0125] [Table 3]

[0126] Some operators in Table 3 can be derived from other operators in Table 3. In rows 2, 4, and 9 of Table 3, the respective Boolean operators do not have corresponding real-valued operators. For implications (rows 2 and 4 of Table 3), the R-implications (or residuals) corresponding to the T-norm may be used (see "Continuous R-implications," B. Jayaram et al., July 20-24, 2009). The probabilistic operators in Table 2 have been found to perform better in testing than the Hamacher T-norm and Einstein sum T-conorm in Table 3.

[0127] Other implementations of real-valued operators may be based on the Frank T-norm, the Yager T-norm, the Aczel-Alsina T-norm, the Dombi T-norm, and the Sugeno-Weber T-norm, in addition to the corresponding T-conorms. Further information on logical operators can be found in "Analyzing Differentiable Fuzzy Logic Operators," van Krieken et al., August 24, 2021.

[0128] Further T-norms and T-conorms that may be used to implement real-valued operators are shown in Tables 4 and 5 below.

[0129] [Table 4]

[0130] [Table 5]

[0131] Exemplary values ​​for the neuron of Figure 1 during training are provided in the following table. Logical operator probabilities during mid-training of neural networks JPEG0007803594000006.jpg72170Logic operator probabilities in the later stages of neural network training, i.e., after convergence JPEG0007803594000007.jpg78170

[0132] The neuron in Figure 1 corresponding to each row of logical operator probabilities shown above is provided at the beginning of the row. For example, a row beginning with "1.1" indicates the value for neuron 1.1 in layer 103. Each column in a of logical operator probabilities corresponds to a sequential ID in Table 2. For example, row 1 of logical operator probabilities corresponds to ID 0 in Table 2, and row 2 of logical operator probabilities corresponds to ID 1. Thus, at mid-training, neuron 3.2's highest probability is 0.678, which corresponds to ID 10 in Table 2, i.e., real-valued operator 1-B.

[0133] Mid-stage training may refer to the number of epochs a network is trained for. Thus, mid-stage training may refer to approximately half the number of epochs. Similarly, late-stage training may refer to epochs within 10% of the final epoch. For example, if a neural network is trained for 200 epochs, mid-stage training may be epochs 90-110. Similarly, late-stage training may be epochs 190-200. Convergence may refer to the stage of training where additional training no longer improves the neural network.

[0134] As can be seen from the example values ​​above, during mid-training, each neuron has a number of logical operator probabilities between 0 and 1. However, after convergence, typically only one or two of the probabilities will have a non-zero value, and at least one value will be close to 1.

[0135] Figure 2 shows a neural network for performing a regression task. In this case, instead of "class 0" or "class 1," the output of the neural network is a value in the output layer 107. More specifically, the output may be an array of Boolean values ​​(i.e., a vector of values ​​where each value may be either 0 or 1). Alternatively, there may be a scalar output.

[0136] More specifically, there may be n neurons in the output layer 107 to generate (i.e., predict) k-dimensional outputs. The outputs may be grouped into k groups (e.g., of size n / k). For each dimension i of the output (prediction), α i A scalar parameter α such that n / k is valid and provides a determined (desired) range for the output (prediction). i There may be.

[0137] For example, if the regression task is to predict rainfall, there may be one dimension, or if the regression task is to predict rainfall and wind speed, there may be two dimensions. The determined range of rainfall may be 0 to 200 mm. In another example, if the regression task is image generation, k may be the number of pixels, so to generate an image with 784 pixels, k may be 784.

[0138] The final output may be determined by counting the 1 values ​​of the output neurons in the output layer 107.

[0139] A bias β is used to shift the determined range (i.e., output space) to generate both positive and negative outputs. i Therefore, the final output may be determined by counting the output neurons and applying an affine transformation as shown in the following equation:

number

[0140] An affine transformation may be used to convert (or shift) the determined range from 0 to n / k to a more suitable application-specific range.

[0141] In some cases, α = α for all i i and β = β i All dimensions of the output have the same range so that

[0142] In some cases, it may be desirable for the determined range to cover all real numbers, which can be achieved using a logit transform:

number

number

[0143] Using the outputs, the mean squared error (MSE) loss can be formulated as follows:

number

[0144] In another respect, the neural network in Figure 2 corresponds to the neural network in Figure 1.

[0145] Figure 3 illustrates a method for training a neural network to perform a machine learning task. The neural network may correspond to the neural networks of Figures 1 and 2 described above.

[0146] After training, the neural network may be referred to as a deep differentiable logic gate network, which performs all calculations as binary operations on Boolean values, rather than the floating-point calculations typically performed in conventional neural networks. The operators for each neuron in the neural network may be realized exclusively using logic gates. This can lead to very sparse networks and increased efficiency when performing machine learning tasks with the trained neural network. Efficiency may be further increased by spreading data for the machine learning task (e.g., test data) across the neurons in the first layer 103.

[0147] In step S201, input data for the neural network is received. The input data may be received as binary-valued inputs 101, such that each neuron in layers 103-105 receives two inputs. The input data may include training data, validation data, and test data. The training data may be used to learn probability distributions for each neuron. The validation data may be used to determine whether the neural network is sufficiently accurate, i.e., whether training is complete. The test data may be used during inference (after training is complete), for example, to evaluate the neural network against other or conventional neural networks.

[0148] In step S203, values ​​for multiple hyperparameters of the neural network are determined. The hyperparameters may include the number of layers (e.g., from about 2 to about 32), the number of neurons in each layer (e.g., from about 12 to about 1,024,000), and a learning rate.

[0149] For example, each layer may have the same number of neurons. Additionally or alternatively, there may be about 4 to about 8 layers. This may lead to the advantage that the architecture / structure of the neural network does not need to be fine-tuned. This may therefore simplify and speed up the determination of hyperparameters.

[0150] The learning rate may have a constant value of approximately 0.01.

[0151] In step S205, a neural network may be constructed according to the hyperparameter values. This may include generating the number of layers (e.g., 4) and the number of neurons per layer (e.g., 8000) according to the hyperparameter values. The constructing may include pseudo-randomly initializing the connections between neurons, i.e., the weights of the network. Other means of initializing the connections (i.e., the weights) are also possible. For example, a trained neural network with weights determined according to a conventional approach may be used as a basis.

[0152] Each neuron may contain probability distributions for multiple logical operators, each of which may have the following signature: f:{0,1}x{0,1}→{0,1}

[0153] Instead of hard binary values ​​p∈{0,1}, the probabilities in a probability distribution are p∈[0,1] 16 This may be the step that makes the neural network differentiable.

[0154] In a neural network, a single neuron may be defined as follows: The two inputs to the neuron may be defined as a, b∈[0,1]. Thus, p is a probability simplex Δ 15 is a probability distribution over logical operators. p is

number

number

[0155] The output o may be ∈[0,1].

[0156] Since neurons in a neural network can be distinguished from one another even with binary inputs, i.e., the network values ​​are no longer constrained to ∈{0.1}, training can be performed (e.g., using a gradient descent-based algorithm) by representing the selection of which logical operators (i.e., logic gates) are present in each neuron by a probability distribution (e.g., a categorical probability distribution).

[0157] Advantageously, the memory requirements for storing a neural network may be significantly less than those for neural networks that perform floating-point operations instead of using logical operators. For example, the logical operators may consist of 16 logical operators. Thus, after the neural network is trained, only four bits are required to represent the logical operator for a given neuron. In other words, only four bits are required to store information specifying the operation a given neuron performs. This may be significantly less memory than that required for neural networks whose neurons perform more complex operations.

[0158] In traditional sparse neural networks, the connections between the neurons (weights) of the sparse neural network may be trained.

[0159] In step S207, the neural network may be trained according to the hyperparameter values. In contrast to traditional sparse neural networks, training a neural network may involve learning which logical operators (i.e., binary functions) to realize at each neuron while keeping the connections between neurons fixed after initialization. Thus, the learning objective may be to determine which of the logical operators should be present at each neuron. The network may therefore be parameterized (continuously) by learning probability distributions for the logical operators at each neuron.

[0160] Neural networks may be relaxed. In other words, instead of fixed logical operators, probability distributions for logical operators may be used, and the neural network may work on probabilities during training. Relaxing the logical operators may be another step in making the neural network differentiable. Learning the probability distributions for each neuron may be accomplished via a (relaxed) softmax parameterization.

[0161] Learning the probability distribution may be accomplished by parameterizing each neuron with 16 floating-point values ​​corresponding to 16 logical operators. Softmax may be used to map the 16 floating-point values ​​to a probability simplex (i.e., a categorical probability distribution where all entries sum to 1 and only non-negative values ​​exist). Referring to the discussion of Equation 4 above, parameterizing neurons during initialization of the neural network may include selecting (depicting) the elements of q independently from a standard normal distribution.

[0162] Training may involve evaluating all 16 logical operators for each neuron and computing their weighted average using categorical probability distributions. Thus, during training, the output of every neuron may be ∈[0,1].

[0163] More specifically, for a classification task with k classes and n neurons in the output layer, the outputs may be grouped into k groups of size n / k. In this way, the classification score for each of the classes may be determined by counting the number of ones in each group. Thus, in the context of a classification task, the output of the neural network may be determined by taking the argmax of the classification scores.

[0164] To determine whether a neural network is sufficiently accurate, rather than counting the number of ones, the probability of the output in each group may be added. Thus, a measure of accuracy may be determined by calculating the classification loss. For example, the softmax cross-entropy classification loss may be calculated as follows:

number

[0165] After training is complete, the two inputs to the neuron may be defined as a, b∈{0,1}. Thus, in contrast to a neuron during training (see Equation 4), where the inputs to the neuron may be floating-point values ​​≧0 and ≦1, the inputs to the neuron after training may be Boolean values, i.e., either 0 or 1. Similarly, the output o of the neuron may be ∈{0,1}. Thus, the output o of the neuron after training may be defined as o=0p γ (a,b) (Equation 6) In the formula, γ=argmax in formula (6) i p i where i refers to the ID in Table 2.

[0166] In step S209, a logical operator may be determined for each neuron. The logical operator may be determined after training is complete, i.e., during inference. The determined logical operator may be the maximum likelihood logical operator, e.g., the logical operator of the probability distribution with the highest probability. In other words, the probability distribution may be discretized by taking its mode. Thus, machine learning tasks may be performed by computing Boolean values ​​rather than floating-point values, which makes classification more efficient compared to neural networks that rely on floating-point arithmetic.

[0167] Before using the trained neural network to perform a machine learning task, the neural network may be compiled into at least one binary executable. The binary executable may be processor-dependent. For example, two binary executables may be compiled: one for the CPU (e.g., from C code) and one for the GPU (e.g., from CUDA).

[0168] Additionally, logical and / or sub-representations may be simplified, e.g., instead of Boolean data types, hardware-implemented data types corresponding to the processor on which the machine learning task is to be performed may be used.

[0169] For example, for a 64-bit CPU, the hardware implementation data type may be int64. Additionally, an aggregate size of 64 may be used, meaning that 64 samples (e.g., images) are processed through the neural network in a given iteration (i.e., epoch). In GPUs, because any write to GPU memory can result in a bottleneck (i.e., reduced performance), output neurons may be directly aggregated using the logic gates that make up each adder. In general, bottlenecks (points in the system that can cause a reduction in speed) may be the data loader and / or transmission speed.

[0170] Thus, bitwise operations may be performed on large batches, which can have a significant impact on the speed at which machine learning tasks can be performed.

[0171] The output layer 107 may generate multiple outputs for each class. The outputs may be aggregated via bit counting, i.e., by counting the ones, resulting in a score for each class. Thus, if the machine learning task is a classification task, the classification task may be completed by providing the class with the highest score as the output of the neural network.

[0172] When performing a machine learning task using binary vectors given as input, pairs of Boolean values ​​may be selected from the binary vectors, logical operators (i.e., binary logic gates) in one of the layers (e.g., layer 103) may be applied to the Boolean values, and then those outputs may be used in subsequent layers of the neural network (e.g., layers 105 or 107).

[0173] After training, the computational cost of performing machine learning tasks may be reduced by at least an order of magnitude compared to traditional binary and sparse neural networks, and in some cases even compared to other types of traditional neural networks.

[0174] An exemplary machine learning task is the Monk problem, as discussed in "The monk's problems: A performance comparison of different learning algorithms," Thrun et al., 1991. The Monk problems, MONK-1, MONK-2, and MONK-3, are three machine learning tasks that have been used to benchmark machine learning algorithms. These tasks consist of three binary classification tasks on data sets with six attributes, each with two to four possible values. Therefore, data points (samples) can be encoded as binary vectors of size 17.

[0175] Tests show that the above-described neural network performs more accurate classification than logistic regression for all three MONK data sets. Additionally, for MONK-3, the above-described neural network (i.e., Figures 1 and 2) is more accurate than a much larger convolutional neural network. Additionally, the above-described neural network is more than three times faster than logistic regression and more than seven times faster than a larger convolutional neural network. Additionally, the above-described neural network requires significantly less storage space than either logistic regression or a convolutional neural network.

[0176] As another example of a machine learning task, consider the adult census (UCI machine learning repository: Adult data set, Kohavi et al., 1996) and breast cancer (UCI machine learning repository breast cancer dataset, Zwitter et al., 1988) data sets. For the adult data set, the machine learning task is to predict whether a given adult earns more than $50,000 per year based on attributes such as education and weekly work hours. For breast cancer, the machine learning task involves binary classification, determining whether the cancer is diagnosed as benign or malignant based on properties of the cell nucleus, including perimeter, area, and smoothness. For these tasks, the classification accuracy achieved by the neural network described above is comparable to that of conventional (convolutional) neural networks and logistic regression. Additionally, the classification speed is more than 10 times faster than logistic regression and more than 40 times faster than conventional neural networks. In addition, the neural network described above requires approximately 20% less storage space than logistic regression and approximately 75% less than traditional neural networks.

[0177] Furthermore, very high frame rates may be achieved for image classification. For example, in the case of the neural network described above, frame rates of over 1 million images per second may be achieved for the Modified National Institute of Standards and Technology (MNIST) dataset (http: / / yann.lecun.com / exdb / mnist / ) and the Canadian Institute for Advanced Research (CIFAR-10) dataset ("Learning Multiple Layers of Features from Tiny Images," Alex Krizhevsky, April 8, 2009). In other words, classification rates of over 1 million images per second may be achieved using a single CPU core. This may exceed the efficiency of any conventional approach.

[0178] More specifically, the above-described neural network may have image classification accuracy on the MNIST dataset comparable to the fastest conventional binary neural network, while requiring less than 10% of the number of binary operations. On a standard GPU (e.g., NVIDIA A6000), the above-described neural network requires only 7% of the GPU utilization, yet the above-described neural network may implement 12 times faster than a conventional binary neural network on dedicated FPGA hardware. Compared to another conventional binary neural network, the above-described neural network may be three orders of magnitude faster. Compared to evolutionarily trained sparse function networks, the above-described neural network is more accurate.

[0179] For image classification using the CIFAR-10 dataset, the accuracy of the neural networks described above can rival that of traditional convolutional neural networks, while requiring less than 0.1% of the memory in some cases and less than 1% of the memory in others. Dedicated fully connected networks are slightly more accurate (less than 4%) but require 64% more memory.

[0180] When performing image classification using the CIFAR-10 dataset, a traditional fully connected neural network that relies on floating-point operations may require 2,000,000 floating-point operations to perform the machine learning task, while the neural network described above, before pruning or optimization, requires 5,000,000 bitwise logical operations. In floating-point hardware-accelerated integrated circuits (e.g., current GPUs and many CPUs), 2,000,000 floating-point operations is approximately 100 times slower than 5,000,000 bitwise logical operations. Without floating-point acceleration, the speed difference would be an order of magnitude larger, i.e., three orders of magnitude.

[0181] Even conventional sparse neural networks, while faster than conventional fully connected neural networks, are still at least an order of magnitude slower than the neural networks mentioned above. A single sparse neural network requires at least twice the storage space of the neural networks mentioned above.

[0182] A possible exemplary architecture for the neural network described above (related to Figures 1 and 2) is shown in Table 6 below.

[0183] [Table 6]

[0184] Table 7 shows the configuration of the fully connected ReLU network that was used as a basis for comparison with the neural network configurations in Table 6.

[0185] [Table 7]

[0186] Additionally, in each clock cycle (e.g., per Hertz) of a CPU in a typical general-purpose desktop or laptop computer, it may be possible to compute an average of approximately 250 binary logic gates for each core of the CPU. This is possible because a typical CPU executes many instructions per clock cycle, even with a single core. This may be significantly faster than would be possible if relying on floating-point arithmetic to execute neural networks. Additionally, the CPU may be able to spread data for a machine learning task across the neurons of the first layer 103 by grouping bits from multiple samples of data (e.g., images) into a single integer (e.g., an integer having data type int64). Grouping bits from multiple samples into a single integer is sometimes referred to as Single Instruction Multiple Data (SIMD). Additional efficiency gains may be possible using Advanced Vector Extensions (AVX).

[0187] The computational cost of some machine learning tasks performed by a layer with n neurons (e.g., layers 103, 105, or 107) may be 0(n), including fine fixed costs, because the layer requires only Boolean logic gates (i.e., operations are performed exclusively through logical operators and can therefore be implemented very efficiently). In comparison, a fully connected layer with m input neurons has a computational cost of O(n m), with significantly higher fixed costs, especially because fully connected layers generally require floating-point operations. Overall, performing machine learning tasks using neural networks trained according to the methods described with reference to Figures 1 and 2 can lead to inference speeds that are two orders of magnitude faster than fully connected ReLU neural networks (neural networks that use a ReLU activation function). In addition, neural networks trained according to the methods of Figures 1 and 2 may be more than 13 times faster than conventional binary neural networks and two to three orders of magnitude faster than the theoretical speed of conventional sparse neural networks.

[0188] Figure 4 shows the distribution of logic gates for a neural network trained to perform a machine learning task. In this case, the machine learning task involves image classification. The neural network has four layers with 12,000 neurons per layer. For layers 1 and 2, the values ​​on the x-axis correspond to the IDs in Table 2, as well as the values ​​on the x-axis for layers 3 and 4. The values ​​on the y-axis provide a count of the number of neurons in which the logical operator on the x-axis resides.

[0189] From Figure 4, we can see that the logical operators for constants "0" and "1" (corresponding to IDs 0 and 15 in Table 2) are used very infrequently and do not occur at all in layer 4. In the first layer, "and", "nand", "or", and "nor" have relatively strong probabilities.

[0190] In the second and third layers, one of the two inputs is ignored and the other is passed forward (possibly in a modified form) to the next layer, resulting in more "A", "B", "┐A", and "┐B" that can be seen as residual direct connections. This may allow the neural network to model lower-order dependencies more efficiently by representing them in fewer than a predefined number of layers.

[0191] In layer 4, the most frequent operations are "xor" and "xnor," which can create conditional dependencies on activations in previous layers. As shown, implication is used less frequently. However, tests show that using only a proper subset of logical operators (rather than all 16 operators in the probabilistic interpretation in Table 2) can lead to reduced accuracy.

Claims

1. 1. A computer-implemented method for training a neural network to perform a machine learning task, such as image classification, comprising: receiving input data for the neural network (S201); Determining values ​​for a plurality of hyperparameters of the neural network (S203); constructing (S205) the neural network comprising a plurality of neurons (1.1-3.4) according to the hyperparameter values, constructing said neural network, wherein each neuron (1.1-3.4) contains a probability distribution for each of a plurality of logical operators, said neuron (1.3-3.4) containing a corresponding probability for each of said logical operators; training the neural network according to the hyperparameter values ​​and the input data by learning the probability distribution of each neuron (1.3-3.4) (S207); determining (S209) a logical operator of the plurality of logical operators for each neuron (1.3-3.4) by selecting a value in the probability distribution; 11. A computer-implemented method comprising:

2. the input data includes an input and a corresponding desired output; the input data includes one or more of training data, validation data, and test data; The method of claim 1.

3. the hyperparameter values ​​are determined to minimize a validation error of the neural network under one or more specified computational resource constraints; The specified computational resource constraints may include processor and / or memory constraints; The method according to claim 1 or 2.

4. the hyperparameters include one or more of neurons per layer, number of layers, number of epochs, batch size, and learning rate; 4. The method according to any one of claims 1 to 3.

5. 5. The method of claim 1, wherein the neural network comprises a plurality of layers, including an output layer (107).

6. the output layer (107) includes n neurons, the machine learning task is a classification task with k classes, The method further includes performing the classification task using the trained neural network, wherein performing the classification task comprises: outputting a classification score for each of the classes by the trained neural network; grouping the output into k groups of size n / k, where the number of ones in each group corresponds to the classification score of the class corresponding to that group; determining a classification, including determining a maximum of the classification scores; The method of claim 5 , comprising:

7. After training the neural network and before performing the machine learning task, the method further comprises: converting the trained neural network into a binary executable on a central processing unit or a graphics processing unit; The executable binary is callable via a program that can handle shared object binaries, and the program may be written in Python.

7. The method according to any one of claims 1 to 6.

8. The output of each neuron is a value between 0 and 1, and / or Each neuron receives 0 to 3 inputs, 8. The method according to any one of claims 1 to 7.

9. constructing the neural network further comprises pseudo-randomly initializing connections between the neurons; During the training, connections between neurons may remain fixed.

9. The method according to any one of claims 1 to 8.

10. The machine learning task is: Binary classification, pattern recognition, image classification, object identification or recognition, character recognition, gesture or face recognition, voice detection or speech recognition, text classification The method of claim 1 , wherein the classification task comprises one or more of:

11. the machine learning task is a regression task, Image generation, text generation, video generation, speech generation, 3D reconstruction, compression, encoding 11. The method of claim 1, comprising one or more of:

12. the plurality of logical operators includes at least two operators or exactly 16 operators; The plurality of logical operators may be real-valued operators; The real-valued operator may conform to one of the following interpretations: probabilistic, Hamacher t-norm, relative Einstein sum t-conorm, Lukasiewicz t-norm and t-conorm; 12. The method according to any one of claims 1 to 11.

13. the input data includes test data having a plurality of samples; performing the machine learning task by accessing the sum of the test data by a first layer (103) of the neural network during inference, the accessing including assigning elements of each sample of the test data to consecutive bits of an integer; During inference, the two inputs of each neuron have numeric data types that correspond to hardware implementation data types of a processor of a computer on which the machine learning task is to be performed.

13. The method according to any one of claims 1 to 12.

14. A computer program comprising instructions that, when said program is run by a computer, cause the computer to carry out the method of any one of claims 1 to 13.

15. 1. A computer system for training a neural network to perform a machine learning task, such as image classification, comprising: At least one processor, the processor comprising: receiving input data for the neural network; determining values ​​for a plurality of hyperparameters of the neural network; constructing the neural network comprising a plurality of neurons (1.3-3.4) according to the hyperparameter values; constructing said neural network, each neuron (1.3-3.4) including a probability distribution for a plurality of logical operators, said neuron (1.3-3.4) including a corresponding probability for each of said logical operators; training the neural network according to the hyperparameter values ​​and the input data by learning the probability distribution for each neuron (1.3-3.4); By selecting a value in the probability distribution, a logical operator of the plurality of logical operators for each neuron (1.3-3.4) is determined. A computer system configured to:

Images