Training a neural network to perform machine learning tasks

Abstract

This provides methods, programs, and systems for training neural networks. [Solution] The method receives input data for a neural network, determines values ​​for several hyperparameters of the neural network, and constructs a neural network comprising several neurons according to the hyperparameter values. Each neuron contains a probability distribution for a logical operator, such that the neuron contains the corresponding probability for each of several logical operators. The method further trains the neural network according to the hyperparameter values ​​and input data by learning the probability distribution of each neuron, and determines the logical operator of the several logical operators for each neuron by selecting values ​​in the probability distribution.

Description

[Technical Field]

[0001] The technical field is artificial intelligence, particularly neural networks. The aspect is machine learning tasks. This relates to training neural networks to perform machine learning tasks. This may be a classification task or a regression task. Classification tasks involve patterns and / or Among sequence recognition, novelty detection (i.e., outlier or anomaly detection), and sequential decision-making, It may include one or more of the following. The regression task may include compression or restoration. In addition The method involves performing machine learning tasks using a trained neural network. It may also be related to this. [Background technology]

[0002] A neural network is an artificial neuron (i.e., a neuron, node, computation unit). It may include a set or unit, and may be used to solve artificial intelligence problems. The connections between neurons are modeled as weights in a neural network. It is fine. A neural network is a neural network that can be trained. The components or elements of the function may include parameters. For example, the weights may be several. These may be parameters in the neural network. However, other parameters Lameter is also possible.

[0003] Activation functions are used to control the magnitude of the output of a neural network. This is also good. Therefore, the activation function limits the value of the output of the neural network, and It may be used to add nonlinearity to a neural network.

[0004] Neural networks may be trained using optimization algorithms. The neural network's performance continues until the neural network is sufficiently accurate. This may include adjusting the meter. The optimization algorithm is based on gradient descent. Or it may depend on. An example of an optimization algorithm based on gradient descent is probability Adaptive Gradient Descent (SGD), and Adam (Adaptive Learning Rate Optimization Algorithm - Root Mean Squared) Examples include propagation and SGD being realized as a combination of momentum. Further information can be found at "Adam: A method for stochastic "Optimization," discovered by Kingma et al. in 2014.

[0005] Gradient descent is a method of error backpropagation (i.e., backprop or reverse propagation). It may also be based on (the method). Gradient descent measures the accuracy of the parameters, and changes in the parameters are negative. Determine the direction in which the network can be improved, that is, the direction in which the parameters can be improved. It may include a cost function. When the output of the cost function is zero, when it is at its minimum value, Sufficient accuracy can be obtained when the value is within the specified range of zero. The specified range is for the machine learning task. It may depend on gradient descent (or an algorithm based on gradient descent). ) is only used to train neural networks with differentiable parameters. Because it cannot be used, binary or Boolean parameters cannot be trained via gradient descent. It's not possible. Boolean parameters (represented as 0 or 1 in neural networks) This can lead to discontinuities (for example, a jump from 0 to 1), in which case, The layout is unclear, and descending the slope is impossible.

[0006] The accuracy of a neural network may be calculated by various methods. For example, in relation to a classification task, the training error (or loss) may be obtained by measuring how accurately the neural network can classify the data used to train the neural network. The classification loss (also called the generalization error or expected loss) may be obtained by measuring how accurately the neural network can perform the classification task on data that was not previously visible. In some cases, the neural network may include multiple layers, i.e., at least one hidden layer and an output layer. In such a case, the neural network may be called a deep learning neural network and may have an infinite number of hidden layers. Thus, multiple layers may be used to incrementally extract higher-level features from the input. The neural network may be implemented as a feed-forward network in which data flows from the previous hidden layer to the output layer without the data looping back. Initialization of the neural network may include assigning random weights to the connections between neurons. Previously, when a neural network did not perform a machine learning task accurately, the weights were adjusted. In this way, the influence of different neurons within the network can be calibrated until the appropriate mechanical operation of the input for performing the machine learning task is determined.

[0007]

[0008]

[0009] ​​​​​​​​​​​​​​​Once trained to perform a machine learning task, a neural network may be used to perform the machine learning task. Performing a machine learning task may also be referred to as inference, or performing inference.

[0010] Neural networks face various problems. For example, overfitting may occur when the training error is low but the classification loss is relatively large. In other words, the neural net work learns an approach specific to the training data used to train the neural network that does not generalize well to other data besides the training data. If the neural network cannot accurately capture the relationship between its inputs and outputs, unlearning may occur, resulting in a high error rate for both the training data and other data.

[0011] Neural networks can be effective but inefficient in performing classification tasks. Therefore, various approaches have been developed to increase the efficiency of neural networks.

[0012] For example, a conventional neural network may be a deep learning neural network that includes at least one dense layer such that part or all of the neural network is fully connected, i.e., each neuron in the dense layer receives an input from all neurons in the previous layer. In this example, the neural network may be referred to as a high-density connected neural network or a fully connected neural network.

[0013] In addition, a conventional neural network has weights and activations of full precision (e.g., double You may also use floating-point calculations with high precision (i.e., 64 bits), and generally, expensive matrix multiplication can be avoided. In short, low-precision floating-point weights and activations (e.g., single-precision or half-precision) However, calculations can still be expensive.

[0014] A binary neural network is one in which 1 bit activation (and / or weight) is followed Replaces the floating-point activation (and / or weights) of conventional neural networks. In this respect, it differs from conventional neural networks. Therefore, Elements propagated through twerking have only one of two values, for example, 0 or 1. It is binary in the sense that it can do that. Therefore, conventional neural networks Compared to the more expensive matrix multiplication, XNOR and bit count operations are faster. It can be simplified to this, which is binary compared to conventional neural networks. Significantly improves neural network inference performance. In some cases, XNOR is bit It can be faster than to-count operations, and both can be faster than floating-point operations.

[0015] The floating-point arithmetic required to train neural networks is a complex task for computers. It may be supported by the software. Specifically, float32 adders and / or A float32 multiplier requires at least 1000 logic gates (or lookup gates). This may require a cable and may have delays of several dozen logical levels. 32 adders and / or float32 multipliers are operated by the central processing unit (CPU) or graphical user. It may be implemented directly on the graphics processing unit (GPU). In any case, float32 processing The arithmetic and float32 multipliers are for float32 and int32 data types. In contrast, even with an optimized GPU, bitwise processing for int64 data types This can sometimes require far more computational resources (e.g., processing cycles) than logical operations. If the GPU is optimized for int32, int64 is supported efficiently. This may not always be the case. Therefore, use the int32 data type for such GPUs. It can be more efficient to do so. In some hardware implementations, logical operations are performed as addition. It may be faster than calling / multiplication.

[0016] The CPU performs approximately 3 to 10 integer 64-bit operations per clock cycle. While it is possible to do so, floating-point operations typically require an entire clock cycle (i.e.) (Only one floating-point operation can be performed per clock cycle.) This refers to the XNOR and Bit counting operations are floating-point operations (e.g., float32 adder or float32 adder). If the AT32 multiplier has dedicated hardware support for the GPU or CPU Even if there are, compared to the floating-point operations performed on conventional neural networks... This can result in significant performance advantages. Considerations regarding floating-point and integer arithmetic. This can also be applied to training neural networks for machine learning tasks. ru.

[0017] A convolutional neural network (CNN) has at least one structured hidden layer. A deep learning neural network that has a limited connection, i.e., is not fully connected. It is also possible. Therefore, a CNN may perform convolution, for example, the hidden layer is hidden The dot product of the convolution kernels is performed using the input matrices of the layers. Only the neurons in the hidden layer covered by the CNN are the neurons in each of the other layers of the CNN. It may connect to.

[0018] In conventional examples, binary neural networks are very small (3x3) convolutions. The depth is increased via a filter, and it includes three fully connected layers, reducing training time. A very deep convolutional neural network that uses ReLU (Reverse Lumen Change) to correct nonlinearity. It may be based on a tech (for example, VGG) architecture. Or, binary nuclear A residual neural network skips some layers using skip connections, leaving residual neural networks. It may be based on the ResNet architecture. VGG and R Both esNet architectures result in very large neural networks. This can sometimes lead to inefficiency.

[0019] A neural network can be sparse (in this example, a sparse neural network). (Sparse neural network). Therefore, a sparse neural network is fully connected. It has an appropriate subset of neural network connectivity. In other words, a fully connected layer. Instead, not all of the neurons in the layers of a sparse neural network Neurons are connected to each other. Sparseness is the number of neurons that each neuron can connect to other neurons. It may also refer to the connectivity of a neural network that accepts input only from [source], or / or, not all neurons in a sparse neural network In a neural network, all such that are active at any given time. It can also refer to the state of a neural network, which describes the level of activity of each neuron. good.

[0020] Sparse neural networks represent 90% of all connected neural networks. It may have full connectivity, i.e., a sparse neural network has at least 10 It may have a sparsity of % (10% to 70% sparsity is low sparsity, 7) 90% sparsity is medium sparsity, 90% to 99.9% sparsity Sparsity is moderate sparsity, and sparsity is greater than 99.9%. (This is sometimes called high sparsity). Neurals with sparsity of less than 10%... Networks are sometimes described as dense.

[0021] Sparse neural networks can be pruned or compressed using techniques such as regularization. The technology can be used to derive results from high-density or fully connected neural networks. For example, one or more analytical methods or heuristics should be retained. Used to determine the weights and the weights that should be removed from the neural network. This may be done (removing weights may involve setting the weights to 0). More specifically The general cost function is obtained by adding the regularization term using L1 or L2 regularization. You may update it. As yet another example, dropout is a neural network Dropout is a regularization technique in which random neurons are canceled or removed. It may also be variational, and the boundary (for example, regarding how many neurons are canceled) It may be limited by the limit. In some cases, instead of removing the weight, new Ron himself may be removed from the neural network. Pruning techniques are as described above. As shown, this can help avoid overfitting and thereby reduce generalization errors. The technique involves removing all layers of a neural network, or neural network This may involve adding an additional cost for workpieces with relatively large weights. Therefore, sparse neural networks are compared to high-density neural networks. This may provide improved accuracy (e.g., less overfitting). Rough networks also require less memory space compared to high-density neural networks. The intervals can be small. However, removing weights from a high-density neural network If a sparse neural network is derived by doing so, then sparse neural A neural network stores the weight positions in addition to the actual weights, so It may require memory.

[0022] In the following two examples of conventional implementations, the connectivity of the neural network was relaxed. It can happen.

[0023] In the first example, the neurons of the neural network are realized as logical modules. Even if that were the case, half of the logical modules would be defined by fuzzy (i.e., relaxed) conjunctions. The remaining half is defined by fuzzy (i.e., relaxed) disjunction. In this example, neural The parameters of a Le network include weights, but the operators (i.e., fuzzy conjunctions and selections) The words are fixed. Therefore, the neuronal weights included in fuzzy conjunctions and disjunctions This may be relaxed from only 0 or 1 to a value within the range of 0 to 1. In this way, Unlike type parameters, the parameters of a logic module are trained via gradient descent. It is possible.

[0024] In the second example, a set of arithmetic operations is predefined for each layer, and the neurons in that layer perform these operations. In contrast, the softmax model is used for learning, and the output of the previous layer is used for the current layer. It is used as input to the corresponding arithmetic operation. In this way, a symbolic representation is derived from the data. It can be taught.

[0025] In another conventional example, the fully connected layer of an artificial neural network was trained on a sparse layer before it was trained. This can be replaced without significantly reducing accuracy (for example, by more than 5% or 10%). The number of parameters is reduced quadratically (without reducing accuracy by more than %). The - topology has two continuous layers and transforms into a scale-free topology during training. Expand. Scale-free topology has a power law frequency distribution P(d)~d -y Approximate This is a sparse graph, and the proportion P(d) from all neurons in the network is the other It has d connections to a neuron, and the parameter γ is usually within the range γ∈(2,3) stay.

[0026] In yet another conventional example, a single shared weight parameter is used for all of the neural networks. It is assigned to each connection. Instead of optimizing the weights, the neural network, It is optimized for network topologies that perform well across a wide range of weights. Neurons and connections are gradually added, and activation functions (e.g., sine, cosine, Gau) are used. (The values ​​of the hyperbolic tangent, sigmoid, inverse, absolute value, and ReLU) are randomly assigned to new neurons. It is determined. The activation function unfolds as the neural network grows. A neural network uses floating-point operators.

[0027] According to another conventional example, a network of binary (2-input) lookup tables A method is provided for storing binary classification datasets using this method. • The network in the table is based on the conditional frequency (e.g., pattern p) in the training data. It is constructed by counting how many times a specific output of the training data (0) is associated with it. Similar to neural networks, look-up tables are placed in continuous layers. Unlike neural networks, training is done through memory, backpropagation, or gradient Descent is not involved. Memory remembers the output that is most commonly associated with the input of the training data. It involves doing so.

[0028] In further conventional examples, neural networks are first trained on machine learning tasks. Then, the neural network is translated into a random forest, and the random forest Forest is then translated into a network of AND-Inverter logic gates, i.e. The network is based on "AND" and "NOT" logic gates. [Overview of the Initiative] [Problems that the invention aims to solve]

[0029] This is more efficient than the conventional examples mentioned above, while maintaining comparable accuracy, in mechanical engineering. It is sometimes desirable to provide a neural network to deliver learning tasks. .

[0030] For example, a logic gate network may be used. A logic gate network is Each neuron is a logical operator (i.e., a logical gate such as "AND" or "NAND"). It may also be a neural network, which includes each logical operator having an input of 0 to 2. It may be. A logic gate network is better than a fully connected neural network. It is sparse, and all neurons in the previous layer provide input to each neuron in the current layer. It is provided in this way. In other words, it takes n inputs (where n is the number of neurons in each layer). However, because each neuron in a logic gate network receives only two inputs, logic Gate networks are sparse. Therefore, they compute logical operations very quickly. This is possible (for example, if logical operations are built into the processor architecture). (Because there are many) machine learning tasks may be performed efficiently. In addition, the input of each neuron Since there are at most two forces, logic gate networks are very sparse, for example, at least two. This refers to the sparsity between the segments, which leads to improved accuracy compared to denser networks. There is.

[0031] Optionally, a logic gate network has at least one operation with three or more inputs. Children may be included.

[0032] Advantageously, logic gate networks operate in binary (e.g., after training) and are nonlinear. Therefore, logic gate networks may not require activation functions.

[0033] However, conventional logic gate networks use SGD or Adam (mentioned above) Since it is not possible to train using any gradient descent-based optimization algorithm, There are problems with implementing a neural network as a logic gate network. This is because the values ​​of conventional logic gate networks are always either 0 or 1. Therefore, the loss function with respect to the network layer parameters does not have a derivative. Therefore, it is impossible to calculate the gradient in conventional logic gate networks. This is a logic game that can be trained using an optimization algorithm based on gradient descent. Providing a network is sometimes desirable.

[0034] Furthermore, conventional logic gate networks have a single binary output (e.g., recursive task). (In the case of k) or k binary outputs for k classes (for example, in the case of a classification task) ) may be provided. This is because graded prediction, i.e., class ranking relative to each other is impossible. Yes, it exists. In other words, this is the "maximum activation" classification scheme ("How a Neural Network Works, Arkin Gupta, May 27, 2018 A corresponding classification is impossible. [Means for solving the problem]

[0035] According to one embodiment, a neural network is trained to perform a machine learning task. A computer implementation method for this is provided. The method is for neural networks. This includes receiving input data. The method further involves multiple neural networks. This includes determining the values ​​for the hyperparameters. The method further involves the hyperparameters This includes building a neural network according to the values, The node has multiple neurons. Each neuron has multiple logical operators. This includes a probability distribution for logical operators that includes the corresponding probability. Furthermore, by learning the probability distribution of each neuron, the hyperparameter values ​​and inputs can be determined. This involves training a neural network according to force data. The method further involves probability By selecting values ​​in the distribution, the logic of multiple logical operators for each neuron This includes determining the operator.

[0036] A neural network is a binary neural network (neural network). Once the network is trained, only one of two values ​​will propagate within the network. (Because they may have) logic gate networks (neurons realize logical operators) (For this purpose), or sometimes called a binary logic gate network. Children are sometimes called logic gates.

[0037] In some cases, each neuron has up to two inputs. If there are two inputs and n neurons per layer, the neural network is It can have sparsity of up to 1-(2 / n).

[0038] The values ​​of multiple hyperparameters are the received hyperparameters (for example, similar machines) Used for performing machine learning tasks, or similar neural networks (used to perform the same machine learning task) is determined from, or from the initial It may be defined without receiving any values.

[0039] In some cases, the probability distribution of each neuron is such that all entries in the probability distribution sum to 1. This is a categorical probability distribution derived from multiple floating-point values, such that it is non-negative. It is acceptable to do so. In a probability distribution, the floating-point value is located at the probability unit, and therefore, the categorical probability Even if the rate distribution is derived from floating-point values ​​via softmax, Good. Since the probability distribution is trained, the probability distribution of the neural network is neural. They are sometimes also called network parameters.

[0040] The probabilities of the probability distribution may be ≥0 and ≤1.

[0041] Softmax may be a conventional function used to convert values ​​into probabilities. Softmax may include temperature (i.e., a temperature parameter), where temperature is a low probability. The sensitivity may be increased. In other words, temperature is a way of controlling the entropy of the probability distribution. It may also be in stages (higher temperatures increase entropy and make the probability distribution more uniform, but lower temperatures (This reduces entropy and highlights high probabilities.)

[0042] Choosing a value for a probability distribution can be done by selecting the maximum likelihood value or a random value. This may be achieved by: Selecting a value, finding a suitable value This may involve repeatedly selecting random values ​​until a certain value is reached. In some cases, the selection may be certain. This may include replacing the rate distribution with the mode of the probability distribution. Therefore, trained nu In a neuronal network, the input and output of each neuron are of Boolean type (0 or 1). It may be, and / or the operator of each neuron is fixed (i.e., not a probability distribution) (See also: single operator)

[0043] Therefore, by using a probability distribution, a curve can be used to minimize the loss function. Since it becomes possible to calculate the distribution, the logic gate network can calculate the probability of each neuron. It can be trained by learning the distribution. Therefore, the conventional logic gates mentioned above Compared to a network, it is possible to calculate the derivative of the loss function and the gradient vector. This can be obtained. The derivative of the loss function is in the direction in which the neural network improves. Therefore, by being able to calculate the derivative of the loss function, the neural network can be trained. This will make it possible to improve the network.

[0044] The input data may include the input and the corresponding desired output. The input data is used for training. May include training data, validation data, and test data. The test data may be mutually exclusive or separate. In other words, training None of the samples in the data appear in the form of validation data or test data. • None of the samples in the data appear in the form of validation or training data. When evaluating hyperparameters using validation data, and test data When testing the accuracy of a neural network using data, determining overfitting is important. This is possible. In some cases, it is desirable to avoid overfitting. Other examples (good In what is sometimes called sexual overfitting, an accuracy of approximately 100% is achieved on the training data. Even in that case, the neural network still has limitations regarding validation and test data. It can be relatively accurate as such. A sample is sometimes also called a data point.

[0045] In some cases, the training data is processed according to the determined hyperparameter values. • May be used to train the network. Validation data is determined hyper Compare one set of parameter values ​​with another set of determined hyperparameter values. It may be used to determine the final hyperparameters (e.g., test data). Once the most accurate value is determined, the neural network performs the machine learning task. It may be used to determine the precision of the workpiece.

[0046] The desired output is sometimes called a label. The input data is provided as a set of tuples. Each tuple may have the form of (input or sample, corresponding output). The vector may be provided as a fixed-size vector (e.g., a binary vector). Alternatively, it may have a known topological structure (e.g., an image).

[0047] In some cases, neural networks are constrained by one or more specified computing resource limitations. The hyperparameter values ​​may be determined in such a way as to minimize the verification error, and the specified Computational resource constraints may include processor and / or memory constraints.

[0048] There are at least 5 hyperparameters or at least 10 hyperparameters You may do so. Hyperparameter values ​​are related to the structure of the neural network, and the neural You may decide how the RAL network is trained. The hyperparameters are , the computation time and memory cost of inference performed via neural networks It can have an effect. In addition, hyperparameters are the finesse of the neural network. It can have an effect on each layer. For example, having too many neurons in each layer can lead to overfitting. This can happen, and if there are too few layers or too few neurons in each layer, it can lead to underlearning. It can happen.

[0049] In some cases, multiple sets of hyperparameter values ​​may be determined, The neural network may be trained according to each set of parameter values. If accurate, the most accurate one in the trained neural network is used for inference. It's okay.

[0050] Hyperparameter values ​​(values ​​for multiple hyperparameters) are randomly optimized. Through algorithms (e.g., grid search or automated machine learning (AutoML)) The selection may be made by or via random search. The optimization algorithm is user This may be achieved using constraints provided by the machine learning task. That's also good. In relation to grid search, constraints are sometimes called grids. Grid The search may be performed using a conventional grid search algorithm.

[0051] In some cases, the input data may include training data and validation data. Then, for each set in the multiple sets of hyperparameter values, the neural network Twerks may be trained using training data and a set of hyperparameter values. Next, the neural network uses a set of hyperparameter values ​​and validation data. The accuracy may be verified using the following hyperparameter values, which have the smallest error with the verification data. The set of values ​​(i.e., the lowest verification error and the most accurate hyperparameter values) is determined. These may be hyperparameter values.

[0052] Hyperparameters include the number of neurons per layer, the number of layers, the number of epochs, and the batch size. This may include one or more of the learning rates. In relation to the classification task, the hyperparameter The meter may also include a softmax temperature for training the probability distribution. Max temperature is used to increase as the number of neurons per layer increases. Good. Therefore, the softmax temperature may depend on the number of outputs per class (for example) (The softmax temperature may increase as the number of outputs per class increases.) .

[0053] The number of neurons per layer in a neural network is {100, 1000} , 10,000, 100,000, 1,000,000, 10,000,000 It may be determined according to the constraint} (for example, by selecting from a grid). The number of layers in a loop network is determined according to the constraint {4, 5, 6, 7, 8}. It may be selected (for example, from a grid). Alternatively, {2,3,4,5,6,7 A grid of ,8,9 may be used. Neural systems with fewer neurons... The network is designed for specific machine learning tasks, such as classification tasks involving tabular data. It can be done better. In such cases, fewer neurons help avoid overfitting. This can sometimes happen. Conversely, more neurons (e.g., 64,000) can be used in image processing. This may be better in some cases due to the classification system.

[0054] The softmax temperature is determined according to the constraint {1,3,10,30,100}. (For example, it can be selected from a grid.) Or, {1,1 / 0.3,1 / 0. A grid of {1,1 / 0.03,1 / 0.01} may be used. Each layer contains neurons. Performance may be improved by increasing the softmax temperature as the number of elements increases. For example, if the number of neurons per layer is 2,000 to 10,000, then approximately 10 softmaps The softmax temperature is more accurate compared to other softmax temperature values ​​in a neural network (low This can lead to verification errors. The number of neurons per layer is 12,000 to 100,0 In the case of 00, a softmax temperature of approximately 30 leads to an accurate neural network. Sometimes, if the number of neurons per layer exceeds 100,000, approximately 100 software Maximum temperature can lead to an accurate neural network. If the number of Rons is 100, then a softmax temperature of less than 3 (e.g., 1 or 3) is accurate. It can sometimes connect to a neural network.

[0055] Training a neural network according to hyperparameter values ​​is a batch process. This may include training with a learning rate relative to the number of epochs, where size is the factor.

[0056] The number of epochs is determined by the number of training data points that are displayed to the neural network during training. It could also be the number of times it is performed (for example, the number of times it is provided to a neural network as input). For example, when determining hyperparameters, if the training error is decreasing. You may increase the number of epochs until the validation error starts to increase (overfitting). A suitable value for the number may be approximately 200.

[0057] The batch size is the input that propagates through the neural network with each training iteration. The number of samples (i.e., instances, observations, input vectors, or features) from force data. It may also refer to a vector. For example, in relation to an image classification task, each of the input data The sample may include an image and a label for that image. Possible batch size values ​​are: It may be 32, 64, 128, 256, and other powers of 2. In some cases, approximately A batch size of 100 may be used.

[0058] The learning rate is how much the neural network adjusts in response to the validation error for each epoch. You can also specify whether to change the degree. When setting the learning rate, you can specify the convergence rate to a low validation error and the training rate. There may be a trade-off between the learning rate and the increase in training error. The learning rate is {0.1, 0. 01,10 -3 ,10 -4 ,10 -5 It is determined according to the constraint} (for example, Grid (It may be selected from D.) For example, a learning rate of approximately 0.01 may be used.

[0059] Training a neural network involves continuously parameterizing the probability distribution. It may also include the following. Therefore, the probability distribution of each neuron is (for example, softmax Even if elements are independently selected from a standard normal distribution and parameterized during training, That's fine. For example, the result could be calculated from each probability distribution in a neural network. For example, the result may be calculated between each epoch (i.e., each learning stage). The result is This could be the mean, a weighted mean, or another statistical function of the probability distribution.

[0060] A neural network may include multiple layers, including an output layer. The network may include at least one hidden layer. The first layer may precede the output layer. The input to the neural network is the first layer, for example. For example, it may be accessed by the first of several hidden layers.

[0061] In the case of multiple hidden layers, each hidden layer may have the same number of neurons.

[0062] A neural network may have the same number of neurons in its hidden layer and output layer. In addition, or by alternative means, the neural network may have 4 to 8 layers. .

[0063] When a machine learning task is a classification task, the classification task may have at least two classes. Good. In some cases, the number of neurons per layer is such that each class in a classification task is multiplied in the output layer. They may be selected so as to be associated with a number of neurons. Therefore, per class By using multiple neurons and summing them up, we can categorize them by class. A stepwise classification can be performed, with the same number of grades (levels) as the number of neurons. Each of Rass's neurons acquires evidence (or aspects) of a different class. This enables finer grading compared to conventional logic gate networks. can.

[0064] The method further involves determining the logical operators for each neuron, and then training the neural network. This may include performing machine learning tasks using a network. In total, the output layer may contain n neurons, and the classification task may have k classes. Therefore, using a trained neural network to perform classification tasks is possible. That is, A trained neural network generates a classification score for each class. To exert force and The output is grouped into k groups of size n / k, and each group The number of 1s corresponds to the classification score of the class corresponding to the group, and the grouping process , This may include determining the classification, which may include determining the maximum classification score.

[0065] Determining the maximum classification score involves using calculations to find the maximum probability (i.e., the maximum predicted probability). This may include determining the class having the rate. Therefore, the maximum classification score is This may be determined using the argmax operation.

[0066] In some cases, the output bits for each class (i.e., each classification score) are totaled and then... By converting it to a number, the memory bandwidth required to return the prediction (i.e., the classification score) can be reduced. This may be represented via a fixed logic gate network. Specifically, 1 An adder is implemented that can add bits to produce a binary number with a logic gate. It is also possible that the number of adders corresponds to the number of classification scores, with each adder being for each classification. It can also be implemented for scores. An adder adds the output bits of the corresponding class to equalize them. It may be modified to be suitable for numerical calculation. In this way, the total is very efficient. Specifically, summing is faster than storing unsummed results in VRAM.

[0067] After determining the logical operator for each neuron, and before outputting the classification score, the method Furthermore, the trained neural network can be controlled by a central processing unit (CPU) or graph This may include converting it into a binary executable on a graphics processing unit (GPU). Depending on the context, the executable binary may be a shared object binary, such as Python. It can be called through a program that can handle it. A binary that can run on the CPU. The code can be compiled from C code. The binary that can run on the GPU is CUDA. It may also be compiled from (the previous Unified Computation Device Architecture).

[0068] The inputs and outputs of a neuron may be real numbers.

[0069] In some cases, during training, the output of all neurons is in the range of 0 to 1 (i.e., between 0 and 1). The values ​​below may also be acceptable. Similarly, the input to the neuron may be between 0 and 1. Good. In addition, or alternatively, each neuron may receive an input of 0 to 3. More specifically, each neuron may accept inputs from 0 to 2. For example, the "0" operation. The child may accept an input of 0, and the "A" operator may accept only one input. The "A-AB" operator may accept two inputs.

[0070] Building a neural network also involves the connections between neurons (i.e., nu This may include initializing the weights of the network in a pseudo-random manner. The system initialization does not need to remember the connection and can be performed as needed. It may have advantages. In particular, the connection initializes the seed (i.e., the pseudorandom number generator). It is generated from a random seed used for this purpose, and may be reproduced from the same seed. The code is derived from the computer's state (e.g., time) or from hardware-based specifics. It can also be generated from a random number generator. Building a neural network is also an option. Using heuristics, for example, to initiate pseudo-random connections within a group. To optimize or to create a structure that enables faster computation for specific hardware. This may include making it a planned item.

[0071] Therefore, once a neural network is trained, the neural network To remember, you need to specify which logical operator is used for each neuron. It is only about storing information in bits. In other words, it is about storing information in a neural network. To do this, we need n × 4 bits (where n is the number of neurons in the neural network), It may be necessary to add a relatively small constant value. The constant value is the seed and layer value. It may include (or be limited to, for example, the number of inputs and the number of outputs). If the layers are the same size (i.e., all layers are the same size), then one size Only the value of 's' is required. In this way, the memory requirements of a neural network are conventionally... Neural networks (for example, high-density neural networks, convolutional neural networks) Neural networks, sparse neural networks, or even binary It can be significantly less than a neural network.

[0072] During training, the connections between neurons may remain fixed. In other words, neural • The network does not need to be trained by adjusting the weights of neurons.

[0073] Classification tasks include binary classification, pattern recognition (e.g., feature classification), image classification, and object recognition. Distinguishing or recognizing, character recognition, gesture or facial recognition, voice detection or speech recognition, text It may include one or more of the classifications.

[0074] For example, the input data for a binary classification task is a binary vector of size 17. It may be included. Binary vectors may each be classified as either class 0 or class 1. In another example, the input data for a pattern recognition (specifically, image classification) task is: The image may include 0-9 digits, and a corresponding label for each image. Each image may be classified by identifying which digit it corresponds to.

[0075] Binary classification is used in medical testing (to determine whether a patient has a disease) and quality control. (determining whether the technical specifications are met), and information retrieval (the page is a set of search results) It may include one of the following (which determines whether or not it should be inside the box).

[0076] Pattern recognition may include image processing (e.g., image classification). Pattern recognition is a form of identification. and / or may be used for one of the following: authentication, medical diagnosis, defense, and mobility. Identification and / or authentication include license plate recognition, fingerprint analysis, facial recognition, and voice recognition. This may include certification of a medical diagnosis. Medical diagnoses may include screening for cervical cancer, chest tumors, or heart sounds. This may include navigation and guidance systems, target recognition systems, and shape recognition technologies. That's fine. Mobility is provided by driver assistance systems (which assist with driving and parking) and / or This refers to autonomous vehicle technology (safe movement with little or no human input). This may include ground vehicles that can perform the following actions.

[0077] A machine learning task may also be a regression task, and a regression task can involve generating images, text, etc. Includes one or more of the following: generation, image generation, speech generation, 3D reconstruction, compression, and encoding. That is also acceptable. In particular, regression tasks may include compression of video, audio, or images. The coding may include speech coding.

[0078] Training a neural network further improves the accuracy of the neural network. By comparing it to a specified accuracy, we can determine whether the neural network is sufficiently accurate. This may include making decisions. Training is carried out in a differentiable form, i.e., neural network The probability distribution of logical operators for twerk neurons is learned during training. Training involves determining the loss or error in the neuronal probability distribution. Furthermore, inference (i.e., the use of a trained neural network) is not necessarily differentiable. It is also fine to use fixed logical operators, and hard values ​​(0 or 1) rather than relaxed values ​​(probability between 0 and 1). This can be done using 1).

[0079] The accuracy of a neural network can be conventionally determined using a conventional loss function. That's good. For example, a softmax cross-entropy classification loss function may be used. In relation to this, cross-entropy is a measure of the divergence of the difference between two probability distributions. Alternatively, Mean squared error loss may be used for regression.

[0080] The specified accuracy may be determined according to the machine learning task. For example, for binary classification... The accuracy of the designation may be approximately 100%, and the accuracy of the designation for image classification must be at least 9%. It may be 5% or at least 97%.

[0081] Training a neural network is also necessary if the neural network is sufficiently If the result is not accurate, this may include determining new values ​​for the hyperparameters. The new values ​​for the hyperparameters are determined iteratively, for example, using Bayesian optimization. It may be set. Therefore, the neural network is rebuilt according to the new value. The reconstructed neural network may be trained according to the new values. After performing a specified number of iterations (or reaching a specified precision), the neural network From the reconstructed neural network, the most accurate neural network The value may be determined. The most accurate neural network is a trained neural network. It may be offered as a network.

[0082] There can be a trade-off between accuracy and efficiency. In particular, when processing more neurons... Neural networks that include this may not be efficient in inference. Therefore, In terms of mental efficiency, it is sometimes desirable to reduce the precision of the specification. In addition, from the previous iteration Using this knowledge, new hyperparameter values ​​for subsequent iterations may be determined. For example, the constraints for determining the hyperparameter values ​​may be modified.

[0083] In some examples, multiple logical operators are at least two operators and at least eight operators It may contain operators, or exactly 16 operators.

[0084] In some examples, multiple logical operators are real numbers (i.e., relaxed counterparts of logic gates) ) is also acceptable.

[0085] Logical operators are probabilistic, Hamacher t-norm, and relative. Einstein sum t-conorm, Lucasiewicz t-norm and t-cono You may adhere to one of the interpretations, such as Rum.

[0086] Real-valued logical operators may be derived from fuzzy logic.

[0087] The t-norm may be a binary operation used in fuzzy logic to represent conjunctions. Similarly, the t-conorm may be used to represent disjunction in fuzzy logic.

[0088] The use of real-valued logical operators and / or real-valued inputs allows for finer control over neural networks. It may contribute to making it divisible. Conventional logic gate networks have their parameters Since the meter is limited to 0 and 1, it is not differentiable. The use of real values ​​is limited to the use of real values. The use eliminates discontinuities, such as jumps from 0 to 1, so the gradient is networked. The goal is to enable calculation of the derivative of the loss function (for example, a multidimensional derivative) for the lamellar function. It is also possible. Therefore, a neural network is a differentiable logic gate network. It is sometimes called a "ku". By making neural networks differentiable, gradient descent Training using algorithms based on this may be made easier.

[0089] Furthermore, the inputs and outputs of neurons are real values ​​during training, but after training, In a Lewy network, it may be Boolean (fixed). Similarly, trained neural networks... The logical operators of the network are set so that each probability distribution is the logical operator that has the highest value of the probability distribution. They may be discretized by taking on those modes so that they can be interchanged.

[0090] Real-valued logical operators may include the following: 0, A·B, A-AB, A, B-AB, B, A+B-2AB, A+B-AB, 1-(A+B-AB),1-(A+B-2AB),1-B,1-B+AB, 1-A, 1-A+AB, 1-AB, 1.

[0091] In the real-valued logical operators described above, even if A and B are inputs from the corresponding neurons... Good. In other words, the real-valued logical operators of a given neuron are the inputs from the first neuron. Calculations may be performed on force A and input B from the second neuron. 0 and 1 In this case, these values ​​may be output regardless of the input. The real-valued logical operators described above are certain This may also correspond to a probabilistic interpretation (i.e., a stochastic product T-norm and a stochastic sum T-conorm). However, other interpretations (as described above) may also be used.

[0092] In some cases, the input data may include test data with multiple samples. The method further involves the first layer of the neural network during inference (i.e., after training). This involves accessing the total test data to perform machine learning tasks. It may include, and accessing it may be a sequence of integers for each sample of the test data. This involves assigning bits. Therefore, during inference (i.e., after training), each neuron The two inputs are the hardware of the computer's processor on which the machine learning task should be performed. It has numerical data types that correspond to the data types implemented in the form.

[0093] Therefore, the total size of the test data is the total size, i.e., the total size of the batch during inference. It may have. The total size is the processor's hardware implementation data type (i.e., A It corresponds to the Sembri language data type and is therefore efficient. For example, total size 64 is the hardware implementation data type for int64, and the total size 32 is for int32. It can be used for hardware implementation data types.

[0094] Specifically, the test data samples may be images, and the elements of the samples are images. It may also be a pixel. The numerical data type is binary neural network. Instead of the Boolean data type traditionally used with the 'k', an integer data type, for example, It may also be an int64. Therefore, each neuron input is a different integer with a total of 64 values. It may be part of an int64 integer, as it consists of neuron inputs. The total size is 6. It may also be 4. Therefore, each batch of images provided to the machine learning task has 64 images. The image may include 784 binary pixels (i.e., 28x28 bits). Since it has a top image, the image is of the int64 hardware implementation data type. The data is batched into an array of 784 integer variables (i.e., spread throughout). It is possible. The processor of the computer on which the machine learning task is to be performed is the CPU or It can also be a GPU.

[0095] Accessing test data by neurons in the first layer is possible. It may also be possible to distribute test data across the entire course. Access This means that each sequential image is spread across all integers of the first layer, so that image 1 All values ​​of image 1 are assigned to integer bit 1, and all values ​​of image 2 are assigned to integer bit 2. It may include guessing. In other words, the boolean value i of image k is the bit of the int64 value i. It may be assigned to tok.

[0096] Therefore, since the operation is the same for each image, all logical operations are performed on integers. Therefore, typically, bitwise operators are used with Boolean data types. Applying bitwise operators to a value of one in the same (or comparable) processing cost as applying them to a single value It can be applied to 64 bits of t64 simultaneously.

[0097] In this way, compared to the performance of operations on values ​​with Boolean data types, Performance can be significantly improved. In particular, this performance improvement is beneficial for Boolean data types. A value having the following characteristics, and the processor's hardware implementation data type (e.g., int64). Bitwise operations on values ​​that have an integer data type corresponding to the data type. Many processors performing the task (for example, many CPUs, if not many) at the same time It relies on the fact that it can apply a large amount efficiently, meaning it requires only one instruction. By using larger data types, for example, advanced vector graphics. Using instruction sets such as AVX, AVX2, AVX-512, Further efficiency may be achieved.

[0098] Dedicated hardware, such as ASICs or FPGAs, can be used as an alternative to CPUs or GPUs. It may be used. In this case, Boolean data types may be used efficiently.

[0099] In another embodiment, a computer program is provided. When a program is executed by a computer, it uses the methods described above. Includes instructions to be executed by the user. Computer programs are computer programs made by computer program creators. It may be part of (or included in) the product. Computer programs are computer programs. Computer programs may be materialized (tangibly) in a computer-readable medium. It may be implemented in hardware or software. In particular, computer programs F This may be implemented using PGA or ASIC. The computer program implements a neural network to perform specific tasks. Processors (CPUs) including hardware encoding (e.g., FPGA or ASIC) Alternatively, this may be implemented using a GPU component.

[0100] In yet another aspect, a neural network is used to perform a machine learning task. A computer system is provided for training. The computer system is small. It has at least one processor. The processor is the input to the neural network. It receives force data and values ​​for multiple hyperparameters of the neural network. Determine the hyperparameter values ​​and construct a neural network according to them. It is accomplished. A neural network consists of multiple neurons. Each neuron is a A logical operator such that each of the logical operators has a corresponding probability. This includes the probability distribution for each neuron. The processor further learns the probability distribution for each neuron. This allows the neural network to be generated according to hyperparameter values ​​and input data. It is configured to be trained. The processor further selects values ​​in the probability distribution. Therefore, it is configured to determine the logical operators of multiple logical operators for each neuron. ru.

[0101] The processor is a neural network for performing machine learning tasks (using the technology of this disclosure). To accelerate the training of a new network or to accelerate the performance of machine learning tasks It may also be a GPU that includes features that can be used.

[0102] In addition or by another method, the computer system has at least one FPGA This may be implemented using a call and / or ASIC.

[0103] In addition or by another method, computer systems utilize logic gate networks. This may include hardware acceleration (for example, in ASICs). stomach.

[0104] The subject matter described herein may include one or more computer programs. In the form of (for example, a computer program product), as a method, or on a device Such computer programs may be one of those described in this disclosure or Multiple calculations may be performed by the data processing device.

[0105] The subject matter described herein is to be realized in data signals or on machine-readable media. This can be done using tapes, CD-ROMs, DVD-ROMs, semiconductor memory, or hard drives. • It is materialized on one or more information carriers, such as disks. In particular, the subject matter to be disclosed is: It may be materialized in a tangible form using a machine (computer)-readable medium.

[0106] In addition, the subject matter described herein includes at least one processor and coupled to the processor. It can be implemented as a system, including the memory. The processor is a central processing unit. It may be a CPU or a graphics processing unit (GPU). Memory is as described in this application. One or more methods described above are used to cause the processor to perform one or more of the methods described above. The program may be encoded. Further subject matter described herein can be put into practice using various machines. It can be expressed. The CPU and / or GPU provide hardware acceleration. n, for example, an integrated circuit having hardware acceleration for floating-point arithmetic. It may include, or the processor may use hardware acceleration (for example) General-purpose hardware without floating-point acceleration is also acceptable.

[0107] Furthermore, the subject matter of this disclosure is at least one field-programmable gate array It may be implemented using (FPGA). FPGAs are used for artificial intelligence and / or in particular FPGAs may be specifically designed to implement neural networks. If a defined task is represented via a logic gate, then the predefined task (or task's A configurable hardware acceleration processor that can efficiently perform (a task). It may also be used. FPGAs are particularly suitable for tasks that require high speed with limited complexity, such as the mining of cryptocurrencies (e.g., Bitcoin) or the realization of an oscilloscope scope. Therefore, in the case of operations on neural networks, an FPGA specially designed to process neural networks may be 10 to 100 times faster than a conventional CPU

[0108] In addition or alternatively, the subject matter of the present disclosure may be implemented using application - specific integrated circuits (ASICs) customized for a particular use. For example, an ASIC may be developed to support artificial intelligence. Specifically, Google's Tensor Processing Unit or Fujitsu's Deep Learning Unit may be used

[0109]

[0110] In addition or alternatively, a hardware implementation of a logic - gate network may be used. Specifically, an FPGA or ASIC may be used to implement a logic - gate network, or a GPU may include features that can be utilized to accelerate the training of neural networks for machine - learning tasks (using the techniques of the present disclosure) or to accelerate the performance of machine - learning tasks

[0111] [Figure 1] [Figure 2] [Figure 3] The details of one or more implementations are described in the illustrative drawings and the following description. Other features will be apparent from the description, the drawings, and the claims [Figure 4]

Brief Description of the Drawings

[0112] This figure shows a simplified neural network, including an input layer, two hidden layers, and an output layer. [Figure 2] This is a diagram of another simplified neural network. [Figure 3] This diagram shows the steps of a method for training a neural network to perform a machine learning task. [Figure 4] This figure shows the distribution of logical operators in a 4-layer neural network after training. [Modes for carrying out the invention]

[0112] The following document provides a detailed description of the example with reference to the drawings. Various modifications to the example are available. This may be done. In particular, if one or more elements of one example are combined and used in other examples. This may lead to the formation of new examples.

[0113] Figure 1 shows an exemplary neural network for performing a machine learning task. Yes. More specifically, neural networks are used to perform classification tasks. Yes, a neural network is constructed according to the determined hyperparameter values. This is also acceptable. Therefore, the hyperparameter values ​​define the structure of the neural network. It is also possible. The hyperparameter values ​​include multiple layers and multiple neurons in each layer. That is also fine. Therefore, the neural network is a neural network that takes input 1 01, including a first layer 103, a second layer 105, and an output layer 107, hyper - It may be constructed according to the parameters. Furthermore, the output layer 107 includes four neurons. Hmm. In addition, layers 103 and 105 each contain four neurons.

[0114] Therefore, layers 103 and 105 of the neural network and the output layer 107 Each of these may contain the same number of neurons. In other words, a neural network Each layer of the matrix may contain the same number of neurons.

[0115] The simplified neural network classification task involves two classes: Class 0 and Class 1. It has two classes. Therefore, the classification task involves binary classification. For example, neuro Neurons 3.1 and 3.2 may each output "1", and neurons 3.3 and 3.2 may each output "1". Each of the 4s may output "0". In this case, the number of class k is 2, and the outer layer 10 Since there are 4 neurons (n) in the 7th generation, we divide the output into two groups of size 2. Therefore, the classification score for class 0 is 2, and for class 1... The classification score will be 0. Therefore, the classification will be determined according to the maximum classification score. Therefore, class 0 will be determined as the classification. In other examples, more classes are used. That's fine.

[0116] Each neuron in the neural network, 1.1-3.4, consists of two different neurons. It receives two inputs. For example, neuron 1.4 in the first layer 103 receives input 101. Receives forces of 0.5 and 0.6.

[0117] Each neuron may contain a corresponding probability for each of the multiple logical operators. The probability of a neuron may be learned as the neural network is trained. , may be part of the probability distribution of the neuron. Logical operators are real numbers (i.e., slow (They may be added together.) Real-valued logical operators are T-norm (relaxation of "and") and T-co It may be based on a norm (relaxation of "or"). The real-valued logical operator may be differentiable and / or continuous. Thus, either an extreme T-norm or a minimum T-norm may not provide a suitable basis for real-valued logical operators.

[0118] A real-valued logical operator may be regarded as an extension of a conventional Boolean logical operator in that the real-valued logical operator is defined not only for inputs of 0 and 1 but also for inputs between 0 and 1.

[0119] An exemplary probability distribution for neuron 1.1 of layer 103 may include the following probabilities as shown in Table 1.

[0120]

Table 1

[0121] Table 2 shows the probabilistic interpretations of real-valued logical operators (probabilistic logical operators) and their corresponding Boolean interpretations.

[0122]

Table 2

[0123] In Table 2, the ID column identifies each row. The operator column shows the Boolean operator. The rows of the real-valued column show the real-valued operator corresponding to the Boolean operator of that row. The rows of the columns "00", "01" ,"10", "11" show the output of the operator (Boolean or real-valued) corresponding to that row given the column header value (e.g., "00") as input. A test is shown that reducing the number of operators in Table 2 may lead to a decrease in performance. In other words ​​​​By using 16 logical operators instead of fewer than 16, machine learning tasks can be performed. To achieve this, we can improve the efficiency of the methods for training neural networks. stomach.

[0124] The values ​​in Table 2 represent probabilistic logical operators. Other interpretations of real-valued logical operators are used. It may also be done. For example, the Hamacher product T-norm and its dual Einstei The n-sum T-conorm may be differentiable and provides a suitable foundation for real-valued logical operators. Provided. Based on the Hamacher product T-norm and the Einstein sum T-conorm. The real-valued logical operators are shown in Table 3 below.

[0125] [Table 3]

[0126] Some operators in Table 3 can be derived from other operators in Table 3. Row 2 of Table 3, In 4 and 9, the respective Boolean operators do not have corresponding real-valued operators. Regarding the implications (rows 2 and 4 in Table 3), the R-implication (or residual) corresponding to the T-norm is May be used ("Continuous R-implications", BJ) (See ayaram et al., July 20-24, 2009). The probabilistic operators in Table 2 are tested. At that time, from the Hamacher T-norm and Einstein sum T-conorm in Table 3 It has also been found that this can be implemented successfully.

[0127] Other real-valued operators include the Frank T-norm, in addition to the corresponding T-conorm. Mu, Yager T-norm, Aczel-Alsina T-norm, Dombi T - Norms and Sugeno-Weber T-norms may also be used. Logical operators Further information on this topic can be found at "Analyzing Differentiable Fu zzy Logic Operators”, van Krieken et al., 20218 It can be found on the 24th of the month.

[0128] Further T-norms and T-conors may be used to implement real-valued operators. The terms are shown in Tables 4 and 5 below.

[0129] [Table 4]

[0130] [Table 5]

[0131] Exemplary values ​​for the neurons in Figure 1 during training are provided in the following table. Logical operator probabilities in the mid-stage of neural network training JPEG2026053632000007.jpg72170 Logical operator probability in the later stages of training of a neural network, i.e., after convergence. JPEG2026053632000008.jpg78170

[0132] The neurons in Figure 1 corresponding to each row of the logical operator probability shown above are provided at the beginning of the row. For example, a row starting with "1.1" shows the value for neuron 1.1 in layer 103. Each column in row a of the logical operator probability corresponds to a consecutive ID in Table 2. For example, logical Row 1 of the operator probability corresponds to ID 0 in Table 2, and row 2 of the logical operator probability corresponds to ID 1. Therefore, in the mid-term of training, the highest probability for neuron 3.2 is 0.678. This corresponds to ID 10 in Table 2, i.e., real-valued operator 1-B.

[0133] The mid-stage of training may also refer to the number of epochs during which the network is trained. The middle stage of training may refer to about half of the number of epochs. Similarly, the later stage of training refers to the final epoch. It may also refer to an epoch within 10% of the total. For example, if the neural network is 200 If training is conducted for a given epoch, the mid-term of training may be epochs 90-110. Good. Similarly, the later stages of training may be epochs 190 to 200. Convergence is, This can also refer to the stage of training where the neural network no longer improves even with additional training. .

[0134] As can be seen from the example values ​​above, 0-1 for each neuron during the mid-stage of training. There are a number of probabilities for the logical operators. However, after convergence, generally, one or two probabilities remain. Only one of the values ​​is non-zero, and at least one of the values ​​is close to 1.

[0135] Figure 2 shows a neural network for performing a regression task. In other words, instead of "Class 0" or "Class 1", the output of the neural network is This is the value of force row 107. More specifically, the output is an array of Boolean values ​​(i.e., each value is 0 or (This can be a vector of values ​​that are either 1 or any other value.) Or, a scalar output. It's okay to have that.

[0136] More specifically, in order to generate (i.e., predict) a k-dimensional output, n ni There may be a euron. The output is divided into k groups (for example, of size n / k). It may be divided. For each dimension i of the output (prediction), αi ·n / k is valid, and the output A scalar parameter α that provides a determined (desired) range for (the prediction) i may exist.

[0137] For example, if the regression task is to predict rainfall, there may be one dimension, and if the regression ta sk is to predict rainfall and wind speed, there may be two dimensions. The determined range of rainfall may be 0 to 200 mm. In another example, if the regression task is image generation case, k may be the number of pixels, and thus to generate an image of 784 pixels , k may be 784.

[0138] The final output may be determined by counting the value of 1 of the output neurons in the output layer 107 .

[0139] To generate outputs for both positive and negative values, a bias β is used to shift the determined range (i.e., the output space). i Thus, the final output may be determined by counting the output neurons and applying an affine transformation as shown in the following equation.

Equation

[0140] The affine transformation may be used to transform (or shift) the determined range of 0 to n / k to a more suitable range for a specific application .

[0141] In some cases, for all i, α = α i and β = β i such that all dimensions of the output have the same range. ​​

[0142] In some cases, it is desirable for the determined range to cover all real numbers. This can be achieved using a logit transformation, as shown in the following equation.

number

number

[0143] Using the output, the mean squared error (MSE) loss can be formulated as follows: .

number

[0144] From another perspective, the neural network in Figure 2 is equivalent to the neural network in Figure 1. handle.

[0145] Figure 3 shows the process for training a neural network to perform a machine learning task. The method is shown. The neural network is the neural network shown in Figures 1 and 2 above. • Network compatibility is also acceptable.

[0146] Neural networks, after training, generally perform the same tasks as conventional neural networks. Instead of performing floating-point calculations, all calculations should be performed as binary operations on Boolean values. This is sometimes called a deep differentiable logic gate network. The operators for each neuron in the network are exclusively implemented using logic gates. This is also acceptable. This involves performing machine learning tasks using a pre-trained neural network. When this happens, it can lead to a very sparse network and increased efficiency. Efficiency is The data for the machine learning task (e.g., test data) is in the first layer 103. It may be further increased by spreading it over a distance of 100 meters.

[0147] In step S201, the input data for the neural network is received. The input data is binary, such that each neuron in layers 103-105 receives two inputs. The value may be received as input 101. The input data consists of training data, validation data, and The training data may also include test data. The training data learns the probability distribution for each neuron. It may be used for training. The validation data is sufficient to ensure that the neural network is sufficiently accurate. Even if used to determine whether or not, that is, whether or not the training is complete. Good. Test data can be used during inference (after training is complete), for example, with other or conventional data. Used to evaluate neural networks against neural networks That's good too.

[0148] In step S203, for multiple hyperparameters of the neural network The value is determined. The hyperparameters are the number of layers (for example, about 2 to about 32), and in each layer This may include the number of neurons (e.g., approximately 12 to approximately 1,024,000) and the learning rate. .

[0149] For example, each layer may have the same number of neurons. In addition or otherwise, about It's acceptable to have 4 to approximately 8 layers. This relates to the architecture of neural networks. This could also lead to the advantage of not needing to fine-tune the structure. Therefore, this is high The determination of per-parameters may be simplified and sped up.

[0150] The learning rate may have a constant value of approximately 0.01.

[0151] In step S205, the neural network is constructed according to the hyperparameter values. This may be done according to the hyperparameter values, the number of layers (e.g., 4) and the number of layers This may include generating the number of neurons per unit (e.g., 8000). This means that the connections between neurons, i.e., the network weights, are initialized in a pseudo-random manner. It may be included. Other means of initializing the connection (i.e., weights) are also possible. For example, conventional A trained neural network with weights determined according to the approach is the basis. It may be used in this way.

[0152] Each neuron may contain probability distributions for multiple logical operators. Each may have the following characteristics (signature): f:{0,1}x{0,1}→{0,1}

[0153] Instead of a hard binary value p∈{0,1}, the probability of the probability distribution is p∈[0,1] 16 This may be relaxed. This is a step that makes the neural network differentiable. That's fine.

[0154] In a neural network, a single neuron may be defined as follows: The two inputs to the euron may be defined as a, b ∈ [0, 1]. Therefore, p The probability is Δ 15 p is a probability distribution across logical operators.

number

number

[0155] The output o may be ∈[0,1].

[0156] Even when using binary input, the neurons of a neural network are connected to each other. Since they can be distinguished, that is, the network values ​​are no longer constrained to ∈{0.1}. Therefore, depending on the probability distribution (for example, a categorical probability distribution), which logical operator (i.e., logical) By representing the choice of whether a gate is present in each neuron, (for example, in gradient descent) Training can be performed using the underlying algorithm.

[0157] Advantageously, the memory requirements for a neural network are less stringent than those of a neural network, as it uses logical operators. It is considerably less than in the case of neural networks that perform floating-point operations. This is also acceptable. For example, multiple logical operators may consist of 16 logical operators. Therefore, After the neural network is trained, it represents logical operators for a given neuron. Only 4 bits are needed for this. In other words, it specifies the operation that a given neuron will perform. Only 4 bits are needed to store information. This allows neurons to perform more complex calculations. Even if the amount of memory required is significantly less than what the neural network being implemented needs. stomach.

[0158] In conventional sparse neural networks, The connections between neurons (weights) may be trained.

[0159] In step S207, the neural network is trained according to the hyperparameter values. It may be done. In contrast to conventional sparse neural networks, neural... Training a network involves keeping the connections between neurons fixed after initialization, and then arguing which theory This includes learning whether to implement logical operators (i.e., binary functions) in each neuron. That's fine. Therefore, the learning objective is to determine which of the logical operators should be present in each neuron. It may also be to determine the function. Therefore, the network is such that each neuron The probability distribution for the logical operators is learned and then (continuously) parameterized. That's good too.

[0160] Neural networks may be relaxed. In other words, instead of fixed logical operators In addition, the probability distribution of logical operators may be used, and the neural network during training It may also work on probabilities. Relaxing logical operators allows neural networks to Another step in making it differentiable is to learn the probability distribution of each neuron. This may be done through parameterization of (relaxed) softmax.

[0161] Learning probability distributions involves using 16 floating-point values ​​corresponding to 16 logical operators. This may be achieved by parameterizing each neuron. Softmax is 1 It may be used to map the floating-point value of 6 to a probability unit (i.e., all (A categorical probability distribution where the sum of entries is 1 and only non-negative values ​​exist). Referring to the consideration of Equation 4, during the initialization of the neural network, the neurons are parameterized Data modeling involves selecting (depicting) elements of q independently of the standard normal distribution. That's good too.

[0162] The training involves evaluating all 16 logical operators for each neuron, and categorically. This may also include calculating their weighted mean using probability distributions. During training, the output of all neurons may be ∈[0,1].

[0163] More specifically, a classification task using k classes and n neurons in the output layer In this case, the output may be grouped into k groups of size n / k. The classification score for each class is calculated by counting the number of 1s in each group. Therefore, it may be determined. Thus, in relation to the classification task, neural network The output of the class may be determined by taking the argmax of the classification score.

[0164] To determine whether a neural network is sufficiently accurate, we count the number of ones. Instead, the probabilities of the output in each group may be added together. Therefore, classification The measure of accuracy may be determined by calculating the loss. For example, softmax The differential entropy classification loss may be calculated as follows:

number

[0165] After training is complete, the two inputs to the neuron are defined as a,b ∈ {0,1}. This is also acceptable. Therefore, even if the input to the neuron is a floating-point value ≥ 0 and ≤ 1 Good, in contrast to neurons during training (see Equation 4), input to neurons after training The force can be a Boolean value, i.e., either 0 or 1. Similarly, the output of the neuron o The range may be ∈{0,1}. Therefore, the output o of the trained neuron is given by the following equation. It may also be defined as follows. o=0p γ (a,b) (Equation 6) In equation (6), γ = argmax i p i Here, i refers to the ID in Table 2.

[0166] In step S209, a logical operator may be determined for each neuron. The child may be determined after training is complete, i.e., during inference. The determined logical operator is the most likely. It may also be a logical operator of degree, for example, a logical operator of the probability distribution with the highest probability. In other words Therefore, the probability distribution may be discretized by taking those modes. The machine learning task was performed by calculating Boolean values ​​rather than floating-point values. This is also true, and as a result, it is more efficient compared to neural networks that rely on floating-point arithmetic. The process becomes more efficient.

[0167] Before performing a machine learning task using a trained neural network, The RAL network can be compiled into at least one binary executable file. The binary executable file may be processor-dependent. For example, one might be for a CPU. For example, two binaries: one for the GPU (for example, from CUDA) and one for the GPU. The line files may be compiled.

[0168] In addition, logical and / or sub-expressions may be simplified. For example, Boolean type Instead of data types, use hardware corresponding to the processor on which the machine learning task should be performed. A software implementation data type may be used.

[0169] For example, in the case of a 64-bit CPU, the hardware implementation data type is int64. It is also acceptable to use a total size of 64, that is, 64 samples (example). For example, an image is processed through a neural network in a given iteration (i.e., epoch). In a GPU, arbitrary writes to GPU memory can become a bottleneck (i.e., Because this can lead to a reduction in ability, output neurons constitute each adder. Logic gates may be used to directly sum the calculations. Generally, bottlenecks (speed reductions) are a factor. The point in the system where this may occur is the data loader and / or transmission speed. That's fine.

[0170] Therefore, bitwise operations may be performed on large batches, However, this can have a significant impact on the speed at which machine learning tasks can be performed. .

[0171] The output layer 107 may generate multiple outputs for each class. The outputs are obtained via bit counting. In other words, by counting the 1s that produce a score for each class, the total can be calculated. Therefore, if the machine learning task is a classification task, the classification task will have the highest score. Even if it is completed by providing the class it possesses as the output of the neural network good.

[0172] When performing a machine learning task using a binary vector given as input, The pair of value values ​​may be selected from binary vectors, one of the layers (for example, layer 1) The logical operators (i.e., binary logic gates) in 03) may also be applied to Boolean values, and then Those outputs are then passed to subsequent layers of the neural network (e.g., layer 105 or 107). It can be used for this purpose.

[0173] After training, the computational cost of performing machine learning tasks is different for traditional binary and sparse binary. Compared to neural networks, and in some cases even other types of conventional networks, Compared to neural networks, the efficiency may be reduced by at least an order of magnitude.

[0174] An exemplary machine learning task is "The monk's problems: A per formance comparison of different learning As discussed in "g algorithms" by Thrun et al., 1991, Mon This is the Monk problem. The Monk problem, MONK-1, MONK-2, and MONK-3 are problems involving machines. These are three machine learning tasks that have been used to benchmark learning algorithms. These tasks involve a data set with six attributes, each with two to four possible values. It consists of three binary classification tasks for the data points (samples). It can be encoded as a binary vector of size 17.

[0175] The test involved the aforementioned neural network on all three MONK datasets. In contrast, it shows that it performs more accurate classification than logistic regression. In addition, MO In the case of NK-3, the neural network described above (i.e., Figures 1 and 2) is It is more accurate than a much larger convolutional neural network. In addition, the aforementioned RAL networks are more than three times faster than logistic regression and can handle larger convolutions. This is more than seven times faster than a complex neural network. In addition, the aforementioned neural... The network is either a logistic regression or a convolutional neural network. It requires significantly less memory space.

[0176] As another exemplary machine learning task, the Adult Census ("UCI machine learning") “earnings repository:Adult data set”, Kohav i et al., 1996) and breast cancer ("UCI machine learning re “pository breast cancer dataset”, Zwitter et al. The dataset from 1988 may also be considered. Regarding the dataset for adults, The machine learning task assumes that a given adult has 5 years of experience based on attributes such as education and weekly working hours. The prediction is whether or not to earn more than $0,000. Regarding breast cancer, machine learning Suku includes binary classification and is based on characteristics of the cell nucleus, including perimeter, area, and smoothness, for cancer This involves determining whether the diagnosis is benign or malignant. Therefore, the (classification) accuracy achieved by the above-mentioned neural network is different from that of conventional (convolutional) (Including) Comparable to neural networks and logistic regression. In addition, classification speed This is more than 10 times faster than logistic regression, compared to conventional neural networks. It is more than 40 times faster than [another method]. In addition, the aforementioned neural network requires [something]. The memory space is about 20% smaller than that of logistic regression, and compared to conventional neural networks... It is also about 75% less.

[0177] Furthermore, a very high frame rate may be achieved in image classification. For example, the above New In the case of the RAL Network, Modified National Institute of Standards and Technology (MNIST) dataset (http: / / yann.lecun.com / exdb / mnist / ) and C anadian Institute For Advanced Research( The CIFAR-10 dataset has a frame rate exceeding 1 million images per second. It may be achieved ("Learning Multiple Layers of Fe atures from Tiny Images” by Alex Krizhevsky (April 8, 2009). In other words, a classification rate exceeding 1 million images per second is possible with a single CPU. This may be achieved using the core, even if it outperforms the efficiency of any conventional approach. good.

[0178] More specifically, the neural network described above is the fastest conventional binary network It has image classification accuracy on the MNIST dataset that is comparable to that of a neural network. It is permissible to do so, while the number of necessary binary operations is less than 10%. (Example: Standard GPU) However, in the NVIDIA A6000, the G required by the aforementioned neural network Although PU utilization is only 7%, the aforementioned neural network uses a dedicated FPGA hardware. It performs the task 12 times faster than conventional binary neural networks. It may also be possible. Compared to another conventional binary neural network, the above neural The network can be about three orders of magnitude faster. Compared to a simple network, the neural network described above is more accurate.

[0179] For image classification using the CIFAR-10 dataset, the neural network described above... The accuracy of the workpiece may be comparable to that of conventional convolutional neural networks, while on the other hand, The required memory is less than 0.1% in some cases and less than 1% in others. The connected network offers slightly (less than 4%) higher accuracy but requires 64% more memory.

[0180] When performing image classification using the CIFAR-10 dataset, floating-point arithmetic is required. Traditional fully connected neural networks, which rely on this, perform machine learning tasks. This may require 2,000,000 floating-point operations, but the above-mentioned neural network... The twerk has 5,000,000 bits of logic before pruning or optimization. This requires computation. Floating arithmetic requires hardware-accelerated integrated circuits (e.g., modern GPUs and many others). In a CPU, a floating-point operation of 2,000,000 is equivalent to a floating-point operation of 5,000,000 bits. It is about 1 / 100th the speed of unit-level logical operations. In total, the difference in speed becomes an order of magnitude larger, meaning the difference becomes three orders of magnitude.

[0181] Even with conventional sparse neural networks, or conventional fully connected neural networks... It is faster than a network, but still less so than the neural network mentioned above. At the very least, it is an order of magnitude slower. A single sparse neural network is, as mentioned above, It requires at least twice the memory space of a RAL network.

[0182] Possible exemplary examples for the neural network described above (related to Figures 1 and 2) The architecture is shown in Table 6.

[0183] [Table 6]

[0184] Table 7 was used as a basis for comparison with the neural network configurations in Table 6. This shows the configuration of a fully connected ReLU network.

[0185] [Table 7]

[0186] In addition, a general-purpose desktop computer or laptop computer In each clock cycle of the CPU (for example, per Hertz), each of the CPU's It may be possible to compute an average of approximately 250 binary logic gates for A. This is because a typical CPU, even a single-core one, has many life cycles per clock cycle. This is possible to execute the command. This relies on floating-point arithmetic and neural networks. It may be significantly faster than what is possible when performing the work. In addition, the CPU, Extracting bits from multiple samples of data (e.g., images) into a single integer (e.g., data...). By grouping them into integers of type int64, the data for machine learning tasks can be processed. The data may be spread across neurons in the first layer 103. Grouping bits from a sample into a single integer is called Single Instruction Multiplexing (SI). It is sometimes called MD. Additional efficiency improvements are Advanced Vector Extension. It may be possible using AVX.

[0187] The process is carried out by layers having n neurons (e.g., layers 103, 105, or 107). The computational cost of some of the machine learning tasks performed requires only Boolean logic gates (immediately Furthermore, the operations are performed exclusively through logical operators, and therefore are performed very efficiently. (This is possible), so it can be 0(n) including small fixed costs. Compared to that, m A fully connected layer with a number of input neurons is particularly useful for floating-point operations. Because it requires a significantly higher fixed cost, it has a computational cost of O(n·m). Overall A neural network trained according to the method described with reference to Figures 1 and 2. Performing machine learning tasks using a fully connected ReLU neural network It is two orders of magnitude faster than a neural network using the ReLU activation function. This can lead to faster reasoning. In addition, trained Newtypes according to the methods in Figures 1 and 2 The RAL network is more than 13 times faster than conventional binary neural networks. This speed is two to three orders of magnitude faster than the theoretical speed of conventional sparse neural networks. High speed is also acceptable.

[0188] Figure 4 shows the relationship between a neural network trained to perform a machine learning task. This shows the distribution of logic gates. In this case, the machine learning task includes image classification. The RAL network has four layers, each containing 12,000 neurons. For 1 and 2, the values ​​on the X-axis are the IDs in Table 2, and the X-axis for layers 3 and 4. Corresponds to the values ​​above. The value on the Y-axis is a count of the number of neurons where a logical operator exists on the X-axis. To provide.

[0189] Figure 4 shows the constants "0" and "1" (corresponding to IDs 0 and 15 in Table 2). Logical operators are used very infrequently and are not seen at all in layer 4. In layer 1, "and", "nand", "or", and "nor" have relatively strong probabilities. It has.

[0190] In the second and third layers, one of the two inputs is ignored, and the other input is the next Since it is sequentially passed through the layers (in some cases in a modified form), it can be viewed as a direct residual connection. The combinations "A", "B", "┐A", and "┐B" occur more frequently. By representing it with fewer layers than the predefined number, the neural network This could allow for more efficient modeling of lower-order dependencies.

[0191] In layer 4, the most frequent operations can create a conditional dependency on the activations of the previous layer. These are "xor" and "xnor". As illustrated, the implication is used infrequently. However, the test is theoretical (not all 16 operators of the probabilistic interpretation in Table 2). This demonstrates that using only an appropriate subset of precision operators can lead to a decrease in precision. They are doing it.

Claims

1. Train neural networks to perform machine learning tasks such as image classification. A computer implementation method for that purpose, Receiving input data for the neural network (S201), Determining values ​​for multiple hyperparameters of the neural network ( S203) and, The system comprises multiple neurons (1.1 to 3.4) according to the hyperparameter values. The task is to construct a neural network (S205), Each neuron (1.1-3.4) is a neuron (1.3-3.4) with multiple logic For each operator, a probability distribution for that logical operator is defined, which includes the corresponding probability. This includes constructing the neural network, By learning the probability distribution of each neuron (1.3 to 3.4), the hyperspace Training the neural network according to the parameter values ​​and the input data (S 207) and, By selecting a value in the probability distribution, each neuron (1.3 to 3.4) (S209) The logical operators of the plurality of logical operators are determined Computer implementation methods, including those mentioned above.

2. The aforementioned input data includes an input and a corresponding desired output. The aforementioned input data is one of the training data, validation data, and test data. This includes multiple The method according to claim 1.

3. The hyperparameter values ​​are determined under one or more specified computational resource constraints for the neural It was decided to minimize the validation error of the network, The specified computing resource constraints may include processor and / or memory constraints. The method according to claim 1 or 2.

4. The aforementioned hyperparameters include the number of neurons per layer, the number of layers, the number of epochs, and batch size. Including one or more of the following: is and learning rate, The method according to any one of claims 1 to 3.

5. Claim 1, wherein the neural network includes a plurality of layers, including an output layer (107). The method described in any one of items (a) 4.

6. The output layer (107) includes n neurons, and the machine learning task has k classes. This is a classification task that has the following characteristics: The method further uses the trained neural network to perform the classification task. Performing the classification task, including carrying it out, The aforementioned trained neural network performs classification for each of the classes. Outputting the core, The output is grouped into k groups of size n / k, and each group The number of 1s in the group corresponds to the classification score of the class corresponding to that group, thus grouping the data. Toto, Determining a classification includes determining the maximum value of the classification score. The method according to claim 5, including the method described in claim 5.

7. After training the neural network and before performing the machine learning task Furthermore, the above method further, The aforementioned trained neural network is used by a central processing unit or graphics processing unit. This includes converting it to an executable binary, The executable binary is a program that can handle shared object binaries. It can be called via and the program may be written in Python. The method according to any one of claims 1 to 6.

8. The output of each neuron is a value in the range of 0 to 1, and / or Each neuron receives an input of 0 to 3. The method according to any one of claims 1 to 7.

9. Constructing the aforementioned neural network further raises suspicions about the connections between the neurons. This includes initializing in a similarly random manner. During the aforementioned training, the connections between neurons may remain fixed. The method according to any one of claims 1 to 8.

10. The aforementioned machine learning task, Binary classification, pattern recognition, image classification, object identification or recognition, character recognition, gesture recognition - Or facial recognition, voice detection or speech recognition, text classification A classification task comprising one or more of the following, as described in any one of claims 1 to 9. The method.

11. The aforementioned machine learning task is a regression task, and this task is, Image generation, text generation, video generation, speech generation, 3D reconstruction, compression, encoding The method according to any one of claims 1 to 10, comprising one or more of the above.

12. The plurality of logical operators include at least two operators or exactly sixteen operators. fruit, The aforementioned multiple logical operators may be real-valued operators, The real-valued operator is probabilistic, follows the Hamcher t-norm, and is relative. c Einstein and t-conorm, Lukasiewicz t-norm and t- It can be based on one of the interpretations, such as the Conorm. The method according to any one of claims 1 to 11.

13. The aforementioned input data includes test data having multiple samples, During inference, the first layer (103) of the neural network performs the test... Performing the machine learning task by accessing the total amount of data, This involves assigning the elements of each sample of the test data to consecutive bits of an integer. Including, During inference, the two inputs of each neuron are used to determine the machine learning task on which the task should be performed. Numeric data types corresponding to the hardware implementation data types of the computer's processor possess, The method according to any one of claims 1 to 12.

14. A computer program, wherein the program is executed by a computer. Then, an instruction to cause the computer to carry out the method described in any one of claims 1 to 13. A computer program that includes [this].

15. Train neural networks to perform machine learning tasks such as image classification. A computer system for that purpose, It comprises at least one processor, and the processor is The neural network receives input data, Determine the values ​​for multiple hyperparameters of the neural network, The system comprises multiple neurons (1.3 to 3.4) according to the hyperparameter values. It is about building a neural network, Each neuron (1.3-3.4) is a neuron (1.3-3.4) with multiple logic For each operator, a probability distribution for the logical operator, which includes the corresponding probability. The neural network is constructed, including the following: By learning the probability distribution of each neuron (1.3 to 3.4), the hyperspace The neural network is trained according to the parameter values ​​and the input data. By selecting a value in the probability distribution, each neuron (1.3 to 3.4) Determine the logical operators of these multiple logical operators. A computer system configured in such a way.

Images