Device and method for training binary deep neural network

Abstract

There is disclosed an apparatus (600) for training a binary deep neural network, the apparatus being configured to: generate (501) a training signal (302) from an error between an output and an expected output of a prototype version of the binary deep neural network (301), the prototype version of the binary deep neural network having a plurality of binary weights, each binary weight having a respective value; in accordance with the training signal (302), for each binary weight of the prototype version of the binary deep neural network (301), a respective decision to reverse or maintain the respective value of the respective binary weight is output (502). This may cause the device to train a deep neural network including binary parameters directly in the binary domain without the need for a gradient processing method. Thus, the training process may require less memory and computing power, since the training process does not use memory-intensive gradient signals.

Description

Technical Field

[0001] The present invention relates to machine learning models, and in particular to the training of binary deep neural networks. Background Art

[0002] Deep learning is at the root of many of the successes of the past decade in computer vision and natural language processing. It has taken center stage in technology and society, with applications far beyond computer science.

[0003] Deep learning uses deep neural networks (DNNs), which are complex nonlinear systems inspired by the biological brain that are able to manipulate high-dimensional objects, learn autonomously from given examples without being programmed with any task-specific rules, and achieve state-of-the-art performance.

[0004] Figure 1 The basic operation scheme of a DNN is shown, which is generally divided into two phases. In the training phase, the DNN (labeled as 'model' in the figure) learns its own parameters. A starting version of the model to be trained is trained using the provided training data 102, as shown in 101, to form a trained model 103. Then, in the inference phase, the DNN is used to output one or more predictions 104 for unseen input data 105.

[0005] Deep neural networks are generally considered to be highly intensive in terms of memory and computation. On the one hand, DNNs usually consist of a very large number of parameters, reaching hundreds of millions in today's applications. This requires a large memory footprint to represent the model. In addition to this, the training phase requires a large amount of training data, and a large number of other temporary variables (also called parameters) used to optimize the model weights, such as gradient information. Therefore, training DNNs usually requires dedicated powerful infrastructure, which may limit the potential of artificial intelligence.

[0006] A promising approach to solving this memory wall problem is to design DNNs with binary parameters, which means that model parameters are represented by binary numbers, occupying only 1 bit, instead of using 32-bit floating point numbers. This can not only greatly reduce the model footprint, but also reduce training memory and reduce computational complexity.

[0007] However, binary parameters are discrete and cannot be optimized using existing deep learning theory of gradient descent (see, e.g., https: / / en.wikipedia.org / wiki / Gradient_descent).

[0008] Binarization is an important method for training binary deep neural networks. Figure 2A schematic operation diagram of the process is shown, where a floating point model 201 is a starting DNN with floating point parameters, and a floating point gradient descent optimizer 202 is a block that optimizes the model's floating point parameters using the principle of gradient descent. A binarization block 203 converts the model parameters from floating point to binary form, for example by using symbol extraction that follows some predefined performance criteria. The binarized model 204 is updated during the training process, and the final model obtained at the end of the training process will be used for inference.

[0009] The main limitation of this approach is that the training phase relies entirely on floating-point training. Not only does it not solve the memory and computational complexity issues, it also adds more complexity to the training process.

[0010] There is a need to develop a method to overcome these problems. Summary of the invention

[0011] According to a first aspect, a device for training a binary deep neural network is provided, the device being used to: generate a training signal based on an error between an output of a prototype version of the binary deep neural network and an expected output, the prototype version of the binary deep neural network having a plurality of binary weights, each binary weight having a corresponding value; and output, based on the training signal, a corresponding decision of whether to invert or maintain the corresponding value of the corresponding binary weight for each binary weight of the prototype version of the binary deep neural network.

[0012] This enables the device to train and optimize DNNs including binary weights directly in the binary domain without the need for gradient processing methods. As a result, the training process may require less memory and computing power because the training process does not use memory-intensive gradient signals and floating-point multiplications.

[0013] The apparatus may be configured to generate the training signal according to a predefined optimization objective. This may enable the training signal to reflect the performance of the prototype version of the DNN in predicting the results compared to the expected results.

[0014] The predefined optimization objective may be minimization of a loss function. This may be a convenient implementation for training a deep neural network.

[0015] The training signal may include one or more quantities. The device may be configured to output the corresponding decision for each binary weight based on the one or more quantities so as to meet the predefined optimization objective. This may enable the device to use the training signal to determine whether to invert a particular weight to meet a predefined optimization objective, such as to minimize a predetermined loss function.

[0016] The device may also be used to track the state of the predefined optimization target. This may enable the device to incorporate the state of the optimized entity into the optimization signal during the iterative optimization process. This may enable the trained deep neural network to achieve better prediction accuracy during inference.

[0017] The corresponding decision to invert or maintain the corresponding value of each binary weight of the prototype version of the binary deep neural network can be based on an optimization signal calculated based on the training signal. This can enable the device to make a decision to invert or maintain each binary weight of the prototype version of the DNN.

[0018] Each binary weight can have only two possible values. This allows the model parameters to be represented by binary numbers, occupying only 1 bit, instead of using 32-bit floating point numbers. This not only reduces the model footprint, but also reduces training memory and computational complexity.

[0019] The apparatus may be configured to receive a training data set for forming the output of the prototype version of the binary deep neural network, the training data set comprising input data and corresponding expected outputs. A training signal may be formed based on an error between the output of the prototype version of the binary deep neural network and the expected output. This may enable the prototype version of the binary DNN to process the input data to form a predicted output, which may then be compared to the corresponding expected output. This may enable the apparatus to evaluate the performance of the prototype version of the binary DNN.

[0020] The device may also include a memory for storing an accumulator updated according to a predefined optimization function. This may support controlling the amount of binary weights to be inverted during each training iteration, thereby enhancing training convergence and performance.

[0021] The device may be configured to reset the memory in accordance with the decision. For example, for each binary weight that is inverted in a particular iteration of the training process, the device may instruct a memory reset for the stored output of the accumulator function corresponding to the inverted weight. This may cause the accumulator to be reset each time the corresponding binary weight is inverted.

[0022] The device can also be used to update the binary weights of the prototype version of the binary deep neural network according to the corresponding decision. This can enable the device to optimize the weights and update the DNN for use in the next iteration of the training process.

[0023] The device can be used to iteratively update the binary weights of the prototype version of the binary deep neural network until a predefined convergence level is reached. This can enable the resulting trained binary deep neural network to reach a predefined performance level during inference.

[0024] The binary deep neural network can be a Boolean deep neural network including Boolean neurons. Using the Boolean neuron design, Boolean layers and networks (such as linear layers and convolutional layers) can be directly constructed in the same way as floating-point layers and networks are constructed from floating-point neurons.

[0025] According to another aspect, a method for training a binary deep neural network is provided, the binary deep neural network comprising a plurality of binary weights, the method comprising: generating a training signal based on an error between an output of a prototype version of the binary deep neural network and an expected output, the prototype version of the binary deep neural network having a plurality of binary weights, each binary weight having a corresponding value; and outputting, based on the training signal, for each binary weight of the prototype version of the binary deep neural network, a corresponding decision as to whether to invert or maintain the corresponding value of the corresponding binary weight.

[0026] This method can support optimizing or training DNNs including binary weights directly in the binary domain without the need for gradient processing methods. Therefore, the training process may require less memory and computing power because the training process does not use memory-intensive gradient signals and floating-point multiplications.

[0027] According to another aspect, a computer readable storage medium storing computer readable instructions is provided, which, when executed at a computer system, causes the computer system to perform the method as described above. The computer system may include one or more processors. The computer readable storage medium may be a non-transitory computer readable storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The present invention is described below by way of examples with reference to the accompanying drawings.

[0029] In the attached picture:

[0030] Figure 1 The operation scheme of DNN is schematically shown.

[0031] Figure 2 Schematic showing the binarization of a model trained using a floating point gradient descent optimizer.

[0032] Figure 3 Schematic illustration of the training of a binary DNN in the binary domain.

[0033] Figure 4 An example of processing the flows and signals of one layer of a binary DNN is shown.

[0034] Figure 5 A flowchart of an example of a method for training a binary DNN including multiple binary weights provided by an embodiment of the present invention is shown.

[0035] Figure 6 An example of a device for training a binary DNN is shown.

[0036] 7(a) and 7(b) show examples of results achieved in terms of memory consumption and memory reduction, respectively, using embodiments of the present invention. DETAILED DESCRIPTION

[0037] A weight is a parameter in a neural network that is used to transform the input data in the hidden layers of the network. A neural network consists of a series of nodes, or neurons. In each neuron, there is a set of inputs, a set of weights, and a bias value. When an input enters a node, it is multiplied by the corresponding weight value. Before passing the data to the next layer, it can optionally add a bias. The resulting output is observed, or passed to the next layer of the network.

[0038] Weights are learnable parameters in a network. DNNs can randomize weights before they initially begin learning. As training continues, weights are adjusted to desired values ​​to give the "correct" output.

[0039] Embodiments of the present invention may support direct training and optimization of DNNs with binary weights in the binary domain without the need for gradient processing.

[0040] The 'weights' of a DNN can also be interchangeably called 'parameters' with the same meaning.

[0041] Each binary weight has only two possible values. This allows the model weights to be represented by binary numbers, occupying only 1 bit, instead of using 32-bit floating point numbers. This not only reduces the model footprint, but also reduces training memory and computational complexity.

[0042] Given a binary DNN, a logic rule can be established to decide whether to invert or retain each binary weight. This logic rule makes decisions by using the signals calculated from the binary model to achieve a predefined optimization goal, such as minimizing the learning loss function.

[0043] When a decision to invert a binary weight is given, the binary weight is inverted from its current binary value to another possible binary value. This avoids the combinatorial nature of discrete optimization problems, i.e., nondeterministic polynomial time hard (NP-Hard), and also takes advantage of the binary constraint, i.e., a binary parameter has only two values, so inverting or keeping the current value is the ultimate question.

[0044] Figure 3 An exemplary embodiment of the present invention is shown. A binary optimizer is shown at 300 .

[0045] The binary deep neural network 301 is a DNN to be trained. The DNN 301 includes an input layer, an output layer, and one or more hidden layers. The DNN 301 has binary weights. Initially, the weights of the DNN 301 may be randomized.

[0046] The training process follows a standard iterative process that iterates through a cycle of a forward pass followed by a backward pass. In the forward pass, the training data is injected into the prototype version of the DNN (i.e., the current version of the DNN in this iteration of the training process) to evaluate the error between the output produced by the prototype version of the DNN and the true data labels. In the backward pass, the error is propagated from the output layer to the entire network and back to the input layer. This can be done relying on an optimization objective (e.g., minimizing a predetermined loss function).

[0047] During this back-propagation, different information is calculated, including the training signal indicated at 302. For example, in a standard full-precision DNN, the training signal includes the gradient of the loss function with respect to each quantity to be optimized.

[0048] In this implementation, the training signal 302 is generated based on the error between the output of the prototype version of the binary deep neural network 301 and the expected output (the true data label of the training data).

[0049] Based on the training signal, for each binary weight of the prototype version of the binary deep neural network 301, a corresponding decision is output as to whether to invert or maintain the corresponding value of the corresponding binary weight, as described in more detail below.

[0050] The training signal may include a specially defined quantity to express how a predefined optimization objective (e.g., a loss function) changes when the binary weights are inverted. These quantities are not necessarily real-valued signals. These training signals are sent to and used by the processor 303. The processor 303 is used to implement at least one function that outputs an optimization signal from the received training signal for each binary weight as an input to the function. The optimization signal is then fed to the decision logic 304 and the optimizer memory 305.

[0051] Calculating the optimization signal from the training signal may be implementation specific.A general way of calculating the optimization signal from the training signal is to calculate the accumulation of the training signal over multiple iterations of the training process.

[0052] The decision logic 304 uses the received optimization signal to make a decision to invert or maintain each binary weight of the current version of the DNN. Depending on the instruction from the decision logic 304, the binary inverter 306 can perform a binary inversion of the binary weight or can maintain its value.

[0053] The controller 307 can control the optimization process. For example, the controller can consider the decisions made by the decision logic 304 to indicate a memory reset, or adjust the way the processor 303 calculates the optimization signal. The controller 307 can also track the status of the optimization target.

[0054] The updated binary weights 308 are sent back to the binary DNN, and the weights of the prototype version of the binary DNN 301 are updated as needed, i.e., inverting the weights of decisions that have already been output, or keeping them.

[0055] In a preferred embodiment, the binary DNN is a Boolean deep neural network composed of Boolean neurons. The Boolean neurons have a Boolean input b 1 、b 2 ,……,b m and the Boolean weight w 0 、w 1 ,……,w m , where m is the number of inputs and in one specific example the output is a Boolean value given by:

[0056]

[0057] Wherein, XOR is Boolean exclusive OR logic, T is a predefined threshold, and in this example, is set to T=(m+1) / 2.

[0058] Using Boolean neuron design, Boolean layers and networks (such as linear layers and convolutional layers) are directly constructed in the same way as floating-point layers and networks are constructed from floating-point neurons.

[0059] For the above Boolean network, Figure 4 An example of the signals and flows for processing a layer of a binary DNN provided by the preferred embodiment is shown. A Boolean optimizer is indicated at 401. A layer of the DNN is indicated at 402.

[0060] exist Figure 4 , the arrows from left to right indicate forward processing, the arrows from right to left indicate backward processing, and the vertical arrows are between layer 402 and optimizer 401. Specifically, W is the weight, Z is the back-propagated signal received from the downstream layer, U = MAJ(XOR(W, Z) is the signal to be sent to the upstream layer. X is the feedforward input of the layer, and Y = MAJ(XOR(X, W) is the signal to be sent to the downstream layer.

[0061] The weights W and feed-forward inputs X may be stored in a layer, as shown in 403 .

[0062] In this example, the training signal Q is provided by Q=MAJ(XOR(X,Z)), where XOR(X,Z) is an element-wise XOR operation on appropriately matched dimensions X and Z, and MAJ(A) is a majority vote of the Boolean array A, outputting TRUE if A includes more TRUE and FALSE, otherwise outputting FALSE.

[0063] In this example, for a Boolean neuron with XOR logic, the rule for optimizing binary weights is 'if XOR(W, XOR(X, Z)) = FALSE, then invert weight W'. In this example, the training signal is Q: = XOR(X, Z). The signal Q is a quantity used to determine whether to invert W to minimize a predefined loss function.

[0064] Other logic rules may also be used to determine whether to invert W, as appropriate.

[0065] Therefore, the training signal is used to determine whether to invert the values ​​of the binary weights in the prototype version of the binary DNN.

[0066] Back to Figure 3 , in one example, the memory 305 stores the accumulator M. The controller 307 controls the optimizer operation. The controller 307 can specify a first scalar parameter ALPHA and a second scalar parameter BETA, which can be fixed or adjusted during the training process as needed. ALPHA can control the amount of binary weights to be changed in each iteration. BETA behaves like a forgetting parameter that reflects the system evolution during the training process, thereby mimicking the brain plasticity phenomenon.

[0067] In one embodiment, the controller 307 adjusts BETA as the ratio between the number of non-inverted binary weights of the layer and the total number of binary weights of the layer in each training cycle. Here, the number of non-inverted binary weights is obtained from the decision logic 304.

[0068] For each binary weight that is inverted in a particular iteration of the training process, the controller 307 may instruct a memory reset for the accumulated value M corresponding to the inverted weight.

[0069] The processor 303 may perform M←BETA*M+ALPHA*Q, where '*' represents a standard real-valued multiplication. The updated M for the inverted weights is stored in the memory 305.

[0070] If W=True and M>=1, or M=False and M<=−1, then the decision logic 304 gives a reversal instruction for the weight W.

[0071] The binary inverter 306 executes W=NOT W according to the inversion instruction. The values ​​of the weights are then updated in the prototype version of the DNN 301 for use in the next training iteration. Alternatively, if the trained model has converged, the DNN with these updated weights will be used in the inference phase.

[0072] Figure 5 An example of a method for training a binary deep neural network provided by an embodiment of the present invention is shown. As described above, the binary deep neural network includes a plurality of binary weights. At step 501, the method includes generating a training signal according to an error between an output of a prototype version of the binary deep neural network and an expected output, the prototype version of the binary deep neural network having a plurality of binary weights, each weight having a corresponding value. At step 502, the method includes outputting a corresponding decision to invert or maintain a corresponding value of the corresponding binary weight for each binary weight of the prototype version of the binary deep neural network according to the training signal.

[0073] Figure 6 An example of an apparatus for implementing the above method is shown.

[0074] Device 600 may include at least one processor (eg, processor 601) and at least one memory (eg, memory 602). The memory stores code in a non-transitory manner that can be executed by one or more processors to implement the device as described herein.

[0075] The device 600 can also be used to implement a binary deep neural network trained according to the method 500, and has optional additional features described with reference to the above embodiments. This can enable the device to use the binary DNN to perform reasoning for various tasks.

[0076] The main advantage of the approach described in this paper is that it enables direct training of binary DNNs in the binary domain without the need for floating-point (or full-precision) gradients, thus reducing memory usage and computational complexity many times while approaching the prediction accuracy of full-precision training, as shown in Table 1 below.

[0077] Table 1: Test accuracy on the Canadian Institute for Advanced Research (CIFAR10) image dataset using the visual geometry group (VGG) small architecture (batch size 100)

[0078]

[0079] FIG. 7( a ) and FIG. 7( b ) respectively show the advantages of the above preferred embodiment in terms of memory consumption and memory reduction compared with full-precision training.

[0080] Therefore, embodiments of the present invention can support training and optimizing binary DNNs directly in the binary domain without gradients. Compared with existing solutions that require DNNs to calculate gradient signals, the training process requires less memory and computing power because the training process does not use memory-intensive gradient signals. This approach is native in deep architectures.

[0081] The method works directly on the binary weights of binary DNNs. Existing schemes require two versions of each quantity (full precision and binarized), such as weights and neuron inputs and outputs. Not using full precision signals (during training as described in this article) results in much less memory and computational power required.

[0082] The optimizer controller reflects the decision results on the processor and memory, and supports the use of a binary training process to achieve better prediction accuracy. This is an innovative component that incorporates the state of the optimized entity into the optimization signal during the iterative optimization process. This can support the use of a binary training process to achieve better prediction accuracy.

[0083] Applicants hereby disclose separately each individual feature described herein and any combination of two or more such features. With the common knowledge of those skilled in the art, such features or combinations can be implemented as a whole according to this specification, without considering whether such features or combinations of features can solve any problem disclosed herein; and without limiting the scope of the claims. This application shows that various aspects of the present invention can be composed of any such individual features or combinations of features. In view of the above description, it is obvious to those skilled in the art that various modifications can be made within the scope of the present invention.

Claims

1. A device (600) for training a binary deep neural network, characterized in that: The device is used for: generating (501) a training signal (302) based on an error between an output of a prototype version of the binary deep neural network (301) and an expected output, the prototype version of the binary deep neural network (301) having a plurality of binary weights, each binary weight having a corresponding value; Based on the training signal (302), for each binary weight of the prototype version of the binary deep neural network (301), a corresponding decision of whether to invert or maintain the corresponding value of the corresponding binary weight is output (502).

2. The device (600) according to claim 1, characterized in that The device is configured to generate the training signal according to a predefined optimization objective (302).

3. The device (600) according to claim 2, characterized in that The predefined optimization objective is minimization of a loss function.

4. The device (600) according to claim 2 or 3, characterized in that The training signal (302) includes one or more quantities, and the device (600) is used to output the corresponding decision for each binary weight according to the one or more quantities so as to meet the predefined optimization goal.

5. The device (600) according to any one of claims 2 to 4, characterized in that The device (600) is also used to track the status of the predefined optimization goal.

6. The device (600) according to any one of the preceding claims, characterized in that The respective decision to invert or maintain the respective value of each binary weight of the prototype version of the binary deep neural network (301) is based on an optimization signal calculated from the training signal (302).

7. The device (600) according to any one of the preceding claims, characterized in that Each binary weight has only two possible values.

8. The device (600) according to any one of the preceding claims, characterized in that The device (600) is used to receive a training data set for forming the output of the prototype version of the binary deep neural network (301), wherein the training data set includes input data and corresponding expected output.

9. The device (600) according to any one of the preceding claims, characterized in that The device (600) also includes a memory (305, 602) for storing an accumulator updated according to a predefined optimization function.

10. The device (600) according to claim 9, characterized in that The device is used to reset the memory (305, 602) according to the corresponding decision.

11. The device (600) according to any one of the preceding claims, characterized in that The device (600) is also used to update the binary weights of the prototype version of the binary deep neural network (301) according to the corresponding decision.

12. The device (600) according to claim 11, characterized in that The device (600) is used to iteratively update the binary weights of the prototype version of the binary deep neural network (301) until a predefined convergence level is reached.

13. The device (600) according to any one of the preceding claims, characterized in that The binary deep neural network (301) is a Boolean deep neural network including Boolean neurons.

14. A method (500) for training a binary deep neural network, characterized in that: The binary deep neural network includes a plurality of binary weights, and the method includes: generating (501) a training signal based on an error between an output of a prototype version of the binary deep neural network (301) and an expected output, the prototype version of the binary deep neural network (301) having a plurality of binary weights, each binary weight having a corresponding value; Based on the training signal (302), for each binary weight of the prototype version of the binary deep neural network (301), a corresponding decision of inverting or maintaining the corresponding value of the corresponding binary weight is output (502).

15. A computer-readable storage medium (602) storing computer-readable instructions, characterized in that: The computer readable instructions, when executed at a computer system (601), cause the computer system to perform the method (500) according to claim 14.

Images