Design and training of binary neurons and binary neural networks with error correcting codes

Abstract

A data processing system having a binary neural network architecture for receiving binary network inputs and propagating signals via a plurality of binary processing nodes in dependence on the network inputs to form network outputs in dependence on respective binary weights, the data processing system being configured to train each node of the plurality of binary processing nodes by implementing the node function as an error correcting code (e.g. an rth order Reed-Muller code, such as a 1st order Reed-Muller code or a coset of a 1st order Reed-Muller code) function to identify, by channel decoding (e.g. a fast Walsh-Hadamard transform algorithm), a set of binary weights for a given input to the node that minimises any error between an output of the node formed in dependence on current binary weights of the node and a preferred output from the node, and updating the weights of the node to the identified set of binary weights. Such training is performed without storage and / or use of any higher-precision weights or other elements.

Description

Technical Field

[0001] This invention relates to deep neural networks, and more particularly to the design and training of binary neurons and binary neural networks. Background Technology

[0002] Deep Neural Networks (DNNs) are computational systems inspired by the biological neural networks that make up the biological brain. DNNs "learn" to perform tasks by considering examples that are typically not programmed using any task-specific rules. For example, in image recognition, they can learn to recognize images containing cars by analyzing example images that are manually labeled "car" or "no car" and using the results to identify cars in other images. They do this without any prior knowledge about cars. Alternatively, they automatically generate recognition features based on the learning material they process. Figure 1 The general scheme for this process is shown.

[0003] DNNs are based on a collection of connection units, or nodes, called artificial neurons, which loosely model neurons in the biological brain. Each connection, like a synapse in the biological brain, transmits signals from one artificial neuron to another. The receiving artificial neuron can process the signal and then activate other artificial neurons connected to it. Typically, artificial neurons are clustered into layers, where different layers can perform different kinds of transformations on their inputs.

[0004] In common DNN implementations, the signals at the connections between artificial neurons are real numbers, and the output of each artificial neuron is calculated using some nonlinear function of the sum of its inputs. The connections between artificial neurons are called "edges." Edges typically have weights that adjust as learning progresses. These weights increase or decrease the strength of the signal at the connection. Artificial neurons can have thresholds, such that a signal is only sent when the accumulated signal exceeds that threshold.

[0005] Single artificial neuron, such as Figure 2 As shown. Input (x1, x2, ..., x...) n The input signal (x) can be either the input signal or the output of the previous neuron layer. Each input x i First multiply by the weight w ij , where j represents the neuron exponent, and then the products are summed. In some cases, this sum is added to the bias w. 0j The net input is then passed through a non-linear activation function, and finally a single output y is produced. j .

[0006] DNNs are now ubiquitous, permeating every aspect of modern communication networks. Therefore, it's no surprise that DNNs are expected to play a crucial role in future network devices such as smartphones, sensors, and wearables. Especially in smartphones, DNNs already have a significant market presence with a wide range of applications, including image recognition, portrait photography, text prediction, user profiling, noise reduction, and camera enhancement.

[0007] However, its implementation on such resource-constrained devices is limited by high resource consumption (memory and energy). In standard DNN implementations, all inputs, outputs, weights, biases, and activation functions are treated as real numbers and are typically represented in the computer using floating-point arithmetic, meaning each number is approximated by a 64-bit sequence. In deep learning applications, DNNs with thousands of neurons can exist, where each neuron can have many inputs, resulting in millions of such floating-point numbers.

[0008] For this reason, recent research has focused on compressing DNNs using smaller bitwise operations (i.e., binary, ternary, or 8 bits). This approach essentially uses fewer bits to represent each number (input, output, weight) than floating-point numbers would use. However, reducing the number of bits negatively impacts accuracy. For the same number of neurons, each neuron is now less efficient at describing relationships, and this can introduce errors. On the other hand, the number of neurons can be increased, theoretically achieving the same accuracy as before. For example, in the worst case, 64 binary neurons can simulate floating-point neurons.

[0009] However, recent research suggests that this situation could be improved if binary neural networks can be designed appropriately.

[0010] Previous work has established that reducing computational precision can significantly improve the performance of DNNs without sacrificing accuracy. However, training low-precision DNNs has become a new bottleneck and is one of the key issues in the field of machine learning today.

[0011] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv and Y. Bengio's "Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations", Journal of Machine Learning Research, 18(187), pp. 1-30, 2017, and M. Rastegari, V. Ordonez, J. Redmon and A. Farhadi's "XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks", European Conference on Computer Vision, pp. 525-542, 2016, provide designs for binary neural networks (BNNs). Compared to DNNs with continuous weights, BNNs reduce memory usage by 32 times, achieve a tenfold reduction in energy consumption, and significantly reduce inference time. These results demonstrate the necessity of developing low-precision neural network architectures and training algorithms that achieve accuracy close to that achieved through their counterparts with continuous weights.

[0012] However, training neural networks (even single-layer neural networks) with low-precision weights is known to be an NP-hard optimization problem. In particular, due to the discretization of weights, the backpropagation algorithm used in continuous DNNs, which is based on calculating the gradient of the loss function, cannot be used efficiently. Furthermore, unstable gradients become problematic once the number of digits used to represent numbers is less than 8.

[0013] Therefore, most previous work on efficient training of low-precision neural networks was based on the use of gradient approximation, which required storing floating-point weights and low-precision weights, and performing some form of backpropagation using floating-point algorithms (thus giving up some discretization advantages).

[0014] To address this issue, these works maintain a full-precision value for each weight and then (i) binaryize that value, and (ii) update the full-precision value using a pass-through estimator for the gradients evaluated for the binaryized value.

[0015] The algorithm described in “Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM” by C. Leng, Z. Dou, H. Li, S. Zhu, and R. Jin, The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), pp. 3466-3473, also stores two versions of weights: low-precision weights and high-precision weights, and seeks to optimize a function that severely penalizes the distance between the two, thus forcing the high-precision weights to take values ​​very close to the low-precision weights.

[0016] Alternative methods based on Bayesian modeling are described in the following literature: D. Soudry, I. Hubara, R. Meir, “Expectation Backpropagation: parameter-free training of multilayer neural networks with continuous or discrete weights,” 27th International Conference on Neural Information Processing Systems (NIPS), 2014. In this approach, each weight is represented by a probability distribution updated during training. The BNN is then specified by sampling the weights from a given distribution. While the process of following these methods can generate BNNs during training, they still require operation on full-precision real numbers, making them unsuitable for use on battery-powered devices.

[0017] Another approach to training BNNs is through algorithms inspired by statistical physics. Methods in the following literature provide training algorithms that maintain and update integer “hidden weights” for each binary weight: C. Baldassi and A. Braunstein, “AMax-Sum algorithm for training discrete neural networks,” *Journal of Statistical Mechanics: Theory and Experiments*, August 2015; and C. Baldassi, “Generalization Learning in a Perceptron with Binary Synapses,” *Journal of Statistical Physics*, Vol. 136, No. 5, pp. 902-916, September 2009. It has been shown that these integer hidden weights can also have finite magnitudes. This property significantly reduces operational complexity, making these algorithms more attractive for device-level use. However, these algorithms are designed for specific types of networks and cannot be directly applied to more general and deeper architectures.

[0018] The work in the following literature provides an efficient algorithm based on belief propagation, but as with previous work, it applies to a single perceptron and it is unclear how to apply it to networks with multiple layers: C. Baldassi, A. Ingrosso, C. Lucibello, L. Saglietti, R. Zecchina, “Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses”, Physical Review Letters, Vol. 115, No. 12, September 2015.

[0019] An alternative approach to overcoming the lack of gradient information is to treat training as a combinatorial optimization problem and apply known solutions to such problems. For example, evolutionary algorithms (EAs) have been proposed as training methods for general neural networks, as described in G. Morse and KOStanley, “Simple Evolutionary Optimization Can Rival Stochastic Gradient Descent in Neural Networks,” The Genetic and Evolutionary Computation Conference (GECCO), July 2016. In particular, evolutionary algorithms have been proposed as training methods for low-precision DNNs, as described in R. Ito and T. Saito, “Dynamic binary neural networks and evolutionary learning,” The International Joint Conference on Neural Networks (IJCNN), July 2010.

[0020] However, EA suffers from performance and scalability issues. First, even considering only low-precision weights, the number of weights stored in memory multiplies by the population size, and therefore, even for binary weights and a modest population size of 100, the memory footprint is significantly larger than storing decimal weights with 16, 32, or 64-bit floating-point representations. Second, a single forward propagation during training must be performed on all members of the population. While this can be done in parallel, having as many parallel processors as the population size is impractical for training on mobile devices. Finally, it is well known that the population size should increase with the number of weights (the dimension of the optimization parameter space), making scaling large neural networks problematic.

[0021] Therefore, no known scalable and efficient algorithm exists in the literature for training low-precision DNNs solely on smartphones. The existing approach involves performing the computationally demanding training phase using floating-point operations in the cloud, where resource constraints are less stringent. A low-precision version of the DNN is then provided on the smartphone to perform the less demanding inference (feedforward) phase on intangible data. However, this process can only produce fixed, pre-trained low-precision DNNs, which is the main limitation of the existing technology.

[0022] Low-precision neural networks can be implemented in software or hardware. Binary neural networks, in particular, are known to have very efficient implementations based on bitwise operations (in software) or digital logic circuits (such as NOR or XNOR (in hardware)).

[0023] WO 1992012497 A1 and US 10089577 B2 describe the use of specific hardware designs for neurons used to implement BNNs. WO 1992012497 A1 relates to a neural network used as an associative memory, and implements each neuron as a "NOR" ("NOR") logic function such that its output is active only when all of its inputs are inactive. In US10089577 B2, an XNOR (Exclusive NOR) is performed between the input signal of each neuron and the corresponding weight, and then the number of 1s obtained is compared with a threshold to determine the output (activation function). Furthermore, no training method is proposed.

[0024] The goal is to design binary neurons and neural networks to fully binary DNNs, thereby enabling artificial intelligence (AI) in smartphones to have compact AI chipsets, reduced power consumption, and accelerated training. Summary of the Invention

[0025] According to a first aspect, a data processing system with a neural network architecture is provided. This neural network architecture receives binary network input and propagates signals via multiple processing nodes based on the network input to form a network output according to corresponding binary weights. The data processing system is configured to train nodes by: implementing an error correction function to identify binary weight reconfigurations for a given input that minimize any error between the node's output formed according to the node's current weights and a preferred output from the node; and updating the node's weights to the identified weights. This training can be performed in the binary domain without storing and / or using any weights or other elements with higher computational precision.

[0026] Therefore, this method can train "binary domain" neurons and / or neural networks without storing and / or using any weights or other elements with higher computational precision. Depending on the needs of mass production, this method can be implemented on AI chipsets in a simple, stand-alone, and modular manner. This solution may suffer a small loss in accuracy but offers significant advantages in simplicity, space, thermal properties, memory requirements, and energy consumption, and can be used for federated learning, user configuration, privacy, security, and optimal control. Inspired by the characteristics of channel coding theory, this method is closer to the exact form of the Boolean function to be used.

[0027] The node can be configured to receive node inputs based on network inputs and form a node output that is binary and functionally equal to the sum of the binary fields of: (i) a first weight, (ii) the sum of the binary fields of the product of each bit in the node input and (b) the corresponding weight in the second weighting, and (iii) a value indicating whether all bits of the node input are non-zero. This provides an efficient way to implement flexible computation nodes.

[0028] One of the nodes can be configured to form the node output of a coset functionally equal to a first-order Reed-Muller code, where the kernel of the coset takes the bits of the node input as input. This allows the node to efficiently process binary data.

[0029] This node can be implemented as a group of logic circuits, which is configured such that:

[0030] (i) Each bit of the node input is logically ANDed with the corresponding weight to form the corresponding first intermediate value;

[0031] (ii) All bits of the node input are combined using a logical AND function to form a second intermediate value; and

[0032] (iii) The other weight, the first intermediate value, and the second intermediate value are combined and subjected to a logical XOR function to form the node output. This facilitates efficient processing of binary data.

[0033] The network can include multiple blocks, each block comprising at least three logic circuits as described above. These logic circuits are configured such that a first and a second logic circuit receive block inputs based on network inputs and form a first intermediate block value and a second intermediate block value, and a third logic circuit receives the intermediate block values ​​as inputs and forms an output from that block. This form of block can efficiently form components of more complex computational structures. The interconnections between blocks can depend on the desired Boolean function to be implemented.

[0034] The first logic circuit can be configured to apply weights as weights to a binary perceptron, and the second logic circuit can have fixed weights. The fixed weights can be zero. This allows the first logic circuit to efficiently process the input data.

[0035] The third logic circuit can receive (i) a single instance of the first intermediate block value and (ii) multiple instances of the second intermediate block value as input. This allows the third logic circuit to process data efficiently.

[0036] The third logic circuit can be configured to apply fixed weights to all inputs of the third logic circuit except for one instance of the second intermediate block value. This allows the third logic circuit to process data efficiently.

[0037] The third logic circuit can be configured to apply a weight equal to the sum of the weights of the binary fields of the second intermediate block value to the instance. This allows the third logic circuit to process data efficiently.

[0038] The data processing system may include multiple subsystems, each including the system described above, wherein the system is configured to provide the output of at least one subsystem as the input of at least one subsystem in other subsystems. The connections between the multiple subsystems can be configured according to the desired Boolean function to be implemented.

[0039] The system can be configured to adjust the weights for nodes based on the expected set of node outputs for the given set of node inputs through the following steps: forming a set of values ​​for each node input indicating that the expected node output is (i) zero, (ii) one, or (iii) indifferent; identifying the i-th row of the set that best matches the value in the Hadamard matrix; and adjusting the weights for the nodes based on this identification. Any procedure used for decoding error correction codes can be used to adjust the weights for the nodes. This allows the system to learn effectively in response to input data.

[0040] The system can be configured to form a set of values ​​for each node input, indicating whether the corresponding node output, after weight adjustments, matches the expected node output. This can be performed using any procedure used for decoding error correction codes. This can provide feedback to the learning process.

[0041] The system can be configured to adjust the weights of nodes based on the expected set of node outputs anticipated for the set of node inputs, using a Fast Walsh-Hadamard function as input. This provides feedback to the learning process.

[0042] According to a second aspect, a communication terminal is provided, comprising a sensor and a data processing system as described above, the terminal being configured to sense data by means of the sensor to form a network input. The sensor may be a camera.

[0043] The terminal can be configured to use the data processing system to perform error correction on data received through the communication link.

[0044] According to a third aspect, a computer program is provided implemented in a system having a neural network architecture for receiving binary network input and propagating signals via multiple processing nodes based on the network input to form a network output according to corresponding binary weights. When executed by a computer, the computer program causes the computer to perform a method comprising training nodes by: implementing an error correction function to identify a binary weight reconfiguration for a given input that minimizes any error between the node's output formed according to the node's current weights and a preferred output from the node; and updating the node's weights to the identified weights. The computer program may be provided on a non-transitory computer-readable storage medium. Attached Figure Description

[0045] The invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0046] Figure 1 The diagram shows the stages of neural network training.

[0047] Figure 2 An artificial neuron is shown.

[0048] Figure 3 A flow graph of the fast Walsh-Hadamard transform is shown.

[0049] Figure 4 An example circuit for implementing the activation function of a binary domain neuron with three inputs is shown.

[0050] Figure 5 The symbol for a binary domain neuron indicating M inputs is shown.

[0051] Figure 6 (a) shows the circuit of a binary domain perceptron.

[0052] Figure 6 (b) shows Figure 6 The symbol for the circuit in (a).

[0053] Figure 7 (a) shows a circuit that can implement Boolean monomials, Boolean terms, or weighted OR gates.

[0054] Figure 7 (b) shows the symbol indicating a Boolean monomial.

[0055] Figure 7 (c) shows the symbol indicating a Boolean item.

[0056] Figure 8The circuit shown can implement any Boolean function in the algebraic normal form (ANF). The number of neurons in each layer, from layer 4 to the output neuron (a total of n layers), is indicated at the top.

[0057] Figure 9 (a) shows the symbol for an OR gate (“weighted OR”) used to indicate an input to which an OR operation can be selectively performed.

[0058] Figure 9 (b) shows the OR gate (equivalent to α) in which the operation is applied considering all inputs. i The sign of the weighted OR of i=1, i=1,…,M.

[0059] Figure 10 The circuit for DNF that can implement any Boolean function is shown. From layer 4 to the output neuron (a total of n layers of OR gates), the number of OR gates in each layer is indicated at the top.

[0060] Figure 11 An example of a data processing system is shown. Detailed Implementation

[0061] This invention proposes a pure binary neural network by introducing an architecture based on binary field operations, and transforms the training problem into a communication channel decoding problem.

[0062] Artificial neurons have many inputs and weights, and return an output. They are typically viewed as functions parameterized by the weights, taking the neuron's inputs as variables to return the output. In the context of BNNs, particular attention is paid to the development of binary neurons, where the inputs, weights, and output are all binary numbers. Interestingly, this structure of binary neurons conforms to Boolean functions.

[0063] Let m be the number of variables, and It is a binary m-tuple. A Boolean function is any function f(x) = f(x1, x2, ..., xn) that takes values ​​from F2. m It can be defined by all 2. m The truth table specifies the value of f at each input combination. Table 1 provides an example of the truth table for the binary sum function (XOR function).

[0064]

[0065] Table 1: Example of a truth table for the Boolean function (XOR)

[0066] The Boolean function f consists of a length of 2 m The binary vector is specified, therefore there are a total of A function of m variables. Since for x... i ∈F2, Therefore, any Boolean function is a binary combination of basis functions:

[0067] {1, x i x i x j，j≠i x i x j，j≠i x k，k≠i，j ,…,x1x2…x m} (1)

[0068] Boolean functions provide a natural approach to modeling binary neurons, particularly from the perspective that a neuron is like a feedforward function that produces an output based on a given input. From this perspective, training a binary neuron is equivalent to fitting a Boolean function, i.e., finding a function that best approximates the target output given a specific input. As illustrated below, this formulation allows the application of well-known tools to solve function fitting, and thus also neuron training.

[0069] Boolean functions of expression error correction codes have emerged as specific and suitable candidates for implementation of binary neurons due to their key advantage: as codes, they have built-in structures that directly transform function fitting into decoding tasks.

[0070] This invention proposes binary domain neurons that can be conveniently trained using channel coding theory, as well as several circuits specifically constructed using this type of neuron, which are capable of solving any learning problem with discrete inputs and discrete outputs.

[0071] The latter is possible thanks to (i) the ability of the proposed neuron activation function to generate a nonlinear function of its input, and (ii) the fact that any Boolean function can be implemented by the network by conveniently connecting a sufficient number of such neurons. This means that the system can learn any combination of input and label.

[0072] Furthermore, for a fixed input, the function implemented by the neurons is linear in the neuron parameters, which allows the weight vector of each neuron to be viewed as a specific error correction code, and thus the neuron training problem can be formulated as a channel decoding problem.

[0073] Details of an exemplary binary domain neuron and several circuits built from it, capable of solving any binary classification problem with binary inputs, will now be described. Since any M-label classification problem can be formulated as M one-against-all binary classifications, the circuits given here can be extended to construct multi-label classifiers with binary inputs.

[0074] In this example, the weight vector of each binary domain neuron is a Reed-Muller code, a type of error correction code that can be easily defined using a Boolean function. For m>0 and 0≤r≤m, the r-order Reed-Muller code denoted as R(r,m) is defined according to the Boolean function f(x1,x2,…,x…). m The obtained n=2 m All vectors The set of is a Boolean polynomial of at most degree r.

[0075] An r-order Reed-Muller code consists of all linear combinations of column vectors corresponding to basis functions of at most r degrees in equation (1). There exists... Such basis vectors are linearly independent. Therefore, the dimension of the code is d and corresponds to a set of basis vectors of length 2. m 2 d A Boolean function for a truth table consisting of columns.

[0076] The r-order Reid-Müller code has a minimum distance of 2. m-r Higher code order increases codebook size and approximate accuracy, but reduces minimum distance and makes neurons less robust to misfitting data, which is an important requirement.

[0077] A particularly interesting case is the first-order Reed-Müller code R(1,m), which has the largest minimum distance among Reed-Müller codes. It consists of 2... m+1 It consists of several linear Boolean functions, given by the following formula:

[0078]

[0079] in, It is an input variable, and The parameter is the parameter specifying f. First-order Reed-Muller codes not only possess the largest minimum distance, but they are also computationally efficient, thanks to their Hadamard structure codebook which allows for computationally efficient decoding. However, it is worth noting that first-order Reed-Muller codes are purely linear, see equation (2), and conceptually, artificial neurons are required to be able to express both linear and nonlinear functions. Fortunately, the advantages of first-order Reed-Muller codes can be preserved while simultaneously satisfying the requirement of nonlinear expressibility. This is achieved by using the coset of the first-order Reed-Muller code CR(1,m), defined by the following equation:

[0080]

[0081] The right-hand product is the cogroup nucleus, which provides the neuron with the ability to express nonlinear components. Clearly, if... The kernel is 1 if the kernel is true and 0 otherwise. The truth table of CR(1,m) is obtained directly from the truth table of R(1,m) by taking only the two's complement of the last row. As a result, the Hadamard structure codebook can be used as easily as in R(1,m) for efficient decoding of CR(1,m).

[0082] Following this line of thought, training neurons is equivalent to finding the optimal training variable *w* that corresponds to the best realizable function, based on a given optimization criterion. This problem is similar to a decoding problem, where the optimization criterion could be, for example, minimizing the error between the expected truth table and the realizable function, and maximizing the accuracy for the input training dataset.

[0083] Table 2 shows examples of implementable functions for CR(1,2). This represents a decodeable and error-free 2 m +1 =A set of 8 implementable functions. Generally, there exists... There are 100 possible functions, which means that any truth table function that does not belong to the set in Table 2 has one or more errors for a particular x.

[0084]

[0085] Table 2: Examples of implementable functions for m = 2 binary inputs

[0086] An example of a method for decoding neurons will now be described.

[0087] Given the target truth table, the optimal training variable w can be found using the Hadamard structure codebook of the CR(1,m) code described in equation (3) with its Boolean function. This is achieved by using the Walsh-Hadamard transform.

[0088] The Walsh matrix is ​​a matrix with dimension 2 for n = {1, 2, ...}. n A specific square matrix. For each item h of the matrix. ik h ik ∈{-1,1} and all rows and columns are orthogonal, i.e., the dot product is zero. The natural-order Walsh matrix is ​​called the Hadamard matrix, which is given by the recursive formula:

[0089]

[0090] The Hadamard matrix provides a method for finding the training parameters w that minimize the distance to the desired truth table. For this purpose, the target truth table is defined by y. Next, based on y, a modified version can be obtained through the following transformation.

[0091]

[0092] Performing this modification makes the realizable functions shown in Table 2 represented by rows (or columns) of a Hadamard matrix. Note that y i The last case, -1, represents erasure, i.e., an input vector x with no target value. As a final step, to account for the nonlinear part of the neuron function, The sign of the last term is reversed. Then, the optimal row i of the Hadamard matrix is... * It was found to be i * =argmax|d|, where, This shows the distance of each row of H from the target vector. How close are the vectors? Then, if max(d) > max(-d), the optimal bias w0 of the neuron is 0; otherwise, the optimal bias w0 of the neuron is 1. Furthermore, the weights of the neuron (w1, w2, ..., w...) m Simply by integer i * The binary representation is given.

[0093] Previous methods suffer from high memory requirements due to the need to construct a full Hadamard matrix and the computational demands. The Fast Walsh-Hadamard Transform avoids the construction of the Hadamard matrix and reduces O(m²) computation time. m This addresses these two drawbacks by reducing the complexity of [the technology / mechanism].

[0094] In short, such as Figure 3 As shown, the Fast Walsh-Hadamard Transform works as a decoder that, given a target truth table, provides the optimal training variable w and also indicates bits where decoding errors exist (i.e., the implemented function differs from the target truth table).

[0095] An example of a circuit that implements a binary domain neuron and a perceptron will now be described.

[0096] The neuron activation function in equation (3) can be implemented in hardware with AND and OR logic gates.

[0097] Figure 4 An example of a possible implementation for M=3 inputs is shown.

[0098] In the following text, neurons with M inputs having the activation function given by equation (3) will be used... Figure 5 The symbols shown represent the weights w1,…,w. M The specific value can be indicated inside the symbol, and the bias w0 can be indicated at the top of the symbol. If only the Π symbol appears, the weights are not fixed and can be obtained during training. Symbol 1 inside the neuron. M or 0 M Indicator vectors (w1,…,w) M) is a vector consisting entirely of 1s or 0s. In both cases, the bias is w0 = 0.

[0099] Small networks can be constructed specifically using neurons with activation functions given by equation (3) to have circuits that implement the function given by equation (2), i.e., a linear combination of inputs plus biases. This function is called a binary perceptron.

[0100] Figure 6 (a) shows the circuit of a binary domain perceptron that implements the function in equation (2).

[0101] Figure 6 (a) illustrates how to combine three binary domain neurons 601, 602, and 603 to obtain a binary perceptron. The network has two layers, with two neurons 601 and 602 in the first layer and one neuron 603 in the second layer. To implement the function in equation (2) at the network output, the weights of the first neuron 601 in the first layer are taken as perceptron parameter values, while the weights of the second neuron 602 are all taken as 0. Note that this gives all input variables x1, ..., x at the output of the second neuron. M The product of the input variables is calculated. The output of the first neuron 601 is connected to the first input of the second neuron 603, while all other inputs are connected to the output of the second neuron 602. The weight multiplied by the output of the first neuron 601 is 1, while the weight multiplied by the output of the second neuron 602 is 0 except for one of its values, β. The value of β is chosen such that at the output of the last neuron 603, the product of all input variables cancels out. To achieve this, β takes the following values:

[0102]

[0103] Figure 6 (b) shows the representation Figure 6 (a) The symbol of the network shown.

[0104] An example circuit implementing Boolean monomials will now be described. A Boolean monomial is an input variable x1,…,x M The product of subsets of. Any monomial μ k It can be written as:

[0105]

[0106] Where, if variable x m Appearing in monomial μ k In the middle, then Otherwise Note that there exists K=2. M One reason This monomial is given by all possible values ​​of .

[0107] Figure 7 (a) shows a circuit consisting of M perceptrons and an output neuron, which can implement any Boolean monomial. Since the perceptrons can compute any linear function of their inputs, in order to obtain the monomial μ... k Choose the parameters for each perceptron i (i = 1, ..., M) to... When x is given i and in The value is given as 1. The weight of the last neuron is set to 0. M This ensures that the last neuron only calculates the product of the perceptron outputs.

[0108] Figure 7 (b) illustrates the selection of neuron coefficients to produce any monomial μ at the output. k When, in the remaining part, the meaning used to represent Figure 7 (a) The symbol of the network shown.

[0109] An example of a circuit that implements a Boolean function in polynomial algebraic normal form (ANF) will now be described.

[0110] Algebraic Normal Form (ANF) is a Boolean expression composed of the XOR operation of Boolean monomials. For certain parameter values ​​a0,…,a that depend on f(x),… 12…M The ANF of the Boolean function f(x) can be written as:

[0111]

[0112] Figure 8 A network constructed using binary domain neurons is shown, which can compute the ANF at most M times. Since any Boolean function can be written in ANF (see O'Donnell R., "Analysis of boolean functions", Cambridge University Press, June 5, 2014), this network can implement any Boolean function with M inputs indicated at 801.

[0113] The first three layers of the network are arranged such that all possible monomials μ are generated at the output of the third layer. k k = 1, ..., 2 M There exists an M that produces 0 at its output. (n-1)Another monomial, whose sole function is to have an output for each neuron from the fourth layer to the last layer that takes a value of 0 in the previous layer and can therefore be connected to the last input of each neuron. This allows the product terms in equation (3) to be canceled, and thus has a network that (starting from the fourth layer) only computes the XOR of all its inputs to produce a unique value at the output. Therefore, in order to all 2 M The terms are added together, and it is ensured that there is at least one neuron with an output of 0 in the previous layer in all layers, and the parameter n satisfies (M-1)M (n-1) >2 M .

[0114] In this network, the first four layers must generate 2 M Each layer has a singlet, so the parameters of all neurons in these layers are fixed. Similarly, starting from the fifth layer, all inputs are summed simply by XOR, so the coefficients of all neurons in these layers are set to 1, and the biases are all set to 0. Therefore, the only free parameter is the weights of the neurons in the fourth layer, which are weights on their input μ. k The monomial at a certain position should take the value 1 when it appears in f(x), otherwise it should take the value 0.

[0115] An example of a circuit that implements a Boolean term will now be described.

[0116] A literal is an inverted form. Or not negate (x) i The terms are Boolean variables, and the Boolean terms are products of literals. Specifically, a term in which all M variables appear is called a full term, and it can be represented as:

[0117]

[0118] Where, if x m In γ k If an item is not negated, then... otherwise Note that there exists K=2. M One reason All possible values ​​of such a term are given.

[0119] A circuit with M perceptrons and one output neuron, used to generate arbitrary monomials, can also be used to generate any total term. For this purpose, each input variable x... iEach input variable is assigned to a perceptron, and the weight of each perceptron is set to 0, except for one of the assigned input variables, whose perceptron weight is set to 1. If a variable appears in a term without being inverted, the bias in the corresponding perceptron is set to 0; otherwise, the bias in the corresponding perceptron is set to 1. Finally, the last neuron only calculates the product of all the outputs of the perceptrons, so its bias and all weights are set to 0.

[0120] Figure 7 (c) shows the symbol used below to indicate the Boolean whole item.

[0121] The circuit that implements Boolean functions in Disjunctive Normal Form (DNF) will now be described.

[0122] Any Boolean function can be written as the disjunction (OR) of all its inputs. This standard expression for a Boolean function is called Total Disjunctive Normal Form (DNF), and its general expression can be written as:

[0123] f(x) = b1γ 1 V…Vb K γ K (10)

[0124] Where, if the corresponding term γ k If it appears in the entire DNF of f(x), then the coefficient b k k = 1, ..., 2 M The value is 1; otherwise, the coefficient b k k = 1, ..., 2 M The value is 0.

[0125] To build a circuit that implements any full DNF using only binary domain neurons, it is necessary to specifically use this neuron to build the OR gate. This can also be achieved through... Figure 7 Implement this using the circuit shown in (a). Consider a function that calculates the OR between some of its inputs. The general expression for this function is:

[0126] g(x) = α1x1V…Vα M x M (11)

[0127] Where, if variable x i If it appears in an OR operation, then α i =1, otherwise α i = 0. The operation calculated by g(x) is called the weighted OR below. Applying De Morgan's law, the function calculated by g(x) can be rewritten as:

[0128]

[0129] The expression in the last line of equation (12) can be derived from... Figure 7 The network implementation shown in (a) allows for convenient selection of the weights for each neuron. Figure 9 The symbols shown in (a) are used to indicate parameters α1, ..., α M Some choices compute a network of weighted ORs of its M inputs, and Figure 9 The symbols in (b) are used to indicate a "standard" OR operation for M input variables, or equivalently to indicate the operation in α. i Weighted OR in the case of i=1, i=1, ..., M.

[0130] For practical reasons, it may be more efficient to implement OR gates directly in the hardware using actual OR gates, rather than combining many neurons that themselves require many basic Boolean gates.

[0131] Figure 10 This illustrates a full DNF network specifically constructed using binary domain neurons, capable of computing any full DNF with M input variables. Since any Boolean function can be written in full DNF, this network can implement any Boolean function with M inputs.

[0132] Similar to computing any ANF network, Figure 10 The first three layers of the network are used to generate all 2^M variables. M There are 10 possible terms. The terms that will make up the DNF are selected using parameters from a weighted OR gate constructed through the next three layers. After these, the network is simply a cascade of OR gates that compute the disjunctions of all selected terms.

[0133] Unlike ANF, in ANF the number of monomials generated is M. n M should be satisfied n >2 M +M (n-1) To propagate zero, in this case, the number of monomials generated should satisfy M. n >2 M This is because the tree structure previously generated using neurons has been replaced with OR (or weighted OR) gates. Consequently, the depth of the tree has been multiplied by 3, and since there are n layers of OR gates, 3n layers of neurons are needed.

[0134] Figure 11A schematic diagram of a data processing system 1100 configured to implement the network and its associated components described above is shown. The system may include a processor 1101 and a non-volatile memory 1102. The system may include more than one processor and more than one memory. The memory may store data executable by the processor. The processor may be configured to operate according to a computer program stored in non-transitory form on a machine-readable storage medium. The computer program may store instructions for causing the processor to perform its methods in the manner described herein.

[0135] In one particular implementation, the aforementioned data processing system can be implemented as part of a communication terminal such as a smartphone. The smartphone may include sensors, such as a camera, from which sensed data is provided to form network input. The terminal can be configured to use the data processing system to perform error correction on data received via a communication link. This data processing system can be used for applications including image recognition, portrait mode photography, text prediction, user profiling, noise reduction, and camera enhancement.

[0136] The system can be implemented as multiple subsystems, each including a network as described in one of the above embodiments. In this arrangement, the output of the network of at least one subsystem can be provided as input to the network of at least one other subsystem. The connections between the multiple subsystems can be configured according to the desired Boolean function to be implemented.

[0137] The method of this invention trains "binary domain" neurons and / or neural networks that do not utilize continuous elements. This is made possible by the special design of the neurons. The invention models the weight vector of each neuron as an error correction code and formulates the neuron training problem as a channel decoding problem. The binary domain artificial neurons have parameters that implement the error correction code. Methods for decoding this type of neuron and decoding BNNs are also described, along with logic circuits that can implement such neurons and BNNs based on this type of neuron, and their corresponding implementations in a virtual environment.

[0138] Compared with existing technologies, the proposed solution has many advantages.

[0139] To meet the needs of mass production, this method can be implemented on AI chipsets in a simple, independent, and modular manner.

[0140] This solution suffers a minor loss in accuracy but offers significant advantages in simplicity, space, thermal properties, memory requirements, and energy consumption, and can be used for federated learning, user profiles, privacy, security, and optimal control.

[0141] Inspired by the characteristics of channel coding theory, this method is closer to the exact form of the Boolean function to be used.

[0142] The applicant has individually disclosed each individual feature described herein, as well as any combination of two or more such features, to a certain extent that such features or combinations can be implemented based on this specification as a whole, according to ordinary common sense known to those skilled in the art, regardless of whether such features or combinations of features solve any problem disclosed herein, and without limiting the scope of the claims. The applicant notes that aspects of the invention can consist of any such individual features or combinations of features. In view of the foregoing description, it will be apparent to those skilled in the art that various modifications can be made within the scope of the invention.

Claims

1. A data processing system having a neural network architecture, the neural network architecture being used to receive binary network input via a communication link and, based on the network input, propagate signals via a plurality of processing nodes to form a network output according to corresponding binary weights, the data processing system being configured to train nodes by: implementing an error correction function to identify, for a given input to the node, a binary weight reconfiguration that minimizes any error between the node's output formed according to the node's current weights and the desired output from the node; and updating the node's weights to the identified weights; wherein, The node is implemented as a group of logic circuits.

2. The data processing system according to claim 1, wherein, The node is configured to receive node inputs based on the network inputs and to form node outputs that are binary and functionally equal to the sum of the binary fields of the following three items: (i) a first weight; (ii) the sum of the binary fields of the product of (a) each bit of the node input and (b) a corresponding weight in the second weighting; and (iii) a value indicating whether all bits of the node input are non-zero.

3. The data processing system according to claim 2, wherein, One of the nodes is configured to form the node output of a coset functionally equal to a first-order Reed-Muller code, wherein the kernel of the coset takes the bits of the node input as input.

4. The data processing system according to claim 2, wherein, The logic circuit group is configured such that: (i) Each bit of the node input is logically ANDed with the corresponding weight to form the corresponding first intermediate value; (ii) All bits of the node input are combined using a logical AND function to form a second intermediate value; as well as (iii) The other weight, the first intermediate value, and the second intermediate value are subjected to a logical XOR function to form the node output.

5. The data processing system according to claim 4, wherein, The neural network architecture includes multiple blocks, each block including at least three logic circuits. The at least three logic circuits are configured such that a first logic circuit and a second logic circuit among the at least three logic circuits receive block inputs based on the network inputs and form a first intermediate block value and a second intermediate block value, and a third logic circuit among the at least three logic circuits receives the intermediate block values ​​as inputs and forms an output from the block.

6. The data processing system according to claim 5, wherein, The first logic circuit is configured to apply weights as weights for a binary perceptron, and the second logic circuit has fixed weights.

7. The data processing system according to claim 6, wherein, The fixed weight is zero.

8. The data processing system according to claim 6, wherein, The third logic circuit receives (i) a single instance of the first intermediate block value and (ii) multiple instances of the second intermediate block value as input.

9. The data processing system according to claim 8, wherein, The third logic circuit is configured to apply fixed weights to all inputs of the third logic circuit except for one instance of the second intermediate block value.

10. The data processing system according to claim 9, wherein, The third logic circuit is configured to apply a weight equal to the sum of the binary fields of the weights of the first logic circuit to the instance of the second intermediate block value.

11. A data processing system comprising multiple subsystems, each of the subsystems comprising the system according to any one of claims 5 to 10, wherein, The system is configured to provide the output of at least one subsystem in the subsystem as the input of at least one subsystem in other subsystems.

12. The data processing system according to claim 11, wherein, The connections between the multiple subsystems are configured according to the desired Boolean function to be implemented.

13. The data processing system of claim 11, wherein the system is configured to adjust the weights for nodes based on the expected node output set for the node input set by the following steps: Form a group of values, wherein for each node input, the group of values ​​represents whether the expected node output is (i) zero, (ii) one, or (iii) irrelevant; and Identify the i-th row of the group in the Hadamard matrix that best matches the stated value; and The weights for the nodes are adjusted based on this identification.

14. The data processing system of claim 13, wherein the system is configured to form a group of values, the group of values ​​representing, for each node input, whether the corresponding node output after the weights have been adjusted matches the expected node output.

15. The data processing system of claim 11, wherein the system is configured to adjust the weights of nodes based on the expected set of node outputs for the node input set by operating a Fast Walsh-Hadamard function, the Fast Walsh-Hadamard function taking the expected set of node outputs as input.

16. A communication terminal comprising a sensor and a data processing system according to any one of claims 1-10, the terminal being configured to sense data by means of the sensor to form the network input.

17. The communication terminal according to claim 16, wherein, The sensor is a camera.

18. The communication terminal of claim 16, wherein the terminal is configured to use the data processing system to perform error correction on data received through the communication link.

19. A computer program product implemented in a system having a neural network architecture for receiving binary network input received via a communication link and propagating signals through a plurality of processing nodes based on the network input to form a network output according to corresponding binary weights, the computer program product comprising computer instructions that, when executed by a computer, cause the computer to perform a method comprising training a node by: implementing an error correction function to identify a binary weight reconfiguration for a given input to the node that minimizes any error between the node's output formed according to the node's current weights and the node's desired output; and updating the node's weights to the identified weights; wherein, The node is implemented as a group of logic circuits.

20. A computer-readable storage medium for storing a computer program, which, when executed by a computer, causes the computer to perform a method implemented in a system having a neural network architecture for receiving binary network input received via a communication link and propagating signals via a plurality of processing nodes according to the network input to form a network output according to corresponding binary weights, the method comprising training nodes by: implementing an error correction function to identify a binary weight reconfiguration for a given input to the node that minimizes any error between the node's output formed according to the node's current weights and the node's desired output; and updating the node's weights to the identified weights; wherein, The node is implemented as a group of logic circuits.

21. A data processing method using a neural network architecture, the neural network architecture being used to receive binary network input received via a communication link, and to propagate signals via a plurality of processing nodes based on the network input to form a network output according to corresponding binary weights, the data processing method comprising training nodes by: implementing an error correction function to identify a binary weight reconfiguration for a given input to the node that minimizes any error between the output of the node formed according to the node's current weights and the desired output from the node; and updating the weights of the node to the identified weights; wherein, The node is implemented as a group of logic circuits.

22. A data processing system comprising a processor and a memory, the memory for storing instructions, the processor, when executing the instructions, for receiving binary network input received via a communication link, and for propagating signals via a plurality of processing nodes based on the network input to form a network output according to corresponding binary weights, the data processing system being configured to train nodes by: implementing an error correction function to identify a binary weight reconfiguration for a given input to the node that minimizes any error between the node's output formed according to the node's current weights and the desired output from the node; and updating the node's weights to the identified weights; wherein, The node is implemented as a group of logic circuits.

Images