Deep neural network with multiple layers formed by multi-terminal logic gates

Abstract

A deep neural network circuit having a plurality of layers formed by a multi-terminal logic gate is provided. In one aspect, a neural network circuit includes a plurality of logic gates disposed into a plurality of layers and a plurality of logic connectors disposed between each pair of adjacent layers. Each of the logic connectors connects an output of the first logic gate to an input of the second logic gate, and each of the logic connectors has one of a plurality of different logic connector states. The neural network circuitry is configured to be trained to implement a function by finding a set of logical connector states of logical connectors that cause the neural network circuitry to implement the function.

Description

Cross-reference to priority application This application claims priority to U.S. Provisional Patent Application No. 63 / 364,405, filed on May 9, 2022, entitled “DEEP NEURAL NETWORK WITH MULTIPLELAYERS FORMED OF MULTI-TERMINAL LOGIC GATES,” the disclosure of which is incorporated herein by reference in its entirety and for all purposes. Technical Field The present disclosure generally relates to neural networks. More specifically, the present disclosure relates to deep neural networks implemented using multiple layers formed by multi-terminal logic gates. Background Art Neural networks can be implemented on various types of hardware such as central processing units (CPUs) and field programmable gate arrays (FPGAs), and on specialized hardware designed for neural networks, such as distributed architectures like graphics processing units (GPUs) or tensor processing units (TPUs). Summary of the invention The innovations recited in the claims each encompass multiple aspects, no single aspect of which is solely responsible for its superior performance. Without limiting the scope of the claims, some of the salient features of the disclosure will now be briefly described. One innovative aspect is a neural network circuit comprising: a plurality of logic gates arranged into a plurality of layers, each of the logic gates having a plurality of inputs and having an output; and a plurality of logic connectors arranged between each pair of adjacent layers, each of the logic connectors determining a relationship of an output of a first logic gate to one of a plurality of inputs of a second logic gate, and each of the logic connectors having one of a plurality of different logic connector states, wherein the neural network circuit is configured to be trained to implement a function, the training being performed by finding a set of logic connector states of the logic connectors that enables the neural network circuit to implement the function. In some embodiments, the logic connector state includes a first state and a second state, wherein in the first state, the output of the first logic gate is connected to the input of the second logic gate via a NOT gate, and in the second state, the output of the first logic gate is connected to the input of the second logic gate via a short circuit. In some embodiments, the logic connector state further includes a third state in which the output of the first logic gate is connected to the input of the second logic gate via an open circuit. In some implementations, the logic gates and logic connectors are implemented in complementary metal oxide semiconductor (CMOS) technology. In some embodiments, the neural network circuit is formed on a single chip. In some embodiments, each of the logic connectors includes: an input; a shorting circuit connected to the input; an inverter arranged in parallel with the shorting circuit and connected to the input; an output; and at least one switch configured to connect one of the shorting circuit and the inverter to the output. In some implementations, each of the logic connectors further includes an open circuit connected to the input, wherein the at least one switch is configured to connect one of the short circuit, the inverter, and the open circuit to the output. In some embodiments, the at least one switch includes a first switch and a second switch connected in series, the first switch is configured to be electrically connected to one of a short circuit and an inverter, and the second switch is configured to operate in an open circuit or a short circuit state. In some embodiments, each of the logic connectors includes at least one of a short circuit or an inverter, and each of the logic connectors connects an output of a logic gate of a previous layer to an input of a logic gate of a current layer. In some implementations, each of the logic gates comprises a multi-terminal NOR gate. In some embodiments, the neural network further includes a training circuit configured to generate a set of logical connector states. In some embodiments, each of the logical connectors has a fixed one of the logical connector states. In some embodiments, each of the logic connectors includes a single one of a short circuit, an inverter, and an open circuit corresponding to a fixed one of the logic connector states. On the other hand is a method for calculating a function using a neural network circuit, comprising: providing a neural network circuit; and calculating the function for an input signal using the neural network, the neural network comprising: a plurality of logic gates arranged into a plurality of layers, each of the logic gates having a plurality of inputs and having an output; and a plurality of logic connectors including a logic connector group arranged between each pair of adjacent layers, each of the logic connectors having one of a plurality of different logic connector states, wherein the plurality of logic connectors are programmed to implement the function. In some implementations, the method further includes finding a set of logic connector states for the plurality of logic connectors such that the neural network circuit implements the function. In some implementations, the method further includes: generating a set of integer linear programming (ILP) problems based on the function; and solving the set of ILP problems to produce a set of logical connector states. In some embodiments, the method further includes: determining a set of inequalities describing the state of a logical connector; and linking the output from a previous layer to the input of a subsequent layer through the set of inequalities, wherein generation of the ILP problem is based on linking the output from the previous layer to the input of the subsequent layer through the set of inequalities. In some embodiments, the logic connector state includes a first state and a second state, wherein in the first state, the output of the first logic gate is connected to the input of the second logic gate via a NOT gate, and in the second state, the output of the first logic gate is connected to the input of the second logic gate via a short circuit. Yet another aspect is a single chip comprising: a plurality of logic gates arranged into a plurality of layers of a neural network circuit, each of the logic gates having a plurality of inputs and having an output; a plurality of logic connectors arranged between the logic gates of each pair of adjacent layers, each of the logic connectors determining a relationship between an output of a first logic gate to one of a plurality of inputs of a second logic gate, and each of the logic connectors having one of a plurality of different logic connector states; a plurality of input terminals; and at least one output terminal, wherein the chip is configured to calculate a function between the input terminals and the output terminals. In some embodiments, the chip is configured to compute the function for a given input provided to the input terminal in 10 clock cycles or less. For purposes of summarizing the present disclosure, certain aspects, advantages, and novel features of the innovations have been described herein. It should be understood that not all such advantages may be achieved according to any particular implementation. Thus, the innovations may be implemented or advanced in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other advantages as taught or suggested herein. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a circuit diagram illustrating a digital neural network according to aspects of the present disclosure. Figure 2 An implementation of a multi-terminal logic gate and a plurality of logic connectors connected thereto according to aspects of the present disclosure is shown. Figure 3 Another embodiment of a multi-terminal logic gate and a plurality of connected logic connectors connected thereto according to aspects of the present disclosure is shown. Figure 4 An implementation of a multi-terminal NOR gate according to aspects of the present disclosure is shown. Figure 5 An implementation of a multi-terminal NOR gate with thresholds according to aspects of the present disclosure is shown. Figure 6 is v / V evaluated according to aspects of the present disclosure with the th and th+1 switches closed DD . Figure 7 An exemplary neural network according to aspects of the present disclosure is shown. Fig. 8A and Figure 8B An implementation of a logical connector according to aspects of the present disclosure is shown. DETAILED DESCRIPTION The following description of certain embodiments presents various descriptions of specific embodiments. However, the innovations described herein may be implemented in a variety of different ways, such as defined and covered by the claims. In this specification, reference is made to the accompanying drawings in which the same reference numerals may represent identical or functionally similar elements. It should be understood that the elements shown in the drawings are not necessarily drawn to scale. In addition, it should be understood that certain embodiments may include more elements than shown in the drawings and / or a subset of the elements shown in the drawings. In addition, some embodiments may combine any suitable combination of features from two or more drawings. The titles provided herein are for convenience only and do not necessarily affect the scope or meaning of the claims. Although current processor architectures are adequate for relatively small data sets, for certain computationally intensive applications, they may not be able to keep up with the growth in the amount of information required in real time. For example, vehicles could benefit greatly from more computing power to decipher incoming sensor data and quickly make important real-time decisions. Aspects of the present disclosure relate to cost-effective and scalable computing systems and methods that can improve computing power for a variety of applications, including combating future gaps in vehicle data processing capabilities. Current edge processors based on central processing units (CPUs), graphics processing units (GPUs), and field programmable gate arrays (FPGAs) are unable to run algorithms on certain large sensor data streams (e.g., those in vehicles) sufficiently to meet certain computing goals. Data movement represents one of the most important limitations in certain computing architectures. This limitation is known as the von Neumann bottleneck, and it has two main implications: it limits computational throughput and requires considerable energy to move data between memory and processing devices. Machine learning and artificial intelligence (AI) techniques, such as neural networks, are increasingly being adopted to seek to improve processing capabilities for larger data sets and achieve automation. However, in many cases, these techniques are still limited by the von Neumann architecture on which they are based. Specifically, there are challenges in implementing ultra-low power electronic architectures for real-time evaluation / inference of neural networks. This remains an unsolved problem for neural networks running on traditional hardware, because the nonlinear (artificial neuron activation functions can be sigmoid functions, rectified linear units (ReLUs), etc.) and depth (several layers to be evaluated in sequence) characteristics of neural networks may involve a series of unavoidable calculations, and therefore include several unavoidable clock cycles, even in the case of state-of-the-art distributed architectures such as GPUs or tensor processing units (TPUs). This translates into intolerably high evaluation / inference times for many applications (e.g., autonomous vehicles). Despite efforts to propose mitigation of this problem, digital architectures (GPUs, CPUs, TPUs) may not be able to solve the underlying problem due to the physical and computational limitations of modern neural network designs. Aspects of the present disclosure relate to improved neural network designs, which may be referred to as memory-computing (MemComputing) neural networks (MEMC-NN) or more generally referred to as digital neural networks. The digital neural networks disclosed herein may be implemented by circuits. In certain embodiments, the disclosed neural networks may be easily integrated on a chip using only logic gates, switches, and / or selectors. Aspects of the present disclosure may partially or completely address some or all of the above limitations, thereby providing real-time neural network evaluation / inference while using negligible power. The neural network solutions provided herein may be virtually integrated at all levels in the system, from servers and clouds to smaller devices such as smart watches and glasses, or Internet of Things (IOT) and edge computing such as those described herein. Advantageously, aspects of the present disclosure have the potential to provide unprecedented AI processing capabilities that adhere to size, weight, power, environment, and cost targets. The digital neural network systems and methods disclosed herein may be applied to any other suitable application and / or meet any other suitable purpose. Digital Neural Network Design and Implementation Figure 11 is a circuit diagram illustrating a digital neural network 100 according to aspects of the present disclosure. The digital neural network 100 may be implemented as a circuit formed on a single chip. The digital neural network 100 is a deep neural network that includes a plurality of inputs 102, a plurality of layers 104, a plurality of outputs 106, and an optional training circuit 111. Each of the layers 104 includes a plurality of multi-terminal logic gates 108 (also referred to as "logic gates"). Each multi-terminal logic gate 108 may be logically associated with one or more of the multi-terminal logic gates 108 in the previous layer 104. For example, a logical association between two multi-terminal logic gates 108 may apply a logical operation to the output of the multi-terminal logic gate 108 in the previous layer 104, and then provide the result of the logical operation to the multi-terminal logic gate 108 in the current layer 104. These logical associations may be implemented by a logic connector 110 (also referred to as a "logic operator", "binary logic operator" or "logic circuit"). Each of the logic connectors 110 is configured to define a relationship between a pair of multi-terminal logic gates 108. The logic connectors 110 may not always physically connect the logic gates. For example, the logic connectors 110 may implement an open circuit between two multi-terminal logic gates 108. Figure 2 and Figure 3 As shown and discussed in greater detail below, the relationship between two multi-terminal logic gates 108 may include a logical NOT (eg, implemented by inverter 118), an open circuit 120, and / or a short circuit 122, although other relationships are possible. In some embodiments, the logic connector 110 can receive a single input and output a single output. In some embodiments, the topology of the connections implemented by the logic connector 110 defines a network layer of the digital neural network 100, such as fully connected, convolutional, pooling, etc. Optional training circuit 111 can be used to train digital neural network 100. Depending on the implementation, optional training circuit 111 can be included as part of the same chip as digital neural network 100, or can be implemented on a separate chip. Digital neural network 100 can be trained to implement the functions disclosed herein. After digital neural network 100 is trained, it can calculate the function for a given input in a single clock cycle, several clock cycles, single-digit clock cycles. For example, digital neural network 100 can be configured to calculate the function in less than two, three, four, five, six, seven, eight, nine, or ten clock cycles, or in several clock cycles, depending on the implementation.

[0043] Figure 2An embodiment of a multi-terminal logic gate 108 and a plurality of logic connectors 110 connected thereto according to aspects of the present disclosure is shown. In the illustrated embodiment, each of the logic connectors 110 includes an input 112, an output 114, a switch 116, and a plurality of selectable parallel paths between the input 112 and the output 114. Figure 2 In the top logic connection 110 shown in FIG. 1 , the parallel path includes an inverter 118 (e.g., a NOT gate), an open circuit 120, and a short circuit 122. The switch 116 is configured to define the state (also referred to as a “logic state”) of the logic connector 110 by defining a logical relationship between the input 112 and the output 114 of the logic connector 110 via one of the inverter 118, the open circuit 120, and the short circuit 122. Although Figure 2 The open circuit 120 is shown as a separate path that the switch 116 can select, but in some implementations, there may not be a physical path with the open circuit 120. For example, when the open circuit is selected, the switch 116 can be disconnected from each of the inverter 118 and the short circuit 122. Therefore, the logic operation applied by a given logic connector 110 can depend on the path (e.g., the inverter 118, the open circuit 120, and the short circuit 122) used to connect the input 112 and the output 114 of the logic connector 110. Some logic connectors 110 may include two parallel paths to select, such as an inverter and a short circuit. Figure 2 The two logic connectors 110 at the bottom of the illustrative embodiment show such logic connectors 110. According to the embodiment, the digital neural network 100 may include logic connectors 110 having substantially the same structure, or the digital neural network 100 may include a plurality of logic connectors 110 having different structures, for example Figure 2 Thus, in various embodiments, the logical connectors 110 used for a particular application may be of the same kind or of different kinds. In addition, Figure 2 In the example of , the multi-terminal logic gate 108 is implemented as a multi-terminal OR gate. However, aspects of the present disclosure are not limited in this regard, and the multi-terminal logic gate 108 may be implemented using other types of multi-terminal logic gates 108 depending on the implementation. As described in more detail herein, the states of the logic connectors 110 may be similar to the weights of a traditional neural network. In some other embodiments, each of the logic connectors 110 may be implemented using only a single path (e.g., one of the inverter 118, the open circuit 120, and the short circuit 122) without the switch 116, which represents the trained state of the logic connector 110 as described herein. Figure 3Another embodiment of a multi-terminal logic gate 108 and a plurality of connected logic connectors 110 connected thereto according to aspects of the present disclosure is shown. In addition to the logic connector 110 not including an open circuit 120 path, Figure 3 The implementation is similar to Figure 2 implementation method. Figure 2 and Figure 3 Exemplary implementations of the logic connector 110 are provided, but aspects of the present disclosure are not limited thereto, and the logic connector 110 may include a greater number of logic states that implement one or more different logic operations. The universality of digital neural networks As used herein, a neural network can be used as a universal machine if it can compute any computable function in the Turing sense. A computable function can be defined as a function that is computed by a Turing machine. Figures 1 to 3 The digital neural network 100 shown in FIG. 1 can be used to create a universal machine. In fact, the OR gate and the NOT gate (e.g., inverter) can form a complete basis in the sense that any Boolean function can be written as a set of OR and NOT gates. Although the digital neural network 100 described herein is not limited to Figures 1 to 3 The configuration shown in Figures 1 to 3 The digital neural network 100 is able to use any basis of Boolean functions or mixtures thereof to create layers that allow great flexibility. The versatility of the digital neural network 100 described herein can also be viewed from an operational point of view. Given N input terminals and M output terminals, there is a minimum number of layers (depending on the topology of the connections) that allow any possible function y=f(x) to be calculated for any input x of length N and a binary output y of length M. The states of the logic connectors 110 can be used to implement the function f. In other words, for each function f, there is at least one set of states of the logic connectors 110 that configures the digital neural network 100 to accurately evaluate the function f. Training digital neural networks As used herein, training a digital neural network 100 generally refers to finding a configuration of logic states for logic connectors 110 that is used to map a function f to the digital neural network 100. This training can be performed without analytical knowledge of the function f nor the output results y for all possible inputs x. Typically, a set of data output results y (labels) for a given input x (training set) is available for training. Training circuitry 111 can perform training to train the digital neural network 100. For conventional neural networks, gradient descent-based techniques may be used to find a configuration of weights that allows for a good representation of the function f. However, such techniques may not be applicable to training the digital neural network 100 described herein because there are an integer number of states available for the logic connector 110, and therefore gradients may not be defined for the logic connector 110. This is one reason why fully digital neural networks are difficult to implement. Typical training methods for neural networks encounter technical challenges for training digital circuits. In such typical training methods, continuous parameters are used for training. However, digital circuits can work based on binary values ​​instead of continuous parameters. Methods that work for continuous parameter values ​​are generally not well suited for digital circuits that use binary values. Training the digital neural network 100 disclosed herein may involve determining associations between logic gates 108 in different layers of the neural network 100 (e.g., as defined by the digital neural network 100). For example, this may involve determining whether to connect the logic gates 108 in different layers via a short circuit 122 or via an inverter 118. As another example, training may involve determining whether to connect the logic gates 108 in different layers via a short circuit 122 or via an inverter 118, or to not connect the two logic gates 108 (e.g., by connecting an open circuit 120 to the input of one of the logic gates 108). One technique disclosed herein for training a neural network 100 involves using integer linear programming (ILP) to solve a training problem. For example, for each input x, a set of linear inequalities can be defined to represent the propagation of the input x through the neural network 100, where the binary variables representing the states of the logical connectors 110 are unknowns. Thus, the training of the digital neural network 100 can be transformed into solving an ILP problem. However, solving ILP may not be a simple task. ILPS belongs to the combinatorial class of problems, also known as non-deterministic polynomial (NP) problems, which are notorious for their difficulty. The present disclosure relates to a virtual memory-computing machine (VMM) in which a non-temporary computer readable memory stores instructions that, when executed by one or more processors, emulate an innovative computing architecture and solve large ILP problems very efficiently. The VMM can be used to solve ILP problems related to training a digital neural network 100 and provide a configuration for the state of a logic connector 110 representing a function f. The VMM can be a software emulation of a self-organizing algebraic gate (SOAG), which can be used as a training solution for the digital neural network 100. International Patent Application No. PCT / US2022 / 053781 filed on December 22, 2022 and / or International Patent Application No. PCT / US2016 / 041909 filed on July 12, 2016 and published as International Publication No. WO 2017 / 011463 can be used to solve problems associated with training any neural network disclosed herein (e.g., ILP problems), and the disclosure of each of these patent applications is incorporated herein by reference in its entirety and for all purposes. For example, training circuit 111 can be implemented according to any suitable principles and advantages disclosed in these international patent applications. Any other suitable training method can be applied to the digital neural network disclosed herein. The digital neural network disclosed herein can be trained multiple times to calculate different functions. In some applications, the digital neural network disclosed herein can be trained once. This is useful for some applications, such as Internet of Things devices. Design NOR gate as activation function In some embodiments, NOR gates may be used to implement the multi-terminal logic gate 108 of the digital neural network 100. A multi-terminal NOR gate may be defined with thresholds having the following general relationship: In Equation 1, o is the output of the NOR gate, and i j is the jth input of the NOR gate, and th is the threshold value. In some embodiments, a NOR gate can replace an OR gate to implement the multi-terminal logic gate 108 because the NOR gate is easily implemented in complementary metal oxide semiconductor (CMOS) technology. However, aspects of the present disclosure are not limited to this. In order to obtain an OR gate, a NOT gate can be added after the NOR gate. However, when used in certain embodiments of the digital neural network 100 described herein, the OR gate and the NOR gate can be completely interchangeable because the input ends of the multi-terminal logic gate 108 can be connected to a logic connector 110, which can include a switch 116 that can select an inverter 118 or a short circuit 122 (for example, as Figure 3 shown). Figure 4 An implementation of a multi-terminal NOR gate 200 according to aspects of the present disclosure is shown. Figure 5 An embodiment of a multi-terminal NOR gate 250 with thresholds according to aspects of the present disclosure is shown. Each of the multi-terminal NOR gates 200 and 250 may be implemented in CMOS technology. refer to Figure 4 The multi-terminal NOR gate 200 includes a plurality of inputs i1, i2, ..., i n , output o, first power supply terminal V DD, a second power supply terminal GND and a plurality of transistors 202. The transistors 202 are arranged to implement a NOR function. exist Figure 5 In the embodiment, the multi-terminal NOR gate 250 includes a plurality of input switches i1, i2, ..., i n , output o, first power supply terminal V DD , two second power supply terminals GND, a pair of transistors 252, a plurality of first resistors R and a threshold resistor R th , each first resistor R is connected to a corresponding input switch i1, i2, ..., i n When the threshold resistor R th is set to R / 2 or any larger value, Figure 5 The function of the multi-terminal NOR gate 250 can be basically the same as Figure 4 The function of the multi-terminal NOR gate 200 is the same as that of the NOR gate 250. The working principle of the NOR gate 250 can be as follows. th The size of can be set as follows: if a number of switches strictly greater than th are closed, the output voltage o is set to 0, otherwise it is set to V DD Therefore, NOR gate 250 implements Equation 1, where the state of the switch is the input i j , and the output o is the voltage at node o. Even if it is fully implemented in CMOS technology, Figure 5 The multi-terminal NOR gate 250 can be implemented using a combination of digital and analog circuit elements. This can provide certain advantages over all-digital implementations in certain applications, such as those discussed herein. Continue to refer Figure 5 The multi-terminal NOR gate 250, the input is configured to open or close the input switches i1, i2, ..., i n . Input switches i1, i2, ..., i n is configured to realize input switches i1, i2, ..., i n The control terminal of the corresponding transistor (e.g., the gate of the CMOS component) is used to open or close. n It can be directly controlled by the output of other NOR gates 250 or other common CMOS-based logic gates. Figure 5 The implementation of can be one of the most compact implementations for a NOR gate with a threshold. For example, Figure 5 An implementation of may use as few as 3(n+1) transistors, where n is the number of inputs. A fully digital implementation may include more complex digital circuits that primarily perform a summation of the inputs and compare them to a threshold. According to an embodiment, the magnitude of the resistance of the resistor R th or ratio may be an important design consideration for achieving the desired functionality of NOR gate 250. For example, a standard CMOS transistor may have a cutoff gate voltage V DD / 2. In this case, the ratio can be configured as In order to make NOR gate 250 work properly. In one example, if the th switch is closed, the voltage v is When the th+1 switch is closed, Figure 6 is v / V evaluated according to aspects of the present disclosure with the th and th+1 switches closed DD . Here, th may be a parameter characterizing the multi-terminal NOR gate 250. th may be a threshold number of inputs that may be asserted for a state of logic 1 (the output of the multi-terminal NOR gate 250 is in a state of logic 1). For example, th may characterize the multi-terminal NOR gate 250 as defined in Equation 1: wherein, if at most the th input is 1 (e.g., at most the th switch is closed), the output is 1. Otherwise, if the th+1 or more inputs are 1 (e.g., the th+1 switch or more switches are closed), the output is 0. Therefore, the multi-terminal NOR gate 250 with parameter th may be a generalization of a multi-terminal NOR gate. A standard multi-terminal NOR gate may be obtained by setting th=0. However, when implemented in CMOS technology, there is resistance variability and the NOR gate 250 may not follow a perfect step function. Figure 6 As shown, if the threshold is small, th≤2, then the gap is about 10%, which is sufficient to cope with the variability mentioned above. This relationship may not depend on the number of inputs to the NOR gate 250. However, if the threshold is higher, the variability may be an inherent aspect of the NOR gate 250, and this may be addressed in different ways in the training of the digital neural network 100. ILP formulation of NOR gate According to aspects of the present disclosure, an ILP formulation may be used to train a digital neural network 100 including logic gates 108 and / or threshold logic gates 250. Starting from equation 1 above for output o, the ILP formulation of a multi-terminal NOR gate 250 with a threshold may be written as a pair of inequalities of the following form: In equation 2, i jare n inputs, o is the output and th is the threshold. These two inequalities can be used to completely describe the multi-terminal NOR gate 250 with thresholds in ILP format. To account for possible variability from CMOS implementations of such multi-terminal NOR gates 250, the NOR relationship can be expressed in terms of a ratio higher than The additional gap δ + and below the ratio The additional gap δ - In terms of ILP formulation, this changes Equation 2 to: Equation 3 prohibits for Thus, a gap δ is created above the threshold th + and below the threshold th creates a gap δ - This is a good size to compensate for the variability introduced by the CMOS implementation. ILP Formulation of the Training Problem Figure 7 An example of a digital neural network 300 according to aspects of the present disclosure is shown. Figure 7 The digital neural network 300 of FIG. 1 is used as an example with a simple and small topology in order to describe and capture aspects of the ILP formulation of the training problem that can be applied to digital neural networks 100 of any size and topology. Figure 7 As shown, the neural network 300 includes a plurality of multi-terminal logic gates 308 arranged in a plurality of layers 304. The neural network 300 also includes a plurality of input i 1,1 、i 1,2 ,…,i 1,12 and output o 3,1 The neural network 300 also includes a plurality of logic connectors 310 having states defining relationships or connections between adjacent layers 304. Also shown are internal outputs for the multi-terminal logic gates 308. 1,1 , o 1,2 ,…,o 2,3 , internal input i 2,1 、i 2,2 ,…,i 3,3 and for the internal input i' of the logic connector 310 connecting each of the layers 304 to its adjacent layer 304 2,1 、i' 2,2 、…、i' 3,3 The neural network 300 may also include an optional training circuit 111 . Fig. 8A and Figure 8B An embodiment of a logical connector according to aspects of the present disclosure is shown. In particular, Fig. 8AThe logic connector 311 is basically similar to Figure 2 The logic connector 110 shown in FIG. The logic connector 311 includes a switch 116 , an inverter 118 , an open circuit 120 , and a short circuit 122 . Figure 8B The logic connector 313 is implemented with an alternative design Fig. 8A The logic connector 313 has the same function as the logic connector 311. In particular, the logic connector 313 includes the inverter 118, the short circuit 122, the first switch 314 and the second switch 316. The first switch 314 is configured to select (e.g., connect to) one of the inverter 118 and the short circuit 122, and the second switch 316 is configured to operate in an open circuit or a short circuit state. Therefore, the combination of the first switch 314 and the second switch 316 can realize three different states of the logic connector 313 (e.g., open circuit, closed circuit or inversion). An important aspect of training the neural network 300 is to identify the variants of the ILP problem. Training involves finding the states of a set of logic connectors 310 at the input of each of the multi-terminal logic gates 308 such that for a set of inputs {I}, the neural network 300 will return output Thus, for each I∈{I}, the propagation of input I through neural network 300 can be accomplished by linking I to The following discussion applies to an implementation in which the multi-terminal logic gate 308 is implemented as a NOR gate. However, the principles and advantages of the discussion can be modified to be equally applicable to an OR gate implementation of the multi-terminal logic gate 308 or any other suitable implementation. For example, the binary variable x l,j ∈{0,1} can describe the partial state of the j-th logical connector 313 of layer l. l,j =1, the logic connector 310 is in the first state (eg, the first switch 314 is connected to the inverter 118). l,j =0, the logic connector 313 is in the second state (eg, the first switch 314 is connected to the short circuit 122). For the binary variable y describing the complementary state of the j-th logic connector 313 of layer l l,j ∈{0,1}, if y l,j =1, the logic connector has a third state (eg, the second switch 316 is open). l,j = 0, then the logic connector 313 has a value depending on x l,j The state of the logic connector 313 at the input of the multi-terminal logic gate 308 may satisfy the following inequality set: In equation 4, i l,j is the input applied to the input terminal of the multi-terminal logic gate 308, and i′ l,j is the input applied to the logic connector 313. The set of inequalities in Equation 4 can completely describe the state of the logic connector 313 in the neural network 300. The training process through ILP can be described based on the inequality group in Equation 4. For example, for I∈{I}, the input i 1,1 ,…,i 1,12 is set equal to the components of I. The output o of the first layer is then evaluated 1,1 ,…,o 1,4 The input i of the first layer 304 is 1,1 ,…,i 1,12 and output o 1,1 ,…,o 1,4 No logical connector 313 is involved, and therefore, input i 1,1 ,…,i 1,12 and output o 1,1 ,…,o 1,4 is a parameter that will be input in the next equation. For the second layer 304, the first multi-port gate 308 will be described in detail, since the remaining multi-port gates 308 operate similarly. Input i' 2,1 ,…,i′ 2,4 Can be set to o 1,1 ,…,o 1,4 , because the input i′ 2,1 ,…,i′ 2,4 is directly connected to the output o 1,1 ,…,o 1,4 . The state x of the logical connector 313 2,1 ,…,x 2,4 and 2,1 ,…,y 2,4 , using inequality (4), input i′ 2,1 ,…,i′ 2,4 Can be linked to input i 2,1 ,…,i 2,4 The output o on the multi-port gate 308 2,1 can be linked to input i using Equation 3 2,1 ,…,i 2,4 : A similar process can be used for the final multi-port gate 308, where the output o of the second layer 2,1 ,…,o 2,3 Linked to input i' of logic connector 313 3,1 ,…,i′ 3,3 , i′3,1 ,…,i′ 3,3 In turn, it is connected to the input i of the multi-terminal gate 308 through inequality (4). 3,1 ,…,i 3,3 , and finally the input of multi-terminal gate 308 can be linked to the output o through inequality (4) 3,1 Output 3,1 is set to It corresponds to the input I∈{I}. Therefore, for each pair The training process may include generating a set of ILP inequalities that propagate inputs through the neural network 300 and simultaneously backpropagate outputs through the neural network 300. For example, because the layers 304 of the neural network 300 are linked by inequality (4), the outputs from the previous layer 304 and the inputs to the next layer 304 will affect the state of the logical connector 313 that connects the current layer 304 to the previous layer 304 and the next layer 304. Therefore, for all pairs simultaneously Solving ILP, the training process will return the reproducible function For large neural networks 300, rather than generating a very large LIP problem, a mini-batch approach is used to solve randomly extracted pairs of and iterate the process over multiple training cycles until the training process reaches a threshold accuracy. Applications The adoption of neural networks for other applications such as data mining, image recognition, and signal identification is driving this growth in the neural network market, as there is a need to detect complex, non-linear relationships between variables and patterns. In fact, more efficient and less energy-intensive digital neural network architectures are highly desired to support current and future computational research. There are huge commercial applications in various industries (e.g., fintech, artificial intelligence, life sciences, manufacturing, governance and defense, transportation logistics). Cloud-based training and edge deployment using digital neural network solutions are expected to grow, primarily due to their advantages such as easy maintenance of generated data, cost-effectiveness, scalability, and efficient management. The digital neural network described in this article has a high potential to disrupt this market and bring strategic competitive advantages to early adopters. Evaluation of Digital Neural Networks By using training to find the configuration of the logic connectors 110 that maps the function f to the digital neural network 100, the digital neural network 100 can be programmed so that the state of the logic connectors 110 (e.g., the state of the switches 116) is set following the training output. In implementations where the digital neural network 100 can be retrained, updates to the training of the digital neural network 100 can be achieved by simply reprogramming the state of the switches 116. For example, if more labeled data is available, this can allow offline training and simple reconfiguration of the switches 116 to achieve updates. Once the switches 116 are set, the digital neural network 100 is ready to evaluate for any possible input x. By design, the evaluation of the digital neural network 100 can be performed in a single clock cycle. In addition, using CMOS technology to design the multi-terminal logic gates 108, the power and energy used for evaluation is very small, several orders of magnitude smaller than current neural networks. In fact, since there is no data movement from the memory (only bringing inputs and returning outputs) to the processing unit (e.g., the digital neural network 100), CMOS technology allows the implementation of extremely low power multi-terminal logic gates 108. Theoretical aspects and practical performance of the digital neural network 100 can be analyzed against established benchmarks. The scale of specific applications (such as classification, predictive analysis, image recognition, etc.) in terms of logic gates, transistors, and interconnect complexity can be quantified. To this end, for a given topology and input and output sizes, the minimum number of layers used to obtain and / or ensure versatility can be determined. The training efficiency in terms of accuracy measured on established benchmarks can be evaluated. In order to efficiently train the digital neural network 100, a virtual storage and computing integrated computer can be used. In some embodiments, training can be performed using a full training group and using a minimum batch to reduce the size of the ILP to be solved, as this can allow for accelerated training under certain conditions. Performance in terms of energy / power and speed can be evaluated. This can be evaluated by considering how it is implemented in CMOS technology. Trained digital neural network As discussed herein, the digital neural network 100 may be trained by solving a first set of ILP problems to determine a first set of states that map a first function f1 to the logical connectors 110 of the digital neural network 100. The digital neural network 100 may also be retrained by solving a second set of ILP problems corresponding to a second function f2 to determine a second set of states of the logical connectors 110 to map the second function f2 into the digital neural network 100. Thus, the digital neural network 100 may be retrained to implement substantially any function f by generating a corresponding set of ILP problems. However, for some applications, it may not be necessary to retrain the digital neural network 100. For example, when the digital neural network 100 is designed to implement a single function f, it may not be necessary to retrain the digital neural network 100. Therefore, it is not necessary to include a parallel path (e.g., Figure 2 Each of the inverter 118, the open circuit 120, and the short circuit 122) or the switch 116. For example, if the trained state of a given logic connector 110 is the inverter 118 path, the logic connector 110 may include only the inverter 118 and no other components. By implementing the logic connector 110 only with components corresponding to the trained state of the logic connector 110, the digital neural network 100 can be implemented with a significantly smaller number of components. In certain implementations, the digital neural network 100 may be implemented on a single chip, for example, when incorporated into certain devices (e.g., in an autonomous vehicle). Because the digital neural network 100 may be implemented entirely in CMOS technology, the digital neural network 100 may be more easily implemented on a chip than other neural networks that rely on the von Neumann architecture. Advantageously, this enables the digital neural network 100 to be more easily used in a variety of different applications than conventional neural networks. The digital neural network disclosed herein can provide fast computations. The digital neural network can be implemented on a single chip. In some cases, an input can be loaded, the digital neural network can calculate a function in a single clock cycle, and then the output can be read out. This is a significant improvement in speed relative to some existing neural network function calculations. The digital neural network disclosed herein can also use very little energy compared to some existing neural network calculations. in conclusion The foregoing disclosure is not intended to limit the present disclosure to the precise forms of use or specific fields of use disclosed. Therefore, it is contemplated that various alternative embodiments and / or modifications to the present disclosure are possible, whether explicitly described or implied herein, based on the disclosure. Thus, having described embodiments of the present disclosure, one of ordinary skill in the art will recognize that changes may be made in form and detail without departing from the scope of the present disclosure. Therefore, the present disclosure is limited only by the claims. In the foregoing description, the present disclosure has been described with reference to specific embodiments. However, as will be appreciated by those skilled in the art, without departing from the spirit and scope of the present disclosure, the various embodiments disclosed herein may be modified or otherwise implemented in various other ways. Therefore, this description is considered to be illustrative, and the purpose is to teach those skilled in the art to make and use the methods of various embodiments. It should be understood that the form of the disclosure shown and described herein is considered to be a representative embodiment. Equivalent elements, materials, processes or steps may replace those representedly shown and described herein. In addition, some features of the present disclosure may be utilized independently of the use of other features, all of which will be apparent to those skilled in the art after benefiting from the specification of the present disclosure. Expressions such as "including", "comprising", "incorporating", "consisting of", "having", "being" and the like used to describe and claim the present disclosure are intended to be interpreted in a non-exclusive manner, i.e., allowing for the existence of items, components or elements that are not explicitly described. Mentioning the singular is also interpreted as involving the plural. In addition, various embodiments disclosed herein will be considered as illustrative and explanatory meanings, and should never be interpreted as limiting the present disclosure.All joint references (e.g., attachment, bonding, connection, connection and the like) are only used to help readers understand the present disclosure, and may not produce restrictions, particularly with respect to the position, direction or use of the system and / or method disclosed herein.Therefore, joint references (if any) will be widely interpreted.In addition, such joint references are not necessarily inferred to be two elements directly connected to each other.In addition, all digital terms, such as but not limited to "first", "second", "third", "primary", "minor", "main" or any other common and / or digital terms, should also be regarded as identifiers only, to help readers understand the various elements, embodiments, variations and / or modifications of the present disclosure, without any restrictions, particularly any element, embodiment, variation and / or modification relative to or superior to the order or preference of another element, embodiment, variation and / or modification. It should also be understood that one or more of the elements shown in the figures / drawings may also be implemented in a more separate or more integrated manner, or even removed or rendered inoperable in some cases, as may be useful for certain applications.

Claims

1. A neural network circuit, comprising: a plurality of logic gates arranged into a plurality of layers, each of the logic gates having a plurality of inputs and having an output; as well as a plurality of logic connectors disposed between each pair of adjacent layers, each of the logic connectors determining a relationship between an output of a first logic gate to one of a plurality of inputs of a second logic gate, and each of the logic connectors having one of a plurality of different logic connector states, The neural network circuit is configured to be trained to implement a function, and the training is performed by finding a set of logical connector states of the logical connectors that enables the neural network circuit to implement the function.

2. The neural network circuit according to claim 1, wherein: The logic connector state includes a first state in which the output of the first logic gate is connected to the input of the second logic gate via a NOT gate and a second state in which the output of the first logic gate is connected to the input of the second logic gate via a short circuit.

3. The neural network circuit according to claim 2, wherein: The logic connector states also include a third state in which the output of the first logic gate is connected to the input of the second logic gate via an open circuit.

4. The neural network circuit according to claim 1, wherein: The logic gates and the logic connectors are implemented using complementary metal oxide semiconductor (CMOS) technology.

5. The neural network circuit according to claim 1, wherein: The neural network circuit is formed on a single chip.

6. The neural network circuit according to claim 1, wherein: Each of the logical connectors comprises: enter; a short circuit connected to the input; an inverter arranged in parallel with the shorting circuit and connected to the input; Output; and At least one switch is configured to connect one of the shorting circuit and the inverter to the output.

7. The neural network circuit according to claim 6, wherein: Each of the logical connectors further comprises: Open circuit, connected to the input, Wherein, the at least one switch is configured to connect one of the short circuit, the inverter and the open circuit to the output.

8. The neural network circuit according to claim 6, wherein: The at least one switch includes a first switch and a second switch connected in series, The first switch is configured to be electrically connected to one of the short circuit and the inverter, and The second switch is configured to operate in an open circuit or a short circuit state.

9. The neural network circuit according to claim 1, wherein: Each of the logic connectors includes one of a shorting circuit or an inverter, and Each of the logic connectors connects the output of a logic gate of a previous layer to the input of a logic gate of a current layer.

10. The neural network circuit according to claim 1, wherein: Each of the logic gates comprises a multi-terminal NOR gate.

11. The neural network circuit according to claim 1, further comprising: A training circuit is configured to generate the set of logical connector states.

12. The neural network circuit according to claim 1, wherein: Each of the logical connectors has a fixed one of the logical connector states.

13. The neural network circuit according to claim 12, wherein: Each of the logic connectors includes a single one of a short circuit, an inverter, and an open circuit corresponding to the fixed one of the logic connector states.

14. A method for computing a function using a neural network circuit, comprising: A neural network circuit is provided, the neural network circuit comprising: a plurality of logic gates arranged into a plurality of layers, each of the logic gates having a plurality of inputs and having an output; and a plurality of logic connectors, including a set of logic connectors disposed between each pair of adjacent layers, each of the logic connectors having one of a plurality of different logic connector states, wherein the plurality of logic connectors are programmed to implement the function; and The function is calculated for an input signal using the neural network.

15. The method according to claim 14, further comprising: A set of logic connector states of the plurality of logic connectors that enables the neural network circuit to implement the function is found.

16. The method according to claim 15, further comprising: generating a set of integer linear programming (ILP) problems based on the function; as well as The set of ILP problems is solved to produce the set of logical connector states.

17. The method according to claim 16, further comprising: determining a set of inequalities describing a state of the logical connector; as well as The output from the previous layer is linked to the input of the next layer through the set of inequalities, The generation of the ILP problem is based on linking the output from the previous layer to the input of the next layer through the inequality group.

18. The method according to claim 14, wherein: The logic connector state includes a first state in which the output of the first logic gate is connected to the input of the second logic gate via a NOT gate and a second state in which the output of the first logic gate is connected to the input of the second logic gate via a short circuit.

19. Single chip, including: a plurality of logic gates arranged into a plurality of layers of the neural network circuit, each of the logic gates having a plurality of inputs and having an output; a plurality of logic connectors disposed between each pair of logic gates of adjacent layers, each of the logic connectors determining a relationship between an output of a first logic gate to one of a plurality of inputs of a second logic gate, and each of the logic connectors having one of a plurality of different logic connector states; Multiple input ports; as well as At least one output, The chip is configured to calculate a function between the input terminal and the output terminal.

20. The chip according to claim 19, wherein: The chip is configured to compute the function for a given input provided to the input terminal in ten clock cycles or less.

Images