A deep neural network having multiple layers formed from multiple terminal logic gates

Abstract

A deep neural network circuit having a plurality of layers formed from multi-terminal logic gates is provided. In one aspect, the neural network circuit includes a plurality of logic gates disposed in a plurality of layers and a plurality of logic connectors disposed between each pair of adjacent layers. Each of the logic connectors connects an output of a first logic gate to an input of a second logic gate, and each of the logic connectors has one of a plurality of different logic connector states. The neural network circuit is configured to be trained to perform a function by finding a set of logic connector states for the logic connectors such that the neural network circuit performs the function.

Description

Technical Field

[0001] Cross - reference to related applications This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 364,405, filed on May 9, 2022, entitled "DEEP NEURAL NETWORK WITH MULTIPLE LAYERS FORMED OF MULTI - TERMINAL LOGIC GATES", the disclosure of which is incorporated herein by reference in its entirety for all purposes.

[0002] The present disclosure generally relates to neural networks. More particularly, the present disclosure relates to deep neural networks implemented using multiple layers formed of multi - terminal logic gates.

Background Art

[0003] Neural networks can be implemented on various types of hardware such as central processing units (CPUs) and field - programmable gate arrays (FPGAs), as well as on specialized hardware designed for neural networks such as distributed architectures like graphics processing units (GPUs) or tensor processing units (TPUs).

Summary of the Invention

[0004] The innovations described in the claims have several aspects each, and no single one of them solely bears the desired attributes. Without limiting the claims, some of the prominent features of the present disclosure are briefly described here.

[0005] One aspect of the present invention is a neural network circuit, the neural network circuit comprising a plurality of logic gates arranged in a plurality of layers, each of the logic gates being a plurality of logic gates having a plurality of inputs and outputs, and a plurality of logic connectors arranged between each pair of adjacent layers, each of the logic connectors determining a relationship between an output of a first logic gate and one of a plurality of inputs of a second logic gate, each of the logic connectors being a plurality of logic connectors having one of a plurality of different logic connector states, the neural network circuit being configured to be trained to perform a function by finding a set of logic connector states for the logic connectors such that the neural network circuit performs the function.

[0006] In some embodiments, the logic connector states include a first state in which the output of the first logic gate is connected to the input of the second logic gate via a NOT gate, and a second state in which the output of the first logic gate is connected to the input of the second logic gate via a short circuit.

[0007] In some embodiments, the logic connector states further include a third state in which the output of the first logic gate is connected to the input of the second logic gate via an open circuit.

[0008] In some embodiments, the logic gates and the logic connectors are implemented in complementary metal-oxide-semiconductor (CMOS) technology.

[0009] In some embodiments, the neural network circuit is formed on a single chip.

[0010] In some embodiments, each of the logic connectors comprises an input, a short circuit connected to the input, an inverter arranged in parallel with the short circuit and connected to the input, an output, and at least one switch configured to connect either the short circuit or the inverter to the output.

[0011] In some embodiments, each of the logic connectors further comprises an open circuit connected to the input, and at least one switch is configured to connect one of a short circuit, an inverter, and an open circuit to the output.

[0012] In some embodiments, at least one switch comprises a first switch and a second switch connected in series, the first switch is configured to be electrically connected to one of a short circuit and an inverter, and the second switch is configured to operate in either an open circuit or a short circuit state.

[0013] In some embodiments, each of the logic connectors comprises at least one of a short circuit or an inverter, and each of the logic connectors connects the output of the logic gate of the previous layer to the input of the logic gate of the current layer.

[0014] In some embodiments, each of the logic gates includes a multi-terminal NOR gate.

[0015] In some embodiments, the neural network further comprises a training circuit configured to generate a set of logic connector states.

[0016] In some embodiments, each of the logic connectors has one of the fixed logic connector states.

[0017] In some embodiments, each of the logic connectors comprises one of a short circuit, an inverter, and an open circuit corresponding to one of the fixed logic connector states.

[0018] Another aspect is a method of computing a function using a neural network circuit, the method comprising providing a neural network circuit including a plurality of logic gates arranged in a plurality of layers, each of the logic gates having a plurality of inputs and outputs, and a set of logic connectors arranged between each pair of adjacent layers, each of the logic connectors having one of a plurality of different logic connector states, the plurality of logic connectors being programmed to perform a function; and computing a function of an input signal using the neural network.

[0019] In some embodiments, the method further comprises finding a set of logic connector states of the plurality of logic connectors such that the neural network circuit performs a function.

[0020] In some embodiments, the method further comprises generating a set of integer linear programming (ILP) problems based on the function, and solving the set of ILP problems to generate a set of logic connector states.

[0021] In some embodiments, the method further comprises determining a set of inequalities that describe the states of the logic connectors, and linking the outputs from a previous layer to the inputs of a subsequent layer via the set of inequalities, wherein generating the ILP problems is based on linking the outputs from a previous layer to the inputs of a subsequent layer via the set of inequalities.

[0022] In some embodiments, the logic connector states include a first state in which the output of a first logic gate is connected to the input of a second logic gate via a NOT gate, and a second state in which the output of the first logic gate is connected to the input of the second logic gate via a short circuit.

[0023] Yet another aspect is a single chip, the single chip being a plurality of logic gates arranged in a plurality of layers of a neural network circuit, each logic gate being a plurality of logic gates having a plurality of inputs and outputs, and a plurality of logic connectors arranged between each pair of logic gates in adjacent layers, each of the logic connectors determining the relationship between the output of a first logic gate and one of the plurality of inputs of a second logic gate, each of the logic connectors having one of a plurality of different logic connector states, a plurality of input terminals, and at least one output terminal, the chip comprising at least one output terminal configured to calculate a function between the input terminal and the output terminal.

[0024] In some embodiments, the chip is configured to calculate a function in 10 clock cycles or less for a given input supplied to the input terminals.

[0025] For the purposes of summarizing the present disclosure, certain aspects, advantages, and novel features of the innovation are described herein. It should be understood that not all such advantages may necessarily be achieved in accordance with any particular embodiment. Thus, the innovation may be embodied or carried out in a manner that achieves or optimizes one advantage or group of advantages taught herein without necessarily achieving other advantages that may be taught or suggested herein.

Brief Description of the Drawings

[0026]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8A

Figure 8B

DETAILED DESCRIPTION OF THE INVENTION

[0027] The following description of specific embodiments presents various descriptions of specific embodiments. However, the technological innovations described herein can be implemented in many different ways, for example, as defined and encompassed by the claims. In this description, reference is made to the drawings, in which like reference numerals can indicate the same or functionally similar elements. It will be understood that the elements shown in the drawings are not necessarily drawn to scale. Further, it will be understood that a particular embodiment can include more elements and / or a subset of the elements shown in the drawings than are shown. Additionally, some embodiments can incorporate any suitable combination of features from two or more of the drawings. The headings provided herein are for convenience only and do not necessarily affect the scope or meaning of the claims.

[0028] Current processor architectures are suitable for relatively small datasets, but may not be able to handle the increasing amount of information required in real time for certain computationally intensive applications. For example, vehicles can benefit greatly from more computing power to interpret received sensor data and make important decisions quickly in real time. Aspects of the present disclosure relate to cost-effective and scalable computing systems and methods that can improve the computing power of various applications, including addressing future gaps in vehicle data processing capabilities.

[0029] Current edge processors based on central processing units (CPUs), graphics processing units (GPUs), and field programmable gate arrays (FPGAs) cannot execute algorithms efficiently enough to achieve specific computational goals for certain large-scale sensor data streams, such as data streams within a vehicle. Data movement is one of the most significant limitations in a particular computing architecture. This limitation, known as the von Neumann bottleneck, has two main effects: it limits computing throughput and requires significant energy to move data between memory and processing.

[0030] The adoption of machine learning and artificial intelligence (AI) technologies such as neural networks is increasing, aiming to improve the processing ability of larger datasets and enable automation. However, in many cases, these technologies are still restricted by their underlying von Neumann architecture. Specifically, there is an issue of realizing an ultra-low-power electronic architecture for real-time evaluation / inference of neural networks. The non-linearity of neural networks (the artificial neuron activation functions can be sigmoid, rectified linear unit (ReLU), etc.) and the property of deepness (several layers are evaluated in sequence) may involve an inevitable series of calculations, so even with state-of-the-art distributed architectures such as GPUs or tensor processing units (TPUs), there may be multiple inevitable clock cycles included, which remains an unsolved problem for neural networks executed on conventional hardware. This is translated into an unacceptably long evaluation / inference time for many applications (e.g., autonomous vehicles). Despite efforts to propose mitigation strategies for this problem, due to the physical and computational limitations of modern neural network designs, the fundamental problem may not be solved by digital architectures (GPUs, CPUs, TPUs).

[0031] Aspects of the present disclosure relate to improved neural network designs, which may be referred to as MemComputing Neural Network (MEMC-NN) or more generally digital neural networks. The digital neural networks disclosed herein can be implemented by circuits. In certain embodiments, the disclosed neural networks can be easily integrated onto a chip using only logic gates, switches, and / or selectors. Aspects of the present disclosure can address some or all of the above limitations, thereby providing real-time neural network evaluation / inference using very little power. The neural network solutions provided herein can be practically integrated at every level within a system, from servers and clouds to small devices such as smartwatches and glasses, or the Internet of Things (IoT) and edge computing as described herein. Advantageously, aspects of the present disclosure have the potential to provide unprecedented AI processing capabilities compliant with size, weight, power consumption, environmental, and cost goals. The digital neural network systems and methods disclosed herein can be applied to any other suitable application and / or can meet any other suitable purpose.

[0032] Embodiments of digital neural network design FIG. 1 is a circuit diagram showing a digital neural network 100 according to an aspect of the present disclosure. The digital neural network 100 can be implemented as a circuit formed on a single chip. The digital neural network 100 is a deep neural network including a plurality of inputs 102, a plurality of layers 104, a plurality of outputs 106, and an optional training circuit 111. Each of the layers 104 includes a plurality of multi-terminal logic gates 108 (also referred to as "logic gates"). Each multi-terminal logic gate 108 can be logically associated with one or more of the multi-terminal logic gates 108 in the previous layer 104. For example, the logical association between two multi-terminal logic gates 108 can apply a logical operation to the output of the multi-terminal logic gate 108 from the previous layer 104 before providing the result of the logical operation to the multi-terminal logic gate 108 in the current layer 104. These logical associations can be implemented by logical connectors 110 (also referred to as "logical operators", "binary logical operators", or "logical circuits").

[0033] Each of the logical connectors 110 is configured to define the relationship between a pair of multi-terminal logic gates 108. The logical connector 110 does not necessarily physically connect the logic gates. For example, the logical connector 110 can implement an open circuit between two multi-terminal logic gates 108. As shown in FIGS. 2 and 3 and described in more detail below, the relationship between two multi-terminal logic gates 108 can include a logical NOT (implemented, for example, via an inverter 118), an open circuit 120, and / or a short circuit 122, but other relationships are also possible.

[0034] In certain embodiments, the logical connector 110 can receive a single input and output a single output. In certain embodiments, the connectivity topology implemented by the logical connector 110 defines the network layer of the digital neural network 100, such as full connection, convolution, pooling, and the like.

[0035] An optional training circuit 111 can be used to train the digital neural network 100. Depending on the embodiment, the optional training circuit 111 may be included as part of the same chip as the digital neural network 100, or may be implemented on a separate chip. The digital neural network 100 can be trained to perform the functions as disclosed herein. After the digital neural network 100 is trained, for a given input, it can calculate a function in 1 clock cycle, several clock cycles, or 1-digit clock cycles. For example, the digital neural network 100 can be configured to calculate a function in less than 2, 3, 4, 5, 6, 7, 8, 9, or 10 clock cycles, or several clock cycles, depending on the implementation.

[0036] FIG. 2 shows an embodiment of a multi-terminal logic gate 108 and a plurality of logic connectors 110 connected thereto according to an aspect of the present disclosure. In the illustrated embodiment, each of the logic connectors 110 includes an input 112, an output 114, a switch 116, and a plurality of alternative parallel paths between the input 112 and the output 114. In the first logic connection 110 shown in FIG. 2, the parallel paths include an inverter 118 (e.g., a NOT gate), an open circuit 120, and a short circuit 122. The switch 116 is configured to define the state (also referred to as the "logic state") of the logic connector 110 by defining the logical relationship between the input 112 and the output 114 of the logic connector 110 through one of the inverter 118, the open circuit 120, and the short circuit 122. FIG. 2 shows the open circuit 120 as a separate path that the switch 116 can select, but in a particular embodiment, there may be no physical path having the open circuit 120. For example, when selecting the open circuit, the switch 116 is an It can be disconnected from each of the inverter 118 and the short - circuit 122. Thus, the logical operation applied by a given logic connector 110 may depend on the path (e.g., inverter 118, open circuit 120, and short - circuit 122) used to connect the input 112 and the output 114 of the logic connector 110. Some logic connectors 110 can include two alternative parallel paths such as an inverter and a short - circuit. The two lower logic connectors 110 in FIG. 2 show such logic connectors 110. Depending on the embodiment, the digital neural network 100 may include logic connectors 110 having substantially the same structure, or the digital neural network 100 may include a plurality of logic connectors 110 having different structures such as the logic connector 110 shown in FIG. 2. Thus, in various embodiments, the logic connectors 110 used for a particular application may be of the same or different types.

[0037] Furthermore, in the example of FIG. 2, the multi - terminal logic gate 108 is implemented as a multi - terminal OR gate. However, aspects of the present disclosure are not limited to this, and the multi - terminal logic gate 108 can be implemented using other types of multi - terminal logic gates 108 depending on the implementation. As will be described in more detail herein, the state of the logic connector 110 can be similar to the weighting of a conventional neural network. In some other embodiments, each of the logic connectors 110 can be implemented using only a single path (e.g., one of the inverter 118, open circuit 120, and short - circuit 122) without the switch 116 representing the training state of the logic connector 110, as described herein.

[0038] FIG. 3 shows another embodiment of the multi-terminal logic gate 108 and the plurality of connected logic connectors 110 connected thereto according to an aspect of the present disclosure. The embodiment of FIG. 3 is similar to the embodiment of FIG. 2, except that the logic connector 110 does not include an open circuit 120 path. FIGS. 2 and 3 provide exemplary embodiments of the logic connector 110, but aspects of the present disclosure are not limited thereto, and the logic connector 110 can include more logic states that perform one or more different logical operations.

[0039] Universality of Digital Neural Networks As used herein, a neural network can function as a universal machine if it can compute any computable function in the sense of Turing. A computable function can be defined as a function that can be computed by a Turing machine. The digital neural network 100 embodied in FIGS. 1-3 can be used to create a universal machine. In fact, combining an OR gate and a NOT gate (e.g., an inverter) can form a complete basis in the sense that any Boolean function can be written as a set of OR gates and NOT gates.

[0040] Note that the digital neural network 100 described in this specification is not limited to the configurations shown in FIGS. 1 to 3. The digital neural networks 100 in FIGS. 1 to 3 can create layers that allow for great flexibility using any basis of Boolean functions or mixtures thereof. The universality of the digital neural network 100 described in this specification can also be seen from an operational perspective. Given N input terminals and M output terminals, there exists a minimum number of layers (depending on the connectivity topology) that enables the calculation of any possible function y = f(x) for any arbitrary input x of length N and binary output y of length M. The states of the logic connectors 110 can be used to implement the function f. In other words, for each function f, there exists at least one set of states of the logic connectors 110 that make up the digital neural network 100 to accurately evaluate the function f.

[0041] Training of Digital Neural Network As used in this specification, training the digital neural network 100 generally refers to finding the configuration of the logical states of the logic connectors 110 that map the function f to the digital neural network 100. This training can be performed without analytical knowledge of the function f or the results y for all possible inputs x. Generally, a set of data results y (labels) for a given input x (training set) is available for training. The training circuit 111 can perform the training to train the digital neural network 100.

[0042] In conventional neural networks, gradient descent-based techniques can be used to find a weighting configuration that allows for a good representation of the function f. However, since the number of available states of the logic connector 110 is an integer, it may not be possible to define the gradient of the logic connector 110, so such techniques may not be applicable to the training of the digital neural network 100 described in this specification. This was one of the reasons why it was difficult to fully implement a digital neural network.

[0043] Various common training methods for neural networks may face technical challenges for training digital circuits. In such common training methods, continuous parameters are used for training. However, digital circuits can function based on binary values rather than continuous parameters. Methods of acting on continuous parameter values are generally not very suitable for digital circuits that use binary values.

[0044] The training of the digital neural network 100 disclosed herein can include determining associations (e.g., defined by the digital neural network 100) between logic gates 108 in different layers of the neural network 100. For example, this can include determining whether to connect logic gates 108 of different layers via short - circuit circuit 122 or via inverter 118. As another example, training can include determining whether to connect to logic gates 108 within different layers via short - circuit circuit 122 or via inverter 118, or not to connect to two logic gates 108 (e.g., by connecting open - circuit 120 to one of the inputs of one of the logic gates 108).

[0045] One technique for training the neural network 100 disclosed herein includes selecting a training problem using integer linear programming (ILP). For example, for each input x, a set of linear inequalities with binary variables representing the states of logic connectors 110 as unknowns can be defined to represent the propagation of input x through the neural network 100. Thus, the training of the digital neural network 100 can be transformed into solving an ILP problem.

[0046] However, solving ILP may not be a trivial task. ILP belongs to a class of combinatorial problems, also known as non-deterministic polynomial (NP) problems, and is notoriously difficult. The present disclosure relates to a virtual memory computing machine (VMM), and a non-transitory computer-readable storage device stores instructions that, when executed by one or more processors, emulate a novel computing architecture and solve large-scale ILP problems very efficiently. The VMM can be used to solve ILP problems related to the training of the digital neural network 100 and provide a configuration of the states of the logic connectors 110 representing the function f. The VMM can be a software emulation of a self-organizing algebraic gate (SOAG) that can be used as a training solution for the digital neural network 100.

[0047] Any suitable principles and advantages disclosed in International Patent Application No. PCT / US2022 / 053781 filed on December 22, 2022 and / or International Patent Application No. PCT / US2016 / 041909 filed on July 12, 2016 and published as International Publication No. 2017 / 011463 can be used to solve problems related to the training of any of the neural networks disclosed herein (e.g., ILP problems), and the disclosure of each of these patent applications is hereby incorporated by reference in its entirety for all purposes. For example, the training circuit 111 can be implemented in accordance with any suitable principles and advantages disclosed in these international patent applications. Any other suitable training method can be applied to the digital neural networks disclosed herein.

[0048] The digital neural networks disclosed herein can be trained multiple times to compute different functions. In certain applications, the digital neural networks disclosed herein can be trained only once. This can be useful for certain applications such as Internet-of-Things devices.

[0049] Design of NOR Gate as Activation Function In certain embodiments, a NOR gate can be used to implement the multi-terminal logic gate 108 of the digital neural network 100. The multi-terminal NOR gate can be defined using a threshold having the following general relationship.

Equation

[0050] In Equation 1, o is the output, i j is the j-th input of the NOR gate, and th is the threshold. In certain embodiments, a NOR gate can be used to implement the multi-terminal logic gate 108 in place of an OR gate, because the NOR gate can be easily implemented in complementary metal-oxide-semiconductor (CMOS) technology. However, aspects of the present disclosure are not limited thereto. To obtain an OR gate, a NOT gate can be added after the NOR gate. However, when used in certain embodiments of the digital neural network 100 described herein, the OR gate and the NOR gate may be fully interchangeable, because the input terminals of the multi-terminal logic gate 108 can be coupled to a logic connector 110 that can include a switch 116 that can select an inverter 118 or a short circuit 122 (e.g., as shown in FIG. 3).

[0051] FIG. 4 shows an embodiment of a multi-terminal NOR gate 200 according to an aspect of the present disclosure. FIG. 5 shows an embodiment of a multi-terminal NOR gate 250 having a threshold according to an aspect of the present disclosure. Each of the multi-terminal NOR gates 200 and 250 can be implemented in CMOS technology.

[0052] Referring to FIG. 4, the multi-terminal NOR gate 200 includes a plurality of inputs i 1 , i 2 , …i n , and an output o, and a first power supply terminal V DDand a second power supply terminal GND, and a plurality of transistors 202. The transistor 202 is configured to perform a NOR function.

[0053] In FIG. 5, the multi-terminal NOR gate 250 includes a plurality of input switches i 1 , i 2 , …i n , an output o, a first power supply terminal V DD , two second power supply terminals GND, a pair of transistors 252, and a plurality of first resistors R in series with each input switch i 1 , i 2 , …i n , and a threshold resistor R . When the threshold resistor R th is set to R / 2 or any value greater than that, the function of the multi-terminal NOR gate 250 in FIG. 5 can be substantially the same as the function of the multi-terminal NOR gate 200 in FIG. 4. The operating principle of the NOR gate 250 is as follows. The resistor R th is sized such that when a number of switches greater than strictly th are closed, the output voltage o is set to 0, and otherwise to V th . Thus, the NOR gate 250 implements Equation 1 where the state of the switches is the input i DD and the output o is the voltage at node o. j

[0054] Even when the multi-terminal NOR gate 250 in FIG. 5 is implemented entirely in CMOS technology, it can be implemented using a combination of digital circuit elements and analog circuit elements. This can provide certain advantages over a fully digital implementation in certain applications, for example, as described herein.

[0055] Continuing to refer to the multi-terminal NOR gate 250 in FIG. 5, the inputs are configured to open and close the input switches i 1 , i 2 , …i n . The input switches i 1 , i 2 , …i nis the input switch i 1 i 2 ...i n is configured to be opened and closed by applying a voltage to the control terminals of the respective transistors (e.g., the gates of CMOS components) used to implement it. Therefore, the input switch i 1 i 2 ...i n can be directly controlled by the output of other NOR gates 250 or other general-purpose CMOS-based logic gates. The embodiment of FIG. 5 can be one of the most compact embodiments for a NOR gate with a threshold. For example, the embodiment of FIG. 5 can use at least 3(n + 1) transistors, where n is the number of inputs. A complete digital implementation can include a more complex digital circuit that essentially performs the sum of the inputs and compares the sum with a threshold.

[0056] Depending on the implementation, the resistance value, or ratio th of the resistor R

Number

Number

Number

Number

Number

[0057] However, when implemented in CMOS technology, there may be variations in resistance, and the NOR gate 250 may not follow a perfect step function. As shown in Figure 6, when the threshold is small, i.e., when th ≤ 2, the gap

Number

[0058] ILP Formulation of NOR Gate According to an aspect of the present disclosure, the digital neural network 100 including the logic gate 108 and / or the threshold logic gate 250 can be trained using ILP formulation. Starting from Equation 1 for the output o above, the ILP formulation of the multi-terminal NOR gate 250 with a threshold can be written as a pair of inequalities of the following form. [Number]

[0059] In Equation 2, i j represents n inputs, o represents the output, and th represents the threshold. Using these two inequalities, the multi-terminal NOR gate 250 with a threshold in the ILP format can be fully described. To address the possible variations caused by the CMOS implementation of such a multi-terminal NOR gate 250, the NOR relationship has a special gap of δ [Number] above the ratio + and δ - below it, and can be implemented. Regarding the ILP formulation, Equation 2 becomes as follows. [Number]

[0060] Equation 3 [Number] is prohibited from becoming [Number] and creates a gap of δ + above the threshold th and δ - below the threshold th. This is of appropriate size and can be used to compensate for the variations caused by the CMOS implementation.

[0061] ILP Formulation of Training Problems FIG. 7 shows an example of a digital neural network 300 according to an aspect of the present disclosure. The digital neural network 300 of FIG. 7 is provided as an example with a simple and small topology to illustrate and capture aspects of the ILP formulation of training problems applicable to digital neural networks 100 of any size and topology. As shown in FIG. 7, the neural network 300 includes a plurality of multi-terminal logic gates 308 arranged in a plurality of layers 304. The neural network 300 also includes a plurality of inputs i 1,1 、i 1,2 、...、i 1,12 and an output o 3,1 . The neural network 300 further includes a plurality of logic connectors 310 having states that define the relationships or connections between adjacent layers 304. Also, internal outputs o 1,1 、o 1,2 、...o 2,3 to the multi-terminal logic gates 308, internal inputs i 2,1 、i 2,2 、...、i 3,3 , and internal inputs i' 2,1 、i’ 2,2 、...、i’ 3,3 to the logic connectors 310 connecting each layer 304 to its adjacent layer 304 are also shown. The neural network 300 can further include an optional training circuit 111.

[0062] FIGS. 8A and 8B show embodiments of logic connectors according to aspects of the present disclosure. In particular, the logic connector 311 of FIG. 8A is substantially the same as the logic connector 110 shown in FIG. 2. The logic connector 311 includes a switch 116, an inverter 118, an open circuit 120, and a short circuit 122.

[0063] The logic connector 313 in FIG. 8B realizes the same function as the logic connector 311 in FIG. 8A with an alternative design. In particular, the logic connector 313 includes an inverter 118, a short - circuit circuit 122, a first switch 314, and a second switch 316. The first switch 314 is configured to select (e.g., connect) one of the inverter 118 and the short - circuit circuit 122, and the second switch 316 is configured to operate in either an open - circuit or a short - circuit state. Thus, the combination of the first switch 314 and the second switch 316 can realize three different states (e.g., open - circuit, closed - circuit, or inversion) for the logic connector 313.

[0064] An important aspect for training the neural network 300 is to identify the variables of the ILP problem. The training involves finding a set of states of the logic connector 310 at each input terminal of the multi - terminal logic gate 308 such that the neural network 300 returns an output

Number

Number

[0065] ∈ {0, 1} can describe the partial state of the j - th logic connector 313 in layer l. x l,j ∈ {0, 1} can describe the partial state of the j - th logic connector 313 in layer l. x l,jWhen = 1, the logic connector 310 is in a first state (e.g., the first switch 314 is connected to the inverter 118). x l,j When = 0, the logic connector 313 is in a second state (e.g., the first switch 314 is connected to the short - circuit 122). The binary variable y that describes the complementary partial state of the j - th logic connector 313 in layer l l,j For y ∈ {0, 1} l,j When = 1, the logic connector has a third state (e.g., the second switch 316 realizes an open circuit). y l,j When = 0, the logic connector 313 has a state that depends on x l,j (e.g., this depends on the state of the first switch 314). Thus, the state of the logic connector 313 at the input terminal of the multi - terminal logic gate 308 can satisfy the following set of inequalities.

Equation

[0066] In Equation 4, i l,j is the input applied to the input terminal of the multi - terminal logic gate 308, and i’ l,j is the input applied to the logic connector 313. The set of inequalities in Equation 4 can completely describe the state of the logic connector 313 within the neural network 300.

[0067] The training process by ILP can be explained based on the set of inequalities in Equation 4. For example, when I ∈ {I}, the inputs i 1,1 ,..., i 1,12 are set equal to the components of I. Subsequently, the outputs of the first layer o 1,1 ,...., o 1,4 are evaluated. The inputs i 1,1 ,..., i 1,12 and outputs o 1,1 ,...., o 1,4 do not include any logic connector 313. Thus, the inputs i 1,1 ,..., i 1,12 and outputs o1,1 ,..., o 1,4 is a parameter that enters the following equation. For the second layer 304, since the remaining multi-terminal gates 308 also operate in the same way, the first multi-terminal gate 308 will be described in detail. The input i' 2,1 ,..., i' 2,4 is the output o 1,1 ,..., o 1,4 is directly connected to, so the input i' 2,1 ,..., i' 2,4 is o 1,1 ,..., o 1,4 and can be set. The input i' 2,1 ,..., i' 2,4 is linked to the input i 2,1 ,..., x 2,4 and y 2,1 ,..., y 2,4 through the states of, using the inequality (4). 2,1 ,..., i 2,4 The output o on the multi-terminal gate 308 2,1 can be linked to the input i 2,1 ,..., i 2,4 using the following equation 3.

Number

[0068] The same process can be used for the last multi-terminal gate 308, and the output o of the second layer 2,1 ,..., o 2,3 is linked to the input i' 3,1 ,..., i' 3,3 of the logic connector 313, and the logic connector is linked to the input i 3,1 ,..., i 3,3 of the multi-terminal gate 308 through the inequality (4), and finally, the input of the multi-terminal gate 308 can be linked to the output o 3,1 through the inequality (4). The output o 3,1 corresponds to the input I ∈ {I}

Number

[0069] Thus, for each pair

Number

Number

Number

Number

[0070] Application In particular, neural networks have been adopted in applications such as data mining, image recognition, and signal recognition, and this growth in the neural network market is being driven because it is necessary to detect complex non-linear relationships between variables and patterns. In fact, there is a great need for more efficient and low-energy digital neural network architectures to support current and future computing efforts.

[0071] There are large commercial applications across various industries (e.g., fintech, i.t., life sciences, manufacturing, government and defense, transportation logistics). The adoption of cloud-based training and edge deployment of digital neural network solutions is expected to expand mainly due to advantages such as easy maintenance of the generated data, cost-effectiveness, scalability, and effective management. The digital neural networks described herein have the potential to bring about significant changes in this market and provide strategic competitive advantages to their early adopters.

[0072] Evaluation of Digital Neural Networks Using training to find the configuration of the logic connector 110 that maps the function f to the digital neural network 100, the digital neural network 100 can be programmed such that the state of the logic connector 110 (e.g., the state of the switch 116) is set according to the training result. In an embodiment where the digital neural network 100 can be retrained, the update of the training of the digital neural network 100 can be implemented by simply reprogramming the state of the switch 116. This enables, for example, offline training when more labeled data is available and easy reconfiguration of the switch 116 to implement the update. Once the switch 116 is set, the digital neural network 100 is ready to be evaluated for any possible input x. By design, the evaluation of the digital neural network 100 can be executed in a single clock cycle. Furthermore, using CMOS technology to design the multi-terminal logic gate 108, the power and energy for evaluation are extremely small, orders of magnitude smaller than current neural networks. In fact, since there is no movement of data from memory to the processing unit (e.g., for the digital neural network 100), CMOS technology enables the implementation of extremely low-power multi-terminal logic gates 108.

[0073] The theoretical aspects and practical performance of the digital neural network 100 can be analyzed against established benchmarks. The size regarding the complexity of gates, transistors, and interconnections for a specific application (e.g., classification, predictive analytics, image recognition, etc.) can be quantified. For this purpose, for a given topology and input / output size, the minimum number of layers to achieve and / or guarantee universality can be determined. The training efficiency can be evaluated regarding the accuracy measured by established benchmarks. To efficiently train the digital neural network 100, a virtual memory computing machine can be used. In some embodiments, the training can be performed not only on the complete training set but also using mini - batches to reduce the size of the ILP to be solved, as there is a possibility of accelerating the training under specific conditions. The performance regarding energy / output and speed can be evaluated. This can be evaluated by considering the implementation in CMOS technology.

[0074] Trained digital neural network As described herein, the digital neural network 100 can be trained by solving a first set of ILP problems to determine a first set of states of the logic connector 110 that map the first function f 1 to the digital neural network 100. The digital neural network 100 can also be retrained to map the second function f 2 by solving a second set of ILP problems corresponding to the second function f to determine a second set of states of the logic connector 110. 2 Thus, the digital neural network 100 can be retrained to implement substantially any function f by generating a corresponding set of ILP problems.

[0075] However, in certain applications, it may not be necessary to retrain the digital neural network 100. For example, if the digital neural network 100 is designed to implement a single function f, there is no need to retrain the digital neural network 100. Therefore, each logic connector 110 need not include each parallel path (e.g., the inverter 118, open circuit 120, and short circuit 122 of FIG. 2) or switch 116. For example, if the trained state of a given logic connector 110 is the inverter 118 path, the logic connector 110 can include only the inverter 118 without any of the other components. By implementing the logic connector 110 with only the components corresponding to the trained state of the logic connector 110, the digital neural network 100 can be implemented with a significantly smaller number of components.

[0076] In certain embodiments, the digital neural network 100 can be embodied on a single chip, for example, when incorporated into a particular device (e.g., an autonomous vehicle). Since the digital neural network 100 can be fully implemented in CMOS technology, the digital neural network 100 can be more easily implemented on a chip than other neural networks that rely on the von Neumann architecture. Advantageously, this allows the digital neural network 100 to be more easily adopted in a variety of different applications compared to conventional neural networks.

[0077] The digital neural network disclosed in this specification can provide high-speed computing. The digital neural network can be embodied on a single chip. In a specific example, an input can be loaded, the digital neural network can calculate a function in a single clock cycle, and then the output can be read out. This represents a significant improvement in speed compared to the calculation of specific existing neural network functions. The digital neural network disclosed in this specification also uses almost no energy compared to specific existing neural network calculations.

[0078] Conclusion The foregoing disclosure is not intended to limit the present disclosure to the exact forms disclosed or to a particular field of use. Accordingly, various alternative embodiments and / or modifications to the present disclosure, whether explicitly described or implied herein, are considered possible in light of the present disclosure. Although embodiments of the present disclosure have been described thus far, those skilled in the art will recognize that changes can be made in form and detail without departing from the scope of the present disclosure. Accordingly, the present disclosure is limited only by the claims.

[0079] In the foregoing specification, the present disclosure has been described with reference to specific embodiments. However, as will be understood by those skilled in the art, the various embodiments disclosed herein can be modified or otherwise implemented in various other ways without departing from the spirit and scope of the present disclosure. Accordingly, this description should be regarded as illustrative and is for the purpose of teaching those skilled in the art how to make and use the various embodiments. It should be understood that the forms of the disclosure shown and described herein are to be construed as representative embodiments. Equivalent elements, materials, processes, or steps can be substituted for the elements, materials, processes, or steps typically illustrated and described herein. Further, certain features of the present disclosure can be utilized independently of the use of other features, although all of these will become apparent to those skilled in the art after obtaining the benefit of this disclosure. Expressions such as "comprising," "including," "incorporating," "consisting of," "having," "being," etc., used to describe and claim the present disclosure are to be construed in a non-exclusive manner, i.e., intended to allow for the presence of items, components, or elements not explicitly recited. References to the singular are also to be construed as relating to the plural.

[0080] Furthermore, the various embodiments disclosed herein should be construed in an illustrative and explanatory sense only and should in no way be construed as limiting the present disclosure. All references to couplings (e.g., attachment, fixation, coupling, connection, etc.) are used only to assist the reader's understanding of the present disclosure and, in particular, do not impose limitations with respect to the position, orientation, or use of the systems and / or methods disclosed herein. Accordingly, references to couplings should be construed broadly. Furthermore, references to such couplings do not necessarily imply that two elements are directly connected to each other. Additionally, without limitation, all numerical terms such as "first," "second," "third," "primary," "secondary," "main," or any other ordinary and / or numerical terms should also be construed only as identifiers to assist the reader's understanding of the various elements, embodiments, variations, and / or modifications of the present disclosure and, in particular, do not impose any limitations with respect to the order or preference of any element, embodiment, variation, and / or modification with respect to another element, embodiment, variation, and / or modification or beyond another element, embodiment, variation, and / or modification.

[0081] One or more of the elements shown in the overall / detailed drawings may also be implemented in a more separated or integrated manner, or in certain cases removed or rendered inoperable, as may be useful depending on the particular application.

Claims

1. A neural network circuit, comprising: a plurality of logic gates arranged in a plurality of layers, each of the logic gates having a plurality of inputs and outputs; a plurality of logic connectors arranged between each pair of adjacent layers, each of the logic connectors determining a relationship between the output of a first logic gate and one of the plurality of inputs of a second logic gate, and each of the logic connectors having one of a plurality of different logic connector states; and the neural network circuit is configured to be trained to perform the function by finding a set of the logic connector states for the logic connectors such that the neural network circuit performs the function. A neural network circuit.

2. The neural network circuit according to claim 1, wherein the logic connector states include a first state in which the output of the first logic gate is connected to the input of the second logic gate via a NOT gate, and a second state in which the output of the first logic gate is connected to the input of the second logic gate via a short circuit.

3. The neural network circuit according to claim 2, wherein the logic connector states further include a third state in which the output of the first logic gate is connected to the input of the second logic gate via an open circuit.

4. The neural network circuit according to claim 1, wherein the logic gates and the logic connectors are implemented in complementary metal oxide semiconductor (CMOS) technology.

5. The neural network circuit according to claim 1, wherein the neural network circuit is formed on a single chip.

6. Each of the logic connectors includes: an input; a short circuit connected to the input; an inverter arranged in parallel with the short circuit and connected to the input; an output; and at least one switch configured to connect one of the short circuit, the inverter, and the open circuit to the output. The neural network circuit according to claim 1.

7. Each of the logic connectors further includes an open circuit connected to the input, and the at least one switch is configured to connect one of the short circuit, the inverter, and the open circuit to the output. The neural network circuit according to claim 6.

8. The at least one switch includes a first switch and a second switch connected in series, The first switch is configured to be electrically connected to one of the short - circuit circuit and the inverter, The second switch is configured to operate in either an open - circuit state or a short - circuit state The neural network circuit according to claim 6.

9. Each of the logic connectors includes at least one of a short - circuit circuit or an inverter, Each of the logic connectors connects the output of the logic gate in the previous layer to the input of the logic gate in the current layer, The neural network circuit according to claim 1.

10. Each of the logic gates includes a multi - terminal NOR gate, the neural network circuit according to claim 1.

11. The neural network circuit according to claim 1, further comprising a training circuit configured to generate the set of the logic connector states.

12. Each of the logic connectors has one of the fixed ones of the logic connector states, the neural network circuit according to claim 1.

13. Each of the logic connectors includes one of a short - circuit circuit, an inverter, and an open - circuit corresponding to the fixed one of the logic connector states, the neural network circuit according to claim 12.

14. A method of calculating a function using a neural network circuit, comprising: Providing a neural network circuit including a plurality of logic gates arranged in a plurality of layers, each of the logic gates having a plurality of inputs and outputs, and a plurality of logic connectors including a set of logic connectors arranged between each pair of adjacent layers, each of the logic connectors having one of a plurality of different logic connector states, and the plurality of logic connectors being programmed to perform a function; Calculating the function for an input signal using the neural network. Including the step of providing a neural network circuit; And the step of calculating the function for the input signal using the neural network. A method.

15. The method according to 14, further including the step of finding a set of the logic connector states for the plurality of logic connectors such that the neural network circuit performs the function. Including the step of finding a set of the logic connector states for the plurality of logic connectors such that the neural network circuit performs the function.

16. generating a set of integer linear programming (ILP) problems based on the function; solving the set of ILP problems to generate the set of logical connector states; The method according to claim 15, further comprising.

17. determining a set of inequalities describing the states of the logical connectors; linking the output from the previous layer to the input of the subsequent layer via the set of inequalities; further comprising the step of generating the ILP problem is based on the step of linking the output from the previous layer to the input of the subsequent layer via the set of inequalities; The method according to claim 16.

18. The method according to claim 14, wherein the logical connector state includes a first state in which the output of the first logic gate is connected to the input of the second logic gate via a NOT gate, and a second state in which the output of the first logic gate is connected to the input of the second logic gate via a short circuit.

19. A single chip, a plurality of logic gates arranged in a plurality of layers of a neural network circuit, each of the logic gates having a plurality of inputs and outputs; a plurality of logical connectors arranged between each pair of adjacent layer logical gates, each of the logical connectors determining the relationship between the output of a first logical gate and one of the plurality of inputs of a second logical gate, each of the logical connectors having one of a plurality of different logical connector states; a plurality of input terminals; at least one output terminal comprising the chip is configured to calculate a function between the input terminal and the output terminal; A single chip.

20. The chip according to claim 19, wherein the chip is configured to calculate the function for a given input provided to the input terminal in 10 clock cycles or less.

Images