Variational inference circuit, electronic neuron comprising the same and manufacturing method thereof

Abstract

A circuit with hardwired electronic components that perform mathematical operations such as a generalize product without executing programming instructions during the calculation of the mathematical operations allows you to generate a variational message passing solution of a multiparametric mathematical problem factorizable according to Forney-factor. Such a circuit can be part of a circuit neuron with which neuronal networks can be created.

Description

[0001] " Variational inference circuit , electronic neuron comprising the same and manufacturing method thereof"

[0002] DESCRIPTION

[0003] TECHNICAL FIELD OF THE INVENTION

[0004] The present invention refers to the hardware implementation of a circuit without a processor, a microcontroller or any hardware executing programming instructions , to carry out Bayesian inference based on message passing, understood as the estimation of probability distributions through deterministic approximation, through the algorithm called Variational Message Passing (VMP ) and to automate a Bayesian inference , such as for example to achieve the minimi zation of the variational free energy (VFE ) and to an electronic neuron comprising such a circuit . More speci fically, the present invention refers to a circuit architecture that implements a plurality of electronic neurons made in hardware and connected to each other .

[0005] PRIOR ART

[0006] It is known to use generative algorithms of synaptic inputs including a variational free energy (VFE ) minimi zation step, as example of a VMP algorithm, to create neuronal networks comprising connected neurons . It is known to emulate in hardware such a neuronal network by connected microcontrollers wherein each microcontroller is programmed to execute instructions of the algorithm

[0002] . However, it does not appear to have been implemented in a dedicated electronic circuit i . e . in the absence of processors , controllers or any hardware capable of executing programmed instructions through the automatic decomposition of such instructions into a plurality of executed tasks e . g . by the processor and whose individual results are processed to provide execution of the instruction . It is not known an electronic circuit with the above features that performs message-passing-based Bayesian inference , nor is there known a methodology for designing an electronic circuit starting from the mathematical description of the message-passing problem to be solved .

[0007] The use of VMP as a discrete- or continuous-valued Bayesian inference technique for free energy minimi zation represents an ef fective approach for solving complex computational problems . For example , a publicly accessible resource has been developed that allows the generation of message-passing-based Bayesian algorithms starting from the definition of a generic problem [ 3- 4 ] .

[0008] It is therefore very useful to have a dedicated electronic circuit available that can perform Bayesian inference based on message-passing so that such approximate inference technique can be used with low energy consumption and high performance . The possibility of having an electronic circuit that can execute the inferential process in hardware ( and not through interpretation during the execution of the calculation cycles of programming instructions via a microcontroller, microprocessor or other programmable machine ) , speci fically allows the circuit-based reali zation, as a non-limiting example , of corresponding neural networks suitable for solving complex problems with high performance and low energy consumption .

[0009] SCOPE AND SUMMARY OF THE INVENTION

[0010] The scope of the present invention is to reali ze an electronic circuit that , without executing programmed instructions and avoiding having to perform iterative calculations , implements in hardware the mathematical formulation of a problem solvable through the VMP approach .

[0011] The VMP approach can be used for example in the mathematical description of the activity of a neuron, modeled as a Bayesian inference activity . In this case the electronic circuit coincides with the circuit of an inferential electronic neuron capable of encoding the expectations regarding the causes that generate the synaptic inputs . This circuit solves a variational problem in message-passing in a non-iterative way through a dedicated circuit that implements the mathematical operation of convergence to an equilibrium value of some variables that characteri ze the electronic circuit .

[0012] A further scope of the present invention is to create an electronic circuit , preferably an electronic neuron circuit , comprising a circuital branch or node wherein the associated electric signal ( e . g . current or voltage ) converges , through VMP algorithm, to an exact or approximate solution of the problem, an example of which is the minimi zation of the free energy defined as

[0013] VFE = / WQCOIPCWJ) - lnp(Ot) where DKL is the Kullback-Leibler operator, q and p a priori and a posteriori probability distributions , respectively, Ot synaptic inputs received and St the hidden states , so q represents the probability that a hidden state St generates the input received by the neuron . Such circuit solves a variational problem and, in particular, the prediction of synaptic inputs in an inferential neuron model . More generally, it is possible to apply this circuit to reali ze convergence to an approximate solution through the VMP algorithm .

[0014] The scope of the present invention is achieved through an electronic circuit designed according to claim 1 .

[0015] BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The invention is described below on the basis of nonlimiting examples illustrated by way of example in the following figures , which respectively refer to : Fig . la, lb, 1c show Forney graphs of the mathematical problem defined by equations 7- 8 ( Figg , la and lb ) and by equations 9- 10- 11 ( Fig . 1c ) ;

[0017] Fig . 2a, 2b, 2c, 2d, 2e show the block diagram of the computational core of an VMP algorithm according to Forney Graph in Fig . lb ;

[0018] Fig . 3a, 3b, 3c show block diagram of a computational core for an VMP algorithm according to Forney graph of Fig . lb including the memory element and connections to the computational core thereof ;

[0019] Fig . 4 shows two possible embodiments of the circuit according to Fig . 3c ;

[0020] Fig . 5 shows the block diagram of a single electronic neuron including a computational core according to one of Fig . 1 - 4 , a pre-proces sing block processing the inputs and a stochastic decisional output block;

[0021] Fig . 6 shows a VMP mathematical model of a neuron;

[0022] Fig . 7 shows the functional block diagram of the preprocessing circuit block with n multiplicity of outputs in the embodiment of an electronic neuron;

[0023] Fig . 8 shows a possible functional block diagram of the preprocessing circuit block with 2 multiplicity of outputs in the embodiment of an electronic neuron;

[0024] Fig . 9 shows a possible functional block diagram of the pre-processing circuit block with 2 multiplicity of outputs where the possible implementation of the internal blocks ai...anis explained;

[0025] Fig . 10 and 11 show a possible functional block diagram of the implementation of block b inside the pre-processing circuit block;

[0026] Fig . 12 shows a possible functional block diagram of the multiplier internal to the pre-processing circuit block; Fig . 13 shows the functional decision block diagram of the electronic neuron output block;

[0027] Fig . 14 shows a plurality of possible analogic embodiments of the logarithm circuit ;

[0028] Fig . 15 shows a plurality of possible analogic implementations of the exponential circuit ;

[0029] Fig . 16 shows a possible analogic implementation of the multiplier / divider block;

[0030] Fig . 17 shows a plurality of possible analogic implementations of the softmax block;

[0031] Fig . 18 shows a plurality of possible analogic implementation of the weighed sum circuit ;

[0032] Fig . 19 shows a possible analogic implementation of a normali zing circuit ;

[0033] Fig . 20 shows a possible analogic implementation of a multiplicative inverse circuit ;

[0034] Fig . 21 shows a possible analogic implementation of the subtractor circuit ;

[0035] Fig . 22 shows a possible analogic implementation of the vector-to-matrix multiplier circuit ;

[0036] Fig . 23 shows a possible analogic implementation of the circuit that simultaneously implements the softmax and product operations for a constant matrix ;

[0037] Fig . 24 shows a possible analogic implementation of the memory circuit or memory element ;

[0038] Fig . 25 shows a possible mixed digital / analog implementation of the output decision block;

[0039] Fig . 26 shows a possible analogic implementation of the function g that approximates the softmax function;

[0040] Fig . 27 shows a possible analogic implementation of the multiplier circuit present in the pre-processing circuit for inputs ; Fig . 28 shows a schematic of a neural network to simulate a delayed eyeblink protocol ;

[0041] Fig . 29 shows a schematic of a neural network that encodes a plurality m of inputs and returns a plurality n of outputs ;

[0042] Fig . 30 illustrates a computational core comprising a series of n connected circuits

[0043] Fig . 31 shows a scheme of the circuit in Fig . 30 where n=2 ;

[0044] Fig . 32 shows a block diagram of Fig . 3c according to the scheme in Fig . 31 .

[0045] DETAILED DESCRIPTION OF THE INVENTION

[0046] A VMP algorithm is a technique for solving a very large class of mathematical problems . It is based on variational inference performed by passing messages in nodes that implement the description of the variational process thanks to local operations on each node . The VMP optimi zes a variational limit using a series of local computations obtained thanks to the passing of messages between said nodes .

[0047] Among the various possible applications , the excitability of the biological neuron can be modelled as an optimi zation process based on the minimi zation of a cost function to be understood as variational free energy (VFE ) through a VMP algorithm

[0001] . The minimi zation of the variational free energy allows the updating of the internal state of the neuron consistently with the graph schemati zation of the VMP algorithm reported in Fig . 1 .

[0048] In this example , an inferential model of the synaptic inputs received by a neuron at a certain instant of time ( ot according to the nomenclature of Fig . la ) can be described in terms of relations between the internal and external states of a system by means of likelihood matrices A defined as the probability of observing a given outcome based on the values of the system input parameters ( ot) and previous beliefs about the internal states ( st) that allows decisions to be made even in the presence of limited information or uncertainty . This implies that the inferred value of the internal state can lead the neuron to one of the two alternative states : on or of f . The adoption of an inferential model of this type therefore provides for modelling the excitability of a single neuron as a random event whose posterior probability distribution ' a' is updated on the basis of the inputs received . In particular, inference can be reformulated as an optimization process , where the belief update corresponds to the convergence of the free energy towards a local minimum, using an iterative descent of the VFE gradient . The neuron therefore acts as an inference element and at each point in time the expectation of the internal states is updated through an error ( st) . The expectation of the internal states is associated with the membrane potential (vt) of the neuron as : vt= ln(st) (Eq. 1)

[0049] In the speci fic example of the VMP algorithm applied to the case of the neuron, the mathematical formulation of the problem comprises the following equation, whose solution provides the estimate of the internal states ( st) : where B is the transition matrix (possibly also a function of time ) that indicates the probability that the internal state evolves , over time , into another . The solution consists in estimating the value of the internal states ( st) that makes the equation above true . It can be expressed in variational terms by rewriting the equation by introducing the prediction error of the state ( st ) , which is equal to the free energy gradient . st= a(vt) (Eq.5)

[0050] In the above equation, the softmax operator ( o) is a sigmoidal activation function modeling the depolari zation responsible for the neuronal action potential . This explains the belief updating and neuronal dynamics during each time step, i . e . inference . The message passing representation of the problem via a Forney factori zation graph is illustrated in Figure la .

[0051] Each node performs a speci fic function ( e . g . sum-product algorithm) of the variables represented by the vertices converging to it (blue labels in Fig . la ) and of the input messages ( orange arrows ) to provide an output message , optionally divided by an arbitrary constant . In the context of the schemati zation in terms of a graph and the solution of a given problem (which includes the determination or choice of an ordered sequence in which messages are exchanged on the vertices of the graph) , such as the one depicted above in Fig . la, the graph can always be reformulated in such a way as to avoid the presence of vertices with respect to which there is an interest in calculating messages in both directions , and at the same time it is obvious for the skilled man to carry out such reformulation, knowing the operations performed by the various nodes of the graph and the ordered sequence in which messages are exchanged on the vertices of the graph . In the speci fic example , the graph in Fig . la can be reformulated as in Fig . lb . It should be noted that the present invention applies in all cases in which an mathematical problem described by a Forney factori zation graph presents a graph in which, for each vertex, the messages are calculated in a single direction . Fig . lb illustrates the rewriting of Fig . la in this sense .

[0052] In Fig . lb, the messages exchanged between the various nodes of the graph have also been made explicit by means of orange arrows. It can be seen how the input parameters to the system (o) , which represent external observable variables, are translated by means of the likelihood matrices A into messages (x) that contribute together with the others highlighted in Fig. lb to predict the internal state of the system at time t (st) . xj,t=Aj°j,t (Eq.6)

[0053] This means that, without losing generality, the quantities relevant from a mathematical point of view are the messages that encode the inputs to the system (x) and those relating to the internal state of the system (s) .

[0054] It is emphasized that, in the formal approach reported above (both in terms of equations and in equivalent terms of graph) the message passing aimed at predicting the internal state at time t (st) includes the evaluation of the internal state at the past time t-1 (st-i) and a hypothesis on the future internal state at time t+1 (st+i) . In the example, the hypothesis on the internal state at time t+1 (st+i) is formulated in an identical way to the prediction of the internal state at time t (st) , however, ignoring the hypothesis on the internal state at time t+2, i.e., it is assumed that the message associated with the vertex relating to the variable st+2 is equal to 1 (as is known, see [5] ) . In the equation formulation, this is reflected in the need to iteratively solve two coupled equations: so as to converge to the prediction value of the internal state at time t (st) that minimizes the prediction error (st) . Typically, the iterative process stops when the prediction error (st) is less than some arbitrarily fixed threshold.

[0055] Note that the formal approach to the problem can be extended so that the message passing aimed at predicting the internal state at time t (st) includes the evaluation of the internal state also at times even earlier than the past time t- 1 ( st-i ) , i . e . the evaluation of the internal states at time t-2 ( st-2 ) , t- 3 ( st-3 ) , t-n ( st-n) • An example of such an extension, limited to time t-2 , is reported in the graph in Fig . 1c which shows its reformulation in such a way as to avoid the presence of vertices with respect to which there is interest in calculating messages in both directions .

[0056] In the equation formulation, this is reflected in the need to iteratively solve three coupled equations :

[0057] More generally, for a generic problem to be solved, the variables that populate the graph are partly unknown and that is , they must be inferred . I f the graph is cyclic, like those in Fig . la- lc, the inference is obtained through the iterative resolution of a system of two or more coupled equations , or of multiple instances of such a system .

[0058] Although " software" type solutions of VMP algorithms are known through the use of processors , microprocessors , microcontrollers or any hardware capable of executing programmed instructions automatically divided into simpler tasks whose results are subsequently processed as execution of the instruction, do not result in the state of the art an electronic circuit for solving mathematical problems via VMP algorithm without executing programmed instructions and avoiding having to perform iterative calculations to solve the coupled equations nor a methodology for designing an electronic circuit starting from the mathematical description of the problem to be solved in terms of message-passing . The circuit implementation described below can also be implemented using programmable or reprogrammable hardware ( intended as the ability to program or re-program the hardware and not as the ability to execute programmable instructions , i . e . , software programmability) such as FPGAs . However , this solution involves a higher computational latency caused by the reduced parallelism available on such hardware , higher energy consumption and higher system cost in the face of reduced design complexity .

[0059] In the present invention the electronic circuit is reali zed through the interconnection of speci fic blocks of electronic hardware components as defined in the preceding paragraph, each of which is associated with a speci fic node of the graph that represents the problem to be solved . The interconnections between the blocks follow those speci fied in the graph in terms of message passing .

[0060] Given that :

[0061] A Forney graph associated with a generic problem solvable by the VMP algorithm has k nodes , and that the generic node is identi fied with the symbol n±, with i ranging from 1 to k;

[0062] A number v± of vertices converges at node n±, and that the graph has been formali zed in such a way that v±- l vertices are associated with messages exclusively in input to the node ( input vertices ) and 1 vertex is associated with the message outgoing from the node ( output vertex ) , i . e . , the graph does not contain nodes to which vertices converge for which there is interest in calculating messages both in input to , and output from, the node ; a graph can always be formali zed in such a way as to satis fy this constraint ;

[0063] Node n± operates a known local function f± of the variables identi fied by the v±- l input vertices and of the messages in input to such vertices to determine the message outgoing from the node , associated with the output vertex ; then within the scope of the present invention it is possible to associate with each node n± an electronic block-node (consisting of electronic components that, overall, solve mathematical functions without processing software programming instructions during the resolution) bi having: a number of inputs equal to v±-l, characterized by electrical signals that are representative of the messages entering the node; a number of outputs equal to 1, the latter characterized by an electrical signal that is representative of the message exiting the node; an electronic circuit having at most all the inputs of the block as input and the output of the block as output, and realizing the function f±;

[0064] The functions f± are to be understood as those mainly used in the resolution of probabilistic problems, i.e. aimed at determining the joint probability density of random variables, i.e. the functions of product, generalized product, sum, weighted sum.

[0065] It is specified that in this document and in the claims, by : product, it is intended any mathematical function of at least n variables f (xi, ..., xn) whose value in at least one noninfinitesimal region of its domain differs by no more than a factor of 5 from generalized product, it is intended any mathematical function of at least 2n variables f (xi, ..., xn, ai, ..., an) whose value in at least one non-inf initesimal region of its domain differs by no more than a factor of 5 from sum, it is intended any mathematical function of at least n variables f (xi, ..., xn) whose value in at least one noninfinitesimal region of its domain differs by no more than a factor of 5 from weighed sum, it is intended any mathematical function of at least 2n variables f (xi, xn, ai, an) whose value in at least one non-inf initesimal region of its domain differs by no more than a factor of 5 f rom 52Xi at ■ xtsubtraction, it is intended any mathematical function of at least 2 variables f (xi, x2) whose value in at least one noninfinitesimal region of its domain differs by no more than a factor of 5 from x1—x2, i.e., logarithm to base n, it is intended any mathematical function of at least 1 variable f (xi) whose value in at least one non-inf initesimal region of its domain differs by no more than a factor of 5 from raising to a power of base n, it is intended any mathematical function of at least 1 variable f (xi) whose value in at least one non-inf initesimal region of its domain differs by no more than a factor of 5 from nX1, i.e., ratio, it is intended any mathematical function of at least 2 variables f (xi, x2) whose value in at least one noninfinitesimal region of its domain differs by no more than a factor of 5 from the value its domain deviates by no more than a factor of 5 from multiplicative inverse, it is intended any mathematical function of at least 1 variable f (xl) whose value in at least one non-inf initesimal region of its domain deviates by no more than a factor of 5 from

[0066] Fig. la-lc show Forney graphs for achieving the convergence of a VMP algorithm in which a difference between an observed value of predetermined parameters and a prediction of such values , performed by means of a mathematical model , must converge to a stable and minimum value . In the following, an embodiment of how to implement an electronic circuit capable of achieving such convergence and calculating an output value when such convergence is achieved will be described .

[0067] Fig . 2a shows a block diagram of a possible implementation in terms of an electronic circuit of the VMP algorithm applied to the problem illustrated in the graphs of Fig . la and lb . Speci fically, in Fig . lb a speci fic portion of the graph diagram, the one dedicated to solving the mathematical problem related to the coupled equations ( 7 ) and ( 8 ) , has been highlighted by a dotted border that is also found in Fig . 2a to indicate the corresponding electronic block-node . Each block, with respect to the node of the graph to which it is associated, respects the criteria stated above , in particular that each electronic blocknode has a single output message . The correspondence , also in terms of nomenclature , between the information input to the graph and that input to the circuit and how the quantities input to and output from the various electronic block-node actually represent the various messages to the associated vertices in the graph .

[0068] Fig . 2b shows a greater detail than that of Fig . 2a . Speci fically, the internal structure of each electronic blocknode is highlighted in terms of functionality, that is , the mathematical relationship established between the input signals and the single output signal , which is obviously identical to that established by the relative node of the Forney graph in Fig . lb between the input messages and the single output message . Speci fically, as known in literature , the function f± associated with a node of type ( = ) in Fig . lb makes the output message correspond to the product of the input messages , possibly dividing the output by an arbitrary factor (normali zation) . This is reported schematically within block bi, which highlights a multiplier that takes the input signals to the block as input and returns as output a quantity proportional to their product. As shown in Fig. lb, it is appropriate and convenient to generalize the input product operation to the arbitrary power product operation of the inputs (hereinafter, generalized product) , so as to implement, if useful, a different relative weight for the various input messages to the node of type (=) . This known generalization, which realizes the variant of VMP known as marginal message passing [6] , can be useful to improve convergence to the solution, whether exact or approximate. It should be noted that the choice of the specific circuit implementation of the multiplier as well as of the other elements composing the block and in general any circuit block is known to the expert in the electronics branch and can be made on the basis of specific circuit design criteria and system constraints (e.g., based on the nature of the input and output signals, i.e., voltages or currents) . Non-limiting examples will be provided later in this document. The output of the generalized multiplier enters a normalizing circuit that will output the signal at its input divided by an arbitrary factor (i.e., normalization function) . It should be noted that this operation is often useful especially when the input and / or output signals have a vectorial nature rather than a scalar one, as in the example at stake. In this case, each input and / or output signal will be understood as (and represented by) a plurality of signals and corresponding tracks. One possible choice for the normalization factor is represented by the norm of the vector signal at the input of the normalizing circuit. The function f± associated with a node of type (Bt) or (BtT) in Fig. lb makes the output message correspond to the product of the input message by a predetermined constant, identified by the name of the node itself, possibly normalizing the output. This is reported schematically in block b2 where a multiplier is highlighted that takes the input signal to the block as input and returns as output the product of the same by a constant of value Bt . The same output enters a normali zing circuit that will operate in an identical way to what was discussed in relation to block bi . In the case of scalar inputs , Bt represents a scalar number while for vector inputs Bt will be represented by a matrix . Finally, we note how block ba is structurally identical to block bl because the nodes associated with them :

[0069] - are characteri zed by identical functions ( i . e . they are both nodes of the same type ) ; receive the same number of vertices carrying input messages as input ;

[0070] For the same reason, block ba is structurally identical to block b4 . The only di f ference is that in block b4 the multiplicative constant takes the value BtTinstead of Bt . In the simpli fying hypothesis of considering the multiplicative constant as a scalar or, in case , a unitary or symmetric matrix, then BtT= Bt and the two blocks ba and b4 are identical . Finally, in block ba, the unitary input has been highlighted with a rounded dashed rectangle to indicate that it is optional , as it is a neutral element for multiplication .

[0071] Fig . 2c shows a non-limiting embodiment of the generali zed multiplier within blocks bi and ba shown in Fig . 2b . Speci fically, the output of the generali zed multiplier is obtained by exploiting the properties of logarithms and powers , by raising to a power of base n the weighed sum of the logarithms of the multiplier inputs to a base n, where n is an arbitrary number . The circuits identi fied with " logn" and "nx" represent circuit embodiments that generate output signals that are proportional to the logarithm of base n (hereinafter, logarithmic circuit ) and to the power of base n (hereinafter, exponential circuit ) of the respective input signals . The circuit identi fied with is a circuit manufactured to generates an output signal proportional to the weighed sum of the input signals (hereinafter, weighted sum circuit ) . Finally, in blocks b2 and b4 , the normali zing circuits have been highlighted with rounded dashed rectangles to indicate that they are optional in the case where the characteristic multiplicative constant of blocks b2 and b4 is a unitary matrix ( the product of a normali zed vector by a unitary matrix is itsel f normali zed) .

[0072] Fig . 2d shows a speci fic implementation of the structure in Fig . 2c where n i s chosen equal to e , i . e . , the number of Neper . In this case , the combination of the exponential circuit and the normali zing circuit can be implemented through a single circuit that generates an output signal proportional to the softmax function ( o) of the input (hereinafter, softmax circuit ) .

[0073] Fig . 2e shows how to group the electronic block-nodes of the system into electronic subcircuits , each of which includes one or more electronic block-nodes . In addition, a speci fic implementation of the structure in Fig . 2d is highlighted in which the electronic circuits responsible for generating output signals proportional to the logarithm in base n have been replaced by electronic circuits responsible for generating output signals that are a precise function ( f O , f l , ..., fm+ 1 ) of the input signal such that this function approximates a function proportional to the logarithm in base n of the respective input signal : with c being an arbitrary constant . Similarly, the circuit designed to generate an output signal proportional to the softmax function ( o) of the inputs has been replaced by an electronic circuit designed to generate an output signal that is a precise function ( g) of the input signal such that this function approximates a function proportional to the softmax function of the respective input signal : gM ~ c ■ a(x) with c being an arbitrary constant .

[0074] Fig . 3a shows the content of Fig . lb where , however, the input message called Bt-iSt-i (which represents information relating to the internal state at the past time t- 1 ( st-i ) ) is obtained through the output of an electronic memory element that receives as input and stores the message called BtSt and supplies it as output at the next time while it supplies as output at the current time the message stored at the previous time . It should be noted that the naming of the messages is consistent with the time flow . Consistently with the above , Fig . 3b shows a block diagram of a possible implementation in terms of an electronic circuit in hardware . The circuit , highlighted by a dotted rectangular border, shows a speci fic additional output , called y, and coinciding with the output of block ba .

[0075] Fig . 3c shows a possible speci fic implementation in terms of an electronic circuit without microcontrollers to execute programming instructions during the calculation cycle of the structure in Fig . 3b in line with what is shown in Fig . 2d . The signal present at the output of sub-circuit A, coinciding with the input of the memory element and with one of the inputs of sub-circuit B, is called z , while the signal present at the output of the memory element and coinciding with one of the inputs of sub-circuit A, is called xo . Finally, the signal present at the output of sub-circuit B, coinciding with one of the inputs of sub-circuit A, is called xm+i .

[0076] In the context of the mathematical problem at stake and of its formulation in terms of variational free energy, by minimi zation / convergence of the variational free energy it is intended the updating of the state value z predicted in a current cycle and calculated on the basis of input signals received in a such that its discrepancy or deviation with an internal state Xo of the system calculated in a past cycle is minimi zed / converges to a minimum value . In the application but not limiting case of an electronic circuit representing a neuron, when this neuron is connected within a network the input signals can be the output signals generated by neurons positioned upstream and the generated output can become the input signal for neurons located downstream in the network .

[0077] As shown in Figg . 2d and 3c and coherently with the Forney graph in Fig . lb, such a minimi zation / convergence is achievable via a circuit configured in at least one circuit ring or loop to receive as input the plurality of normali zed signals relating to a current processing cycle and to at least one past processing cycle and a parameter Xo representative of the internal state z ( state variable ) of the neuron in order to produce an output value Y representative of the probability of activation of the neuron and an update of the internal state .

[0078] The normali zation of signals e . g . within the loop of the Forney graph of Fig . lb and implemented in the circuit of Fig . 2e within the dashed box, allows to make the convergence process sel f-consistent , scaling the dynamic range of each signal in a predefined range for all normali zed signals .

[0079] According to the present invention and unlike what is known in the state of the art , the computational block has been implemented in circuit in order to include a minimi zation / convergence circuit capable of minimi zing the VFE ( or in general of achieving the convergence of a VMP algorithm) and therefore to solve variational problems .

[0080] The ability to solve variational problems arises from the need to be able to model and describe many phenomena in all fields of science , from a mathematical point of view, in terms of maximum or minimum principles . For example , the stable equilibrium configurations of a mechanical system subj ect to conservative forces are those that minimi ze the potential energy, or in a transparent medium, the ray of light travels between two given points choosing, among all the possible , the traj ectory along which it takes the least time .

[0081] Other times , however, the problem of controlling a phenomenon can be posed, or an event , to force it to adapt to certain maximum or minimum requirements . For example , in the design step of a certain structure , one would like to build it according to criteria of minimum cost , minimum weight , maximum resistance or minimum heat dispersion, etc . which determine the choice of materials to be used .

[0082] When the solution to a variational problem cannot be formulated analytically in an exact way, the problem arises of determining it with a certain degree of approximation using numerical methods that the advent of the computer has made possible .

[0083] The circuit implementing such a convergence , coherently with Fig . 2e , includes a first subcircuit A receiving as input the plurality of normali zed inputs (Xi,t-i, Xm,t-i ) which refer to a past processing cycle having in series a functional block in which each of the input signals (Xi,t-i, Xm,t-i ) is processed by an arbitrary function fi, . . . , fm, already described, a weighed adder, an arbitrary function g already described, possible normali zer ( coherently with Fig . 2c i f not already included in function g) , multiplier, possible arbitrary function k to be intended as normali zation function ( identi fied by 'Norm. ' in Figg . 2e and 3c ) and a second subcircuit B receiving as input the plurality of normali zed inputs (Xi,t, Xm,t) which refer to a cycle of current processing having in series a functional block in which each of the input signals (Xi,t, Xm,t) is processed by an arbitrary function fi, . . . , fmalready described, a weighed adder, an arbitrary function g, normalizer, multiplier, possible arbitrary function k to be intended as normalizing function (identified by 'Norm.' in Figg. 2e and 3c) .

[0084] The circuit, in addition to the two plurality of inputs, provides that the output of the arbitrary function k to be intended as normalizing function (identified by 'Norm.' in Figg. 2e and 3c) of subcircuit B is connected by circuit to the input of subcircuit A in order to apply a further input (Xm+i) , obtained through the processing by said second branch of the plurality of current inputs (Xi,t,Xm,t) , which is processed by a further arbitrary function fm+i already described of subcircuit A.

[0085] Furthermore, the output of the arbitrary function k to be intended as normalizing function (identified by 'Norm.' in Figg. 2e and 3c) of subcircuit A is connected by circuit to subcircuit B so as to apply the signal (z) as a further input, representing the internal state of the minimization circuit as already indicated in the previous paragraphs, which is processed by a further arbitrary function fo already described of subcircuit B. The output of the arbitrary function k to be intended as normalizing function (identified by 'Norm.' in Figg. 2e and 3c) of subcircuit A is also connected by circuit via the function h() to be intended as memory function (identified with 'memory' in Fig. 3c) to the input of subcircuit A in order to provide the further input Xo which is processed by a further arbitrary function fo already described of subcircuit A. Similarly, even if not shown in the figure, also the value calculated in subcircuit B representing the probability of activation of the neuron for the current cycle (y) can be stored via an arbitrary function j () to be understood as storage function.

[0086] The multiplier of subcircuits A and B is preceded by a possible normalizer, if the normalization is not already implemented in function g, which has the purpose of representing at the output the data received at the input so that they satisfy unitarity, a condition for which the output data can be, if necessary, correctly interpreted as probability, e.g. in the case of a neuron the probability that the internal state of the neuron can be active or inactive.

[0087] Fig. 4a shows a possible specific implementation in terms of electronic circuit of the structure in Fig. 3c. Specifically: the implementation of the generalized multiplier circuits for both subcircuits is that of Fig. 2e; in the weighed sum circuit of subcircuit A some weights have been chosen equal to while the others have been chosen equal to 1; in the weighted sum circuit of subcircuit B the weights have all been chosen equal to 1; the input of value 1 to the generalized multiplier circuit of subblock B has been omitted; the normalizing circuits in both subcircuits have been omitted. Fig. 4b shows how it is possible to optimize the circuit in Fig. 4a with equal functionality while saving the implementation of a logarithmic circuit. Specifically, the two logarithmic circuits highlighted in Fig. 4a by thick borders are eliminated and replaced by a single logarithmic circuit, highlighted in Fig. 4b by thick borders. It is highlighted how, in the circuit in Fig. 4b, the memory element stores a signal proportional to the logarithm of the signal z, rather than the signal z itself.

[0088] According to different embodiments, weights can be implemented in different locations of the circuit such as, e.g., upstream or downstream the adding electronic block-node, preserving the functionality of the circuit. Furthermore, in the two subcircuits A and B, the arbitrary functions previously identified as g() and multiplier in this embodiment are the exponential and product functions between vector and matrix, respectively .

[0089] In particular, the sequence of the exponential function and the normalization function coincides with the softmax function (identified with 'o' ) , which is a sigmoidal activation function suitable for modeling of arti ficial neurons . As is known, arti ficial neurons are provided with data and weights as inputs with which to calculate the weighted sum of the inputs which are converted into outputs via an activation function . Then an activation function is used to map the input to the output . This activation function is used by a network of neurons to learn complex relationships and patterns in the data .

[0090] The softmax function is not the only possible activation function, but compared to other functions such as the sigmoid, the hyperbolic tangent , or the ReLU ( recti fied linear unit ) it has the advantage of being able to capture the relationships between the inputs , emphasi zing ef fective the relatively high inputs and minimi zing the relatively weak ones . At the same time , softmax allows to convert the results into values that can directly be interpreted in terms of probability as they are normali zed .

[0091] Circuit block B represents the multiplication of the softmax output by a matrix, structured to guarantee unitarity, which identi fies the transition probability between layers .

[0092] Always in Fig . 4 and coherently with the above paragraphs , a memory element is shown in which the value of the internal state z of the neuron is saved ( in Fig . 4a, or a value proportional to its natural logarithm, in Fig . 4b ) in a past processing cycle and how the value Xo as input to subcircuit A is updated via the value present in memory at the end of the current cycle to be taken again as input , appropriately processed by a certain previously described function fo ( e . g . identi fied by natural logarithm) , of the first subcircuit A at the next cycle .

[0093] Similarly, even i f not shown in the figure , the value calculated on subcircuit B representing the probability of activation of the neuron for the current cycle ( y) can also be stored within a memory element , previously identi fied by function j ( ) .

[0094] The blocks shown in Fig . 4 are made using hardwired or electronically ( re ) -programmable circuit blocks , where by hardwired circuit it is intended a circuit whose connections are permanently determined during the design step to perform a speci fic complex function and in so that they cannot be altered and / or broken down into simpler sub- functions . This speci fic function is therefore not achieved through the execution of programmed instructions ( software ) . Said functional blocks can be implemented in hardware using at least one of an integrated circuit for speci fic applications (AS ICS ) , a programmable logic device ( PLD) such as , for example , a re-programmable integrated circuit in hardware ( FPGA) , a complex programmable logic device ( CPLD) . Furthermore , the lines illustrated in Figure 4 are electrical conduction lines for current or voltage signals .

[0095] The hardware implementation of the circuit blocks allows for better performance in terms of energy ef ficiency and processing latency, especially with AS ICs , since unlike a function that can be managed via software by a " general purpose" processor it must necessarily be interpreted and codi fied in machine language through a set o f elementary operations before it can be executed, the circuit created in dedicated hardware is instead speci fically designed to solve a speci fic function .

[0096] Functional blocks to implement an inferential neuron

[0097] Figure 5 shows an application example of the circuit in figures 1-4 , called computational core , to solve via a VMP algorithm a mathematical problem to simulate the activity of a neuron capable of updating its internal generative model , minimi zing the variational free energy, an operation performed in a core computational method appropriately developed starting from the diagrams of figures 1-4 and also to generate an output signal ( y) . The scheme in Fig . 5 is inspired by the algorithm for simulating the behavior of a neuron described in

[0001] . In addition to the computational core , there are : an additional electronic circuit called " input preprocessing" configured to receive as input a plurality of inputs coming from outside the neuron and relating to a current processing cycle and a signal representing the convergence calculated to the previous processing cycle coming from the computational core ( y) , and a further additional electronic circuit called " stochastic output decision block" generating a neuron activation signal when the value of the neuron activation probability ( y) , received as input , is greater than a second value generated by a random, pseudo-random or chaotic number generation function . Speci fically, the " input pre-processing" electronic circuit receives from the outside a multiplicity of m voltage or current input signals and provides as output at least two multiplicities of m output signals . These at least two multiplicities of m output signals constitute the input signals of the "computational core" electronic circuit . As previously noted, each input and / or output signal is to be understood as a signal potentially characteri zed by being defined by a set of values ( e . g . , a signal of vector, matrix or tensor nature ) .

[0098] Fig . 6 shows the mathematical model of inferential neuron proposed by

[0001] . Upper left panel : I llustrates a probabilistic graphical model from which observations are generated . The circles indicate hidden states (" s1" ) or observable states ( output , "o1" ) ; the squares or nodes , the probability distribution functions that generate these states ; and arrows , the causal relations that connect functions to states . Bottom left panel : mathematical description of the generative model . The first equality speci fies the form of the generative model (neural beliefs about the causal relationship between hidden causes and observations ) . This can be expressed in terms of matrices A, D and B, which respectively define the probability of observing a given output given the state of the network, the probability of the initial state of the network and probabilistic state transitions. Each A-matrix or connection between network and neuron has an associated precision, which represents its synaptic strength. Middle panel: The generative model is different from the process of generating real outputs by the neuron network. Lower left panel: each presynaptic neuron is a hidden state sit-i -where s1represents its activation state- capable of causing an o1output in the post-synaptic neuron. Upper right panel: in this Forney factor graph, the squares represent factors or conditional probabilities on random variables (hidden states) ; the edges correspond to the random variables (higher marginal probability densities) that are passed between the nodes; and equal signs connect edges that instantiate the same random variable. This scheme summarizes the model inversion process well as it shows the convergence (equals signs) of messages (edges) from all factors (squares) contributing to the inference of the same random variable. Right panel 'VMP' : messages are compared to the actual belief about the hidden state of the network. The resulting prediction error is then used to update this belief. Lower right panel: once the prediction error has been minimized, the neuron decides whether or not to generate an action potential st based on the updated belief st, which then feeds back into the network (arrow R) , generating a new observation at t+1.

[0099] To complete the embodiment wherein the minimization of the VFE is applied to an inferential algorithm of a neuron, there now follows a detailed description of what, in addition to the computational block that minimizes the VFE, is implemented in circuit, as well as for the computational block purposes of obtaining neuronal functions.

[0100] Input preprocessing block As previously described, the computational core circuit obj ect of the present invention is configured in such a way as to be able to solve variational problems while also managing n plurality of inputs calculated according to a plurality of subsequent processing cycles managing a given processing depth at starting from a previous cycle to a current processing cycle . It follows that the pre-processing block also requires appropriate measures to be able to manage said plurality of inputs in such a way that the activity of an electronic neuron can evolve according to a time step that takes synaptic integration into account .

[0101] Therefore , each signal belonging to the input pluralities must be weighted according to likelihood matrices A representing a synaptic connection existing between a neuron and the network to which it is connected, and where synaptic connection means the probability o f activation of a corresponding neuron starting from the inputs weighted by the likelihood matrix A. The likelihood matrices have an associated precision that represents the synaptic strength and said matrices are periodically updated as will be described below via the b ( ) functions , one for each input m .

[0102] According to a possible embodiment , the A matrices for the excitatory synapses are initiali zed as follows : while the A matrices for the inhibitory synapse are initiali zed as :

[0103] The o denotes the softmax operator (normali zed exponential ) and the quantities of e~1 / 2and e~2are weights added to di f ferentiate the strength of the inhibitory and excitatory connections . From the perspective of the generative model , these weights control the precision of the likelihood (i.e. the weight of the presynaptic inputs) and are updated over time. A±nh initially presents a higher precision (i.e. 2) than Aexc(i.e. M to mimic the greater effectiveness of inhibitory synapses compared to excitatory synapses.

[0104] The synaptic precision parameter (i.e., the exponent initially equal to 2 or k) , which fundamentally weights the likelihood matrix A, determines the impact of presynaptic inputs on the updating of neuronal beliefs. Thus, the neuronal circuit is equipped with synaptic plasticity because synapse-specific fine-tuning is implemented based on prediction errors.

[0105] Excitatory synapses that transmit information about the state of presynaptic neurons, which correspond to predicted values, will experience an increase in (estimated) precision, leading to more accurate predictions in the future.

[0106] Synaptic precision, however, will decrease when predicted values conflict with sensory evidence. Inhibitory synapses will be regulated in the opposite way: they will experience an increase in precision when the inferred values conflict with the input evidence and a decrease in precision when the inferred values and the input are well matched. This configuration mimics the homeostatic role of inhibitory neurotransmission.

[0107] Fig. 78 shows a possible implementation of the input preprocessing block. The plurality of m voltage or current input signals is supplied as input to a series of n+1 memory elements (ao, ..., an) , each capable of supplying, at the subsequent time (t+1) , to its own m outputs the value assumed, at the current time (t) , by its own m inputs. The output signals of each memory element therefore represent the input signals to the system at progressively more remote times, starting from the current ones (oi,t, ..., om,t) to the most remote ones considered (oi,t-n, ..., om,t- n) . The output signals of each memory element are sent as input to one of n+1 circuits, each of which is made up of a set of m multiplier circuits. Each of these n+1 circuits also receives as input a set of descriptive signals of m matrices (Ai, Am) and will provide as output m signals, each proportional to the product of a specific input signal by the matrix associated with it (the correspondence between matrix and input signal is given by the correspondence of the index of the matrix with the first index of the input) . Specifically, the output signals of the i- th circuit among the n+1 is the multiplicity, with second index equal to t-1, of m output signals to the electronic preprocessing circuit on the inputs (Xi,t-i, ..., Xm,t-i) . Each multiplier circuit will be configured to provide as output a signal proportional to the product of one of the input signals by a matrix, also specified by certain input signals. The descriptive signals of said m matrices (Ai, ..., Am) are determined by m circuits (bi, ..., bm) that receive as input the signal y. The relationship between the signal y and the descriptive signals of said m matrices is detailed in Fig. 10.

[0108] Furthermore, the pre-processing block receives binary signals as input from outside, they are naturally transformed by the pre-elaboration block to be non-binary for the purposes of subsequent processing.

[0109] For example, according to a first embodiment, shown in Figs. 7, 8 whereby n=l, the plurality of signals belonging to the past (t-1) and current (t) processing cycles are read from memory elements (ao, ai) .

[0110] Fig. 9 shows a possible implementation of electronic circuit of Fig. 8 wherein two memory elements (ao, ai) are implemented by two groups of m flip-flops in parallel, i.e. two FIFO registers having dimension m.

[0111] Each of the individual signals belonging to the pluralities, both those of a current cycle (t) and those of a past cycle (t- 1) , is coupled to the respective likelihood matrix A, each stored within one of the blocks bi()...bm() (described in Fig 10) , which can be of an excitatory or inhibitory type depending on whether the signal must be enhanced or not , as already indicated previously . In this way the input signals are weighted and normali zed before being provided as input to the computational core .

[0112] The likelihood matrices A are representative of a probability of activation of the neuron based on the inputs provided .

[0113] A likelihood matrix A is an excitatory ( left ) or inhibitory ( right ) matrix of the type :

[0114] The peak values representing the activation of the neuron are encoded with two complementary binary values of the type : where the peak, spike , is represented by the binary value ( 1 0 ) while its complementary, no spike , is represented by the binary value ( 0 1 ) .

[0115] Fig . 10 shows a possible implementation of one of the m circuits (bi, ..., bm) that receive as input the signal y described in Fig . 8 . The circuit bi receives as input the signal o±,t generated inside the circuit described in Fig . 8 and the signal y supplied by the "computational core" circuit , providing as output signals describing the cells of the matrix A± . The input signal o±,t and the signal present at the output of the memory element "memory 1" are supplied as input as the first and second input signals , respectively, to an electronic circuit that will be configured to provide as output a signal proportional to the di f ference between the first and the second input signals (hereinafter, subtractor circuit ) . It is speci fied that , in the case in which the nature of one of the two input signals is vectorial and the nature of the other input signal is matrix, the circuit wil l be configured to provide an output signal representing a matrix whose values present on each column ( or row) are proportional to the di f ference between the values of the column ( or row) vector associated with the input signal and the values of the respective column ( or row) of the matrix associated with the other input signal . The output signal of this circuit is provided as input , together with the input signal y, to a circuit configured to provide an output signal proportional to the scalar product between the signal y and the output signal of the subtractor circuit . This signal is provided as input to a multiplier circuit configured to provide an output signal proportional to the product of the input signal by a known matrix Mi . The output signal from this circuit is supplied as an input to a circuit configured to output a signal proportional to the sum of the elements that make up the input signal , which, as previously explained, can potentially be characteri zed by being defined by a set of values ( e . g . , a signal of a vector, matrix or tensor nature ) . The output signal from this circuit is supplied as an input to a circuit configured to output a signal proportional to the ratio of the input signal and a known coef ficient "c" (hereinafter, divider circuit ) . The output signal from this circuit , called "p" , and the signal present at the output of the memory element "memory 2" are supplied as the first and second input signals , respectively, to a second subtractor circuit . The output signal from this circuit is supplied as an input signal both to the memory element "memory 2" and to a circuit configured to output a signal proportional to the reciprocal of the input signal (hereinafter, reciprocal circuit ) . The output signal from this circuit is supplied as input to a multiplier circuit configured to output a signal proportional to the product of the input signal and a known matrix M2 . , the shape of which depends on the nature of the synapse in question, whether inhibitory or excitatory . The output signal from this circuit is supplied as input to a softmax circuit . The output signal from this circuit is supplied as an input signal to the memory element "memory 1" and constitutes the output signal of the entire circuit bi . It is understood that this signal actually represents , due to its vector nature , a set of signals , representative of the matrix A± . The memory elements "memory 1" and "memory 2" are conf igured to receive as input and store a corresponding signal of state 1 and state 2 as explained above to output it at the next time , outputting at the current time the signal stored at the previous time .

[0116] Fig . 11 shows , by way of example and without limitation, a possible alternative but functionally equivalent implementation of the circuit of Fig . 10 in which the output signal from the memory element "memory 2" is supplied as an input to a further multiplicative inverse circuit , and the output signal from the latter is supplied as an input signal to the second subtractor circuit . Furthermore , the output signal from this circuit is supplied as an input signal exclusively to a multiplicative inverse circuit and not also to the memory element "memory 2" . The output signal from this multiplicative inverse circuit is supplied as an input both to a multiplier circuit configured to supply as an output a signal proportional to the product of the input signal and a known matrix M2 and to the memory element "memory 2" .

[0117] Referring to Figg . 8 and 9 , i f the pre-elaboration block receives from outside binary input signals , the product of each matrix A with the binary value thereof therefore corresponds to selecting the first or second column of matrix A, which can be implemented through analog multiplexer as known for the skilled man ( as schematically shown in Fig . 12 to exempli fy inputs of the actual elaboration cycle ( t ) ) . In the electronic neuron implementation, since a plurality of neurons are networked, the m current inputs to the preprocessing block are corresponding binary output values generated by upstream connected neurons or inputs that come from outside the system .

[0118] As shown in Figs . 7- 9 , the input s ignals are multiplied by the elements of the associated matrix A to be subsequently supplied to the computational core respectively as a first plurality of normali zed signals relating to a current processing cycle and a second plurality of normali zed signals relating to a cycle of past processing ( and possibly other pluralities of normali zed signals relating to even more remote processing cycles ) .

[0119] As mentioned previously, the likelihood matrices A are representative of a synaptic connection existing between a neuron and the network to which it is connected and therefore the stored values of said matrices evolve over time during the functioning of the neuron . However, a likelihood matrix initially set in the preprocessing block as excitatory or inhibitory does not change its nature during operation .

[0120] The updating of the weights of the matrices stored in the bi. .bmblocks occurs on the basis of the y signal provided as output by the computational core . The new calculated values are subsequently stored in the memory blocks of the pre-processing block to be used in the next processing cycle . Normali zed current and past inputs (Xi,t, Xm,t) ... (Xi,t-n, Xm,t-n) are therefore generated via the bi. .bmblocks which are subsequently processed in the computational core .

[0121] Output bl ock

[0122] Finally, Fig . 13 shows a block diagram of the stochastic decisional output block which is configured in such a way as to receive as input signal y output from computational core and generate the binary activation signal , e . g . spike or no spike , when the value received as input is greater than a second value generated by a random, pseudo-random or cahotic number generation function. The output signal from this circuit and the input signal y are provided as the first and second input, respectively, (i.e. as the second and first input, respectively) to a comparator circuit identified by the symbol "<?" configured to supply as an output a signal representing the sign of the difference between the two inputs.

[0123] It should be noted that, in the context of this description, the meaning of non-explicit variables, e.g., VTH, is known to the skilled man in electronics. It is also specified that the design and implementation of:

[0124] - an electronic circuit generating current, or voltage, with a predetermined value

[0125] - a digital PIPO register of size n an electronic circuit comparing voltage signals, or current

[0126] - an analog multiplexer

[0127] - a non-linear feedback shift register (NLFSR)

[0128] - a current sense amplifier (CSA) circuit

[0129] - a digital-to-analog converter (D / A) are known to the skilled man, known in literature, and outside the scope of this description.

[0130] Implementation of the neuron using electronic circuits

[0131] Each of the functional blocks described previously can be created using analog circuits or digital circuits whose operation is not necessarily accompanied by an interpretation phase of stored software instructions.

[0132] An implementation using digital circuits is generally useful in the study and prototyping phase to verify operation, simplifying the implementation process and reducing implementation costs. An implementation using analog circuits instead allows the creation of circuits that operate with very low currents , allowing high energy ef ficiency to be achieved .

[0133] Analog implementation of the neuron

[0134] Each of the functional blocks described can be implemented using speci fic analog circuits that can be created, for example , with field ef fect transistors ( FET ) and / or other active or passive electronic components . To achieve the greatest possible energy ef ficiency, signals can be encoded and processed as very small currents ( e . g . in the current range from a few pA to a few nA) . For this reason, most of the transistors used in the circuits exempli fied below work subthreshold .

[0135] Fig . 14a shows a possible analog implementation of the logarithmic circuit . This circuit exploits the relationship between the gate voltage and the drain current of a transistor VGS~VTH working below threshold IDS« IoenVt . In the circuit of Fig . 14a, the transistor Ml converts the input current ( T in) into a gate voltage (VR) proportional to the natural logarithm of l in . The voltage drop on the resistor, VR, ensures that the current flowing between the source and drain contacts of transistors M2 and M3 is proportional to the logarithm of the input current . Such a current is mirrored at the output through the current mirror constituted by the transistors M3 and M4 . Based on the circuit si zing, the output current Iout can be written as : and can therefore be seen as consisting of two terms , one linearly proportional to the natural logarithm of the input current , and a constant ( k) : lout & ' frlGin) + k The circuit diagram of Fig. 14b, known in literature, includes a resistor, a diode and an operational amplifier connected as shown in the diagram. The equation that links the output voltage signal (Vout) to the input voltage signal (V±n) is shown in the figure. If, in any of the proposed circuit variants, the input signal were characterized by being defined by a set of values (e.g., a vector, matrix or tensor signal) , each value would in fact be represented in turn by an individual signal, and each of these individual signals could be processed separately by a dedicated copy of the logarithmic circuit.

[0136] Fig. 15a shows a possible analog implementation of the exponential circuit. The operation of the circuit that implements the exponential function is similar to that of the logarithmic circuit, but in this case the input current flows on a resistor which converts it into a voltage (VR) . This voltage is connected to the gate of Ml, ensuring that the current flowing in Ml and M2 is proportional to the exponential of the input current. This current is mirrored out of M2 and M3.

[0137] The circuit provides an output current lout which can be written as :

[0138] RIin~vTH ~VTHRIin

[0139] Iout ~ I0- env< = Io■ enVt ■ enVt

[0140] Therefore, the current lout is proportional to the exponential of an input current lin according to the following formula:

[0141] IIou ,t = P B ■ eeaIin

[0142] Note that it is possible, with an adequate resistor design, to arbitrarily choose the value of the constant a, e.g., a = 1. The electrical diagram of Fig. 15b, known in the literature, includes a resistor, a diode and an operational amplifier connected as shown in the diagram. The equation that links the output voltage signal (Vout) to the input voltage signal (V±n) is shown below. Note that it is possible to arbitrarily choose the value of the constant a, e.g., a = 1, determining the voltage value that represents the unit (e.g., if chosen equal to Vt, then when V±n = Vt the signal associated with it represents the unit and therefore a = 1) . If, in any of the proposed circuit variants, the input signal were characterized by being defined by a set of values (e.g., a vector, matrix or tensor signal) , each value would in fact be represented in turn by an individual signal, and each of these individual signals could be processed separately by a dedicated copy of the exponential circuit.

[0143] Fig. 16a shows a possible analog implementation of the divider or multiplier circuit, known in literature [7] , that allows to calculate the ratio between two input currents IA and IB. By exchanging the role of the inputs and the ISCALE generator, such a circuit can be used to multiply two input currents. The product between two currents is for example used in the input pre-processing block of the electronic neuron.

[0144] The current Iout can therefore be written as:

[0145] In other words, the circuit diagram of Fig. 16a, provides 4 transistors (Mi, ..., M4) operating in subthreshold mode and connected as shown in the diagram. The circuit provides for the use of an additional current signal called ISCALE provided by a specific current generator circuit. In order to maintain selfconsistency, the value of the ISCALE current must be set to the one that represents unity and which can be arbitrarily chosen. Note how the same circuit can function as a multiplier when the role of one of the two input signals and the current signal called ISCALE is swapped, as exemplified in the circuit diagram of Fig. 16b. Consequently, the divider circuit can be used to implement the divider circuit of Fig. 10 (where one of the two input signals is supplied by a current source of predetermined value equal to c) and the reciprocal circuit of Fig. 10 (where one of the two input signals is supplied by a current source of predetermined value and representing unity, see Fig. 20) .

[0146] The analog implementation of the divider circuit can be implemented using the previously described exponential and logarithmic circuits (A / B = exp(ln(A / B) ) = exp ( In (A) -In (B) ) ) .

[0147] Fig. 17a shows a possible implementation of the softmax circuit that exploits the exponential and divisor circuits. Two exponential circuits calculate the exponentials of the input currents, e.g. representative of a vector with two components. Two current mirrors duplicate the outputs of the two exponentials, and one of the copies of each mirror is supplied as input to a divider circuit, while the others are summed in a node of the circuit and connected to the other input of the divider circuits. At the output of each divider circuit, each of the components of the softmax output is calculated.

[0148] In other words, the circuit diagram in Fig. 17a provides for input (Iini and I±n2) and output (Iouti and Iout2) current signals and uses 2 exponential circuits as shown in Fig. 15 for which a = 1 has been chosen, 2 divider circuits as shown in Fig. 16, 3 current mirrors (as shown in Fig. 18 below) needed to replicate a current signal on multiple branches, and a summing node which for current signals simply corresponds to short-circuiting the input and output lines in a single circuit node, as is known to skilled man in electronics. The circuit diagram shows the connections between the various circuits listed above. The circuit diagram in Fig. 17b includes voltage input signals (Viniand V±n2 ) and output signals (Vouti and Vout2) and uses 2 exponential circuits as shown in Fig. 15 for which a = 1 has been chosen, 2 divider circuits as shown in Fig. 16, and a summing circuit as shown in Fig. 18. The circuit diagram shows the connections between the various circuits listed above. The circuit diagram in Fig. 17c, known in literature, includes voltage input signals (Vini and V±n2 ) and current output signals (louti and Iout2) and uses 2 transistors (Mi, M2) operating in the subthreshold regime and connected as shown in the diagram. The circuit includes the use of an additional current signal called ISCALE provided by a special current generator circuit. The circuit diagram shows the connections between the various circuits listed above. Fig. 18 shows, as a non-limiting example, a possible circuit implementation of the weighed sum circuit in which the input signals are, as a non-limiting example, two. The electrical diagram above includes input signals (Iini and Iin2) and current output signals (Iouti and Iout2) and uses 4 transistors (Mia, Mib, M2a, M2b) connected as shown in the diagram. Each pair of transistors M±a, M±b (with the generic index i) forms a current mirror (highlighted by a dotted border) which provides at its output a current signal (or more than one current signal, in the general case, as is obvious to the skilled man in electronics, proportional to the current signal at its input and which can therefore be used as a circuit suitable for multiplying the input current signal by a predetermined constant value (or multiplying in parallel the input current signal by several predetermined constant values) , e.g., as an implementation of the divider circuit shown in Fig. 10-11. The output current signals of the individual current mirrors are then added together by converging the corresponding output lines into a single output circuit branch, as is obvious to the skilled man. The weighed summation circuit can also be used as an implementation of each of the two multiplier circuits shown in Fig. 10-11 and of the multiplier circuit shown in blocks b2 and b4 of Fig. 10-11. To this end, in the case where the input signal is scalar in nature, a current mirror is sufficient in which the transistor pair is appropriately sized to provide an output current signal proportional to the input current signal where the proportionality factor is the desired multiplicative constant, as per the equation below . I f the input signal is vectorial and the multiplicative factor therefore corresponds to a matrix, it is possible to provide a weighted summation circuit for each row ( column) of the matrix . All the weighted summation circuits would receive the same input vector s ignal as input , appropriately replicated by additional current mirrors . In each weighed summation circuit , each transistor pair relating to a single current mirror will be appropriately si zed to provide an output current signal proportional to the input current signal where the proportionality factor is the relative entry of the matrix . A non-limiting example is shown in the circuit diagram at lower left for a two-component ( Iini and I±n2 ) input vector signal ( Iin) that is multiplied by a 2x2 square matrix (A, with values an, ai2 , a2i, 022 ) , resulting in an output current vector signal ( lout ) with two components ( louti and Iout2 ) . The weighed sum circuit can also be used to implement the circuit configured to output a signal proportional to the sum of the elements that make up the input signal in Fig . 13 . Finally, several instances of the divider circuit in Fig . 16 , followed by a weighed sum circuit in Fig . 18 , can be used to implement the scalar product circuit in Fig . 10- 11 . As an example , the scalar product of a vector current signal of si ze n and a matrix current signal of si ze m x n will require n current mirrors with m identical outputs each, and m x n divider circuits (used as multipliers ) each of which will take as input one and only one of the m output s ignals of one and only one of the n current mirrors , and one and only one component of the input matrix current signal . The output current signals from the m x n divider circuits (used as multipliers ) will then be summed in groups of n ( each group of n outputs converging at m separate nodes ) to provide a vector output signal of si ze m . The circuit diagram at the bottom right , known in the literature , is related to an implementation of the weighted summation circuit characteri zed by n input voltage signals (Vi, Vn) and one output voltage signal (Vout) and involves n+1 resistors (Ri, Rn, Rf) and an operational amplifier connected as shown in the scheme.

[0149] Fig. 19 shows as a non-limiting example a possible circuit implementation of the normalizing circuit in the exemplified case of a 2-component vector input signal (of dimension 2) , to be understood therefore as composed of 2 signals. Said two signals, in this non-limiting example to be understood as current signals (Iini and I±n2) , are supplied to as many current mirrors each with two output branches and corresponding signals. One output signal of each current mirror is supplied as the first input signal to a divider circuit while the other output signal of each current mirror is supplied as an input signal to a weighed sum circuit configured to have equal weights. The output signal from the weighted sum circuit is supplied as input to a further current mirror with two output branches and corresponding signals, which will be supplied as second input signals to the two divider circuits. The output signals from these divider circuits will constitute the component elements (Touti and Iout2) of the output vector signal.

[0150] Fig. 20 shows, as a non-limiting example, a possible circuit implementation of the multiplicative inverse circuit, as shown in Fig. 16. A divider circuit is used for this purpose in the configuration shown in the circuit diagram. It should be noted that ISCALE is a current value that represents unity.

[0151] Fig. 21 shows, as a non-limiting example, a possible circuit implementation of the subtractor circuit shown in Fig. 10-11. As already specified in the description of Fig. 10-11, where one of the two input signals is vectorial and the other input signal is a matrix, the circuit will be configured to provide an output signal representing a matrix whose values present in each column (or row) are proportional to the difference between the values of the column (or row) vector associated with the input signal and the values of the respective column (or row) of the matrix associated with the other input signal. In this case, the m components of the vector input signal, which in turn are to be understood as input signals, must be supplied as input to m current mirrors having n output branches, where m represents the dimension of the vector while m and n those of the matrix. In this way, each of the m n output signals from said current mirrors can be supplied as the first input signal (Iini) to one and only one of m n circuits like the one shown in the circuit diagram, each of which will receive as the second input signal (Im) one and only one of the components of the matrix input signal. As a non-limiting example, the implementation in the case of current input signals is shown. The input current signals are appropriately replicated by current mirrors. In two separate circuit nodes, a replica of the first input signal is subtracted from a replica of the second input signal and vice versa. The currents resulting from the subtraction cannot, however, be negative by circuit construction. These currents are further replicated by additional current mirrors and further subtracted from the output circuit node, providing an output current signal proportional to the difference between the first and second input current signals.

[0152] Fig. 22 shows, as a non-limiting example, a possible circuit implementation of each of the two multiplier circuits in Figs. 10-11 and of the multiplier circuit in blocks b2 and b4 of Fig. 2 and subsequent ones, added to the one already outlined in Fig. 18. In this circuit implementation, a divider circuit, used as a multiplier, is present for each element of the multiplicative constant (which can be characterized by being defined by a set of values, e.g. a matrix) . As a non-limiting example, the implementation is shown in the case of a vectorial current input signal of size 2 that is multiplied by a matrix-type multiplicative constant of size 2x2 (A, with values an, ai2, a2i, «22) • The input current signal is composed of two current signals (Iini and I±n2) that are each sent to a current mirror with two output branches that each carry a copy of the input signal. The two current mirrors are formed by the transistors Mu, Mis, M21, M25. The output signals to the current mirrors are sent as inputs to four divider circuits used as multipliers formed by the transistors MAI, ..., MA4, MBI, ..., MB4, ..., MDI, ..., MB4. Each of these divider circuits has as an additional input, appropriately positioned, a further current signal representing a specific value of the matrix A, that is, one and only one among the values an, ai2, a2i, a22. The output signals from these divider circuits are supplied as inputs in pairs to two weighted sum circuits, formed by the transistors MEI, ..., ME4, MEI, ..., ME4, whose output signals (IOuti and IOut2) represent the values of the two-component output current vector signal (Tout) .

[0153] Fig. 23 shows as a non-limiting example a possible circuit implementation of a circuit that combines together the functions of the softmax circuit shown in blocks bi and bBof Fig. 2e followed by the multiplier circuit shown in blocks b2 and b4 of Fig. 2e, as also shown in the subcircuit B of Fig. 3c. This implementation applies in specific cases, such as the case in which the matrix multiplicative constant indicated as B in Figs. 2e and 3c is equal to its transpose indicated as BTin the same Figs, and that is, in the case in which B is symmetric, or when such a matrix has the same value of the sum of the elements for each row (or column) . In this case, the circuit diagram in Fig. 17c relating to a possible implementation of the softmax circuit, shown as a non-limiting example for a vector input signal of size 2, which provides voltage input signals (Viniand V±n2) and current output signals (Touti and IOut2) , shown again on the left in the current Figure, can be modified as shown in the diagram on the right (Fig. 23b) . This specific implementation uses 4 transistors (Mu, M12, M21, M22) operating in the subthreshold regime, connected as shown in the diagram and sized in such a way that the aspect ratio of a given transistor coincides with one and only one element of the matrix. The circuit provides for the use of an additional current signal called ISCALE provided by a specific current generator circuit that will represent the unit .

[0154] Fig. 24 shows as a non-limiting example a possible circuit implementation of a memory element. The circuit diagram includes two cascaded "sample and hold" circuits, driven by opposing synchronization signals. Specifically, the circuit includes two capacitors (Ci and C2) , two switches (Si and S2) typically implemented via transistors, and three operational amplifiers (Ai, ..., A3) connected as shown in the diagram. In addition to the output voltage signal (Vout) and the input voltage signal (V±n) , it is also necessary to provide a digital synchronization voltage signal (Vc) and its logical complement (Vcn) , aimed at determining the state of the switches, which will work in mutual exclusion over time. This periodic signal must remain at the high logic level (switch closed) at least for a time sufficient to charge capacitor Ci, and then switch to the low logic level (switch open) at least for a time sufficient to charge capacitor C2. The period of the synchronization signal will coincide with the time difference between time t and time t-1 as discussed in the introduction and in Fig. 3a. In the case where the input signal is a voltage and digital, a possible implementation of a memory element is that of a FIFO register of size n where n is the number of elements needed to represent the input signal (for example, a vector of size n) as for example in Fig. 9.

[0155] Fig. 25 shows a possible circuit implementation of the output decisional block. Any circuit for generating random numbers generates a sequence of random bits, which is stored in a digital register. The digital value stored in the register is converted into an equivalent analog current e. This current is compared using a current sense amplifier (CSA) with the current signal calculated by the variational cycle, generating or not a peak in output.

[0156] In other words, Fig. 25a provides a digital circuit configured to output a parallel n-bit digital voltage signal representing a random, pseudo-random, or chaotic number, designated "Digital RNG." A well-known example of such a circuit is the non-linear feedback shift register (NLFSR) that generates an n-bit pseudo-random number that is stored in a digital register. The digital value stored in the register can then be converted to an equivalent analog current via an n-bit digital- to-analog (D / A) converter, such as a current output. A nonlimiting embodiment of such a D / A circuit is shown in the circuit diagram in Fig. 25b, enclosed by a rounded rectangle. In this scheme there are n transistors such that the i-th transistor receives at its gate terminal the signal relating to the i-th bit of the digital value and at its drain terminal a current of predetermined value, whose value is shown in the figure, generated by a special current generator circuit. The currents at the source terminals of the n transistors are supplied to the input of a current mirror, whose output current (Imd) represents the current equivalent of the digital value stored in the NLFSR register. The current signal output from the D / A circuit is then compared by means of a current sense amplifier (CSA) with the current signal "y" , generating or not a peak on the output voltage signal (output) .

[0157] Fig. 26 shows as a non-limiting example a possible circuit implementation of a circuit that approximates the natural logarithm function. Specifically, in the non-inf initesimal portion of the domain shown in the figure, i.e., x between 0.035 and 1, the function indicated by p(x) differs from the natural logarithm function ln(x) by less than 14%. In the proposed scheme, the input current signal (Ix) is supplied as input to a current mirror having two output branches sized in such a way as to supply on them a current equal to 3.6 and 3 times the input current respectively. From the first of these a current representative of the value 3.6 (I3.6) is subtracted, generated by a special constant current generator, while to the second of these a current representative of the value 1.1 (I1.1) is added, also generated by a special constant current generator. The resulting current signals are supplied respectively as the first and second input signals to a divider circuit that supplies as output a current signal (Ig) proportional to p(Ix) .

[0158] Fig. 27 shows as a non-limiting example a possible circuit implementation of a "set of m multipliers" circuit as shown in Fig. 7. In fact, each of the m multipliers can be implemented according to the scheme shown in Fig. 22. However, in the case in which the input vector signal to a generic multiplier is digital and voltage, it is possible to implement this multiplier according to the scheme in Fig. 27a, already specified in Fig. 12, in which the digital voltage input signal constitutes the selection signal of an analog multiplexer which receives as analog inputs the descriptive signals of the A± matrix, as explained in Fig. 7, so that the output signal coincides with a specific row (or column) of the Ai matrix, identified by the value of the vector input signal. A specific example in which the input vector signal has two components and complementary components and is multiplied by a 2x2 matrix is shown in Fig. 27b. It is specified that in each scheme proposed in the drawings above it is possible, for any sequence of electronic circuits or blocks :

[0159] - identify the number and type of input signals to and output from the sequence of circuits or blocks

[0160] - for each output signal (outi) identify the mathematical relationship (r±) that links it to the input signals - identi fy an alternative electronic circuit that has the same number and type of input and output signals as the sequence of circuits or blocks and that establishes for each output signal ( outi ) the same mathematical relationship with the input signals that the sequence of circuits or blocks ( r± ) establishes , or a di f ferent mathematical relationship ( s± ) that does not deviate from r± by more than one order of magnitude ( i . e . for which si / ri is between 0 . 1 and 10 ) for at least a non-inf initesimal subset of the domain of r± replace the sequence of circuits or blocks with the alternative electronic circuit .

[0161] Digital implementation of the neuron

[0162] The previously described functional blocks that define a single neuron can be implemented using digital circuits in standard CMOS technology . Unlike the analog implementation in which the results of the various blocks are calculated simultaneously, in the digital implementation the operations of the various functional blocks must or can be performed in sequence . Furthermore , the operations by the computational core must be solved using an iterative approach, repeating the set of operations a suf ficient number of times to guarantee the convergence of the result .

[0163] From a circuit point of view, the digital version of the neuron can be implemented as a finite state machine that sequentially performs the operations described in the di f ferent functional blocks . The circuit that implements a neuron must include at least the following components :

[0164] • At least one adder block

[0165] • At least one multiplier block, unless multiplication is implemented as a repeated sum

[0166] • At least one synchronism signal ( clock)

[0167] • Registers to store parameters and partial results

[0168] • A control logic that manages the finite state machine • May or may not include dedicated circuits that perform the arbitrary functions, e.g. logarithm, exponential and sof tmax

[0169] • May or may not include lookup tables to calculate the results of arbitrary functions, e.g. logarithm, exponential and softmax

[0170] • At least one random number generator

[0171] In the digital implementation of a neuron, the hardware can also be reused to process the information of multiple neurons sequentially, to balance the trade-off between occupied area and latency, according to the specifics of the application.

[0172] Neuronal network

[0173] According to an aspect of the present invention, once the electronic neuron has been created in circuit as indicated e.g. in Fig. 5, it is possible to set up a network of electronic neurons by connecting the neurons to each other even in multiple layers. These networks are capable of carrying out operations without a previous training step. Figure 28, for example, illustrates a neuronal network to simulate one of the most widely adopted experimental protocols to characterize learning mechanisms is delayed eyeblink classical conditioning (dEBCC) , a form of cerebellar-dependent associative memory depending on the cerebellar activity to anticipate the closure of eyelid in response to a conditioning stimulus [2] .

[0174] The temporal evolution of the system was simulated in cycles (i.e. time steps) , where each time step can be thought of as equivalent to a period of two milliseconds, in accordance with known neuronal dynamics. The frequency calculation and the stimuli must therefore be interpreted in this reference time frame. Associative learning was induced by pairing a generic sensory stimulus (the conditioned stimulus (CS) ) with an unconditioned stimulus (US) that caused closure of eyelid. The first was produced by repetitive stimulation ( 10 stimuli at 2 Hz ) of PN neurons ( see Fig . 21 ) with bursts of 120 ms at 400 Hz , while the second was delivered as repetitive stimulation ( 10 stimuli at 2 Hz ) of TN neurons ( simulated) with 30 ms bursts at 400 Hz . TN stimulation was always delivered within the burst stimulation window of the PNs : more precisely, the ends of the two stimulation windows coincided . Each simulation involved the successive and independent repetition of three conditions : US , CS and combined stimulation (USCS ) . The latter, thanks to the combined presentation of the US and CS input in a defined temporal sequence , allows associative learning to be established .

[0175] To quanti fy the emergence of conditioning, the firing probability of output neurons (RNs ) was assessed in terms of the total number of firing events generated in the time window corresponding to the CS . The number of firing events associated with a speci fic time interval was obtained by introducing an upper threshold of 90% for the probability of triggering the output of the RNs . The histogram representation of the RN peak was also divided into two-time windows for each CS stimulus window, one in the first hal f of the window and the other in the second hal f . This allowed us to identi fy earlier RN activation in the USCS condition compared to the CS and US alone .

[0176] Variational free energy minimi zation of each electronic neuron of the network was implemented according Forney factori zation that provides the reformulation of the inference as an optimi zation process , where belief updating corresponds to the convergence of free energy towards a local minimum, using iterative gradient descent ( see equation ( 3 , 4 , 5 ) ) . In other words , we treat neuronal dynamics as a gradient flow on VFE , which define a special case of the generic class of problems represented already described in the introduction : VFE = DKL($(St)\\p(St\Oty) ~ lnp(Ot) and solvable with the proposed invention by approximating the cancellation of the Kullback-Leibner operator with the estimation of the potential of membrane indicated in equation 7. The a posteriori distribution on the hidden states is derived from the application of the sum product rule, i.e. the product of all messages arriving at the associated node. In the present context, where each neuron acts as an inferent element, at any time the messages update the logarithmic expectation of the hidden states via an error term St. The logarithmic expectation of hidden states is associated with voltage in the soma of the neuron .

[0177] At each time step, the neuron's belief about the state of the network, in which it participates, is encoded in the probability of actuating an action potential (y) .

[0178] The conditional dependence of st on st-i is parameterized by a probability transition matrix (B in Fig. 1) . Matrix B is set equal to the softmax of the identity matrix [1,0; 0,1] , since the circuit does not have specific temporal dynamics, such as oscillations, which are prescribed by a conditional dependence between states. Intuitively, this means that each neuron thinks that the hidden or latent states that generate its observations do not change as its beliefs are updated.

[0179] So the softmax operator is used here to mimic a sigmoid actuation rate depolarization activation function. This explains the early belief updating and neural dynamics during each time step (i.e., inference) .

[0180] In a more general case of an electronic neuron network, Fig. 29 shows, as a non-limiting example, a possible implementation of a network that encodes a multiplicity m of inputs and returns a multiplicity n of outputs. Within an electronic neuron network, a multiplicity m of input signals to a given neuron can be provided from the outside or be the output signals generated by other neurons, and the output signal generated by the neuron can become an input signal for other neurons. The input signals are responsible for the outputs of the k input neurons. These outputs can constitute the inputs of a first layer of w electronic neurons (grey box) . The connections indicated by the arrows can be complete, e.g., each electronic neuron sends its output signal to each of the neurons of the layers affected by its connections, sparse, e.g., each electronic neuron sends an output signal to a number of neurons lower than the number of neurons belonging to the layers affected by its connections, and recurrent, e.g., each neuron can send an output signal to one or more neurons of the layer to which it belongs and can receive an input signal from one or more neurons of the same layer. In the case shown, the electronic neurons of the first layer (red box) of an m-th layer of as many w neurons whose output in turn constitutes an input for the neurons of the first layer (blue arrows) .

[0181] It is stated that, an electronic circuit according to the scope of claim 1 can be manufactured according to an embodiment where in general it is possible to use a plurality of instances of the electronic circuit executing the convergence, e.g. the circuit in Fig. 2e, connected in cascade. In general, the specific characteristics of the graph associated with the problem to be solved may include the need to process multiple inputs relating to various processing cycles progressively earlier in time. Consequently, according to the scope of claim 1, it is possible to generally provide for a number n of electronic circuits, each of which is designed to receive as input at most all the multiple inputs relating to various processing cycles progressively earlier in time, in addition to any accessory inputs. As the cascade of circuits created in a given circuit solution increase towards an infinite number, the solution will tend to become a continuous time solution.

[0182] These circuits can then be connected to each other in such a way as to respect the connections dictated by the graph associated with the problem. An example is shown in Fig. 30, where a specific signal of each of the n cascaded circuits (for example the signal called y) represents a further input for the downstream circuit (for example the input Xo) . In the diagram in Fig. 30 it is possible to note how the connection between the various cascaded circuits can provide a storage and feedback scheme for the subsequent calculation cycle such that the feedback, implemented between the circuit that realizes the function h() to be understood as a storage function, is present only in the first upstream circuit, while for the subsequent n- 1 cascaded circuits the feedback of the state value (Xo) at the input to an n-th stage is replaced by the output signal (in this example the signal y) calculated in the circuit placed upstream with respect to the circuit considered.

[0183] As explained previously, each cascaded circuit generates a signal representing the internal state value (z) which is used during the minimization of the discrepancy between the internal state value (Xo) calculated in a previous processing cycle and the state value (z) calculated in the current cycle where the internal state value (Xo) represents an additional input to the circuit. An example with n=2 is shown in Fig. 31.

[0184] In the specific case of the mathematical problem represented by the coupled equations 7 and 8, if one intends to extend the formal treatment of the problem in such a way that the message passing aimed at predicting the internal state at time t (st) includes the evaluation of the internal state also at times even earlier than the past time t-1 (st-i) , i.e. the evaluation of the internal states at time t-2 (st-2) , t-3 (st-3) , ..., t-n (st-n) , it is possible to implement the related electronic circuit according to claim 1 or, equivalently, as described above and illustrated in Fig. 32. In this example, in which this extension is limited to time t-2, consistently with the graph in Fig. 1c, it provides a number equal to 2 of cascaded circuits, each of which is configured to receive as input at most all the multiplicities of inputs relating to various processing cycles progressively earlier in time, i.e., t, t-1, t-2. In the specific example, each circuit is configured to receive as input two multiplicities of inputs relating to two adjacent processing cycles (e.g. t and t-1) . The downstream circuit receives as input a first pair of pluralities of inputs relating to the current processing cycle (t) and the previous processing cycle (t-1) , while the upstream circuit receives as input a second pair of pluralities of inputs relating to the previous processing cycle (t-1) and the even previous processing cycle (t-2) . In the case of n cascaded circuits in which each circuit receives as input two multiplicities of inputs relating to two adjacent processing cycles (e.g. t and t-1) successively.

[0185] The number of cascaded circuits determines the time depth 'n' of the past inputs obtained during the processing of n-1 previous cycles with respect to a current processing cycle to arrive at a convergence of the error and the solution of a given variational problem. Coherently with the case of the past processing cycle (t-1) , it is important that the corresponding circuit, depicted in Fig. 2e, receives as well as input two input pluralities representative of two adjacent processing cycle e.g. t and t-1.

[0186] In Fig. 32, it is to note the connection between the 2 cascade circuits via the state signal (z) , coherently to the Forney graph in Fig. 1c. Said signal (z) calculated in each block is representative of the discrepancy between the value XO and the set of plurality of received inputs. Therefore, for each cascade circuito, the state signal (z) represents a further input for the downstream circuit (Xo) while the output signal (y) of the last stage represents the output circuit of the computational core which represents the convergence achieved based on n+1 number of input sets. Is is noted that the further connection between the n cascade circuits via the signal called xm+i, so that, for each cascade circuit, the signal called xm+i also represents a further input for the upstream circuit and replaces the optional signal set to unity as shown in Fig. 2e, as it represents, for the upstream circuit, a return message from the future.

[0187] In the diagram of Fig. 32 it is possible to see how the connection between the various circuits can provide, coherently with Fig. 31, a storage and feedback scheme for the subsequent calculation cycle such that the feedback is present only in the first upstream circuit, while for the subsequent n -1 circuits in cascade, the feedback of the state value (Xo) input to an n- th circuits is replaced by the state signal (z) calculated in the circuit located upstream from the circuit considered.

[0188] This scheme generally allows the approximate or exact convergence of a VMP algorithm to be achieved by considering a plurality of inputs relating to the current calculation cycle (t) and a plurality of inputs relating to the previous calculation cycle (t-1) , and a further plurality of inputs relating to the even previous calculation cycle t-2.

[0189] BIBLIOGRAPHY

[0190] [1] Palacios, E.R., Isomura, T., Parr, T. et al., "The emergence of synchrony in networks of mutually inferring neurons", Scientific Reports 9, 6412 (2019) . https : / / doi . org / 10.1038 / s 41398 - 019- 42821 - 7

[0191] [2] D. Gandolfi, F. M. Puglisi, G. M. Boiani, G. Pagnoni, K. J. Friston, E. D'Angelo and J. Mapelli, "Emergence of associative learning in a neuromorphic inference network", Journal of Neural Engineering, Volume 19, Number 3, 036022, DOI 10.1088 / 1741- 2552 / ac6ca7.

[0192] [3] M. Cox, T. van de Laar, B. de Vries, "A factor graph approach to automated design of Bayesian signal processing algorithms", International Journal of Approximate Reasoning, Volume 104, January 2019, Pages 185-204, doi: 10.1016 / j . i j ar .2018.11.002

[0193] [4] ForneyLab, https : / / github . com / biaslab / ForneyLab . j l?tab=readme-ov-f ile

[0194] [5] H.-A. Loeliger, "An introduction to factor graphs," in IEEE Signal Processing Magazine, vol. 21, no. 1, pp . 28-41, Jan. 2004, doi: 10.1109 / MSP.2004.1267047.

[0195] [6] Parr, T., Markovic, D., Kiebel, S.J. et al., "Neuronal message passing using Mean-field, Bethe, and Marginal approximations", Scientific Reports 9, 1889 (2019) . https : / / doi .org / 10.1038 / s41598- 018-38246-3

[0196] [7] M. Vatalaro, T. Moposita, S. Strangle, L. Trojman, A. Vladimirescu, M. Lanuzza and F. Crupi, "A Low-Voltage, Low-Power Reconfigurable Current-Mode Softmax Circuit for Analog Neural Networks", Electronics 2021, 10, 1004. https: / / doi.org / 10.3390 / electronicsl 0091004

Claims

CLAIMS1 . A solution circuit , within a current calculation cycle ( t ) , in variational message passing of a multiparametric mathematical problem factori zable according to the Forney- f actor, said circuit comprising :* a first plurality of m* r electrically conductive input tracks , related to m electrical input signals (Xi,t, ..., Xm,t) of dimension r related to the current calculation cycle ( t ) and representative of said parameters* and a first block of electrical and electronic components connected to each other in which :* there is a second plurality o f m* r electrical ly conductive input tracks , related to m electrical input signals (Xi,t-i, ..., Xm,t-i ) of dimension r related to a previous calculation cycle ( t- 1 ) and representative of said parameters* there are k node-circuits corresponding to k nodes of a graph of the Forney factori zation of the multiparametric mathematical problem* there are at least v tracks to connect the node circuits together in order to exchange between the node circuits current and / or voltage signals representing the output / input messages via vertices of the graph( a ) Wherein(b ) At least one node circuit is a ' = ' node circuit , configured to receive at least two tracks as input and generate , on a single first output track in the case of a scalar output signal or on a single first multiplicity of r output tracks in the case of a signal of dimensionality r greater than or equal to 2 , avoltage or current output signal representing said output message , proportional to the generalized product of the input signals ,( c ) At least one node circuit is a node circuit corresponding to a 'product ' type node in the graph, configured to receive at least one track as input and generate , on a single second output track in the case of a scalar output signal or on a single second multiplicity of r output tracks in the case of a signal of dimensionality r greater than or equal to 2 , a voltage or current output signal representing said output message , proportional to the product of the at least one input by a constant quantity,( d) Wherein one of the tracks between the node circuits and downstream of said node circuit ' = ' and the 'product ' node circuit provides a voltage or current signal y relating to the message associated with it which represents the variational message passing solution of said mathematical problem and wherein the node circuits are configured to generate the corresponding output signals by means of one or more hardwired or ( re- ) conf igurable hardware components , preferably made using at least one of the following technologies : application-speci fic integrated circuit (AS IC ) , programmable logic device ( PLD) , FPGA device . 2 ) An electronic circuit according to claim 1 , called computational core ,- wherein the block comprises at least a first subcircuit (A) comprising at least :* said circuit-node '=' or a first electronic circuit of equivalent functionality ( function describing the ratio between output and inputs ) , a second electronic normali zation circuit configured to provide as anoutput a signal proportional to the input signal (i.e., multiplied by an arbitrary factor) which receives as an input the current or voltage output of the first electronic circuit, and said circuit-node 'product' receives as an input the current or voltage output of the electronic normalization circuit; said first subcircuit (A) receiving as an input:(1) the first plurality of m*r electrically conductive input tracks;(2) a first further plurality of r electrically conductive input tracks relating to a feedback signal (Xo) of size r; and(3) a second further plurality of r electrically conductive input tracks relating to a signal called (Xm+i) of size r; to provide a first and single electrical output signal (z) of size r;- an electronic memory element that receives as input said first and single output signal (z) of size r to provide as output to the next calculation cycle (t+1) , by feeding said first plurality of r electrically conductive input tracks relating to a feedback signal (Xo) of size r, the value assumed by its input to the current calculation cycle (t) , or to provide as output to the current calculation cycle (t) , by feeding said first plurality of r electrically conductive input tracks relating to a feedback signal (Xo) of size r, the value assumed by its input to the previous calculation cycle (t-1) ;- a second sub-circuit (B) defining a loop with the first sub-circuit (A) and equal to the first subcircuit (A) , receiving at input:(1) said second plurality of m*r electrically conductive input tracks;(2) a third further plurality of r electrically conductive input tracks powered by said first and unique output signal (z) of size r coming from said first sub-circuit (A) ; to provide a second and unique output electrical signal of size r that powers said second plurality of r electrically conductive input tracks relative to the signal (Xm+i) of size r; wherein the output voltage or current signal (y) of size r coincides with the output of the normalization circuit of the second sub-circuit (B) and is representative of the convergence to the minimum of a discrepancy between the first and the second unique output signal for the current calculation cycle (t) .

2. Circuit according to the previous claim, wherein the first electronic circuit of at least one sub-circuit is structured as a series of a circuit (Li) that provides at the output a set of signals, each one proportional to the logarithm in base n (or its approximation) of one and only one of the signals received at the input; a circuit (L2) that, receiving at the input the outputs of the previous circuit, provides at the output a signal proportional to the weighted sum of the signals received at the input; a circuit (L3) that, receiving at the input the output of the previous circuit, provides at the output a signal proportional to the base power p and of exponent proportional to the input signal (or its approximation) .

3. Circuit according to claim 3, wherein, in at least one sub-circuit, the series of said circuit L3 and ofthe second electronic circuit capable of providing an output signal proportional to the input signal ( i . e . , multiplied by an arbitrary factor ) is implemented by an electronic circuit configured to perform the p- based softmax operation, i . e . normali zed exponential in base p, of the input .4 . Circuit according to claim 3 , wherein, in at least one sub-circuit , the series of said circuit L3, of the second electronic circuit capable of providing an output signal proportional to the input signal ( i . e . , multiplied by an arbitrary factor ) and of the third electronic circuit capable of providing an output signal proportional to the product of the input signal and a predetermined matrix that receives as input the current or voltage output of the second electronic circuit , is implemented by a single electronic circuit configured to perform the operation equivalent to the product of the result of the p-based softmax operation, i . e . of normali zed exponential in base p, of the input for said predetermined matrix .5 . A circuit according to any preceding claim wherein the mathematical problem is representative of a neuron model , wherein said circuit further comprises an output circuit block configured to generate an activation signal of said neuron when the output voltage or current signal (Y) from the computational core is greater than a second value generated by an electronic circuit for generating random, pseudorandom, or chaotic numbers .6 . Circuit according to claim 5 , further comprising a pre-processing block configured to generate , by means of one or more hardwired or ( re- ) conf igurable hardware components , preferably implemented using at least oneof the following technologies : application-speci fic integrated circuit (AS IC ) , programmable logic device ( PLD) , FPGA device , at least said first and second plurality of electrical input signals to the computational core (Xi,t, . . . , Xm,t, Xi,t+i, . . . , Xm,t+i ) , and to :* receiving as input a plurality of signals ( Inputi, . . . , Inputm) coming from outside the circuit and relating to a current calculation cycle ( t ) ;* receiving an output voltage or current signal (Y) from the computational core calculated at the previous calculation cycle ( t- 1 ) ;* reading from memory at least a plurality of past inputs relating to at least one previous processing cycle ( t- 1 ) ; where said pre-processing block provides said computational core with said first and second plurality of electrical input signals (Xi,t, • • • , Xm,t, Xi,t+i, Xm,t+i ) normali zed, on the basis of said plurality of inputs , of corresponding excitatory or inhibitory matrices and of the output voltage or current signal ( y) from the computational core calculated at the previous processing cycle ( t- 1 ) .7 . Network of electronic neurons made according to any of the preceding claims .8 . Method of manufacturing an electronic circuit comprising the steps of :Performing the Forney factori zation of a multiparametric mathematical problem by reali zing a graph comprising a number k of nodes each of which performs a mathematical operation, receives at least one input message and generates an output message , each node presenting one or more vertices associatedwith messages exclusively in input to the node and a vertex associated with a message exclusively outgoing from the node wherein the graph comprises a number k of nodes and a number v of vertices , and wherein at least one node is an '=' node defining a generali zed product between at least two input messages to generate a single output message- manufacturing an electronic circuit having* a first plurality of m* r electrically conductive input tracks , relating to m electrical input signals (Xi,t, ..., Xm,t) of dimension r relating to the current calculation cycle ( t ) and representing said parameters ;And a block of electrical and electronic components connected to each other in which :* there is a second plurality o f m* r electrical ly conductive input tracks , relating to m electrical input signals (Xi,t-i, ..., Xm, t-i ) of dimensionality r relating to a previous calculation cycle ( t- 1 ) and representative of said parameters* there are at least k node circuits corresponding to the k nodes of the graph,* there are at least v electrical tracks each of which connects at least two node circuits carrying corresponding voltage or current signals representing the messages exchanged between the nodes according to the graph* at least one node-circuit is a '=' node-circuit , configured to receive at least two tracks as input and generate , on a single first output track in the case of a scalar output signal or on a single first multiplicity of r output tracks in the case of a signalof dimensionality r greater than or equal to 2 , a voltage or current output signal representing said output message , proportional to the generali zed product of the input signals ,* at least one node-circuit is a node-circuit corresponding to a 'product ' type node in the graph, configured to receive at least one track as input and generate , on a single second output track in the case of a scalar output signal or on a single second multiplicity of r output tracks in the case of a signal of dimension r greater than or equal to 2 , a voltage or current output signal representative of said output message , proportional to the product of the at least one input by a constant quantity,* wherein one of the tracks between the node-circuits and downstream of said '=' node-circuit or 'product ' node-circuit provides a voltage or current signal y relating to the message associated with it which represents the variational message passing solution of said mathematical problem and wherein the nodecircuits are configured to generate the corresponding output signals by means of one or more hardwired or ( re- ) conf igurable hardware components , preferably implemented using at least one technology between : application-speci fic integrated circuit (AS IC ) , programmable logic device ( PLD) , FPGA device .

Images