Recurrent neural network, recurrent neural network equipment and method for training recurrent neural network

Abstract

The invention discloses a recurrent neural network comprising a plurality of n-damped harmonic oscillators (DHOi), each of the n-damped harmonic oscillators being a unit (nci) of the neural network, an input unit (IU) adapted to receive and input time series input data (S (t)), a recurrent connection unit (RCU) comprising, for each unit (nci), a plurality of n-damped harmonic oscillators (DHOi), each of the n-damped harmonic oscillators (DHOi) being adapted to receive and input time series input data (S (t)). At least one connection (wi, j) between the input / output node (IOi) of the respective unit (nci) and the input / output node (IOj) of at least one other unit (ncj) for transmitting the damped harmonic oscillations (hi) output from the input / output node of the respective unit (nci) to the output / input node of the other unit (ncj).

Description

[0001] The present invention relates to a recursive neural network and a corresponding recursive neural network device, as well as a method for training such a recursive neural network.

[0002] Technical background and problems to be solved

[0003] Neural networks are well known in the art, for example, feedforward networks, deep convolutional neural networks (DNNs), memory computation networks, recurrent neural networks (RNNs), and other types of neural networks.

[0004] Despite their attractive properties, recurrent networks have been less studied compared to pure feed-forward DNNs because they suffer from exploding and vanishing gradient (EVG) problems and are difficult to train using the back-propagation through time (BPTT) algorithm. Many methods have been proposed to make the training of recurrent networks easier, ranging from gradient scaling under the constraints of the recurrent weight matrix to gated architectures.

[0005] N. Benjamin Erichson et al., “LIPSCHITZ RECURRENT NEURAL NETWORKS”, arXiv:2006.12070v3 [cs.LG] 24 April 2021, describes one approach, which treats the RNN as a continuous-time dynamical system described by linear components and a Lipschitz nonlinearity.

[0006] In other recent approaches, attempts have been made to exploit the complex dynamics of coupled oscillator networks for computation. For example, in an approach aimed at hardware implementation, M. Romera et al., "Vowel recognition with four coupled spin-torque nanooscillators," Nature 563, 230-234, DOI: 10.1038 / s41586-018-0632-y (2018), and M. Romera et al., "Reciprocal synchronization of binding events in spintronic nanoneurons," Nat. Commun. 13, 883, DOI: 10.1038 / s41467-022-28159-1 (2022), describe spin-torque nanooscillators as natural candidates for building hardware neural networks composed of coupled nanoscale oscillators, as potential hardware implementations.

[0007] T. Konstantin Rusch and Siddhartha Mishra, "Coupled Oscillatory Recurrent Neural Networks (CORNNs): An Accurate and (Gradient) Stable Architecture for Learning Long-Term Dependencies", arXiv:2010.00951v2 [cs.LG] 14 Mar 2021, (hereinafter Prior Art 1 = PA1) describes another approach that aims to analyze the performance of the corresponding algorithm, where a coupled oscillator model is used in an RNN to create a coupled oscillatory recurrent neural network (coRNN). The coRNN is trained using the BPTT algorithm, and the paper introduces that the coRNN can alleviate the EVG problem.

[0008] W. Moy et al., “A 1,968-node coupled ring oscillator circuit for solving combinatorial optimization problems,” Nature Electronics (2022), DOI: 10.1038 / s41928-022-00749-3, (hereafter referred to as Prior Art 2 = PA2) describes a scalable integrated circuit based on ring oscillators that could serve as an alternative to quantum, optical, and spintronic computing architecture approaches.

[0009] Various methods in the prior art either only focus on task performance without any hardware implementation, or focus on hardware implementation of relatively simple tasks, or focus on solving specific application problems. At least some of these methods require high computing power and / or specific temperatures, resulting in high energy consumption.

[0010] Therefore, the object of the present invention is to provide a computing architecture design solution that can exploit the complex dynamics of coupled damped harmonic oscillator networks and can be implemented equally efficiently in hardware, such as electronic digital or analog hardware, for example, in complementary metal oxide semiconductor (CMOS) technology, and in software, and can be reliably trained through a back-propagation through time (BPTT) algorithm with low energy consumption.

[0011] Solutions and Advantages

[0012] This object can be achieved by the recursive neural network device according to claim 1 or the method for training a recursive neural network according to claim 15.

[0013] Further developments are given in the dependent claims.

[0014] The RNN of this teaching may have several different features from known RNNs, one very powerful feature is that the natural frequency of the damped resonator is different.

[0015] Other optional powerful features include time delays in connections and / or different damping coefficients and / or multi-layer structures with specific connection structures in the same layer and / or different feedforward and feedback connections between layers and / or different distributions of natural frequencies between layers. By introducing oscillatory units, high-dimensional dynamic phenomena such as resonance, entrainment, synchronization, phase shift and desynchronization can be generated and high-dimensional dynamic landscapes in state space can be generated. The potential introduction of heterogeneity = inhomogeneity will enhance these effects.

[0016] Designs employing interactions of oscillatory units in recurrent networks allow for the design of computational architectures that can be implemented equally in hardware or software, and / or the networks can be efficiently trained using BPTT, and / or can be implemented in semiconductor devices where the final setting of the recurrent network parameters can be done after training using a software algorithm.

[0017] Other features and advantages can be seen from the description of the embodiments with reference to the accompanying drawings:

[0018] Figure 1 A connection diagram of a first embodiment of a recurrent neural network having n damped harmonic oscillators and connections between all n damped harmonic oscillators;

[0019] Figure 2 a) Schematic diagram of a damped translational mechanical harmonic oscillator; b) Schematic diagram of a damped electrical RLC harmonic oscillator; c) Schematic diagram of a damped harmonic oscillator implemented with analog electronic components;

[0020] Figure 3 A circuit diagram of a second embodiment of a neural network having n damped harmonic oscillators, showing only the connections between the first oscillator and the other oscillators;

[0021] Figure 4 Figure 4 A schematic circuit diagram of a damped harmonic oscillator in the second embodiment of a neural network, showing details of its input / output nodes;

[0022] Figure 5 a) Schematic diagram of VCO; b) Schematic diagram of VCO output power and frequency; c) Example diagram of CMOS implementation of VCO;

[0023] Figure 6 a) is the power spectral density variance of the MNIST data of the number 7 shown in b);

[0024] Figure 7 a) Waveform of one sample of spoken digital speech data with a length of 1 second; b) Calculation of the variance of the power spectral density of 1000 samples; c) Schematic diagram of the performance improvement of a single-layer NHHORNN and a double-layer NHHORNN with the same number of units in processing such data;

[0025] Figure 8 A schematic diagram of a recurrent neural network of a third embodiment, wherein a) is a general representation of the diversity of connectivity between network unit groups, and b) to d) are different representations for explaining connectivity;

[0026] Fig. 9 Schematic diagram of a double-layer recursive neural network in the fourth embodiment, where the first layer is Figure 8 Corresponding to the third embodiment;

[0027] Fig.10 A schematic diagram of a recurrent neural network of a fifth embodiment, wherein a) is a general representation of the diversity of connectivity between groups of network units, and b) is a different representation for explaining connectivity;

[0028] Fig.11 Schematic diagram of a three-layer recursive neural network of the sixth embodiment, wherein the first layer in a) is Figure 1 Corresponding to the first embodiment, b) is a schematic diagram of each layer, and

[0029] Fig.12 Example of recognition certainty levels in MNIST recognition using a small NHHORNN using this teaching.

[0030] In the following description of the embodiments, terms written with one or more bold letters, such as U or y or LC, are shown as ∈R n , which means that there are n vector elements U i or i or LC i (i = 1, 2, ..., n) whose elements are ∈ an element of the real number R. Terms written with one or more bold letters, such as W, are shown as W∈R n×m , represents an (nxm) element matrix (i=1,2,...,n;j=1,2,...,m), whose matrix element w i,j is an element of ∈ real number R.

[0031] The inventors started with biological neuronal systems such as the cerebral cortex. Neurobiological studies of the cerebral cortex have shown that these neuronal networks include highly complex nonlinear dynamics that can generate high-dimensional spatiotemporal patterns in a self-organizing manner. These patterns manifest as oscillations of varying frequencies, transient synchronization or desynchronization phenomena, resonances, entrainment, phase shifts, and traveling waves.

[0032] The present invention implements these principles implemented in natural neuronal systems in a new way, focusing on oscillatory units interacting in recurrent networks, for the design of computational architectures that can be implemented equally in hardware or software.

[0033] The superior performance of the present invention compared to prior art systems has been demonstrated by quantitative evaluation of the performance on a number of standard pattern recognition benchmark tasks. Specifically, the recurrent networks of the present teachings were tested on the well-known MNIST handwritten digits dataset, the more challenging EMNIST dataset consisting of handwritten digits and letters, the more difficult Omniglot dataset consisting of 1623 different handwritten characters of 50 different alphabets, and the speech recognition task performance on the free spoken digits dataset consisting of 3000 spoken digit recordings from 6 speakers. Different types of tasks have shown that the recurrent neural networks of the present teachings perform well in recognizing patterns in time series data, regardless of the actual content type of the task.

[0034] The biologically inspired recurrent neural networks of this teaching are particularly suitable for low-energy hardware implementation, and they are superior in learning speed, noise tolerance, hardware requirements, and number of system variables compared to prior art RNNs with gated architectures such as long short-term memory (LSTM) and gated recurrent unit (GRU) networks. In addition, by analyzing these recurrent neural networks in this teaching, intrinsic mechanisms that can improve task performance can also be identified.

[0035] The recurrent neural network (RNN) of this teaching will be explained starting from an RNN developed by the inventor as a basic model. Figure 1 It is explained in the form of a schematic diagram.

[0036] The basic model of RNN can be viewed as consisting of n damped harmonic oscillators (DHOi, i = 1, 2, ..., n) that have a common basic dynamics, i.e., an oscillation characteristic defined by a natural frequency ω0 and a damping coefficient γ. Such an RNN is a homogeneous recurrent neural network composed of identical DHO units. These units are connected in an all-to-all manner, without conduction delays, and all nodes are tuned to the same preferred frequency (see Figure 1 ). In this teaching, such a homologous recurrent network composed of identical DHO units will be referred to as a homologous harmonic oscillator RNN (HHORNN), representing a first non-limiting embodiment of the present invention. Such HHORNN has been trained using the back propagation through time (BPTT) algorithm on a set of benchmark tasks and allows gain adjustment of all connections (including input connections, recurrent connections, and output connections of network nodes).

[0037] The oscillation of each single DHO unit of such a HHORNN can be viewed as obeying the following general second-order differential equation of a damped resonator (1)

[0038]

[0039] βi = Damping coefficient, ω 0i = natural frequency of undamped oscillation, e.g. β i =c i / 2m i and ω 0i 2 =k i / m i , m i = mass, k i = spring constant, c i = viscous damping constant for the corresponding damped translational mechanical resonator DHOi (see Figure 2 a). or β i =R i / L i and ω 0i 2 =1 / L i C i , R i = Resistance, L i = Inductance and C i = Capacity for the corresponding damped electrical RLC harmonic oscillator DHOi (see Figure 2 b) Another example of a damped harmonic oscillator is Figure 3 The following Figure 2 c), where DHO is implemented by analog components such as operational amplifiers, inverters, and potentiometers. In the example shown, for example, two operational amplifiers are used as integrators I1 and I2, an inverter and multiple potentiometers are used as weighting coefficients wi,j, and 2β is used. i Represents (and sets) the damping coefficient and ω 0i represents (and sets) the natural frequency. Since potentiometers cannot achieve coupling / weighting coefficients wi,j greater than 1, operational amplifiers or other amplification elements can be used instead of potentiometers if necessary. In HHORNN, for all i, j, β i =β j and ω 0i 2 =ω 0j 2 However, the above notation with indices i, j is also used for HHORNN, since this more general description can also be used to describe DHO for other RNNs in this teaching.

[0040] The electrical characteristics of a coupled RLC circuit network designed to have some of the characteristics of a biological neural network can be defined as follows: A series RLC circuit driven by an external input has Kirchhoff's equation for voltage valid, which is

[0041] U L +U R +U C=U ext (2)

[0042] Introducing the resistor R, inductor L and capacitor C into formula (2), we can get formula (3)

[0043]

[0044] An n additively coupled DHO system implemented by an RLC circuit described by equation (3) is composed of a time-varying external input U ext (t)∈R n Drive, differentiate once to get the ordinary differential equation system, which can be input I(t)∈R n The vector form of is written as equation (4)

[0045]

[0046] The following are the important features of the RNN of this teaching. We assume and design such a neural network: any external input to any DHO is determined by the nonlinear coupling of the activity of the entire recurrent neural network with the external signal. This nonlinear coupling can be expressed by formula (5)

[0047]

[0048] Where W∈R n×n Denotes the pairwise coupling weight w i,j As an (nxn) element matrix (i=1,2,...,n; j=1,2,...,n), S ext =S ext (t)∈R n is the vector of input signals.

[0049] Substituting the first-order differential equations x=I(t) and y=dI(t) / dt into equation (4), we obtain the final equation (6):

[0050]

[0051] Equation (6) describes Figure 1 The HHORNN shown in Figure 1, where DHO is implemented as an RLC circuit, is Figure 2 b. However, in order to train HHORNN using BPTT, a discrete-time description of HHORNN is required. This in turn requires a numerical integration scheme. To this end, the well-known Euler discretization method can be used to discretize (6) with a time constant τ, resulting in equation (7)

[0052]

[0053] x t+1 =xt +τy t (7)

[0054] like Figure 1 As shown, the above equations describe a HHORNN where the DHO is implemented as an RLC circuit, as Figure 2 As shown in b, it can be trained as a recurrent neural network. Using BPTT to train HHORNN means adjusting W∈R n×n The pairwise coupling weights w in i,j , so that the error (cost) is minimized. In other words, the connection w between the units DHOi i,j Will be in Figure 1 This training includes the coupling weights of the input and output, which can also be seen from the description of the input unit and the output unit below.

[0055] Each connection w between units DHOi i,j Can be set (adjusted) to the transmission characteristic T i,j =w i,j xh i , where w i,j ∈R is the pairwise coupling weight, |w i,j |≤10. If the pairwise coupling weight is set to |w i,j |=0, the connection from unit DHOi to unit DHOj does not exist or is interrupted.

[0056] The recurrent neural network of the present teachings includes not only the HHORNN described above. The recurrent neural network of the present teachings also includes recurrent neural networks composed of non-homogeneous DHO. Non-homogeneous means that they are not composed of only one type of DHO units, which have the same β = damping coefficient and the same ω0 = natural frequency of the DHO undamped oscillation, but the multiple n damped harmonic oscillators (DHOi, i = 1 to n) of the RNN include at least two different types of damped harmonic oscillators, wherein each of the at least two different types of damped harmonic oscillators has at least one damped harmonic oscillator in ω. 0i =The natural frequency parameters of undamped oscillation are different from those of other types of damped harmonic oscillators.

[0057] In the present teachings, such a recurrent network consisting of DHO units having at least two different types of damped harmonic oscillators will be identified as a non-homogeneous harmonic oscillator RNN (NHHORNN). Such a NHHORNN represents a second non-limiting embodiment of a recurrent neural network in the present teachings. Figure 3 and Figure 4 A NHHORNN with n damped harmonic oscillators is described.

[0058] exist Figure 3In the diagram, n damped harmonic oscillators DHOi (i = 1, 2, ..., n) are shown in FIG. Figure 1 A circular arrangement similar to Figure 3 The circular arrangement is only for illustration purposes to simplify the connection diagram. The number of damped harmonic oscillators n ≥ 8 in this teaching, therefore, if n = 8 is selected, Figure 3 The representation in shows all 8 damped harmonic oscillators DHOi, i=1,2,...,8. The damped harmonic oscillators DHOi are shown as damped electrical RLC harmonic oscillators. Each damped harmonic oscillator comprises a resistor R i , an inductor L i and a capacity C i , can be set or predetermined individually for each harmonic oscillator DHOi. Thus, in particular the damping factor β of each damped harmonic oscillator i and the natural frequency of the undamped oscillation ω 0i can be set or predetermined independently of the corresponding parameters of each other damped harmonic oscillator DHOj. A NHHORNN comprises at least two damped harmonic oscillators DHOi, whose natural frequency ω of undamped oscillation 0i It should be emphasized that HHORNN and NHHORNN are not limited to being implemented using RLC type electrical oscillators, but can also be implemented using mechanically damped harmonic oscillators such as DHO. In this case, Figure 2 a) Schematic diagram of the corresponding damped translational mechanical harmonic oscillator. Following the same general second-order differential equation (1) for the damped translational mechanical harmonic oscillator DHOi, β i =c i / 2m i and ω 0i 2 =k i / m i ,m i = mass, k i = spring constant, c i = viscous damping constant. Other alternatives are, for example, the use of secondary (electroplating) cells, chemical oscillating systems such as coupled Belousov-Zabodzinsky reactions, and of course implementation in computer programs / software. Figure 3 under Figure 2 c) shows the representation of an analog DHOi, it is obvious that one or more RLC type electrical oscillators can be replaced by such an analog DHOi.

[0059] exist Figure 3In the NHHORNN embodiment shown, full-to-full connectivity including self-connectivity is provided, i.e., the connection between DHO and itself, which means that the input / output nodes IO of all damped harmonic oscillators DHOi are i All other damped harmonic oscillators DHOj have input / output nodes IO j And its own input / output node IO i Make input and output connections. Figure 3 In FIG. 1 , for ease of illustration, only the connections from the first damped harmonic oscillator DHO1 to all other damped harmonic oscillators DHO2 to DHOn, i.e., w1,2, w1,3, ... to w1,n, and the connections from the input / output nodes of all other damped harmonic oscillators DHOj (j=2, 3, ..., n) to DHO1, i.e., w2,1, w3,1, ... to wn,1, and the connection w1,1 to itself are listed. Other connections in the full-to-full connection may also exist, but are not shown to simplify the illustration. The characteristics of the wi,j connection and its implementation will be further described below.

[0060] See also Figure 4 , showing in more detail a cell nc of the damped harmonic oscillator DHOi i and its input / output nodes IO i . A single dynamic harmonic oscillator DHOi and its input / output nodes IO i A unit nc that constitutes a recurrent neural network i . Input / output node IO i It consists of a generating unit input x i Input connection IC i . Input connection IC i Including adder ICA i , receives all inputs from itself and all other units of the recurrent neural network through the connection wj,i, if the corresponding input / output node IO of the corresponding unit i Connected to the input receiving the external signal S to be processed, the adder ICA i The corresponding input signal S will be received in due time i (t). Therefore, S i The inputs of (t) are shaded arrow lines to indicate potential connections. Figure 2 The same is true for c), where the input S i (t) is indicated by a shaded arrow line. The signal S and the input of the signal S will be further described below.

[0061] Adder ICA i The output terminal is connected to the input element ICS i, the input element implements a nonlinear, preferably saturating nonlinear, transfer function such as a sigmoid-like, or more preferably a hyperbolic tangent-like, transfer function with an optional offset, so as to obtain the output from the adder ICA i The output signal generation unit input x i For example, US2018 / 0307977 A1 describes a semiconductor circuit for realizing a nonlinear saturation such as a sigmoid-like transfer function in a single-type transistor technology or a CMOS technology. Therefore, since US2018 / 0307977 A1 includes a large number of examples, the input element ICS is omitted here. i A description of the specific hardware implementation.

[0062] Input / output node IO i Includes output connection OC i , which will generate the damped harmonic oscillation h i Output to the corresponding connection wi,j. Figure 4 The inputs and outputs of the DHO are not shown in detail. As a non-limiting example only, Figure 5 a) to c) show a voltage controlled oscillator (VCO), which can be an implementation of a DHO. This VCO can be implemented using CMOS technology, such as Figure 5 c). Figure 5 c) Input voltage V in i and output voltage V out i are displayed at the corresponding nodes. In this regard, it must be noted that in the case of this complementary CMOS VCO, the input and output voltages are phase-shifted by 180°, so an inverter can be added at the input or output if both are in phase. However, for the purpose of implementing HHORNN and NHHORNN and calculating the characteristics of HHORNN / NHHORNN, it does not matter whether there is a phase shift between the input and output, as both are possible in this implementation.

[0063] Look again Figure 3 and Figure 4 , the representation of input connection wj,i and output connection wi,j also includes and These symbols represent optional time delays in the connection, which may be present but do not have to be present (= optional), as will be explained further below.

[0064] All these connections wi,j form a recursive connection unit RCU, which includes the input / output nodes IO of all units of the recursive neural network i All connections wi,j between them. The recursive connection unit RCU is composed of Figure 3The shaded box RCU in the figure represents the recursive connection unit RCU. All connections of the recursive connection unit RCU can be described by a (nxn) matrix W, and the matrix elements are wi,j (i,j = 1, 2, ..., n).

[0065] These recursive neural networks implemented as HHORNN or NHHORNN are suitable for effectively processing time series input data S(t). The recursive neural network includes an input unit IU for receiving and inputting such time series input data S(t) and an output unit OI suitable for outputting a result, i.e., output data O(t).

[0066] In its simplest form, an input unit IU for receiving and inputting such time series input data S(t) is an input connection IC to an input / output node of one of the units DHOi. i A single connection, such as Figure 4 The simplest form of an output unit OI for outputting a result, ie, output data O(t), is an output connection OC connected to the input / output node of one of the units DHOi. i , which unit can be the same as or different from the unit connected to the input unit. In addition, the output unit can also start outputting at a start time OST, which start time is set later than the first input of the input data S(t), and after receiving the input data S(t), the time series interval ti1 of the input data S(t) is at least five times, preferably ten times, and more preferably twenty times ti1. The output time interval to2 of the output data can be the same as ti1 or different. The number of units constituting the input unit or connected to the input unit, and the number of units constituting the output unit or connected to the output unit can be freely selected. For example, if the recursive neural network is composed of n=16 damped harmonic oscillators, all 16 units can be connected to receive the input data S, and 8 of the 16 units can come from the output unit. Or 8 of the 16 units are connected to receive the input data S, and the other 8 of the 16 units receive data from the output unit, and so on.

[0067] The simplest form of the recurrent neural network of the present teaching is a network with n=8 damped harmonic oscillators. There is no limit to the number of damped harmonic oscillators per se, but for practical considerations the number of DHOs in a network layer may be limited to 10. 6 , or preferably 250000 or 102400 or 40000 or 10000 or 2500 or 1024 or 512.

[0068] Figure 3 The above RNN in can be described by formula (8)

[0069]

[0070] This description corresponds to equation (6), except that the scalars R / L and 1 / LC and 1 / L are replaced by the corresponding vectors. In order to train the NHHORNN, a numerical solution method is required. For this purpose, the well-known Euler discretization method can be used to discretize equation (8) with time constant τ, resulting in equation (9)

[0071]

[0072] x t+1 =x t +τy t (9)

[0073] The above equation describes Figure 3 , Figure 4 The NHHORNN shown in Figure 1 can be trained as a recurrent neural network. Of course, if all i,j, β i =β j and ω 0i 2 =ω 0j 2 , the above equation can also be used to describe Figure 1 HHORNN shown.

[0074] The original setup of this recursive neural network and its training in this teaching will be explained below.

[0075] As mentioned earlier, training recurrent neural networks using BPTT techniques is fundamentally difficult due to the widely studied exploding and vanishing gradient problem (EVG problem). The EVG problem occurs when the gradient of the loss function used to train long-term dependencies cannot be properly propagated through the recursive iterations of the recursive network. However, HHORNN and NHHORNN do not have this problem. The mathematical proof is given in PA1 and will not be discussed here.

[0076] For electrical hardware implementations, especially CMOS or other semiconductor implementations, one conclusion can be drawn: for given values ​​of R and L, the value of the capacitance C determines the characteristics of the DHOs. Especially in the later stages of semiconductor implementation, when the overall structure of the HHORNN or NHHORNN has been determined, but the HHORNN or NHHORNN still needs to be trained, the value of L may be difficult to set, both initially and to learn "new things". However, at a later stage, the capacitance can be set gradually or even continuously over a relatively wide range relatively easily by means of a set of varactors (=variable capacitors), which are mature semiconductor components and can be changed at any time by programming or fixed by fusion or other means at a later stage of hardware implementation.

[0077] Therefore, in the case of HHORNNN, the same parameters β and ω0 are chosen for all DHOs, or the possibly different parameters β of NHHORNNN are chosen for the original or initial setup of HHORNN / NHHORNN to be trained. i and ω 0i , is a problem of the original setting. In general, it is possible and effective to just choose a ω0 for the DHOs of the HHORNN so that it is within (= overlaps) the expected frequency range of the input signal. In general, it is possible and effective to just choose a different ω0 for the DHO of the NHHORNN so that it can cover a wider frequency range, preferably overlapping the reception frequency range of the DHO. Overlapping reception frequency range means that, with reference to Figure 5 b) The corresponding DHO has an output power amplitude of at least 1 / 10 or 1 / e of the maximum output power when oscillating at the corresponding frequency. Of course, this wide frequency range preferably overlaps with the expected frequency range of the input signal. For HHORNN / NHHORNN, the above overlap is not strictly required, but we prefer it to improve performance.

[0078] Thus, an input signal S(t) is generated. The present invention is intended to process time series input data (S(t)). These time series input data (S(t)) can be either discrete or continuous time series input data (S(t)), having a time series interval ti1 and a time series length TI1, wherein the time series interval ti1 of the time series input data (S(t)) represents the time interval between subsequent discrete values ​​in the case of discrete time series input data (S(t)), and represents the time interval between subsequent sampled values ​​in the case of continuous time series input data (S(t)) such as a sinusoidal signal. In other words, the data to be processed is represented by such time series input data (S(t)).

[0079] If the data to be processed is the MNIST dataset consisting of 60,000 samples of handwritten digits, each of which consists of 28×28 intensity values, representing a handwritten digit from 0 to 9, then by collecting the pixels of each digit from top left to bottom right in scan line order, these data can be converted into a time series dataset D of length TI1=784. The clock frequency 1 / ti1 used to input this time series data sets the time interval ti1. Note that the clock frequency corresponding to the time series interval ti1 can be freely selected.

[0080] If the data to be processed is speech data, where each data set consists of real-time speech data of a certain length and the relevant frequency range is 500 to 16000Hz, it is sampled at a specific sampling rate and converted into a time series data set with a time series interval of ti1 and a time series length of ti1.

[0081] No matter what the contents of the data to be processed are, once they are originally in the form of time series input data (S(t)) or are converted into time series input data (S(t)), the time series input data (S(t)) to be processed, such as the MNIST dataset or any other dataset, represents multiple samples of the time series input data (S(t)) to be processed, and the multiple samples of the time series input data (S(t)) to be processed have power spectral densities and variances of these power spectral densities.

[0082] Parameter ω 0i =NHHORNN The natural frequencies of the undamped oscillations of the damped harmonic oscillators of different types are preferably set to be based on the determination of the power spectral density of multiple samples of the time series input data (S(t)) to be processed and the variance of these power spectral densities, and the natural frequencies are distributed on the peak of the variance, wherein one natural frequency is set to correspond to the peak of the variance with the lowest frequency, and the other natural frequencies are distributed within the frequency range of the variance. As for HHORNN, the parameter ω0=NHHORNN's natural frequency of the undamped oscillations of the damped harmonic oscillator can be selected based on the power spectral density of multiple samples of the time series input data (S(t)) to be processed and the variance of these power spectral densities. The natural frequency ω0 (for the case of HHORN) or the natural frequency ω 0i It is preferably set to correspond to the peak value of the input power spectral density variance.

[0083] The input / output node IO of a corresponding unit of the recursive connection unit RCU i I / O node with another unit j The connection wi,j between them is set to have the transmission characteristic T i,j =w i,j xh i , w i,j ∈R,|w i,j |≤10. In other words, the output strength / amplitude of a unit can be expressed by |w i,j |=0 is set to 0, or 1<|w i,j |≤10 is set to amplify, or |w i,j |=1 is set to keep the same level, or |w i,j |<1 is set to attenuate, and the output can be inverted with the corresponding amplitude. You can also set |w i,j|≤5 or ≤3 or ≤2. This can be achieved by using a correspondingly damped harmonic oscillator output h i The PA2 can be realized by a voltage divider, transmission gate coupler or other known circuits. Figure 2 An example of such a transmission gate coupling is shown, which allows setting 5 different inversion levels. Other circuits can be implemented using amplifiers with preset amplification ratios, which are preset by values ​​stored in a memory bank used to set the connection values.

[0084] exist Figure 1 HHORNN and Figure 3 In the training of NHHORNN, these connections and their features are the elements that need to be trained.

[0085] As mentioned earlier, despite Figure 3 The NHHORNN in does not include such transmission delays, but the input / output node IO of a corresponding unit in the recursive connection unit RCU i The input / output node IO of another unit j The connection wi,j between them can not only be set to the above transmission characteristic T i,j , can also be optionally set as a transmission delay. Such an output time delay k i,j =0,1,....,k,0≤k≤kmax,ti1 / 10≤Δt≤ti1 and k i,j xΔt≤50 ti1, for example, can be connected via a series connection of an inverter or used to transmit the corresponding damped harmonic oscillator output h i If this propagation delay is provided, NHHORNN can be realized using the matrix W k Multiply by the corresponding time t k =t-(k xΔt) to replace the matrix W term in equation (9), thereby obtaining equation (10).

[0086]

[0087] x t+1 =x t +τy t (10)

[0089] The maximum value kmax of k can be set to 1000 or 500 or 100 or less than 100, such as 50, 40, 30, 20, 16 or 12 or 10 or 8 or 5 or 4 or 3 or 2. Preferably, the kmax value is in the range of 30 to 10. It is worth noting that the introduction of delay does not increase the complexity of the system description, because for all k, the matrix W kAt each position i, j contains only one element ≠ 0. Therefore, the computational workload of training does not increase, but the performance of HHORNN / NHHORNN is improved.

[0090] The input signal S(t) is input through the input unit IU, which is connected to the input connection IC of the input / output node of one or more units nci. i The input unit IU can consist of selected units from the units nci of the neural network itself or of individual units. The input signal can be input to one unit or to nI units in this way, nI≤n. The data forming the input signal can be input to multiple units serially or in parallel. Therefore, the input signal can be summarized as

[0091] S ext (t) = WI·D(t) (11)

[0092] Among them, S ext =S ext (t)∈R n is the vector of input signals, WI∈R n×q Denotes the pairwise coupling weight wi i,j , as a (n×p) element matrix (i=1,2,...,n; p=1,2,...), for input into one or more units nci of the selected (predetermined) neural network to receive input from the input unit. For example, if only one unit ncr in the neural network unit nci receives input, it can be described as follows: only the matrix elements wi,j for i=r (i-th row) can be different from 0, and for i≠r (all other rows), the matrix elements are 0. If only one data value is input per input cycle, then the vector D(t)∈R q There is only one potential value of vector element d in R ≠ 0, in other words, q = 1. If q ≠ 1 data values ​​are to be input in the same input cycle (time interval ti1), then the vector D(t)∈R q There are q elements with potential value ≠ 0.

[0093] The above description of HHORNN and NHHORNN provides a design tool for studying how the electrical properties of DHOs and their connections, i.e., "neural circuits", determine the learning ability of the system, and thus the overall ability to train the network, and which design changes will lead to which ability or performance changes. The above description of HHORNN and NHHORNN also provides a tool for giving DHOs and their connections inputs and outputs, in other words, HHORNN / NHHORNN can be implemented in software and run on a correspondingly programmed data processing device. HHORNN / NHHORNN can also be implemented in hardware through mechanical elements such as spring-supported weights, and important nonlinear saturation input elements can be implemented through viscoelastic input elements or hydraulic or pneumatic input elements with such force transmission. However, hardware implementation in the form of circuit implementation is currently preferred, especially semiconductor circuit implementation.

[0094] The training of HHORNN / NHHORNN is performed on a series of classification benchmark tasks commonly used to evaluate the performance of RNNs. The same training is also performed on LSTM and GRU gated networks, which are the mainstream RNN architectures in the current machine learning field.

[0095] In all cases, the input data was presented sequentially to the network in the form of time series data, and the networks were trained in a supervised manner using backpropagation through time (BPTT). All networks were implemented in the PyTorch framework and trained using BPTT. To achieve comparability between different network architectures, the network size was chosen so that the effective number of trainable parameters for all networks was the same, and 10,000 parameters were chosen by default unless otherwise stated. 128 training batches and the AdamW optimizer were used for all tasks. For network architectures with gradient explosion and gradient vanishing problems (GRU, LSTM), parameter searches were performed in the space of learning rate and gradient scaling values ​​to determine the best configuration of learning rate and gradient scaling for each model. As a measure of the network's performance on the task, the classification accuracy on the test set was calculated. To evaluate the learning rate, these accuracy rates were calculated throughout the training process, uniformly sampled across m training batches.

[0096] For HHORNN and NHHORNN, which do not have the problem of gradient vanishing or exploding, a learning rate of 0.01 was used in all experiments and no gradient scaling was applied. In order to determine the impact of the intrinsic parameters β and ω0 on the performance of HHORNN, either β was simultaneously grid-searched as described below using the power spectral density information of multiple samples to estimate the optimal ω0 value from the processed dataset; or a grid search was performed on each dataset to evaluate the learning performance of the HORNN network, where all nodes had the same intrinsic parameters β and ω0 selected from the grid. For each dataset, the parameter set of the network with the highest learning performance was selected, and this HHORNN network with the same intrinsic parameters for all nodes was named the optimal HHORNN network HHORNNopt. In addition, the distribution of each intrinsic parameter in the set of 10 HHORNN networks with the highest learning performance was determined to obtain their minimum and maximum values ​​in the considered set. Using these minimum and maximum values, non-homogeneous HORNN networks were generated, in which each node was assigned a random value, such as β i ∈U([βmin,βmax]) and ω 0i ∈U([ω0min,ω0max]), where U([a,b]) denotes a uniform distribution on the interval [a,b]. The inhomogeneous HORNN network and the optimal HHORNN network HHORNNopt were tested to evaluate the impact of the intrinsic oscillation parameters. In addition, overdamped, critically damped, and underdamped networks were tested, but these tests are not relevant to this paper.

[0097] The connections wi,j are initially set according to different strategies to evaluate their impact. The connections wi,j are initially set to be drawn from a uniform distribution A random value selected from , where n represents the number of DHO units in the network. Alternatively, according to a Gaussian distribution Choose weights. These different choices do not have a big impact on the final task performance and learning speed. It is also possible to initially set all connections wi,j to wi,j = 1 or wi,j = 0. In addition, multiple network instances with different initial weights were trained on the same task, and the final task performance was found to be not very related to the weight initialization.

[0098] Of course, the choice of the discrete constant value has a significant impact on the learning performance. For example, values ​​between 0.01 and 0.5 were tested for MNIST (0.01, 0.05, 0.1, 0.2, 0.3, 0.4, and 0.5), and 0.2 was chosen because it performed best.

[0099] With the same number of learnable parameters, HHORNN has far outperformed the latest RNNs with gated units such as LSTM and GRU in terms of learning speed, absolute performance, and noise tolerance, especially in the low parameter count region.

[0100] Accordingly, NHHORNN outperforms HHORNN in all aspects and thus outperforms state-of-the-art RNNs such as LSTM and GRU.

[0101] The inventors can identify several reasons for the enhanced functionality of HHORNN and NHHORNN:

[0102] (i) The time constant introduced by the discretization of the damped resonance equation plays the role of smoothing prior and regularized gradient in the state space, making BPTT learning more stable;

[0103] (ii) The specific dynamics of DHO provides each node with its own memory function, which helps to encode temporal relationships and makes the node sensitive to the sequential order of the input;

[0104] (iii) The prototypical properties of coupled DHOs can generate dynamical phenomena such as resonance, entrainment, synchronization, phase shift, and desynchronization, thereby generating high-dimensional dynamical landscapes in the state space; and

[0105] (iv) The introduction of inhomogeneous DHO, in other words, the increase in network heterogeneity, amplifies the effects of (i), (ii) and (iii), enabling this heterogeneous = inhomogeneous HORNN to respond to stimuli in more complex and asynchronous patterns.

[0106] Reason (iv) even causes NHHORNN to classify stimuli significantly above chance in the untrained state before learning / training. Although HHORNN also has this property, the effect of NHHORNN is stronger. Heterogeneity = Heterogeneous HORNN projects the input into a high-dimensional pre-structured state space even in the untrained state and randomly selects connections wi,j, which simplifies training.

[0107] In other words, the learning / training process exploits the initial diversification of response patterns, requiring fewer trials to reach higher performance levels.

[0108] Regarding reason (i), it is important to emphasize that the discretization of the time constant τ does not affect the hardware implementation. Regardless of whether HHORNN or NHHORNN is implemented in hardware or software, discretization is merely a tool for numerically computing the model and thus training it via BPTT. HHORNN and NHHORNN and any of their constituent units are capable of converting any input signal into oscillations in both hardware and software. Digital signals such as triangular pulses are converted into analog signals, and these analog signals (oscillations) make the resonance phenomenon more intense. This fact stems from the design of HHORNN and NHHORNN and is an inevitable inherent characteristic of their hardware and software implementations and has nothing to do with the discretization of the time constant τ, which is only used for efficient training. The above-mentioned introduction of delays in the wi,j connection increases heterogeneity and improves overall performance for the same reason as the different preferred frequencies of DHO. As a possible rule for the initial setting of such delays, a coupling delay dispersion can be introduced around a median value that is approximately equivalent to 1 / 5 of the median cycle time of the preferred oscillation frequency. However, this is only a non-limiting example. Other distributions of delay values ​​are possible, such as a uniform distribution of delay values ​​between zero delay and 20 times the time series interval ti1. The introduction of delay has two effects: first, it increases heterogeneity; second, it introduces a potential reaction delay of the network, which can be viewed as a potential delay time interval before providing an output and may represent an intermediate processing result.

[0109] The following explanation may provide clues to better understand the consequences of the choice and distribution of the DHO natural frequencies. Each DHO is actually a bandpass filter, whose "receptive field" is defined by the resonant frequency window where amplitude amplification occurs (see Figure 5 b). A single isolated DHO unit of a HHORNN / NHHORNN without feedback connection can be considered to produce a time convolution of the input signal and act as a bandpass filter that selectively amplifies signals near its natural frequency ω through resonance. When self-connections are possible and foreseeable in the HHORNN / NHHORNN through potential connections wi,i, adjusting the gain of the self-connection can shift the natural frequency ω to some extent. In other words, a DHO with self-connection can adjust the input to some extent and change its receptive field during the learning process by changing the value of wi,i.

[0110] As described above, whether HHORNN / NHHORNN is implemented by hardware or software, the input for training and detecting performance can be a pulse input or an analog input, such as an oscillating input in a narrow sense. The HHORNN and NHHORNN in this teaching can be used for both discrete input and analog oscillating input.

[0111] As mentioned above, if the dataset has a clear structure, such as the MNIST data, the units can be pre-tuned according to the characteristic frequency. The best results are achieved if the resonant frequency is set at the peak of the variance, because this makes the network resonate with the difference in the input signal (i.e. the stimulus). Figure 6 a) shows Figure 6 b) The power spectral density variance of the MNIST data of the number 7 shown, which is the time series input data S(t) of the MNIST dataset, consisting of 28 times 28 intensity values ​​of the number 7. As mentioned above, the pixels of each number are collected in scan line order from the upper left to the lower right and converted (transformed) into time series data D with a length of TI1=784.

[0112] In such an MNIST dataset, a vertical line has a period of 28, a line tilted 45 degrees has a period of 27 or 29, etc. The period is "converted" to a frequency by inputting a time interval.

[0113] This example also helps to understand different strategies for choosing DHO eigenfrequencies. If the data to be processed has a clear structure, one can choose a high resonance at a precisely chosen frequency; if there is no clear structure, one can choose a broader tuning. The design and effectiveness of multilayer HHORNN / NHHORNN are explained below.

[0114] Using speech data, performance was tested with spoken digits. The "Free Spoken Digits Dataset" used contains 3000 samples of spoken digits from 0 to 9, where each sample represents a 1 second mono recording with a sampling rate of 8000Hz and a subsampling rate of 4000Hz. These 3000 samples represent spoken digits from 6 male speakers, with each speaker having 50 samples per digit. This approach produces a relatively small (and therefore more difficult to learn) dataset of 3000 samples. 10% of the data is used as test data, i.e. 300 samples, while 90% of these data are used as training.

[0115] Figure 7 Shown are a) a waveform of one sample of length 1 second, and b) the calculated variance of the power spectral density of 1000 samples.

[0116] The amplitude values ​​are fed directly to the network as time series data, in other words, as a 4000-dimensional vector.

[0117] The first NHHORNN trained and fed with data had a network size of n = 256 units arranged in a single layer with all-to-all connectivity.

[0118] The second NHHORNN has n=256 units but is divided into two layers and is given the same task, i.e. training and input with the same data. The second NHHORNN is divided into a first layer with n1=128 units and a second layer with n2=128 units.

[0119] Before describing the layers and initial setup in detail, some definitions are given to describe multi-layer HHORNN / NHHORNN. Assume a two-layer network has a total of n units, of which n1 units are located in the first layer, which receives the input signal and is therefore the upstream layer in the entire input-output direction; n2 units (n=n1+n2) are located in the second layer, which is the downstream layer in the entire input-output direction, and only the units in the second layer are connected to provide output signals. The all-to-full connectivity within a layer is the same as the single-layer network described above. Forward connection / connectivity refers to the input / output node IO of at least one unit in the n1 units nci of the upstream (e.g., first) network layer. i The input / output node (IO) of at least one unit in the downstream (such as the second) network layer unit ncj j ) for connecting the node IO of at least one of the n1 units nci in the upstream (eg, first) network layer i To the input / output node (IO) of at least one of the n2 units (ncj) in the downstream network layer j ) The damped harmonic oscillation generated by the transmission i Output. Feedback connection / connectivity refers to the input / output node IO of at least one unit in the downstream (such as the second) network layer n2 units ncj j The input / output node (IO) of at least one unit in the unit nci in the upstream (eg, first) network layer i ) for connecting the node IO of at least one of the n2 units ncj in the downstream (eg, second) network layer j To the input / output node (IO) of at least one of the n1 units (nci) in the upstream network layer i ) The damped harmonic oscillation generated by the transmission j Output. The maximum / minimum forward connectivity of X / Y% refers to the maximum or minimum X% of 100% of the input / output nodes in the n1 units nci of the upstream network layer that are likely to be connected to the maximum or minimum Y% of the input / output nodes in the n2 units ncj of the upstream network layer, for outputting the resulting damped harmonic oscillation h of the corresponding input / output nodes of the n1 units nci of the upstream network layer. i Passed to the input / output node IO of the downstream network layer n2 unit ncj jThe maximum / minimum feedback connectivity of X / Y% accordingly means that the maximum or minimum X% of 100% of the input / output nodes in the n2 units ncj of the downstream network layer are likely to be connected to the maximum or minimum Y% of the input / output nodes in the n1 units nci of the downstream network layer, for outputting the resulting damped harmonic oscillation h of the corresponding input / output nodes of the n2 units ncj of the downstream network layer j Passed to the input / output node IO of the upstream network layer n1 unit nci i . Accordingly, if the corresponding maximum / minimum forward or feedback connectivity is N / M, such as 32 / 64, then the corresponding maximum and minimum values ​​are integers rather than percentages. The potential connection refers to the matrix W in equation (9) or the matrix W in equation (10) k includes corresponding matrix elements wi,j that may be different from 0, even though as a result of training the corresponding elements may eventually be set to 0. Accordingly, in a hardware implementation, there are corresponding connection elements wi,j that, as a result of training, may be set to the corresponding transfer characteristic T i,j =w i,j xh i , where w i,j ∈R and |w i,j |≤10.

[0120] The preferred frequency of the DHO in the first layer of the second NHHORNN is set to be the same as that of the first single-layer NHHORNN, that is, distributed within the input data frequency range of the input data power density spectrum variance. In the second NHHORNN, the preferred frequency of the DHO in the second layer is set to be lower than the frequency of the first layer, that is, the resonant frequency in the second layer is set to 1 / 5 (20%) of the resonant frequency of the first layer. Other options are also possible, such as 1 / 2, 1 / 3, 1 / 4, 1 / 6, 1 / 8, 1 / 10 or 1 / 20 of the resonant frequency. The preferred frequency of the second layer DHO can also be higher than the frequency of the first layer, such as 2, 3, 4 or 5 times the resonant frequency of the first layer.

[0121] In the second 128+128NHHORNN, the 128 units in the first layer have all-to-all connectivity (= 100% forward connectivity and 100% feedback connectivity and self-connectivity within the layer), and the 128 units in the second layer have all-to-all connectivity (= 100% forward connectivity and 100% feedback connectivity and self-connectivity within the layer). The forward connectivity of the units in the first layer is set to 100%, and the feedback connectivity of the units in the second layer is set to 50%. In other words, the forward connectivity between the two layers is 100%, and the feedback connectivity is 50%.

[0122] Figure 7c) shows the performance improvement between a single-layer NHHORNN and a two-layer NHHORNN with the same number of units. One reason for the improved performance is that the lower resonant frequency of the second layer acts as a low-pass filter, suppressing high-frequency components in the activity pattern of the first layer. These high-frequency components are usually caused by irrelevant input signals and therefore influence rather than support classification. Therefore, if the task becomes more complex and requires more units, then a multi-layer NHHORNN can further improve the already significantly improved performance of a single-layer NHHORNN without increasing the number of units.

[0123] NHHORNNs are generally composed of multiple n units with n ≥ 8. These units can be connected in an all-to-all manner, but for a large number of n units, this all-to-all connectivity does not bring sufficient performance improvement compared to the exploding complexity of the connections. Therefore, if n exceeds 512, the all-to-all connectivity is preferably limited to groups or subgroups of na units, where na ≤ 512, preferably na ≤ 256, more preferably na ≤ 128, even more preferably na ≤ 64 or na ≤ 32 or na ≤ 16 or na = 9 or na = 4.

[0124] As discussed above for the 128+128 two-layer NHHORNN, starting from a certain number of units, building a two-layer or more network is more effective than increasing the number of units in a single-layer network. Alternatively or in combination, limiting the number of connections within the same network layer is also an effective approach. In addition, it may be preferable to provide more feedforward connections from the upstream layer to the downstream layer than to provide more feedback connections from the downstream layer to the upstream layer. In addition, setting the parameter ω 0i = the natural frequency of the undamped oscillation of the damped resonator of the downstream network layer unit, so that the highest natural frequency of the upstream network layer unit is higher than the highest natural frequency of the downstream network layer unit, and the lowest natural frequency of the upstream network layer unit is higher than the lowest natural frequency of the downstream network layer unit, it may be advantageous. Preferably, the natural frequency of the downstream network layer is set to 20% to 80% of the frequency of the upstream network layer, more preferably, 20% to 50%, such as 20% or 30% or 40% or 50%.

[0125] For example, in the first network layer L1, a first group of multiple n1 units are arranged, at least one of which is connected to the input unit, so this layer is the most upstream layer in terms of forwarding data and processing results; in the second network layer L2, a second group of multiple n2 units are arranged, at least one of which is connected to at least one unit of the first layer, so this layer is more downstream than the upstream (first) layer in terms of forwarding data and processing results. Of course, you can also choose to add a third, fourth, fifth, sixth, ....x layers (x≤1000), with n3, n4, n5, n6, .....nx units respectively.

[0126] In particular, if the data to be processed has potentially strong correlations between adjacent data points and potentially weak correlations between distant data points, such as in two-dimensional image data, a potentially effective setting for designing a two-layer network is to introduce many connections wi,j between smaller DHO groups in the first (upstream) network layer, such as full-to-full connectivity within the smaller groups, and fewer connections wi,j between these smaller DHO groups in the first (upstream) network layer, so that there are 25% to 75% potential connections between units in different smaller groups. The second (downstream) layer can have the same number of units as the first layer or a smaller number of units. The number of feedforward connections from units of the first (upstream) layer L1 to units of the second (downstream) layer L2 is preferably higher than the number / percentage of feedback connections from units of the second (downstream) layer L2 to units of the first (upstream) layer L1, such as 25% or 50% or 75% or 100% or 125% or 150% higher. The same is true if a third, more downstream layer is added.

[0127] Figure 8 and Fig. 9 An example of such a two-layer network is described.

[0128] exist Figure 8 In a), a number of n1=100 cells are shown arranged in a 10x10 matrix CB. The cells can be indexed by row k and column l, where k, l=1,2,…,5, i.e. k, l. The matrix CB is only used to explain the organization and connection of the cells and does not mean that the physical arrangement of the cells in the matrix CB is required for hardware implementation. The 5x5 chessboard pattern CBG1 in the figure divides the 100 cells into 25 first groups G1, each with 4 cells. Figure 8 b), the 4 cells in each of the 25 first groups G1 are marked alternately in rows and columns, and each second group is marked with a cross, which is done only to facilitate identification of the cells in the first group. The recursively connected unit includes all-to-all connections of all cells in each of the 25 first groups G1.

[0129] Figure 8c) A 5x5 checkerboard pattern CBG2 is displayed, which is shifted by 1 cell in both the row and column directions compared to CBG1, in other words, 50% of the cells in the first group are shifted in both directions. 25 second groups G2 of 4 cells each are created using this shift, as follows: The shift produces a new set of 16 second groups G2 of 4 cells each, where each of the 4 cells is part of a different first group G1. At the edges of the matrix CB, each "group" created by the shift of the checkerboard pattern has only two cells, while at the four corners of the matrix, each "group" created by the shift of the checkerboard pattern has only one cell. In order to sort these 100 cells into 25 second groups G2, each of which has 4 cells created by shifting the checkerboard pattern, the 8 second groups G2 at the edge of the matrix are "filled up" in the direction of the shift with the two remaining cells from the first group G1 on either side of the matrix diagonal / edge, which are not included in the 16 second groups G2 that have completed 4 cells. The second group G2, however, includes only one unit (10,10) at the corner located in the direction of matrix movement, and the other three corners of the matrix are filled with three units (1,1,1,10 10,1). Figure 8 d) shows the corresponding second grouping. The recursively connected unit comprises all-to-fully connected connections for all units in each of these 25 second groups G2.

[0130] This organization provides many connections wi,j between the cells / DHOs of the first group G1 in the first (upstream) network layer L1, for example in the described case, all-to-all connectivity within 25 first groups G1, and fewer connections wi,j between the cells / DHOs of the first group G1 in the first (upstream) network layer L1, i.e. 25% of the connections between the second group and each "neighboring" first group G1, the all-to-all connectivity in the second group G2 each including 4 (=25%) cells from four different first groups G1.

[0131] Of course, the organization of the first network layer L1 described above is not limited to 10x10=100 units, grouped into 5x5=25 groups. The number of units per group g1 can be any square of an integer that divides 2 ≤ 256, i.e. 4, 16, 36, 64, ..., 256, and the grouping can be any grouping in the (uxv) matrix, u, v = u, v = 2, 3, 4, 5, 6, ..., provided that the product (g1 uv) ≤ nmax, such as 102400. In this description, we only discuss square matrix organization (u = v), although the principle also applies to rectangular organization (u ≠ v). The restriction of g1 ≤ 256 is specifically chosen to limit the complexity of the connections in the full-to-full connectivity case. The above organization by "chessboard pattern movement" and the subsequent allocation to incomplete groups by diagonal allocation and 4 corner groupings can complete the second grouping of all these organizations and provide the expected connection diversity within the first (upstream) layer.

[0132] Fig. 9 The two-layer HORNN in Figure 8 The first (upstream) layer L1 and the second (downstream) layer L2, the second (downstream) layer L2 has a plurality of n2 units among n units, where 8≤n2. Fig. 9 In the example of , the second (downstream) layer L2 comprises the same number of n2=n1 cells as the first (upstream) layer, which is only a non-limiting example to explain the principle.

[0133] As with all multilayer RRNs in this teaching, each layer can generally be implemented as a HHORNN or NHHORNN. This means that DHO within a layer in a multilayer network can be implemented as an HHORNN layer with the same intrinsic parameters β and ω0, or with possibly different parameters β i and at least some different parameters ω 0i To implement the NHHORNN layer, this fact will not be mentioned every time for each example of a multi-layer RNN in this teaching.

[0134] The recursive connection unit RCU comprises an input / output node IO of each unit nci of the first group G1 of the first network layer L1 i At least one forward connection wi,j between at least one of the first group G1 of cells nci of the first network layer L1, in the example shown, the input / output node IO of each cell nci of each first group G1 i The input / output node IO of at least one unit of the unit ncj of the second network layer L2 j A forward connection wi,j between the first network layer and the corresponding unit nci is used to connect the damped harmonic oscillation h output by the input / output node of a unit in the first network layer i passed to the input / output nodes of the corresponding unit ncj in the second network layer, and

[0135] The input / output node IO of one of the cells nci of each second group G2 of the first network layer i and an input / output node IO of at least one of the units ncj of the second network layer L2 j At least one feed-forward connection wi,j between the first network layer and the corresponding unit nci is used to output the damped harmonic oscillation h i The input / output node IO of a cell of the corresponding cell ncj of the second network layer (in the example shown, this is natural because the input / output node IO of each cell nci of each first group G1 of the first network layer L1 i are all connected, and therefore inevitably also connected to a cell of the corresponding second group G2), and

[0136] From the input / output node IO of the corresponding number of units ncj of the second network layer L2 j To the input / output node IO of one unit of each unit nci of the first group G1 of the first network layer i at least one feedback connection wj,i of the second network layer L2 for transmitting the resulting damped harmonic oscillation h outputted from the input / output node of the corresponding unit ncj of the second network layer L2 i .

[0137] Setting parameters ω 0i = the natural frequency of the undamped oscillation of the damped harmonic oscillator of the second network layer n2 units, so that the highest natural frequency of the first network layer n1 units is higher than the highest natural frequency of the second network layer n2 units, and optionally, the lowest natural frequency of the first network layer n1 units is higher than the lowest natural frequency of the second network layer n2 units. As a result of this arrangement, the second network layer acts as a low-pass filter for the signal / oscillation transmitted from the first network layer, suppressing potential irrelevant high-frequency components, thereby improving the performance of the corresponding HHORNN / NHHORNN.

[0138] The arrangement of the subgroups of cells with full-to-full connectivity in each layer is also not limited to the above-mentioned chessboard pattern arrangement. For example, the well-known king diagram arrangement derived from the possible moves of the king in the game of chess can be used. This arrangement is used as a third embodiment in Fig.10, where the rows and columns of cells are arranged in an assumed (or actual) matrix, and one cell, cell 2,2 (=king) in the example, is connected to all eight "neighboring cells", namely a cell in the same column of the previous row (cell 1,2) and its two direct neighboring cells in the previous row (cells 1,1 and 1,3), a cell in the same column of the next row (cell 3,2) and its two direct neighboring cells in the next row (cells 3,1 and 3,3), and two direct neighboring cells in the same row (cells 2,1 and 2,3), as shown in Figure 8 a). Figure 8 b) shows the king graph arrangement of cell 3,3 as the "king", which has all-to-all connectivity with its eight neighboring cells 2,2, 2,3, 2,4, 3,2, 3,4, 4,2, 4,3, and 4,4.

[0139] Figures 8 to 10 Networks of the type shown and other improved networks can be implemented with a hierarchical structure of 2, 3, 4, 5, 6, 7, 8, ..., x layers (x ≤ 1000). If there are more than two layers, skip-layer connections / connections can be made. It is particularly useful that skip-layer feedback connections from more downstream layers to more upstream layers can skip one or more intermediate upstream layers, for example,

[0140] In a 4-layer network, a skip-feedback connection from the 4th (most downstream) layer to layer 2 or layer 1 can skip layer 3 or layers 3 and 2. This skip-feedback connection helps to re-feed back intermediate recognition "results", thus establishing a structure in which intermediate recognition results trigger activation of specific regions in the network. This skip-feedback enhances the possibility of resonance, entrainment, synchronization, phase shift, and desynchronization. Fig.11 a) shows a schematic diagram of a network with skip-layer feedback connections, where there are three layers L1, L2, and L3. The layers can be the same or different types of layers (HHORNN or NHHORNN), and / or have the same or different connectivity within different layers. For example, the most upstream layer L1 can be implemented as an HHORNN with Fig.10 The second layer L2 can be implemented as HHORNN with all-to-all connectivity in the king graph arrangement. Figure 8 The full-to-full connectivity in the chessboard arrangement of , and the most downstream (third layer) L3 can be implemented as NHHORNN with Fig.10 All-to-all connectivity in the King graph arrangement of , in separate groups of 100 units each within the third layer L3. There is feed-forward connectivity (wi,j,L1-L2 and wi,j L2-L3) and feedback connectivity (wj,i L3-L2 and wj,i L2-L1) between adjacent layers, for example, Fig.11As shown in a), the connectivity between the selected unit area in the first layer L1 and the selected unit area in the second layer L2 and between the selected unit area in the second layer L2 and the selected unit area in the third layer L3 is enhanced, and as shown in Fig.11 As shown in a), the connection wj,i L3-L1 between the third layer L3 and the first layer L1 has a skip-layer feedback connection.

[0141] By limiting the full-to-full connectivity of the unit subgroups as described above, the number of units per layer can be greatly increased without exploding the complexity of the software or hardware connections. If the complexity of the task requires expansion, networks with 100,000 units or more at the top layer are possible, and 10,000 units per layer is no problem. Fig.11 b), schematically shows Fig.11 a) A three-layer network, where the first layer L1 has 4000 units, the second layer L2 has 2000 units, and the third layer L3 has 1000 units.

[0142] If the information to be processed is of geometric nature, such as image data, or more generally expresses spatially invariant statistical information, then using HHORNN as the first layer L1 on the input side is advantageous. The reason is that the first layer L1 of the network can process the input in a spatially invariant manner.

[0143] The following explains how to implement NHHORNN in semiconductor hardware to set the connection values ​​and potential time delays after training the network using the BPTT technique.

[0144] Assume that in conventional CMOS technology, n is 1296 oscillators DHOi arranged in a 36 x 36 matrix array on a semiconductor chip. Each oscillator DHOi is implemented as a ring oscillator in CMOS. Figure 8 As described above, the oscillators DHOi are all-to-all connected in a king graph arrangement. The connection weights of the connections are set by a multi-stage transmission gate coupling block that provides a connection wi,j between the oscillators DHOi and DHOj. This multi-stage transmission gate coupling block can be as shown in PA2 Figure 2 If the coupling weights should be reprogrammable, the coupling values ​​of this multi-stage transmission gate coupling block are set by values ​​stored in SRAM memory, otherwise by ROM. Figure 2 and Figure 3 A corresponding example arrangement of an SRAM is shown. Of course, the reference to PA2 is only a non-limiting example, and a memory block that controls the amplification of a CMOS-implemented amplifier to successively set connection values ​​may be used, wherein a potential inversion of the sign of the connection value is achieved by a single inverter that switches to connection wi,j or not.

[0145] The connection delay can be achieved by selecting / setting the number of grading inverters, which are also controlled by the corresponding SRAM or ROM memory blocks. If such delays are provided by grading inverters, they can also be used to set the potential weight inversion.

[0146] Obviously, in the above case, the nonlinear saturation transfer function input component ICS i It can be implemented as an input element in CMOS as shown in US2018 / 0307977 A1, where a CMOS-implemented adder ICA i Connect to each input element ICS i .

[0147] Such an implementation in CMOS ASIC will provide extremely low power consumption for trained HHORNN / NHHORNN.

[0148] The training of such a CMOS NHHORNN is implemented using the well-known BPTT, and the values ​​of the connected cells can be obtained in this way and then stored in a memory block as rewritable values ​​when using a rewritable memory such as SRAM, or if the final NHHORNN product is not reprogrammable, then stored in a ROM (or by blowing fuses, etc.). The above description of the HHORNN and NHHORNN using equations (9) or (10) provides a tool to implement the DHO, the connections between them and the inputs and outputs, in other words the HHORNN / NHHORNN, in software and run them on a correspondingly programmed data processing device (such as a general-purpose computer). This means that the method for calculating the oscillations performed by the DHO is implemented in the following manner: (i) using equation (9) or (10) to create a description of the corresponding RNN to define the output data O(t) in response to the input data S(t), (ii) inputting the input data S(t) representing the data to be processed into the description, thereby calculating the oscillation caused by the input data (S(t)), and then calculating the output data O(t) based on the description and the calculated oscillation, and (iii) outputting the processing result based on the output data generated by the input data. If no training is performed, equation (9) or (10) captures the initial settings of the RNN. In this case, the description can be used for training. If the training result is used to create a description of the RNN, then the created description represents the trained RNN, thereby representing a data processing tool that provides a processing result for identifying the object / event represented by the corresponding input data S(t). Therefore, when running on a correspondingly programmed computer, a computer program including program code implementing the above method is a possible implementation of the HHORNN / NHHORNN of the present teaching.

[0149] The output data will typically provide the result of processing before all the input data S(t) is fed in. For example, a 16-unit NHHORNN with all-to-all connectivity including self-connectivity trained for MNIST data can always determine the digit to recognize when less than 50% of the total input data S(t) is fed in. Fig.12 All 10 digits are shown, with a line representing the input data that has been fed in from the top of the image as the RNN determines the outcome.

[0150] HHORNN and NHHORNN are also highly resilient to the noise introduced in the input data S(t), outperforming the state-of-the-art RNNs in this respect as well.

[0151] The recursive neural network and data processing method and apparatus of the present invention can be used for a variety of data processing tasks that are challenging for the prior art.

[0152] The first example is image segmentation and classification, such as detecting objects in a digital image, or segmenting an image into a set of objects contained in the image. In the prior art, this task is usually implemented by a feedforward neural network, i.e. a network with an input layer, intermediate layers and an output layer (deep neural network). For example, a multilayer perceptron, a convolutional neural network or a transformer network. Using the intermediate layers, the network can transform the input pixel values ​​into a representation suitable for solving classification / detection / segmentation tasks, such as detecting traffic signs in self-driving (automatic) cars. With the recursive neural network / data processing device of the present teaching, a completely new and completely different approach becomes possible. HORNN (both types NHHORNN and HHORNN) are recursive in nature and receive input in the form of a time series. The network is implemented by several layers: the first layer is a geometrically organized HHORNN, followed by a layer of HHORNN or NHHORNN, which can be geometrically organized or all-to-all, or a combination of both. The geometric organization refers to the connection structure (connectivity) within the layer, with a limited number of connections to other units, which can be regarded as next neighbors, and therefore a geometric pattern, such as the king graph or chessboard pattern mentioned above or other corresponding next neighbor connection patterns. To present an image to the network, a matrix of nxm pixel values ​​(each with an intensity value between v_min and v_max, possibly for different color channels R, G, B) is converted into a time series of length N, by scanning the value range [v_min, v_max] from top to bottom (or bottom to top) in discrete steps v_1=v_max,...,v_N=v_min. This produces a time series of length N, where each time point is represented by an nxm matrix Mi (1≤i≤N), whose entries are set to 1 when the corresponding pixel value exceeds a threshold v_i, and to 0 elsewhere. In other words, the topography of nxm pixel values ​​is sampled into N nxm matrices. The units of the first layer of the network are stimulated according to the values ​​in the time series. For a certain processing time, the network processes the input data. Multiple layers can form a more abstract representation of the data, which can be geometrically organized (respecting the geometric organization of the input data or all-vs-all). After a period of time, the state of the network (the answer) is read out. The network can develop internal measures, such as by measuring the entropy of its dynamic state, to "know" when it has found a good answer for a given input.

[0153] We note that the type of processing described in the first example is not limited to image data with inherent spatial relationships, but can be applied in the same way to other spatially organized data (such as sensor data obtained from spatially distributed sensors).

[0154] A second example is keyword spotting, which refers to identifying keywords in speech, such as detecting the presence of a wake-up word ("Hey, Siri," "OK, Google") in a continuously recorded audio stream, with the goal of getting a smart device (smartwatch, smartphone, smart speaker) into mode to accept commands.

[0155] In the prior art, this task is usually achieved by re-encoding, digitizing, and segmenting the microphone data (sound pressure level), and extracting MFCC (see https: / / de.wikipedia.org / wiki / Mel_Frequency_ Cepstral_Coefficients ) is executed to perform the mapping to the feature vector, and then the feature vector sequence is fed to the detection / classifier unit (such as a recurrent neural network) to detect whether the keyword is detected in the recorded audio. The program example is shown at: https: / / github.com / MycroftAI / mycroft-precise#how-it-works . Software implementation see: https: / / github.com / Picovoice / porcupine,https: / / github.com / MycroftAI / mycroft-precise Utilizing the recurrent neural network / data processing apparatus of the present teachings, a completely new and radically different approach can be taken, namely feeding the raw microphone data (sound pressure level) directly into the corresponding trained NHHORN.

[0156] The third example is speech recognition. In the prior art, this task is usually implemented similarly to keyword recognition, i.e., the microphone data (sound pressure level) is re-encoded, digitized, and segmented, and MFCC is extracted (see https: / / de.wikipedia.org / wiki / Mel_Frequency_Cepstral_Coefficients ) is executed, a mapping to a feature vector is performed, and then the sequence of feature vectors is fed to a detection / classifier unit (such as a recurrent neural network, SVM, etc.) to detect the word sequence. Using the information of the language model, a possible word sequence (such as a sentence) is output. With the recurrent neural network / data processing device taught by this invention, a completely new and completely different approach becomes possible. The raw microphone data (sound pressure level) can be fed into the (appropriately trained) multi-layer NHHORN without any preprocessing. The properties of the NHHORNN provide a potential tension organization for the first layer, that is, units encoding similar frequencies are more likely to be connected during training, and therefore are connected during training, resulting in a "geometric" organization of the frequency space. Therefore, over the course of several layers, the network is able to form an abstract representation of the input data, for example, it can transcribe syllables, words and sentences.

[0157] A fourth example is anomaly detection, which is the identification of rare events in time series data that deviate from expected or normal behavior (according to most data definitions). In state-of-the-art systems, there are different approaches to this task, ranging from simple statistical tests to neural network-based methods. https: / / en.wikipedia.org / wiki / Anomaly_detection. With the recurrent neural network / data processing device of this teaching, a completely new and completely different approach becomes possible. Now time series data can be directly fed value by value into a (correspondingly trained) HHORNN / NHHORNN, which may be multi-layered. Simple tasks such as EKG, EEG, MEG or other time series data from organisms, machines or processes. When using HHORNN / NHHORNN, there is no need to process the data by chunking, etc.

[0158] A fifth example is closed-loop or feedback control. A closed-loop or feedback control system is a system designed to maintain a prescribed relationship between two sets of system variables by comparing functions of the two sets of system variables and using the difference as a means of control. https: / / en.wikipedia.org / wiki / Control_theory ). Application examples include controlling chemical reactions (by controlling valves) or directional control of moving objects, etc. With the recurrent neural network / data processing device of the present teachings, a completely new and radically different approach becomes possible. Time series data representing the measured variables can now be directly input into a (correspondingly trained) latent multi-layer HHORNN / NHHORNN that is trained to predict the dependent variable.

[0159] The sixth example is time series forecasting and predictive analytics. Given the historical values ​​of a time series (such as sensor data), the task is to predict the development of future data ( https: / / en.wikipedia.org / wiki / Predictive_ analytics). With the recurrent neural network / data processing device of the present teaching, a completely new and completely different approach becomes possible. Now the time series data can be directly fed value by value into the (correspondingly trained) HHORNN / NHHORNN, which may be multi-layered.

[0160] The above teachings can be implemented as described in the following aspects of the present invention:

[0161] Aspect 1 A recurrent neural network is used to process time series input data (S(t)), comprising

[0162] A plurality of n damped harmonic oscillators (DHOi, i=1 to n), n≥8, wherein the oscillation of each oscillator follows The general second-order differential equation for a damped harmonic oscillator is i = Damping coefficient, ω 0i = natural frequency of undamped oscillation, such as β i =c i / 2m i And ω 0i =k i / m i , m i = mass, k i = spring constant, ci = the viscous damping constant of the corresponding damped translational mechanical harmonic oscillator DHOi, or β i =R i / L i ,ω 0i 2 =1 / L i C i , R i = Resistance, L i = Inductance, C i = the capacity of the corresponding damped electrical RLC resonator DHOi, and similarly for other types of damped harmonic oscillators, each of the n damped resonators being a unit of the neural network (nci),

[0163] Each unit (nci) has an input and output node (IO i ), which is used to receive the unit input (x i ), such as the translation of a corresponding damped translational mechanical harmonic vibrator or the electrical input of a corresponding damped electrical RLC harmonic oscillator, and outputting the resulting damped harmonic oscillation (h i ),

[0164] Input / output nodes (IO i )include

[0165] Input Connections (IC i ), adapted to receive any input of the unit, and outputs the same value as the unit input (x) through a nonlinear transfer function, preferably a saturated Sigmoid-like function or a hyperbolic tangent-like function with an optional offset. i ) same input, such as through the viscoelastic input element for the corresponding damped translational mechanical harmonic oscillator or through the sigmoid curve input implemented by the transistor (ICS i ) for the corresponding damped electrical RLC harmonic oscillator DHOi, and

[0166] Output Connection (OC i ), suitable for damping the resonance generated by the output (h i ),as well as

[0167] Recursively connected units (RCU), including, for each unit (c i ), at least one connection (wi,j) is connected to the input / output node (IO) of the corresponding unit (nci) i ) and at least one other cell (ncj) input / output node (IO j ) between the input / output nodes (IO i ) output damped harmonic oscillation (h i) is transferred to another unit (c j ) of the input and output nodes (IO j ).

[0168] 2. A neural network according to claim 1, wherein

[0169] The input / output node (IO i ) and the input / output node (IO) of another unit (ncj) of the recursive connection unit (RCU) j ) is set to have a transmission characteristic T i,j =w i,j xh i , w i,j ∈R and |w i,j |≤10, wherein the connection (wi,j) may be such as for transmitting the output (h i ) is a viscous element or is used to transmit the output of the corresponding damped electrical RLC harmonic oscillator (h i ) of an electrical voltage divider or transmission gate coupler.

[0170] 3. A neural network according to claim 1 or 2, wherein

[0171] The input / output node (IO i ) and the input / output node (IO) of another unit (ncj) of the recursive connection unit (RCU) j ) is set to have a transmission characteristic T i,j , and delay the transmission by an output time delay k i,j =0,1,…,k,0≤k≤kmax,ti1 / 10≤Δt≤ti1 and k i,j xΔt≤50ti1, where ti1 is the time series interval of the time series input data (S(t)), representing the time interval between subsequent discrete values ​​of the discrete time series input data (S(t)) or the time interval between subsequent sampled values ​​of the continuous time series input data (S(t)), where the output time delay can be, for example, the output of the corresponding damped translation mechanical harmonic vibrator (h i ) or an elastic element for transmitting the output of a corresponding damped electrical RLC harmonic oscillator (h i )'s inductor or clock output gate.

[0172] 4. A neural network according to any preceding aspect, wherein

[0173] A recursive connection unit (RCU) comprises at least one unit (c i ) of the corresponding damped harmonic oscillator DHOi input / output node IO i The connections with itself, for input and output, provide self-connectivity, preferably the recursively connected unit (RCU) comprises connections with 5% self-connectivity of the unit, more preferably 10%, more preferably 25%, more preferably 50%, more preferably 75%, more preferably 100%.

[0174] 5. A neural network according to any preceding aspect, wherein

[0175] A plurality of n1 units among the n units are arranged in a (first) network layer (L1), 8≤n1 and n1≤n, and a recursive connection unit (RCU) comprises, for n1 units (c i ) has at least one connection (wi,j) to an input / output node (IO) of a corresponding unit (nci) in a (first) network layer (L1). i ) and an input / output node (IO) of at least another unit (ncj) of a (first) network layer (L1) j ) between the input / output nodes (IO i ) produces a damped harmonic oscillation (h i ) to another unit (c j ) of the input / output nodes (IO j ).

[0176] 6. A neural network according to any preceding aspect, wherein

[0177] A plurality of n2 units of the n units are arranged in a second (downstream) network layer (L2), 8≤n2 and preferably n2≤n1, a recursive connection unit (RCU) comprising, for the n2 units (c i ) in each unit, at least one connection (wi,j) in the input / output node (IO) of the corresponding unit (nci) in the second network layer (L2) i ) and an input / output node (IO) of at least another unit (ncj) of a second network layer (L2) j ) between the input / output nodes (IO i ) produces a damped harmonic oscillation (h i ) to another unit (c j ) of the input / output nodes (IO j ); and the recursive connection unit (RCU) comprises a feed-forward connection (wi,j) between the input / output nodes (IO) of nl units (nci) in a (first) network layer i) and the input / output nodes (IO j ), for transmitting the damped harmonic oscillation (hi) generated and outputted from the input / output node of the corresponding unit of the n1 units (nci) of a (first) network layer to the input / output node (IO) of the corresponding unit of the n2 units (ncj) of the second network layer j ) to establish a minimum potential feed-forward connectivity of 10% and a maximum potential feed-forward connectivity of 100%; and

[0178] Feedback connection (wj,i) is the input / output node (IO) of n2 units (ncj) in the second network layer. j ) and an input / output node (IO) of a (first) network layer nl unit (nci) i ) for generating a damped harmonic oscillation (h j ) is transmitted to the input / output node (IO) of the corresponding unit of a (first) network layer nl unit (nci) i ) to establish a minimum potential feedback connectivity of 10% and a maximum feedback connectivity of 100%; and

[0179] Optionally, a plurality of nr units among the n units are arranged in the rth network layer (Lr), wherein r=3 or 4 or 5 or 6, and 8≤nr, preferably nr≤n(r-1), and the recursive connection unit (RCU) comprises, for the nr units (c i ) has at least one connection (wi,j) between the input / output nodes (IO) of the corresponding unit (nci) in the rth network layer (Lr). i ) and an input / output node (IO j ) between the input / output nodes (IO i ) produces a damped resonance at the output (h i ) to another unit (c j ) of the input / output nodes (IO j ); The recursive connection unit (RCU) includes the feed-forward connection (wi,j) at the input / output node (IO) of the n(r-1)th unit (nci) of the n(r-1)th network layer. i ) and the input / output node (IO j ), for generating an output damped resonance (h) from the input / output node of the corresponding unit (nci) of the n(r-1)th unit (nci) of the n(r-1)th network layer i) is transmitted to the input / output nodes of the corresponding units of the nr units (ncj) of the nr-th network layer to establish a minimum potential feedforward connectivity of 10% and a maximum potential feedforward connectivity of 100%; and the feedback connection (wj,i) is transmitted to the input / output nodes (IO j ) and the input / output node (IO) of the n(r-1)th unit (nci) of the n(r-1)th network layer i ) for generating an output damped resonance (h j ) is transmitted to the input / output node (IO) of the corresponding unit (nci) of the n(r-1)th network layer i ) to establish a minimum potential feedback connectivity of 10% and a maximum potential feedback connectivity of 100%;

[0180] 7. A neural network according to any preceding aspect, wherein

[0181] The recursive connection unit (RCU) optionally includes, for a plurality of nr units arranged in the same r-th network layer, where r=1, 2, ..., 6, 8≤nr, nr≤n, potential connections (wi,j) for full connectivity of each unit with at least 8 and at most 512 other units (ncj) of the nr units (nci) of the same r-th network layer to transmit a damped harmonic oscillation (h) output from an input / output node of the corresponding unit (nci) i ) to the input / output nodes of the corresponding other unit in the n1 units, or full connectivity in the king graph layout, or when the units are arranged in the ((g1 / 2u)x(g1 / 2v)) matrix CB, g1=4,16,36,64,100,144,256 and u,v=2,3,4,5,6,… and (g1 uv)≤102400, full connectivity for a first group (G1) of g1 cells, wherein these first groups (G1) are arranged in a first chessboard in the (uxv) matrix CBG1, full connectivity for a second group (G2) of g1 cells, wherein these second groups (G2) are arranged in a second chessboard in the (uxv) matrix CBG2, CBG2 being shifted by g1 / 2 cells in both row and column directions relative to the first chessboard arrangement CBG1, and the second groups (G2) at the edges of the matrix CB in the shifted direction are completed by cells at diagonally opposite positions at the edges of the matrix CB that are not covered by the second (shifted) chessboard arrangement CBG2, and the second groups (G2) at the corners in the shifted direction are completed by cells in the three corners of the matrix CB that are not covered by the second (shifted) chessboard arrangement CBG2.

[0182] 8. A neural network according to claim 6 or 7, wherein

[0183] The recursive connection unit (RCU) comprises at least one unit (c i ), to the unit (c i ) of at least one unit, where s=2 or 3 or 4 or 5, preferably at least 10%, more preferably at least 20% of the potential skip-layer feedback connectivity.

[0184] 9. A neural network according to any preceding aspect, wherein

[0185] The plurality of n damped harmonic oscillators (DHOi, i=1 to n) comprises at least two different types of damped harmonic oscillators, wherein each of the at least two different types of damped harmonic oscillators has at least one damped harmonic oscillator having a certain characteristic parameter ω. 0i =The natural frequency of the undamped oscillation is different from that of other types of damped harmonic oscillators.

[0186] 10. A neural network according to any one of aspects 6 to 9, wherein

[0187] Setting parameters ω 0i = the undamped oscillation natural frequency of the damped harmonic oscillator of the nr units of the nr-th network layer, r = 2, 3, ..., 6, so that the highest natural frequency of the n(r-1)-th unit of the n(r-1)-th network layer is higher than the maximum natural frequency of the nr units of the nr-th network layer, and the lowest natural frequency of the n(r-1)-th unit of the n(r-1)-th network layer is higher than the minimum natural frequency of the nr units of the nr-th network layer.

[0188] 11. A neural network according to any preceding aspect, wherein

[0189] Provide at least three different types of damped harmonic oscillators and set the parameters ω 0i = the natural frequencies of undamped oscillations of at least three different types of damped harmonic oscillators, based on the determination of the power spectral densities of multiple samples of the time series input data (S(t)) to be processed and the variance of these power spectral densities, so that the natural frequencies are distributed on the peak values ​​of the variance, one of the at least three natural frequencies is set to correspond to the variance peak value with the lowest frequency, and one of the at least three natural frequencies is set to correspond to the variance peak value with the largest amplitude, and the remaining natural frequencies are preferably set to frequencies that are basically evenly distributed between the frequencies of the two variance peaks, or are set to a normal distribution with a selected mean and variance.

[0190] 12. A neural network according to any of the preceding aspects, further comprising

[0191] Input unit (IU), an input connection (IC) connected to the input / output node of at least one unit (nci) i ), adapted to receive and input time series input data (S(t)) of length Tl1 to input the input data (S(t)) to an input connection (IC) of an input / output node of at least one unit (ci) i ), and an output unit (OU) connected to an output connection (OC) of an input / output node of at least one unit (nci) i ) is adapted to output output data (O(t)) at an output start time (OST) set to a predetermined time interval after receiving an input start of input data (S(t)).

[0192] 13. A neural network according to any preceding aspect, wherein

[0193] The input / output nodes of each unit (nci) are suitable for discrete time steps t d =t0+dτ (τ=discrete time value) Output damped harmonic oscillation (h i ), ti1 / 20≤τ≤ti1 / 2, 2≤d≤20, where ti1 is the time series interval of the time series input data (S(t)), which represents the time interval between subsequent discrete values ​​of the discrete time series input data (S(t)) or the time interval between subsequent sampling values ​​of the continuous time series input data (S(t)).

[0194] It is expressly pointed out that all features disclosed in the specification and / or aspects and / or claims are intended to be separate and independent from each other, for the purpose of original disclosure and for the purpose of limiting the claimed invention, regardless of the combination of features in the embodiments and / or claims. It is expressly pointed out that all value ranges or indications of groups of entities disclose every possible intermediate value or intermediate entity, both for the purpose of original disclosure and for the purpose of limiting the claimed invention, in particular as limits of value ranges.

Claims

1. A neural network device, comprising a neural network for processing time series input data (S(t)), comprising A plurality of n damped harmonic oscillators (DHOi, i=1 to n), n≥8, wherein the oscillation of each oscillator follows The general second-order differential equation for a damped harmonic oscillator is i =R i / L i ,ω 0i 2 =1 / L i C i , R i = Resistance, L i = Inductance, C i = the capacitance of the corresponding damped electrical RLC resonator DHOi, and β corresponding to other types of damped electrical harmonic oscillators i and ω 0i 2 , each of the n damped resonators is a unit of the neural network (nci), Each unit (nci) has an input and output node (IO i ), which is used to receive the unit input (x i ), such as the electrical input of the corresponding damped electrical harmonic oscillator, and outputting the generated damped harmonic oscillation (h i ), Input / output nodes (IO i )include Input Connections (IC i ), adapted to receive any input of the unit, and outputting the same value as the unit input (x) through a saturated nonlinear transfer function, preferably a sigmoid-like function with an optional offset or more preferably a hyperbolic tangent-like function. i ) the same input, such as a sigmoid or hyperbolic tangent input implemented by a transistor (ICS i ) outputs the same input for the corresponding damped electrical harmonic oscillator DHOi, and Output Connection (OC i ), suitable for damping the resonance generated by the output (h i ),as well as Recursively connected units (RCU), including, for each unit (c i ), at least one connection (wi,j) is connected to the input / output node (IO) of the corresponding unit (nci) i ) and at least one other cell (ncj) input / output node (IO j ) between the input / output nodes (IO i ) output damped harmonic oscillation (h i ) is transferred to another unit (c j ) of the input and output nodes (IO j ), in, i) one or more of the plurality of n damped harmonic oscillators (DHOi, i=1 to n) are respective damped electrical RLC harmonic oscillators DHOi, implemented as voltage controlled oscillators, as CMOS cross-coupled differential oscillators or CMOS ring oscillators, each of the n damped harmonic oscillators being a unit (nci) of a neural network, and / or ii) One or more of the plurality of n damped harmonic oscillators (DHOi, i=1 to n) are respective damped electrical RLC harmonic oscillators DHOi, implemented as voltage controlled oscillators using analog components including an integrator and a potentiometer, each of the n damped harmonic oscillators being a unit (nci) of a neural network.

2. The neural network device according to claim 1, wherein Each unit (nci) has input / output nodes (IO i ) for implementing a CMOS inverter stage circuit (ICS i ) is a Sigmoid transfer function that takes the unit input (x i ) as the electrical input to a correspondingly damped electrical harmonic oscillator.

3. The neural network device according to claim 1 or 2, wherein Each unit (nci) has input / output nodes (IO i ), used to pass the adder (ICA i ) receives all inputs to the cell, adds all inputs to the cell, and outputs the added input as input to a CMOS inverter stage circuit (ICS) i ) such as saturated nonlinear transfer functions like Sigmoid or hyperbolic tangent, the unit input (x i ) as the electrical input output of the corresponding damped electrical RLC harmonic oscillator.

4. The neural network device according to any one of claims 1 to 3, wherein The input / output node (IO i ) and the input / output node (IO) of another unit (ncj) of the recursive connection unit (RCU) j ) between which at least one connection (wi,j) is set to have a transmission characteristic T i,j =w i,j xh i , w i,j ∈R and |w i,j |≤10, where The connection (wi,j) may be used to transmit the output (h i ) of an electrical voltage divider or transmission gate coupler.

5. The neural network device according to any one of claims 1 to 4, wherein The input / output node (IO i ) and the input / output node (IO) of another unit (ncj) of the recursive connection unit (RCU) j ) is set to have a transmission characteristic T i,j , and delay the transmission by an output time delay xΔt,k i,j =0,1,…,k,0≤k≤kmax,ti1 / 10≤Δt≤ti1 and k i,j xΔt≤50ti1, where ti1 is the time series interval of the time series input data (S(t)), which represents the time interval between subsequent discrete values ​​of the discrete time series input data (S(t)) or the time interval between subsequent sample values ​​of the continuous time series input data (S(t)), wherein the output time delay can be, for example, an inductor or a clock output gate for transmitting the output of the corresponding damped electrical RLC harmonic oscillator (h i ).

6. The neural network device according to any one of claims 1 to 5, wherein Each connection (wi,j) of the recursive connection unit (RCU) is implemented as a (2p+1)-stage transmission gate, coupled to a first chain of at least 2 serially connected NOT gates to receive an output of an output input / output node, and a second chain of at least 2 serially connected NOT gates coupled to another input input / output node, and 4p transmission gate blocks are connected between the output terminals of the NOT gates of the first chain and the input terminals of the NOT gates of the second chain, p≤10, so that if no transmission gates are connected to the input and output of the NOT gates, the transmission characteristic T i,j =w i,j xh i The transmission weight w i,j = 0, if the 2w transmission gates are correctly connected in phase between the output of the NOT gate of the first chain and the input of the NOT gate of the second chain, then w i,j = +w, if 2w transmission gates are connected with a phase shift between the output of the NOT gate of the first chain and the input of the NOT gate of the second chain, then w i,j =-w, Optionally with transmission output time delay and k i,j xΔt≤50ti1, where The output time delay is achieved by serially connecting l clock NOT gates between the output terminal of the output input / output node and another input / output node after or before the NOT gate of the first chain and the second chain, wherein the number of clock gates l is determined as l=(k i,j xΔt) / clock period, and optionally having a weight setting block including a setting device for each (2p+1) stage transmission gate coupling, the setting device being adapted to close or interrupt the connection via each corresponding 4p transmission gate, either by permanent closing or interruption such as by melting or etching the corresponding connection wire, or by programmable closing or interruption of a CMOS transistor switched according to data stored in an SRAM or ROM memory block storing the setting weights, and Optionally have a delay setting block including a setting device for setting the time delay of each transmission output The setting device is adapted to close or interrupt the connection by means of a corresponding number of l clocked NOT gates, either by permanent closing or interruption by, for example, melting or etching the corresponding connection wires, or by programmable closing or interruption, by means of CMOS transistors switched according to data stored in an SRAM or ROM memory block, storing the set transmission output time delay 7. A neural network device according to any preceding claim, wherein A recursive connection unit (RCU) comprises at least one unit (c i ) of the corresponding damped harmonic oscillator DHOi input / output node IO i A connection to itself, for input and output, providing self-connectivity, Preferably, the recursively connected unit (RCU) comprises 5% of the connections of the unit being self-connected, more preferably 10%, more preferably 25%, more preferably 50%, more preferably 75%, more preferably 100%.

8. A neural network device according to any preceding claim, wherein A plurality of n1 units among the n units are arranged in a (first) network layer (L1), 8≤n1 and n1≤n, and a recursive connection unit (RCU) comprises, for n1 units (c i ) has at least one connection (wi,j) to an input / output node (IO) of a corresponding unit (nci) in a (first) network layer (L1). i ) and an input / output node (IO) of at least another unit (ncj) of a (first) network layer (L1) j ) between the input / output nodes (IO i ) produces a damped harmonic oscillation (h i ) to another unit (c j ) of the input / output nodes (IO j ).

9. A neural network device according to any preceding claim, wherein A plurality of n2 units of the n units are arranged in a second (downstream) network layer (L2), 8≤n2 and preferably n2≤n1, a recursive connection unit (RCU) comprising, for the n2 units (c i ) in each unit, at least one connection (wi,j) in the input / output node (IO) of the corresponding unit (nci) in the second network layer (L2) i ) and an input / output node (IO) of at least another unit (ncj) of a second network layer (L2) j ) between the input / output nodes (IO i ) produces a damped harmonic oscillation (h i ) to another unit (c j ) of the input / output nodes (IO j ); and the recursive connection unit (RCU) includes The feed-forward connections (wi,j) are connected between the input / output nodes (IO i ) and the input / output nodes (IO j ), for transmitting the damped harmonic oscillation (hi) generated and outputted from the input / output node of the corresponding unit of the n1 units (nci) of a (first) network layer to the input / output node (IO) of the corresponding unit of the n2 units (ncj) of the second network layer j ) to establish a minimum potential feed-forward connectivity of 10% and a maximum potential feed-forward connectivity of 100%; as well as Feedback connection (wj,i) is the input / output node (IO) of n2 units (ncj) in the second network layer. j ) and an input / output node (IO) of a (first) network layer nl unit (nci) i ) for generating a damped harmonic oscillation (h j ) is transmitted to the input / output node (IO) of the corresponding unit of a (first) network layer nl unit (nci) i ) to establish a minimum potential feedback connectivity of 10% and a maximum feedback connectivity of 100%; as well as Optionally, a plurality of nr units among the n units are arranged in the rth network layer (Lr), wherein r=3 or 4 or 5 or 6, and 8≤nr, preferably nr≤n(r-1), and the recursive connection unit (RCU) comprises, for the nr units (c i ) has at least one connection (wi,j) between the input / output nodes (IO) of the corresponding unit (nci) in the rth network layer (Lr). i ) and an input / output node (IO j ) between the input / output nodes (IO i ) produces a damped resonance at the output (h i ) to another unit (c j ) of the input / output nodes (IO j ); the recursive connection unit (RCU) includes The feed-forward connection (wi,j) is connected to the input / output node (IO) of the n(r-1)th unit (nci) of the n(r-1)th network layer. i ) and the input / output node (IO j ), for generating an output damped resonance (h) from the input / output node of the corresponding unit (nci) of the n(r-1)th unit (nci) of the n(r-1)th network layer i ) is transmitted to the input / output nodes of the corresponding units of the nr units (ncj) of the nr-th network layer to establish a minimum potential feedforward connectivity of 10% and a maximum potential feedforward connectivity of 100%; and the feedback connection (wj,i) at the input / output node (IO j ) and the input / output node (IO) of the n(r-1)th unit (nci) of the n(r-1)th network layer i ) for generating an output damped resonance (h j ) is transmitted to the input / output node (IO) of the corresponding unit (nci) of the n(r-1)th network layer i ) to establish a minimum potential feedback connectivity of 10% and a maximum potential feedback connectivity of 100%.

10. A neural network device according to any preceding claim, wherein The recursive connection unit (RCU) optionally includes, for a plurality of nr units arranged in the same r-th network layer, where r=1, 2, ..., 6, 8≤nr, nr≤n, potential connections (wi,j) for full connectivity of each unit with at least 8 and at most 512 other units (ncj) of the nr units (nci) of the same r-th network layer to transmit a damped harmonic oscillation (h) output from an input / output node of the corresponding unit (nci) i ) to the input / output nodes of the corresponding other cell among the n1 cells, or full connectivity in the king graph layout, or when the cells are arranged in the ((g1 / 2u)x(g1 / 2v)) matrix CB, g1=4,16,36,64,100,144,256 and u,v=2,3,4,5,6,… and (g1 uv)≤102400, for the first group (G1) of g1 cells, full connectivity, where these first group (G1) are arranged in the first chess board in the (uxv) matrix CBG1, for the second group (G2) of g1 cells, full connectivity, where, These second groups (G2) are arranged in a second chess board in the (uxv) matrix CBG2, which is shifted by g1 / 2 units in both row and column directions relative to the first chess board arrangement CBG1, and the second groups (G2) at the edges of the matrix CB along the shift direction are completed by the units at the diagonal relative positions at the edges of the matrix CB that are not covered by the second (shifted) chess board arrangement CBG2, and the second groups (G2) at the corners in the shift direction are completed by the units in the three corners of the matrix CB that are not covered by the second (shifted) chess board arrangement CBG2.

11. The neural network device according to claim 9 or 10, wherein The recursive connection unit (RCU) comprises at least one unit (c i ), to the unit (c i ) of at least one unit, where s=2 or 3 or 4 or 5, preferably at least 10%, more preferably at least 20% of the potential skip-layer feedback connectivity.

12. A neural network device according to any preceding claim, wherein The plurality of n damped harmonic oscillators (DHOi, i=1 to n) comprises at least two different types of damped harmonic oscillators, wherein each of the at least two different types of damped harmonic oscillators has at least one damped harmonic oscillator having a characteristic parameter ω. 0i =The natural frequency of the undamped oscillation is different from that of other types of damped harmonic oscillators.

13. The neural network device according to any one of claims 9 to 12, wherein Setting parameters ω 0i = the undamped oscillation natural frequency of the damped harmonic oscillator of the nr units of the nr-th network layer, r = 2, 3, ..., 6, so that the highest natural frequency of the n(r-1)-th unit of the n(r-1)-th network layer is higher than the maximum natural frequency of the nr units of the nr-th network layer, and the lowest natural frequency of the n(r-1)-th unit of the n(r-1)-th network layer is higher than the minimum natural frequency of the nr units of the nr-th network layer.

14. A neural network device according to any preceding claim, wherein Input unit (IU), an input connection (IC) connected to the input / output node of at least one unit (nci) i ), adapted to receive and input time series input data (S(t)) of length Tl1 to input the input data (S(t)) to an input connection (IC) of an input / output node of at least one unit (ci) i ),as well as Output unit (OU), an output connection (OC) connected to the input / output node of at least one unit (nci) i ) is adapted to output output data (O(t)) at an output start time (OST) set to a predetermined time interval after receiving an input start of input data (S(t)).

15. A method for training a recursive neural network device, according to any one of the preceding claims, comprising the steps of: a) setting the natural frequencies of a plurality of n damped harmonic oscillators (DHOi, i=1 to n), optionally such that they generate at least three different types of damped harmonic oscillators, wherein each of the at least three different types of damped harmonic oscillators has at least one characteristic parameter ω. 0i = The natural frequency of the undamped oscillation is different from that of the other two damped harmonic oscillators, and optionally, Based on the determination of power spectral densities of a plurality of representative samples of the time series input data (S(t)) to be processed and the variance of these power spectral densities, the natural frequencies are distributed on the peaks of the variance, wherein one of the at least three natural frequencies is set to correspond to the variance peak with the lowest frequency, and one of the at least three natural frequencies is set to correspond to the variance peak with the highest amplitude; b) Set the initial weight w of the connection (wi,j) of the recursive connection unit i,j ∈R and |w i,j |≤10, optionally set the initial connection output time delay of the connection (wi,j) of the recursive connection unit c) performing a training sequence by processing a plurality of representative samples of the time series input data (S(t)) to be processed; d) evaluating the results of the training sequence of step c) by comparing them with the expected results; e) performing a back propagation through time (BPTT) technique based on the result of the training sequence in step c), and adjusting the weight w of the connection (wi, j) of the recursive connection unit based on the BPTT result i,j , and optionally adjust the output time delay of the connection (wi,j) of the recursively connected unit f) Repeat steps c) to e) until f1) the result of step d) is that the result obtained is consistent with the expected result, or f2) the preset maximum number of repetitions of steps c) to e) has been reached, but step f1) has not been reached, and g) Fix the weights of the recursive connection units and optionally determine the weights of the connection delays based on the weights and connection delays that resulted in the conformance of f1), or terminate training.

Images