Recurrent neural network, recurrent neural network device, and recurrent neural network learning method

Abstract

A plurality of n damped harmonic oscillators (DHOi), each of which is one cell (nci) of the neural network, an input unit (IU) adapted to receive and input time-series input data (S(t)), and an input / output node (IO) of the corresponding cell (nci) for each cell (nci). i ) and at least one other cell's (ncj) input / output node (IO j ) and a recurrent connection unit (RCU) having at least one connection (wi,j) between the corresponding cell (nci), and the connection (wi,j) is connected to the corresponding input / output node of the corresponding cell (nci) to generate a damped harmonic oscillation (h i ) to the input / output nodes of one other cell (ncj).

Description

[Technical Field]

[0001] The present invention relates to a recurrent neural network and a corresponding recurrent neural network device, as well as a method for training such a recurrent neural network. [Background technology]

[0002] Neural networks are widely known in the art, including, for example, feedforward networks, deep convolutional neural networks (DNNs), reservoir computing networks, recurrent neural networks (RNNs), and other types of neural networks.

[0003] Despite their attractive properties, RNNs have been less studied than pure feedforward DNNs due to the difficulty of training them with the backpropagation through time (BPTT) algorithm due to the exploding and vanishing gradient (EVG) problem.To make recurrent networks more amenable to training, various approaches have been proposed, ranging from gradient scaling to limiting the recurrent weight matrix to gate architectures.

[0004] "Lipschitz Recurrent Neural Networks" by N. Benjamin Erichson et al. (arXiv:2006.12070v3[cs.LG], April 24, 2021) describes one such approach, viewing RNNs as continuous-time dynamical systems described by linear components and Lipschitz nonlinearities.

[0005] Other recent approaches have attempted to exploit the complex dynamics of coupled oscillator networks for computation. For example, one approach towards hardware implementation is “Vowel recognition with four coupled spin-torque nano-oscillators” by M. Romera et al. (Nature 563, 230-234, DOI: 10.1038 / s41586-018-0632-y (2018)) and “Binding events through the mutual synchronization of spintronic nano-neurons” by M. Romera et al. (Nat. Commun. 13, 883, DOI: 10.1038 / s41467-022-28159-1 (2022)). These authors discuss spin-torque nano-oscillators as a natural candidate for building hardware neural networks composed of coupled nanoscale oscillators.

[0006] Yet another approach is described in "COUPLED OSCILLATORY RECURRENT NEURAL NETWORK (CORNN): AN ACCURATE AND (GRADIENT) STABLE ARCHITECTURE FOR LEARNING LONG TIME DEPENDENCIES" (arXiv:2010.00951v2 [cs.LG], March 14, 2021 (hereafter, Prior Art 1 = PA1)) by T. Konstantin Rusch and Siddhartha Mishra. They provide a performance analysis of the corresponding algorithm and describe another of these approaches, in which a model of coupled oscillators is used in an RNN to create a coupled oscillatory recurrent neural network (coRNN). The coRNN is trained using the BPTT algorithm, and the EVG problem is mitigated in this coRNN.

[0007] W. Moy et al.'s paper, "A 1,968-node coupled ring oscillator ciRCUit for combinatorial optimization problem solving" (Nat Electron (2022), DOI:10.1038 / s41928-022-00749-3 (hereinafter referred to as Prior Art 2 = PA2)), describes a scalable ring oscillator-based integrated circuit as an alternative to quantum-based, optical-based, and spintronic-based approaches for computing architectures.

[0008] Various prior art approaches are directed to mere task performance without hardware implementation, or to hardware implementation of relatively simple tasks, or to solving specific application problems. At least some approaches require high computational power and / or specific temperatures, both of which lead to high energy consumption. Summary of the Invention [Problem to be solved by the invention]

[0009] Therefore, the objective of the present invention is to provide a solution for the design of a computational architecture that exploits the complex dynamics of networks coupled with damped harmonic oscillators, and that can be implemented equally efficiently in hardware, e.g., electrical digital or analog hardware, e.g., in complementary metal-oxide-semiconductor (CMOS) technology, and software, and that can be trained accurately using the backpropagation through time (BPTT) algorithm, and that is energy-efficient. [Means for solving the problem]

[0010] This object is achieved by a recurrent neural network device according to claim 1 or a method for training such a recurrent neural network according to claim 15.

[0011] Further refinements are set forth in the dependent claims.

[0012] The RNN of the present teachings may differ from known RNNs in several features, not the least of which is the different natural frequency of the damped harmonic oscillator.

[0013] Any other significant differences may include time delays in the connections, and / or different damping coefficients, and / or multi-layer structures with specific connection structures in the same layer, and / or different feedforward and feedback connections between layers, and / or different distribution of natural frequencies between layers, etc.

[0014] The introduction of oscillatory units enables high-dimensional dynamical phenomena such as resonance, entrainment, synchronization, phase shift, and desynchronization, enabling the generation of high-dimensional dynamical landscapes in state space. The potential introduction of heterogeneity enhances these effects.

[0015] The design of interacting oscillatory units in a recurrent network allows for the design of computational architectures that can be implemented equally well in hardware or software, and / or allows for efficient training of networks using BPTT, and / or allows for realization in semiconductor devices where the final setting of the recurrent network parameters can be performed after training using software algorithms.

[0016] Further features and advantages will become apparent from the following description of the embodiments with reference to the drawings. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a schematic connection diagram of a first embodiment of a recurrent neural network having n damped harmonic oscillators and connections between all the n damped harmonic oscillators. [Figure 2] a) Schematic of a damped translational mechanical harmonic oscillator, b) Schematic of a damped electrical RLC harmonic oscillator, c) Schematic of a damped harmonic oscillator implemented with analog electrical elements. [Figure 3] FIG. 10 is a schematic circuit diagram of a second embodiment of a neural network with n damped harmonic oscillators, showing only the connections between the first oscillator and the other oscillators. [Figure 4] FIG. 4 is a schematic circuit diagram of one of the damped harmonic oscillators of the second embodiment of the neural network of FIG. 3, showing details of its input and output nodes. [Figure 5] a) A schematic diagram of a VCO; b) A schematic graph of output power versus frequency of such a VCO; c) An exemplary schematic CMOS implementation of such a VCO. [Figure 6] a) shows the variance of the power spectral density of MNIST data number 7 shown in b). [Figure 7] a) Waveform of one sample of one-second-long speech data, b) variance of the power spectral density calculated for the power spectral density of 1000 samples, c) performance increase of a one-layer NHHORNN over a two-layer NHHORNN with the same number of cells when processing such data. [Figure 8] 10A and 10B are schematic diagrams of a recurrent neural network according to the third embodiment, where a) shows a general representation, and b) to d) show different representations for explaining connectivity, showing the diversity of connectivity between groups of network cells. [Figure 9] FIG. 9 is a schematic diagram of a two-layer recurrent neural network of a fourth embodiment, having a first layer corresponding to the third embodiment of FIG. 8; [Figure 10] 10A and 10B are schematic diagrams of a recurrent neural network according to a fifth embodiment, where a) shows a general representation and b) shows another representation for illustrating connectivity. [Figure 11] 1. FIG. 1 a) is a schematic diagram of a three-layer recurrent neural network of a sixth embodiment, having a first layer corresponding to the first embodiment of FIG. 1, and FIG. 1 b) is a schematic diagram of each layer. [Figure 12] An example of recognition confidence in MNIST recognition using the small NHHORNN of this teaching. DETAILED DESCRIPTION OF THE INVENTION

[0018] In the following description of the embodiment, ∈R n A term written in one or more bold letters, such as U (bold) or y (bold) or LC (bold), represents a vector, and n vector elements U are elements ∈ R of the real number i or y i or LC i (i=1,2,...,n) (The letters written in bold in the international patent application corresponding to this case are immediately followed by (bold)). Also, W(bold)∈R n×m One or more bolded terms, such as W (bold), denoted as i,j represents an (n × m)-element matrix (i = 1, 2, … , n; j = 1, 2, … , m) with

[0019] We started by considering biological neuronal systems such as the cerebral cortex. Neurobiological studies of the cerebral cortex have shown that these neuronal networks comprise highly complex, nonlinear dynamics that can self-organize to generate high-dimensional spatiotemporal patterns. These patterns manifest as frequency-varying oscillations, transient synchronous or desynchronous phenomena, resonance, entrainment, phase shifts, and traveling waves.

[0020] The present invention focuses on interacting oscillatory units in recurrent networks and implements these principles, realized in natural neuronal systems, in a novel way to design a computational architecture that can be implemented in hardware and software alike.

[0021] The superior performance compared to prior art systems has been demonstrated through quantitative evaluation of task performance in numerous standard benchmark tests for pattern recognition. Specifically, recurrent networks of the present teachings have been tested for performance on pattern classification tasks from the well-known MNIST handwritten digit dataset, the more challenging EMNIST dataset consisting of handwritten digits and letters, and the more challenging Omniglot dataset consisting of 1623 handwritten character classes from 50 alphabets. They have also been tested on speech recognition tasks based on the Free Spoken Digit Dataset, which consists of recordings of 3000 spoken digits from six speakers. The various types of tasks have already demonstrated that recurrent neural networks of the present teachings excel at recognizing patterns in time-series data, regardless of the actual content type of the task.

[0022] The biologically inspired recurrent neural networks of the present teachings are particularly suitable for hardware implementation with low energy consumption and are superior to prior art RNNs with gated architectures, such as Long Short-Term Memory (LSTM) networks and Gated Recurrent Unit (GRU) networks, in terms of learning speed, noise immunity, hardware requirements, and number of system variables. Furthermore, by analyzing these recurrent neural networks of the present teachings, the underlying mechanisms that result in this improved task performance can be identified.

[0023] The recurrent neural network (RNN) of the present teachings will be described using the RNN developed by the inventors and shown schematically in FIG. 1 for illustrative purposes as a basic model.

[0024] The basic model of an RNN can be viewed as consisting of n damped harmonic oscillators (DHOi, i = 1, 2, ..., n) that share an essential dynamic feature: an oscillation characteristic defined by a natural frequency ω and a damping coefficient γ. Such an RNN is a homogeneous recurrent neural network composed of multiple identical DHO cells. The cells are connected in an all-to-all fashion without conduction delays, and all nodes are tuned to the same preferred frequency (see Figure 1). In the present teachings, such a homogeneous recurrent network composed of multiple identical DHO cells may be referred to as a homogeneous harmonic oscillator RNN (HHORNN) and represents a first non-limiting embodiment of the present invention. Such an HHORNN has been trained on a set of benchmark tasks using a backpropagation-through-time (BPTT) algorithm, allowing gain adjustment of all connections to the network nodes, including input, recurrent, and output connections.

[0025] The oscillation of each single DHO cell of such an HHORNN can be considered to follow the general second-order differential equation (1) of a damped harmonic oscillator:

[0026]

number

[0027] β i = damping coefficient, ω 0i = the natural frequency of the undamped oscillation. For example, for the corresponding damped translational mechanical harmonic oscillator DHOi (see Fig. 2a), β i =c i / 2m i and ω 0i 2 =k i / m i and m i = mass, k i = spring constant, c i = viscous damping constant, or for the corresponding damped electrical RLC harmonic oscillator DHOi (see Fig. 2b), for other types of damped harmonic oscillators, βi =R i / L i and ω 0i 2 =1 / L i C i and R i = resistance, L i = inductance, C i = capacitance. An example of another type of harmonic oscillator is shown in Figure 2c) below Figure 3, where the DHO is implemented using analog elements such as operational amplifiers, inverters, and potentiometers. In the example shown, for example, two operational amplifiers are used as integrators I1 and I2, one inverter and several potentiometers are used as metering coefficients wi and j, and the damping coefficient is set to 2β i , the natural frequency is ω 0i Since potentiometers cannot achieve coupling / weighing coefficients wi,j greater than 1, operational amplifiers or other amplifying elements can be used instead of potentiometers if desired and necessary. In the HHORNN, β i =β j and ω 0i 2 = ω 0j 2 corresponds to all i,j. However, the above more general description can also be used to describe the DHOs of other RNNs in the present teachings, so the above notation with subscripts i,j is also used for HHORNNs.

[0028] The electrical properties of a network of electrical RLC circuits designed to have certain properties of biological neural networks can be defined as follows: A series connection of electrical RLC circuits driven by external inputs for which the Kirchhoff equation for voltage holds:

[0029]

number

[0030] Equation (3) is derived by substituting resistance R, inductance L, and capacitance C into equation (2).

[0031]

number

[0032] It is implemented by an RLC circuit described by equation (3) and has a time-varying external input U (bold type). ext (t)∈R n A system of n additively coupled DHOs driven by and differentiated once to obtain a system of ordinary differential equations is given by the input I(bold)(t)∈R. n can be expressed in vector form as equation (4).

[0033]

number

[0034] The following is an important feature of the RNN of this teaching: We assume and design a neural network so that any external input to any DHO is determined by a nonlinear combination of the activity of the entire recurrent neural network and an external signal. This nonlinear combination is expressed as Equation (5).

[0035]

number

[0036] where W∈R n×n is the pairwise connection weights w as an (n × n) element matrix (i = 1, 2, … , n; j = 1, 2, … , n). i,j indicates, S (bold) ext =S (bold) ext (t)∈R n is a vector of input signals.

[0037] To obtain the final system of first-order differential equations, we substitute x(bold) = I(bold)(t) and y = dI(bold)(t) / dt into equation (4) to obtain equation (6).

[0038]

number

[0039] Equation (6) describes the HHORNN shown in Figure 1, and the DHO is implemented as an RLC circuit as shown in Figure 2b. However, to train the HHORNN using BPTT, a discrete-time description of the HHORNN is required, which requires a numerical integration scheme. For this purpose, we can use the well-known Euler discretization of (6) with time constant τ, resulting in equation (7).

[0040]

number

[0041] The above equation describes the HHORNN as shown in Figure 1, and the DHO is implemented as an RLC circuit as shown in Figure 2b), which can be trained as a recurrent neural network. n×n The connection weight of the pair w i,j is adjusted to minimize the error (cost). In other words, the connection w between cells DHOi i,j is adjusted in the representation shown in Figure 1. This learning involves the weights of the input and output connections, which will be apparent from the description of the input and output units below.

[0042] The connections between these cells DHOi i,j Each of these has a transmission characteristic T i,j =w i,j ×h i can be set (adjusted) to w i,j ∈R is the pairwise connection weight, and |w i,j |≦10. The pair weight is |w i,j If set to |=0, the connection from cell DHOi to cell DHOj does not exist or is interrupted.

[0043] The recurrent neural network of the present teachings does not only target the above-mentioned HHORNN. The recurrent neural network of the present teachings also targets a recurrent neural network composed of non-homogeneous DHOs. "Non-homogeneous" refers to the fact that the multiple n damped harmonic oscillators (DHOi, i = 1 to n) of the RNN are composed of at least two different types of damped harmonic oscillators, rather than being composed of only one type of DHO cell having the same β = damping coefficient and the same ω = DHO natural frequency of undamped oscillation, and each of these at least two different types of damped harmonic oscillators has at least a parameter ω 0i = The natural frequency of the undamped oscillation is different from that of one or more other types of damped harmonic oscillators.

[0044] Such a recurrent network composed of DHO cells with at least two different types of damped harmonic oscillators is identified in the present teachings as a non-homogeneous harmonic oscillator RNN (NHHORNN). Such an NHHORNN represents a second non-limiting embodiment of a recurrent neural network of the present teachings. An NHHORNN with n damped harmonic oscillators is described with reference to FIGS. 3 and 4.

[0045] In FIG. 3, n damped harmonic oscillators DHOi (i=1, 2, ..., n) are shown schematically in a kind of circular arrangement similar to the arrangement in FIG. 1, but rectangular in FIG. 3. The circular arrangement is shown for illustrative purposes only to simplify the schematic representation of the connections. The number n of damped harmonic oscillators of the present teachings is n≧8, and if n=8 is selected, the representation in FIG. 3 shows all eight damped harmonic oscillators DHOi, i=1, 2, ..., 8. The damped harmonic oscillators DHOi are shown as damped electrical RLC harmonic oscillators. Each of the damped harmonic oscillators is connected to a resistor R i , inductance L i , and capacitance C iThese capacitances can be individually set or predetermined for each of the harmonic oscillators DHOi. As a result, the damping coefficients β of each of the damped harmonic oscillators are i and the natural frequency of the undamped vibration ω 0i can be set or predetermined independently from the corresponding parameters of each of the other damped harmonic oscillators DHOj. NHHORNN is the natural frequency ω of the undamped oscillation 0i The harmonic oscillator comprises at least two damped harmonic oscillators DHOi having different amplitudes.

[0046] It should be noted that HHORNNs and NHHORNNs are not limited to implementation in RLC-type electrical oscillators, but can also be implemented in, for example, damped mechanical harmonic oscillators (DHOs). In this case, the corresponding damped translational mechanical harmonic oscillator, shown schematically in Fig. 2a), can be implemented by the general second-order differential equation (1) for damped harmonic oscillators (β i =c i / 2m i and ω 0i 2 =k i / m i , m i = mass, k i = spring constant, c i = viscous damping constant). Other options include, for example, a second-order (galvanic) cell, a chemical oscillator system such as the Belousov-Zhabotinsky reaction, or an implementation in a computer program / software. By the representation of the analog DHOi in Fig. 2 c) below Fig. 3, it is understood that one or more RLC-type electrical oscillators can be replaced, for example, by such analog DHOi.

[0047] In the embodiment of NHHORNN shown in Fig. 3, all-to-all connections are provided, including self-connections, i.e., connections of DHO to itself, which are connected to the input / output nodes IOi of all damped harmonic oscillators DHOi. i is the input / output node IO of all other damped harmonic oscillators DHOj. j , and its own input / output node IOi This means that the first damped harmonic oscillator DHO1 may be connected at both the input and output to DHO1. For purposes of illustration, Figure 3 only shows the connections from the first damped harmonic oscillator DHO1 to all other damped harmonic oscillators DHO2 through DHO1, i.e., w1,2, w1,3, ..., w1,n, as well as the connections from the input / output nodes of all other damped harmonic oscillators DHOj (j = 2, 3, ..., n) to DHO1, i.e., w2,1, w3,1, ..., wn,1, and to itself, w1,1. Other connections, i.e., all-to-all connections, potentially exist but are not shown for the sake of simplicity. The properties of connections wi,j and their implementation are further described below.

[0048] Referring to FIG. 4, one cell nci of the damped harmonic oscillator DHOi and its input / output node IO i is shown in more detail. One such dynamic harmonic oscillator DHOi and its input / output node IO i constitutes one cell nci of a recurrent neural network. i is the cell input x i Input connection IC that generates i Input connection IC i is an adder ICA that receives all inputs from itself and all other cells of the recurrent neural network via connections wi,j. i and the corresponding input / output nodes IO of the corresponding cells. i is connected to the input that receives the external signal S to be processed, the adder ICA i is the corresponding input signal S, if applicable. i (t) is additionally adapted to receive S i The inputs of (t) are shown with hatched arrows representing possible connections. i The input of (t) is shown by a hatched arrow. The input of signal S and signal S will be described later.

[0049] Adder ICA i The output of the adder ICA i output signal to cell input xi an input element ICS implementing a transfer function of a nonlinear type, preferably a saturation nonlinear type such as a sigmoid type optionally with an offset, or more preferably of a tanh type, to generate i A semiconductor circuit for implementing a nonlinear saturation, such as a sigmoidal transfer function, either in single-transistor or CMOS technology is described, for example, in US 2018 / 0307977 A1. Thus, this input element ICS i A description of specific hardware implementations of the above is omitted here, as US 2018 / 0307977 A1 contains numerous examples.

[0050] Input / Output Node IO i is output connection OC i This output connection is OC i is the resulting damped harmonic oscillation h i is output to the corresponding connection wi,j.

[0051] The inputs to and outputs from the DHO are not shown in detail in Figure 4. By way of non-limiting example only, a voltage controlled oscillator (VCO) that may be an implementation of a DHO is shown in Figures 5a-c. Such a VCO may be implemented in CMOS technology, for example, as shown schematically in Figure 5c. In Figure 5c, the input voltage Vin i and the output voltage Vout i are shown at the corresponding nodes. In this respect, for such a complementary CMOS VCO, the input and output voltages are 180° out of phase, so to make them in phase, simply add an inverter to the input or output side. However, for the purposes of implementing HHORNN and NHHORNN and calculating the characteristics of HHORNN / NHHORNN, the presence or absence of such a phase shift between the input and output is irrelevant.

[0052] Returning to Figures 3 and 4, the input connection wj,i and the output connection wi,j are further expressed as ∂t i and ∂t jThese designations indicate optional time delays in the connection, which may or may not be present (=optional configuration), and are described in more detail below.

[0053] All these connections wi,j are input / output nodes IO of all cells in the recurrent neural network. i The recurrent connection unit RCU consists of all connections wi,j between i,j and wi,j. The recurrent connection unit RCU is represented by the hatched box RCU in Figure 3. All connections of the recurrent connection unit RCU can be described by an (n × n) matrix W (bold) with matrix elements wi,j (i, j = 1, 2, ..., n).

[0054] These recurrent neural networks, implemented as HHORNNs or NHHORNNs, are suitable for efficient processing of time-series input data S(t). An RNN consists of an input unit IU for receiving and inputting such time-series input data S(t), and an output unit OI adapted to output the result, i.e., output data O(t).

[0055] In the simplest form, the input unit IU for receiving and inputting such time series input data S(t) is an input connection IC of one input / output node of the cell DHOi, as exemplarily shown in FIG. i The simplest form of an output unit OI adapted to output the result, i.e. the output data O(t), is the output connection OC iThe output unit is further adapted to start outputting at a start time OST after receiving the first input of the input data S(t), which is set to be at least 5 times, preferably 10 times, and more preferably 20 times ti1 later than the first input of the input data S(t) (as further described below). The output data is output at an output time interval to2, which may be the same as or different from ti1. The number of cells forming the input units or connected to the input units and the number of cells forming the output units or connected to the output units can be freely selected. For example, if a recurrent neural network is composed of n=16 damped harmonic oscillators, all 16 cells may be connected to receive the input data S, and 8 of the 16 cells may be output from the output unit. Alternatively, 8 of the 16 cells may be connected to receive the input data S, and the other 8 of the 16 cells may be connected to receive the input data S from the output unit.

[0056] The simplest form of a recurrent neural network of the present teachings is a network with n = 8 damped harmonic oscillators. While the number of damped harmonic oscillators is not limited per se, for practical reasons the number of DHOs in a network layer may be limited to 10^6, or preferably 250,000, 102,400, 40,000, 10,000, 2,500, 1,024, or 512.

[0057] The RNN in FIG. 3 described above can be described by the following equation (8).

[0058]

number

[0059] This statement corresponds to equation (6) with the scalars R / L, 1 / LC, and 1 / L replaced by the corresponding vectors. To train the NHHORNN, a numerical solution is required. For this purpose, we use the well-known Euler discretization of equation (8) with time constant τ, which gives us equation (9) below:

[0060]

number

[0061] The above equation describes the NHHORNN shown in Figures 3 and 4, which can be trained as a recurrent neural network. i =β j and ω 0i 2 =ω 0j 2 exists for all i,j, the above formula can equally be used to describe an HHORNN such as Figure 1.

[0062] The following describes the original setup of such a recurrent neural network of the present teachings and its training.

[0063] As mentioned at the beginning, training recurrent neural networks using BPTT techniques is fundamentally difficult due to the well-studied exploding and vanishing gradient problem (EVG problem). This EVG problem arises when the gradient of the loss function used to learn long-term dependencies cannot be properly propagated through the recurrent network's iterations. However, HHORNN and its derivative NHHORNN do not face this problem. The mathematical proof is presented in PA1 and will not be described here.

[0064] In electrical hardware, especially in CMOS and other semiconductor implementations, one conclusion can be drawn: for given values of R and L, the value of capacitance C determines the characteristics of the DHO. In particular, L can be difficult to set in the later stages of semiconductor implementation, when the overall structure of the HHORNN or NHHORNN is defined but the HHORNN or NHHORNN still needs to be trained for initial learning or to learn "new things." On the other hand, capacitance can be set relatively easily in steps or, in later stages, continuously over a relatively wide range using well-established semiconductor varactor (variable capacitor) banks. In later stages of hardware implementation, capacitance can be programmable so that it can be changed at any time, or it can be fixed by fusion or other methods.

[0065] Therefore, during the original or initial setup of the HHORNN / NHHORNN to be trained, it is important to choose the same parameters β and ω0 for all DHOs in the case of HHORNN, and potentially different parameters β for the case of NHHORNN. i and ω 0i Selecting the ω0 for the HHORNN DHO is a challenge. It is generally possible and effective to simply select a single ω0 for the HHORNN DHO that lies within (i.e., overlaps) the expected frequency range of the input signal. It is also generally possible and effective to simply select a different ω0 for the NHHORNN DHO, preferably covering a wide frequency range where the DHO's receptive frequency range overlaps. Referring to Figure 5b), the corresponding DHO can be expected to oscillate with an associated output power amplitude of at least 1 / 10 or 1 / e of the maximum output power when excited at its corresponding frequency. This wide frequency range, of course, should preferably overlap with the expected frequency range of the input signal. While the overlap is not strictly necessary for the HHORNN / NHHORNN, it is highly desirable for improved performance.

[0066] This results in an input signal S(t). The present invention is designed to process time series input data (S(t)). The time series input data (S(t)) is either discrete or continuous time series input data (S(t)) having a time series interval ti1 and a time series length TI1, where the time series interval ti1 of the time series input data (S(t)) represents either the time interval between successive discrete values in the case of discrete time series input data (S(t)), or the time interval between successive sampled values in the case of continuous time series input data (S(t)) such as a sinusoidal signal. In other words, the data to be processed is represented as such time series input data (S(t)).

[0067] If the data to be processed is, for example, the MNIST dataset consisting of 60,000 samples of handwritten digits, each consisting of 28x28 intensity values and representing one handwritten digit from 0 to 9, the same can be converted into a time series dataset D of length TI1=784 by collecting the pixels of each digit in scan line order from the top left to the bottom right. The clock rate 1 / ti1 for inputting this time series data sets the time interval ti1. Note that the clock rate corresponding to the time series interval ti1 can be freely selected.

[0068] If the data to be processed are audio data and each data set consists of a certain length of audio data in real time with associated frequencies ranging for example from 500Hz to 16000Hz, the same are sampled at a particular sampling rate and converted into a time series data set having time series interval ti1 and time series length TI1.

[0069] Whatever the content of the data to be processed, whether originally in the form of time-series input data (S(t)) or converted to time-series input data (S(t)), such as the MNIST dataset, represents a plurality of samples of the time-series input data (S(t)), where the samples of the time-series input data (S(t)) have power spectral densities and variances of these power spectral densities.

[0070] The parameter ω is the natural frequency of undamped oscillations of different types of damped harmonic oscillators of NHHORNN. 0i is preferably set based on the determination of the power spectral density of multiple samples of the time series input data (S(t)) to be processed and the variance of these power spectral densities so that the natural frequencies are distributed at the peak of this variance. One of the natural frequencies is located at the peak of this variance, and the lowest frequency and the other natural frequencies are distributed in the frequency range of the variance. In the case of the HHORNN, the undamped oscillation parameter ω0 = natural frequency of the damped harmonic oscillator of the NHHORNN can be selected according to the power spectral density of multiple samples of the time series input data (S(t)) to be processed and the variance of these power spectral densities. The natural frequency ω0 (in the case of the HHORNN) or the natural frequency ω 0i is preferably set to correspond to the peak of the variance of the power spectral density of the input.

[0071] The input / output node IO of one cell corresponding to the recurrent connection unit RCU i and another cell's input / output node IO j The connection wi,j between i,j =w i,j ×h i Toshi, w i,j ∈R, |w i,j |≦10. In other words, |w i,j Set the output strength / amplitude of one cell to 0 by |=0 or 1<|w i,j Amplify by |≦10 or |w i,j |=1 to keep it at the same level, or |wi,j You can either set it to |<1 to weaken it or invert the output with the corresponding amplitude. i,j |≦5 or |w i,j |≦3 or |w i,j It can also be set to |≦2, which means that the output of the corresponding damped harmonic oscillator, h i This can be achieved using known circuits such as voltage dividers or transmission gate coupling. An example of such transmission gate coupling is shown in Figure 2 for PA2. Other circuits can be implemented using amplifiers with preset gain ratios. This gain ratio is preset via values stored in a memory bank for setting the connection values.

[0072] These connections and their characteristics are the elements learned in the training of the HHORNN in Figure 1 and the NHHORNN in Figure 3.

[0073] As already mentioned, the NHHORNN in FIG. 3 does not include such a transmission delay, but the input / output node IO of one corresponding cell in the recurrent connection unit RCU i and another cell's input / output node IO j The connection wi,j between i,j In addition, the transmission delay can be set arbitrarily. i,j =k i,j ×Δt, k i,j =0,1,…,k, 0≦k≦kmax, ti1 / 10≦Δt≦ti1, k i,j × Δt ≦ 50ti1 is, for example, the output h of the corresponding damped harmonic oscillator i This can be implemented by a series connection of inverters or a clock output gate for the transmission of . When such a transmission delay is provided, NHHORNN converts the terms of matrix W (in bold) in equation (9) into the corresponding time t k = matrix W (boldface) multiplied by the corresponding vector of input change at t-(k×Δt) k By substituting the sum of k products of

[0074]

number

[0075] The maximum value of k, kmax, can be set to 1000, 500, 100 or a value smaller than 100, such as 50, 40, 30, 20, 16, 12, 10, 8, 5, 4, 3, 2, etc. kmax is preferably in the range of 30 to 10. The important point is that the matrix W (bold) k contains only one element ≠ 0 at each position i,j for all k, so introducing delays does not increase the complexity of the system description. Therefore, the performance of HHORNN / NHHORNN improves without increasing the amount of computation required for training.

[0076] The input signal S(t) is input to one or more input / output nodes of the cell nci via input connections IC i The input signal is input via an input unit IU connected to the neural network. The input unit IU is formed by selected ones of the cells nci of the neural network or by separate units. The input signal can thus be input to one single cell or to nI cells, where nI ≤ n. The data forming the input signal can be input to several cells either serially or in parallel. The input signal can therefore be written in general form as follows:

[0077]

number

[0078] Here, S (bold) ext =S (bold) ext (t)∈R n is a vector of input signals, WI(bold)∈R n×q are the pairwise connection weights wi as an (n × p) element matrix (i = 1, 2, ..., n; p = 1, 2, ...) for input to one or more cells nci of the neural network that are selected (predetermined) to receive input from the input unit. i,jFor example, if only one cell ncr among the cells nci of the neural network should receive input, it can be written so that only the matrix element wi,j at i=r (i-th row) is different from 0, and the matrix elements at i≠r (all other rows) are 0. If only one data value should be input per input cycle, then the vector D(bold)(t)∈R q Only one vector element d of the vector D(t)∈R can potentially have value ≠ 0, in other words, q = 1. If q ≠ 1 data value should be input in the same input cycle (time interval ti1), then the vector D(bold)(t)∈R q has q elements, potentially with values ≠ 0.

[0079] The above description of HHORNN and NHHORNN provides design tools for studying how the electrical characteristics of DHOs and their connections—in other words, the “neural network”—determine the system’s learning ability and thus the overall performance of the trained network, and how design changes result in different capabilities and performance. The above description of HHORNN and NHHORNN also provides tools for providing inputs and outputs to DHOs and their connections—in other words, for implementing HHORNN / NHHORNN in software running on a correspondingly programmed data processing device. HHORNN / NHHORNN can also be implemented in hardware with mechanical elements such as spring-supported weights, and the critical nonlinear saturating input elements can also be implemented with viscoelastic input elements, or hydraulic or pneumatic input elements with such force transmission. However, hardware implementations in the form of electrical circuitry, particularly semiconductor circuitry, are currently preferred.

[0080] HHORNN / NHHORNN were trained on a number of classification benchmark tasks commonly used to evaluate the performance of RNNs, and the same was done for LSTM and GRU gate networks, which are the de facto state of the art for RNN architectures in current machine learning.

[0081] In all cases, input data was presented to the network sequentially in the form of time-series data, and the network was trained in a supervised manner using backpropagation through time (BPTT). All networks were implemented in the PyTorch framework and trained using BPTT. To enable comparability across different network architectures, the network size was chosen so that all networks had the same number of learnable parameters; unless otherwise noted, 10,000 parameters was the default choice. For all tasks, a training batch size of 128 and the AdamW optimizer were used. For network architectures that suffer from exploding or vanishing gradients (GRU, LSTM), a parameter search in the space of learning rates and gradient scaling values was used to determine the optimal configuration of the learning rate and gradient scaling for each model. Classification accuracy on the test set was calculated as a measure of the network's task performance. To evaluate learning speed, these accuracies were calculated throughout the entire training process, evenly sampled across m training batches.

[0082] For HHORNN and NHHORNN, where vanishing and exploding gradients are not an issue, a learning rate of 0.01 was used throughout all experiments, and no gradient scaling was applied. To determine the impact of the intrinsic parameters β and ω0 on HHORNN performance, the optimal ω0 value was estimated from the dataset to be processed using information from the power spectral density of a large number of samples along with a grid search of β, as described below. Alternatively, a grid search was performed on each dataset, and the learning performance of HHORNN networks with all nodes having the same intrinsic parameters β and ω0 selected from the grid was evaluated. For each dataset, the parameter set belonging to the network with the highest learning performance was selected, and this HHORNN network with all nodes having the same intrinsic parameters was named the optimal HHORNN network, HHORNNopt. Furthermore, the spread of each intrinsic parameter in the set of 10 HHORNN networks with the highest learning performance was determined, and the minimum and maximum values in the set considered were found. These minimum and maximum values were used to assign β to each node. i ∈U([βmin,βmax]) and ω 0i We generated a non-homogeneous HORNN network with randomly assigned values such that ∈U([ω0min,ω0max]), where U([a,b]) denotes a uniform distribution within the interval [a,b].

[0083] This non-homogeneous HORNN network and the optimal HHORNN network HHORNNopt were tested to evaluate the influence of the natural vibration parameters.

[0084] Additionally, tests were conducted using overdamped, critically damped, and underdamped networks, but were not relevant to this case.

[0085] Connections wi,j were initialized according to different strategies to evaluate their influence. Connections wi,j were initially set to random values chosen from a uniform distribution U([-1 / √n, 1 / √n]), where n represents the number of DHO cells in the network. Alternatively, weights were selected according to a Gaussian distribution N(0, 1 / 3n). These different choices did not have a strong impact on the final task performance and learning speed. We also set all connections wi,j to w i,j = 1 or all wi,j = 0. Furthermore, we trained several networks on the same task with different initial weights, and found that the final task performance did not depend strongly on the weight initialization.

[0086] Of course, the choice of the discretization constant value has a significant impact on training performance. For example, on MNIST, values from 0.01 to 0.5 (0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5) were tested, and 0.2 was selected as the best performing value.

[0087] HHORNN already significantly outperforms state-of-the-art RNNs with gated units such as LSTM and GRU in terms of learning speed, absolute performance, and noise resistance, especially in areas with a small number of parameters, for the same number of learnable parameters.

[0088] NHHORNN outperforms HHORNN in every aspect, and as a result even outperforms state-of-the-art RNNs such as LSTM and GRU.

[0089] The inventors have identified several reasons why HHORNN and NHHORNN have acquired these functions: (i) The time constant introduced by discretizing the damped harmonic oscillator equation acts as a smoothing prior in the state space, regularizing the gradient and making the BPTT learning more stable; (ii) the specialized kinetics of DHO endows each node with a unique memory function, facilitating the encoding of temporal relations and making nodes sensitive to the order of inputs; (iii) the typical properties of coupled DHOs enable dynamical phenomena such as resonance, entrainment, synchronization, phase shift, and desynchronization, allowing the generation of high-dimensional dynamical landscapes in the state space; (iv) The introduction of non-homogeneous DHOs, in other words, increased network heterogeneity, amplifies the effects of (i), (ii), and (iii), allowing such heterogeneous = non-homogeneous HORNNs to respond to stimuli in more complex and less synchronized patterns.

[0090] Reason (iv) furthermore makes it possible to significantly improve the probability of classification of stimuli in the untrained state of the NHHORNN before learning / training. While the HHORNN has the above properties, the effect is more pronounced in the NHHORNN. Heterogeneous HORNNs are easy to learn because they project inputs into a high-dimensional pre-structured state space and have randomly selected connections wi,j, even in the untrained state.

[0091] In other words, the learning / training process can take advantage of the diversification of initial response patterns, significantly reducing the number of trials required to reach a high performance level.

[0092] Regarding reason (i), it is important to emphasize that the discretization by the time constant τ does not affect the hardware implementation. Whether HHORNN or NHHORNN is implemented in hardware or software, discretization is merely a tool that enables numerical computation of the model and training using BPTT. HHORNN and NHHORNN, and the cells that comprise them, whether in hardware or software, convert any input signal into oscillations. For example, digital signals such as delta pulses are converted into analog signals, and these analog signals (oscillations) further reinforce the resonance phenomenon. This fact is due to the design of HHORNN and NHHORNN and is unrelated to the discretization by the time constant τ, which is necessarily inherent in their hardware and software implementation and serves only for efficient training.

[0093] Introducing delays into connections wi,j as described above increases heterogeneity, improving overall performance for the same reasons discussed above regarding the different preferred frequencies of the DHO. A useful rule for initializing these delays is to introduce a dispersion of connection delays around a median value roughly corresponding to one-fifth of the cycle duration of the median preferred oscillation frequency. However, this is a non-limiting example. Other distributions of delay values are also possible, such as an even delay value distribution between zero delay and 20 times the time series interval ti1. Introducing delays has two effects: first, it increases heterogeneity; second, it introduces a potential reaction delay into the network. This can be thought of as a potential delay interval before providing an output, representing a kind of intermediate processing result.

[0094] The following explanation may provide a clue to better understand the results of the selection and distribution of the DHO's natural frequency. Each DHO is effectively a bandpass filter, with a "receptive field" defined by the resonant frequency window within which amplitude amplification occurs (see Figure 5b). An isolated single DHO cell in an HHORNN / NHHORNN without feedback connections can be thought of as a bandpass filter that generates a temporal convolution of the input signal and selectively amplifies signals near its natural frequency ω through resonance. If an HHORNN / NHHORNN may have self-connections predicted by potential connections wi,i, adjusting the gain of the self-connections can shift the natural frequency ω to some extent. In other words, a DHO with self-connections can be tuned to the input to some extent, and its receptive field can be altered during the learning process by changing the value of wi,i.

[0095] As mentioned above, regardless of whether the HHORNN / NHHORNN is implemented in hardware or software, the input used for training and performance verification can be an analog input, such as a pulse input or a strictly vibration input. The HHORNN and NHHORNN of the present teachings work with both discrete inputs and analog vibration inputs.

[0096] As mentioned above, if a dataset has a significant structure, such as the MNIST data, it is possible to pre-tune the cells to characteristic frequencies. In this case, setting the resonant frequency at the peak of the variance yields the best results, since it allows the network to resonate with the difference in the input signal, i.e., the stimulus. Figure 6(a) shows the variance of the power spectral density of MNIST data 7, shown in Figure 6(b). This is the time-series input data S(t) of the MNIST dataset, consisting of 28 × 28 intensity values for 7, converted (transformed) into time-series data D of length TI1 = 784 by collecting the pixels of each digit in scanline order from top left to bottom right, as described above.

[0097] In the MNIST dataset, a vertical line has a period of 28, a 45-degree angled line has a period of 27 or 29. This period is "translated" into frequency via the input time interval.

[0098] This example also helps us understand different strategies for selecting characteristic frequencies for a DHO: it is possible to select a high resonance at a precisely selected frequency if there is a significant structure in the data to be processed, and a broader tuning can be chosen if there is no such significant structure.

[0099] In the following, we explain the design and effectiveness of the multi-layered HHORNN / NHHORNN.

[0100] We used audio data to test performance on spoken digits. The "Free Spoken Digit Dataset" was used, consisting of 3,000 audio samples of the digits 0-9. Each sample was a one-second recording recorded in mono at a sampling rate of 8,000 Hz and subsampled to 4,000 Hz. The 3,000 samples represented the digits spoken by six male speakers, with 50 samples per digit. This approach resulted in a relatively small (and therefore difficult to train) dataset of 3,000 samples. 10% of these samples were used as test data, i.e., 300 samples, and 90% were used for training.

[0101] Figure 7 shows the variance of the power spectral density (PSD) calculated over a) a waveform of one sample with a length of 1 s and b) the power spectral density (PSD) calculated over 1000 samples. The amplitude values were fed into the network directly as time series data, i.e., as a 4000-dimensional vector.

[0102] The first NHHORNN, which was fed with the data and trained, had a network size of n=256 cells, with the cells arranged in one layer and having all-to-all connectivity.

[0103] The second NHHORNN, with n = 256 cells and a two-layer structure, was trained on the same task, i.e., the same data. The second NHHORNN was divided into a first layer with n1 = 128 cells and a second layer with n2 = 128 cells.

[0104] Before describing the layers and initial setup in detail, some definitions are provided to explain the multi-layer HHORNN / NHHORNN. Consider a two-layer network with n cells. Of these, n1 cells are in the first layer, which receives input signals and is therefore the upstream layer in the overall input-output direction, and n2 cells (n=n1+n2) are in the second layer, which are downstream in the overall input-output direction. Only the cells in the second layer are connected to provide output signals. All-to-all connectivity within a layer has the same meaning as in the one-layer network described above. Feedforward connectivity / connectivity means that at least one input-output node IO of the n1 cells nci in the upstream (e.g., first) network layer is connected to the input-output node IO. i and at least one input / output node (IO j ), and at least one node IO of n1 cells nci of the upstream (for example, first) network layer. i At least one input / output node (IO) of n cells (ncj) in the downstream (e.g., second) network layer from j ) and the resulting damped harmonic oscillations h i A feedback connection / connectivity means a connection for transmitting at least one input / output node IO of n cells ncj of a downstream (e.g., second) network layer. j and at least one input / output node (IO) of the cell nci of the upstream (e.g., first) network layer. i ) and at least one node IO of n2 cells ncj of the downstream (for example, second) network layer. j At least one input / output node (IO) of n1 cells (nci) in the upstream (e.g., first) network layer from i ) and the resulting damped harmonic oscillations h jThe maximum / minimum feedforward connectivity of X / Y% means that maximum or minimum X% of 100% of input / output nodes of n1 cells nci in the upstream network layer are potentially connected to maximum or minimum Y% of 100% of input / output nodes of n2 cells ncj in the downstream network layer, and the corresponding input / output nodes IO of n1 cells nci in the upstream network layer are i from the input / output node IO of n2 cells ncj in the downstream network layer j The resulting damped harmonic oscillations h i Correspondingly, the max / min feedback connectivity of X / Y% means that the maximum or minimum X% of 100% of the input / output nodes of the n2 cells ncj in the downstream network layer are potentially connected to the maximum or minimum Y% of the input / output nodes of the 100% of the input / output nodes of the n1 cells nci in the upstream network layer, and the corresponding input / output nodes IO j from the input / output node IO of n1 cells nci in the upstream network layer i The resulting damped harmonic oscillations h j , which refers to the connectivity for transmitting . Thus, if the corresponding max / min feedforward or feedback connectivity is given as N / M, such as 32 / 64, the corresponding max and min values are integers, not percentages. The potential connectivity is the matrix W (bold) in equation (9) or the matrix W (bold) in equation (10). k This means that, contains corresponding matrix elements wi,j that may differ from 0, even if the corresponding elements are eventually set to 0 as a result of training. Correspondingly, in the hardware implementation, there exist corresponding connection elements wi,j, which means that as a result of training, w i,j ∈R and |w i,j |≦10, the corresponding transmission characteristic T i,j =w i,j ×h i can be set to

[0105] The recommended frequency of the DHO in the first layer of the second NHHORNN was set so that the variance of the power density spectrum of the input data was distributed across the frequency range of the input data, similar to the first single-layer NHHORNN. In the second NHHORNN, the recommended frequency of the DHO in the second layer was set lower than the recommended frequency of the DHO in the first layer. Specifically, the resonant frequency of the second layer was set to 1 / 5 (20%) of the resonant frequency of the first layer. Other options for the resonant frequency include 1 / 2, 1 / 3, 1 / 4, 1 / 6, 1 / 8, 1 / 10, and 1 / 20. The preferred frequency of the DHO in the second layer can also be higher than that of the first layer, such as 2x, 3x, 4x, or 5x the resonant frequency of the first layer.

[0106] In the second NHHORNN with 128 cells + 128 cells, the 128 cells in the first layer had all-to-all connectivity between the 128 cells (= 100% feedforward connectivity, 100% feedback connectivity, and self-connectivity within the layer), and the 128 cells in the second layer had all-to-all connectivity between the 128 cells (= 100% feedforward connectivity, 100% feedback connectivity, and self-connectivity within the layer), with 100% feedforward connectivity among cells in the first layer and 50% feedback connectivity among cells in the second layer, in other words, 100% feedforward connectivity and 50% feedback connectivity between the two layers.

[0107] The performance increase between a single-layer NHHORNN and a two-layer NHHORNN with the same number of cells is shown in Figure 7c. One reason for this performance increase is that the lower resonant frequency of the second layer acts as a low-pass filter, suppressing high-frequency components in the activity pattern of the first layer. These high-frequency components typically result from uncorrelated input signals, and therefore tend to undermine rather than support classification. Therefore, as tasks become more complex and require more cells, the already significantly improved performance of the single-layer NHHORNN can be further improved by the multi-layer NHHORNN without increasing the number of cells.

[0108] A NHHORNN typically consists of multiple n cells (n≧8). Cells may be connected with all-to-all connections, but as the number of cells n becomes large, all-to-all connections do not provide sufficient performance improvement for the explosive complexity of the connections. Therefore, when n is greater than 512, all-to-all connections are limited to groups or subgroups of na cells, preferably na≦512, more preferably na≦256, even more preferably na≦128, and even more preferably na≦64, na≦32, na≦16, na=9, or na=4.

[0109] As mentioned above for the 128+128 two-layer NHHORNN, it is more effective to configure a network with two or more layers than to increase the number of cells in a single-layer network after a certain number of cells. It is also effective to limit the number of connections within the same network layer. Furthermore, it may be preferable to have more feedforward connections from the upstream layer to the downstream layer than feedback connections from the downstream layer to the upstream layer. Furthermore, the parameter ω 0i It may be advantageous to set the natural frequencies of the undamped oscillations of the damped harmonic oscillators of the cells of the downstream network layer so that the highest natural frequency of the cells of the upstream network layer is higher than the highest natural frequency of the cells of the downstream network layer and the lowest natural frequency of the cells of the upstream network layer is higher than the lowest natural frequency of the cells of the downstream network layer. Preferably, the natural frequency of the downstream network layer is set to a frequency that is 20% to 80% of the frequency of the upstream network layer, more preferably 20% to 50%, such as 20%, 30%, 40%, or 50%.

[0110] For example, the first network layer L1 has a number of n1 cells, at least one of which is connected to an input unit, and is therefore the most upstream layer in terms of data and processing result transfer. The second network layer L2 has a number of n2 cells, at least one of which is connected to at least one cell in the first layer, and is therefore a layer downstream from the upstream (first) layer in terms of data and processing result transfer. Of course, it is possible to add any number of layers, such as the third, fourth, fifth, sixth, ...xth (x≦1000) layers, each with n3, n4, n5, n6, ...nx cells.

[0111] In particular, when the data being processed has potentially strong correlations between adjacent data points and potentially weak correlations between distant data points, such as 2D image data, a potentially effective configuration for designing a two-layered network is to introduce many connections wi,j between small groups of DHOs in the first (upstream) network layer, making all-to-all connections within the small groups of DHOs in the first (upstream) network layer and introducing fewer connections wi,j between these small groups, resulting in 25% to 75% of the potential connections between cells in different small groups. The second (downstream) layer may have the same number of cells as the first layer, or it may have a smaller number of cells. Preferably, the number of feedforward connections from cells in the first (upstream) layer L1 to cells in the second (downstream) layer L2 is higher than the number / percentage of feedback connections from cells in the second (downstream) layer L2 to cells in the first (upstream) layer L1, for example, by 25%, 50%, 75%, 100%, 125%, or 150%. Similarly, if a third, more downstream layer is added, etc.

[0112] An example of such a two-tier network is shown in FIGS.

[0113] In Figure 8(a), a number of n1 = 100 cells are shown arranged in a 10 × 10 matrix CB. The cells can be indexed by row k and column l, i.e., k,l = 1, 2, …, 5. The matrix CB is shown only to illustrate the cell configuration and connections and does not imply that a hardware implementation requires the physical arrangement of the cells in the matrix CB. A 5 × 5 chessboard pattern CBG1 groups the 100 cells into 25 first groups G1 of four. In Figure 8(b), four cells in each of the 25 first groups G1 are marked with crosses alternating rows and columns for each second group, simply to facilitate identification of the cells in the first group. The recurrent connection unit consists of connections for all-to-all connections of all cells in each of these 25 first groups G1.

[0114] Figure 8c) shows a 5x5 chessboard pattern CBG2 that has been shifted by one row and one column, or 50% of the cells in one first group, relative to CBG1. This shift is used to create 25 second groups G2 of four cells each, as follows: 16 new second groups G2 of four cells each, each of which was part of a different group in the first group G1. At the edges of the matrix CB, there are only two cells in each "group" created by shifting the chessboard pattern, and at the four corners of the matrix, there are only one cell in each "group" created by shifting the chessboard pattern. To organize the 100 cells into 25 secondary groups G2, each containing four cells, created by shifting the chessboard pattern, the eight secondary groups G2 on the shifted edge of the matrix are "filled" with the two remaining cells from the first group G1 on the diagonally opposite side of the matrix that are not included in any of the 16 secondary groups G2 already completed with four cells. The secondary group G2 at the corner of the shifted matrix, which contains only one cell (10,10), is then filled with three cells (1,1, 1,10, 10,1) from the other three corners of the matrix. The corresponding secondary grouping is shown in Figure 8(d). The recurrent connection unit configures the all-to-all connectivity of each cell in these 25 secondary groups G2.

[0115] This configuration provides more connections wi,j between cells / DHOs of the first group G1 in the first (upstream) network layer L1, such as all-to-all connectivity within the 25 first groups G1 and fewer connections wi,j between the first groups G1 of cells / DHOs in the first (upstream) network layer L1, specifically 25% to each of the "adjacent" first groups G1 according to the second grouping, and all-to-all connectivity within the second group G2, each of which includes 4 (=25%) cells from four different first groups G1.

[0116] Of course, the above-described configuration of the first network layer L1 is not limited to a grouping of 10 × 10 = 100 cells into 5 × 5 = 25 groups. The number of cells per group g1 can be any square of an integer divisible by 2 ≤ 256, i.e., 4, 16, 36, 64, ..., 256. The grouping can also be any (u × v) matrix with u, v = 2, 3, 4, 5, 6, ..., where the product (g1 uv) ≤ nmax is 102400, for example. While this discussion focuses only on square matrix organizations (u = v), the principles apply equally well to rectangular organizations (u ≠ v). The g1 ≤ 256 restriction was specifically chosen to limit the connection complexity for all-to-all connections. By "chessboard pattern shifting" and subsequent allocation to incomplete groups via diagonal allocation and grouping of the four corners, the above configuration operates to complete the second grouping for all configurations of groups, achieving the intended diversity of connections within the first (upstream) layer.

[0117] The two-layer HORNN of Figure 9 is composed of a first (upstream) layer L1 as described with reference to Figure 8 and a second (downstream) layer L2 having a number of cells, n, where 8≦n2. In the example of Figure 9, the second (downstream) layer L2 is composed of the same number of cells, n2=n1, as the first (upstream) layer, but this is merely a non-limiting example to explain the principle.

[0118] As with all multi-layer RNNs of the present teachings, it is generally possible to implement each layer as either an HHORNN or an NHHORNN. That is, the DHOs within a layer of a multi-layer network may be implemented with the same intrinsic parameters β and ω as an HHORNN layer, or with potentially different parameters β as an NHHORNN layer. i and at least some different parameters ω 0i This fact is not reiterated for each example of a multi-layer RNN of the present teachings.

[0119] The recurrent connection unit RCU is connected in the illustrated example to each input / output node IO of each of the cells nci of each first group G1 of the first network layer L1. i The damped harmonic oscillation h is output as a result from the input / output node of the corresponding cell nci in the first network layer to the input / output node of the corresponding cell ncj in the second network layer, which is one feedforward connection wi,j between i the input / output nodes IO of the cells nci of each first group G1 of the first network layer L1 for transmitting i and at least one input / output node IO of the cell ncj of the second network layer L2. j At least one feedforward connection between wi,j and the damped harmonic oscillation h output from the input / output node of the corresponding cell nci of the first network layer. i to the input / output nodes IO of the corresponding cells ncj of the second network layer (in the illustrated example, the input / output nodes IO of each cell nci of each first group G1 of the first network layer L1). i (This is natural because one cell of the corresponding second group G2 is also connected to the first network layer), and one input / output node IO of each cell nci of the second group G2 of the first network layer. i and at least one input / output node IO of the cell ncj of the second network layer L2 j and at least one feedforward connection wi,j between the input and output nodes of the corresponding cell ncj of the second network layer L2 and the resulting damped harmonic oscillation h i the input / output node IO of the corresponding number of cells ncj of the second network layer L2 for transmitting j to the input / output node IO of one cell nci among the cells nci of each first group G1 of the first network layer i and at least one feedback connection wi,j to

[0120] Parameter ω 0i= The natural frequencies of the undamped oscillations of the n2 damped harmonic oscillators of the second network layer are set so that the highest natural frequency of the n1 cells of the first network layer is higher than the highest natural frequency of the n2 cells of the second network layer, and optionally the lowest natural frequency of the n1 cells of the first network layer is higher than the lowest natural frequency of the n2 cells of the second network layer. This setting results in the second network layer acting as a kind of low-pass filter for the signals / oscillations transmitted from the first network layer, suppressing potentially irrelevant high-frequency components and improving the performance of the corresponding HHORNN / NHHORNN.

[0121] The arrangement of subgroups of cells with all-to-all connectivity in each layer is not limited to the chessboard pattern described above. For example, the arrangement of the well-known King's graph, which is inferred from the possible moves of a king in a chess game, can also be used. Such an arrangement is exemplarily shown in FIG. 10 as a third embodiment. In this example, cells are arranged in a board-like arrangement consisting of virtual (or real) rows and columns. One cell, for example, cell 2,2 (=king) in this example, is connected to eight "neighboring cells" as shown in FIG. 10(a), namely, one cell (cell 1,2) in the same column of the upper row and its two direct neighboring cells (cells 1,1 and 1,3), one cell (cell 3,2) in the same column of the lower row and its two direct neighboring cells (cells 3,1 and 3,3), and all direct neighboring cells in the same row (cells 2,1 and 2,3). Figure 10b) shows a graph layout of cell 3,3 as the "king" with all-to-all connectivity to the eight adjacent cells 2,2, 2,3, 2,4, 3,2, 3,4, 4,2, 4,3, and 4,4.

[0122] Networks of the types shown in Figures 8-10, as well as other modified networks, can be implemented with a layer structure of 2, 3, 4, 5, 6, 7, 8, ..., x layers (x ≤ 1000). With more than two layers, layer-skipping connectivity / connections are possible. Particularly useful are layer-skipping feedback connections from lower layers to higher layers, skipping one or more intermediate upstream layers. For example, in a four-layer network, a connection from the fourth (lowest) layer to the second or first layer skips the third layer or both the third and second layers. Such skipping feedback connections provide intermediate cognitive "results" and help establish a structure in which intermediate cognitive results can trigger activation of specific regions within the network. Such layer-skipping feedback connections enhance the potential for resonance, entrainment, synchronization, phase shifts, and desynchronization. A schematic example of a network including layer-skipping feedback connections is shown in Figure 11(a). There are three layers: L1, L2, and L3. The layers may be the same or different types of layers (HHORNN or NHHORNN) and / or may have the same or different connectivity within different layers. For example, the most upstream layer L1 may be implemented as an HHORNN with all-to-all connectivity in the King's graph arrangement of FIG. 10 within the first layer L1, the second layer L2 may be implemented as an NHHORNN with all-to-all connectivity in the chessboard arrangement of FIG. 8 within the second layer L2, and the most downstream (third layer) layer L3 may be implemented as an NHHORNN with all-to-all connectivity in the King's graph arrangement of FIG. 10 within separate groups of 100 cells each within the third layer L3. There are feedforward connections (wi,j L1-L2 and wi,j L2-L3) and feedback connections (wj,i L3-L2 and wj,i L2-L1) between adjacent layers, increasing the connections between selected regions of cells in the first layer L1 and selected regions of cells in the second layer L2, and between selected regions of cells in the second layer L2 and selected regions of cells in the third layer L3, as shown, for example, in Fig. 11a).Also, there exists layer-skipping feedback connectivity between the third layer L3 and the first layer L1 as shown in FIG. 11a) by the connection wj,i L3-L1.

[0123] As mentioned above, restricting all-to-all connectivity to a subgroup of cells allows for a large increase in the number of cells per layer without exploding the complexity of the software and hardware connections. Networks of 100,000 cells or more are possible at the highest layer, and even 10,000 cells per layer is acceptable if the complexity of the task requires such expansion. Figure 11(b) shows a schematic representation of the three-layer network shown in Figure 11(a), with 4,000 cells in the first layer (L1), 2,000 cells in the second layer (L2), and 1,000 cells in the third layer (L3).

[0124] When the information to be processed is of a geometric nature, such as image data, or more generally represents spatially invariant statistics, it is advantageous to use an HHORNN as the first input layer L1, since the first layer L1 of the network consequently processes the input in a spatially invariant way.

[0125] Below we describe how to implement NHHORNN in semiconductor hardware to set connection values and potential time delays after being trained with the BPTT technique.

[0126] Assume that 1296 oscillators DHOi, number n, are arranged in a 36x36 matrix on a semiconductor chip using conventional CMOS technology. Each oscillator DHOi is implemented, for example, as a CMOS ring oscillator. The oscillators DHOi are connected all-to-all in a King's graph arrangement, as described above in FIG. 8. The connection weights are set by multi-stage transmission gate coupling blocks providing the connections wi,j between oscillators DHOi and DHOj. Such multi-stage transmission gate coupling blocks can be implemented as shown in FIG. 2 of PA2. The coupling values of such multi-stage transmission gate coupling blocks are set by values stored in memory implemented as SRAM if the coupling weights are to be reprogrammable, or as ROM if not. Corresponding exemplary arrangements using SRAM are shown in FIGS. 2 and 3 of PA2. Of course, this reference to PA2 is a non-limiting example; a memory block controlling the amplification of a CMOS-implemented amplifier could be used for continuous setting of the connection values. The possibility of reversing the sign of the connection value is determined by whether the connection wi,j is implemented with a single switched inverter or not.

[0127] Connection delays can be implemented by selecting / setting the number of tiered inverters controlled by the corresponding SRAM or ROM memory blocks. If such delays are provided by tiered inverters, they can be used simultaneously to set the potential weight inversions.

[0128] In the above case, the nonlinear saturated transfer function input element ICS i can be implemented as an input element in CMOS, as shown in US2018 / 0307977 A1, and the adder ICA implemented in CMOS i is each input element ICS i It is clear that the

[0129] Such a CMOS ASIC implementation can significantly reduce the power consumption of the trained HHORNN / NHHORNN.

[0130] The training of this CMOS NHHORNN is implemented using the well-known BPTT, and the connection unit values are obtained in this way and stored in the memory blocks either as rewritable values if rewritable memory such as SRAM is used, or using ROM (or fusible fuses, etc.) if the final NHHORNN product is not reprogrammable.

[0131] The above description of the HHORNN and NHHORNN using Equations (9) or (10) provides tools for implementing the DHO, their connections, and inputs and outputs. In other words, it provides tools for implementing the HHORNN / NHHORNN in software and running it on a correspondingly programmed data processing device, such as a general-purpose computer. That is, the method for calculating oscillations performed by the DHO is as follows: (i) a description of the corresponding RNN is created using Equation (9) or (10) to define output data O(t) in response to input data S(t); (ii) input data S(t) representing the data to be processed is input to the description, and the resulting oscillations caused by the input data S(t) are calculated; the resulting output data O(t) is calculated according to the description and the calculated oscillations; and (iii) the processing results are output based on the output data obtained from the input data. Equation (9) or (10) captures the initial setting of the RNN if no training has been performed. In this case, the description can be used for training. If the training results are used to create a description of the RNN, the created description represents a trained RNN. Thus, it represents a data processing tool that provides a processing result that recognizes the object / event represented by the corresponding input data S(t). Thus, a computer program comprising program code that implements the above-described method when executed on a corresponding programmed computer is one possible implementation of the HHORNN / NHHORNN of the present teachings.

[0132] The output data often provides a processing result before all the input data S(t) has been input. For example, a 16-cell NHHORNN with all-to-all connectivity including self-connections trained on the MNIST data was always confident in identifying the correct digit when inputting less than 50% of the total input data S(t). Figure 12 shows the top 10th digit of all images with a line representing the input data that had already been input when the RNN was confident in its result.

[0133] HHORNN and NHHORNN are also highly robust to noise in the input data S(t), outperforming state-of-the-art RNNs in this respect as well.

[0134] The recurrent neural network and data processing method and apparatus of the present invention can be used, for example, for a variety of data processing tasks that were difficult to accomplish using conventional techniques.

[0135] A first example is image segmentation and classification, such as detecting objects in a digital image or segmenting an image into a collection of objects contained in the image. In the current state of the art, this task is often implemented using feedforward neural networks, i.e., networks with one input layer, hidden layers, and an output layer (deep neural networks). Examples include multilayer perceptrons, convolutional neural networks, and transformer networks. Using hidden layers, the network converts input pixel values into a representation suitable for solving classification / detection / segmentation tasks, such as detecting traffic signs in autonomous vehicles. The use of recurrent neural networks / data processing devices of the present teachings enables a fundamentally different and novel approach. HORNNs (both NHHORNN and HHORNN types) are inherently recurrent and receive input as a time series. The network is implemented with multiple layers. The first layer is a geometrically constructed HHORNN, followed by geometrically constructed or all-to-all HHORNN or NHHORNN layers, or a combination thereof. "Geometrically structured" refers to the connectivity within a layer, where a cell has a limited number of connections to other cells that are considered next-neighbors, thus referencing a geometric pattern such as the king graph or chessboard pattern mentioned above, or other corresponding next-neighbor connectivity patterns. To present an image to the network, an n × m matrix of pixel values (each pixel can have an intensity value between v_min and v_max for each of the different color channels R, G, and B) is converted into a length-N time series by sweeping the value domain [v_min, v_max] from top to bottom (or bottom to top) in discrete steps v_1 = v_max,…, v_N = v_min. This results in a length-N time series where each time point is represented by an n × m matrix M_i (1 ≤ i ≤ N), where entries of M_i are set to 1 when the corresponding pixel value reaches a threshold v_i and 0 otherwise. In other words, the topography of n × m pixel values is sampled into N n × m matrices.The cells in the first layer of the network are stimulated according to the values of the time series.

[0136] Given some processing time, the network processes the input data. Multiple layers can form more abstract representations of the data, organized geometrically (respecting the geometric organization or all-to-all of the input data). After some time, the state (answer) of the network is read out. The network can develop internal measures to "know" that it has found a good answer for the given input, such as measuring the entropy of its dynamic state.

[0137] The processing described in the first embodiment is not limited to image data having inherent spatial relationships, but can also be applied to other spatially organized data (such as sensor data obtained from spatially distributed sensors).

[0138] A second example is keyword spotting. Keyword spotting deals with identifying keywords in speech, for example detecting the presence of wake words ("Hey Siri," "Okay Google") in a constant stream of recorded audio in order to put a smart device (smartwatch, smartphone, smart speaker) into a mode where it can accept commands. In the current state of the art, this task is often implemented in the following way: microphone data (sound pressure levels) are recoded, digitized, and chunked, MFCC extraction (described at https: / / de.wikipedia.org / wiki / Mel_Frequency_Cepstral_Coefficients) is performed, and a mapping to feature vectors is performed. The sequence of feature vectors is then fed to a detection / classification unit (e.g., a recurrent neural network) to detect whether keywords have been detected in the recorded audio. An example of the procedure is shown below. https: / / github.com / MycroftAI / mycroft-precise#how-it-works The software implementation is shown below. https: / / github.com / Picovoice / porcupine, https: / / github.com / MycroftAI / mycroft-precise. The recurrent neural network / data processing device of the present teachings allows a new and fundamentally different approach: raw microphone data (sound pressure levels) can be directly input into the corresponding trained NHHORNN.

[0139] A third example is speech recognition. In the current state of the art, this task is often implemented similarly to keyword spotting: microphone data (sound pressure levels) are recoded, digitized, and chunked; MFCC extraction (described at https: / / de.wikipedia.org / wiki / Mel_Frequency_Cepstral_Coefficients) is performed; a mapping to feature vectors is performed; and the sequence of feature vectors is fed to a detection / classification unit (recurrent neural network, SVM, etc.) to detect word sequences. Information from a language model is used to output likely word sequences (e.g., sentences). The recurrent neural network / data processing device of the present teachings allows for a fundamentally different and novel approach. Raw microphone data (sound pressure levels) are fed to a multi-layer NHHORNN (trained accordingly) without any preprocessing. The nature of the NHHORNN provides a latent tonotopic organization in the first layer. That is, cells encoding similar frequencies are likely to be connected during training, leading to a "geometric" organization in frequency space. Thus, over several layers, the network can form abstract representations of the input data, writing out, for example, syllables, words, or sentences.

[0140] A fourth example is anomaly detection, i.e., the identification of rare events in time series data that deviate from expected or normal behavior (as defined by the majority of the data). In modern systems, this task is implemented using a variety of approaches, ranging from simple statistical tests to neural network-based techniques. https: / / en.wikipedia.org / wiki / Anomaly_detection The recurrent neural network / data processing device of the present teachings enables a fundamentally different new approach. Time series data can be fed directly into a potentially multi-layered (and correspondingly trained) HHORNN / NHHORNN, value by value. Examples include simple tasks such as ECG, EEG, MEG, or other time series data from living organisms, machines, or processes. HHORNN / NHHORNN does not require chunking of data.

[0141] A fifth example is closed-loop or feedback control. A closed-loop or feedback control system aims to maintain a predetermined relationship between two sets of system variables by comparing a function of the two variables and using the difference as a control measure (https: / / en.wikipedia.org / wiki / Control_theory). Examples of applications include controlling chemical reactions (valve control) and directional control of moving objects. The recurrent neural network / data processing device of the present teachings enables a fundamentally different new approach. Time series data representing measured variables can be directly fed into a potentially multi-layered HHORNN / NHHORNN trained to predict the dependent variable.

[0142] A sixth example is time series forecasting and predictive analytics. Given the past values of a time series (e.g. sensor data), the task is to predict the evolution of that data in the future (https: / / en.wikipedia.org / wiki / Predictive_analytics). The recurrent neural network / data processing device of the present teachings allows a new and fundamentally different approach: it becomes possible to feed the time series data value by value directly into a (correspondingly trained) HHORNN / NHHORNN (potentially with multiple layers).

[0143] The above teachings can be implemented as described in the following aspects of the invention.

[0144] Feature 1. A recurrent neural network for processing time series input data (S(t)), A plurality of n damped harmonic oscillators (DHOi, i=1 to n), n≧8, each of which oscillates according to the general second-order differential equation of a damped harmonic oscillator, Equation (1), as follows: β i = damping coefficient, ω 0i = the natural frequency of the undamped vibration, e.g., β for the damped translational mechanical harmonic oscillator DHOi i =c i / 2m i and ω 0i 2 =k i / m i , m i = mass, k i = spring constant, c i = viscous damping constant, or for other types of damped harmonic oscillators, β if it corresponds to a damped electrical RLC harmonic oscillator DHOi. i =R i / L i , ω 0i 2 =1 / L i C i , R i = resistance, L i = inductance, C i= capacitance, and each of the n damped harmonic oscillators is one cell (nci) of a neural network; Each cell (nci) has a cell input (x), such as the translational input of a corresponding damped translational mechanical harmonic oscillator or the electrical input of a corresponding damped electrical RLC harmonic oscillator. i ) and the resulting damped harmonic oscillations (h i ) to output the input / output node (IO i ) and Input / Output Node (IO i )teeth, Any input to the cell can be connected via a nonlinear, e.g., sigmoidal or tanh-type transfer function with an arbitrary offset, preferably a saturated transfer function, e.g., via a viscoelastic input element for a corresponding damped translational mechanical harmonic oscillator, or a transistor-implemented sigmoidal input for a corresponding damped electrical RLC harmonic oscillator DHOi (ICS i ) through the cell input (x i ) and output this as an input connection (IC i )and, The resulting damped harmonic oscillations (h i ) output connection (OC i )and, Cell (c i ) for each cell (nci), the input / output node (IO i ) and at least one other cell's (ncj) input / output node (IO j ) of the corresponding cell (nci) i ) damped harmonic oscillations (h i ) to another cell (c j ) input / output node (IO j a recurrent connection unit (RCU) having connections (wi, j) for transmitting to the recurrent neural network.

[0145] Feature 2: The input / output node (IOi ) and the input / output node (IO j ) at least one connection (wi,j) between i,j =w i,j ×h i , w i,j ∈R, |w i,j |≦10, and this connection (wi,j) is, for example, the output of the corresponding damped translational mechanical harmonic oscillator (h i ) or the output of a corresponding damped electrical RLC harmonic oscillator (h i 2. The neural network of claim 1, wherein the first input signal is an electrical voltage divider or transmission gate coupling for transmitting a second input signal.

[0146] Feature 3: The input / output node (IO i ) and the input / output node (IO j ) at least one connection (wi,j) between i,j , and the output time delay ∂t i,j =k i,j ×Δt, k i,j =0,1,…,k, 0≦k≦kmax, ti1 / 10≦Δt≦ti1, k i,j ×Δt≦50ti1, where ti1 is the time interval of the time series input data (S(t)), which represents the time interval between subsequent discrete values in the case of discrete time series input data (S(t)), and the time interval between subsequent sampled values in the case of continuous time series input data (S(t)). This output time delay is, for example, the output (h i ) or the output of a corresponding damped electrical RLC harmonic oscillator (h i 3. The neural network of claim 1, wherein the gate may be an electrically inductive or clock output gate for transmitting a signal.

[0147] Feature 4. Recurrent Connection Unit (RCU) is a cell (c i) for at least one of the input / output nodes IO of the corresponding damped harmonic oscillator DHOi i , preferably 5%, more preferably 10%, more preferably 25%, more preferably 50%, more preferably 75%, more preferably 100% of the cells are configured with connections providing self-connectivity.

[0148] Feature 5. Among the n cells, a plurality of n1 cells, where 8≦n1 and n1≦n, are arranged in one (first) network layer (L1), and a recurrent connection unit (RCU) connects the n1 cells (c i ) of the corresponding cell (nci) of one (first) network layer (L1), i ) and an input / output node (IO j ) of the corresponding cell (nci) i ) damped harmonic oscillations (h i ) to one other cell (c j ) input / output node (IO j ) for transmitting to the neural network of any preceding aspect.

[0149] Feature 6. Among the n cells, a plurality of n2 cells, where n2≦n2, preferably n2≦n1, are arranged in a second (downstream) network layer (L2), and a recurrent connection unit (RCU) connects the n2 cells (c i ) for each of the input / output nodes (IO i ) and an input / output node (IO) of at least one other cell (ncj) of the second network layer (L2). j ) of the corresponding cell (nci) i) damped harmonic oscillations (h i ) to one other cell (c j ) input / output node (IO j ), and Recurrent Connection Unit (RCU) The input / output nodes (IO) of n1 cells (nci) in one (first) network layer i ) and the input / output nodes (IO j ) and the damped harmonic oscillations (h i ) to the input / output nodes (IO j ), which establishes a minimum of 10% potential feedforward connectivity and a maximum of 100% potential feedforward connectivity; The input / output nodes (IO j ) and one (first) network layer of n1 cells (nci) with input / output nodes (IO i ) and the damped harmonic oscillations (h j ) to the input / output nodes (IO i ), with feedback connections (wj,i) that establish a minimum of 10% potential feedback connectivity and a maximum of 100% potential feedback connectivity; Optionally, a plurality of nr cells among the n cells, where r=3 or 4 or 5 or 6, and 8≦nr, preferably nr≦n(r−1), are arranged in an r-th network layer (Lr), and a recurrent connection unit (RCU) is configured to connect the nr cells (c i) of the corresponding cell (nci) in the rth network layer (Lr), i ) and the input / output node (IO j ) of the corresponding cell (nci) i ) damped harmonic oscillations (h i ) to one other cell (c j ) input / output node (IO j ), and Recurrent Connection Unit (RCU) The input / output nodes (IO i ) and the input / output nodes (IO j ) and the damped harmonic oscillation (h i ) to the input / output nodes of the corresponding cells of the nr cells (ncj) of the nr-th network layer, and a feedforward connection (wi,j) that establishes a minimum potential feedforward connectivity of 10% and a maximum potential feedforward connectivity of 100%; The input / output nodes (IO j ) and the input / output nodes (IO i ) and the damped harmonic oscillations (h j ) of the input / output nodes (IO i) and establishing a minimum of 10% potential feedback connectivity and a maximum of 100% potential feedback connectivity.

[0150] Feature 7. The recurrent connection unit (RCU) has potential connections (wi, j) for multiple nr cells located in the same r-th network layer, where r = 1, 2, ..., 6 and 8 ≤ nr, and nr ≤ n. The damped harmonic oscillation (h i ) into n1 cells (c j ), each cell has all-to-all connectivity to at least 8 and at most 512 other cells (ncj) of the same r-th network layer (nr cells nci) for transmission to the corresponding input / output nodes of another cell of the same r-th network layer, or All-to-all connectivity in King's graph layout, or If cells are arranged in a ((g1 / 2u)×(g1 / 2v)) matrix CB with g1=4, 16, 36, 64, 100, 144, 256, and u, v=2, 3, 4, 5, 6, …, and (g1 uv)≦102400, then for all-to-all connectivity for a first group (G1) of g1 cells, these first groups (G1) are arranged in a first chessboard arrangement in the (u × v) matrix CBG1, and a second group (G2) of g1 cells have all-to-all connectivity, these second groups (G2) are arranged in a (u × v) matrix CBG2 shifted by g1 / 2 cells in both the row and column directions relative to the first chessboard arrangement CBG1. 2. The neural network of any preceding feature, wherein the second group (G2) at the edge of the matrix CB in the shift direction is completed by cells located at the edge of the matrix CB at diagonally opposite positions of the matrix CB that are not covered by the second (shifted) chessboard arrangement CBG2, and the second group (G2) located at the corners in the shift direction is completed by cells at three corners of the matrix CB that are not covered by the second (shifted) chessboard arrangement CBG2.

[0151] Feature 8. The recurrent connection unit (RCU) is a cell (c i ) for at least one of the cells (c i 8. The neural network of claim 6 or 7, further comprising a latent layer-skipping feedback connection (wi,j) to at least one of the inputs (wi,j), preferably having a latent layer-skipping feedback connectivity of at least 10%, more preferably at least 20%.

[0152] Feature 9: The plurality of n damped harmonic oscillators (DHOi, i = 1 to n) are composed of at least two different types of damped harmonic oscillators, and each of the at least two different types of damped harmonic oscillators has at least a parameter ω 0i 3. The neural network of any preceding feature, wherein the natural frequency of the undamped oscillation is different from other types of damped harmonic oscillators.

[0153] Feature 10. The parameter ω of the damped harmonic oscillator of nr cells in the nr-th network layer, where r=2, 3, …, 6, is 0i = The natural frequencies of the undamped oscillations are set so that the highest natural frequency of the n(r-1) cells in the n(r-1)th network layer is higher than the highest natural frequency of the nr cells in the nrth network layer, and the lowest natural frequency of the n(r-1) cells in the n(r-1)th network layer is higher than the lowest natural frequency of the nr cells in the nrth network layer.

[0154] Feature 11: At least three different types of damped harmonic oscillators are provided, and a parameter ω is a natural frequency of undamped oscillation of the at least three different types of damped harmonic oscillators. 0i wherein, based on determining power spectral densities of a plurality of samples of the time series input data (S(t)) to be processed and variances of these power spectral densities, a natural frequency is set to be distributed at each peak of this variance, wherein one of the at least three natural frequencies is set to correspond to the peak of the variance having the lowest frequency, one of the at least three natural frequencies is set to correspond to the peak of the variance having the highest amplitude, and the remaining natural frequencies are preferably set to frequencies substantially uniformly distributed between the frequencies of these two peaks of the variance, or are set to a normal distribution with a selected mean and variance.

[0155] Feature 12. Input connection of input / output node of at least one cell (nci) (ICi ) and at least one cell (c i ) input connection of the input / output node (IC i an input unit (IU) adapted to receive and input time series input data (S(t)) having a length TI1 for inputting the input data (S(t)) to the Output connection (OC) of the I / O node of at least one cell (nci) i ) and adapted to output output data (O(t)) having an output start time (OST) configured to be a predetermined time interval after receiving an input start of the input data (S(t)).

[0156] Feature 13. The input and output nodes of each cell (nci) have damped harmonic oscillations (h i ) at discrete time step t d =t0+dτ (τ = discretized time value), ti1 / 20≦τ≦ti1 / 2, 2≦d≦20, where ti1 is a time series interval of the time series input data (S(t)), representing the time interval between subsequent discrete values in the case of discrete time series input data (S(t)), and representing the time interval between subsequent sampled values in the case of continuous time series input data (S(t)).

[0157] It is expressly stated that all features disclosed in the specification and / or aspects and / or claims are intended to be disclosed separately and independently from one another, not only for purposes of the inherent disclosure, but also for purposes of limiting the claimed invention, independently of the configuration of features in the embodiments and / or claims. The exemplification of all ranges or groups of entities is expressly stated to disclose all possible intermediate values or intermediate entities, in particular as limits of the ranges, not only for purposes of the inherent disclosure, but also for purposes of limiting the claimed invention.

Claims

1. A neural network device including a neural network for processing time series input data (S(t)), A plurality of n damped harmonic oscillators (DHOi, i=1 to n), n≧8, each oscillating according to the general second-order differential equation for damped harmonic oscillators: [Equation 1] In the case of a damped electrical RLC harmonic oscillator DHOi, β ｉ =R ｉ / L ｉ , ω ０ｉ ２ = 1 / L ｉ C ｉ , R ｉ = resistance, L ｉ = inductance, C ｉ = capacitance and the corresponding β for other types of damped harmonic oscillators ｉ , ω ０ｉ ２ each of the n damped harmonic oscillators being one cell (nci) of the neural network; Each cell (nci) has a cell input (x) similar to the electrical input of the corresponding damped harmonic oscillator. ｉ ) and the resulting damped harmonic oscillations (h ｉ ) to output the input / output node (IO ｉ ) The input / output node (IO ｉ )teeth, Any input to the cell may be coupled via a saturated nonlinear, preferably sigmoidal or more preferably tanh-type transfer function with an arbitrary offset, e.g., a transistor-implemented sigmoidal or tanh-type input for a corresponding damped harmonic oscillator DHOi (ICS ｉ ) through the cell input (x ｉ ) and adapted to output it. ｉ )and, The resulting damped harmonic oscillations (h ｉ ) and an output connection (OC ｉ )and, Cell (c ｉ ) for each input / output node (IO) of the corresponding cell (nci). ｉ ) and an input / output node (IO) of at least one other cell (ncj). ｊ ) of the corresponding cell (nci), ｉ ) damped harmonic oscillations (h ｉ ) into one other cell (c ｊ ) input / output node (IO ｊ a recurrent connection unit (RCU) having a connection (wi,j) for transmitting to i) one or more damped harmonic oscillators (DHOi, i=1 to n) of said plurality of n damped harmonic oscillators are corresponding damped electrical RLC harmonic oscillators DHOi implemented with a voltage controlled oscillator as a CMOS cross-coupled differential oscillator or a CMOS ring oscillator, each of the n damped harmonic oscillators being one cell (nci) of said neural network; and / or ii) one or more damped harmonic oscillators (DHOi, i=1 to n) of the plurality of n damped harmonic oscillators are corresponding damped electrical harmonic oscillators DHOi implemented with analog elements including an integrator and a potentiometer as a voltage-controlled oscillator, and each of the n damped harmonic oscillators is one cell (nci) of the neural network.

2. Each cell (nci) is a CMOS inverter stage circuit (ICS ｉ ) as the electrical input of the corresponding damped harmonic oscillator via a sigmoid transfer function ｉ ) to input / output node (IO ｉ 2. The neural network device of claim 1, further comprising:

3. Each cell (nci) contains an adder (ICA) that adds all the inputs to the cell. ｉ ) and receives all inputs to the cell via the cell input (x ｉ ) a CMOS inverter stage circuit (ICS ｉ an input / output node (IO) for outputting the summed input as an input to a saturated nonlinear transfer function implemented by ｉ 3. The neural network device according to claim 1, further comprising:

4. The input / output node (IO) of the corresponding cell (nci) of the recurrent connection unit (RCU) ｉ ) and the input / output node (IO) of one other cell (ncj). ｊ ) has a transmission characteristic T ｉ，ｊ =w ｉ，ｊ ×h ｉ , w ｉ，ｊ ∈R, |w ｉ，ｊ |≦10, and this connection (wi,j) is, for example, connected to the output (h ｉ 4. The neural network device according to claim 1, wherein the first and second inputs are connected to the first and second inputs via an electrical divider or transmission gate for transmitting the first and second inputs.

5. The input / output node (IO) of the corresponding cell (nci) of the recurrent connection unit (RCU) ｉ ) and the input / output node (IO) of one other cell (ncj). ｊ ) has a transmission characteristic T ｉ，ｊ and the transmission is set to the output time delay ∂t ｉ，ｊ = k ｉ，ｊ ×Δt, k ｉ，ｊ =0,1,...,k, 0≦k≦kmax, ti1 / 10≦Δt≦ti1, k ｉ，ｊ ×Δt≦50ti1, where ti1 is the time series interval of the time series input data (S(t)), which represents the time interval between subsequent discrete values in the case of discrete time series input data (S(t)), or the time interval between subsequent sampled values in the case of continuous time series input data (S(t)), and this output time delay is, for example, the output (h) of a corresponding damped electrical RLC harmonic oscillator. ｉ 5. The neural network device according to claim 1, which can be an electrically inductive or clock output gate for transmitting a signal.

6. Each connection (wi,j) of the recurrent connection unit (RCU) is implemented as (2p+1) levels of transmission gates coupling a first chain of at least two series-connected NOT gates coupled to receive the output of an output-side input / output node, a second chain of at least two series-connected NOT gates coupled to the other input-side input / output node, and a block of 4p transmission gates coupled between the output of the NOT gates of the first chain and the input of the NOT gates of the second chain, where p≦10, and has a transmission characteristic T ｉ，ｊ =w ｉ，ｊ ×h ｉ The transmission weight of is w if no transmission gates are connected to the input and output of the NOT gate. ｉ，ｊ = 0, and 2w transmission gates are connected in phase offset between the output of the NOT gate in the first chain and the input of the NOT gate in the second chain. ｉ，ｊ = +w, and if 2w transmission gates are connected with a phase shift between the output of the NOT gate in the first chain and the input of the NOT gate in the second chain, then w ｉ，ｊ = -w, Optionally, the transmit output time delay ∂t ｉ，ｊ = k ｉ，ｊ ×Δt, k ｉ，ｊ =0,1,...,k,0≦k≦kmax, ti1 / 10≦Δt≦ti1,k ｉ，ｊ ×Δt≦50ti1, and this output time delay is implemented by serially connecting a first clocked NOT gate between the output of the output-side I / O node and the other input-side I / O node, either after or before the NOT gates of the first and second chains, and the first clocked gate has a delay of 1=(k ｉ，ｊ × Δt) / clock time period, Optionally, a weight setting block includes one setting device for each of the a(2p+1) levels of transmission gate couplings, the setting device being adapted to close or block the connection through each of the corresponding 4p transmission gates, either by permanent closing or blocking, for example by melting or etching the corresponding connection line, or by programmable closing or blocking via a CMOS transistor that is switched according to data stored in an SRAM or ROM memory block that stores the set weight; Optionally, the transmit output time delay ∂t ｉ，ｊ , the setting device is adapted to permanently close or interrupt the connection via the corresponding first clocked NOT gate, for example by melting or etching the corresponding connection line, or to set the transmit output time delay ∂t ｉ，ｊ 6. The neural network device of claim 1, adapted to close or block either by programmable closing or blocking via CMOS transistors that are switched according to data stored in an SRAM or ROM memory block that stores:

7. The recurrent connection unit (RCU) includes at least one cell (c ｉ ) to the input / output node IO of the corresponding damped harmonic oscillator DHOi. ｉ a recurrent connection unit (RCU) configured with connections that provide self-connectivity, wherein the recurrent connection unit (RCU) is configured with connections that provide self-connectivity for 5%, more preferably 10%, more preferably 25%, more preferably 50%, more preferably 75%, and more preferably 100% of the cells.

8. Among the n cells, a plurality of n1 cells, where 8≦n1 and n1≦n, are arranged in one (first) network layer (L1), and the recurrent connection unit (RCU) is configured to connect the n1 cells (c ｉ ) for each of the input / output nodes (IO) of the corresponding cell (nci) of said one (first) network layer (L1), ｉ ) and an input / output node (IO) of at least one other cell (ncj) of said one (first) network layer (L1). ｊ ) of the corresponding cell (nci), ｉ ) damped harmonic oscillations (h ｉ ) into another cell (c ｊ ) input / output node (IO ｊ 8. The neural network device according to claim 1, further comprising a connection (wi, j) for transmitting to the neural network.

9. Among the n cells, a plurality of n2 cells, where 8≦n2, preferably n2≦n1, are arranged in a second (downstream) network layer (L2), and the recurrent connection unit (RCU) is configured to connect the n2 cells (c ｉ ) for each of the input / output nodes (IO) of the corresponding cell (nci) of the second network layer (L2), ｉ ) and an input / output node (IO) of at least one other cell (ncj) of said second network layer (L2). ｊ ) of the corresponding cell (nci), ｉ ) damped harmonic oscillations (h ｉ ) into another cell (c ｊ ) input / output node (IO ｊ ), The recurrent connection unit (RCU) The input / output nodes (IO) of the n1 cells (nci) of the one (first) network layer ｉ ) and the input / output nodes (IO) of the n2 cells (ncj) of the second network layer ｊ ), and the damped harmonic oscillations (h) output from the input / output nodes of the corresponding cells of the n1 cells (nci) of the one (first) network layer. ｉ ) to the input / output nodes (IO) of the corresponding cells of the n2 cells (ncj) of the second network layer. ｊ ) that establishes a minimum of 10% potential feedforward connectivity and a maximum of 100% potential feedforward connectivity; The input / output nodes (IO) of the n2 cells (ncj) of the second network layer ｊ ) and the input / output nodes (IO) of the n1 cells (nci) of the one (first) network layer ｉ ) and the damped harmonic oscillations (h) output from the input / output nodes of the corresponding cells of the n2 cells (ncj) of the second network layer. ｊ ) to the input / output nodes (IO) of the corresponding cells of the n1 cells (nci) of the one (first) network layer. ｉ ), and a feedback connection (wj,i) that establishes a minimum of 10% potential feedback connectivity and a maximum of 100% potential feedback connectivity; Optionally, a number nr of the n cells, where r=3 or 4 or 5 or 6, and 8≦nr, preferably nr≦n(r−1), are arranged in the r-th network layer (Lr), and the recurrent connection unit (RCU) is configured to connect the nr cells (c ｉ ) of the corresponding cell (nci) in the r-th network layer (Lr), ｉ ) and an input / output node (IO) of at least one other cell (ncj) of the r-th network layer (Lr). ｊ ) of the corresponding cell (nci), ｉ ) damped harmonic oscillations (h ｉ ) into another cell (c ｊ ) input / output node (IO ｊ ), The recurrent connection unit (RCU) The input / output nodes (IO) of the n(r-1) cells (nci) in the n(r-1)th network layer ｉ ) and the input / output nodes (IO) of the nr cells (ncj) in the nr-th network layer. ｊ ) and the damped harmonic oscillation (h ｉ ) to the input / output nodes of the corresponding cells of the nr cells (ncj) of the nr-th network layer, establishing a minimum of 10% potential feedforward connectivity and a maximum of 100% potential feedforward connectivity; The input / output nodes (IO) of the nr cells (ncj) in the nr-th network layer ｊ ) and the input / output nodes (IO) of n(r-1) cells (nci) in the n(r-1)th network layer ｉ ) and the damped harmonic oscillations (h) output from the input / output nodes of the corresponding nr cells (ncj) in the nr-th network layer. ｊ ) to the input / output nodes (IO) of the corresponding cells of the n(r-1) number of cells (nci) in the n(r-1)th network layer. ｉ 9. The neural network device according to claim 1, further comprising: a feedback connection (wj,i) for transmitting to the neural network (wj,i) and establishing a minimum of 10% potential feedback connectivity and a maximum of 100% potential feedback connectivity.

10. The recurrent connection unit (RCU) has potential connections (wi, j) for a number of nr cells located at the same r-th network layer, where r=1, 2, ..., 6 and 8≦nr, and nr≦n, which is The damped harmonic oscillation (h) output from the input / output node of the corresponding cell (nci) ｉ ) into n1 cells (c ｊ ) to the input / output nodes of the corresponding other cells of the nr cells (nci), or All-to-all connectivity in King's graph layout, or If the cells are arranged in a ((g1 / 2u) x (g1 / 2v)) matrix CB with g1 = 4, 16, 36, 64, 100, 144, 256, and u, v = 2, 3, 4, 5, 6, ..., and (g1 u v) ≤ 102400, then for all-to-all connectivity for a first group (G1) of g1 cells, these first group (G1) are arranged in a first chessboard arrangement in the (u x v) matrix CBG1, and a second group (G2) of g1 cells have all-to-all connectivity, these second group (G2) are arranged in a second chessboard arrangement in the (u x v) matrix CBG2 shifted by g1 / 2 cells in both the row and column directions relative to the first chessboard arrangement CBG1.

10. A neural network device as claimed in any one of claims 1 to 9, wherein the second group (G2) at the edge of the matrix CB in the shift direction is complemented by cells arranged in a board arrangement, the cells being arranged at the edge of the matrix CB at opposite positions on the diagonal of the matrix CB that are not covered by the second (shifted) chessboard arrangement CBG2, and the second group (G2) located at the corners in the shift direction is complemented by cells at the three corners of the matrix CB that are not covered by the second (shifted) chessboard arrangement CBG2.

11. The recurrent connection unit (RCU) receives the r-th network layer (Lr) cell (c ｉ ) for at least one of the cells (c ｉ 11. The neural network device according to claim 9 or 10, further comprising an implicit layer-skipping feedback connection (w i,j ) to at least one of the neural network elements (w i,j ), preferably having an implicit layer-skipping feedback connectivity of at least 10%, more preferably at least 20%.

12. The plurality of n damped harmonic oscillators (DHOi, i=1 to n) are composed of at least two different types of damped harmonic oscillators, and each of the at least two different types of damped harmonic oscillators has at least a parameter ω ０ｉ = The neural network device according to any one of claims 1 to 11, wherein the natural frequency of the undamped oscillation is different from that of other types of damped harmonic oscillators.

13. The parameter ω of the damped harmonic oscillator of the nr cells of the nr-th network layer, where r=2, 3, . . . , 6 ０ｉ = The natural frequencies of the undamped oscillations are set so that the highest natural frequency of the n(r-1) cells in the n(r-1)th network layer is higher than the highest natural frequency of the nr cells in the nrth network layer, and the lowest natural frequency of the n(r-1) cells in the n(r-1)th network layer is higher than the lowest natural frequency of the nr cells in the nrth network layer.

14. The input connection (IC) of the input / output node of at least one cell (nci) ｉ ) and at least one cell (c ｉ ) input connection of the input / output node (IC ｉ an input unit (IU) adapted to receive and input time series input data (S(t)) having a length TI1 for inputting the input data (S(t)) to the input unit (IU); The output connection (OC) of the input / output node of at least one cell (nci) ｉ 14. The neural network apparatus according to claim 1, further comprising: an output unit (OU) connected to the input data (S(t)) and adapted to output output data (O(t)) having an output start time (OST) set to be a predetermined time interval after receiving an input start of the input data (S(t)).

15. A training method for a recurrent neural network device according to any one of claims 1 to 14, comprising: a) setting natural frequencies of said plurality of n damped harmonic oscillators (DHOi, i=1 to n), optionally setting the natural frequencies so as to obtain at least three different types of damped harmonic oscillators, each of the at least three different types of damped harmonic oscillators having at least a parameter ω ０ｉ = the natural frequency of the undamped oscillation is set to be different from the other two types of damped harmonic oscillators, and optionally based on the determination of the power spectral densities of a number of representative samples of the time series input data (S(t)) to be processed and the variance of these power spectral densities, said natural frequencies are distributed over the peaks of this variance, one of at least three natural frequencies being set to correspond to the peak of the variance with the lowest frequency and one of at least three natural frequencies being set to correspond to the peak of the variance with the highest amplitude; b) the initial weight w of the connection (wi, j) of the recurrent connection unit ｉ，ｊ ∈R and |w ｉ，ｊ |≦10, and optionally set an initial connection output time delay ∂t of connection (w i,j) of the recurrent connection unit. ｉ，ｊ setting c) performing a training sequence by processing a plurality of representative samples of the time series input data (S(t)) to be processed; d) evaluating the results of the learning sequence of step c) against expected results; e) Based on the results of the learning sequence in step c), a backpropagation time (BPTT) technique is performed, and based on the BPTT results, the weights w of the connections (wi,j) of the recurrent connection unit are calculated. ｉ，ｊ and optionally the output time delay ∂t of connection (w i,j) of said recurrent connection unit ｉ，ｊ and f) Any of the following: f1) the evaluation of step d) results in a result that conforms to said expected result; or f2) Steps c) to e) are repeated a predetermined maximum number of times without reaching step f1); repeating steps c) to e) until g) modifying the weights and optionally the connection delays of the recurrent connection units according to the weights and connection delays that satisfy the result of step f1) or terminating the learning.

Images