Legendre memory unit in recurrent neural networks

Abstract

A neural network architecture is trained with connection weights determined using Legendre memory unit equations, while optionally keeping the determined weights fixed. The network can use spike or non-spike activation functions, can be stacked or recurrently coupled with other neural network architectures, and can be implemented in software and hardware. Embodiments of the invention provide systems for pattern classification, data representation, and signal processing that compute using orthogonal polynomial basis functions across a sliding time window.

Description

Technical Field

[0001] This invention relates generally to artificial intelligence and deep learning, and more particularly to recurrent neural network architectures that can be implemented in software and hardware. This application claims priority to Provisional Application No. 62 / 814,767, filed March 6, 2019, and Provisional Application No. 62 / 844,090, filed May 6, 2019, the contents of which are incorporated herein by reference. Background Technology

[0002] Deep learning has indeed brought many rapid and profound advances to the field of artificial intelligence. Due to its black-box nature, achieving state-of-the-art performance on a wide range of important problems requires neither domain expertise nor an understanding of the inner workings of neural networks, including image recognition, speech recognition, natural language understanding, question answering, and language translation (see Y. LeCun, Y. Bengio, and G. Hinton, Deep learning. Nature, Vol. 521, No. 7553, pp. 436–444, May 2015). The basic formula is as follows: install a software library for deep learning, choose a network architecture, set its hyperparameters, and then train it using as much data as the hardware (e.g., a graphics processing unit).

[0003] Deep learning architectures, such as multilayer perceptrons, excel at constructing static vector functions that generalize to new examples by automatically discovering "latent representations" (i.e., hidden features) most relevant to the task at hand. However, the opacity of the optimization process for this deep learning architecture is a double-edged sword: while deep learning can be easily applied to many problems with minimal manual engineering, even experts are unaware of the potential impact of most hyperparameter variations on overall performance beforehand.

[0004] Despite breakthroughs, it's clear in the field that feedforward architectures cannot learn relationships across input data arbitrarily in a timely manner, which is essential for other tasks involving video, speech, and sequential time-series data with long-term temporal dependencies. Regardless of network depth, feedforward networks will always have some finite input response, leaving a limited "memory" of previous inputs within the network's state. In other words, functions computed with such networks cannot access inputs beyond the network's depth. The most general solution to overcome this problem is to introduce recurrent connections into the network, thus transmitting current state information back to the network itself, allowing the network to capture information about previous inputs and reuse that information in the future. These networks are called recurrent neural networks (RNNs).

[0005] RNNs are the most computationally powerful type of neural network known to be physically implemented. By using recurrent connections to store state information over time, thus giving the network internal memory, RNNs are able to compute functions outside the class of computation provided by deep feedforward networks: dynamic systems, i.e., functions whose states evolve non-linearly according to the history of their inputs. This allows the network to take advantage of patterns in the input that span across time along arbitrary time scales.

[0006] Specifically, RNNs are used in the discrete-time domain (see AM) (See H. G. Zimmermann, "Recurrent neural networks are universal approximators," *International Conference on Artificial Neural Networks*, Springe, pp. 632-640, September 2006) and "universal approximators for any finite-dimensional, causal, dynamic system in the continuous-time domain" (see K. Funahashi and Y. Nakamura, "Approximation of dynamical systems by continuous-time recurrent neural networks," *Neural Network*, Vol. 6, No. 6, pp. 801-806, November 1992). In practice, RNNs are often the best models for tasks involving sequential input, such as speech recognition, language translation, video processing, caption generation, and decoding human emotions.

[0007] A long-standing challenge of RNNs relates to the difficulty of training initial stochastic recurrent weights to leverage long-term time dependencies (see Y. Bengio, P. Simard, and P. Frasconi, Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, Vol. 5, No. 2, pp. 157-166, March 1994). Many architectural solutions have been proposed, the most successful in history being Long Short-Term Memory (LSTM; see S. Hochreiter and J. Schmidhuber, Long short-term memory. Neural Computation, Vol. 9, No. 8, pp. 1735-1780, November 1997). Several recent but closely related alternatives also exist, such as gated recurrent units (GRU; see Chung, C. Gulcehre, K. Cho and Y. Bengio, Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv: 1412.3555, December 2014) and non-saturating recurrent units (NRU; see S. Chandar, C. Sankar, E. Vorontsov, SE Kahou and Y. Bengio, Towards non-saturating recurrent units for modelling long-term dependencies. In Proceedings of the AAAIConference on Artificial Intelligence, Vol. 33, No. 1, pp. 3280-3287, July 2017).

[0008] LSTM, GRU, NRU, and other related alternatives are all specific RNN architectures designed to alleviate the difficulty of training RNNs by providing connections between nodes configured in the network. These architectures typically train to better levels of accuracy compared to randomly initialized RNNs of the same size. However, these architectures currently cannot learn temporal dependencies across approximately 100 to 5,000 time steps, which severely limits their scalability to applications involving longer input sequences. Therefore, there remains a need for improved RNN architectures that can be trained to accurately maintain longer representations of temporal information (i.e., beyond 100 to 5,000 steps in sequential time series), which inspired the proposed Legendre Memory Cell (LMU). Summary of the Invention

[0009] In one embodiment of the present invention, a method for generating a recurrent neural network having Legendre memory cell (LMU) units is disclosed, comprising: defining a node response function for each node in the recurrent neural network, the node response function representing a state over time, wherein the state is encoded as either a binary event or a real value; each node having node inputs and node outputs; defining a set of connection weights for each node input; defining a set of connection weights for each node output; defining one or more LMU units having a set of circular connections defined as a matrix for determining node connection weights based on the following formula:

[0010] ,

[0011] in

[0012] Where q is an integer determined by the user, and i and j are greater than or equal to 0.

[0013] In one aspect of the invention, the set of input connection weights is defined as a matrix that determines the node connection weights based on the following formula: Where b i =(2i+1)(-1) i .

[0014] In another aspect of the invention, the LMU node connection weights are determined based on the following equation:

[0015] f(A;θ,t)

[0016] Where f is a function of A, θ is a predetermined parameter, and t is time.

[0017] In another aspect of the invention, the predetermined parameter is either selected by the user or determined using the output of the neural network.

[0018] In another aspect of the invention, the LMU node connection weights are determined based on the following equation:

[0019]

[0020] Where is a function of A, θ is a predetermined parameter, t is time, and Δt is a predetermined parameter.

[0021] In another aspect of the invention, one or more connection weights from the node output are determined by estimating Legendre polynomials.

[0022] In another aspect of the invention, the LMU units are stacked, wherein each LMU unit is connected to the next using a connection weight matrix.

[0023] In another aspect of the invention, one or more LMU units include connections to inputs to other network architectures selected from LSTM units, GRU units, NRU units, other LMU units, multilayer perceptrons, S-shaped layers and other linear or nonlinear layers, and to outputs from other network architectures selected from LSTM units, GRU units, NRU units, other LMU units, multilayer perceptrons, S-shaped layers and other linear or nonlinear layers.

[0024] In another aspect of the invention, the network is trained as a neural network by updating multiple parameters of the network.

[0025] In another aspect of the invention, the network is trained as a neural network by fixing one or more parameters while simultaneously updating the remaining parameters.

[0026] According to another embodiment of the invention, a system is provided for pattern classification, data representation, or signal processing in a neural network, the system comprising: one or more input layers that present a vector of one or more dimensions, wherein each dimension is provided to the network via an external input or by using a previous output from the network; one or more intermediate layers coupled to at least one of the input layers, other intermediate layers, or an output layer via a weight matrix; and one or more output layers that generate a vector representation of the data presented at the input layers or a function of the data computed at one or more discrete time points or continuously over time; wherein the system uses the methods described herein to generate a recurrent neural network.

[0027] According to another embodiment of the invention, a circuit is provided, implemented in hardware, having one or more cyclic connections that determine node connection weights, as described herein. Attached Figure Description

[0028] The invention is illustrated in the accompanying drawings, which are intended to be exemplary and not limiting, wherein the same reference numerals are intended to refer to the same or corresponding parts, and wherein:

[0029] Figure 1 A software implementation for a feedforward network is shown, which determines connection weights to deactivate the effects of low-pass filters at each layer.

[0030] Figure 2 A software implementation for a recurrent network according to an embodiment of the invention is shown, which determines connection weights to deactivate the effects of low-pass filters at each layer.

[0031] Figure 3 A circuit implementation of the continuous-time LMU equations for six-dimensional loops and input weights is shown.

[0032] Figure 4 A method according to one embodiment of the present invention is shown.

[0033] Figure 5 This is a schematic diagram of an exemplary neural network on which embodiments of the present invention can be implemented. Detailed Implementation

[0034] The present invention has been outlined above, and some exemplary and detailed embodiments will now be described below, in which the comparison with and benefits of the prior art are described more clearly.

[0035] It will be apparent to those skilled in the art that other configurations, hardware, etc., may be used in any of the foregoing embodiments of the products, methods, and systems of the present invention. It will be understood that the specification is illustrative of the invention, and that other embodiments are themselves suggestions for those skilled in the art. All references cited herein are incorporated herein by reference.

[0036] Implementations of the systems and methods described herein may be implemented in hardware or software or a combination of both. These implementations may be implemented in a computer program that executes on a programmable computer, each computer including at least one processor, a data storage system (including volatile or non-volatile memory or other data storage elements or combinations thereof), and at least one communication interface.

[0037] In this invention, a neural network architecture with connection weights determined using Legendre Memory Cell (LMU) equations is trained, while optionally keeping the determined weights fixed. The network can use spiked or non-spiked activation functions, can be stacked or cyclically coupled with other neural network architectures, and can be implemented in software and hardware. Embodiments of this invention provide systems for pattern classification, data representation, and signal processing, which are computed using orthogonal polynomial basis functions spanning a sliding time window. Recurrent neural networks are well known in the art, and it is assumed that their description and operation are known in this application. This invention provides an improved method and system by which the Legendre Memory Cell (LMU) method and algorithm are used to determine the weights of recurrent network nodes. Each node to which the LMU method is applied is also referred to herein as an LMU unit.

[0038] A neural network architecture with connection weights determined using the Legendre memory cell equations is trained, while optionally keeping the determined weights fixed. The network can use spiked or non-spiculated activation functions, can be stacked or cyclically coupled with other neural network architectures, and can be implemented in software and hardware. Embodiments of the present invention provide systems for pattern classification, data representation, and signal processing that use orthogonal polynomial basis functions across a sliding time window for computation.

[0039] The LMU unit is defined as follows. Let q≥1 be an integer, provided by the user. Let A=[a] ij ∈ Let be a q×q square matrix (0≤i,j≤q-1), where the coefficients are as follows:

[0040]

[0041] The output of each node can be defined as follows. Let Let x be a q×1 matrix, with the following coefficients:

[0042] b i =(2i+1)(-1) i .

[0043] Let θ be a parameter provided by the user or determined using the output of nodes in the neural network.

[0044] Let t be either a continuous time point or a discrete time point. For the discrete-time case, let Δt be a parameter provided by the user or determined using the output of nodes in the neural network.

[0045] LMU circular connections determine node connection weights by estimating the following equation:

[0046] In the continuous-time case, f(A; θ,t); or in the discrete-time case...

[0047] f is A in the continuous-time case (parameterized by θ), t in the discrete-time case, and additionally, for the discrete-time function. The function of Δt

[0048] The LMU connection weights to the node input can optionally be determined by estimating the following equation:

[0049] In the continuous-time case, g(B; θ,t); or in the discrete-time case...

[0050] g is B in the continuous-time case (parameterized by θ), t in the discrete-time case, and additionally, for the discrete-time function. The function of Δt

[0051] The LMU connectivity from the node output can optionally be determined by estimating the Legendre polynomial (see AMLegendre, Recherches sur l'attraction des Mémoires de Mathématiques et de Physique, présentés à l'Académie Royale des Sciences, pp. 411 to 435, 1782).

[0052] This method for determining the weights of recurrent connections is novel in the art and provides an improved recurrent neural network as discussed below.

[0053] Derivation of matrices A and B

[0054] To derive formulas 1 and 2, let This corresponds to a subset of state vectors represented by some LMU units, and lets This corresponds to a subset of the vectors provided as input to the aforementioned LMU unit. Given the choice of the (A,B) matrix, the following continuous-time dynamic system is defined:

[0055]

[0056] This dynamic system uses orthogonal Legendre symbols to represent the memory of u across a sliding time window of length θ, where the coefficients are given by the state x. This provides computational benefits that cannot be used in any other RNN architecture.

[0057] Examples of (A, B) with q = 6 are as follows:

[0058]

[0059] Example of determining continuous-time loops and input weights

[0060] Equation 3 corresponds to the following continuous-time system of the q-ordinary differential equation (ODE):

[0061]

[0062] For example, the following function is defined to determine loop weights:

[0063]

[0064] And define the following function to determine the weight input:

[0065]

[0066] Example of determining discrete-time cycles and input weights

[0067] Equation 3 corresponds to the following discrete-time dynamic system with qODE, discretized in time steps of Δt:

[0068]

[0069] For example, considering zero-order preservation (ZOH; see W.B. Rogan, Modern Control Theory, 3rd Edition, Pearson, October 1990) discretization, the following function is defined to determine the recurrent weights:

[0070]

[0071] And define the following function to determine the weight input:

[0072]

[0073] Other considerations for equations 1 and 2

[0074] When estimating equations 1 and 2 respectively, f(A; θ,t) and f(A; θ,t) are allowed. and g(B; θ, t) or Other possible definitions of θ are used to determine the connection weights. Examples include, but are not limited to, alternative methods using numerical integration of the differential equation, and varying θ with t and Δt.

[0075] Input sequences with irregular intervals (also known as "non-uniformly spaced time series") are supported by using input nodes in a neural network to provide Δt.

[0076] If the output of equation 1 or 2 is constant (i.e., if its parameters are immutable and do not depend on the output of any node in the neural network), then these two equations only need to be estimated once (e.g., to initialize the weights). Otherwise, these two equations may need to be re-estimated when their parameters change.

[0077] Example of determining output weights

[0078] To determine the output connection weights from one or more nodes using Legendre polynomials, the first q-polynomial can be estimated, for example, using the Rodrigues formula for shifted Legendre polynomials (see O. Rodrigues, De l'attraction des Correspondence sur Polytechnique.PhDThesis,University of Paris,1816):

[0079] (6)

[0081] Where r∈[0,1], 0≤i≤q-1, and P i It is a Legendre polynomial of order i. To provide a concrete example, state the following properties:

[0082]

[0083] For the representation x i For each connection extending from a node, θ′ (0 ≤ θ′ ≤ θ) can be selected, and then set... The weights are determined by estimating Equation 6. More generally, any function of these polynomials (e.g., integral transform, such as Fourier transform) can be computed to make the output nodes approximate a function of the sliding window of u.

[0084] refer to Figure 1 and Figure 2 These methods are applied to determine the output connection weights between layers by choosing θ′ = 0. In this example, Equation 4 is used to determine the cyclic connection weights, and Equation 5 is used to determine the input connection weights. In this exemplary implementation, low-pass filters are used to implement the integration required by the dynamic system at each layer, and the choice of θ′ = 0 effectively deconvolutions performed by each low-pass filter. Therefore, through these LMU weights, the system immediately propagates its input signal to the deepest layer, such as... Figure 2 As shown. Without this choice of weights, the signal gradually becomes lower-pass filtered at each layer, as... Figure 1 As shown.

[0085] Training Neural Networks

[0086] The parameters of a neural network can be trained using any available method, such as backpropagation over time (BPTT; see PJ Werbos, Backpropagation through time: What it does and how to do it. Proceedings of the IEEE, Vol. 78, No. 10, pp. 1550-1560, October 1990).

[0087] During training, one or more of the weight parameters generated by estimating Equation 1 or 2 or the Legendre polynomial can be kept constant. Alternatively, one or more of the weights generated by estimating Equation 1 or 2 or the Legendre polynomial can be trained. In either case, when using BPTT, the error may propagate back through the multiplicative accumulation operation of the connected weights.

[0088] Similarly, the parameters of equation 1 or 2 (e.g., θ or Δt, or the parameters of the neural network for determining θ or Δt) can also be trained, for example, by backpropagating the error through the gradient of equation 1 or 2 (see also TQ Chen, Y. Rubanova, J. Bettencourt and DK Duvenaud, Neural Ordinary Differential Equations. In Advances in Neural Information Processing Systems, pp. 6571-6583, December 2018).

[0089] To train an architecture with nodes consisting of spike nonlinearities, any available method for training spike neural networks can be used (see E. Hunsberger, C. Eliasmith, Spiking deep networks with LIFneurons, arXiv:1510.08829, October 2015).

[0090] Software Architecture

[0091] Neural networks with the aforementioned connection weights can be implemented in software. Layers with one or more connection weights determined by estimating Equation 1 or Equation 2 or the Legendre polynomial can be implemented using program code to create LMU units. These layers can be cyclically coupled to other neural network architectures. These layers can also be stacked by connecting each layer to the next using connection weights or other neural networks.

[0092] The program code is applied to the input data to perform the functions described herein and generate output information. The output information is applied to one or more output devices in a known manner.

[0093] Each program may be implemented in a high-level procedural or object-oriented programming or scripting language, or both, to communicate with a computer system. Alternatively, the program may be implemented in assembly language or machine language, if desired. The language may be a compiled or interpreted language. Each such computer program may be stored on a storage medium or device (e.g., read-only memory (ROM), magnetic disk, optical disk) readable by a general-purpose or special-purpose programmable computer for configuring and operating the computer when the storage medium or device is read by the computer to perform the processes described herein. Implementations of the system may also be considered as non-transitory computer-readable storage media configured with computer programs, wherein such a storage medium causes the computer to operate in a specific and predefined manner to perform the functions described herein.

[0094] Furthermore, the systems and methods described in the embodiments can be distributed in a computer program product, which includes a physical, non-transitory computer-readable medium carrying computer-usable instructions for one or more processors. The medium can be provided in various forms, including one or more floppy disks, optical disks, magnetic tapes, chips, magnetic and electronic storage media, etc. Non-transitory computer-readable media includes all computer-readable media but excludes transiently propagated signals. The term non-transitory is not intended to exclude computer-readable media in which data stored thereon is only temporarily stored, such as volatile memory or random access memory (RAM). Computer-usable instructions can also be in various forms, including compiled and uncompiled code.

[0095] Figure 5 A schematic diagram of a neural network 500, which can be implemented in hardware or software, is shown. This neural network has an input layer 508, one or more intermediate layers 512, and an output layer 516. The input layer has multiple nodes 508, 530, and 536. The intermediate layers have recurrent nodes 532 that loop within the intermediate layers, where input weights 518 and output weights 520 couple each of the nodes in the intermediate layers. The recurrent weights provide feedback loops within the nodes of the intermediate layers. The output layer has a node 534. For example, the input to the input layer is shown as an external input 502 or an input from a previous output 504 (derived from 528).

[0096] Hardware architecture

[0097] The neural network with the aforementioned connection weights can be implemented in hardware, including neuromorphic, digital, or analog hardware and / or a hybrid thereof. More specifically, the architecture can be implemented in application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), or in configurations using analog components and other physical primitives, including but not limited to transistors and / or other parallel computing systems.

[0098] refer to Figure 3 An exemplary circuit 300 for implementing a neural network according to the invention is shown, wherein, in continuous time, connection weights are determined by estimating equations 1 and 2 by module 300, where q = 6. Large circles correspond to each dimension of x. Small circles represent elements added (arrows) or subtracted (round heads) from their input. The i-th dimension is time-integrated and scaled against its input (triangle heads) by (2i+1) / θ.

[0099] This design utilizes symbolic alternation and decomposes it into two separate cascaded summation chains, which are then combined by a feedback loop, using intermediate calculations within the upper and lower triangles of A. These same calculations are also reused to implement the connection weights of B by supplying u to the appropriate intermediate nodes.

[0100] Increasing the system's dimension by 1 requires adding O(1) lines, adders, and state variables to the existing circuitry. In total, the circuitry requires O(q) lines, adders, and state variables, thus enabling linear scaling of the circuitry both spatially and temporally.

[0101] Simulation results

[0102] A set of experiments was considered to estimate the memory capacity of a stacked LSTM relative to a stacked LMU with equivalent resource usage. For this purpose, an off-the-shelf Keras implementation of a stacked LSTM was used, and three layers were constructed, each with 50 units. Each layer was fully connected to the next layer and used all default settings (e.g., tanh activation). The last layer similarly consisted of tanh activation units for each output. To estimate the continuous-time memory capacity, the input data was white noise with a bandwidth limited to 30 Hz, starting from 0 and normalized to the absolute range [-1, 1]. The output data was a 50-dimensional vector representing a uniform arrangement of inputs with delays between 0 and 0.2 seconds. The dataset consisted of 256 samples, each 1 second long. These data were randomly split 50% for training and 50% for testing. The training data was further split into a separate random 25% sample, which was used to report validation accuracy during training. Backpropagation over time was performed using the Adam optimizer relative to the mean squared error (MSE) loss function. Training was performed in parallel on four Nvidia Titan Xp GPUs (12GB each) using Keras and TensorFlow.

[0103] The findings revealed that, for a time step of 2 ms, backpropagation could find sufficient parameters to solve the task; that is, LSTM could indeed accurately represent the entire delay interval consisting of θ = 100 time steps, with a normalized root mean square error (NRMSE) of approximately 10%. However, performance degraded after reducing the time step by an order of magnitude to 200 μs while simultaneously increasing the data length by the same factor, so that the data still represented the exact same 1-second signal; at a time step window of θ = 1,000, accuracy decayed exponentially with the delay length. In the worst case, LSTM was no better than random chance, with an NRMSE of approximately 100%. Therefore, even the most successful RNN architectures in history clearly cannot represent increasingly longer time windows, spurring a demand for more powerful RNN architectures.

[0104] Then, the exact same training code and network specifications were used, but each LSTM unit was replaced with a layer of LMU units, where (A,B) matrices were used in the continuous-time case (equivalent to discretizing the system using the Euler method). These matrices were shared between each unit within the same layer (similar to weight sharing in convolutional neural networks). Finally, multiple tanh nonlinearities (one per unit) were included, which received inputs from all state variables on the same layer, thereby supporting mixed nonlinear computation across scaling Legendre codes. For smaller q values ​​(e.g., 9), the network had resource requirements comparable to the aforementioned LSTM.

[0105] Each LMU unit receives a one-dimensional input. The trainable parameters are the weights between layers and the delay length θ within each unit. In this experiment, training on shared (A,B) weights was disabled. The overall architecture is consistent with LSTM, as the LMU contains 50 units stacked 3 times. The final output layer consists of linear activation units, since tanh has already been applied at this point. Finally, q = 9 is set, and for the first layer, the encoded weights of each unit are initialized to 1, and for all subsequent layers, they are initialized to 1 / 50 (i.e., the reciprocal of the fan-in), with the θ values ​​uniformly distributed over U[100, 1000], and the weights projected onto each tanh are set by estimating the Legendre polynomial with r = 1, where the weights of all other state variables outside the unit are zero. In other words, each unit is initialized to approximate tanh(u[t-θ]), where u[·] is the average input of the unit. Then, backpropagation trains the values ​​of θ and learns a weighted nonlinear combination of the input and output mixed between layers.

[0106] Running the exact same code and performing the analysis on identical training, validation, and test data revealed a significant difference in training time between the two methods. It was found that the stacked LMU took 52.5 s per epoch to train, compared to 102.6 s per epoch for the stacked LSTM. Furthermore, the LMU outperformed the LSTM in every accuracy metric. Specifically, MSE was reduced by three orders of magnitude in training and validation while converging to the ideal solution much faster. The LMU architecture achieved a consistent 3% to 4% error across latency intervals, while an equivalent LSTM cell architecture approached 100% error at the end of the window. This illustrates the difficulty of stacked LSTMs in remembering low-frequency signals (relative to the time step) over long time intervals. In contrast, this task is natural for the stacked LMU, whose state representation of the input history is a q-degree Legendre scaling.

[0107] Backpropagation enables stacked LMUs to outperform stacked LSTMs, even on tasks where initial network configurations are not readily supported. To evaluate the performance of each network on continuous-time prediction tasks, a synthetic dataset called Mackey-Glass (MG) is considered: a chaotic time series described by a nonlinear delay differential equation. MG data is generated using a discrete-time delay of τ = 17 (1 unit time per time step). The desired output is a look-ahead (prediction) 15 time steps ahead. This is simulated over 5,000 time steps after removing the first 100-step transients. This is repeated 128 times, each time starting from initial random conditions. The entire dataset is then centered so that its global mean is zero. Next, the dataset is randomly split into 32 training examples, 32 validation examples, and 64 test examples.

[0108] The same network as in previous experiments was used, but with four layers, each with 100 units. For the LMU units, all parameters were made trainable (including the A and B matrices shared across units within the same layer). q was set to 6 and θ ∈ U[25, 50] was initialized to address the shorter timescale of this dataset. The remaining weights were initialized using the standard Keras weight initializer. All three methods were trained using the Adam optimizer over 500 epochs. In this case, to minimize overfitting, only the model from the epoch with the highest validation score was retained.

[0109] The test performance and training time are summarized below. LSTM achieved an error rate of 7.084% with 282,101 parameters, taking 50.0 seconds per training epoch. LMU achieved an error rate of 6.783% with 270,769 parameters, taking 30.5 seconds per training epoch. Therefore, LMU outperforms LSTM in both accuracy and training time. This is presumably because LMU is better able to support delayed embeddings within its 6-dimensional state. Furthermore, LMU provides time-varying improvements over longer continuous time intervals compared to lower frequencies.

[0110] Exemplary Applications

[0111] These methods can be used to create systems that use neural networks in hardware and software for pattern classification, data representation, or signal processing.

[0112] For example, Automatic Speech Recognition (ASR) is a system for computer speech recognition that processes speech (as an audio input waveform) and produces text (as a model output). The input can be preprocessed into audio features (e.g., Mel-frequency cepstral coefficients, FilterBANK coefficients, and feature-space maximum likelihood linear regression coefficients; see M. Ravanelli, T. Parcollet, and Y. Bengio, The PyTorch-Kaldi Speech Recognition Toolkit, In International Conference on Acoustics, Speech and Signal Processing, IEEE, pp. 6465-6469, May 2019) and fed into a neural network consisting of layers with connection weights determined using LMU unit equations, where the output nodes of the neural network are post-processed using available methods for generating text (e.g., contextual bundle search). Thus, the system can be trained as a neural network to construct an ASR system.

[0113] As another example, consider an application of anomaly detection, i.e., identifying outliers or "anomalies" in a dataset. This data can be sequentially fed (one input vector at a time) into a neural network consisting of layers with connection weights determined using LMU unit equations, where the output nodes of the neural network classify the inputs as typical or anomalous. Therefore, the system can be trained to build an anomaly detector using available methods (e.g., using unsupervised, semi-supervised, or fully supervised learning rules).

Claims

1. A computer implementation method for implementing a recurrent neural network, the recurrent neural network having a Legendre memory unit (LMU) for automatic speech recognition, the method comprising: Using a computer processor, a node response function is defined for each node in the recurrent neural network. The node response function represents the state over time, wherein the state is encoded as either a binary event or a real value; each node has a node input and a node output. The computer processor defines a set of connection weights for each node input; The computer processor defines a set of connection weights for each node's output. The computer processor defines one or more LMU units, each having a set of cyclic connections defined by a matrix that determines node connection weights based on the following formula: ， in Where q is an integer determined by the user, i and j are greater than or equal to 0, and The computer processor trains the network into a recurrent neural network by updating multiple parameters or fixing one or more parameters while updating the remaining parameters, generating a recurrent neural network that includes the node response function of each node, the set of connection weights for each node's input, the set of connection weights for each node's output, and LMU units.

2. The method of claim 1, wherein the set of input connection weights is defined as a matrix for determining node connection weights based on the following formula: Where b i =(2i+1)(-1) i .

3. The method of claim 1, wherein the LMU node connection weights are determined based on the following equation: f(A;θ,t) Where f is a function of A, θ is a predetermined parameter, and t is time.

4. The method of claim 3, wherein the predetermined parameter is either selected by a user or determined using the output of a node in the neural network.

5. The method of claim 1, wherein the LMU node connection weights are determined based on the following equation: in It is a function of A, θ is a predetermined parameter, t is time, and Δt is a predetermined parameter.

6. The method of claim 5, wherein each of the predetermined parameters is either selected by a user or determined using the output of a node in the neural network.

7. The method of claim 1, wherein one or more connection weights from the node output are determined by estimating Legendre polynomials.

8. The method of claim 1, wherein the LMU units are stacked, and each LMU unit is connected to the next using a connection weight matrix or another neural network.

9. The method of claim 1, wherein one or more LMU units include connections to inputs to other network architectures selected from LSTM units, GRU units, NRU units, other LMU units, multilayer perceptrons, sigmoid layers and other linear or nonlinear layers, and to outputs from other network architectures selected from LSTM units, GRU units, NRU units, other LMU units, multilayer perceptrons, sigmoid layers and other linear or nonlinear layers.

10. The method of claim 1, further comprising training a recurrent neural network; and Use trained recurrent neural networks for pattern classification, data representation, or signal processing.

11. The method of claim 10, wherein the network is trained as a neural network by updating a plurality of parameters of the network.

12. The method of claim 10, wherein the network is trained as a neural network by fixing one or more parameters while simultaneously updating the remaining parameters.

Images