Method and system for processing temporal data with a linear artificial neural network layer - Patents.com
Patent Information
- Application Number
- JP2024550148
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-02-24
- Filing Date
- 2023-02-23
- Publication Date
- 2025-03-26
Smart Images
Figure 2025508465000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates generally to the field of processing temporal data with artificial neural networks, and more specifically to improving the efficiency of these networks by configuring them to implement linear dynamical systems that compute state updates in linear time. [Background technology]
[0002] Modern machine learning systems are widely used to perform tasks in which time-varying sequences of input data are mapped to one or more output predictions. Examples of such tasks include natural language processing (e.g., mapping a sequence of words in one language to a sequence of words in another language), automatic speech recognition (e.g., mapping a sequence of samples of speech waveforms to a much shorter sequence of linguistic symbols), and general-purpose signal processing (e.g., mapping a sequence of readings from a wearable wristband to a sequence of heartbeat event detection). A common approach to building such systems involves the use of artificial recurrent neural network models, which use a set of recurrently connected weights to iteratively map each item in the input sequence items to a continuously evolving internal state representation. The ability to process input sequences of any length in this manner has made recurrent neural networks (RNNs) a standard choice for classification, modeling, or other processing of time series data.
[0003] Due to their versatility, RNN models have been widely studied from a theoretical perspective and also widely deployed in practical situations. From these theoretical and practical studies, two central challenges for RNNs have emerged. First, most RNNs exhibit a significant degradation in performance when tasked with processing long input sequences. Second, most RNNs also exhibit a significant degradation in their performance when their weights are quantized to reduce to some numerical precision on specialized computing devices designed to be energy efficient (e.g., computing devices that use 8-bit integer representations rather than 32-bit floating-point representations). The reason for these degradations lies in the fact that the state updates of RNNs implement a kind of dynamical system, and the behavior of such systems can change chaotically with small state perturbations (i.e., when the system is driven by long input sequences) over long time spans (i.e., when the system is driven by long input sequences). To address these challenges and improve the stability of machine learning models designed to process time-series data, many innovative neural network systems have been defined in the prior art. As such, the following publications and patents are provided for their supporting teachings and are incorporated by reference in their entirety: https: / / doi.org / 10.1162 / neco.1997.9.8.1735 discusses how to add gating mechanisms to RNN models, allowing more stable and controlled updates to the model's internal state representation, allowing it to reliably process sequences of up to about 1000 input items. Importantly, gated RNNs of this kind are sequentially bottlenecked, with each sequential update taking O(n 2 ) (where n is the number of dimensions of the input items), they are often time consuming to train and run on computer hardware.
[0004] Further prior art https: / / arxiv.org / pdf / 1706.03762.pdf describes a method for training neural networks to process sequential data at scale by using a pure feed-forward "Transformer" network architecture, which exploits an attention mechanism to model the relationships between different sequence elements. Transformers are implemented by a large number of dense matrix multiplications that are almost perfectly suited for GPU-based parallelization, thus allowing them to be trained on large amounts of data. This scalability, together with the effectiveness of the attention mechanism to learn long-term data dependencies, has led Transformer-based architectures to the state of the art for many time series modeling tasks, especially in the area of natural language processing. However, Transformers are inherently not suited to work with streaming inputs. Also, these networks are computationally extremely inefficient, frequently requiring a huge number of parameters to achieve good task performance. Relatedly, they have a linear rather than quadratic O(N 2 ) execution time.
[0005] On the topic of efficient RNN algorithms, the prior art article http: / / compneuro.uwaterloo.ca / files / publications / voelker.2019.lmu.pdf describes a recurrent neural network architecture that combines one or more layers implementing a linear time-invariant (LTI) dynamical system with one or more nonlinear layers processing sequential input data. The weights governing this LTI system are analytically derived to compute the optimal delay of the input signal over a time window, and the nonlinear components of the network are read from the state of this system to compute any function of the data in the input window. The resulting network is called a "Legendre Memory Unit" (LMU) because the LTI system represents the data using a Legendre basis, and experimental evidence shows that the LMU can efficiently handle time dependencies spanning hundreds of thousands of time steps, greatly exceeding the capabilities of alternative recurrent network architectures. Overall, the LMU is an important example of a linear recurrent network with strong performance characteristics, but is nevertheless constrained in that it only implements certain LTI systems, i.e. systems that achieve a mathematically optimal reconstruction of an input signal delayed by a certain number of time steps for a given number of state variables.
[0006] The methods and systems described in the above-mentioned publications and many similar publications do not clearly address how to design recurrently connected artificial neural networks that implement a wide variety of linear dynamical systems that compute state updates in linear time. More specifically, the existing state of the art does not provide a means to configure linear recurrent neural network layers to efficiently perform streaming spectral decomposition of an input signal over a fixed window of input time steps, where such decomposition is encoded by the state of some linear dynamical system driven by a given input signal.
[0007] This application features a method and system for improving the efficiency of recurrent neural networks by configuring the recurrent neural network to implement a linear dynamical system that computes state updates in linear time, thereby addressing the aforementioned concerns and shortcomings. These efficiency improvements result from defining the weights of a linear recurrent neural network layer by first selecting a set of basis functions that define a desired impulse response associated with a particular spectral decomposition, and then deriving a matrix of recurrent weights A that produces this impulse response over a time window of a specified length. A second matrix of recurrent weights, Γ, is also derived to cause this impulse response to decay to zero outside the time window, and the recurrent weight matrix on the network layer is set to be equal to the sum of A and Γ. Specifically, this Γ matrix can be derived for any A matrix that produces a basis as the impulse response, and further, it is guaranteed that the product of Γ and the state vector can be computed in O(n). This general method for deriving Γ allows more freedom than prior work in the selection of the set of basis functions produced by the A matrix. Correspondingly, A can also be computed in O(n) and designed to be more robust to quantization (in state) and discretization (in time). Summary of the Invention
[0008] Summary of the Invention for Processing Temporal Data Using Linear Artificial Neural Network Layers In view of the aforementioned limitations inherent in known methods for processing time data with artificial neural network layers, the present invention provides a method and system for improving the efficiency of neural networks by configuring the neural network to implement a linear dynamical system that computes state updates in linear time. These efficiency improvements are achieved by specifying recurrent weight matrices for the network layers that are optimized to compute impulse responses that decay to zero outside of a specified time window. Importantly, when the resulting linear recurrent neural network layer is applied to a time series of input data elements, each update to the layer's underlying state representation can be computed in linear time with respect to the dimension of these input data elements (by comparison, conventional recurrent neural network layers compute such updates in quadratic time). The resulting state representations of the linear recurrent layer are then provided as inputs to at least one nonlinear neural network layer, which computes a network output that provides a solution to some computational task of interest involving the original input sequence. Thus, a general object of the present invention, which is described in more detail below, is to provide a method and system for improving the efficiency of neural networks by configuring them to implement a linear dynamical system that computes state updates in linear time. In applications where recurrent layers are undesirable, the methods described herein can be used to derive weights for linear-time convolutional layers that perform the same as recurrent networks.
[0009] A main aspect of the invention features a method and system for efficiently processing time series data using an artificial neural network model. The method defines and configures at least one linear recurrent or temporal convolutional layer and at least one other layer implementing any nonlinear layer type, such as a perceptron layer, a self-attention layer, a convolutional layer, or a gated recurrent layer. The method further comprises defining recurrent or convolutional weights of at least one linear layer by (a) selecting a set of basis vectors that define a desired impulse response of the layer, (b) deriving a matrix that produces the desired impulse response over a target time window of length θ, (c) deriving a matrix that decays the impulse response to zero outside the target time window, and (d) combining these two matrices together through addition to obtain recurrent weights of the layer, or alternatively, using the impulse response of the LTI system obtained by summing two feedback matrices as a convolution kernel for the linear temporal convolutional layer. The method further includes applying the resulting linear recurrent layer to at least one time series of input data elements, thereby computing at least one state vector, and then applying a non-linear layer to this state vector, thereby generating at least one output data element corresponding to a result of performing at least one pattern classification, signal processing, data representation or data generation task involving said time series of input data elements.
[0010] In this regard, before describing at least one embodiment of the invention in detail, it is to be understood that the invention is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. The invention is capable of other embodiments and of being practiced and carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting.
[0011] These, together with other objects of the invention, together with the various features of novelty which characterize the invention, are particularly pointed out in this disclosure. For a better understanding of the invention, its operating advantages, and specific objects attained by its uses, reference should be made to the accompanying drawings and descriptive matter in which preferred embodiments of the invention are illustrated. [Brief description of the drawings]
[0012] The invention will be better understood and further objects thereof will become apparent from a consideration of the following detailed description, which refers to the accompanying drawings, in which: [Figure 1] FIG. 1 is an illustration of different orthonormal basis functions generated as impulse responses of a neural network constructed in accordance with the method disclosed herein. [Diagram 2] FIG. 2 is an illustration showing the accuracy of a generated impulse response (relative to a target impulse response) generated by a neural network configured in accordance with the method disclosed herein. [Diagram 3] FIG. 3 is a diagram illustrating the accuracy of window-truncated impulse responses (relative to a target impulse response) generated by a neural network configured in accordance with the methods disclosed herein. [Figure 4] FIG. 4 is an illustration showing the accuracy of signal delays calculated by different recurrent neural network layers configured in the manner disclosed herein. [Diagram 5] FIG. 5 is an illustration showing the accuracy of signal delay per time step over a delay window computed by different recurrent neural network layers configured in the manner disclosed herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof, and in which is shown by way of illustration specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable one skilled in the art to practice the invention, and it should be understood that the embodiments may be combined or other embodiments may be utilized, and structural and logical changes may be made without departing from the spirit and scope of the present invention. Therefore, the following detailed description is not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims and their equivalents.
[0014] The present invention will now be briefly described with reference to the accompanying drawings, in which: Reference will now be made in more detail to the exemplary drawings for the purposes of illustrating non-limiting embodiments of the present invention.
[0015] As used herein, the term "comprising", as well as its derivatives including "comprises" and "comprise", includes each of the listed integers or elements but does not exclude the inclusion of one or more additional integers or elements.
[0016] As used herein, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. For example, reference to "a device" includes not only a single device but also two or more devices, and so forth.
[0017] As used herein, the terms "for example," "like," "such as," or "including" are meant to introduce examples that further clarify a more general subject matter. Unless otherwise specified, these examples are provided only as an aid in understanding the illustrated applications of this disclosure and are not meant to be limiting in any way.
[0018] As used herein, the terms "may," "can," "could," or "might" do not require that a particular component or feature be included or have that characteristic.
[0019] The exemplary embodiments are described in more detail below with reference to the accompanying drawings in which exemplary embodiments are shown. These exemplary embodiments are provided for illustrative purposes only so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. However, the disclosed invention may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein.
[0020] Various modifications will be readily apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present invention. Moreover, all statements herein describing embodiments of the present invention, and specific examples thereof, are intended to encompass both structural and functional equivalents. Moreover, such equivalents are intended to include not only currently known equivalents, but also equivalents developed in the future (i.e., any elements developed that perform the same function, regardless of structure). Additionally, the terms and expressions used are for the purpose of describing exemplary embodiments and should not be considered limiting. Thus, the present invention is to be accorded the widest scope encompassing numerous alternatives, modifications and equivalents consistent with the principles and features disclosed. For clarity, details relating to known technical material in the art related to the present invention have not been described in detail so as not to unnecessarily obscure the present invention.
[0021] Thus, for example, those skilled in the art will appreciate that diagrams, schematics, illustrations, etc., represent conceptual diagrams or processes illustrating systems and methods embodying the present invention. The functionality of the various elements shown in the figures can be provided through the use of dedicated hardware as well as hardware capable of executing associated software. Similarly, any switches shown in the figures are conceptual. Their functionality can be performed through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, with the particular technique being selected by the entity implementing the present invention. Those skilled in the art will further appreciate that the exemplary hardware, software, processes, methods, and / or operating systems described herein are for illustrative purposes and thus are not intended to be limited to any particular named elements.
[0022] Each of the appended claims defines a separate invention and, for infringement purposes, is to be recognized as including equivalents to the various elements or limitations specified in the claims. Depending on the context, all references below to the "invention" may in some cases refer only to certain embodiments. In other cases, it will be recognized that references to the "invention" refer to subject matter recited in one or more, but not necessarily all, of the claims.
[0023] All methods described herein can be performed in any suitable order unless otherwise indicated herein or clearly contradicted by context. The use of any and all examples or exemplary language (e.g., "such as") provided with respect to specific embodiments herein is intended merely to clarify the invention and does not pose a limitation on the scope of the invention as otherwise claimed. No language in the specification should be construed as indicating any non-claimed element essential to the practice of the invention.
[0024] Various terms used in this specification are set forth below. To the extent that a term used in the claims is not defined below, that term should be given the broadest definition given to that term by those of ordinary skill in the relevant art as reflected in publications and issued patents at the time of filing.
[0025] Groupings of alternative elements or embodiments of the invention disclosed herein are not to be construed as limitations. Each group member may be referred to and claimed individually or in any combination with other members of the group or other elements found herein. One or more members of a group may be included in or deleted from a group for reasons of convenience and / or patentability. When such inclusions or deletions are made, the specification shall be deemed to include the group as modified so as to satisfy all group descriptions used in the appended claims.
[0026] For simplicity and clarity of explanation, numerous specific details are set forth in order to provide a thorough understanding of the exemplary embodiments described herein. However, it will be understood by those skilled in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the embodiments generally described herein.
[0027] Additionally, this specification is not intended to in any way limit the scope of the embodiments described herein, but rather should be considered merely as an illustration of implementations of various embodiments as described.
[0028] The embodiments of the artificial neural networks described herein can be implemented in configurable hardware (i.e., FPGAs) or custom hardware (i.e., ASICs), or a combination of both with at least one interface. Input signals are consumed by digital circuitry to perform the functions described herein and to generate output signals. The output signals are provided to one or more adjacent or peripheral systems or devices, in a known manner.
[0029] As used herein, the term "node" in the context of artificial neural networks refers to the basic processing element that implements the functionality of a simulated "neuron," which may be a spiking neuron, a continuous rate neuron, or any linear or nonlinear component used to construct distributed systems.
[0030] The described system can be implemented using adaptive or non-adaptive components, and can be efficiently implemented on a wide variety of distributed systems including multiple non-linear components whose individual outputs can be combined together to implement particular aspects of the system, as described in more detail below.
[0031] The main embodiment of the present invention is a set of systems and methods for efficiently processing time series data using artificial neural network models. The methods are configured by defining at least one linear recurrent or linear-time convolutional layer and at least one other layer implementing any nonlinear layer type such as a perceptron layer, a self-attention layer, a convolutional layer or a gated recurrent layer. The weights of at least one linear layer are configured by first selecting a set of basis vectors that define a desired impulse response and then deriving a matrix that produces this desired impulse response over a target time window of length θ. A second matrix is then derived that decays this desired impulse response to zero everywhere outside the target time window of length θ, and the connection weights of the linear layer are set to be equal to the sum of these two matrices in the case of a linear recurrent layer and to the impulse response of the corresponding LTI system in the case of a linear-time convolutional layer. The resulting method of operation of the linear neural network layer is configured to apply it to at least one time series of input data elements to compute at least one state vector that is iteratively updated as each item of the input sequence is provided to the layer. The methods include applying at least one nonlinear layer to this state vector to generate at least one output data element that solves at least one pattern classification, signal processing, data representation, or data generation task involving the time series of input data elements.
[0032] Here, the term "artificial recurrent neural network model" refers to an artificial neural network model that includes at least one set of weighted connections that transform the output of one or more nodes in a given network layer as input to one or more nodes in the same layer. These weighted connections are called "recurrent connections" and they typically introduce complex state dynamics into the artificial neural network model, since the node outputs of the model can evolve dramatically over time as a result of the feedback loops introduced by said recurrent connections. A standard implementation of an artificial recurrent neural network model performs O(n) computations with respect to the length of the input sequence to which it is applied, and O(n) with respect to the number of dimensions of each item in this input sequence. 2 ) calculation.
[0033] Here, the term "activation function" refers to any method or algorithm for applying a linear or nonlinear transformation to some input values to generate output values in an artificial neural network. Examples of activation functions include identity, rectified linear, ReLU, leaky-ReLU, parametric ReLU, sigmoid, tanh, softmax, log-softmax, maxpool, polynomial, sine, gamma, softsine, Hebeside, swish, exponential linear, scaled exponential linear, and Gaussian error linear functions. Here, the term "linear network layer" refers to any layer in an artificial neural network that computes its output values using a linear activation function, such as the identity function.
[0034] The activation function may optionally output a "spike" (i.e., a 1-bit event), a "multi-valued spike" (i.e., a multi-bit event with a fixed or floating bit width), a continuous quantity (i.e., a floating point value with some level of precision determined by a given computing system - typically 16, 32, or 64 bits), or a complex value (i.e., a pair of floating point numbers representing rectangular or polar coordinates). These aforementioned functions are generally referred to by those skilled in the art as "spiking", "multi-bit spiking", "non-spiking", and "complex-valued" neurons, respectively. When using spiking neurons, the real and complex values may also be represented by any of a number of encoding and decoding schemes, including the relative timing of the spikes, the frequency of the spikes, and the phase of the spikes. However, it will be understood by those skilled in the art that the embodiments described herein may be practiced without these specific details.
[0035] Here, the term "dynamical system" refers to any system whose system state can be characterized with a set of numbers corresponding to a point in a geometric space, and for which a function is defined that relates this system state to its derivative with respect to time. In other words, a dynamical system includes a state space together with a function that defines the transition between states over time. The term "linear time-invariant system" refers to a particular class of dynamical systems in which the relationship between the input of the system at a given time and its output is a linear mapping; moreover, this mapping is time-invariant in the sense that a given input is mapped to the same output regardless of the time the input is applied. LTI systems have the advantage of being relatively easy to analyze mathematically compared to more complex nonlinear systems. In the context of the present invention, a particularly important form of mathematical analysis specifies how to write the state update equations of an LTI system in a non-sequential form. Every linear recurrent neural network layer implements an LTI system, and the configuration of the connection weights of the layer determines which particular LTI system is implemented. Thus, every linear recurrent neural network layer also implements a dynamical system.
[0036] Here, the term “impulse response” refers to the mathematical description of the output of an LTI system in response to an instantaneous input of unit magnitude. The impulse response of a dynamic system more generally defines how it behaves as a function of time under certain input conditions. For any LTI system, the behavior of the system can be fully characterized in terms of the impulse response, since an instantaneous pulse of unit magnitude includes all possible combinations of input frequencies, thereby stimulating the system response to all possible input frequencies. Thus, subject to the constraints of linearity and time invariance, the response exhaustively defines the behavior of the system for all possible inputs over time. Mathematically, an LTI system is characterized in terms of the input u(t) being mapped through a matrix B to generate a state representation m(t), which is then mapped through a recurrent matrix A, with the instantaneous change to m being characterized as described by the following relationship:
number
[0037] The term "basis vector" herein refers to a vector that belongs to a set of vectors that span a vector space. The term "set of basis vectors" herein refers to a set of basis vectors such that none of the individual vectors in the set can be expressed as a linear combination of any of the other vectors in the set. The term "basis function" herein refers to a function that belongs to a set of functions that span a similar function space. The term "spectral decomposition" herein refers to the process of taking a sliding window over a time-varying one-dimensional signal and decomposing the signal within the window into a weighted combination of a set of selected basis functions or basis vectors. Often these basis functions or basis vectors correspond to different signal frequency components that are present within the sliding window, in which case the basis is a Fourier basis. Other common choices for basis include cosine functions over a range of frequencies (i.e., "cosine basis") and a set of orthogonal Legendre polynomials (i.e., "Legendre basis"). Other polynomials that can be used as a basis include Legendre, Chebyshev, Laguerre, Hermite, and Jacobi polynomials.
[0038] The nonlinear components of the aforementioned systems can be implemented using a combination of adaptive and non-adaptive components. Examples of nonlinear components that can be used in various embodiments described herein include simulated / artificial neurons, FPGAs, GPUs, and other parallel computing systems. The components of the systems may be implemented using various standard techniques, such as by using a microcontroller. Furthermore, the nonlinear components may be implemented in various forms, including software simulation, hardware, or any neuronal fiber. The nonlinear elements may also be implemented using neuromorphic computing devices such as Neurogrid, SpiNNaker, Loihi, and TrueNorth.
[0039] As an exemplary embodiment of the proposed system and method, we consider the problem of constructing an LTI system that approximates the sliding-window spectrum for an arbitrary basis. In standard form, this kind of LTI system can be described in terms of the input to the system, u, being mapped through a matrix B, while the system state is mapped through a recurrent matrix A at each time step. We define a set of q orthonormal basis functions b i To realize an LTI system with t, the state m(t) of the system can be described by the integral:
number
[0040] Referring to FIG. 1, if q is set to 6 and Fourier series are used as the basis, then m(t) is a linear combination of sine and cosine functions that reconstructs the signal u over a window of length θ. Alternatively, if cosine series or Legendre polynomials are used as the basis, then m(t) is also a linear combination of basis functions that optimally reconstructs the signal u over a window of length θ, but a different basis is used in each case. Importantly, the A and B matrices of the LTI system that generate the above basis can be derived over N time steps using only q state dimensions. The orthonormal basis transformation matrix M∈R qxNwhere the i-th column of M, i∈{1, ,N}, is a q-dimensional vector representing the impulse response of the desired system at time t = Δti, where Δt = θ / N. If the state evolution is indeed the result of a time-invariant linear process, then for all 1≦i <Nについて、
number
number
number
[0201] ), cosine (
[0202] , and Legendre (
[0203] ) bases.
[0041] However, the impulse response generated from the derivation of A and B is not limited to a target window of length θ. To constrain the response to that window, we can decode a delayed version of the input signal u(t-θ) from the system state m(t). Furthermore, the specific contribution of u(t-θ) to m(t) can be calculated by m It is also possible to calculate m(t) from mSubtracting (t) effectively "erases" information about u(t-θ) from the state vector, resulting in a rapidly decaying and nearly finite impulse response. The resulting "information erasure" update is linear and can be expressed as a rank-1 matrix Γ. Referring to Figure 3, the effect of this "information erasure" process on an LTI system generating modified Fourier, cosine, and Legendre bases is shown. To generate the modified Fourier basis, the oscillations of the Fourier basis are slowed by 10%, making the basis non-periodic and compatible with information erasure. This new modified Fourier basis is no longer orthonormal and is therefore theoretically suboptimal information. For both the modified Fourier and Legendre bases, this method does indeed approximate a rectangular window, but introduces ringing artifacts. Although this method stabilizes the cosine basis, oscillations still remain for t>θ - presumably because the cosine basis is not well realized by the underlying LTI system.
[0042] In theory, the various function bases just mentioned have the same expressive power. In continuous form, they are 2 (0,θ) and in the discrete case, the N-dimensional vector space R NHowever, when the basis is truncated to include only q terms, differences arise with respect to the accuracy with which the function can be decoded, especially when generating the basis as the impulse response of an LTI system of order q. One way to quantitatively characterize these differences is to measure how accurately a delayed version of the input signal u(t) can be decoded from the state m(t) of the system. Therefore, the results of several benchmark experiments are described below to provide a demonstration of the herein disclosed method and system for processing time data with recursively connected artificial neural networks. With reference to FIG. 4, a 1 second delay is calculated from m(t) using a dynamic system that computes a low pass filter on the input signal u(t) and a system that computes the impulse responses corresponding to the modified Fourier basis u(t), the cosine basis u(t) and the Legendre basis u(t). The modified Fourier basis provides the lowest level of decoding error overall, improving the cosine and Legendre basis by more than 25% in comparison. With reference to FIG. 5, it is also possible to compare the accuracy of calculating a 1 second delay per time step using different approaches to truncating the system's impulse response to a target window length under varying values of q using the Fourier, cosine, modified Fourier and Legendre bases. Using an ideally truncated impulse response
[0501] naturally produces the lowest overall level of error for all bases, while truncating the response
[0502] using an approximated Bartlett window produces low error for the modified Fourier bases but higher levels for the other bases. Using the information elimination method
[0503] finally results in the modified Fourier bases producing the lowest overall level of error. Notably, the Legendre bases produce only a slightly higher level of error, but are still much more accurate when comparing inputs from early in the delay period with inputs from later in the delay period; in comparison, the modified Fourier bases are more uniformly accurate across all points in the delay period.Overall, these analyses clearly demonstrate the advantages of the proposed method and system for efficiently processing time series data.
[0043] It should be understood that the above description is intended to be illustrative, and not restrictive. For example, the above-described embodiments may be used in combination with each other. Many other embodiments will be apparent to those of ordinary skill in the art upon reviewing the above description.
[0044] Benefits and advantages that may be provided by the present invention have been described above with respect to specific embodiments. These benefits and advantages, as well as any elements or limitations that may cause them to occur or that may make them more pronounced, should not be construed as critical, essential, or essential features of any or all of the embodiments.
[0045] Although the present invention has been described with reference to specific embodiments, it should be understood that the embodiments are exemplary and that the scope of the present invention is not limited to these embodiments. Many variations, modifications, additions, and improvements to the above-described embodiments are possible. These variations, modifications, additions, and improvements are intended to be included within the scope of the present invention.
Claims
1. 1. A computer-implemented method for efficiently processing time series data using an artificial neural network model, comprising: a. defining at least one linear layer having one or more dimensional inputs; the linear layer may be implemented as a recurrent linear layer or as a linear-in-time convolutional layer; b. defining at least one other layer that implements any nonlinear layer type, such as a perceptron layer, a self-attention layer, a temporal convolutional layer, or a gated recurrent layer; c. The recurrent and input weights of the at least one linear recurrent layer or the weights of the linear temporal convolutional layer are adjusted by: i. selecting a set of basis vectors that define a desired impulse response of the at least one linear recurrent layer over a time window of length θ; ii. Derive a matrix A of recurrent weights that produces the desired impulse response over a time window of length θ, such that multiplication of at least one vector by matrix A can be computed in O(n) time, where n is the dimensionality of the at least one vector; iii. Derive a matrix A of recurrent weights that decays the impulse response to zero outside a time window of length θ, such that multiplication of at least one vector by the matrix A can be computed in O(n) time, where n is the dimensionality of the at least one vector; iv. setting the recurrent weights of the at least one linear recurrent layer to the sum of the weights A and Γ, or setting the weights of the linear-temporal convolution layer to the impulse response of the LTI system defined by the sum of the weights A and Γ; It is defined by d. applying said at least one linear layer to at least one time series of input data elements to compute at least one state vector that represents said at least one time series of input data elements as a linear combination of said basis vectors; e. applying said at least one non-linear layer to said at least one state vector to generate at least one output data element, thereby performing at least one pattern classification, signal processing, data representation, or data generation task involving at least one time series of said input data elements.
2. The method of claim 1 , wherein the basis functions are modified Fourier, cosine, Haar, Legendre, Chebyshev, Laguerre, Hermite, or Jacobi bases.
3. The method of claim 1 , wherein one or more layers are implemented as a spiking neural network.
4. The method of claim 1 , wherein the length of the window can be adjusted during run-time of the network.
5. The method of claim 1 , wherein the window is given an arbitrary weight.
6. 1. A system for efficiently processing time series data using an artificial neural network model, said system comprising: a. at least one linear layer having a weight matrix, constructed according to claim 1; b. at least one other layer implementing any nonlinear layer type, such as a perceptron layer, a self-attention layer, a temporal convolutional layer, or a gated recurrent layer; Including, The system is operated to perform at least one pattern classification, signal processing, data representation, or data generation task by first passing a time series of at least one input data element through the at least one linear layer to compute at least one state vector, and then passing this at least one state vector through the at least one non-linear layer to generate at least one output data element, thereby performing at least one pattern classification, signal processing, data representation, or data generation task involving the at least one time series of the input data elements.
7. 7. The system of claim 6, wherein one or more layers are implemented as a spiking neural network.
8. The system of claim 6 , wherein the length of the window can be adjusted during the running of the network.
9. The system of claim 6 , wherein the windows are given arbitrary weights.