Time series forecasting using machine learning with random projections

Abstract

Machine learning models for spatial-temporal forecasting systems based on a multi-layer perceptron (MLP) model or other feedforward network can incorporate one or more random projection layers. Such models need not compute or rely on information representing relationships between different spatial nodes. In addition, the model can include temporal mixer layers that process time series data in the frequency domain, thereby leveraging the periodicity of many spatial-temporal data series.

Description

TIME SERIES FORECASTING USING MACHINE LEARNING WITH RANDOM PROJECTIONSCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 550,438, filed February 6, 2024, the disclosure of which is incorporated by reference herein.BACKGROUND

[0002] This disclosure relates generally to time series forecasting and more particularly to time series forecasting using machine learning models that incorporate random projection layers.

[0003] “Spatial -temporal forecasting” refers generally to making predictions about the future state of a network or other system having a number of interconnected nodes. One example is a road network on which vehicles travel. The road network can be instrumented with vehicle counters (or other sensors) at various locations, and each vehicle counter can correspond to a node in the network. It is desirable to use data collected by the counters to model traffic behavior (e.g., number of vehicles per hour at various locations within the road network) and make predictions regarding future traffic flow. Such predictions can assist with traffic management and planning.

[0004] Various machine learning models have been developed to automate spatial-temporal forecasting. For instance, graph-based models incorporate information about the network connectivity between nodes. (As one example, for a network having a number (n) of nodes, an n x n adjacency matrix can be used to identify adjacent pairs of nodes.) However, for large networks, such connectivity information, which is quadratic in the number of nodes, can give rise to significant computational challenges. Accordingly, models that do not incorporate explicit connectivity information have been explored. In such models, the spatial -temporal forecasting problem can be treated as a multidimensional time-series forecasting problem. However, this approach can lose useful information about the networkconnectivity, resulting in less accurate predictions. Further improvements are therefore desirable.SUMMARY

[0005] Certain embodiments of the invention relate to machine learning models for spatial- temporal forecasting systems. The models can be based on a multi-layer perceptron (MLP) model or other feedforward model that does not explicitly compute or rely on an adjacency matrix or other similarity matrix defining spatial or logical relationships among nodes of a network. The model can incorporate one or more random projection layers, which can improve the model’s ability to capture relationships between nodes. In addition, the model can include temporal layers that process time series data in the frequency domain, thereby leveraging the periodicity of many spatial-temporal data series.

[0006] Certain embodiments relate to computer-implemented methods, which can comprise: obtaining input spatial-temporal data for a network having a plurality of nodes, the input spatial-temporal data including a measurement associated with each node of the plurality of nodes at each of a plurality of past times; providing the input spatial -temporal data to a machine learning model that has been trained to forecast future measurements for each of the plurality of nodes in the network at a plurality of future times, wherein the machine learning model includes a sequence of mixer blocks, wherein each mixer block applies a temporal mixer followed by a spatial mixer that includes a random projection layer and a spatial linear layer that operates on an output of the random projection layer to produce a block-level result; and producing output data from the machine learning model, the output data including a forecast of a measurement for each of the plurality of nodes at one or more future times.

[0007] In these and other embodiments, the machine learning model can also include a final linear mapping layer that operates on the block-level result from a last mixer block in the sequence of mixer blocks.

[0008] In these and other embodiments, the machine learning model can also include an identity mapping connection applied at each temporal mixer and each spatial mixer.

[0009] In these and other embodiments, the temporal mixer for each mixer block can operate in the frequency domain. For instance, in some embodiments, applying the temporal mixer can include: performing a Fourier transform on a received data matrix to produce acomplex- valued input in a frequency domain; applying a first frequency-domain layer to a real part of the complex-valued input to produce a first output component; applying the first frequency-domain layer to an imaginary part of the complex- valued input to produce a second output component; applying a second frequency-domain layer to the real part of the complex- valued input to produce a third output component; applying the second frequencydomain layer to the imaginary part of the complex- valued input to produce a fourth output component; combining the first, second, third, and fourth output components to produce a frequency-domain output; and performing an inverse Fourier transform on the frequencydomain output.

[0010] In these and other embodiments, the random projection layer in each mixer block can comprise a matrix of values randomly selected from a statistical distribution. For example, the statistical distribution can be a Gaussian distribution. In these and other embodiments, for at least one of the mixer blocks, the matrix can have dimensions n X nrand, wherein n is equal to a number of nodes in the network and nrandis less than n. In some embodiments, nrandcan be different for the random projection layers in different mixer blocks.

[0011] Certain embodiments relate to computer systems that can comprise: a memory to store input data including spatial-temporal data for a network having a plurality of nodes, the spatial -temporal data including a measurement associated with each node of the plurality of nodes at each of a plurality of past times; and a processor coupled to the memory and configured to implement a machine learning model that operates on the spatial-temporal data to produce output data including a forecast measurement for each of the plurality of nodes at each of a plurality of future times, wherein the machine learning model includes: a plurality of fully connected layers arranged in a plurality of mixer blocks, each mixer block including a complex linear temporal mixer followed by a spatial mixer that includes a random projection layer and a spatial linear layer that operates on an output of the random projection layer; and a final linear mapping layer that operates on an output of a last one of the mixer blocks.

[0012] In these and other embodiments, the machine learning model can also include an identity mapping connection applied at each temporal mixer and each spatial mixer.

[0013] In these and other embodiments, the temporal mixer can be configured to operate in the frequency domain. For example, the temporal mixer can include: a Fourier transformmodule to operate on an input matrix to produce a complex-valued transformed matrix; a first frequency-domain layer to operate on a real part and a complex part of the complex-valued transformed matrix to produce a first output component and a second output component; a second frequency-domain layer to operate on the real part and the complex part of the complex- valued transformed matrix to produce a third output component and a fourth output component; arithmetic circuits to combine the first, second, third, and fourth output components to produced a frequency-domain output; and an inverse Fourier transform module to operate on the frequency-domain output to produce a time-domain output.

[0014] In these and other embodiments, the random projection layer in each mixer block can comprise a different matrix of values randomly selected from a statistical distribution.

[0015] In these and other embodiments, the spatial mixer can include: a first matrix transpose module that operates prior to the random projection layer; and a second matrix transpose module that operates after the spatial linear layer.

[0016] Certain embodiments relate to a computer-readable storage medium having stored therein program code instructions that, when executed by a processor in a computer system, cause the computer system to perform a method that can comprise: obtaining input spatial- temporal data for a network having a plurality of nodes, the input spatial-temporal data including a measurement associated with each node of the plurality of nodes at each of a plurality of past times; providing the input spatial-temporal data to a machine learning model that has been trained to forecast future measurements for each of the plurality of nodes in the network at a plurality of future times, wherein the machine learning model includes a sequence of mixer blocks, wherein each mixer block applies a temporal mixer followed by a spatial mixer that includes a random projection layer and a spatial linear layer that operates on an output of the random projection layer to produce a block-level result; and producing output data from the machine learning model, the output data including a forecast of a measurement for each of the plurality of nodes at one or more future times.

[0017] In these and other embodiments, the machine learning model can further include a final linear mapping layer that operates on the block-level result from a last mixer block in the sequence of mixer blocks.

[0018] In these and other embodiments, the machine learning model can further include an identity mapping connection applied at each temporal mixer and each spatial mixer.

[0019] In these and other embodiments, for each mixer block, the temporal mixer can operate in the frequency domain.

[0020] In these and other embodiments, the random projection layer in each mixer block can comprise a matrix of values randomly selected from a Gaussian distribution.

[0021] The following detailed description, together with the accompanying drawings, will provide a better understanding of the nature and advantages of the claimed invention.BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG. 1 shows a high-level block diagram of a machine learning model according to some embodiments.

[0023] FIG. 2 shows a block diagram of a mixer block that can be used in a machine learning model according to some embodiments.

[0024] FIG. 3 shows graphs illustrating the preservation of daily and weekly patterns in a random projection layer according to some embodiments.

[0025] FIG. 4 shows a block diagram of a layer structure that can be used in a temporal mixer according to some embodiments.

[0026] FIGs. 5 A and 5B illustrate unraveling of a mixer model according to some embodiments. FIG. 5A shows a high-level block diagram of a mixer model according to some embodiments. FIG. 5B shows an unraveling of a forward pass through the mixer model of FIG. 5 A

[0027] FIG. 6 shows a flow diagram of a process for operating a machine learning model in inference mode according to some embodiments.

[0028] FIG. 7 shows a table of graphs illustrating performance of a random projection mixer model according to some embodiments.DETAILED DESCRIPTION

[0029] The following description of exemplary embodiments of the invention is presented for the purpose of illustration and description. It is not intended to be exhaustive or to limit the claimed invention to the precise form described, and persons skilled in the art will appreciate that many modifications and variations are possible. The embodiments have been chosen and described in order to best explain the principles of the invention and its practicalapplications to thereby enable others skilled in the art to best make and use the invention in various embodiments and with various modifications as are suited to the particular use contemplated.Machine-Learning Model Overview

[0030] FIG. 1 shows a high-level block diagram of a machine learning model 100 according to some embodiments. Machine learning model 100 can be implemented in a computer system of generally conventional design. Machine learning model 100 can be a multilayer perceptron (MLP) or other feed-forward neural network that can be trained to predict future states of a network based on input data 110 representing past states of the network. For example, for a network having a number (n) of nodes, measurements of activity or other status at each node can be collected at some numberof time points (e.g., hourly data for a week or a month). Input data 110 can be, for example, a matrix having n rows and tpastcolumns. (It should be understood that some data may be missing.) The arrangement of the n rows can be but need not be reflective of a physical arrangement or connectivity of different nodes in the network.

[0031] Machine learning model 100 includes a number (M) of mixer blocks 102-1 through 102-Af that are sequentially applied to input data 110. The value of M can be chosen as desired, based on tradeoffs between performance and computational cost. In one example implementation, M=8. The output, or block-level result 112-z (where 1 < i < M), of each mixer block 102 can be a matrix of dimension n X tpastthat is provided as input to the next mixer block 102. The block-level result 1 12- / W of the last mixer block 102-Af is input to a linear mapping layer 104 that resizes the number of columns according to the number of future time points (tfuture) for which predictions are to be made. Thus, the final output (or forecast) 120 can be a matrix of dimension n X tfuture, where tfuturecan be the same as or different from tpast. In some embodiments, machine learning model 100 incorporates some or all of the following: identity mapping connections achieved with a pre-activation design; random projection layers to increase diversity among the outputs of different mixer blocks; and processing of time series in the frequency domain using complex linear layers. Examples are described below.Mixer Block Components

[0032] FIG. 2 shows a block diagram of a mixer block 200 that can be used in machine learning model 100 according to some embodiments. In some embodiments, each of mixer blocks 102-1 through 102- can be a separate instance of mixer block 200, and in some respects, mixer blocks 200, as used in model 100, can serve a similar function to base learners in an ensemble learning model. Input to mixer block 200 can be a matrix 202 of dimension n X tpast, and output of mixer block 200 can be a (modified) matrix 204 of the same dimensions as input matrix 202. In machine learning model 100, the input matrix 202 for first mixer block 102-1 is the input data 110; the input matrix 202 for each of mixer blocks 102-2 through 102-A / is the block-level result 112-(z— 1) through 1 12- / V7 of the previous mixer block 102-1 through 102-(A — 1).

[0033] Mixer block 200 includes a temporal mixer 210 followed by a spatial mixer 212. Temporal mixer 210 can include a ReLU activation function 214 applied to each element of the input matrix 202 and a temporal linear layer 216, which can be implemented as a fully connected weight matrix with trainable weights. Temporal mixer 210 can linearly combine representations of the same node at different times. In some embodiments, temporal linear layer 216 can be a complex-valued layer that operates in the frequency domain to capture periodic features of the input data; an example of a frequency-domain implementation of temporal linear layer 216 is described below. Weights for temporal linear layer 216 can be learned during training.

[0034] Spatial mixer 212 can include a random projection layer 218 followed by a spatial linear layer 220. In some embodiments, random projection layer 218 is preceded by a matrix transpose operation 222 and a ReLU activation function 224. Spatial linear layer 220 can be preceded by a ReLU activation function 226 and followed by a matrix transpose operation 228. Transpose operation 222 allows spatial mixer 212 to linearly combine representations of different nodes at a given time step, rather than representations of the same node at different times, and transpose operation 228 restores the original dimensions of the matrix prior to the next mixer block (or final linear layer).

[0035] Random projection layer 218 can be implemented as a rectangular matrix of fixed weights having dimension n x nrand, where nrandis less than n. The number nrandis a configuration parameter that can be chosen as desired. In some embodiments, nrandcan scale with / n. Different mixer blocks in the same machine learning model 100 can have thesame or different values for nrandas desired. Unlike other layers in the machine learning model, random projection layer 218 has fixed weights (in other words, weights that are not updated during training). In some embodiments, each element of the matrix implementing random projection layer 218 can be selected during initial setup of the machine learning model by randomly sampling a statistical distribution such as a Gaussian distribution. Where a machine learning model includes multiple instances of mixer block 200, each random projection layer 218 can be independently generated so that different random projection layers 218 have different weights (and optionally different values of nrand).

[0036] Each random projection layer 218 can be a fixed, randomly-initialized linear layer that performs random projection on the input. If the input matrix is viewed as a number tpastof vectors of dimension n, random projection layer 218 can compute nrandrandom combination of the vector elements to form new vectors of dimension nrand. Keeping the weights for random projection layer 218 fixed during training has the effect of encouraging different mixer blocks 200 to concentrate on different nodes of the network, provided that the randomly initialized weights for random projection layers 218 in different mixer blocks 200 are different.

[0037] Use of random projection layers 218 can improve model efficiency by reducing the vector dimension from n to a smaller nrand. The reduction may be particularly significant when dealing with spatial-temporal data with a large number of nodes.

[0038] It is expected that a random projection layer of the kind described herein will preserve the relationships between the tpastvectors with high probability and that the dynamics of the overall spatial -temporal data will largely be retained, even when projected into a significantly smaller space. FIG. 3 shows graphs illustrating the preservation of daily and weekly patterns in a random projection layer according to some embodiments. Data was drawn from the Caltrans PeMS dataset, which is part of the publicly-available LargeST Benchmark Dataset (available via https: / / github.com / liuxu77 / LargeST; described in X. Liu et al., “LargeST: A Benchmark Dataset for Large-Scale Traffic Forecasting,” arXiv:2306.08259 [cs.LG] (October 28, 2023)). Nine nodes from the Caltrans PeMS dataset were sampled to form a nine-dimensional time series, represented by nine plots 302, where each plot represents relative traffic volume as a function of time. Random projection was used to reduce the nine-dimensional time series to a three-dimensional time series, represented by three plots 304. As can be seen, daily and weekly patterns in the time plots were preserved.

[0039] Referring again to FIG. 2, spatial linear layer 220 can be implemented as a fully connected weight matrix with trainable weights that operates on a matrix 230 of reduced dimension tpastX nrandto produce an output matrix of dimension tpastX n. Following spatial linear layer 220, transpose operation 228 yields an output matrix 204 of dimension n X tpast, matching the dimensions of input matrix 202 and facilitating the sequential operation of multiple mixer blocks 200.

[0040] In some embodiments, an identity mapping connection can be provided between layers of mixer block 200. As shown in FIG. 2, identity mapping can be supported by addition operations 232, 234. Addition operation 232 adds the input matrix 202 to the output of complex linear layer 216 at the output of temporal mixer 210, and spatial mixer 212 receives the output of addition operation 232. Similarly, addition operation 234 adds the input of spatial mixer 212 to the output of transpose operation 228. This creates an identity mapping connection as indicated by line 236. Examples of implementations and advantages of such identity mapping connections are discussed below.

[0041] Each layer in mixer block 200 can be fully connected; for instance, each layer can be a weight matrix of appropriate dimensions. Weights for all layers except random projection layer 218 can be learned using a training process as described below. Other activation functions can be substituted for ReLU.

[0042] As noted above, in some embodiments, temporal mixer 210 can operate in the frequency domain. FIG. 4 shows a block diagram of a layer structure 400 that can be used to implement temporal linear layer 216 in temporal mixer 210 according to some embodiments. Layer structure 400 includes a Fourier transform module 402, a first frequency-domain layer (also referred to as a “linear real layer”) 404, a second frequency-domain layer (also referred to as a “ linear imaginary layer”) 406, and an inverse Fourier transform module 408. Fourier transform module 402 and inverse Fourier transform module 408 can be implemented using Fast Fourier Transform (FFT) or other discrete Fourier transform algorithms. The output of Fourier transform module 402 is generally complex-valued, with a real part and an imaginary part. As shown, both the real and imaginary parts can be provided to first frequency-domain layer 404 and to second frequency-domain layer 406. First frequency-domain layer 404 can contain the real parts of the model weights, while second frequency-domain layer 406 contains the imaginary parts of the model weights. Outputs of first frequency-domain layer 404 and second frequency-domain layer 406 are recombined using arithmetic operations asshown to produce a complex- valued intermediate matrix on which inverse Fourier transform module 408 operates to produce a time-domain output.

[0043] The design of layer structure 400 reflects the fact that, unlike regular linear layers, the weights for a complex linear layer are complex numbers. For instance, input data x is converted to frequency domain using a Fourier transform to obtain frequency-domain input data xreai+ ximagi. This frequency-domain input data is to be multiplied with a complex weight matrix Wreai+ Wimagi. This multiplication can be expressed as:In layer structure 400, arithmetic operators 411 (subtraction), 412 (addition), 413 (multiplication by i, the imaginary unit), and 414 (addition) implement the algebra of Eq. (1).Identity Mapping ConnectionReferring again to FIG. 2, mixer block 200 incorporates an identity mapping connection (line 236). Some benefits of incorporating this identity mapping connection can be understood by regarding model 100 as analogous to an ensemble learning model, with the identity mapping connection facilitating the creation of shorter paths. For instance, weighted paths for temporal mixer 210 and spatial mixer 212 can be defined as:Ftemp(X):= COMPLEXLlNEAR(ReLU(X)) , (2)Given an input X, the operation of mixer block 200, including addition operations 232, 234, can be expressed as:Mixer(Defining a function G as:the mixer block operation of Eq. (4) can be simplified to:Mixer (X) = G(X) + X . (6)In Eq. (6), the first term represents the weighted path and the second term is the identity mapping connection (indicated by line 236 in FIG. 2).

[0044] The analogy to an ensemble model is illustrated in FIGs. 5 A and 5B. FIG. 5 A shows a high-level block diagram of a mixer model 500 according to some embodiments. Mixer model 500 is an implementation of mixer model 100, with three mixer blocks 502-1 through 502-3, and an output linear layer 504. Each mixer block 502-z computes a function Gtas defined in Eq. (5), and addition operations 534-1 through 534-3 correspond to addition operations 234 in mixer block 200. A forward pass through mixer model 500 can be unraveled into multiple paths, as shown in FIG. 5B. The outputs of mixer blocks 502-1, 502- 2, and 502-3 are denotedrespectively. Output Y can be expressed as:Y = D(X + H1+ H2+ H3) (7)Since D is a linear function, Eq. (7) can be rewritten as:Defining Fo:= £)(%), Fithe output (prediction) Y can be understood as the sum of the predictions from each path, that is Y = Yo+ Y + Y2+ f3. In this manner, the identity mapping connections in mixer blocks 200 can provide ensemble-like behavior of machine learning model 100.Operation of the Model

[0045] Training of a machine learning model such as machine learning model 100 involves automated processes to determine, or “learn,” optimal values for internal parameters of the model, such as the weights for each node or coefficients of a parametric function such as a curve-fitting function or a transform function. A standard approach to training of MLP machine learning models involves iteratively processing data samples through the model and adjusting the parameters of the model, with the goal of minimizing a loss function that characterizes a difference between the output of the model for a given input and an expected (or ground truth) result determined from a source other than the model. Loss functions can be selected based in part on the particular model, and optimization of loss functions can proceed using various techniques. For example, machine learning model 100 can be trained to “predict the past.” That is, the training data can include measured data covering a range of times. A time point within the range can be selected as a nominal “present” time, separating the data into nominal “past” and nominal “future.” Nominal past data is used as input, andnominal future data is used as ground truth, to which predictions (forecasts) output from the model can be compared. A loss function can be defined based on a distance metric between the prediction and the ground truth Training typically occurs across multiple “epochs,” where each epoch corresponds to a pass through the training sample set. Adjustment to parameters of the model (e.g., weights or coefficients) can occur multiple times during an epoch; for instance, the training data can be divided into “batches” or “mini-batches” and weight adjustment can occur after each batch or mini-batch. Aspects of machine learning models and training that are relevant to understanding the present disclosure are described herein; any other aspects can be modified as desired.

[0046] After training, machine learning model 100 can be used in an “inference” mode to generate predictions from previously unseen input data sets. FIG. 6 shows a flow diagram of a process 600 for operating machine learning model 100 in inference mode according to some embodiments. Process 600 can be implemented using any computer system with sufficient computational capability. At block 602, input spatial-temporal data is obtained. The input spatial-temporal data can include data obtained from a network having some number of nodes and can include a measurement associated with each of the nodes at each of a number of past times. For instance, in the case of vehicular traffic modeling, the network can be a road network, and each node can be a traffic counter (located a particular point along the roadway) that reports number of vehicles per some time interval (e.g., per hour or per 30 minutes); the past times can cover a range such as a week, a month, a calendar quarter (or three months), a year, or the like. In the case of a computer network, the nodes may correspond to different network components (e.g., servers, routers, clients), and the measurements can represent volume of traffic (e.g., data packets or bytes) or activity (e.g., CPU use, memory use) at that node. In some instances, data values for some nodes at some time points may be missing. In some embodiments, such missing data values can be set to a default value or reconstructed (e.g., using interpolation from adjacent time points). The input data can be presented as a matrix (such as matrix 110 of FIG. 1).

[0047] At block 604, the input spatial-temporal data can be provided to a machine learning model (e.g., machine learning model 100) that has been trained to forecast future measurements for each of the plurality of nodes in the network at a plurality of future times. As described above, the machine learning model can include a sequence of mixer blocks. Each mixer block can be an instance of mixer block 200, which applies a temporal mixer 210 followed by a spatial mixer 212. As described above, temporal mixer 210 can include atemporal linear layer (which can operate in the frequency domain), and spatial mixer 212 can include a random projection layer and a spatial linear layer that operates on an output of the random projection layer. Each mixer block 200 can produce a block-level result (e.g., output matrix 204 of FIG. 2). The block-level result from one mixer block can be provided as input to the next mixer block in the sequence. As shown in FIG. 2, identity mapping connections can be incorporated between layers and / or mixer blocks.

[0048] At block 606, output data is produced from the machine learning model. The output data can include a forecast (or prediction) of a measurement for each of the plurality of nodes at one or more future times, e.g., in the form of an output matrix 120. It should be understood that the number (tfuture) of future times for which a forecast is provided can be smaller or larger than the number of past times for which input data is provided. Where forecasts are made for multiple future times, the interval between future times can correspond to the interval between the past times in the input data. (For instance, if the input data represents number of vehicles per hour, then output data would also represent number of vehicles per hour.) Further manipulation of the output data can be performed as desired.

[0049] Output data (forecasts) can be used in a variety of applications, depending on the particular network or environment being modeled. For instance, in the case of a computer network, forecasts of future demand surges or lulls may be used to perform dynamic resource allocation. In the case of a road network, forecasts may be used to determine when to implement traffic control measures (e.g., metering lights or alternative traffic routing), when to perform construction or maintenance activities, priorities for future development, or the like.Example Implementations

[0050] To illustrate the effectiveness of machine learning models according to some embodiments, an implementation of machine learning model 100 was trained, and performance of the trained model was compared to a number of different conventional (baseline) models. Training data was obtained from the LargeST benchmark dataset, more specifically from the Caltrans PeMS dataset, which consists of traffic data collected between 2017 and 2021 from 8,600 sensors in California. Four sub-datasets were defined: “SD” (sensors located in the San Diego region); “GBA” (sensors located in the Greater Bay Area region), “GLA” (sensors located in the Greater Los Angeles region), and “CA” (all 8,600 sensors). For these studies, traffic data from 2019 was used. Sensor readings, originallyrecorded at 5-minute intervals, were aggregated into 15-minute windows, yielding 96 windows per day. Each sub-dataset was chronologically divided into training, validation, and test sets at a ratio of 6:2:2. The benchmark task was to predict the next 12 steps for each sensor at each timestamp. Performance was measured using mean absolute error (MAE), root mean squared error (RMSE), and mean absolute percentage error (MAPE).

[0051] Random projection mixer models were implemented leveraging PyTorch and a high-performance server system. The number of mixer blocks was configured to 8, and the random projection dimension was set to nrand= Vn, where n is the number of sensors (nodes) in the network. It was found that the random projection mixer models outperform conventional models in the benchmark task. For example, for the CA dataset, across the 12 time steps, a random projection mixer model yielded MAE of 17.50, RMSE of 28.90, and MAPE of 13.03%. By way of comparison, the conventional models that were tested yielded MAE of 19 or higher, RMSE of 33 or higher, and MAPE of 15% or higher. For the smaller datasets (SD and GBA), the random projection mixer model outperformed all conventional models that were tested, across all performance metrics and all time horizons. It is also noted that the memory complexity of the random projection mixer model scales linearly with respect to the number of nodes, making it practical to implement for the full CA dataset, unlike some conventional models.

[0052] FIG. 7 shows a table 700 of graphs illustrating performance of a random projection mixer model according to some embodiments. Each row corresponds to a different sample (node of the network), and each column corresponds to output at a different mixer block (first, third, fifth, seventh, and final (eighth) block.) The graph in each table cell shows plots for a series of twelve time steps of: (1) predictions from a mixer model with random projection layers according to an embodiment (labeled “proposed”); (2) predictions from a mixer model without random projection layers (labeled “w / o random”); and (3) the ground truth (labeled “ground truth”). It should be noted that ground truth for a given row (node) does not depend on number of blocks. As FIG. 7 shows, the random projection mixer model converges toward ground truth at the final block for all nodes, while the model without random projection layers shows significant error even after eight blocks.

[0053] Ablation studies were also performed to asses the effect of various design decisions, including: presence or absence of identity mapping connection; replacement of random projection layers with regular linear layers with trainable weights; and processing of the timeseries in time or frequency domain (e.g., removing the Fourier transforms). Removing any of these features resulted in some degree of performance degradation. For instance, as described above, the random projection layers can be viewed as analogous to weak learners in an ensemble model; removing the identity mapping connection weakens this behavior, degrading overall performance. Removing Fourier transforms had less effect, which may be because the Fourier transform is linear and much of the benefit it provides can therefore be learned during training.

[0054] Additional studies were performed related to the effect of hyperparameter choices, specifically the number of mixer blocks and the number of neurons in the random projection layers. For the SD, GBA, GLA, and CA datasets, it was found that eight mixer blocks was optimal; increasing the number to 16 provided minor performance improvements on some datasets, with the tradeoff being a doubling of the model size. Varying the number of neurons in the random projection layers was implemented by defining the number as mV where m is a scaling factor that was varied from 1.0 to 2.0. It was found that increasing m above 1.0 did not significantly improve performance, and small improvements might not justify the additional computational cost.

[0055] It should be understood that the example implementation is intended as illustrative and not limiting.Additional Embodiments

[0056] While the foregoing description makes reference to specific embodiments, those skilled in the art will appreciate that the description is not exhaustive of all embodiments. Many variations and modifications are possible. For example, embodiments described above implement frequency-domain processing for the time series, to exploit periodicity in the data. Many time series datasets (e.g., data tied to human activity such as travel) exhibit periodic characteristics; however, some time series data may be only weakly or not at all periodic. For modeling of non-periodic time series activity, frequency-domain processing might not be desirable. Further, even where periodic characteristics are present, it is possible for a machine learning model to learn such characteristics from time-domain data. Accordingly, in some embodiments, transforming to frequency domain (and back) can be omitted.

[0057] The number of mixer blocks and size of each mixer block can be varied, e.g., depending on characteristics of the data set, such as number of nodes in the network, amountof time series data available, and the like. As noted above the dimension of each random projection layer (nrand)canbe smaller than the number of nodes; for instance, the dimension can scale with the square root of the number of nodes. Weights for each random projection layer can be established by randomly sampling from a Gaussian distribution or other distribution. It is expected that each mixer block will have different weights for the random projection layer; dimensions of the random projection layers in different mixer blocks can be the same or different as desired. As noted above, weights for the random projection layers can be fixed at initialization time while other weights are learnable parameters of the model.

[0058] In various embodiments, training of the model can be ongoing, or the model can be retrained from time to time as new data becomes available. In some embodiments, old data points can expire and be removed from the training set prior to retraining. Further, while the data sets are described as matrices with rows corresponding to network nodes and columns corresponding to times, matrices can be transposed if desired. Training and forecasting can be specific to a particular network. To provide forecasting for multiple networks, multiple instances of a machine learning model as described herein can be constructed. (Further, for networks with a very large number of nodes, the network can be divided into sub-networks, each associated with a different instance of the machine learning model. This may reduce computational complexity.)

[0059] While traffic prediction is used as an example, those skilled in the art with the benefit of this disclosure will appreciate that models of the kind described herein can be used in a variety of forecasting applications. For example, techniques used herein can be used to construct machine learning models to predict data volume or transaction volume at nodes of a computer network (e.g., particular servers or IP addresses or the like). In addition, because the techniques described herein do not require or rely on adjacency or connectivity information between different nodes, these techniques can be applied to any type of multidimensional time series forecasting, without requiring any particular spatial or other relationship between the different quantities (nodes) whose time evolution is being predicted. Examples include forecasts related to currency exchange activity, consumer behavior (e.g., sales volume at various online and / or brick-and-mortar locations), multidimensional weather forecasting (e.g., temperature, precipitation, wind), and so on. Correlations between quantities or nodes can be learned by the machine learning model.

[0060] Techniques described herein can be implemented by suitable programming of general-purpose computers. In some embodiments, a computer system includes a single computer apparatus, where the subsystems can be components of the computer apparatus. The computer apparatus can have a variety of form factors including, e.g., a smart phone, a tablet computer, a laptop computer, a desktop computer, etc. In other embodiments, a computer system can include multiple computer apparatuses, each being a subsystem, with internal components. Techniques of the kind described herein can improve the performance of various tasks in which prediction (or forecasting) from spatial-temporal data sets is desired.

[0061] A computer system can include a plurality of components or subsystems, e.g., connected together by external interface or by an internal interface. In some embodiments, computer systems, subsystems, or apparatuses can communicate over a network. In such instances, one computer can be considered a client and another computer a server, where each can be part of a same computer system. A client and a server can each include multiple systems, subsystems, or components.

[0062] It should be understood that any of the embodiments of the present invention can be implemented in the form of control logic using hardware (e.g., an application specific integrated circuit or field programmable gate array) and / or using computer software with a generally programmable processor in a modular or integrated manner. As used herein a processor includes a single-core processor, multi-core processor on a same integrated chip, or multiple processing units on a single circuit board or networked. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will know and appreciate other ways and / or methods to implement embodiments of the present invention using hardware and a combination of hardware and software.

[0063] Any of the software components or functions described in this application may be implemented as software code to be executed by a processor using any suitable computer language such as, for example, Java, C, C++, C#, Objective-C, Rust, Golang, Swift, or scripting language such as Perl or Python using, for example, conventional or object-oriented techniques. The software code may be stored as a series of instructions or commands on a computer readable storage medium; suitable media include random access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a compact disk (CD) or DVD (digital versatile disk), flash memory,and the like. The computer readable storage medium may be any combination of such storage devices or other storage devices capable of retaining stored data.

[0064] Such programs may also be encoded and transmitted using carrier signals adapted for transmission via wired, optical, and / or wireless networks conforming to a variety of protocols, including the Internet. As such, a computer readable transmission medium according to an embodiment of the present invention may be created using a data signal encoded with such programs. Computer readable media encoded with the program code may be packaged with a compatible device or provided separately from other devices (e.g., via Internet download). Any such computer readable medium may reside on or within a single computer product (e.g. a hard drive, a CD, or an entire computer system), and may be present on or within different computer products within a system or network. A computer system may include a monitor, printer or other suitable display for providing any of the results mentioned herein to a user.

[0065] Any of the methods described herein may be totally or partially performed with a computer system including one or more processors, which can be configured to perform the steps. Thus, embodiments can involve computer systems configured to perform the steps of any of the methods described herein, potentially with different components performing a respective steps or a respective group of steps. Although presented as numbered steps, steps of methods herein can be performed at a same time or in a different order. Additionally, portions of these steps may be used with portions of other steps from other methods. Also, all or portions of a step may be optional. Additionally, and of the steps of any of the methods can be performed with modules, circuits, or other means for performing these steps.

[0066] The specific details of particular embodiments may be combined in any suitable manner without departing from the spirit and scope of embodiments of the invention. However, other embodiments of the invention may be involve specific embodiments relating to each individual aspect, or specific combinations of these individual aspects.

[0067] A recitation of “a”, “an” or “the” is intended to mean “one or more” unless specifically indicated to the contrary. The use of “or” is intended to mean an “inclusive or,” and not an “exclusive or” unless specifically indicated to the contrary.

[0068] All patents, patent applications, publications and description mentioned herein are incorporated by reference in their entirety for all purposes. None is admitted to be prior art.

[0069] The above description is illustrative and is not restrictive. Many variations of the invention will become apparent to those skilled in the art upon review of the disclosure. The scope of patent protection should, therefore, be determined not with reference to the above description, but instead should be determined with reference to the following claims along with their full scope or equivalents.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method comprising: obtaining input spatial-temporal data for a network having a plurality of nodes, the input spatial-temporal data including a measurement associated with each node of the plurality of nodes at each of a plurality of past times; providing the input spatial-temporal data to a machine learning model that has been trained to forecast future measurements for each of the plurality of nodes in the network at a plurality of future times, wherein the machine learning model includes a sequence of mixer blocks, wherein each mixer block applies a temporal mixer followed by a spatial mixer that includes a random projection layer and a spatial linear layer that operates on an output of the random projection layer to produce a block-level result; and producing output data from the machine learning model, the output data including a forecast of a measurement for each of the plurality of nodes at one or more future times.

2. The method of claim 1 wherein the machine learning model further includes a final linear mapping layer that operates on the block-level result from a last mixer block in the sequence of mixer blocks.

3. The method of claim 1 wherein the machine learning model further includes an identity mapping connection applied at each temporal mixer and each spatial mixer.

4. The method of claim 1 wherein, for each mixer block, the temporal mixer operates in the frequency domain.

5. The method of claim 4 wherein applying the temporal mixer includes: performing a Fourier transform on a received data matrix to produce a complex- valued input in a frequency domain; applying a first frequency-domain layer to a real part of the complex- valued input to produce a first output component; applying the first frequency-domain layer to an imaginary part of the complexvalued input to produce a second output component;applying a second frequency-domain layer to the real part of the complexvalued input to produce a third output component; applying the second frequency-domain layer to the imaginary part of the complex- valued input to produce a fourth output component; combining the first, second, third, and fourth output components to produce a frequency-domain output; and performing an inverse Fourier transform on the frequency-domain output.

6. The method of claim 1 wherein the random projection layer in each mixer block comprises a matrix of values randomly selected from a statistical distribution.

7. The method of claim 6 wherein the statistical distribution is a Gaussian distribution.

8. The method of claim 6 wherein, for at least one of the mixer blocks, the matrix has dimensions n X nrand, wherein n is equal to a number of nodes in the network and nrandis less than n.

9. The method of claim 6 wherein, for each of the mixer blocks, the matrix has dimensions n X nrand, wherein n is equal to a number of nodes in the network,nrand is less than n, and nrandis different for the random projection layers in different mixer blocks.

10. A computer system comprising: a memory to store input data including spatial-temporal data for a network having a plurality of nodes, the spatial-temporal data including a measurement associated with each node of the plurality of nodes at each of a plurality of past times; and a processor coupled to the memory and configured to implement a machine learning model that operates on the spatial-temporal data to produce output data including a forecast measurement for each of the plurality of nodes at each of a plurality of future times, wherein the machine learning model includes: a plurality of fully connected layers arranged in a plurality of mixer blocks, each mixer block including a complex linear temporal mixer followed by a spatial mixer that includes a random projection layer and a spatial linear layer that operates on an output of the random projection layer; anda final linear mapping layer that operates on an output of a last one of the mixer blocks.

11. The computer system of claim 10 wherein the machine learning model further includes an identity mapping connection applied at each temporal mixer and each spatial mixer.

12. The computer system of claim 10 wherein the temporal mixer is configured to operate in the frequency domain.

13. The computer system of claim 12 wherein the temporal mixer includes: a Fourier transform module to operate on an input matrix to produce a complex- valued transformed matrix; a first frequency-domain layer to operate on a real part and a complex part of the complex-valued transformed matrix to produce a first output component and a second output component; a second frequency-domain layer to operate on the real part and the complex part of the complex- valued transformed matrix to produce a third output component and a fourth output component; arithmetic circuits to combine the first, second, third, and fourth output components to produced a frequency-domain output; and an inverse Fourier transform module to operate on the frequency-domain output to produce a time-domain output.

14. The computer system of claim 10 wherein the random projection layer in each mixer block comprises a different matrix of values randomly selected from a statistical distribution.

15. The computer system of claim 10 wherein the spatial mixer includes: a first matrix transpose module that operates prior to the random projection layer; and a second matrix transpose module that operates after the spatial linear layer.

16. A computer-readable storage medium having stored therein program code instructions that, when executed by a processor in a computer system, cause the computer system to perform a method comprising:obtaining input spatial-temporal data for a network having a plurality of nodes, the input spatial-temporal data including a measurement associated with each node of the plurality of nodes at each of a plurality of past times; providing the input spatial-temporal data to a machine learning model that has been trained to forecast future measurements for each of the plurality of nodes in the network at a plurality of future times, wherein the machine learning model includes a sequence of mixer blocks, wherein each mixer block applies a temporal mixer followed by a spatial mixer that includes a random projection layer and a spatial linear layer that operates on an output of the random projection layer to produce a block-level result; and producing output data from the machine learning model, the output data including a forecast of a measurement for each of the plurality of nodes at one or more future times.

17. The computer-readable storage medium of claim 16 wherein the machine learning model further includes a final linear mapping layer that operates on the block-level result from a last mixer block in the sequence of mixer blocks.

18. The computer-readable storage medium of claim 16 wherein the machine learning model further includes an identity mapping connection applied at each temporal mixer and each spatial mixer.

19. The computer-readable storage medium of claim 16 wherein, for each mixer block, the temporal mixer operates in the frequency domain.

20. The method of claim 1 wherein the random projection layer in each mixer block comprises a matrix of values randomly selected from a Gaussian distribution.

Images