Efficient time series forecasting in large scale multivariate time series data

Abstract

Methods, systems, and machine learning models for efficiently performing time series forecasting for multivariate time series datasets are disclosed, particularly multivariate time series datasets that comprise a large number of variates (or "time series signals") and for which time series signals may emerge and disappear over the sampling time period. A computer system according to embodiments can generate signal embeddings corresponding to time series signals in a multivariate time series dataset and use an efficient (e.g., linear) latent attention module to generate an attention matrix. The computer system can then generate a time series forecast using the attention matrix and a forecasting model. Methods according to embodiments can also involve the use of random projection, reducing the risk of overfitting and improving accuracy for large and complex spatial-temporal datasets.

Description

PATENT Attorney Docket No.: 079900-1513063 Client Reference No.: 9719WO01 EFFICIENT TIME SERIES FORECASTING IN LARGE SCALE MULTIVARIATE TIME SERIES DATA BACKGROUND

[0001] This is a PCT application which claims priority to U.S. Provisional Application No.63 / 701,200, filed on September 30, 2024, which is herein incorporated by reference. BACKGROUND

[0002] A “time series” generally comprises a chronologically ordered series of data values. For example, an electrical utility may record a time series of the electrical power demand for a particular service region at a periodic rate. A “multivariate time series” can comprise a time series where multiple data values can be recorded in the same time period. For example, a multivariate time series could record both the temperature and electrical power demand over a period of time. As another example, a multivariate time series could record the power demand associated with 1000 different households. Spatial-temporal time series data is a type of multivariate time series data, in which data corresponds to both time and space, including both physical spaces (e.g., the traffic volume at different physical locations within a given city) and mathematical spaces. For example, different households that receive power from an electrical utility could be considered points in a “household space”, and thus time series power demand data for such households can comprise spatial-temporal data.

[0003] Time series forecasting generally relates to the task of predicting future time series data values based on historical time series data values. As an example, an electrical utility may predict future demand for electricity based on a time series recording historical demand for electricity. Forecasting is useful as it enables people and organizations to prepare for events in the future, e.g., by increasing electrical power supply to accommodate increasing demand.

[0004] Machine learning can be used to perform time series forecasting. A machine learning model can be trained to take historical time series data values asan input and produce predicted time series data values as an output. Such predicted time series data values can correspond to a period of time in the future. Recently, “sequence machine learning models”, i.e., machine learning models that are designed to process data comprising sequential data elements (including time series), have shown strong performance in a variety of tasks. Sequence machine learning models may have a variety of underlying model architectures, including those based on the concept of model “attention”, e.g., “transformer” models.

[0005] However, transformer models can show poor performance for multivariate time series data, as the computational workload scales quadratically with either the length of time series data signals or the number of time series data signals, e.g., doubling the number of time series data signals increases processing time by a factor of four. Additionally, various existing time series forecasting machine learning models (including some transformer models) have difficulty in situations in which time series data signals may appear or disappear from multivariate time series data, e.g., a machine learning model may produce an inaccurate forecast for a new household for an electrical utility (appearance of a new time series data signal) or if an electrical power utility stops servicing a household (disappearance of a time series data signal). Generally, the lack of historical data for new time series signals, and correlations between existing time series signals and disappearing time series signals can lead to inaccurate predictions. For example, if there is a strong correlation between the electrical power usage at two restaurants (e.g., because those restaurants have similar hours of operation), termination of electrical power service at one restaurant could lead a machine learning model to inaccurately forecast electrical power usage at the other restaurant, even though their electrical power usage is independent.

[0006] As a result, many existing machine learning models do not perform well for multivariate time series forecasting, particularly those involving the appearance or disappearance of time series data signals. This can make it difficult for people and organizations to accurately predict and prepare for future events using transformer models.

[0007] Embodiments address these and other problems, individually and collectively. 2 KILPATRICK TOWNSEND 788665001SUMMARY

[0008] This Summary is provided to introduce a selection of concepts in a simplified form that are further described herein in the Detailed Description. This Summary is not intended to identify key factors or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0009] Embodiments of the present disclosure are directed to methods, systems, and related transformer-based machine learning models for efficient and accurate forecasting for multivariate time series data (e.g., spatial-temporal data), particularly multivariate time series data with a large number of dimensions, variates, or “time series signals” (e.g., tens of thousands or more). Unlike other transformer- based machine learning models (e.g., iTransformer

[0021] ), which scale quadratically with the number of time series signals, methods according to embodiments scale linearly with the number of time series signals and can be used to generate accurate time series forecasts in situations where other transformer machine learning models cannot (e.g., due to out-of-memory issues). As an example, and as discussed with reference to FIG.12B in the detailed description below, methods according to embodiments could be used to perform time series forecasting for multivariate time series comprising millions of time series signals, whereas previous machine learning techniques may encounter out of memory issues at around 10,000 time series signals. As a result, embodiments of the present disclosure are more scalable and better suited for large and complex multivariate time series datasets. The advantages of embodiments over previous machine learning based methods of time series forecasting are demonstrated by experiments, described in the Detailed Description below.

[0010] Methods according to embodiments employ various novel combinations of an “inverted transformer” approach, efficient latent attention, and random projection to perform accurate multivariate time series forecasting. In general terms, the inverted transformer approach involves generating embeddings for multivariate time series data based on individual time series signals rather than based on temporal “slices” or “patches”. As an example, if a multivariate time series comprises time series signals corresponding to electrical power usage by 200 3 KILPATRICK TOWNSEND 788665001households over 365 timestamps (e.g., days of a year), the “standard” transformer approach involves generating 365 embeddings, each corresponding to the electrical power usage by all households in a corresponding patch or slice (e.g., a corresponding day). By contrast, the inverted transformer approach can involve generating 200 embeddings, each corresponding to the electrical power usage of a single household over the entire time period (e.g., the entire year). In general, the use of inverted transformer approach enables the production of accurate forecasts even when time series data signals appear or disappear from multivariate time series data, as individual embeddings do not “mix” data from multiple time series signals, and are therefore less affected by the appearance or disappearance of individual time series signals.

[0011] While the inverted transformer approach has been applied to time series forecasting (see e.g., iTransformer

[0021] ), embodiments of the present disclosure are the first to employ the inverted transformer approach and the use of efficient latent attention and / or the use of random projection to the problem of multivariate time series data forecasting. These aspects of embodiments are described in more detail in the Detailed Description below. In general terms however, the use of efficient latent attention reduces the time and space complexity from quadratic to linear with respect to the number of time series data signals, and the use of random projection can enable machine learning models according to embodiments to capture a wide range of spatial-temporal patterns and reduce the risk of overfitting.

[0012] In more detail, one embodiments is directed to a method for generating a time series forecast using a machine learning model comprising a latent attention module and a forecasting module comprising a multilayer perceptron. The method can be performed by a computer system. The computer system can obtain a multivariate time series dataset comprising a plurality of time series signals. Each time series signal can comprise a plurality of time series data values corresponding to a plurality of time steps. The computer system can generate a signal embedding for each time series signal based on the plurality of time series data values in that time series signal. In this way, the computer system can generate a plurality of signal embeddings. Each signal embedding can comprise a number of embedding 4 KILPATRICK TOWNSEND 788665001values equal to an embedding dimension. The computer system can form a query comprising the plurality of signal embeddings. The query can comprise a matrix with the embedding dimension and a signal dimension equal to a number of time series signals in the plurality of time series signals. The computer system can generate a latent attention matrix using the latent attention module, the query, a key matrix and a value matrix. The key matrix can have a key embedding dimension equal to the embedding dimension and a key input-independent dimension. The key input- independent dimension can be less than the signal dimension of the query. The value matrix can have a value embedding dimension equal to the embedding dimension and a value input-independent dimension equal to the key input- independent dimension. The computer system can generate a time series forecast corresponding to the multivariate time series dataset using the latent attention matrix and the multilayer perceptron of the forecasting module.

[0013] Another embodiment is directed to a method for training a machine learning model comprising a latent attention module and a forecasting module comprising a multilayer perceptron, the method comprising performing an iterative training process until a terminating condition has been met. The method can be performed by a computer system. The computer system can sample a multivariate training time series dataset comprising a plurality of training time series signals. Each training time series signal can comprise a plurality of training time series data values and a plurality of ground truth forecast time series data values corresponding to a plurality of training time steps. The computer system can generate a training signal embedding for each training time series signal based on the plurality of training time series data values in that training time series signal. In this way, the computer system can generate a plurality of training signal embeddings. Each training signal embedding can comprise a number of training embedding values equal to an embedding dimension. The computer system can form a training query comprising the plurality of training signal embeddings, the training query can comprise a training matrix with the embedding dimension and a signal dimension equal to a number of training time series signals in the plurality of training time series signals. The computer system can generate a training latent attention matrix using the latent attention module, the training query, a key matrix, and a value matrix. The key matrix can have a key embedding dimension equal to the embedding dimension 5 KILPATRICK TOWNSEND 788665001and a key input-independent dimension. The key input-independent dimension can be less than the signal dimension of the training query. The value matrix can have a value embedding dimension equal to the embedding dimension and a value input- independent dimension equal to the key input-independent dimension. The computer system can generate a training time series forecast using the training latent attention matrix and the multilayer perceptron of the forecasting module. The computer system can determine a loss value by comparing the training time series forecast and the plurality of ground truth forecast time series data values. The computer system can update a parameter set of the machine learning model based on the loss value, thereby training the machine learning model. The computer system can repeat the iterative training process until the terminating condition has been met.

[0014] In addition to the methods described above (and other methods), some embodiments are directed to computer systems or other devices that can be configured to perform the methods described above or other methods. For example, one embodiment is directed to a computer system comprising a processor and a non-transitory computer readable medium coupled to the processor. The non- transitory computer readable medium can comprise instructions executable by the processor for performing the methods described above (or other methods described in the detailed description below).

[0015] A better understanding of the nature and advantages of embodiments of the present disclosure may be gained with reference to the following detailed description and the accompanying drawings. Reference to the remaining portions of the specification, including the drawings and claims, will realize other features and advantages of the present disclosure. Further features and advantages of the present disclosure, as well as the structure and operation of various embodiments of the present disclosure, are described in detail below with respect to the accompanying drawings. In the drawings, like reference numbers can indicate identical or functionally similar elements. 6 KILPATRICK TOWNSEND 788665001TERMS

[0016] A “server computer” may refer to a computer or cluster of computers. A server computer may be a powerful computing system, such as a large mainframe. Server computers can also include minicomputer clusters or a group of servers functioning as a unit. In one example, a server computer can include a database server coupled to a web server. A server computer may comprise one or more computational apparatuses and may use any of a variety of computing structures, arrangements, and compilations for servicing requests from one or more client computers.

[0017] A “client computer” may refer to a computer or cluster of computers that receives some service from a server computer (or another computing system). The client computer may access this service via a communication network such as the Internet or any other appropriate communication network. A client computer may make requests to server computers including requests for data. As an example, a client computer can request a video stream from a server computer associated with a movie streaming service. As another example, a client computer may request data from a database server. A client computer may comprise one or more computational apparatuses and may use a variety of computing structures, arrangements, and compilations for performing its functions, including requesting and receiving data or services from server computers.

[0018] A “memory” may refer to any suitable device or devices that may store electronic data. A suitable memory may comprise a non-transitory computer readable medium that stores instructions that can be executed by a processor to implement a desired method. Examples of memories including one or more memory chips, disk drives, etc. Such memories may operate using any suitable electrical, optical, and / or magnetic mode of operation.

[0019] A “processor” may refer to any suitable data computation device or devices. A processor may comprise one or more microprocessors working together to achieve a desired function. The processor may include a CPU that comprises at least one high-speed data processor adequate to execute program components for executing user and / or system generated requests. The CPU may be a 7 KILPATRICK TOWNSEND 788665001microprocessor such as AMD’s Athlon, Duron and / or Opteron; IBM and / or Motorola’s PowerPC; IBM’s and Sony’s Cell processor; Intel’s Celeron, Itanium, Pentium, Xenon, and / or Xscale; and / or the like processor(s).

[0020] A “feature” can be an individual measurable property or characteristic of a phenomenon. One or more features can be described using a “feature vector,” e.g., a structured list of data (such as numerical data) representing those features. A feature can be input into a model to determine an output. As an example, in pattern recognition and machine learning, a feature vector can comprise an n-dimensional vector of numerical features that represent some object. In some machine learning contexts, a numerical representation of objects can facilitate processing and statistical analysis. For image processing, for example, feature values might correspond to the pixels of an image. As another example, when feature vectors represent text, the features may comprise occurrence frequency of textual terms. Feature vectors can be equivalent to the vectors of explanatory variables used in statistical procedures such as linear regression.

[0021] A “dataset” may include any set of one or more “observations” or “data values.” A “data value” can include any data element. A “data element” can refer to a set of data that can be grouped into a single unit, enabling comparison between that data element and other data elements. For example, a data element can comprise a single numerical value (e.g., the speed of a vehicle in miles per hour) or could comprise multiple numerical values (e.g., 60 speed recordings of a vehicle corresponding to each minute of an hour-long period). Data elements comprising multiple data values can be organized into various forms or structures, including “data tables,” e.g., comprising data elements organized in rows and columns (or any other suitable structure). A data element may comprise the input to a machine learning model, and individual data values within that data element may comprise features values. A data value can comprise a “data vector,” one or more values (represented in vector form) corresponding to a data element or observation.

[0022] A “time series” or “time series dataset” may refer to a chronologically ordered sequence of data values or observations. Such data values or observations can correspond to one or more quantities. For example, a time series can correspond to an individual’s height (measured in a quantity of centimeters) and 8 KILPATRICK TOWNSEND 788665001weight (measured in a quantity of kilograms) over a period of time. In some cases, there can be equal intervals of time between successive data values or observations, for example, each observation in a time series may be spaced one month apart in time. In such a case, the rate at which data or observations are collected or sampled can be referred to as the sampling rate. Each data value or observation may be associated with a timestamp, time value, or index, which may enable the chronological ordering of the data values in the time series. A “segment” or “subsequence” of time series data can comprise a subset of consecutive data values from a time series dataset. Segments may be produced, identified, or defined using timestamps, time values, or indices.

[0023] “Sampling” may include any process or method used to collect data values. Sampling can be used to collect data values from an existing dataset. The act of sampling may result in a “sample,” one or more data values collected from the dataset during sampling. Datasets can be sampled via a variety of means. For example, “random sampling” involves sampling data values from a dataset randomly.

[0024] A “forecast” may refer to a prediction or estimate of future events or trends. “forecast data” or a “forecast dataset” may refer to data corresponding to such a prediction or estimation. For example, the predicted outdoor temperature in Fahrenheit or Celsius from a weather forecast may comprise forecast data. A “time series forecast” may refer to a forecast that comprises a time series (or a time series dataset, a dataset comprising time series data, etc.).

[0025] The term “artificial intelligence model” or “machine learning model” can include a model that may be used to predict outcomes to achieve a pre-defined goal. A machine learning model may be developed using a learning process, in which training data is classified based on known or inferred patterns.

[0026] "Machine learning" can include an artificial intelligence process in which software applications may be trained to make accurate predictions through learning. The predictions can be generated by applying input data to a predictive model (or “prediction model”) formed from performing statistical analyses on aggregated data. A model can be trained using training data, such that the model may be used to make accurate predictions. The prediction can be, for example, a 9 KILPATRICK TOWNSEND 788665001classification of an image (e.g., identifying images of cats on the Internet), or as another example, a recommendation (e.g., a movie that a user may like or a restaurant that a consumer might enjoy).

[0027] A “machine learning model” (ML model) can refer to a software module configured to be run on one or more processors to provide a classification or numerical value of a property of one or more samples. An ML model can include various parameters (e.g., for coefficients, weights, thresholds, functional properties of function, such as activation functions). As examples, an ML model can include at least 10, 100, 1,000, 5,000, 10,000, 50,000, 100,000, or one million parameters. An ML model can be generated using sample data (e.g., training samples) to make predictions on test data. Various numbers of training samples can be used, e.g., at least 10, 100, 1,000, 5,000, 10,000, 50,000, 100,000, or at least 200,000 training samples.

[0028] A machine learning model can comprise an “unsupervised learning model”. Examples of unsupervised learning models include hidden Markov model (HMM), clustering (e.g., hierarchical clustering, k-means, mixture models, model- based clustering, density-based spatial clustering of applications with noise (DBSCAN), and OPTICS algorithm), approaches for learning latent variable models such as Expectation–maximization algorithm (EM), method of moments, and blind signal separation techniques (e.g., principal component analysis, independent component analysis, non-negative matrix factorization, singular value decomposition), and anomaly detection (e.g., local outlier factor and isolation forest).

[0029] A machine learning model cam comprise a “supervised learning model”. Example supervised learning models may include different approaches including analytical learning, statistical models, artificial neural network (e.g. including convolutional and / or transformer layers) that may have 1-10 layers as examples, recurrent neural network (e.g., long short term memory, LSTM), boosting (meta-algorithm), bootstrap aggregating (bagging) such as random forests, support vector machine (SVM), support vector (SVR), Bayesian statistics, case-based reasoning, decision tree learning, inductive logic programming, linear regression, logistic regression, Gaussian process regression, genetic programming, group method of data handling, kernel estimators, learning automata, learning classifier 10 KILPATRICK TOWNSEND 788665001systems, minimum message length (decision trees, decision graphs, etc.), multilinear subspace learning, naive Bayes classifier, maximum entropy classifier, conditional random field, nearest neighbor algorithm, probably approximately correct learning (PAC) learning, ripple down rules, a knowledge acquisition methodology, symbolic machine learning algorithms, subsymbolic machine learning algorithms, minimum complexity machines (MCM), ordinal classification, data pre-processing, handling imbalanced datasets, statistical relational learning, or Proaftn (a multicriteria classification algorithm), or an ensemble of any of these types. Supervised learning models can be trained in various ways using various cost / loss functions that define the error from the known label (e.g., least squares and absolute difference from known classification) and various optimization techniques, e.g., using backpropagation, steepest descent, conjugate gradient, and Newton and quasi- Newton techniques.

[0030] The process of “training” a machine learning model may include any steps used to prepare a machine learning model to perform some task. Often training involves determining or optimizing a set of “parameters” (which characterize the machine learning model) that result in acceptable model performance. Training can be performed in a series of “training rounds” during which training data is used to update the parameters of the machine learning model, for example, based on a loss value.

[0031] A “loss value” or “error value” may include any value that indicates the deviation between a result of some process, method, or function and an expected, desired, or correct result. For example, if a machine learning model can detect anomalies in a dataset comprising 100 data values, 17 of which are anomalous, if the machine learning model only detects 15 of the 17 anomalous data values, the loss value could comprise, e.g., 2 (17 – 15). Loss values can be used to train and evaluate the training of machine learning models, e.g., by optimizing machine learning model parameters by minimizing the loss value, using processes such as stochastic gradient descent or backpropagation.

[0032] A “hyperparameter” can include any value used to configure a machine learning model that is external to the machine learning model. Typically, a 11 KILPATRICK TOWNSEND 788665001hyperparameter is set, and is not estimated or determined from the training data that is used to train the machine learning model.

[0033] A machine learning model may comprise multiple “sub-models”, “layers,” or “modules”, which may refer to parts of a larger machine learning system. For example, a machine learning model could comprise a long short-term memory layer (which itself can comprise multiple layers), in addition to an attention layer and a linear layer. Layers can be organized in series, such that the input to a machine learning system is processed by a first set of layers, which produces an output that is then processed by a subsequent set of layers, and so forth until the output of the machine learning model is produced by the final layer in the series.

[0034] An “embedding” can refer to a representation of data, usually within an “embedding space”, a theoretical region in which embeddings can be compared via vector operations. For example, an embedding can comprise a vector representation of an image, which can be used to evaluate the similarity of that image to other images, or e.g., determine whether that image contains or depicts a particular subject (e.g., a cat). Embeddings can be used within the field of machine learning, thereby enabling machine learning models to perform certain tasks, particularly tasks that are difficult or subjective. In some cases, an embedding can comprise a lower-dimensional representation of a corresponding element of data, enabling more efficient processing due to the reduced logical size of the embedding relative to the logical size of the original element of data.

[0035] An “encoder” can refer to something (e.g., a physical device, software element, component of a machine learning model, etc.) that produces output data representative of some input, usually in the form of a “code” or signal. An encoder can be used to convert data from one format to another, which may facilitate data processing. For example, a nominal data encoder can be used to convert nominal data (such as the name of a city) into a numeric code, facilitating such data to be processed using numerical methods. As another example, a physical rotary encoder can be used to convert the motion or position of a shaft or axle into electrical signals, facilitating computer-based odometry. A code (or other output of an encoder) can be used as an embedding. Some encoders can be implemented via applications of 12 KILPATRICK TOWNSEND 788665001machine learning, e.g., via a linear layer, a “transformer” machine learning model or via another model that uses the principle of machine learning “attention.”

[0036] An “interaction” can refer to a reciprocal action or influence or an event or phenomenon corresponding to such an action or influence. As an example, an individual posting a comment on a social media website could be considered an interaction between that individual (or e.g., a personal computer system associated with that individual) and the social media website (or e.g., a server computer hosting that social media website). As another example, the collision of a fast-moving cold air front and a slow-moving warm air front (e.g., an “occluded front”) could be considered an interaction between the two air fronts. As another example, a credit card transaction between a cardholder and a business could be considered an interaction. In some cases, measurable properties of interactions can be analyzed or evaluated (e.g., using machine learning) to quantify or qualify those interactions.

[0037] An “access device” may be any suitable device that provides access to a remote system. An access device may also be used for communicating with a resource provider computer, an authorizing computer, or any other suitable system. An access device may generally be located in any suitable location, such as at the location of a resource provider or merchant. An access device may be in any suitable form. Some examples of access devices include POS or point of sale devices (e.g., POS terminals), cellular phones, PDAs, personal computers (PCs), table PCs, hand-held specialized readers, set-top boxes, electronic cash registers (ECRs), automated teller machines (ATMs), virtual cash registers (VCRs), kiosks, security systems, access systems, terminals, and the like. An access device may use any suitable contact or contactless mode of operation to send or receive data from, or associated with, a user communication device. In some embodiments, where an access device may comprise a POS terminal, any suitable POS terminal may be used and may include a reader, a processor, and a computer-readable medium. A reader may include any suitable contact or contactless mode of operation. For example, exemplary card readers can include radio frequency (RF) antennas, optical scanners, bar code readers, or magnetic stripe readers to interact with a payment device and / or mobile device. Other examples of access devices include devices (e.g., locks, gates, access control boxes, etc.,) that control physical 13 KILPATRICK TOWNSEND 788665001access to locations (e.g., venues, transit stations, homes, offices, buildings, etc.,) as well as software devices that control access to data or information. In embodiments, the access device may be configured to charge or recharge a user device via contact or contactless modes of operation.

[0038] A “resource” can generally refer to something that can be acquired or provided to an entity, e.g., by a “resource provider.” Examples of resources include material resources, such as iron, monetary resources, such as dollars, or other resources such as cleaning supplies, clothing, food, etc. Resources may also refer to services, such as cleaning services. Access to something may also qualify as a resource, e.g., access to a secure building. Resources can also include data stored on a computer, such as video files, audio files, databases, data tables, etc. A resource provider can operate a “resource provider computer”, e.g., an entity associated with a digital streaming service could operate a resource provider computer (e.g., a server computer).

[0039] The term “access request” generally refers to a request to access a resource, e.g., provided by a resource provider. An access request and a transaction can be used interchangeably. The access request may be received from a requesting computer, a user device, or a resource computer, for example. The access request may include authorization information, such as a username, account number, or password. The access request may also include access request parameters, such as an access request identifier, a resource identifier, a timestamp, a date, a device or computer identifier, a geo-location, or any other suitable information. The access requests can be received in real time. For example, a user of a user device (e.g., a client computer such as a laptop, smartphone, or tablet) can use that user device to make an access request to a server computer, e.g., to access a service provided by that server computer (e.g., an image hosting service, a video streaming service, an online banking service, etc.), e.g., by transmitting the access request to the server computer over the Internet. In some cases, an access request may be made via an “access device”, which may comprise any device used to access something (e.g., a network or a computer system). This could include a user device as described above, but could also include, e.g., a point-of-sale terminal used to gain access to a payment processing network. An access device may 14 KILPATRICK TOWNSEND 788665001include a means by which it can interface with other devices. For example, a point- of-sale terminal access device may include a chip card reader that can interface with a smartcard operated by a user. An access request may include a “credential” or “identifier”, which may comprise data used to demonstrate or prove that an entity is authorized to access a resource, or which may comprise data used to identify the entity, such that an access device (or another device or computer system, e.g., an “authentication computer” or “authorization computer”) can authenticate the entity (e.g., by verifying its identity) and verify that the entity is authorized to access the resource.

[0040] The term “access request outcome” may include any determination of whether to grant access to the resource. The access request outcomes may include “accept,” “reject,” or “review.” In some embodiments, an access request outcome for an access request may be “reject” if any of the access rules have an access rule outcome of “reject.” In some embodiments, an access request outcome for an access request may be “accept” if any of the access rules have an access rule outcome of “accept,” regardless of any outcome being “reject.” An access request outcome of “accept” may cause the access request to be granted. An access request outcome of “reject” may cause the access request to be denied. The “review” outcome may initiate a review process for the access request. In various embodiments, other outcomes or other meanings of these outcomes may be used. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] FIG.1 shows an exemplary multivariate time series dataset in which time series signals appear or disappear over a time period.

[0042] FIG.2 shows a diagram of an example machine learning model architecture (which includes an encoder) according to some embodiments.

[0043] FIG.3 shows a diagram of a standard attention machine learning model architecture.

[0044] FIG.4 shows an example client-server system that can be used to perform machine learning methods according to embodiments. 15 KILPATRICK TOWNSEND 788665001

[0045] FIG.5 shows a diagram of a machine learning model architecture according to some embodiments.

[0046] FIG.6 shows a flowchart of an example machine learning model training method according to some embodiments.

[0047] FIG.7 shows a flowchart of an example inference method according to some embodiments.

[0048] FIG.8 shows a bar graph detailing the number of time series signals in various multivariate datasets and subsets used in experiments, including four regional subsets of the LargeST

[0020] dataset and a processing network dataset.

[0049] FIGs.9A-9B show a table comparing experimental performance for various models (including machine learning models according to embodiments) on the LargeST dataset.

[0050] FIG.10 shows a table providing a forecasting performance comparison for various models on a processing network dataset.

[0051] FIGs.11A-11B show a qualitative comparison of various time series forecasting methods for various experimental time series signals, including time series forecasting methods according to embodiments.

[0052] FIGs.12A and 12B show graphs detailing experimental results related to the computational cost associated with a forward pass for runtime and GPU memory usage for various numbers of time series signals.

[0053] FIGs.13A-13D show graphs detailing the relationship between mean absolute error (MAE) and learning rate for two machine learning models according to embodiments.

[0054] FIGs.14A-14D show graphs detailing the relationship between mean absolute error (MAE) and the number of neural network layers for two machine learning models according to embodiments.

[0055] FIGs.15A-15D show graphs detailing the relationship between mean absolute error (MAE) and the number of neurons for two machine learning models according to embodiments. 16 KILPATRICK TOWNSEND 788665001

[0056] FIGs.16A-16C show a table providing a benchmark comparing methods according to embodiments with established spatial-temporal baselines.

[0057] FIG.17 shows an example resource security system for authorizing access to resources in accordance with some embodiments.

[0058] FIG.18 shows a system block diagram of a computer system according to some embodiments. DETAILED DESCRIPTION

[0059] Before the present invention is described in greater detail, it should be understood that this invention is not limited to particular embodiments described, as such may vary. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting, since the scope of the present invention will be limited only by the appended claims. Efforts have been made to ensure accuracy with respect to numbers used (e.g., amounts, temperature, etc.) but some experimental errors and deviations should be accounted for.

[0060] As summarized above, embodiments of the present disclosure relate to multivariate (e.g., spatial-temporal) time series forecasting methods and systems using an inverted transformer approach, efficient latent attention, and random projection. Computer time and memory requirements scale linearly using methods according to embodiments, in contrast to quadratic scaling for many existing transformer-based time series forecasting methods, which is a large improvement in efficiency. As a result, methods according to embodiments can be used to generate forecasts for large-scale multivariate time series, comprising e.g., hundreds of thousands or millions of time series data signals, which are multivariate time series that existing time series forecasting methods may be unable to process due to time or memory constraints.

[0061] Additionally, embodiments of the present disclosure can produce accurate forecasts even for multivariate time series datasets with appearing or disappearing time series signals, e.g., enabling accurate modeling or forecasting of electrical power demand for new users of an electrical utility, transaction volume for 17 KILPATRICK TOWNSEND 788665001new merchants on a payment processing network, new users of a resource security system, etc. As described above, many existing multivariate time series forecasting methods fail to produce accurate forecasts for multivariate time series in which time series signals appear or disappear. FIG.1 generally illustrates the emergence or termination of time series data signals in a multivariate time series dataset, showing a continuously active signal 102 (i.e., a time series data signal for which there is meaningful time series data for the entirety of a time period), a late-emerging signal 104 (i.e., a signal that starts part way through the time period), and a terminating signal 106 (i.e., a signal that ends part way through the time period). In a processing network dataset used in experiments involving embodiments of the present disclosure, of 23,849 time series signals (corresponding to entities that are members of the processing network), 14,252 remained active throughout a two year period, while 1,974 emerged after the first week of the two year period and 5,880 ceased activity before the final week of the two year period. The large number of emerging and vanishing time series signals in real-world processing network data demonstrates the need for multivariate time series forecasting methods that can handle the appearance or disappearance of time series data signals (i.e., embodiments).

[0062] Experiments (described in further detail below) were conducted using the publicly available LargeST dataset

[0020] as well as a processing network dataset (comprising time series transaction data for members of a payment processing network). These experiments include three benchmark experiment settings to evaluate the ability of methods according to embodiments to handle large-scale spatial-temporal time series data with emerging and vanishing time series signals. Experiments additionally demonstrate the scalability and accuracy of methods according to embodiments in comparison with other methods, validated using both emulated and real datasets. As demonstrated by such experiments, embodiments outperform alternative methods in both scalability and predictive capabilities and can effectively manage emergent and vanishing time series signals.

[0063] It is assumed that a potential practitioner of methods according to embodiments has some general knowledge of time series data analysis (including time series forecasting), machine learning, sequence machine learning models 18 KILPATRICK TOWNSEND 788665001(including transformers), etc. However, in order to better orient the reader, a summary of some concepts related to embodiments of the present disclosure is provided below.

[0064] “Time series data” can generally comprise a series of observations or time series data values related to some phenomena, process, event, system, object, etc., typically ordered chronologically, and often rendered as a line or “waveform” on a graph. Data in a time series may be organized by timestamps (e.g., indicating the time and date at which a particular observation took place) and / or by indices (indicated, e.g., the relative location of an observation within a data sequence via a numerical value). As an example, periodic outdoor air temperature measurements (e.g., recorded by a weather station) could comprise a time series. Various useful analytics and processes based on those analytics can be performed using time series data or other sequences of data. For example, a weather forecasting service may use historical time series weather data to forecast the weather, e.g., the outdoor temperature, chance of rain, wind speed, etc., over the next 10 days. As another example, a power utility could use historical power demand data to determine whether a particular rapid increase in power demand is anomalous or unusual or is within usual variations in demand over a particular period of time.

[0065] “Multivariate time series data” may comprise time series data in which multiple different observations or time series data elements are recorded, e.g., in which multiple time series data values correspond to a given timestamp. As an example, a time series observation corresponding to patient health data may comprise a heart rate data value, a pulse oximetry data value, a blood pressure data value, etc. Likewise, a time series observation corresponding to transaction activity in a payment processing network (e.g., number of transactions in a given timeframe) could comprise transaction activity data corresponding to any number of banks, merchants, or other entities involved in the payment processing network. In the context of multivariate time series, a “time series signal” may refer to a time series corresponding to a single variate, object, entity, etc. from that multivariate time series. For example, a multivariate time series could comprise a heart rate signal, a pulse oximetry signal, and a blood pressure signal, or a first merchant signal, a 19 KILPATRICK TOWNSEND 788665001second merchant signal, a third merchant signal, etc. Signals 102-106 of FIG.1 could comprise time series signals in a single multivariate time series dataset.

[0066] “Spatial-temporal” data can comprise a type of multivariate time series data, in which data observations are associated with both a timestamp (or index) and a location (e.g., a point) in space. For example, the electricity demand in twenty different cities could comprise spatial-temporal data, e.g., represented by a multivariate time series with twenty time series signals. A “space” can refer to either a physical space or a mathematical space. For example, for multivariate time series data that tracks the transaction rate (measured in transactions per hour) for twenty thousand merchants that are part of a payment processing network, an identifier corresponding to each merchant could comprise a point in an abstract “merchant space”, and thus the multivariate transaction rate time series could be considered a form of spatial-temporal data. As such, the terms “multivariate time series data” and “spatial-temporal data” are used largely interchangeably herein, e.g., spatial- temporal data can be represented using a multivariate time series and, in most scenarios, some type of space can be constructed such that a given multivariate time series can be considered a form of spatial-temporal data.

[0067] Time series forecasting generally involves predicting future time series data values based on historical time series data values and includes both multivariate time series forecasting and spatial temporal forecasting. For example, multivariate time series forecasting can be used to predict the electrical power demand for multiple households in view of historical power demand from those households. As another example, multivariate time series forecasting can be used to predict the number of transactions that a credit card holder will perform next month based on a multivariate time series of credit card transactions performed by various cardholders over previous months. Other examples of multivariate time series forecasting include predicting human motion [28, 29] and traffic [3, 30] which exhibit both spatial and temporal dependencies. Expressed mathematically, givenhistorical observation ^^ ∈ ℝுൈேൈ^ containing ^^ frames of ^^ time-dependentvariables with ^^ features (or “channels”), the general goal of multivariate or spatial-temporal forecasting is to predict the subsequent value ^^ ∈ ℝிൈேൈ^ of ^^ followingsteps. 20 KILPATRICK TOWNSEND 788665001

[0068] Within many systems or networks modeled using multivariate time series, there may be interrelationships (which may be referred to as “spatial relationships”) between entities or objects in those systems or networks, which may result in correlations between time series signals. These correlations can be exploited for forecasting purposes, as future values of a given time series signal can be predicted not only in view of historical values of that time series signal, but also in view of historical values of highly correlated (or anti-correlated) time series signals. However, the interrelationships between entities increase the complexity of performing multivariate time series forecasting and additionally increase computing power and memory requirements.

[0069] Various techniques are used to model the spatial relationships between variates in multivariate time series, including grids

[0039] in which feature maps are split spatially into non-overlapping equal-sized patches, or graphs

[0020] , where variates (or corresponding entities, objects, etc.,) are represented as graph nodesconnected by edges, e.g., for a graph ^^ ൌ ^^^, ^^, ^^ୟ^୨^, ^^ and ^^ can comprise sets of^^ variables from ^^ ∈ ℝுൈேൈ^ (and their relations respectively) and ^^ୟ ∈ ℝேൈே^୨ candenote an adjacency matrix representing variable connectivity. In such a model, thespatial-temporal forecasting problem can be formally defined as ^^ఏ^^^, ^^ୟ^୨^ → ^^,where ^^ఏ^∙^ denotes the parameterized forecaster. Some works, such as

[0017] , rely on a predefined ^^ as part of the model input, while others, like

[0033] , aim to learn the structural dependencies between variables by constructing ^^ during the learning process.

[0070] While both patch-based and graph-based methods have their merits, graph-based methods have seen increased use due to their flexibility in modeling irregular structures, e.g., using graph neural networks (GNNs) [1, 7, 9, 14, 15, 17, 24, 26, 30, 33, 34, 37], transformers

[0035] or multilayer perceptrons (MLPs) [2, 36] to identify spatial dependencies among nodes. Temporally, such methods can use recurrent neural networks (RNNs), temporal convolutional networks (TCNs) or transformers for sequence models. Despite their effectiveness, these methods often cannot scale effectively to large multivariate datasets, and cannot effectively address situations where time series signals may appear or disappear over time. 21 KILPATRICK TOWNSEND 788665001

[0071] The task of spatial-temporal forecasting is sometimes approached using multivariate time series forecasting if a spatial input graph is absent or not clearly defined. In such cases, transformer-based models [16, 19, 32, 40, 41] can be used due to their sequence modeling abilities. While transformer-based models (including “inverted transformers”

[0021] ) offer strong performance, they typically suffer from quadratic complexity with respect to sequence length (or quadratic complexity with respect to the number of variates).

[0072] As such, approaches such as [5, 36, 38], including “mixer-based” approaches, have become popular recently due to their linear complexity with respect to sequence length. RPMixer

[0036] , for example, employs a mixer layer (i.e., a fixed feature multilayer perceptron) to model relationships among time series signals in a multivariate time series. This is in contrast to other spatial-temporal forecasting models that explicitly compute the correlation matrix [14, 21, 23, 25]. Despite their efficiency, mixer-based models often have difficulty with situations in which time series signals may appear or disappear over time (e.g., the appearance of time series signals corresponding to new cardholders in a payment processing network or the disappearance of time series signals corresponding to departing cardholders). As correlations between time series signals change due to the appearance or disappearance of time series signals, fixed feature multilayer perceptrons may generate inaccurate forecasts. As an example, forecasts corresponding to an emerging time series signal could be incorrect as the feature multilayer perceptron does not consider the correlation between the new time series signal and existing time series signals. Likewise, when a time series signal disappears, forecasts for time series signals that were strongly correlated with the disappearing time series signal may be adversely affected due to the unusual signal activity (e.g., a string of consecutive zeros or “not a number” values) following the disappearance of the time series signal.

[0073] As described in more detail further below, embodiments of the present disclosure address these limitations by achieving linear complexity in both space and time while remaining inductive to the number of variates, thus providing a more comprehensive, scalable forecasting solution, increasing efficiency while maintaining forecast accuracy. 22 KILPATRICK TOWNSEND 788665001

[0074] A brief summary of machine learning may facilitate a better understanding of embodiments of the present disclosure. At a high level, a machine learning model generally produces output data responsive to received input data. Such input data may comprise “features”, e.g., of some object, event, or phenomena being analyzed by a machine learning model, which may be quantified or qualified by the output data. For example, for a machine learning model used to evaluate comments posted on a social media website, the length of a comment (measured, e.g., in the number of words or the number of characters) could comprise a feature. The output of such a machine learning model could comprise scores used to describe or define such comments, e.g., a score indicating how “malicious” or “benign” a particular comment is. Such scores could be used as part of an automatic content moderation system, e.g., a system that automatically deletes malicious comments based on their scores.

[0075] Machine learning models are often defined by sets of parameters, which generally control how the machine learning model produces output data responsive to received input data. As an example, a support vector machine (SVM) is a type of machine learning model that divides data points using a hyperplane. Input data positioned on one “side” of the hyperplane is classified as one class (e.g., normal) while input data on the other side of the hyperplane is classified as another class (e.g., anomalous). The parameters of the support vector machine can comprise the coefficients used to define the hyperplane. Changing these parameters changes the shape of the hyperplane, and thus changes which data points the SVM classifies as normal or anomalous.

[0076] In general terms, the process of training a machine learning model can involve determining the set of parameters that achieve the “best” performance, usually based on a loss or error function. A loss function relates the expected or ideal performance of the machine learning model to its actual performance on a training dataset. In “supervised learning”, the training dataset may be labeled, and the labels may indicate expected or ideal outputs of the machine learning model corresponding to data in the training dataset. For example, for a machine learning model used to classify images, training data may comprise images and labels indicating what is depicted in those images. In “self-supervised learning”, the 23 KILPATRICK TOWNSEND 788665001training dataset may not be labeled, and the model may instead learn to e.g., generate embeddings that accurately encode the information contained in the training dataset.

[0077] The loss function typically decreases in value as the model’s performance improves. As such, training a machine learning model often involves determining the set of parameters that minimize a loss function corresponding to that model. In some cases, a random parameter estimate is generated as an initial parameter “guess,” and then a process such as gradient descent is used to iteratively refine the parameter estimate, eventually resulting in a final set of parameters associated with the machine learning model.

[0078] This iterative refinement process can be performed in a series of training “rounds”, “epochs”, or other appropriate divisions. In each round, a machine learning model’s performance can be evaluated using the loss function, and the parameters can be updated based on this evaluation, e.g., with the goal of reducing the result over time. As an example, the gradient of the loss function can be determined in parameter space and can be used to reduce the value of the loss function in successive training rounds. Such a gradient corresponds to a change in model parameters that achieves the greatest immediate reduction in the loss function. By changing the model parameters based on the gradient, the loss function can be reduced during each successive training round.

[0079] The terms “reward” and “penalize” (or “punish”) are sometimes used in the context of training, often to generally describe the objectives of training or the ideal behavior of a machine learning model without needing to specifically reference a loss function or any of the more technical details associated with training. Generally, a model is “rewarded” during training when its performance is more consistent with the expected or ideal performance (or is “acceptable” in view of the expected or ideal performance) and is “penalized” when its performance is less consistent with the expected or ideal performance (or is “unacceptable” in view of the expected or ideal performance).

[0080] What exactly constitutes a “reward” or “penalty” depends on particular training processes or machine learning implementations. However generally, in the 24 KILPATRICK TOWNSEND 788665001context of training via loss function minimization, a “reward” can refer to a reduction in the value of a loss function resulting from machine learning model performance that is consistent with ideal or desired performance, and a “penalty” can generally refer to an increase in the value of the loss function resulting from machine learning model performance that is inconsistent with ideal or desired performance.

[0081] Regardless, training processes can be repeated until a terminating condition has been met, at which point training can end (although a machine learning expert could choose to continue or restart model training after a terminating condition has been met). In embodiments of the present disclosure, one type of terminating condition is a defined number of training rounds. This terminating condition can be met if the number of training rounds performed (e.g., by a computer system training the machine learning model) equals or exceeds the defined number of training rounds, at which point the iterative training process has been completed. Another type of terminating condition in embodiments is a convergence condition. This terminating condition can be met if the machine learning model parameters “converge.” In general terms, convergence is achieved when the value of the loss function, and / or the values of the model parameters change in increasingly small amounts with each successive training round. For example, a convergence condition can be achieved if the value of the loss function decreases by less than 0.1% in two successive training rounds.

[0082] It is possible to train multiple machine learning models or multiple trainable components of a machine learning model simultaneously, e.g., by updating parameter sets corresponding to each of those machine learning models simultaneously, e.g., using a single training dataset and / or a combined loss function. For example, Generative Adversarial Networks (GANs) can comprise a “generator” sub-model and a “discriminator” sub-model, which can be trained concurrently.

[0083] Components of a machine learning model (trainable or otherwise) may be organized into "layers” or “modules”, usually based on their relative “proximity” to either the model’s inputs (e.g., features) or the model’s outputs (e.g., predictions based on those features). For example, an initial layer may operate on feature inputs, a final layer may produce the model’s output, and an intermediate layer may 25 KILPATRICK TOWNSEND 788665001operate on (as well as generate) intermediate data used by the model to produce the model output.

[0084] Some multilayer machine learning models (including some machine learning models according to embodiments) may include “encoders,” which can encode a feature input (or intermediate data) into an “embedding” or “latent space embedding” corresponding to a “latent space”. Such an embedding can generally represent the information contained in the encoder’s input in a different form (e.g., as a point or vector in a latent space), which may facilitate further processing, e.g., by subsequent layers of a machine learning model, ultimately resulting in more accurate or higher quality outputs. For example, in embodiments of the present disclosure, time series signals from a multivariate time series can be encoded to produce signal embeddings representative of those time series signals. For example, if a multivariate time series records electrical power usage for twenty households over 1000 timesteps, that multivariate time series could be encoded to produce twenty signal embeddings, each corresponding to a single household over all time steps. These embeddings can then be input into subsequent layers of a machine learning model, which can predict future time series data values (e.g., future electrical power usage) based on signal embeddings rather than directly based on a multivariate time series dataset.

[0085] There are various advantages of using embeddings as part of machine learning. For example, embeddings may have a smaller logical size than the data used to produce such embeddings. As such, processing embeddings may have lower computational complexity than processing the data used to generate such embeddings, improving the speed at which a machine learning model can be trained or used during inference. As another example, the use of embeddings may enable various machine learning tasks to be performed using vector operations. For example, for a novelty entertainment application used to “fuse” the appearance of two individuals via photographs of those individuals, an encoder machine learning model could be used to convert two photographs into two embedding vectors of equal length. Such embedding vectors could then be added together (e.g., using vector operations) and the resulting vector sum could be decoded using a decoder machine learning model to produce a fused image of the two individuals. 26 KILPATRICK TOWNSEND 788665001

[0086] FIG.2 shows a diagram of a general multilayer machine learning model 214. The multilayer machine learning model 214 comprises an encoder 204 (or “encoding layer”, “embedding layer”, “encoder module”, “embedding module”, etc.), final layer(s) 208, intermediate model layers 212, and additional intermediate model layers 216. The model inputs 202 to machine learning model 214 can be processed by any number of intermediate model layers 212 prior to being processed by encoder 204. Each of these intermediate model layers 212 may produce its own output, which may comprise the input to a successive intermediate model layer 212 (or encoder 204). Such intermediate model layers 212 (and likewise, additional intermediate model layers 216) may depend on the nature of machine learning model 214, and can comprise any variety of layers that could be included in a machine learning model, such as pooling layers, subsampling layers, normalization layers, other neural network layers (e.g., fully connected layers), other encoders, etc. A convolutional neural network, for example, can comprise an alternating series of convolution layers and subsampling layers (each of which could be considered intermediate model layers) prior to a final fully connected layer. In some machine learning models, encoder 204 may comprise either the first layer in the machine learning model 214 or the second to last layer in the machine learning model 214, e.g., such that there are no intermediate model layers 212 preceding encoder 204, or no additional intermediate model layers 216 following encoder 204. In other machine learning models, there may be any number of intermediate model layers 212 preceding encoder 204 and any number of additional intermediate model layers 216 following encoder 204.

[0087] As stated above, an encoder (such as encoder 204) can produce embeddings or encodings (e.g., embeddings 206, which may comprise “embedding vectors”) based on its inputs. Such embeddings 206 can generally comprise generalizable alternative representations of the encoder’s 204 inputs, which may facilitate processing by subsequent components of the machine learning model 214 (e.g., additional intermediate model layers 216, final layer(s) 208, etc.). After being processed by any number of additional intermediate model layers 216 (including zero), the final layer(s) 208 can be used to produce a model output 210 based on embeddings 206 (or based on any intermediate data generated from embeddings 206). Such an output of final layer(s) 208 could comprise, e.g., a time series 27 KILPATRICK TOWNSEND 788665001forecast comprising future time series data values corresponding to an input multivariate time series dataset, e.g., a forecast or prediction indicating the transaction volume for a payment processing network (or e.g., some set of merchants in the payment processing network) some number of minutes, hours, days, weeks, months, years, etc., in the future. The final layer(s) 208 could comprise (as an example) a fully connected layer (also known as a “dense layer”), a layer in a neural network in which every “neuron” in one layer is connected to each neuron in the next layer. The final layer(s) 208 may be defined by their own parameter set and could be trained simultaneously with encoder 204 based on a combined loss function, e.g., by updating a parameter set corresponding to the finals layers and a parameter set corresponding to the encoder in each training round of an iterative training process.

[0088] As described herein, embodiments of the present disclosure relate to performing multivariate time series forecasting using sequence machine learning models. Perhaps the currently most well-known example of a sequence machine learning model is the “transformer”, a class or architecture of machine learning model that has been used in many of the better performing large language models (LLMs). Transformers can use a mechanism known as “attention” in order to perform machine learning tasks. As some embodiments of the present disclosure can be implemented using transformer models and attention, a brief summary of the concept of attention may facilitate a better understanding of embodiments of the present disclosure. More information about attention can be found in the literature, e.g., Vaswani et al.2017 “Attention is all you need” (arxiv.org / abs / 1706.03762).

[0089] Attention is often used in the context of language models, in which each “token” (e.g., data representative of a word or part of a word, such as the suffix “-ing”) in a sequence (e.g., a sentence) can “interact” with each other token in the sequence for the purpose of performing the task associated with the language model. This is in contrast to previous types of machine learning models, such as recurrent neural networks, which have a temporal window and incremental interaction. Using attention, language models can learn which tokens interact with which other tokens, to which degree, and how, enabling such language models to learn the relationships between words in sentences. Language models that use 28 KILPATRICK TOWNSEND 788665001attention often achieve better performance than language models that use recurrent neural networks, as they can model more complex relationships between input tokens in sentences.

[0090] Generally, attention is a “sequence to sequence” operation, for which a sequential output is generated based on one or more sequential inputs (e.g., sequences of time series data values or sequences of embedding vectors or “tokens” based on such time series data values). Such attention outputs (often “attention vectors”) can then be processed in some manner to perform a machine learning task. Two notable types of attention are self-attention and cross-attention. Generally, in self-attention, the attention can be computed between a single input sequence and itself. In cross-attention there can be multiple input sequences, and the attention can be computed between these input sequences to produce the output sequences.

[0091] In more detail, in self-attention, each output in the output sequence can comprise a weighted average of the inputs in the input sequence, e.g., for inputs ininput sequence ^^^ and outputs in output sequence ^^^, ^^^ ൌ ∑^ ^^^,^^^^ . However,unlike other machine learning models based on weighted averages, in which weights are parameters of the system, the weights ^^^,^in self-attention are often derived from a function of the inputs in the input sequence, often based on similarity, such that the weight corresponding to a pair of similar inputs may be greater than the weight corresponding to a pair of dissimilar inputs. Often, for sets of vector inputs,the weights are derived from the dot product between vector inputs, e.g.,ൌ ^^^ ∙^^^, and in some cases, functions such as the softmax function can be used to mapthe weights to a defined range, such as ^0, 1^, i.e.:

[0092] Cross-attention is similar to self-attention, except the attention is not computed between members of a single sequence, but rather between members of multiple (often two) sequences. In such cases, the cross-attention weights may be based off the dot products of different sequences of inputs rather than a single sequence. 29 KILPATRICK TOWNSEND 788665001

[0093] Attention operations are often framed in the context of “queries”, “keys”, and “values” (which may also be referred to as “attention queries”, “attention keys”, and “attention values”). For example, in a case of self-attention, queries ^^^, keys ^^^, and values ^^^can be derived from inputs ^^^via a query matrix ^^^, key matrix ^^^, and value matrix ^^௩, e.g.: ^^^ ൌ ^^^^^^ ^^^ ൌ ^^^^^^ ^^^ ൌ ^^௩^^^^^^ ൌ ^ softmax൫^^^ ∙ ^^^൯^^^^

[0094] These weight matrices can be controlled in order to modify any input vectors based on the machine learning task being performed. In addition, the query, key, value framing of attention can be useful for implementing cross-attention, e.g., by deriving the queries, keys, and values from different input sources. Typically, the size of the query, key, and value matrices are proportional to the length of the sequence of input token, e.g., for a sequence of ^^ input tokens, the query, key, andvalue matrices can comprise ^^ ൈ ^^ matrices. This however leads to quadraticscaling with regards to the number of input tokens for attention-based (e.g., transformer) machine learning models. As described above, this can cause issues for large multivariate time series datasets, as existing transformer models may encounter out-of-memory errors when generating time series forecasts for such datasets.

[0095] FIG.3 shows a “standard” attention model that can be used to process time series datasets using attention (e.g., to generate time series forecasts for such time series datasets). Embeddings 302 can be generated from time series data using an encoder or any other suitable module or component. If necessary, such embeddings 302 can be normalized using a layer normalization block 304 and used to produce a query 306, key 308, and value 310. The query 306, key 308, and value 310 can be applied to an attention layer 312 to produce attention vector(s), which can be combined with the embeddings if necessary and normalized (e.g., via layer normalization block 314). The result can be input into a neural network 316 (e.g., a multilayer perceptron). If necessary, the output of the neural network 316 can be projected, e.g., via a projection layer 318 to produce a model output 320. Forexample, if the dimensions of the neural network output are ^^ ൈ 16 (e.g.,30 KILPATRICK TOWNSEND 788665001corresponding to ^^ time series signals), but the desired model output 320 is an^^ ൈ 7 array (e.g., corresponding to the predicted electricity usage by ^^ householdsover the next 7 days), then the projection layer 318 could be used to project the^^ ൈ 16 neural network output down to an ^^ ൈ 7 array. As another example, if thedimensions of the neural network output are 1 ൈ 16, but the desired model output320 is a Boolean value (e.g., a binary prediction indicating whether a winter storm will occur given time series weather data), then the projection layer 318 could beused to project the 1 ൈ 16 neural network output down to a single Boolean value.

[0096] It should be understood that the formulas provided above are intended as examples in order to generally summarize the concepts of attention and are not intended to be limiting, complete, or even mathematically precise. There are various ways that one could implement e.g., attention, position embeddings, etc., and more detailed descriptions of such implementations can be found in the literature.

[0097] Having described some concepts related to embodiments of the present disclosure, it may now be helpful to describe some systems according to embodiments of the present disclosure, including computer systems that can be used implement methods according to embodiments.

[0098] FIG.4 shows a computer system 402 that can be used to perform methods according to embodiments. Such a computer system can comprise a server computer that can perform methods according to embodiments for client computer(s) 410, e.g., as part of a “machine learning as a service” (MLaaS) system. As described in more detail below with reference to FIG.18, computer system 402 can comprise one or more processors (not pictured) and a non-transitory computer readable medium (e.g., a hard drive) coupled to the one or more processors (also not pictured). The non-transitory computer readable medium can comprise code or instructions, executable by the one or more processors for performing methods according to embodiments described herein.

[0099] As summarized further above, embodiments of the present disclosure are directed to various methods and systems for performing multivariate time series forecasting using an inverted transformer approach, random projection, and efficient latent attention. As a general summary, computer system 402 can retrieve a 31 KILPATRICK TOWNSEND 788665001multivariate time series dataset comprising a plurality of time series signals. Computer system 402 can generate a plurality of signal embeddings corresponding to that plurality of time series signals, e.g., using encoder module 416 (which may comprise part of a machine learning model 404). Computer system 402 can then produce an initial query based on the plurality of embeddings (e.g., via layer normalization) then process the plurality of queries using random projection module 418, which may comprise part of machine learning model 404 and include a random key matrix and a random projection value matrix. The result can comprise (or be used to generate) a second query, which can be processed using latent attention model 420, which may comprise part of machine learning model 404 and include a key matrix and a value matrix. The computer system can process the output of latent attention model 420 using forecasting model 422, which also may comprise part of machine learning model 404 and can include a neural network component (e.g., a multilayer perceptron). The output of forecasting model 422 may comprise a time series forecast corresponding to the multivariate time series input, which can then be used for some task, such as classifying an access request as legitimate or fraudulent based on a comparison between that access request and a time series access request forecast.

[0100] If necessary, computer system 402 can train machine learning model 404 using training module 426 and training time series data 414, which may be obtained by computer system 402 from a data source 408 (e.g., a database, data stream, the Internet, etc.). Computer system 402 may process time series data as a service for one or more client computer(s) 410, e.g., by receiving the time series data from those client computer(s) 410 and returning any generated outputs to those client computer(s) 410.

[0101] As a more concrete example, computer system 402 could use machine learning model 404 to forecast the electrical power demand for a large number of households one month in the future, e.g., on behalf of a client computer 410 associated with an electrical utility. Computer system 402 could receive a multivariate time series dataset comprising a plurality of time series signals, each signal associated with a different household. Computer system 402 could generate signal embeddings corresponding to each time series signal using encoder module 32 KILPATRICK TOWNSEND 788665001416, then process those embeddings using the random projection module 418, latent attention module 420, and forecasting module 422 (as outlined above) to generate a multivariate time series forecast, forecasting the electrical power demand for those households one month into the future. As another example, computer system 402 could comprise an access server that can be part of a resource security system, e.g., a system that is used to control access to digital and physical resources. An example resource security system is described in more detail below with reference to FIG.17. For example, computer system 402 could comprise a computer system associated with controlling access to digital resources, such as movies or videos provided by a digital streaming service. Computer system 402 could use machine learning model 404 to evaluate a multivariate time series of historical access requests in order to classify whether a recently made access request is legitimate or fraudulent. As yet another example, computer system 402 could comprise a computer system associated with a payment processing network, which could forecast future transaction volumes for various banks and merchants (which may comprise client operators of client computer(s) 410) based on a multivariate time series of historical transaction data.

[0102] If necessary, computer system 402 can also use inference module 424 in order to process multivariate time series data and produce any applicable outputs. For example, if machine learning model 404 outputs time series forecast comprising sequences of numerical values, computer system 402 could use inference module 424 to generate an output waveform image representative of that time series, which may be more useful to human operators of computer system 402 and / or client computer(s) 410.

[0103] In some embodiments, machine learning model 404 may be part of a larger segmentation model comprising multiple machine learning models. In general terms, for a segmentation model, relevant feature data from an input dataset may be segmented into data subsets, each of which may be processed by a different machine learning model. For example, an input multivariate time series dataset could be split into several time series subsets, which could each be input into a different machine learning model in a larger segmentation model. The segmentation model can combine the outputs of the different machine learning models to produce 33 KILPATRICK TOWNSEND 788665001a single output, e.g., a single time series forecast. Segmentation models may be useful in situations in which it would be computationally slow, difficult, or infeasible to process a large set of features with a single model.

[0104] Regardless, even though FIG.4 depicts only a single machine learning model 404, in some embodiments the computer system 402 may instantiate, train, and / or utilize multiple machine learning models, each of which may have any number of components. It should be appreciated more generally that the numbers of devices, entities, and components shown in FIG.4 were selected for simplicity of illustration and exposition. It should be understood that systems according to embodiments of the present disclosure can include more than one of each device, entity, component, computer system, etc. In addition, some systems according to embodiments may include a lesser number of devices, entities, and / or components or a greater number of devices, entities, and / or components than those shown in FIG.4. For example, computer system 402 may comprise a distributed computing system comprising several computers collectively performing methods according to embodiments. Likewise, there may be multiple data sources 408 from which computer system 402 retrieves training data 414 for the purpose of training machine learning models, along with multiple communication networks 412 over which computer system 402 communicates with client computers(s) 410. It should be understood that the particular modules depicted in FIG.4 (and their configuration) are intended to represent only one possible implementation, selected for ease of exposition, and are not intended to be limiting. Various other configurations may become apparent upon reading this disclosure, e.g., implementing methods according to embodiments using a single monolithic software module.

[0105] As described above, computer system 402 could be used to implement machine learning based analysis as a service for others, e.g., on behalf of client computer(s) 410 or users of client computer(s) 410 (which may also be referred to as “requestors”). In such scenarios, client computer(s) 410 and computer system 402 can communicate over a communication network 412. A communication network such as communication network 412 can take any suitable form, and may include any one and / or the combination of the following: a direct interconnection; the Internet; a Local Area Network (LAN); a Metropolitan Area Network (MAN); an 34 KILPATRICK TOWNSEND 788665001Operating Missions as Nodes on the Internet (OMNI); a secured custom connection; a Wide Area Network (WAN); a wireless network (e.g., employing protocols such as, but not limited to a Wireless Application Protocol (WAP), I-mode, and / or the like); and / or the like. Messages between computers and devices in the system of FIG.4 and / or over communication network 412 may be transmitted using a secure communication protocol, such as, but not limited to, File Transfer Protocol (FTP); HyperText Transfer Protocol (HTTP); Secure HyperText Transfer Protocol (HTTPS); Secure Socket Layer (SSL), ISO (e.g., ISO 8583) and / or the like. Any suitable communication protocol can be used to communicate over the communication network 412, e.g., for the purpose of creating one or more communication channels. A communication channel may, in some instances, comprise a secure communication channel, which may be established in any known manner, such as through the use of mutual authentication, a session key, and establishment of a Secure Socket Layer (SSL) session.

[0106] In some embodiments, such requests from client computer(s) 410 may contain all the information (e.g., input multivariate time series datasets) necessary for computer system 402 to provide machine learning services using machine learning model 404. However, in other embodiments, computer system 402 may retrieve relevant multivariate time series datasets from another data source 408, e.g., a memory element (e.g., a hard drive), database, or any other data structure or storage element, in order to service requests from client computer(s) 410. As an example, an analyst at a national weather service may request a machine learning based weather forecast in a distance region of their country. The analyst may use a client computer 410 to connect to a computer system 402 that is located at a remote weather monitoring station. Using a time series dataset of historical weather data on a hard drive located in the remote weather monitoring station, computer system 402 can use machine learning model 404 and the various modules of FIG.4 to generate a weather forecast. The weather forecast can be returned to the client computer 410 and can then be interpreted by the weather analyst.

[0107] Methods of training machine learning models according to embodiments and using machine learning models to perform inference are described below with reference to the flowcharts of FIGs.6 and 7 respectively. In addition, a 35 KILPATRICK TOWNSEND 788665001high-level summary of inference methods according to embodiments is provided with reference to FIG.5.

[0108] FIG.5 shows a machine learning model 500 according to some embodiments (sometimes referred to herein as an “efficient inverted transformer” or “EiFormer”). The machine learning model 500 can comprise an encoder module 502 (also referred to as an “embedding module”), a random projection module 504, a latent attention module 506, and a forecasting module 508. However, it should be understood that some machine learning models according to embodiments may lack some of these components, e.g., in some embodiments, machine learning model 500 may not comprise a random projection module 504. A computer system can use machine learning model 500 to generate a model output 510 (e.g., a time series forecast) based on spatial-temporal data 512 (e.g., a multivariate time series dataset comprising a plurality of time series signals 514).

[0109] Using encoder module 502 (which may comprise a linear layer 516), the computer system can generate a plurality of signal embeddings 518 corresponding to the plurality of time series signals 514 in a multivariate time series dataset (spatial-temporal data 512). Similar to the embedding process used by iTransformer

[0021] , the encoder module 502 can be used to embed the spatial- temporal data in the signal dimension. This can take advantage of the inductive bias in the attention mechanism (used by attention layers 520 and 522). The correlations perceived by machine learning model 500 can adapt to both emerging and vanishing time series signals. For emerging time series signals, the resulting correlations with other time series signals will mirror those of existing time series signals exhibiting similar temporal patterns. For vanishing time series signals, their unusual temporal patterns (e.g., consecutive zeros), are disregarded when making forecasts related to other time series signals.

[0110] If necessary, the signal embeddings 518 can be normalized (e.g., using normalization layer 524) and used to produce an initial query 526. The computer system can use random projection module 504 (i.e., attention layer 520, initial query 526, a random key 528 (which may be randomly generated and “frozen”, i.e., not learned during training, in order to provide the random projection operation), and a value 530) to generate a randomly projected latent attention matrix, which can be 36 KILPATRICK TOWNSEND 788665001used as query 532 for latent attention module 506. The use of random projection module 504 can enable machine learning model 500 to capture a wide range of spatial-temporal patterns and reduce the risk of overfitting.

[0111] The computer system can use latent attention module 506 (i.e., attention layer 522, query 532, key 534, and value 536) to generate a latent attention matrix 542, e.g., by combining query 532 and key 536 (e.g., via matrix multiplication and the softmax operation) to generate an attention matrix 540, then combine attention matrix 540 and value 536 to generate latent attention matrix 542. If necessary, the computer system can normalize latent attention matrix 542 using normalization layer 544 (after optionally combining latent attention matrix 542 with signal embeddings 518) then input the (optionally normalized and combined) latent attention matrix 542 into multilayer perceptron 546 of forecasting module 506. The output of multilayer perceptron 546 can optionally be combined with the latent attention matrix 542 and projected (if necessary) using projection layer 548. In this way the computer system can generate model output 510, e.g., a time series forecast, a classification of a time series forecast (e.g., normal or anomalous), etc.

[0112] Machine learning model 500 can comprise a single “EiFormer” block of a machine learning model comprising a chain of EiFormer blocks. Expressed in other words, the output of multilayer perceptron 546 can be input into any number of subsequent machine learning models, each comprising e.g., a subsequent random projection module, a subsequent latent attention module, a subsequent multilayer perceptron, etc. The output of the final multilayer perceptron can be projected down using projection layer 548 to produce module output 510.

[0113] Unlike models such as iTransformer

[0021] , which suffer from poor time / space scalability (i.e., quadratic) due to the necessity of computing a “full-sized” correlation matrix with respect to the number of signals, embodiments of the present disclosure use a form of efficient latent attention, in which an input-independent dimension of the key and values matrices (e.g., random key 528, value 530, key 534, and value 536) are independent of the dimensions of the signal embeddings 518. As a result, forecasting methods according to embodiments have linear time / space complexity with respect to the number of time series signals, akin to RPMixer

[0036] , whilst preserving the inductive bias innate to the attention mechanism. 37 KILPATRICK TOWNSEND 788665001

[0114] Having generally summarized methods according to embodiments, training and inference methods are described in more detail below.

[0115] Training methods according to embodiments can be performed by a computer system and can involve the computer system training a machine learning model by performing an iterative training process. The machine learning model can comprise a latent attention module and a forecasting module comprising a multilayer perceptron (and optionally a random projection module). These modules can comprise sub-models of the machine learning model. The iterative training process can be performed until a terminating condition has been met, e.g., model convergence or a predefined number of training rounds or epochs. Reference is now made to FIG.6, which shows a flowchart corresponding to a training method according to some embodiments.

[0116] At step 602, a computer system can obtain any number of multivariate training time series datasets. Each multivariate training time series dataset can comprise a plurality of training time series signals (which may also be referred to as “training time series datasets”, i.e., subsets of the multivariate training time series dataset). Each training time series signal can comprise a plurality of training time series data values and a plurality of ground truth forecast time series data values corresponding to a plurality of training time steps. For example, a training time series signal could comprise features or observations correspond to 365 time steps, e.g., each day in a given year. The plurality of training time series data values could comprise time series data values corresponding to the first 300 training time steps, while the plurality of ground truth forecast time series data values could correspond to the 301st-365thtraining time steps. In this example, the computer system could train the machine learning model to forecast time series data values corresponding to the last 65 training time steps (i.e., the plurality of ground truth forecast time series data values) based on the first 300 training time series data values (i.e., the plurality of training time series data values).

[0117] In some embodiments, the computer system can retrieve the multivariate training time series dataset from a database, data stream, a local memory such as a hard drive, cloud storage, an I / O interface, or any other appropriate source. In other embodiments, the computer system can comprise a 38 KILPATRICK TOWNSEND 788665001server computer that performs machine learning based services for client computers. In such embodiments, the computer system can receive the training data time series datasets from a client computer (or from any other appropriate source), e.g., over a communication network such as the Internet, as described above with reference to FIG.4.

[0118] A multivariate training time series dataset can comprise a “complete” set of training time series data values from a multivariate training time series or can comprise a subset of training time series data values from a larger multivariate training time series. For example, a multivariate training time series dataset corresponding to an access control system could comprise a multivariate time series tracking all access requests (e.g., to access a computer system or a protected resource) made by all users for the entirety of the operating time of that access control system (e.g., indicating when those users made those access requests, whether those access requests were successful or not, etc.). Alternatively, the multivariate training time series dataset could comprise training time series data values corresponding to access requests made by a subset of users over the previous year.

[0119] Each training time series signal can comprise any number of training time series data values. In some embodiments, each training time series signal can comprise a chronologically ordered sequence of training time series data values. In some embodiments, each training time series signal can comprise any number of features or “channels”, such that each training time series data value corresponds to one or more channels (in addition to the plurality of training time steps). In some embodiments, each training time series signals can comprise a number of training time series data values equal to a product of a number of time steps and a number of channels. In some embodiments, each training time series signal can correspond to a different entity of a plurality of entities or a different entity account of a plurality of entity accounts (e.g., a different electrical utility account, a different paying processing network account, a different hospital patient account, etc.). In some embodiments, the computer system can sort or otherwise generate the multivariate training time series dataset by combining training time series datasets corresponding to individual entities. 39 KILPATRICK TOWNSEND 788665001

[0120] As an example, for a multivariate training time series dataset corresponding to electrical power usage by users of an electrical utility (entities), a given time series signal could correspond to a given user of the electrical utility, some number of training time steps (e.g., 365, corresponding to each day of a given year), and can comprise some number of channels corresponding to different features or other observable or measurable aspects of that user’s electricity usage, such as the total electrical power used, the electrical power used by specific electrical appliances or classes of electrical appliances (e.g., “always on” appliances such as refrigerators, other appliances such as washers, dryers, computers, etc.), outdoor temperature, etc.

[0121] Some training time series signals can correspond to “velocity features”, e.g., corresponding to observations or statistics of observations over a period of time (e.g., the number of access requests made by a user over the last month). However, some training time series data values can comprise non-velocity data values, e.g., recording particular moments in time when an access request was logged by an access control system. Training time series signals may comprise training time series data values that are relevant to a particular machine learning task for which a machine learning model is being trained, or are predictive of (or believed to be predictive of) some expected output of the machine learning model being trained. For example, if a machine learning model is being trained to predict future merchant transaction rates based on a multivariate training time series dataset of merchant transaction rates, that multivariate training time series dataset may include e.g., training time series signals related to historical transaction rates, training time series signals related to current economic conditions, such as changes in a consumer purchasing index, etc. In some embodiments, training time series signals may comprise or correspond to message data (e.g., corresponding to message-based communication between two computer systems over the Internet), transaction data (e.g., corresponding to access requests for resources provided by merchant resource providers), or machine performance data (e.g., corresponding to the performance of a computer system such as a server computer, e.g., relative to some benchmark performance or performance metrics). 40 KILPATRICK TOWNSEND 788665001

[0122] After obtaining any number of multivariate training time series datasets, the computer system can perform a round of an iterative training process. The round of the iterative training process can comprise steps 604-618 of FIG.6. The computer system can perform this iterative training process until a terminating condition has been met, e.g., a set number of training rounds or epochs, a convergence condition, or any other appropriate terminating condition, at which point the iterative training process can be completed.

[0123] At step 604, the computer system can sample a multivariate training time series dataset comprising a plurality of training time series signals, e.g., from among the multivariate training time series datasets retrieved by the computer system at step 602. As described above, each training time series signal can comprise a plurality of training time series data values and a plurality of ground truth forecast time series data values corresponding to a plurality of training time steps. As described above, each training time series signal can correspond to a different entity of a plurality of entities or a different entity account of a plurality of entity accounts. Each training time series signal may be organized chronologically, e.g., in an array, a linked list, or any other suitable data structure.

[0124] Optionally, the computer system can sample more than one multivariate training time series dataset to use in a round of the iterative training process. These one or more multivariate training time series datasets can comprise a batch (or minibatch) used during a training iteration. The computer system can sample any number of multivariate training time series datasets, such as 10, 100, 1,000, 10,000, 100,000, one million, ten million, one hundred million, one billion, or ten billion multivariate training time series datasets. The computer system can sample the one or more multivariate training time series datasets in any appropriate manner, e.g., random sampling, sequential sampling, etc.

[0125] At step 606, the computer system can generate a corresponding training signal embedding for each training time series signal. In this way, the computer system can generate a plurality of training signal embeddings. Embodiments of the present disclosure can adopt the Inverted Transformer’s orthogonal embedding method, in which each training time series signal is represented as a separate token across all training time steps. This allows 41 KILPATRICK TOWNSEND 788665001embodiments to generalize to varying number of time series signals. The generationof training signal embeddings can be expressed as Embedding (ℝ்ൈ^ ↦ ℝ^): h^^ ൌEmbedding൫X ൯, h^:,^,: ^ ∈ ℝ^, where ^^ is a multivariate training time series with ^^ timesteps, ^^ entities, and ^^ channels and ^^ is the embedding dimension. The packed ^ embedding comprising all training signal embeddings can be expressed as ^^ ∈

[0126] The computer system can generate the corresponding training signal embedding for each training time signal based on the plurality of training time series data values in that training time series signal. In some embodiments, the computer system can generate a corresponding training signal embedding for each training time series signal by inputting the training multivariate time series dataset into an encoder module of the machine learning model, e.g., as depicted in FIG.5. In some embodiments, the encoder module can comprise a linear layer.

[0127] Each training signal embedding can comprise a number of training embedding values equal to an embedding dimension. In some embodiments, the embedding dimension may be less than the temporal dimension of each training time series signal (e.g., the number of training time steps associated with each training time series signal). For example, if a training time series signal corresponding to electrical power usage by a household comprises 300 training time series data values corresponding to 300 training time steps, the embedding dimension could some number less than 300.

[0128] The computer system can perform various other processes at step 606 in order to generate the plurality of training signal embeddings. For example, the computer system could normalize numerical training time series data values, quantize numerical training time series data values (e.g., to categories within a normalized range e.g., 0-0.1, 0.1-0.-2, 0.2-0.3, etc., in the normalized range 0.0-1.0), etc.

[0129] At step 608, the computer system can form a training query comprising the plurality of training signal embeddings. The training query can comprise a training matrix with an embedding dimension and a signal dimension equal to the number of training time series signals in the plurality of training time series signals. 42 KILPATRICK TOWNSEND 788665001For example, if the multivariate training time series comprised one million training time series signals corresponding to one million entities in a payment processing network (e.g., merchants, consumers, etc.), if each of these training time series signals was embedded to produce a training signal embedding of embedding dimension 100, then the training query could comprise a training matrix with dimensions 1,000,000 by 100 (or vice versa). In some embodiments, the computer system can apply linear normalization to the plurality of training signal embeddings prior to forming the training query, e.g., using a normalization layer such as normalization layer 524 from machine learning model 500 of FIG.5.

[0130] In some embodiments, the computer system can form the training query using random projection. In such embodiments, the computer system can generate an initial training query based on the plurality of training signal embeddings, which can comprise an initial training query matrix comprising the plurality of training signal embeddings. As described above with reference to FIG.5, the computer system can generate a randomly projected training latent attention matrix using a random projection module, the initial training query, a random key matrix, and a random projection value matrix. The training query can comprise the randomly projected training latent attention matrix.

[0131] Expressed in other terms, after generating the training signal embeddings ^^^, the computer system can use an efficient latent attention module with a frozen random key matrix ^^ (i.e., a random projection module) to produce the randomly projected training latent attention matrix Random_Proj^^^^^, e.g., accordingto ^^ ൌ ^^^^^, Random_Proj^^^^^ ൌ softmax^^^^^ ^^୰୭^^୬⁄ √^^ ^^^. Using a frozen randomkey matrix ^^ produces a randomized ^^^^^in the above equation, which describes the attention between each training time series signal embedding and latent factors. Such a mechanism acts as a noise-equivalent regularization to mitigate model overfitting. It also encourages different machine learning model blocks to concentrate on distinct sets of signal embeddings, thereby producing diverse representations that are then further modeled by subsequent fully-learnable latent attention layers. This diversity improves embodiments’ ability to capture a wide range of spatial-temporal patterns, addresses many of the challenges associated 43 KILPATRICK TOWNSEND 788665001with the scale of many real-world multivariate time series datasets, and reduces the risk of overfitting.

[0132] The random key matrix can have a random key embedding dimension equal to the embedding dimension (^^ as described above) and a random key input- independent dimension ^^ (described in more detail further below). The random key input-independent dimension can be equal to a key input-independent dimension, described further below. Likewise, the random projection value matrix can have a random projection value embedding dimension equal to the embedding dimension and a random projection value input-independent dimension equal to the key input- independent dimension and random key input-independent dimensions. The input- independent dimensions can be less than the signal dimension of the multivariate training time series dataset, resulting in linear computational and memory complexity with regard to the signal dimension, rather than quadratic computational and memory complexity associated with methods such as iTransformer

[0021] .

[0133] The random projection value matrix can be learned during the iterative training process, i.e., a plurality of random projection value elements of the random projection value matrix can comprise parameters of the machine learning model that can be optimized (“learned”) during the iterative training process. The random key matrix can comprise a “frozen” or “fixed” random key matrix, e.g., in some embodiments, the computer system can generate the random key matrix by randomly generating a plurality of random key matrix elements, and then not optimize or otherwise modify the random key matrix after generation (or in other words, “freeze” it).

[0134] At step 610, the computer system can generate a training latent attention matrix using the latent attention module, the training query, a key matrix and a value matrix. The key matrix can have a key embedding dimension ^^ equal to the embedding dimension and input-independent dimension ^^. The key input- independent dimension can be less than the signal dimension of the training query. Likewise, the value matrix can have a value embedding dimension ^^ equal to the embedding dimension and a value input-independent dimension ^^ equal to the keyinput-independent dimension ^^. Expressed in other terms, Let ^^ ∈ ℝேൈ^ be theplurality of training signal embeddings (e.g., after layer normalization), let ^^ ൌ ^^^^ be44 KILPATRICK TOWNSEND 788665001the training query, let ^^ ∈ ℝெൈ^ be the key matrix, ^^ ∈ ℝெൈ^ be the value matrix,and let Latent_Att^^^^ ൌ softmax൫^^^^^⁄ √^^ ൯^^. The matrices ^^ ∈ ℝ^ൈ^, ^^ ∈ ℝெൈ^,^^ ∈ ℝெൈ^ can be learnable matrices. Notably, the attention map ^^ ேൈெୟ^^ ∈ ℝ ∝softmax^^^^^^^produced by latent attention has an input-independent dimension ^^ as described above. By using input-independent latent matrices, embodiments of the present disclosure can reduce the computational complexity from quadratic to linear, while maintaining the ability to model signal dependencies, striking a balance between efficiency and model expressiveness. Various input-independent dimensions can be used. In some embodiments, the key input-independent dimension can be less than 1000. In some embodiments, the key input-independent dimension can be one.

[0135] At step 612, the computer system can generate a training time series forecast using the training latent attention matrix and the multilayer perceptron of the forecasting module. The multilayer perceptron can comprise any number of temporal multilayer perceptron blocks. In some embodiments, the computer system can apply linear normalization to the training latent attention matrix prior to inputting the training latent attention matrix into the multilayer perceptron. In some embodiments, the computer system can combine the training query with the training latent attention matrix prior to inputting the training latent attention matrix into the multilayer perceptron, e.g., as depicted in FIG.5.

[0136] The training time series forecast can correspond to the plurality of ground truth forecast time series data values, and can comprise one or more future training time series signals, each future training time series signal comprising one or more future training time series data values corresponding to one or more future training time steps. For example, a training time series signal could comprise features or observations corresponding to 365 time steps, e.g., each day in a given year. The plurality of training time series data values (used to derive the training query) could comprise training time series data values corresponding to the first 300 training time steps, while the one or more future training time signals could st thcorrespond to the 301 -365 training time steps (i.e., the “future training time steps”).

[0137] As described above with reference to FIG.5, in some embodiments the machine learning model may comprise a single “EiFormer” block (or layer, sub- 45 KILPATRICK TOWNSEND 788665001model, etc.) in a larger machine learning model. In such a case, the output of the multilayer perceptron can be input into sequence of additional EiFormer blocks and the training time series forecast can comprise the output of the final multilayer perceptron.

[0138] Expressed in other words, in some embodiments the latent attention module can comprise a first latent attention module and the training latent attention matrix can comprise a first training latent attention matrix. Likewise, the training query can comprise a first training query, the key matrix can comprise a first key matrix, the value matrix can comprise a first value matrix, and the multilayer perceptron can comprise a first multilayer perceptron. In such embodiments, the computer system can generate the training time series forecast by generating a second training query using the multilayer perceptron (e.g., by inputting the first training latent attention matrix into the multilayer perceptron), then generating a second training latent attention matrix using a second latent attention module, the second training query, a second key matrix and a second value matrix. The computer system can then generate the training time series forecast corresponding to the multivariate training time series dataset using the second training latent attention matrix and a second multilayer perceptron, e.g., by inputting the second training latent attention matrix into the second multilayer perceptron.

[0139] At step 614, the computer system can determine a loss value by comparing the training time series forecast and the plurality of ground truth forecast time series data values, e.g., by calculating a loss value based on the mean absolute error, mean squared error, root mean squared error, etc., between the training time series forecast and the plurality of ground truth training time series forecast values. In some embodiments, the computer system can generate multiple loss values, e.g., each corresponding to a component of the machine learning model (e.g., the random projection module, the latent attention module, the forecast module, etc.). If necessary, the computer system can optionally combine any number of generated loss values. In this way, the computer system can generate a combined loss that can be used to update the parameter set of the machine learning model (e.g., via stochastic gradient descent, backpropagation, etc.). The computer system can 46 KILPATRICK TOWNSEND 788665001combine the losses using any appropriate manner, including, e.g., a weighted combination of the individual losses.

[0140] At step 616, the computer system can update a parameter set of the machine learning model based on the loss value (or any combination of loss values), thereby training the machine learning model. The computer system can use any appropriate technique to update the parameter set, e.g., using backpropagation, stochastic gradient descent, etc.

[0141] In some embodiments, the parameter set of the machine learning model can comprise a plurality of key elements corresponding to the key matrix, a plurality of value elements corresponding to the value matrix, and a plurality of random projection value elements corresponding to the random projection value matrix. The parameter set of the machine learning model can further comprise a multilayer perceptron parameter set (associated with the multilayer perceptron) and a linear layer parameter set (associated with a linear layer of the encoding module). In embodiments in which a machine learning model comprises a sequence of “EiFormer” blocks according to embodiments, the parameter set of the machine learning model can comprise any additional parameters associated with other EiFormer blocks, e.g., a plurality of second key elements corresponding to a second key matrix, a plurality of second value elements corresponding to a second value matrix, etc.

[0142] At step 618, the computer system can determine if a terminating condition has been met. As described above, in some embodiments the terminating condition can comprise a defined number of training rounds, and the terminating condition can be met if a total number of training rounds performed equals or exceeds the defined number of training rounds. In other embodiments, the terminating condition can comprise a convergence condition. This terminating condition can be met if the set of model parameters converge, e.g., exhibit little to no change in consecutive training rounds. If the terminating condition has not been met, the computer system can return to step 604 and repeat the iterative training process until the terminating condition has been met, e.g., by sampling a new batch of training time series datasets. Otherwise at step 620, the computer system can complete the iterative training process. At this point, the machine learning model’s 47 KILPATRICK TOWNSEND 788665001parameters can be fixed for use during inference methods, e.g., as described in more detail further below.

[0143] Training methods according to embodiments can be performed using any other appropriate learning methods, and the examples provided herein are intended to be non-limiting.

[0144] After completing the iterative training process, the trained machine learning model can be used for various inference tasks. Reference is now made to FIG.7, which shows a flowchart corresponding to an inference method according to some embodiments. By performing the inference method, a computer system can generate a time series forecast using a machine learning model comprising a latent attention module and a forecasting module comprising a multilayer perceptron (and optionally a random projection module). As stated above, these modules can comprise sub-models of the machine learning model.

[0145] At step 702, the computer system can obtain a multivariate time series dataset comprising a plurality of time series signals (which may also be referred to as “time series datasets”, i.e., subsets of the multivariate time series dataset). Each time series signal can comprise a plurality of time series data values corresponding to a plurality of time steps. In some embodiments, the computer system can retrieve the multivariate time series dataset from a database, data stream, a local memory such as a hard drive, cloud storage, an I / O interface, or any other appropriate source. In other embodiments, the computer system can comprise a server computer that performs machine learning based services for client computers. In such embodiments, the computer system can receive a request from a client computer and can generate a time series forecast responsive to the request. In some embodiments, the computer system can receive the multivariate time series dataset with the request.

[0146] A multivariate time series dataset can comprise a “complete” set of multivariate time series data values from a multivariate time series or can comprise a subset of time series data values from a larger multivariate time series. For example, a multivariate time series dataset corresponding to an access control system could comprise a multivariate time series tracking all access requests made 48 KILPATRICK TOWNSEND 788665001by all users for the entirety of the operating time of that access control system, or could comprise time series data values corresponding to access requests made by a subset of users over e.g., the previous year.

[0147] Each time series signal can comprise any number of time series data values. In some embodiments, each time series signal can comprise a chronologically ordered sequence of time series data values. In some embodiments, each time series signal can comprise any number of features or “channels”, such that each time series data value corresponds to one or more channels (in addition to the plurality of time steps). In some embodiments, each time series signal can comprise a number of time series data values equal to a product of a number of time steps and a number of channels. In some embodiments, each time series signal can correspond to a different entity of a plurality of entities or a different entity account of a plurality of entity accounts (e.g., a different electrical utility account, a different payment processing network account, a different hospital patient account, etc.). In some embodiments, the computer system can sort or otherwise generate the multivariate time series dataset by identifying time series datasets corresponding to individual entities and combining those time series datasets to create the multivariate time series dataset.

[0148] Time series data signals can comprise both velocity features and non- velocity features, and may comprise time series data values that are relevant to a particular machine learning task for which a machine learning model is being used, or are predictive of (or believed to be predictive of) some expected output of the machine learning model. For example, if a machine learning model is being used to predict future merchant transaction rates based on a multivariate time series dataset of merchant transaction rates, that multivariate time series dataset may include, e.g., time series signals related to historical transaction rates, current economic conditions, etc. In some embodiments, time series signals may comprise or correspond to message data (e.g., corresponding to message-based communication between two computer systems over the Internet), transaction data (e.g., corresponding to access requests for resources provided by merchant resource providers), or machine performance data (e.g., corresponding to the performance of 49 KILPATRICK TOWNSEND 788665001a computer system such as a server computer, e.g., relative to some benchmark performance or performance metrics).

[0149] At step 704, the computer system can generate a corresponding signal embedding for each time series signal. In this way, the computer system can generate a plurality of signal embeddings. As stated above, embodiments of the present disclosure can adopt the Inverted Transformer’s orthogonal embedding method, in which each time series signal is represented as a separate token across all training time steps. This allows embodiments to generalize to varying number of time series signals. The generation of signal embeddings can be expressed asEmbedding (ℝ்ൈ^ ↦ ℝ^): h^ ^^ ൌ Embedding൫X:,^,:൯, h^ ∈ ℝ^, where ^^ is a multivariatetime series with ^^ time steps, ^^ entities, and ^^ channels and ^^ is the embedding dimension. The packed embedding comprising all signal embeddings can beexpressed as ^^^ ∈ ℝேൈ^.

[0150] The computer system can generate the corresponding signal embedding for each time series signal based on the plurality of time series data values in that time series signal. In some embodiments, the computer system can generate a corresponding signal embedding for each time series signal by inputting the multivariate time series dataset into an encoder module of the machine learning model, e.g., as depicted in FIG.5. In some embodiments, the encoder module can comprise a linear layer.

[0151] Each signal embedding can comprise a number of signal embedding values equal to an embedding dimension. In some embodiments, the embedding dimension may be less than the temporal dimension of each time series signal (e.g., the number of time steps associated with each time series signal). The computer system can perform various other processes at step 704 in order to generate the plurality of signal embeddings. For example, the computer system could normalize numerical time series data values, quantize numerical time series data values, etc.

[0152] At step 706, the computer system can form a query comprising the plurality of signal embeddings. The query can comprise a matrix with an embedding dimension and a signal dimension equal to the number of time series signals in the plurality of time series signals. For example, if the multivariate time series comprises 50 KILPATRICK TOWNSEND 788665001one million time series signals corresponding to one million entities in a payment processing network (e.g., merchants, consumers, etc.), if each of these time series signals was embedded to produce a signal embedding of embedding dimension 100, then the query could comprise a matrix with dimensions 1,000,000 by 100 (or vis versa). In some embodiments, the computer system can apply linear normalization to the plurality of signal embeddings prior to forming the query, e.g., using a normalization layer such as normalization layer 524 from machine learning model 500 of FIG.5.

[0153] In some embodiments, the computer system can form the query using random projection. In such embodiments, the computer system can generate an initial query based on the plurality of signal embeddings, which can comprise an initial query matrix comprising the plurality of signal embeddings. As described above with reference to FIG.5, the computer system can generate a randomly projected latent attention matrix using a random projection module, the initial query, a random key matrix, and a random projection value matrix. The query can comprise the randomly projected latent attention matrix.

[0154] Expressed in other terms, after generating the signal embeddings ^^^, the computer system can use an efficient latent attention module with a frozen random key matrix ^^ (i.e., a random projection module) to produce the randomlyprojected latent attention matrix Random_Proj^^^^^, e.g., according to ^^ ൌ^^^^^, Random_Proj^^^^^ ൌ softmax^^^^^ ^^୰୭^^୬⁄ √^^ ^^^. Using a frozen random keymatrix ^^ produces a randomized ^^^^^in the above equation, which describes the attention between each time series signal embedding and latent factors. Such a mechanism acts as a noise-equivalent regularization to mitigate model overfitting. It also encourages different machine learning model blocks to concentrate on distinct sets of signal embeddings, thereby producing diverse representations that are then further modeled by subsequent fully-learnable latent attention layers. This diversity improves embodiments’ ability to capture a wide range of spatial-temporal patterns, addresses many of the challenges associated with the scale of many real-world multivariate time series datasets, and reduces the risk of overfitting

[0155] The random key matrix can have a random key embedding dimension equal to the embedding dimension (^^ as described above) and a random key input- 51 KILPATRICK TOWNSEND 788665001independent dimension ^^. The random key input-independent dimension can be equal to a key input-independent dimension. Likewise, the random projection value matrix can have a random projection value embedding dimension equal to the embedding dimension and a random projection value input-independent dimension equal to the key input-independent dimension and random key input-independent dimensions. The input-independent dimensions can be less than the signal dimension of the multivariate training time series dataset, resulting in linear computational and memory complexity with regard to the signal dimension, rather than quadratic computational and memory complexity associated with methods such as iTransformer

[0021] .

[0156] The random projection value matrix can be learned during the iterative training process, i.e., a plurality of random projection value elements of the random projection value matrix can comprise parameters of the machine learning model that can be optimized (“learned”) during the iterative training process. The random key matrix can comprise a “frozen” or “fixed” random key matrix, e.g., in some embodiments, the computer system can generate the random key matrix by randomly generating a plurality of random key matrix elements during a training process, e.g., as described above.

[0157] At step 708, the computer system can generate a latent attention matrix using the latent attention module, the query, a key matrix, and a value matrix. The key matrix can have a key embedding dimension ^^ equal to the embedding dimension and input-independent dimension ^^. The key input-independent dimension can be less than the signal dimension of the query. Likewise, the value matrix can have a value embedding dimension ^^ equal to the embedding dimension and a value input-independent dimension equal to the key input-independentdimension. Expressed in other terms, Let ^^ ∈ ℝேൈ^ be the plurality of signalembeddings (e.g., after layer normalization), let ^^ ൌ ^^^^ be the query, let ^^ ∈be the key matrix, let ^^ ∈ ℝெൈ^ be the value matrix, and let Latent_Att^^^^ ൌsoftmax൫^^^^^⁄ √^^ ൯^^. The matrices ^^ ∈ ℝ^ൈ^, ^^ ∈ ℝெൈ^, ^^ ∈ ℝெൈ^ can be learnablematrices. Notably, the attention map ^^ ேൈெ ^ ^ୟ^^ ∈ ℝ ∝ softmax ^^^^ ^ produced by thelatent attention has an input-independent dimension ^^ as described above. Structural characteristics can be shared between time series signals, enabling the 52 KILPATRICK TOWNSEND 788665001use of fixed size latent factors to effectively capture inter-signal relationships. By using these input-independent latent factors, embodiments of the present disclosure can reduce the computational complexity from quadratic to linear, while maintaining the ability to model signal dependencies. Various input-independent dimensions can be used. In some embodiments, the key input-independent dimension can be less than 1000. In some embodiments, the key input-independent dimension can be one.

[0158] At step 710, the computer system can generate a time series forecast using the latent attention matrix and the multilayer perceptron of the forecasting module. The multilayer perceptron can comprise any number of temporal multilayer perceptron blocks. In some embodiments, the computer system can apply linear normalization to the latent attention matrix prior to inputting the latent attention matrix into the multilayer perceptron. In some embodiments, the computer system can combine the query with the latent attention matrix prior to inputting the latent attention matrix into the multilayer perceptron, e.g., as depicted in FIG.5.

[0159] In some embodiments, the time series forecast can comprise one or more future time series signals. Each future time series signal can comprise one or more future time series data values corresponding to one or more future time steps. For example, a time series signal from the multivariate time series dataset could comprise features corresponding to 12 time steps, e.g., each month in a given year, and a future time series signal could comprise predicted features corresponding to a subsequent time step, e.g., one month in the future.

[0160] As described above with reference to FIG.5, in some embodiments, the machine learning model may comprise a single “EiFormer” block (or layer, sub- model, etc.) in a larger machine learning model. In such a case, the output of the multilayer perceptron can be input into a sequence of additional EiFormer blocks and the training time series forecast can comprise the output of the final multilayer perceptron.

[0161] Expressed in other words, in some embodiments the latent attention module can comprise a first latent attention module and the latent attention matrix can comprise a first latent attention matrix. Likewise, the query can comprise a first query, the key matrix can comprise a first key matrix, the value matrix can comprise 53 KILPATRICK TOWNSEND 788665001a first value matrix, and the multilayer perceptron can comprise a first multilayer perceptron. In such embodiments, the computer system can generate the time series forecast by generating a second query using the multilayer perceptron (e.g., by inputting the first latent attention matrix into the multilayer perceptron), then generating a second latent attention matrix using a second latent attention module, the second query, a second key matrix and a second value matrix. The computer system can then generate the time series forecast corresponding to the multivariate time series dataset using the second latent attention matrix and a second multilayer perceptron, e.g., by inputting the second latent attention matrix into the second multilayer perceptron.

[0162] At step 712, the computer system can perform various further processing based on the time series forecast. For example, in some embodiments, the computer system may comprise a server computer that performs machine learning as a service for client computers and may have received a request from a client computer. In such embodiments, the computer system can generate the time series forecast responsive to the request. In such embodiments, the computer system can perform further processing by transmitting the time series forecast to the client computer responsive to the request.

[0163] Various actions, interactions, or further processing can be performed (automatically or otherwise) by either the computer system or a client computer responsive to any generated time series forecasts. As another example, in some embodiments, the computer system can receive an access request for a resource from a requestor. The computer system can evaluate the access request using the time series forecast (e.g., by evaluating whether the access request is consistent with the time series forecast), thereby determining an access request classification. The computer system can cause the resource to be provided to the requestor based on the access request classification.

[0164] As another example, in the context of a power demand forecasting system, the computer system or a client computer associated with an electrical utility could perform some action responsive to a model output that indicates increasing forecasted electric power demand. For example, a computer system or a client computer associated with a hydroelectric governor could increase the speed of 54 KILPATRICK TOWNSEND 788665001hydraulic turbines at a hydropower plant, thereby increasing electric power supply to meet the forecasted increased demand.

[0165] Various other actions, automatic or otherwise, can be performed based on the context or application of machine learning models according to embodiments, and the examples provided herein are intended to be non-limiting. Further, various tasks can be performed using machine learning models according to embodiments and model outputs generated using such machine learning models, and non- exhaustive examples are provided herein.

[0166] Various experiments were conducted to assess the effectiveness of methods and systems according to embodiments. These experiments and some of their results are described below. Some of these experiments were conducted using the LargeST dataset

[0020] . In these experiments, four sub-datasets were generated: SD (San Diego), GBA (Greater Bay Area), GLA (Greater Los Angeles), and CA (California). The statistics for these sub-datasets are presented in FIG.8 (along with the statistic for a processing network dataset, used to perform additional experiments described further below). Following the experimental setup described in

[0020] , each sub-dataset was chronologically divided into training, validation, and test sets using a 6:2:2 ratio. The LargeST

[0020] experiments involved forecasting traffic patterns for the next 12 time steps based on historical traffic sensory data. In addition to the standard case (Scenario 0) where node composition remains unchanged, three simulated scenarios from the LargeST dataset

[0020] were created to emulate real- world dynamic conditions. In the first scenario (Scenario 1) 10% of nodes in the test set are new, simulating the emergence of new time series data signals. In the second scenario (Scenario 2) 10% of nodes from the training set are absent in the test set, simulating the disappearance of time series data signals. The third scenario (Scenario 3) generally comprises a combination of Scenarios 1 and 2, with 10% new nodes in the test set and 10% of training set nodes missing from the test set.

[0167] Experiments were also conducted to evaluate methods according to embodiments using a spatial-temporal time series dataset derived from a payment processing network. This spatial-temporal time series dataset comprised per-hour transaction volume time series over a two year period, sampled from 23,849 interconnected entities (e.g., merchants, banks, etc.) within the payment processing 55 KILPATRICK TOWNSEND 788665001network. The dataset was chronologically divided into training, validation, and test sets following a 6:2:2 ratio. The forecasting task involves predicting the transaction volume for the forthcoming 7 days (comprising 168 hour long time steps) based on historical transaction volume data.

[0168] Baseline experiments were also performed in which machine learning models according to embodiments (incorporating the inverted transformer approach, random projection, and efficient latent attention) were evaluated against TSMixer [5], RPMixer

[0036] , and iTransformer

[0021] . Additionally, these experiments evaluated the performance of some of these models when combined with efficient latent attention (labeled “EiM” in the figures) and random projection methods (labeled “RP” in the figures) according to embodiments. Notably, methods were only compared that do not utilize graph structures, as the experimental datasets did not include graph information. Experiments also include a linear model baseline as a performance benchmark. Model performance was evaluated using mean absolute error (MAE), root mean squared error (RMSE), and mean absolute percentage error (MAPE).

[0169] FIGs.9A-9B present experimental results on the LargeST dataset, serving as both a performance comparison and an ablation study for efficient latent attention and random projection methods according to embodiments. The best performing results are underlined and the performance reported is computed by averaging over 12 predicted time steps. The absence of iTransformer (iXFMR) and its RP variant on the Greater Bay Area (GBA), Greater Los Angeles (GLA), and California (CA) subsets indicates that it incurs out-of-memory issues.

[0170] On the smallest dataset (San Diego - SD), an Inverted Transformer using random projection methods according to embodiments (iXFMR+RP) performs well, particularly in Scenarios 1, 2, and 3 where node composition changes, highlighting the importance of inductiveness with respect to the number of entities and the robustness brought by random projection. However, the applicability of the Inverted Transformer is limited to this small-sized dataset due to scalability issues. Second, in the standard case (Scenario 0) with no node changes, efficient latent attention methods according to embodiments enable the Inverted Transformer to process larger datasets effectively, achieving the best performance across most metrics. Third, and perhaps most importantly, for larger datasets and more complex 56 KILPATRICK TOWNSEND 788665001scenarios (the California (CA) dataset is over ten times larger than the San Diego (SD) dataset as shown in FIG.8), machine learning models according to embodiments that use both efficient latent attention and random projection methods according to embodiments demonstrate increasing advantages. The performance gap widens in favor of methods according to embodiments for the most challenging cases, showcasing their effectiveness in handling large-scale, dynamic scenarios. The methods which include the labels “EiM” or “EiFormer” and “+RP” can illustrate embodiments of the invention. As shown, the smaller error values (MAE, RMSE, MAPE) show more accurate model performance. These findings underscore the efficiency and effectiveness of embodiments of the present disclosure.

[0171] FIG.10 presents the quantitative results from experiments performed using a real-world time series transaction volume dataset. The results confirm that methods according to embodiments consistently outperform baseline models across all forecasting horizons, even on challenging large-scale transaction prediction tasks. Notably, the seemingly high errors (e.g., RMSE metric) stems from the inherent challenges of the dataset, which focuses on high-traffic merchants (which can perform millions of hourly transactions, resulting in large absolute errors), and for which the extended prediction horizon (168 hours) amplifies error accumulation. For the 1-day, 3-day, and 7-day horizons, methods according to embodiments show the best performance across all metrics. This consistent performance underscores embodiments’ robustness in modeling complex spatial-temporal behaviors from real- world scenarios. Second, compared with results obtained from the LargeST dataset, methods according to embodiments exhibit larger improvements over all baselines on the transaction dataset. On average across all horizons, methods according to embodiments improves upon the best performing baselines by 24.6% in MAE, 33.0% in RMSE, and 11.2% in MAPE. These gains further highlight the effectiveness of methods according to embodiments in handling complex, large-scale transaction data, validating their potential for practical applications. Underlined data shows the best results.

[0172] The predicted time series for time series forecasting methods according to embodiments (utilizing both efficient latent attention and random projection) and the more competitive alternatives TSMixer+EiM, RPMixer+EiM, and 57 KILPATRICK TOWNSEND 788665001iXFMR+EiM (all of which include efficient latent attention methods according to embodiments) are shown in FIGs.11A-11B. For time series signals 1102, 1104, and 1106, the transaction volume increases during the tested time steps compared to the historical time steps. Conversely, signals 1108 and 1110 exhibit decreasing or vanishing transaction volumes. For signal 1112, a change in the weekly pattern is observable. It is evident that methods according to embodiments (EiFormer) can adapt to these changes more swiftly than the alternative methods, which elucidates why methods according to embodiments outperforms the alternatives quantitatively.

[0173] Further, a network architecture analysis was performed to evaluate the differences between machine learning models incorporating both efficient latent attention and random projection methods according to embodiments (EiFormer) and an inverted transformer using efficient latent attention methods according to embodiments (iXFMR+EiM) using representative similarity measure Linear CKA

[0013] . This network architecture analysis demonstrates the effectiveness of random projection methods according to embodiments. Next, an experimental protocol is presented (refer to

[0012] for more detail, which provides a survey of other representational similarity measures and a more detailed study). Borrowing thenotation from

[0012] , given an input ^^ ∈ ℝெൈேൈ^, where ^^ and ^^ are the number ofspatial-temporal features and time horizon of ^^ independent time series respectively and a model ^^ఏ^∙^. The internal representation of the data is as follows:

[0174] All extracted representations ^^^^^ ∈ ℝெൈேൈி for each network layer ^^are the suitably flattened so that ^^^^^ ∈ ℝெൈேி. Then for two layers ^^, ^^ of a network(or different networks), their similarity is computed as ^^^^^^^^^^^^,^^^^^^ based on theequations below, where ^^^^^ ൌ∈ ℝெൈெ and ^^^^^^^^^^^^^^, ^^^^^^ stands for theHilbert-Schmidt Independent Criterion [8] and ^^ ൌ ^^ெ െis a centeringmatrix.58 KILPATRICK TOWNSEND 788665001

[0175] Based on the earlier description, pairwise layer similarity scores were computed for machine learning models according to embodiments and an inverted transformer using efficient latent attention methods according to embodiments. Unlike the inverted transformer, which has a higher degree of similarity in its learned representations through the layers, machine learning models according to embodiments show a degree of evolution in its internal representations, which leads to the later stages of the model learning distinct representations from its earlier layers, thereby demonstrating the differences in the presentation learning mechanisms of the two architectures.

[0176] Experiments were also conducted that evaluate the runtime and memory cost associated with various multivariate time series forecasting methods and models, including methods according to embodiments. These experiments empirically demonstrate the efficiency advantages of methods according to embodiments over baselines and their variants. To evaluate computational cost, the number of time series signals in the spatial-temporal data instance was varied from 10^to 10^, performing standard 12-step ahead prediction using a single data sample.

[0177] As shown in FIG.12A, methods according to embodiments exhibit near-linear runtime scaling, with performance slightly below the Linear baseline but superior to other methods. Mixer-based methods show runtime efficiency degradation beyond 10ହtime series signals, while the original Inverted Transformer encounters out-of-memory (OOM) issue at approximately 10,000 time series signals.

[0178] Second, regarding memory consumption, illustrated in FIG.12B, methods according to embodiments successfully mitigates the memory bottleneck of the original Inverted Transformer, preventing Out-of-Memory issue for large signal counts. Although the empirical memory profile of methods according to embodiments are higher than Mixer-based models, this is attributable to their larger parameter count, which enables their superior predictive performance. These results demonstrate that methods according to embodiments achieve a favorable balance 59 KILPATRICK TOWNSEND 788665001between computational efficiency and model expressiveness, making them well- suited for dynamic, large-scale spatial-temporal forecasting tasks.

[0179] Additionally, a hyper-parameter sensitivity analysis was conducted on the two most promising methods from the table of FIGs.9A-9B: the Inverted Transformer using efficient latent attention methods according to embodiments (iXFMR+EiM) and a machine learning model according to embodiments using both efficient latent attention and random projection. Such hyper-parameter sensitivity analysis was conducted using the San Diego (SD) subset from LargeST. As shown in FIGs.13A-13D, 14A-14D, and 15A-15D, both models benefit from larger model sizes with an optimal learning rate of 10ିସ. Notably, machine learning models according to embodiments (incorporating both efficient latent attention and random projection) not only shows better mean absolute error (MAE) performance, but also demonstrates lower sensitivity to hyper-parameter settings compared to iXFMR+EiM. This reduced sensitivity underscores the importance of random projection according to embodiments, which generates diverse representations and thereby improves adaptability across different task settings. As such, methods according to embodiments are particularly suitable for large-scale spatial-temporal forecasting, as they maintain consistent performance across various settings and thus reduce tuning efforts.

[0180] For completeness, benchmark results are presented in FIGs.16A-16C on the LargeST dataset, comparing methods according to embodiments with spatial- temporal baselines. In FIGs.16A-16C, the best performing results are underlined, and the performance reported in the “Average” column is computed by averaging over 12 predicted time steps. These experiments reveal that RPMixer exhibits strong performance in simpler cases, while the iXFMR+EiM variant shows advantages in more challenging scenarios. Similar to the synthesis dataset result, methods according to embodiments demonstrate superior performance on the California (CA) subset, highlighting their scalability and effectiveness in modeling dynamic, intricate spatial-temporal dependencies.

[0181] As described further above, machine learning models according to embodiments can be used to improve the accuracy of access review systems and resource security systems. Such systems can include machine learning models 60 KILPATRICK TOWNSEND 788665001used to dynamically evaluate requests for a resource and e.g., automatically approve or deny access requests based on the output of that machine learning model. Such systems can include a fraud detection system, which can be used to evaluate access requests for a computer resource or account (e.g., transactions over the Internet) to determine whether those access requests (e.g., transaction) are authorized or rejected as being invalid (e.g., fraudulent). Thus, a resource security system may receive requests to access a resource. The resource security system may include an access server for determining an outcome for the access request based on access rules.

[0182] FIG.17 shows an exemplary resource security system 1700 for authorizing access to resources in accordance with some embodiments. The resource security system 1700 may be used to provide authorized users access to a resource (e.g., via authentication) while denying access to unauthorized users. In addition, the resource security system 1700 may be used to deny invalid access requests that appear to be legitimate access requests of authorized users. The resource security system 1700 may implement access rules to identify invalid access requests based on parameters of the access request. Such parameter may correspond to fields (nodes) of a data structure that is used to distinguish invalid (e.g., fraudulent) access requests from authentic access requests. Such access rules may be based, wholly or in part on the output of a machine learning model, e.g., a prediction model (or classification model) that produces predictions (or classifications) indicating the likelihood that an access request is valid or invalid. For example, using methods according to embodiments, the resource security system 1700 (or e.g., a component of the resource security system 1700, such as access server 1700) could use a machine learning model to forecast access requests based on a time series dataset corresponding to historical access requests. The resource security system 1700 could evaluate the resulting forecast in order to determine if e.g., a recently received access request is legitimate or fraudulent (e.g., is consistent or inconsistent with the forecast), or to forecast potentially fraudulent future access requests (e.g., associated with a distributed denial of service attack on resource computer 1710). 61 KILPATRICK TOWNSEND 788665001

[0183] The resource security system 1700 includes a resource computer 1710. The resource computer 1710 may control access to a physical resource 1718, such as a building or a lockbox, or an electronic resource 1716, such as a local computer account, digital files or documents, a network database, an email inbox, a payment account, or a website login. In some embodiments, the resource computer may be a webserver, an email server, or a server of an account issuer. The resource computer 1710 may receive an access request from a user 1740 via a user device 1750 (e.g., a computer or a mobile phone) of the user 1740. The resource computer 1710 may also receive the access request from the user 1740 via a request computer 1770 coupled with an access device 1760 (e.g., a keypad or a terminal). In some embodiments, the request computer 1770 may be a resource provider. For example, the request computer 1770 and the resource computer 1710 may be the same and the access request from the user 1740 can be generated directly at the resource computer 1710.

[0184] The access device 1760 and the user device 1750 may include a user input interface such as a keypad, a keyboard, a fingerprint reader, a retina scanner, any other type of biometric reader, a magnetic stripe reader, a chip card reader, a radio frequency identification reader, or a wireless or contactless communication interface, for example. The user 1740 may input authentication information into the access device 1760 or the user device 1750 to access the resource. Authentication information may also be provided by the access device 1760 and / or the user device 1750. The authentication information may include, for example, one or more data elements of a username, an account number, a token, a password, a personal identification number, a signature, a digital certificate, an email address, a phone number, a physical address, and a network address. The data elements may be labeled as corresponding to a particular field, e.g., that a particular data element is an email address. In response to receiving authentication information input by the user 1740, the user device 1750 or the request computer 1770 may send an access request, including authentication information, to the resource computer 1710 along with one or more parameters of the access request.

[0185] In one example, the user 1740 may enter one or more of an account number, a personal identification number, and password into the access device 62 KILPATRICK TOWNSEND 7886650011760, to request access to a physical resource (e.g., to open a locked security door in order to access a building or a lockbox) and the request computer 1770 may generate and send an access request to the resource computer 1710 to request access to the resource. In another example, the user 1740 may operate the user device 1750 to request that the resource computer 1710 provide access to the electronic resource 1716 (e.g., a website or a file) that is hosted by the resource computer 1710. In another example, the user device 1750 may send an access request (e.g., an email) to the resource computer 1710 (e.g., an email server) in order to provide data to the electronic resource 1716 (e.g., deliver the email to an inbox). In another example, the user 1740 may provide an account number and / or a personal identification number to an access device 1760 in order to request access to a resource (e.g., a payment account) for conducting a transaction.

[0186] In some embodiments, the resource computer 1710 may verify the authentication information of the access request based on information stored at the request computer 1770. In other embodiments, the request computer 1770 may verify the authentication information of the access request based on information stored at the resource computer 1710.

[0187] The resource computer 1710 may receive the request substantially in real-time (accounting for delays in computer processing and electronic communication). Once the access request is received, the resource computer 1710 may determine parameters of the access request. In some embodiments, the parameters may be provided by the user device 1750 or the request computer 1770. For example, the parameters may include one or more of: a time that the access request was received, a day of the week that the access request was received, the source-location of the access request, the amount of resources requested, an identifier of the resource being request, an identifier of the user 1740, the access device 1760, the user device 1750, the request computer 1770, a location of the user 1740, the access device 1760, the user device 1750, the request computer 1770, an indication of when, where, or how the access request is received by the resource computer 1710, an indication of when, where, or how the access request is sent by the user 1740 or the user device 1750, an indication of the requested use of the electronic resource 1716 or the physical resource 1718, and an indication of the 63 KILPATRICK TOWNSEND 788665001type, status, amount, or form of the resource being requested. In other embodiments, the request computer 1770 or the access server 1720 may determine the parameters of the access request.

[0188] The resource computer 1710 or the request computer 1770 may send the parameters of the access request to the access server 1720 in order to determine whether the access request is invalid. The access server 1720 may store one or more access rules 1722 for identifying an invalid access request. Each of the access rules 1722 may include one or more conditions corresponding to one or more parameters of the access request. The access server 1720 may determine an access request outcome indicating whether the access request should be accepted (e.g., access to the resource granted), rejected (e.g., access to the resource denied), or reviewed by comparing the access rules 1722 to the parameters of the access request as further described below. In some embodiments, instead of determining an access request outcome, the access server 1720 may determine an evaluation score based on outcomes of the access rules. The evaluation score may indicate the risk or likelihood of the access request being invalid. If the evaluation score indicates that the access request is likely to be invalid, then the access server 1720 may reject the access request. An access server may determine such an evaluation score using machine learning techniques, e.g., using methods and machine learning models according to embodiments described above.

[0189] The access server 1720 may send the indication of the access request outcome to the resource computer 1710 (e.g., accept, reject, review, accept and review, or reject and review). In some embodiments, the access server 1720 may send the evaluation score to the resource computer 1710 instead. The resource computer 1710 may then grant or deny access to the resource based on the indication of the access request outcome or based on the evaluation score. The resource computer 1710 may also initiate a review process for the access request.

[0190] In some embodiments, the access server 1720 may be remotely accessed by an administrator for configuration. The access server 1720 may store data in a secure environment and implement user privileges and user role management for accessing different types of stored data. For example, user privileges may be set to enable users to perform one or more of the following 64 KILPATRICK TOWNSEND 788665001operations: view logs of received access request, view logs of access request outcomes, enable or disable the execution of the access rules 1722, update or modify the access rules 1722, change certain access request outcomes. Different privileges may be set for different users.

[0191] The resource computer 1710 may store access request information for each access requests that it receives. The access request information may include authentication information and / or the parameters of each of the access requests. The access request information may also include an indication of the access request outcome for the access request, e.g., indicating whether the access request was actually valid or invalid. The resource computer 1710 may also store validity information corresponding to each access request. The validity information for an access request may be initially based on its access request outcome. The validity information may be updated based on whether the access request is reported to be invalid. In some embodiments, the access server 1720 or the request computer 1770 may store the access request information and the validity information.

[0192] FIG.18 shows an example computer system 1800 according to some embodiments. Computer system 1800 can comprise any variety of computing device, e.g., a server computer, a desktop or laptop computer, a tablet, a mobile phone, a smartwatch, etc. Computer systems mentioned herein may utilize any suitable number of subsystems. Examples of such subsystems are shown in computer system 1800. In some embodiments, a computer system includes a single computer apparatus and the subsystems can be the components of the computer apparatus. In other embodiments, a computer system can include multiple computer apparatuses with internal components, each being a subsystem of the computer system.

[0193] The subsystems shown in computer system 1800 can be connected via a system bus 1820. Additional subsystems such as a printer 1818, keyboard 1814, computer readable media 1822 (including system memory, storage devices, etc.), data collection device 1816 (e.g., a camera, microphone, accelerometer, GPS unit, fingerprint scanner, etc.), monitor 1810 (e.g., a display screen, such as an LED display), which is coupled to display adaptor 1808, and others, are shown. Peripherals and input / output (I / O) devices, which couple to I / O controller 1806, can 65 KILPATRICK TOWNSEND 788665001be connected to computer system 1800 by any number of means known in the art such as an input / output port 1812 (e.g., USB, FireWire®). Likewise, I / O port 1812 or an external interface 1804 (e.g., Ethernet, Wi-Fi, Near Field Communication, Bluetooth, ZigBee interfaces, etc.) can be used to connect computer system 1800 to various networks, devices, and computer systems, such as a wide area network such as the Internet, a mouse input device, a scanner, an access device (e.g., a point-of-sale terminal), a resource provider computer, etc.

[0194] Computer system 1800 can include a plurality of the same components or subsystems, e.g., connected together by external interface 1804, by an internal interface, or via removal storage devices that can be connected and removed from one component to another component. Additionally, computer system 1800 may lack (even temporarily) some of the components and subsystems depicted in FIG. 18. As such, it should be understood that the various components and subsystems depicted in FIG.18 are optional.

[0195] The interconnection via system bus 1820 allows the processor 1802 (which may comprise a “central processor”, “central processing unit”, “CPU”, graphical processing unit, GPU, or other like terms) to communicate with each subsystem to control the execution of instructions or code from computer readable medium 1822 (e.g., system memory, a fixed disk such as a hard drive, a solid state drive, an optical disk, etc.), as well as the exchange of information between subsystems. Various data and software modules can be stored on computer readable media 1822, including a machine learning model 1824, an inference module 1834, and a training module 1836, which can be used by computer system 1800 to implement various methods described herein. It should be understood that the particular data and software modules depicted in FIG.18 (and their configuration) are intended to represent only one possible implementation, selected for ease of exposition, and are not intended to be limiting. Various other configurations may become apparent upon reading this disclosure, e.g., implementing methods according to embodiments using a single monolithic software module (and any associated data, including machine learning model parameters) rather than using data and software modules depicted in FIG.18. Further, it should be understood that various data and / or software modules that could be included in 66 KILPATRICK TOWNSEND 788665001computer readable media 1822 of computer system 1800 are omitted for brevity. As an example, FIG.18 does not depict an operating system (i.e., system software that manages hardware and software resources, provides common services and task scheduling, etc.), even though many computer systems have operating systems.

[0196] These data and software modules can include a machine learning model 1824. As described above, computer system 1800 can use machine learning model 1824 to perform various data analysis or machine learning tasks. For example, computer system 1800 can use machine learning model 1824 (and optionally inference module 1834) to generate time series forecasts corresponding to multivariate time series data. Machine learning model 1824 can include an encoder module 1826, which can be used by the computer system to generate signal embeddings (e.g., embedding vectors) corresponding to time series signals from a multivariate time series dataset. Machine learning model 1824 can further include a random projection module 1828 and a latent attention module 1830, which can be used by the computer system 1800 to perform processes associated with random projection and latent attention mechanisms according to embodiments, as described above. Machine learning model 1824 can further include a forecasting module 1832, which can be used to produce time series forecasts, e.g., based on latent attention matrices produced via latent attention module 1830. Computer system 1800 may use training module 1836 to train machine learning model 1824 using any applicable training method including training methods described above.

[0197] In closing, it should be understood that any of the computer systems mentioned herein may utilize any suitable number of subsystems. In some embodiments, a computer system includes a single computer apparatus, where the subsystems can be components of the computer apparatus. In other embodiments, a computer system can include multiple computer apparatuses, each being a subsystem, with internal components.

[0198] Further, it should be understood that a computer system can include a plurality of the components or subsystems, e.g., connected together by external interface or by an internal interface. In some embodiments, computer systems, subsystems, or apparatuses can communicate over a network. In such instances, one computer can be considered a client and another computer a server, where 67 KILPATRICK TOWNSEND 788665001each can be part of a same computer system. A client and a server can each include multiple systems, subsystems, or components.

[0199] It should be understood that any of the embodiments of the present invention can be implemented in the form of control logic using hardware (e.g., an application specific integrated circuit or field programmable gate array) and / or using computer software with a generally programmable processor in a modular or integrated manner, and thus a processor can include memory storing software instructions that configure hardware circuitry, as well as an FPGA with configuration instructions or an ASIC. As used herein a processor can include a single-core processor, multi-core processor on a same integrated chip, or multiple processing units on a single circuit board or networked. The computations can be performed in parallel by the different processing units and / or different processing threads of a single processing unit. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will know and appreciate other ways and / or methods to implement embodiments of the present invention using hardware and a combination of hardware and software.

[0200] Any of the software components or functions described in this application may be implemented as software code to be executed by a processor using any suitable computer language such as, for example, Java, C, C++, C#, Objective-C, Swift, or scripting language such as Perl or Python using, for example, conventional or object-oriented techniques. The software code may be stored as a series of instructions or commands on a computer readable medium for storage and / or transmission, suitable media include random access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a compact disk (CD) or DVD (digital versatile disk) or Blu-ray disk, flash memory, and the like. The computer readable medium may be any combination of such storage or transmission devices. In addition, the order of operations may be re-arranged. A process can be terminated when its operations are complete but could have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function of the main function. 68 KILPATRICK TOWNSEND 788665001

[0201] Such programs may also be encoded and transmitted using carrier signals adapted for transmission via wired, optical, and / or wireless networks (including the Internet) conforming to a variety of protocols. As such, a computer readable medium may be created using a data signal encoded with such programs. Computer readable media encoded with the program code may be packaged with a compatible device (e.g., as firmware) or provided separately from other devices (e.g., via Internet download). Any such computer readable medium may reside on or within a single computer product (e.g. a hard drive, a CD, or an entire computer system), and may be present on or within different computer products within a system or network. A computer system may include a monitor, printer or other suitable display for providing any of the results mentioned herein to a user.

[0202] Any of the methods described herein may be totally or partially performed with a computer system including one or more processors, which can be configured to perform the steps. Thus, embodiments can involve computer systems configured to perform the steps of any of the methods described herein, potentially with different components performing a respective step or a respective group of steps. Although presented as numbered steps, steps of methods herein can be performed at a same time or in a different order. Additionally, portions of these steps may be used with portions of other steps from other methods. Also, all or portions of a step may be optional. Additionally, and of the steps of any of the methods can be performed with modules, circuits, or other means for performing these steps.

[0203] Any operation performed with a processor may be performed in real- time. The term “real-time” may refer to computing operations or processes that are completed within a certain time constraint. As examples, a time constraint may be 30 seconds, 1 minute, 10 minutes, 30 minutes, 1 hour, 4 hours, 1 day, or 7 days.

[0204] In some embodiments, computer systems, subsystems, or apparatuses can communicate over a network. In such instances, one computer can be considered a client and another computer a server, where each can be part of a same computer system. A client and a server can each involve multiple systems, subsystems, or components. In various embodiments, methods may involve various numbers of clients and / or servers, including at least 10, 20, 50, 100, 200, 500, 1,000, or 10,000 devices. Methods can include various numbers of communication 69 KILPATRICK TOWNSEND 788665001messages between devices, including at least 100, 200, 500, 1,000, 10,000, 50,000, 100,000, 500,000, or one million communication messages. Such communications can involve at least 1 KB, 1 MB, 10 MB, 100 MB, 1 GB, 10 GB, or 100 GB of data.

[0205] The specific details of particular embodiments may be combined in any suitable manner without departing from the spirit and scope of embodiments of the invention. However, other embodiments of the invention may be directed to specific embodiments relating to each individual aspect, or specific combinations of these individual aspects.

[0206] The above description of exemplary embodiments of the invention has been presented for the purpose of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form described, and many modifications and variations are possible in light of the teaching above. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications to thereby enable others skilled in the art to best utilize the invention in various embodiments and with various modifications as are suited to the particular use contemplated.

[0207] The above description is illustrative and is not restrictive. Many variations of the invention will become apparent to those skilled in the art upon review of the disclosure. The scope of the invention should, therefore, be determined not with reference to the above description, but instead should be determined with reference to the pending claims along with their full scope or equivalents.

[0208] One or more features from any embodiment may be combined with one or more features of any other embodiment without departing from the scope of the invention.

[0209] A recitation of “a”, “an” or “the” is intended to mean “one or more” unless specifically indicated to the contrary. The use of “or” is intended to mean an “inclusive or,” and not an “exclusive or” unless specifically indicated to the contrary. Reference to a “first” component does not necessarily require that a second component be provided. Moreover, reference to a “first” or “second” component 70 KILPATRICK TOWNSEND 788665001does not limit the referenced component to a particular location unless explicitly stated. The term “based on” is intended to mean “based at least in part on.”

[0210] The claims may be drafted to exclude any elements which may be optional. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as “solely”, “only”, and the like in connection with the recitation of claim elements, or the use of a “negative” limitation.

[0211] All patents, patent applications, publications and descriptions mentioned herein are incorporated by reference in their entirety for all purposes. None is admitted as prior art. Wherein a conflict exists between the instant application and a reference provided herein, the instant application shall dominate. 71 KILPATRICK TOWNSEND 788665001I. REFERENCES [1] Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang.2020. Adaptive graph convolutional recurrent network for traffic forecasting. Advances in neural information processing systems 33 (2020), 17804–17815. [2] Arij Bouazizi, Adrian Holzbock, Ulrich Kressel, Klaus Dietmayer, and Vasileios Belagiannis.2022. MotionMixer: MLP-based 3D Human Body Pose Forecasting. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22. International Joint Conferences on Artificial Intelligence Organization, 791–798. [3] Khac-Hoai Nam Bui, Jiho Cho, and Hongsuk Yu.2022. Spatial-temporal graph neural network fro traffic forecasting: An overview and open research issues. Applied Intelligence 52, 3 (2022), 2763-2774. [4] Haoming Chen, Yetain Chen, Jingjing Meng, Yang Jiao, Yikai Ni, Yan Gao, Michinari Momma, and Yi Sun.2023. Improving product search with season-aware query-product semantic similarity. In Companion Proceedings of the ACM Web Conference 2023.864-868. [5] Si-An Chen, Chun-Liang Li, Nate Yoder, Sercan O Arik, and Tomas Pfister. 2023. Tsmixer: An all-mlp architecture for time series forecasting. arXiv preprint arXiv:2303.06053 (2023). [6] Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan Mathur, Rajat Sen, and Rose Yu.2023. Long-term forecasting with tide: Time series dense encoder. arXiv preprint arXiv:2304.08424 (2023). [7] Zheng Fang, Qingqing Long, Guojie Song, and Kunqing Xie.2021. Spatial- Temporal Graph ODE Networks for Traffic Flow Forecasting. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining.364–373. [8] Arthur Gretton, Kenji Fukumizu, Choon Teo, Le Song, Bernhard Schölkopf, and Alex Smola.2007. A Kernel Statistical Test of Independence. In Advances in Neural Information Processing Systems, J. Platt, D. Koller, Y. Singer, and S. Roweis (Eds.), Vol.20. Curran Associates, Inc. 72 KILPATRICK TOWNSEND 788665001https: / / proceedings.neurips.cc / paper_files / paper / 2007 / file / d5cfead94f5350c 12c322b5b664544c1-Paper.pdf [9] Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. 2019. Attention based spatial-temporal graph convolutional networks for traffic flow forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol.33.922–929.

[0010] Yang Jiao, Yan Gao, Jingjing Meng, Jin Shang, and Yi Sun.2023. Learning attribute and class-specific representation duet for fine-grained fashion analysis. In Proceedings of the IEEE / CVF Conference of Computer Vision and Pattern Recognition.11050-11059.

[0011] Yang Jiao, Ning Xie, Yan Guao, Chien-Chih Wang, and Yi Sun.2022. Fine- grained fashion representation learning by online deep clustering. In European conference on computer vision. Springer, 19-35.

[0012] Max Klabunde, Tobias Schumancher, Markus Strohmaier, and Florian Lemmerich.2024. Similarity of Neural Network Models: Asurvey of Functional and Representational Measures. arXiv:2305.06329 [cs.LG] https: / / arxiv.org / abs / 2305.06329

[0013] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. 20219. Similarity of Neural Network Representations Revisited. CoRR abs / 1905.00414. arXiv:1905.00414 htpps: / / arxiv.org / abs / 1904.00414

[0014] Shiyong Lan, Yitong Ma, Weikang Huang, Wenwu Wang, Hongyu Yang, and Pyang Li.2022. Dstagnn: Dynamic spatial-temporal aware graph neural network for traffic flow forecasting. In International conference on machine learning. PMLR, 11906–11917 (2019).

[0015] Fuxian Li, Jie Feng, Huan Yan, Guangyin Jin, Fan Yang, Funing Sun, Depeng Jin, and Yong Li.2023. Dynamic Graph Convolutional Recurrent Network for Traffic Prediction: Benchmark and Solution. ACM Trans. Knowl. Discov. Data 17, 1 (2023).

[0016] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan.2019. Enhancing the locality and breaking the 73 KILPATRICK TOWNSEND 788665001memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems 32 (2019).

[0017] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu.2018. Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting. In International Conference on Learning Representations (ICLR ’18).

[0018] Gang Liu, Fan Yang, Yang Andrew Jiao, Ali Bagheri Garakani, Tian Tong, Yan Gao, and Meng Jiang.2025. Learning attribute as explicit relation for sequential recommendation. (2025).

[0019] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar.2021. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations.

[0020] Xu Liu, Yutong Xia, Yuxuan Liang, Junfeng Hu, Yiwei Wang, Lei Bai, Chao Huang, Zhenguang Liu, Bryan Hooi, and Roger Zimmermann.2024. Largest: A benchmark dataset for large-scale traffic forecasting. Advances in Neural Information Processing Systems 36 (2024).

[0021] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long.2023. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625 (2023).

[0022] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2022. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730 (2022).

[0023] Chao Shang, Jie Chen, and Jinbo Bi.2021. Discrete graph structure learning for forecasting multiple time series. arXiv preprint arXiv:2101.06861 (2021).

[0024] Jin Shang, Yang Jiao, Chenghuan Guo, Minghao Sun, Yan Gao, Jia Liu, Michinari Momma, Itetsu Taru, and Yi Sun.2024. Transitivity-encoded graph attention networks for complementary item recommendations. In 2024 IEEE International Conference on Data Mining (ICDM). IEEE, 430- 439. 74 KILPATRICK TOWNSEND 788665001

[0025] Zezhi Shao, Zhao Zhang, Fei Wang, and Yongjun Xu.2022. Pre-training enhanced spatial-temporal graph neural network for multivariate time series forecasting. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.1567–1577.

[0026] Zezhi Shao, Zhao Zhang, Wei Wei, Fei Wang, Yongjun Xu, Xin Cao, and Christian S. Jensen.2022. Decoupled Dynamic Spatial-Temporal Graph Neural Net- work for Traffic Forecasting. Proc. VLDB Endow.15, 11 (2022), 2733–2746.

[0027] Jiarui Sun, M Ugur Akcal, Wei Zhang, and Girish Chowdhary.2024. MOOSS: Mask-Enhanced Temporal Contrastive Learning for Smooth State Evolution in Visual Reinforcement Learning. Arxiv preprint arXiv:2409.02714 (2024).

[0028] Jiarui Sun and Girish Chowdhary.2023. Toward accurate human motion prediction via iterative refinement. arXiv preprint arXiv:2305.04443 (2023).

[0029] Jiarui Sun and Girish Chowdhary.2024. CoMusion: Towards Consistent Stochastic Human Motion Prediction via Motion Diffusion. In European Conference on Computer Vision. Springer, 18-36.

[0030] Jiarui Sun, Yujie Fan, Chin-Chia Michael Yeh, Wei Zhang, and Girish Chowdhary.2024. Revealing the Power of Spatial-Temporal Masked Autoencoders in Multivariate Time Series Forecasting. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management.

[0031] The Authors.2024. Project Website. https: / / sites.google.com / view / eiformer.

[0032] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long.2021. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems 34 (2021), 22419–22430. 75 KILPATRICK TOWNSEND 788665001

[0033] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang.2020. Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.

[0034] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. 2019. Graph wavenet for deep spatial-temporal graph modeling. In Proceedings of the 28th International Joint Conference on Artificial Intelligence.1907–1913.

[0035] Mingxing Xu, Wenrui Dai, Chunmiao Liu, Xing Gao, Weiyao Lin, Guo-Jun Qi, and Hongkai Xiong.2020. Spatial-temporal transformer networks for traffic flow forecasting. arXiv preprint arXiv:2001.02908 (2020).

[0036] Chin-Chia Michael Yeh, Yujie Fan, Xin Dai, Uday Singh Saini, Vivian Lai, Prince Osei Aboagye, Junpeng Wang, Huiyuan Chen, Yan Zheng, Zhongfang Zhuang, et al.2024. RPMixer: Shaking Up Time Series Forecasting with Random Projections for Large Spatial-Temporal Data. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.

[0037] Bing Yu, Haoteng Yin, and Zhanxing Zhu.2018. Spatio-temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI).

[0038] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu.2023. Are transformers effective for time series forecasting?. In Proceedings of the AAAI conference on artificial intelligence, Vol.37.11121–11128.

[0039] Zijian Zhang, Ze Huang, Zhiwei Hu, Xiangyu Zhao, Wanyu Wang, Zitao Liu, Junbo Zhang, S. Joe Qin, and Hongwei Zhao.2023. MLPST: MLP is All You Need for Spatio-Temporal Prediction. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 3381–3390. 76 KILPATRICK TOWNSEND 788665001

[0040] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang.2021. Informer: Beyond efficient transformer for long sequence time series forecasting. In Proceedings of the AAAI conference on artificial intelligence, Vol.35.11106–11115.

[0041] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. Fedformer: Frequency enhanced decomposed transformer for long- term series forecasting. In International conference on machine learning. PMLR, 27268–27286. 77 KILPATRICK TOWNSEND 788665001

Claims

WHAT IS CLAIMED IS:

1. A method for generating a time series forecast using a machine learning model comprising a latent attention module and a forecasting module comprising a multilayer perceptron, the method comprising: obtaining, by a computer system, a multivariate time series dataset comprising a plurality of time series signals, each time series signal comprising a plurality of time series data values corresponding to a plurality of time steps; generating, by the computer system, for each time series signal, a corresponding signal embedding based on the plurality of time series data values in that time series signal, thereby generating a plurality of signal embeddings, wherein each signal embedding comprises a number of embedding values equal to an embedding dimension; forming, by the computer system, a query comprising the plurality of signal embeddings, wherein the query comprises a matrix with the embedding dimension and a signal dimension equal to a number of time series signals in the plurality of time series signals; generating, by the computer system, a latent attention matrix using the latent attention module, the query, a key matrix, and a value matrix, wherein: the key matrix has a key embedding dimension equal to the embedding dimension and a key input-independent dimension, the key input-independent dimension is less than the signal dimension of the query, and the value matrix has a value embedding dimension equal to the embedding dimension and a value input-independent dimension equal to the key input-independent dimension; and generating, by the computer system, using the latent attention matrix and the multilayer perceptron of the forecasting module, a time series forecast corresponding to the multivariate time series dataset.

2. The method of claim 1, wherein the computer system comprises a server computer, and wherein the method further comprises: 78 KILPATRICK TOWNSEND 788665001receiving, by the computer system, a request from a client computer, wherein the computer system generates the time series forecast responsive to the request; and transmitting, by the computer system, the time series forecast to the client computer responsive to the request.

3. The method of claim 1, further comprising: receiving, by the computer system, an access request for a resource from a requestor; evaluating, by the computer system, the access request using the time series forecast, thereby determining an access request classification; and causing, by the computer system, the resource to be provided to the requestor based on the access request classification.

4. The method of claim 1, wherein: the key matrix comprises a learned key matrix comprising a plurality of learned key elements determined during a machine learning model training process; and the value matrix comprises a learned value matrix comprising a plurality of learned value elements determined during the machine learning model training process.

5. The method of claim 1, wherein the key input-independent dimension is less than 1000.

6. The method of claim 1, wherein the key input-independent dimension is one.

7. The method of claim 1, wherein: the latent attention module comprises a first latent attention module; the latent attention matrix comprises a first latent attention matrix; the query comprises a first query; the key matrix comprises a first key matrix; the value matrix comprises a first value matrix; the multilayer perceptron comprises a first multilayer perceptron; and 79 KILPATRICK TOWNSEND 788665001generating the time series forecast comprises: generating, by the computer system, a second query using the multilayer perceptron, generating, by the computer system, a second latent attention matrix using a second latent attention module, the second query, a second key matrix, and a second value matrix, and generating, by the computer system, using the second latent attention matrix and a second multilayer perceptron, the time series forecast corresponding to the multivariate time series dataset.

8. The method of claim 1, wherein: each plurality of time series data values corresponds to one or more channels in addition to the plurality of time steps, each time series signal thereby comprising a plurality of time series data values equal to a product of a number of time steps and a number of channels; and the embedding dimension is less than the product of the number of time steps and the number of channels.

9. The method of claim 1, wherein generating a corresponding signal embedding for each time series signal comprises inputting, by the computer system, the multivariate time series dataset into an encoder module of the machine learning model, the encoder modules comprising a linear layer.

10. The method of claim 1, wherein the time series forecast comprises one or more future time series signals, each future time series signal comprising one or more future time series data values corresponding to one or more future time steps.

11. The method of claim 1, wherein each time series signal corresponds to a different entity of a plurality of entities or a different entity account of a plurality of entity accounts.

12. The method of claim 1, wherein forming the query comprises: generating, by the computer system, an initial query based on the plurality of signal embeddings; and 80 KILPATRICK TOWNSEND 788665001generating, by the computer system, a randomly projected latent attention matrix using a random projection module, the initial query, a random key matrix, and a random projection value matrix, wherein the query comprises the randomly projected latent attention matrix.

13. The method of claim 12, wherein: the random key matrix has a random key embedding dimension equal to the embedding dimension and a random key input-independent dimension equal to the key input-independent dimension; and the random projection value matrix has a random projection value embedding dimension equal to the embedding dimension and a random projection value input-independent dimension equal to the key input-independent dimension.

14. The method of claim 12, wherein the random projection value matrix comprises a learned random projection value matrix comprising a plurality of learned random projection value elements determined during a machine learning model training process.

15. A method for training a machine learning model comprising a latent attention module and a forecasting module comprising a multilayer perceptron, the method comprising performing an iterative training process until a terminating condition has been met, the method comprising: sampling, by a computer system, a multivariate training time series dataset comprising a plurality of training time series signals, each training time series signal comprising a plurality of training time series data values and a plurality of ground truth forecast time series data values corresponding to a plurality of training time steps; generating, by the computer system, for each training time series signal, a corresponding training signal embedding based on the plurality of training time series data values in that training time series signal, thereby generating a plurality of training signal embeddings, wherein each training signal embedding comprises a number of training embedding values equal to an embedding dimension; 81 KILPATRICK TOWNSEND 788665001forming, by the computer system, a training query comprising the plurality of training signal embeddings, wherein the training query comprises a training matrix with the embedding dimension and a signal dimension equal to a number of training time series signals in the plurality of training time series signals; generating, by the computer system, a training latent attention matrix using the latent attention module, the training query, a key matrix, and a value matrix, wherein: the key matrix has a key embedding dimension equal to the embedding dimension and a key input-independent dimension, the key input-independent dimension is less than the signal dimension of the training query, and the value matrix has a value embedding dimension equal to the embedding dimension and a value input-independent dimension equal to the key input-independent dimension; generating, by the computer system, using the training latent attention matrix and the multilayer perceptron of the forecasting module, a training time series forecast; determining, by the computer system, a loss value by comparing the training time series forecast and the plurality of ground truth forecast time series data values; updating, by the computer system, a parameter set of the machine learning model based on the loss value, thereby training the machine learning model; and repeating, by the computer system, the iterative training process until the terminating condition has been met.

16. The method of claim 15, further comprising, after completing the iterative training process: obtaining, by the computer system, a multivariate time series dataset comprising a plurality of time series signals, each time series signal comprising a plurality of time series data values corresponding to a plurality of time steps; generating, by the computer system, for each time series signal, a corresponding signal embedding based on the plurality of time series data values in 82 KILPATRICK TOWNSEND 788665001that time series signal, thereby generating a plurality of signal embeddings, wherein each signal embedding comprises a number of embedding values equal to the embedding dimension; forming, by the computer system, a query comprising the plurality of signal embeddings, wherein the query comprises a matrix with the embedding dimension and a signal dimension equal to a number of time series signals in the plurality of time series signals; generating, by the computer system, a latent attention matrix using the latent attention module, the query, the key matrix, and the value matrix; and generating, by the computer system, using the latent attention matrix and the multilayer perceptron of the forecasting module, a time series forecast corresponding to the multivariate time series dataset.

17. The method of claim 15, wherein forming the training query comprises: generating, by the computer system, an initial training query based on the plurality of training signal embeddings; and generating, by the computer system, a randomly projected training latent attention matrix using a random projection module, the initial training query, a random key matrix, and a random projection value matrix, wherein the training query comprises the randomly projected training latent attention matrix.

18. The method of claim 17, wherein the parameter set of the machine learning model comprises: a plurality of key elements corresponding to the key matrix; a plurality of value elements corresponding to the value matrix; a plurality of random projection value elements corresponding to the random projection value matrix; and a plurality of weights corresponding to the multilayer perceptron.

19. The method of claim 17, further comprising generating, by the computer system, the random key matrix by randomly generating a plurality of random key matrix elements.

20. A computer system comprising: 83 KILPATRICK TOWNSEND 788665001a processor; and a non-transitory computer readable medium coupled to the processor, the non-transitory computer readable medium comprising instructions executable by the processor for performing a method for generating a time series forecast using a machine learning model comprising a latent attention module and a forecasting module comprising a multilayer perceptron, the method comprising: obtaining a multivariate time series dataset comprising a plurality of time series signals, each time series signal comprising a plurality of time series data values corresponding to a plurality of time steps; generating, for each time series signal, a corresponding signal embedding based on the plurality of time series data values in that time series signal, thereby generating a plurality of signal embeddings, wherein each signal embedding comprises a number of embedding values qual to an embedding dimension; forming a query comprising the plurality of signal embeddings, wherein the query comprises a matrix with the embedding dimension and a signal dimension equal to a number of time series signals in the plurality of time series signals; generating a latent attention matrix using the latent attention module, the query, a key matrix, and a value matrix, wherein: the key matrix has a key embedding dimension equal to the embedding dimension and a key input-independent dimension, the key input-independent dimension is less than the signal dimension of the query, and the value matrix has a value embedding dimension equal to the embedding dimension and a value input-independent dimension equal to the key input-independent dimension; and generating, using the latent attention matrix and the multilayer perceptron of the forecasting module, a time series forecast corresponding to the multivariate time series dataset. 84 KILPATRICK TOWNSEND 788665001

Images