Encoder-decoder architectures

Abstract

Encoder-Decoder Architecture System Control Event data (1024) representing a given event that occurred at a given time is obtained. The event data (1024) is provided as input to an encoder network (1006) to generate encoded event data (1026) comprising an encoded representation of the given event. The encoded event data (1026) is provided as input to a decoder network (1010) to generate decoded event data (1028). The decoded event data (1028) represents one or more further events. Each of the one or more further events occurs at a respective further time. The decoded event data (1028) and reference event data (1030) are provided as inputs to a loss function (1032) to generate loss data. The reference event data (1030) represents the one or more further events. The loss data represents one or more losses of the decoded event data (1028) relative to the reference event data (1030). Encoder configuration data (1032), comprising one or more encoder parameters, is generated using the loss data.

Description

[0001] Encoder-Decoder Architectures

[0002] Technical Field

[0003] The present disclosure relates to encoder-decoder architectures.

[0004] In particular, but not exclusively, the present disclosure relates to measures (for example methods, systems, and computer programs) to control a system having an encoder-decoder architecture.

[0005] Background Art

[0006] Systems having an encoder-decoder architecture include at least one encoder and at least one decoder. Such systems can be configured and used in various different ways, for example depending on the purpose of the system, the type(s) of data the system handles, and so on.

[0007] Existing systems having encoder-decoder architectures suffer from various drawbacks. Examples of such drawbacks include, but are not limited to, restricted downstream utility, lack of data richness, long development and deployment times, overfitting and high latency.

[0008] Summary

[0009] According to first embodiments, there is provided a method of controlling a system having an encoderdecoder architecture, the encoder-decoder architecture having an encoder network and a decoder network, the encoder network comprising at least one encoder, and the decoder network comprising at least one decoder, wherein the method comprises: obtaining event data, the event data representing a given event that occurred at a given time; providing the event data as input to the encoder network to generate encoded event data, the encoded event data comprising an encoded representation of the given event that occurred at the given time; providing the encoded event data as input to the decoder network to generate decoded event data, the decoded event data representing one or more further events, wherein each of the one or more further events occurs at a respective further time; providing the decoded event data and reference event data as inputs to one or more loss functions to generate loss data, wherein the reference event data represents the one or more further events, and wherein the loss data represents one or more losses of the decoded event data relative to the reference event data; and generating encoder configuration data using the loss data, the encoder configuration data comprising one or more encoder parameters.

[0010] According to second embodiments, there is provided a method comprising: providing input event data to an encoder network, the encoder network having been configured based on encoder configuration data generated in accordance with a method according to the first embodiments; and obtaining encoded event data from the encoder network, the encoded event data being based on the input event data. According to third embodiments, there is provided a method comprising using generative autoregression to generate output event data based on input event data, the input event data comprising time-ordered, event-driven data.

[0011] According to fourth embodiments, there is provided a method comprising: using generative autoregression to generate expected event data based on past event data, the expected event data comprising first expected event data representing a first expected event and second expected event data representing a second expected event; and generating a composite loss based on: a difference between the first expected event data and first actual event data, the first actual event data representing a first actual event; and a difference between the second expected event data and second actual event data, the second actual event data representing a second actual event.

[0012] According to fifth embodiments, there is provided a method comprising using generative autoregression to generate expected transaction data based on past transaction data, the past transaction data representing one or more past transactions and the expected transaction data representing one or more expected transactions.

[0013] According to sixth embodiments, there is provided a method comprising using generative autoregression to generate expected event data based on past event data, the past event data representing one or more past events and the expected event data representing one or more expected events.

[0014] According to seventh embodiments, there is provided a system configured to perform a method according to any of the first through sixth embodiments.

[0015] According to eighth embodiments, there is provided a computer program configured to perform a method according to any of the first through sixth embodiments.

[0016] Brief Description of the Drawings

[0017] Certain embodiments of the present disclosure will now be described with reference to the accompanying drawings, in which:

[0018] Figure 1 is a schematic diagram showing an example of a system having an encoder-decoder architecture.

[0019] Figure 2 is a schematic diagram showing an example of a loss function.

[0020] Figure 3 is a visualisation of an example computed embedding space.

[0021] Figure 4 is a schematic diagram showing an example of a system that is useable to create batch embeddings. Figure 5 is a schematic diagram showing an example of a system that is useable to create real-time embeddings.

[0022] Figure 6 is a schematic diagram showing an example of a system comprising the example system shown in Figure 5.

[0023] Figure 7 is a schematic diagram showing another example of a system having an encoder-decoder architecture.

[0024] Figure 8 is a schematic diagram showing another example of a system having an encoder-decoder architecture.

[0025] Figure 9 is a schematic diagram showing another example of a loss function.

[0026] Figure 10 is a schematic diagram showing another example of a system having an encoder-decoder architecture.

[0027] Figure 11 is a schematic diagram showing another example of a system having an encoder-decoder architecture.

[0028] Figure 12 is a schematic diagram showing another example of a system having an encoder-decoder architecture.

[0029] Figure 13 is a diagram showing three graphs of VDR against different FP-ratio thresholds for three different datasets.

[0030] Figure 14 is a visualisation of another example computed embedding space.

[0031] Detailed Description

[0032] Without loss of generality, the present disclosure relates to generative artificial intelligence (Al) systems configured to discover and compute features. In examples, such features comprise behavioral features. A behavioral feature represents behavior of one or more entities.

[0033] Examples described herein use generative autoregression in the context of event data. This presents problems that are not applicable to use of generative autoregression in the context of other types of data. For example, using generative autoregression in the context of event data presents different problems from those associated with use of generative autoregression in the context of large language models (LLMs).

[0034] Examples described herein automate definition and computation of transferrable machine learning (ML) features from input data. Transferrable ML features may be used in multiple downstream ML models. Transferrable ML features therefore differ from non-transferrable ML features which can only be used in one downstream ML model. Different downstream ML models may be associated with different decisioning processes.

[0035] An example of such a downstream decisioning process is decisioning whether or not to approve an incoming request. Accurate predictions about future events improve outcomes of such decisioning processes, i.e., whether or not to process a given incoming request.

[0036] ML models may inform process decisions with an accurate prediction of future outcomes. Such ML models use input features carrying a predictive signal that is pertinent to a given problem. An example of such a predictive signal is a predictive signal that encodes past behavior, decisions, actions and / or activity of an entity such as a person. Such input features are used to operate downstream ML models. Improving the richness of such input features improves outcomes of downstream ML models. Reducing the time to create predictive signals decreases time in successfully developing and deploying ML models.

[0037] In existing systems, features may not be transferrable. Such features may also be referred to herein as being "non-transferrable". For example, such features may be target-encoded features or may be extracted within a neural network trained using supervised learning. If features are computed, whether or not by design, such that they are not transferrable, then the signals cannot be shared between different downstream ML models. This introduces various inefficiencies. Examples of such inefficiencies include, but are not limited to, ongoing computation, development and maintenance inefficiencies.

[0038] In existing systems, features may be defined manually, rather than in an automated manner. Manually defined features are developed by human data scientists using domain knowledge, data analysis and iterative experimentation. Manual definition is time-consuming. This lengthens time to successful development and deployment. Further, human-discovered features are limited in their predictive capacity since they are limited by human ingenuity. This limits the effectiveness and utility of downstream ML models.

[0039] If features are discovered using analysis of a single data source, then both the volume and diversity of data available for analysis is limited. Detectable features may therefore be less granular and, hence, less rich than where data from multiple data sources is used. Further, automated feature discovery may overfit when limited data from a single data source is analysed. For example, automated feature discovery may model specific entities, rather than decomposing into a set of interlinked components, such as behavioral components.

[0040] Examples described herein enable transferrable features to be generated that are automatically defined and tuned by an optimization algorithm and extracted from analysis of data at a large scale.

[0041] Various existing systems will now be described by way of a comparison.

[0042] A typical existing system to generate features from data involves human-defined aggregations over a historic period and pertaining to a particular entity. An example aggregation is an average value of a set of values from a previous six-month period. Another example is a proportion of events that occur overseas. In these examples, the aggregations are performed based on information contained in event data. The definition of features in these systems is manual and is not automated.

[0043] In another existing system, sequential deep learning is used to discover features. For example, recurrent neural networks (RNNs) may be used. In such systems, training typically occurs in a single optimization process. Such optimisation uses a classification or regression head that is adapted to predictions for a specific downstream purpose. Feature extraction in such systems is automated. However, the features are not transferrable.

[0044] In some existing systems, features are designed based on information from a single data source. However, consortium models trained on data collected from multiple data sources may be used.

[0045] Self-supervision has been applied to transactional data. However, this uses contrastive learning to train an embeddings model on a dataset from one data source. The same dataset is used in downstream modelling tasks.

[0046] Autoregression has been used to model certain specific sequence types, for example those found in natural language. An example is an LLM, such as GPT-3 and GPT-4. An embeddings application programming interface (API) may be provided to allow a user to input a sequence and receive an embedding in a non-real time scenario.

[0047] LLMs have recently seen significant potential and adoption. LLMs can leverage large corpora of data and scale to hundreds of billions of parameters. LLMs can be used not only to generate human-level text, but also to produce contextualised embeddings of individual tokens, sentences, and even whole documents that can be fed as input to downstream models. A factor in the rapid adoption of LLMs is the development of self-supervised learning (SSL) methods which have allowed LLMs to learn contextual representations of input tokens using solely structural patterns present in the data. Examples of SSL methods include, but are not limited to, autoregressive and masked language modelling (MLM).

[0048] SSL has already been successfully used with different modalities such as natural language, computer vision, audio, and tabular data. However, SSL has not been adequately adapted to multi-variate timeseries data. One example of multi-variate time-series data is streams of financial transactions. These are sequences of events representing transfer of funds between two entities. Each event can be described by a set of features belonging to both numerical and categorical types. Examples of such features include, but are not limited to, timestamp, amount, card number, merchant name, and merchant category (in the case of card transactions).

[0049] From the perspective of financial institutions, the most important modelling problems in this domain include payment authorization instructions, fraud detection, credit default prediction, customer churn prediction, money laundering detection, scam detection for authorized account holders, synthetic identity detection, and future expenditure modelling. Examples of fraud detection include, but are not limited to, fraudulent transactions and money laundering activity. Most existing techniques to solve these problems are based on supervised learning and rely on timeconsuming feature engineering. Extracted features are typically tailored for specific problems and require deep domain expertise. Existing techniques are, therefore, not amenable to transfer learning. Existing techniques also require redesigning of feature definitions when, for example, new fraud typologies emerge.

[0050] SSL has a potential to remove the above-indicated feature engineering process in favour of contextualised pretrained transaction and / or entity embeddings and to leverage unlabelled data from different domains to increase their generalisability and transferability to different tasks.

[0051] However, efforts in this space have so far been limited.

[0052] Recently, a contrastive learning SSL approach has been designed to generate cardholder embeddings based on the transactional history of the customer. These embeddings have been evaluated on entitylevel classification tasks and shown to perform on par with and, in some cases better than, hand- engineered features. However, autoregressive language modelling techniques have not yet been adapted to the domain of financial transactions, even though the ability to predict future events could be leveraged for predictive tasks such as churn prediction, credit default prediction and / or expenditure prediction.

[0053] Many SSL tasks for sequential data were originally designed for the domain of natural language. Autoregressive language modelling aims to predict the next token based on the previous tokens in a sentence and has been mostly used to pretrain the GPT family of models. In MLM, randomly sampled tokens are masked with a special mask token and the network is tasked with predicting the original token. This method has been successfully adapted to other domains including vision, audio, and tabular data. Next sentence prediction has been used together with MLM and works by feeding the network two sentences and predicting whether they are related. A replaced token detection task is a modification of MLM, where randomly sampled tokens are replaced with candidates generated by a different language model. The task is to predict the identity of the original input token.

[0054] Another class of SSL methods is contrastive learning. Most generally, contrastive learning learns representations that are invariant to data augmentation. Contrastive learning involves generating positive and negative pairs where the positive pairs come from two augmented views of a sample input, while negative pairs come from two different input samples. A contrastive loss function encourages representations to be similar for positive pairs and dissimilar for negative pairs. An existing variation of contrastive learning applicable to autoregressive models seeks to maximize mutual information between hidden state and future events from the same sequence. In the domain of financial transactions, one known system uses contrastive learning to obtain entity embeddings. Such a system uses randomly generated sub-sequences from a transactional history belonging to the same entity as positive pairs and sub-sequences from different entity histories as negative pairs. Noncontrastive methods also exist. Such methods train representations invariant to data augmentations using only positive examples. They avoid representation collapse by using a momentum encoder, by penalizing cross-correlation between positive views, by clustering embeddings with equipartition constraint, and by asymmetrically applied stop-gradient operation. Transactional data has also been used with graph-based methods in which originators and beneficiaries (and, sometimes, transactions) form nodes of a graph. Some early work in this area includes supervised training for fraud detection and generating merchant embeddings. More recently, there has also been progress in inductive representation learning algorithms. A known framework encourages similar representations for nearby nodes and has been used to create embeddings for credit card fraud detection. Link prediction between nodes has been used as an SSL task for anomalous transaction detection and capturing inter-company relationships. However, none of these methods leverages the inherent time-series nature of transactions. The above-described graph-based methods use very different model architectures compared to the previous classes of algorithms, which can all be used to pretrain autoregressive models.

[0055] Examples described herein leverage generative pretraining methods, such as have been used in natural language processing (NLP), but apply associated techniques in the domain of event sequences, such as financial transactional sequences. Examples described herein provide SSL methods and systems for pretraining autoregressive models that can generate event embeddings, such as transaction embeddings. The pretraining method combines two objectives. The first objective is a next event prediction task adapted to handle multi-variate transactional events. The second objective is a weighted-past-events reconstruction task. Use of a next event prediction objective leverages generative modelling and applies such techniques to predictive tasks such as churn prediction, credit default prediction and expenditure prediction. All of these tasks seek to predict a future state of an entity. Solving these tasks involves encoding behavioral features characteristic of previous transactions. The use of a weighted-past-events reconstruction objective further encourages the model to learn longer-term behavioral features and increase predictive performance of the embeddings on downstream tasks.

[0056] Results of evaluating the example techniques described herein on four publicly available datasets of card transactions will be provided. These results show that the generated embeddings can outperform engineered features and other SSL methods on churn prediction, expenditure prediction, and credit default prediction, all of which are of high relevance to the financial industry. Results will also be provided in which example techniques described herein are used to pretrain a model on a private dataset of card transactions from 180 issuing banks and in which the model is used to produce transaction embeddings on three further hold-out datasets from European issuing banks that were excluded from the pretraining consortium. The hold-out issuers operate in a different country from any of the pretraining issuers. The churn embeddings are evaluated on churn prediction, expenditure prediction and fraudulent transaction detection, showing transferability of the model to out-of- domain data and the benefit of pretraining on a large and diversified corpus. Visualisations of the embedding space are provided, which show that the model captures semantic similarity among merchants and learns to encode common behavioral concepts that correlate with known fraud typologies.

[0057] To summarise, examples described herein provide various contributions.

[0058] Firstly, examples provide an SSL method which combines a next event modelling task with a weighted- past-events reconstruction task. Both tasks are adapted to the domain of multi-variate time-series data, such as financial transaction data. Secondly, using evaluations on public datasets, results are provided to demonstrate that example techniques described herein outperform engineered features and existing methods on churn, credit default and expenditure prediction tasks.

[0059] Thirdly, examples pretrain an embedding model on a large, diverse corpus of transactions from multiple card issuers. Results provided herein demonstrate the generalisability and transferability of the embedding model to out-of-domain datasets for modelling churn prediction, expenditure prediction, and fraud detection.

[0060] Fourthly, examples provide visualisations of an embedding space, which demonstrate the ability of the model to capture semantic similarity between merchants and learn behavioral concepts that correlate with known fraud typologies.

[0061] Examples that will now be described enable behavioral feature definitions to be discovered from input data, and to enable those features to be extracted and served to production M L models. Such features may be referred to herein as "embeddings".

[0062] Solely to facilitate an understanding of the present disclosure, specific examples will be described below which relate to transactional data, which generally has a well-defined and standardised schema. However, examples described herein are equally applicable to various other types of data, as will be described in more detail below. As such, any examples and references relating to transactions and transactional data should be understood to be applicable to any other type of event data. For example, event data could be produced by any device operated by a person. Actions performed by the person may have one or more attributes and one or more timestamps. Event data may represent such action(s) and / or timestamp(s). For example, event data may be associated with a telephone call, a session in application software, a change in power consumption in a building corresponding to an electronic device being turned on or off, and so on.

[0063] As explained above, measures described herein may be used in relation to transactional data. Multiple downstream ML models may be attached to various different decisioning processes. Accurate predictions about the future can improve the outcomes of the decision.

[0064] An example of such a downstream decisioning process is decisioning whether to approve a transaction request. In such an example, the downstream ML model may be a fraud prediction model.

[0065] Another example of such a downstream decisioning process is decisioning whether to approve a request for credit line extension. In such an example, the downstream ML model may be a credit risk model.

[0066] Another example of such a downstream decisioning process is decisioning whether to target a customer for intended marketing and increasing retention. In such an example, the downstream ML model may be a churn prediction model. Another example of such a downstream decisioning process is decisioning whether to target advertisements for specific merchant types or locations to a customer. In such an example, the downstream ML model may be a merchant recommendation model.

[0067] In this specific example, transactional data is collected from multiple data sources over a predetermined time period. In this example, the multiple data sources comprise multiple financial institutions. In this example, the predetermined time period is at least one year.

[0068] In this example, the transactional data contains common schema elements related to the standards of inter-bank transactions and / or card transactions.

[0069] For example, for card transactions, the collected transactional data may include the following standard elements:

[0070] # [Schema format : fieldname, data type, # comment] { accountno : "Categorical" , # unique account identifier, typically hashed timestamp : "Datetime" , # datetime of the transaction, to millisecond , merchantid : "Categorical" , # unique merchant identifier sic : "Categorical" , # merchant category code, following ISO 18245 conventions amount : "Numeric", # Transaction amount in USD posentrymode : "Categorical" , # Point of sale type code merchantcountrycode : "Categorical" , # ISO 3166-1 numeric country code for the merchant cardpresent : "Binary" , # Indicator, 1 if card is physically present cardholderpresent : "Categorical" , # Indicator, 1 if cardholder is physically present ecommerce : "Binary" , # Indicator, 1 if the sale is an internet commerce transaction moto : "Binary" , # Indicator, 1 if the sale is machine order or telephone order (keyed) icc : "Binary", # Indicator, 1 if an integrated circuit read failed and the POS fell back to another less secure method like magnetic stripe magfallback : "Binary" , # Indicator, 1 if a magnetic stripe read failed and the POS fell back to manual keying . avsresult : "Categorical" , # Result of Address Verification Service check (eg : ' partial match ' , ' no match ' ) .

[0071] 3ds : "Categorical" , # Result of 3d secure authentication during ecommerce sale cvv : "Categorical" , # Result of check of card verification code during ecommerce sale . }

[0072] The transactional data is collected into a data store on a training server.

[0073] A sequential neural network model is then optimized using stochastic gradient descent. The stochastic gradient descent may use ADAM, for example. The optimization minimises a self-supervised loss or cost function.

[0074] In general, the term "loss' is used herein to refer to a measure of penalty owing to the difference between prediction and reality for a single data point and the term "cost" is used herein to refer to the combined penalty of a group of data points plus any additional penalty contributions, for example from model complexity and / or prior knowledge. As such, the term "cost" as used herein does not imply monetary or financial loss. The term "function" is used herein to mean a logical component of a system. The component may comprise hardware and / or software. For example, the term "loss function" is used herein to mean a logical hardware and / or software component of a system that can determine a loss. Where a function generates data, the function may be referred to as a "generator". For example, a "loss function" may be referred to as a "loss data generator".

[0075] In some architectures, the sequential neural network architecture uses an RNN. Examples of RNN include, but are not limited to, gated recurrent units (GRUs) and long short-term memory (LSTM) networks.

[0076] In some architectures, an attention-mechanism-based architecture may be used. An example of an attention-mechanism-based model is a Transformer.

[0077] In some architectures, an Automated Deep Behavioral Network (ADBN) is used. ADBNs are described in European Patent Application No. 21728524.6 (published as European Publication No. EP3956822) in the name of Featurespace Limited, the entire content of which is hereby incorporated by reference. ADBNs are designed to operate on consumer event sequence data.

[0078] In some architectures, one or more Interaction Modules may be used. Interaction Modules are described in International Patent Application No. PCT / EP2022 / 061405 in the name of Featurespace Limited, the entire content of which is hereby incorporated by reference. Interaction Modules are designed to compute the strength of relationships between payment originators and receivers.

[0079] Any of the above-described architectures can operate on the elements of the raw transaction stream described above.

[0080] However, in some examples, a merchant identifier field is only ever ingested by an Interaction Module. In particular, in some examples, the merchant identifier field is not embedded.

[0081] Such architectures do not operate on human-designed features. As such, the architectures may comprise a pre-processor that performs pre-processing. As part of the pre-processing, numeric features may be normalized. Such normalisation may use the transform sign(x). Zogdxl + 1). As part of the pre-processing, categorical features may be encoded with an embedding layer. Input feature embeddings are different from behavioral embeddings described in more detail below.

[0082] As explained above, a stochastic gradient descent algorithm can operate to minimize a loss or cost function. Example of how the loss or cost function may be defined are provided below.

[0083] A sequence contains a transaction, i, that represents a distinct event at a point in time. In some examples, the context available for predicting at that point in time comprises the information in transaction i and any information aggregated over the sequence of transactions leading up to transaction i. In other examples, the context available for predicting at that point in time comprises the sequence of transactions up to, but not including, transaction i. The latter may be the case if convolutional or attentive architectures are used. Using an RNN or ADBN architecture by way of example only, the aggregated information presents as hidden state passed along a sequence in memory cells of a network.

[0084] In this specific example, the network is tasked with predicting future transactions in the sequence based on the embeddings of transaction i. The future transactions in the sequence are the transactions i + 1, i + 2, and so on. More specifically, in this example, the network is tasked with predicting the values of features in the following k transactions, i.e. the k transactions that follow transaction i. However, it is emphasised again that this is solely by way of a specific example. Other examples are described below in which one or more past transaction are reconstructed and / or in which only one future transaction is predicted.

[0085] This task may be recognised as a form of self-supervision; more specifically generative autoregression.

[0086] In this example, the neural architecture is factored into an encoder network and a decoder network. This provides an encoder-decoder architecture. An encoder encodes data to provide encoded data. A decoder decodes encoded data to provide decoded data.

[0087] In this example, the encoder network is or comprises the above-described RNN or ADBN.

[0088] In this example, the output of the encoder network is a set of behavioral embeddings. The behavioral embeddings may have dimensionality of order 100s, for example.

[0089] In this example, the decoder network is a stateless neural architecture. An example of a stateless neural architecture is a multilayer perceptron (MLP).

[0090] In this example, the input of the decoder network comprises the behavioral embeddings for transaction i, as output by the encoder network.

[0091] In this example, the output of the decoder network comprises predicted event field values for the following k transactions, namely transactions i + 1, i + 2, ... , i + k.

[0092] A reconstruction loss is determined for each of the following k transactions. In this example, for each of the following k transactions, the reconstruction loss is an aggregation of distance metrics computed between actual values of the event fields and the predicted values. The actual values of the event fields are observed values of the event fields. For example, actual transactional data for the actual following k transactions may be obtained and the actual values of the event fields for those actual following k transactions may be used.

[0093] In this example, the aggregation comprises a weighted sum. In this example, the distance metric is categorical cross-entropy (CE) for categorical fields. In this example, the distance metric is mean squared error (MSE) for numerical fields.

[0094] In this example, to compute the total reconstruction loss (also referred to herein as the total reconstruction cost), the reconstruction losses from each of the following k transactions, i, i + 1, i + 2, ... , i + k, are summed together. In this example, the reconstruction losses are weighted by a factor computed as for j + 1 < j < i + k, where t is the transaction timestamp and T is a time-constant hyperparameter. This provides time-decay weighting.

[0095] This weighting up-weights loss contributions from transactions closest in time to the current transaction, i, and down-weights contributions from transactions in the far future.

[0096] Once training has converged, the decoder network is discarded. This is because the decoder network has served its purpose in enabling weights of the encoder network to be determined. In this example, the decoder network is not used in a serving architecture. Discarding the decoder network frees up storage space for other data and reduces complexity. Discarding the decoder network may also improve security by minimising information exposed within the system.

[0097] The weights of the encoder network are returned as the output of the training method.

[0098] The encoder network may also be referred to herein an "embeddings network" since the encoder network generates embeddings.

[0099] Referring to Figure 1, there is shown an example system 100 having an encoder-decoder architecture. In this example, the system 100 has a training architecture arrangement. The system 100 is a generative autoregression arrangement used to train an embeddings model.

[0100] The system 100 comprises an encoder network 102 and a decoder network 104. The encoder network 102 comprises an encoder. The decoder network 104 comprises a set of decoders.

[0101] Data 106 representing a sequence of transactions, tn, tn+1, tn+2, tn+3, including a computed most- recent transaction, tn+3, is input into a set of pre-processors 108. In this example, the number of preprocessors 108 is the same as the number of transactions in the sequence of transactions, tn, tn+i> tn+2> tn+3> namely four. The output of the pre-processors 108 is a set of embeddings 110. In this example, the number of embeddings 110 is the same as the number of transactions in the sequence of transactions, tn, tn+1, tn+2, tn+3, namely four.

[0102] The embedding, [••• ]n+3, for a computed most-recent transaction, tn+3, is fed into the set of decoders in the decoder network 104 to predict the subsequent k transactions. In this example, the subsequent k transactions are subsequent to the computed most-recent transaction, tn+3, but have already taken place as real-world transactions. As will become more apparent from the description below, the predictions for the subsequent k transactions are compared to the actual subsequent k transactions.

[0103] Reconstruction losses 112 from each of the transactions are scaled 114 by a weighting factor that is a function of the time elapsed since the timestamp of the most recent transaction tn+3. The reconstruction losses, lrec, are calculated 112 based on differences between predicted features of the subsequent k transactions and actual features of the subsequent k transactions.

[0104] The system 100 outputs a total loss value 116, L, based on the scaled reconstruction losses. The total loss value, L, is used to train the encoder network 102 and the decoder network 104. Referring to Figure 2, there is shown an example loss function 200. The example loss function 200 may correspond to the loss function 112 shown in Figure 1.

[0105] Figure 2 shows total loss as a weighted sum of reconstruction losses from predicting k future events. The reconstruction loss is a sum of distances between actual and predicted field values, where the actual values are on the left-hand side and the predicted values are on the right-hand side. The sum of distances may be weighted. In this example, the distance metric is CE for categorical fields. A binary field is an example of a categorical field, which has only two categories. In this example, the distance metric is MSE for numerical fields.

[0106] Referring to Figure 3, there is shown an example visualization 300 of computed embeddings in a computed embedding space.

[0107] Embeddings may have a large number of dimensions. For example, embeddings may have 150 dimensions. Enabling effective visualization of embeddings may increase the utility of the systems and methods described herein.

[0108] In this specific example, the three largest principal components are shown. In this specific example, those components account for 20% of the variance in the 150-dimension embeddings space.

[0109] Datapoints are shown in the same manner when they relate to the same account. This indicates how accounts can display consistent behavioral signatures. However, some accounts see migration through the embedding space as behavior evolves.

[0110] Embeddings may be visualised in various different ways in accordance with examples described herein.

[0111] In some examples, an output of a function of a set of embeddings is visualised. For example, the function may average the set of embeddings. Average embeddings associated with certain entities should be similar, and average embeddings associated with certain other entities should be different. For example, average embeddings of multiple different supermarkets should be similar, while an average embedding of a supermarket should be different from an average embedding of an online gambling company.

[0112] A merchant embedding may be a function, such as an average, of a set of embeddings associated with a merchant. A merchant embedding may be used in a downstream task.

[0113] An embedding may be used as part of a recommendation system. For example, the recommendation system may use embeddings to determine how close a set of users is to an embedding of a recommendable item, such as a media item or product.

[0114] Reference embeddings may also be visualised. For example, a reference composite embeddings vector that is aligned with fraud may be visualised. In another example, a reference composite embeddings vector having a high alignment with moving to a different city may be visualised. Examples will now be described in which a set of trained network weights is used to serve outputs from an embedding network as a response to a request to embed one or more transactions. In this example, the set of trained network weights is generated using the above-described training system.

[0115] An embeddings service may be hosted in the cloud. This provides a network endpoint for authenticated users to communicate with the API. Messages sent and received over the API may be encrypted.

[0116] In some examples, an output of a function of a set of embeddings is returned, instead of the full set of embeddings. For example, an average of an age attribute from a set of embeddings may be provided.

[0117] In some examples, the full set of embeddings is returned and an output of a function of a set of embeddings is determined locally.

[0118] Examples described herein may be used for age prediction, where behavioral evolution may not be relevant.

[0119] Two different example request modes, namely batch API and real-time API, will now be described with reference to Figures 4 and 5 respectively.

[0120] Referring to Figure 4, there is shown an example of a batch API 400, which may be used in a batch API request mode.

[0121] In this example, a transaction sequence 402 pertaining to a single account is provided. In this specific example, the transaction sequence 402 is provided as a JavaScript Object Notation (JSON) file. However, JSON is solely an example file format. A different file format may be used in other examples. Embeddings are extracted 404 and returned 406 for each transaction in the transaction sequence 402. This provides a point-in-time snapshot for each transaction based on the context of all the preceding transactions.

[0122] Alternatively, a set of transaction sequences may be provided, for example in a JSON file. In this example, another layer of nesting is provided. For example, the other layer of nesting may be keyed on a hashed account identifier.

[0123] The batch API may be used to provide high-throughput embeddings in scenarios where latency is not a significant constraint. A specific example of this is churn prediction.

[0124] The batch API may be used to generate training datasets to train downstream ML models. The embeddings are the features for those downstream ML models.

[0125] By way of a summary, with reference to Figure 4, a file comprising multiple transaction histories 402 is provided to a cloud embeddings service 404. Embeddings for every transaction in the file are returned 406 in a new file. In this example, no data is persisted within the cloud service 404 between API calls. Referring to Figure 5, there is shown an example of a real-time API 500, which may be used in a realtime API request mode.

[0126] In this example, a single transaction 502 pertaining to specific account is provided to a cloud embeddings service 504, for example as a JSON object. The single transaction 502 may be a current transaction.

[0127] Upon receiving the transaction, hidden state is obtained from a key-value store 506. The key-value store 506 may, for example, comprise a not only SQL (NoSQL) database, where SQL stands for Structured Query Language. The hidden state completes the inputs to an RNN-based network to enable embeddings to be created and returned 508 for the single transaction.

[0128] In an example real-time scenario, an ADBN architecture is used. In such an example, state updates are commutative. This enables fragmented entity processing and state updates. In other words, this enables writes of updated state to the key-value store. In fragmented entity processing, state deltas are applied to state held in the database. This is instead of state being overwritten. This enables significant increases in system throughput via parallelization. Fragmented entity processing is described in International Patent Application No. PCT / IB2022 / 057080 in the name of Featurespace Limited, the entire content of which is hereby incorporated by reference.

[0129] By way of a summary, with reference to Figure 5, a single transaction 502, representing the most recent transaction of a history of transactions, is sent to the cloud API service 504. The embedding service 504 reads state for the appropriate transaction history from a key-value store 506 database, and computes embeddings and additive state update. The embeddings for the single transaction are returned 508, and the state update is applied to the data in the key-value store 506.

[0130] Referring to Figure 6, there is shown an example of how a real-time API may be used in a real-time fraud detection system 600. The real-time API may correspond to the real-time API 500 described above with reference to Figure 5.

[0131] Figure 6 shows an example system arrangement 600 for real-time prediction for a downstream task, namely fraud detetcion in this exmaple.

[0132] The real-time system 602 in this example is indicated by a dashed box. In this example, the real-time system 602 receives raw transaction fields 604 relating to a single transaction. The real-time system 602 generates embeddings for the single transaction. In this example, the real-time system 602 comprises an embedding evaulator 606 and a database 608.

[0133] The embeddings generated by the embedding evaluator 606 are provided to a downstream classifier 610. In this specific example, the downstream classifier 610 comprises an MLP. However, the downstream classifier 610 may not comprise a neural network. For example, the downstream classifier 610 may use XGBoost or a linear regression model. In this example, the downstream classifier 610 is trained on different data from the data on which encoder networks described herein are trained. In this example, other features are also provided to the downstream classifier 610. In this example, the other features are non-behavioral.

[0134] In this example, some features are provided to the downstream classifier 610 directly 612.

[0135] In this example, some features are target-encoded by a target encoder 614 before being provided to the downstream classifier 610. A specific example of such another features is a target encoding for a categorical field, such as merchant identifier.

[0136] In this specific example, the MLP in the downstream classifier 610 is trained after the embeddings evaluator 606, with labels specific to a fraud prediction task.

[0137] Referring to Figure 7, there is shown an example system 700 having an encoder-decoder architecture.

[0138] In this example, event data 702 is obtained. The event data represents, xt, a given event that occurred at a given time, t. The term "obtained" as used herein encompasses receiving, retrieving, generating, and any other type of obtaining.

[0139] The event data is provided as input to an encoder 704.

[0140] The encoder 704 comprises a pre-processor 706. The pre-processor 706 pre-processes the event data 702 and outputs pre-processed event data.

[0141] The pre-processed event data is provided to a first MLP 708. The output of the first MLP 708 is provided as input to a GRU 710. The output of the GRU 710 is provided as input to a second MLP 712. The output of the second MLP 712 comprises an encoded representation 714, et, of the given event that occurred at a given time, t. In this example, the encoded representation 714, et, comprises an embeddings vector.

[0142] In this example, the encoded representation 714, et, is used to perform next event prediction. Next event prediction predicts the next future event. Multiple future events can be predicted in other examples, as described above. In some examples, however, no future events are predicted.

[0143] To perform next event prediction, the encoded representation 714, et, is provided as input to a decoder 716. In this example, the decoder 716 is a next event decoder. A next event decoder is a type of future event decoder.

[0144] The next event decoder 716 outputs a predicted representation 718, xt+1, of the next event, associated with time, t + 1.

[0145] An actual representation 720, xt+1, of the next event is obtained. The predicted representation 718, xt+1, of the next event and the actual representation 720, xt+1, of the next event are provided as inputs to a next event prediction loss function. The output of the next event prediction loss function is a next event prediction loss 722, £fP.

[0146] In this example, the encoded representation 714, et, is used to perform past event reconstruction. Past event reconstruction reconstructs one or more previous events. In this example, multiple previous events are reconstructed. However, in other examples, zero or one previous events may be reconstructed. In this specific example, there are three previous events.

[0147] To perform previous event reconstruction, the encoded representation 714, et, is provided as input to a decoder 724 associated with the earliest of the past three events. In this example, the decoder 724 is a previous event decoder.

[0148] In addition, time difference data 726 indicative of a time difference, 6tt t-3, between the time, t, of the current event and the time, t — 3, of the earliest of the three past events is also provided as input to the previous event decoder 724.

[0149] The previous event decoder 724 outputs a reconstructed representation 728, xt-3, of the earliest of the three past events, which occurred at time, t — 3.

[0150] The time difference data 726 is also used to generate a weighting 730, wt t-3, associated with the earliest of the three past events.

[0151] The reconstructed representation 728, xt-3, of the earliest of the three past events and an actual representation 732, xt-3, of the earliest of the three past events are provided as inputs to a reconstruction loss function 734, lrec.

[0152] The weighting 730, wt t-3, associated with the earliest of the three past events is applied 736 to the output of the reconstruction loss function 734, lrec, to generate a weighted reconstruction loss associated with the earliest of the three past events. In this example, the weighting 730, wt t-3, associated with the earliest of the three past events is applied 736 to the output of the reconstruction loss function 734, lrec, by multiplying the weighting 730, wt t-3, and the output of the reconstruction loss function 734, lrec, together.

[0153] Past event reconstruction process is performed in a similar manner for other past events which, in this example, includes events at previous times t — 2 and t — 1.

[0154] The weighted reconstruction losses associated with the previous events are aggregated together 738. In this example, the weighted reconstruction losses are aggregated 738 by being added together.

[0155] The result of the aggregation is provided as an input to a previous event cost function and the output of the previous event loss function is a total previous event cost 740, £PR. Although, in this example, the previous event cost function is provided with the result of the aggregation, the previous event cost function may be provided with other input in other examples. For example, the previous event cost function may be provided with the set of weighted reconstruction losses associated with the previous events and may aggregate them together itself.

[0156] The previous event cost 740, £PR, and the next event prediction loss 722, £^p, may be aggregated, for example by being added together.

[0157] By way of an explanation of processing performed in the example system 700, suppose = {he} is a set of transactional histories pertaining to one or more entities. Examples of such entities include, but are not limited to, cardholders and account holders.

[0158] A transactional history is a sequence of financial transactions, he= {xt}t>0, each of which can be described by a set of numerical and categorical features, such as amount, merchant name, etc..

[0159] The aim is to train an encoder network 704, E X -> IRd, that creates an embedding vector, ete ]Rd, of a transaction, xt, given past transactions from the same entity up to the current transaction, i.e. et= E xt, xt-, ... , XQ). TO extend this setup to any multi-variate time series data, transactional histories are referred to as "sequences" and individual transactions are referred to as "events".

[0160] As explained above, an example system 700 in which SSL can be applied is shown in Figure 7 and implements two main tasks.

[0161] Next event prediction (NP) is, in this example, the primary task and adapts autoregressive language modelling to the case of multi-variate events. A decoder network 716, -> X, takes an embedding 714, et, of an event, xt, to generate predictions of the next event features 718, xt+1= DNPet~). For numerical features, the predictions are real numbers. For categorical features, the predictions are vectors of probabilities over the distinct categories of a feature.

[0162] An objective function 722 is defined as: £fP= where the summation is f over event features and the reconstruction loss functions, lrec, are either MSE for numerical features or CE for categorical features.

[0163] Past events reconstruction (PR) is another task that aims to guide the encoder 704 towards learning behavioral features with long-term dependencies. A decoder network 724, DPR: -> X, is defined. The decoder network 724 takes, as input, the embedding 714, et, of the event, xt, and a scalar time difference, between the current event and an event from the past, xt-k, and generates a reconstruction 728 of that past event, xt-fc=

[0164] An objective function 740 is a weighted sum of reconstruction losses of past events: £PR= weight function that decays exponentially with the time difference between events at a rate governed by the hyperparameter A. In this example, the summation over the past events is clipped so that at most K past events contribute to the loss. For a sequence of events, he, the total cost is a sum of event losses, which are weighted contributions of the two objective functions 722, 740 defined above, namely, L where a is a hyperparameter.

[0165] Although any autoregressive model architecture can be used as an encoder, examples described herein use a recurrent model based on a GRU 710. Compared to unidirectional transformer models, RNNs are more efficient in production since they only process hidden state and a new event rather than a whole sequence of previous events. This is particularly effective in the financial sector when serving a model in real-time, for example as a fraud detector.

[0166] In the encoder network 704 shown in Figure 7, an event, xt, is first pre-processed 706 into a dense vector. Numerical features are normalized. Categorical features are encoded through an embedding layer. Vectors representing each feature are then concatenated to form a single dense representation of an event. This vector is then passed through a stack of layers 708, 710, 712, (pMLP° <PGRU0(PMLP’ that enriches the vector with the representation of past events. The use of MLPs 708, 712 before and after the GRU layer 710 adds expressivity to the encoder 704 and allows the size of the hidden state to scale independently of the final embedding size.

[0167] In this example, decoder models 716, 724 in the two modelling heads are simple MLPs. Output of the last dense layer in each decoder 716, 724 is split into multiple vectors; one for each feature in the encoded events. Feature vectors corresponding to numerical features have size one (scalar). Feature vectors corresponding to categorical features have size equal to the number of distinct values of that feature. Categorical feature vectors are additionally passed through a softmax activation.

[0168] Referring to Figure 8, there is shown an example system 800 having an encoder-decoder architecture.

[0169] In this example, event data representing four different events is shown. In this example, the third event is the currently embedded event.

[0170] Third event data 802 representing the third event includes a card ID field having a value of "230098", a timestamp field having a value of "2018-09-11 17:38", an amount field having a value of "£102.20" and an mcc field having a value of "Hardware".

[0171] The third event data 802 is pre-processed to generate pre-processed third event data 804. In this example, the pre-processed third event data 804 includes the "£102.20" value from the amount field of the third event data 802 and includes a one-hot encoded version of the "Hardware" value of the mcc field of the third event data 802, namely "0,0, 1,0".

[0172] One-hot encoding results in a sequence of bits in which only one of the bits is allowed to have the value "1" and all other bits have to have the value "0".

[0173] In this example, the pre-processed third event data 804 is in the form [102.20,0,0,1,0],

[0174] The pre-processed third event data 804 is provided as input to an encoder 806. In this example, the output of the encoder 806 is an embedding vector 808, e3, having values [0.1,2.4,10.5,3.8,4.0],

[0175] The embedding vector 808, e3, is provided as input to a next event decoder 810.

[0176] The output of the next event decoder 810 is a set of predicted event features 812 for the next event, namely the fourth event. In this example, the set of predicted event features 812 for the fourth event is [39.81,0.1,0.7,0.1,0.1], Thus, the predicted amount for the fourth transaction is "£39.81" and the mcc probabilities are "0.1, 0.7, 0.1, 0.1".

[0177] Fourth event data 814 representing the fourth event includes a card ID field having a value of "23 00 98", a timestamp field having a value of "2018-09-11 21:34", an amount field having a value of "£32.00" and an mcc field having a value of "Clothing".

[0178] The fourth event data 814 is pre-processed to generate pre-processed fourth event data 816. In this example, the pre-processed fourth event data 816 includes the "£32.00" value from the amount field of the fourth event data 814 and includes a one-hot encoded version of the "Clothing" value of the mcc field of the fourth event data 814, namely "0, 1,0,0". In this example, the pre-processed fourth event data 816 is in the form [32.00,0,1,0,0],

[0179] The set of predicted event features 812 for the fourth event and the pre-processed fourth event data 816 are provided as inputs to a reconstruction loss function 818, Lrec. In this example, an output of the reconstruction loss function 818, Lrec, is in the form [17.1], The value "17.1" is the next event prediction loss value.

[0180] The embedding vector 808, e3, is also provided as input to a past event reconstruction decoder 820 associated with the first event.

[0181] Time difference data 822 indicating the time difference, 8t3 1, is also provided as input to the past event reconstruction decoder 820 associated with the first event. The time difference data 822 has a value of 23 hours, which represents a time difference between the timestamp (2018-09-11 17:38) of the third event and the timestamp (2018-09-10 17:45) of the first event. In this example, the time difference is rounded down to the nearest hour. However, in other examples, no rounding is performed.

[0182] The output of the past event reconstruction decoder 820 associated with the first event is a set of predicted event features 824 for the first event. In this example, the set of predicted event features 824 for the first event is [9.21,0.1,0.1,0.1,0.7], Thus, the predicted amount for the fourth transaction is "£9.21" and the mcc probabilities are "0.1, 0.1, 0.1, 0.7".

[0183] First event data 826 representing the first event includes a card ID field having a value of "23 0098", a timestamp field having a value of "2018-09-10 17:45", an amount field having a value of "£10.98" and an mcc field having a value of "Groceries". The first event data 826 is pre-processed to generate pre-processed first event data 828. In this example, the pre-processed first event data 828 includes the "£10.98" value from the amount field of the first event data 826 and includes a one-hot encoded version of the "Groceries" value of the mcc field of the first event data 826, namely "0,0, 0,1". In this example, the pre-processed first event data 826 is in the form [10.98,0,0,0,1],

[0184] The set of predicted event features 824 for the first event and the pre-processed first event data 828 are provided as inputs to a reconstruction loss function 830,rec. In this example, an output of the reconstruction loss function 830, £rec, is in the form [12.3],

[0185] The time difference data 822 indicating the time difference, 6t3 1, is also provided as input to a weight function 832. In this example, the output of the weight function 832 is a weighting in the form [0.07],

[0186] The weighting, having a value of "0.07", is applied to the value of the output of the reconstruction loss function 830, £rec, namely the value "12.3".

[0187] In this example, the weighting is applied to the reconstruction loss value by multiplying the weighting and the reconstruction loss value together to give a weighted reconstruction loss value of 12.3 x 0.07 = 0.9.

[0188] In this example, the weighted reconstruction loss value is added to weighted reconstruction loss value of all previous events. However, since, in this example, the first event is the earliest event in the event sequence, an initial past reconstruction value of "0.0" is added to the weighted reconstruction loss value of "0.9" to give a combined weighted reconstruction cost value of "0.9".

[0189] The embedding vector 808, e3, is also provided as input to a past event reconstruction decoder 834 associated with the second event.

[0190] Time difference data 836 indicating the time difference, 6t3 2, is also provided as input to the past event reconstruction decoder 834 associated with the second event. The difference data 836 has a value of 2 hours, which represents the time difference between the timestamp (2018-09-11 17:38) of the third event and the timestamp (2018-09-11 14:58) of the second event.

[0191] The output of the past event reconstruction decoder 834 associated with the second event is a set of predicted event features 838 for the second event. In this example, the set of predicted event features 838 for the second event is [30.12,0.6,0.1,0.1,0.2], Thus, the predicted amount for the fourth transaction is "£30.12" and the mcc probabilities are "0.6, 0.1, 0.1, 0.2".

[0192] Second event data 840 representing the second event includes a card ID field having a value of "23 00 98", a timestamp field having a value of "2018-09-11 14:58", an amount field having a value of "£24.70" and an mcc field having a value of "Cafe".

[0193] The second event data 840 is pre-processed to generate pre-processed second event data 842. In this example, the pre-processed second event data 842 includes the "£24.70" value from the amount field of the second event data 840 and includes a one-hot encoded version of the "Cafe" value of the mcc field of the second event data 840, namely "1,0, 0,0". In this example, the pre-processed second event data 842 is in the form [24.70,1,0,0,0],

[0194] The set of predicted event features 834 for the second event and the pre-processed second event data 842 are provided as inputs to a reconstruction loss function 844,rec. In this example, an output of the reconstruction loss function 844, Lrec, is in the form [17.1],

[0195] The time difference data 836 indicating the time difference, 6t3 2, is also provided as input to a weight function 846. In this example, the output of the weight function 846 is a weighting in the form [0.96],

[0196] The weighting, having a value of "0.96", is applied to the value of the output of the reconstruction loss function, Lrec, namely the value "17.1".

[0197] In this example, the weighting is applied to the reconstruction loss value by multiplying the weighting and the reconstruction loss value together to give a weighted reconstruction loss value of 17.1 x 0.96 = 16.4.

[0198] In this example, the weighted reconstruction loss value is added to the weighted reconstruction loss values of all previous events. In this example, the weighted reconstruction loss value of "0.9" from the first event is added to the weighted reconstruction loss value of "16.4" from the second event, giving a combined weighted reconstruction cost value of 0.9 + 16.4 = 17.3. The combined weighted reconstruction cost value may be considered to be a final past reconstruction cost value.

[0199] Referring to Figure 9, there is shown an example of how a reconstruction loss can be determined using a reconstruction loss function 902.

[0200] In this example, actual event features 904 associated with a transaction may be in the form [24.70,1,0,0,0] where the actual amount of the transaction is "£24.60" and where the one-hot encoded mcc vector, based on the actual mcc of the transaction, is "1,0, 0,0".

[0201] In this example, predicted event features 906 associated with the transaction may be in the form [25.12,0.6,0.1,0.1,0.2] where the predicted amount of the transaction is "£25.12" and where the vector of mcc probabilities is "0.6, 0.1, 0.1, 0.2".

[0202] In this example, the amount event features, which are numerical features (scalars), are subject to squared error processing in the form I = (x — x)2, where I is the squared error loss, x is the actual amount event feature value, and x is the predicted amount event feature value. In this example, the squared error loss has a value of "0.18".

[0203] In this example, the mcc event features, which are categorical features (vectors), are subject to CE processing in the form Z = — 2; P(xi) log p(.x> where I is the CE loss, p(Xj) is the jthactual one-hot encoded mcc vector value, and p(xj) is the jthmcc probability vector value. In this example, the CE loss has a value of "0.51". In this example, the squared error loss and the CE loss are added together to give a reconstruction loss, £rec, of 0.18 + 0.51 = 0.69.

[0204] Referring to Figure 10, there is shown an example system 1000.

[0205] Various methods and techniques described herein may be implemented in the example system 1000. Such methods and techniques may involve controlling the example system 1000. Controlling the example system 1000 may comprise obtaining input data, providing input data to one or more elements of the system, obtaining output data from one or more elements of the system, providing output data from one or more elements of the system to one or more other elements of the system, configuring one or more elements of the system, and so on. However, methods and techniques described herein may be implemented in a different system in other examples.

[0206] The example system 1000 has an encoder-decoder architecture. The term "encoder-decoder architecture" is used herein to mean an architecture that includes both an encoder and a decoder. In other words, the example system has both encode and decode functionality.

[0207] In this example, the encoder-decoder architecture includes multiple encoder-decoder subsystems. More specifically, in this example, the encoder-decoder architecture includes a first encoder-decoder subsystem 1002 and a second encoder-decoder subsystem 1004.

[0208] In this example, the first encoder-decoder subsystem 1002 comprises an encoder network 1006. The encoder network 1006 comprises at least one encoder 1008. In this example, the first encoderdecoder subsystem 1002 comprises a decoder network 1010. The decoder network 1010 comprises at least one decoder 1012.

[0209] In this example, the second encoder-decoder subsystem 1004 comprises an encoder network 1014. The encoder network 1014 comprises at least one encoder 1016. In this example, the second encoderdecoder subsystem 1004 comprises a decoder network 1018. The decoder network 1018 comprises at least one decoder 1020.

[0210] In this example, the first encoder-decoder subsystem 1002 is a training encoder-decoder subsystem 1002. The training encoder-decoder subsystem 1002 is used to train one or more elements of the example system 1000.

[0211] In this example, the second encoder-decoder subsystem 1004 is a predictive encoder-decoder subsystem 1004. The predictive encoder-decoder subsystem 1004 is used to generate predicted data 1022.

[0212] Although, in this example, the predictive encoder-decoder subsystem 1004 comprises both the encoder network 1014 and the decoder network 1018, the encoder network 1014 may not generate predicted data as such, and might may generate data that enables the decoder network 1018 to generate the predicted data 1012. Input data 1024 is provided as input to the encoder network 1006 in the training encoder-decoder subsystem 1002. As explained above, the input data 1024 may take various different forms. For example, the input data 1024 may comprise event data. A specific example of event data is transactional data.

[0213] The encoder network 1006 in the training encoder-decoder subsystem 1002 encodes the input data 1024 and generates (which may also be referred to herein as "outputs", "provides" or the like) encoded data 1026. As explained above, the encoded data 1026 may take various different forms. For example, the encoded data 1026 may comprise one or more embeddings vectors.

[0214] The encoded data 1026 is provided as input to the decoder network 1010 in the training encoderdecoder subsystem 1002.

[0215] The decoder network 1010 in the training encoder-decoder subsystem 1002 decodes the encoded data 1026 and generates decoded data 1028. As explained above, the decoded data 1028 may take various different forms. The term "generates" as used here encompasses both predicting and reconstructing. In particular, the decoded data 1028 may comprise predicted future data 1028 and / or reconstructed past data 1028.

[0216] In this example, the decoded data 1028 and reference data 1030 are provided as input to a loss function 1032. As explained above, the reference data 1030 may take various different forms. For example, the reference data 1030 may comprise actual event data.

[0217] The loss function 1032 generates loss data.

[0218] In this example, the loss data is used to train the encoder network 1006 in the training encoderdecoder subsystem 1002. In this example, the loss data is used to train the decoder network 1010 in the training encoder-decoder subsystem 1002.

[0219] Encoder configuration data 1032 is generated. In this example, the configuration data 1032 comprises one or more neural network weights.

[0220] In this example, the encoder network 1014 in the predictive encoder-decoder subsystem 1004 is configured using the encoder configuration data 1032. For example, the encoder network 1014 in the predictive encoder-decoder subsystem 1004 may be configured to use one or more neural network weights comprised in the encoder configuration data 1032.

[0221] Input data 1034 is provided as input to the encoder network 1014 in the predictive encoder-decoder subsystem 1004. The input data 1034 may take various different forms. Some or all of the input data 1034 may be different from some or all of the input data 1024. However, some or all of the input data 1024 may be the same as some or all of the input data 1024.

[0222] The encoder network 1014 in the training encoder-decoder subsystem 1004 encodes the input data 1034 and generates encoded data 1036. The encoded data 1036 may take various different forms. Some or all of the encoded data 1036 may be different from some or all of the encoded data 1026. However, some or all of the encoded data 1036 may be the same as some or all of the encoded data 1026.

[0223] The encoded data 1036 is provided as input to the decoder network 1018 in the predictive encoderdecoder subsystem 1004.

[0224] The decoder network 1018 in the predictive encoder-decoder subsystem 1004 decodes the encoded data 1036 and generates decoded data in the form of the predicted data 1022. The predicted data 1022 may take various different forms. Some or all of the predicted data 1022 may be different from some or all of the decoded data 1028. However, some or all of the predicted data 1022 may be the same as some or all of the decoded data 1028.

[0225] Solely to facilitate understanding, the dash-dot line 1038 depicts a logical divide between training elements of the example system 1000 (above the dash-dot line 1038) and predictive elements of the example system 1000 (below the dash-dot line 1038). Although the encoder configuration data 1032 is shown below the dash-dot line 1038 in Figure 1, the encoder configuration data 1032 is generated in the training portion of the example system 1000 and is used in the predictive portion of the example system 1000.

[0226] Referring to Figure 11, there is shown another example system 1100.

[0227] The example system 1100 includes elements that are the same as or otherwise correspond to elements of the example system 1000 described above with reference to Figure 10. Such features are indicated in Figure 11 using the same reference signs as in Figure 10 but incremented by 100.

[0228] Input data 1124 is obtained. The input data 1124 may be obtained in various ways. For example, the input data 1124 may be received, generated, retrieved, or otherwise.

[0229] The input data 1124 comprises event data 1124. The event data 1124 represents, xt, a given event that occurred at a given time, t.

[0230] The event data 1124 is provided as input to the encoder network 1108. The encoder network 1108 generate encoded data 1126. The encoded data comprises encoded event data 1126. The encoded event data 1126 comprises an encoded representation, et, of the given event that occurred at the given time, t. The encoded representation, et, of the given event is different from the representation, xt, of the given event in the event data 1124.

[0231] The encoded event data 1126 is provided as input to the decoder network 1110. The decoder network 1110 generates decoded data 1128. The decoded data 1128 comprises decoded event data 1128.

[0232] The decoded event data 1128 represents, xt., one or more further events. Each of the one or more further events occurs at a respective further time, ti.

[0233] A further event may occur in the past relative to the given event such that tL< t. In such an example, the further event may be referred to as a "past event", a "previous event", or the like. A further event may occur in the future relative to the given event such that tt > t. In such an example, the further event may be referred to as a "future event", a "subsequent event", or the like. A future event may occur in the future relative to the given event but may still have occurred in the past relative to a current, real-world time.

[0234] A further event may occur at the same time as the given event such that tL= t. In such an example, the further event may be referred to as a "simultaneous" event or the like. However, some systems may safeguard against simultaneous events. For example, such systems may hold an event for a short amount of time to enable distinct timestamps to be provided. For example, an event may be held for a few milliseconds.

[0235] The decoded event data 1128 and reference data 1130 are provided as inputs to a loss function 1132. The reference data 1130 comprises reference event data 1130. Similar to the decoded event data 1128, the reference event data 1130 represents, xt., the one or more further events. The representation, xt., of the one or more further events in the decoded event data 1128 may be the same as or different from the representation, xt., of the one or more further events in the reference event data 1130.

[0236] The loss function 1132 generates loss data 1140. The loss data 1140 represents one or more losses of the decoded event data 1128 relative to the reference event data 1130.

[0237] In some instances, the representation, xt., of a further event in the decoded event data 1128 is the same as the representation, xt., of the further event in the reference event data 1130. In such instances, the loss data 1140 may indicate zero loss.

[0238] In some instances, the representation, xt., of a further event in the decoded event data 1128 is different from the representation, xt., of the further event in the reference event data 1130. In such instances, the loss data 1140 may indicate non-zero loss.

[0239] Encoder configuration data 1132 is generated using the loss data 1140. The encoder configuration data 1132 comprises one or more encoder parameters.

[0240] In this example, the encoder configuration data 1132 is generated by an encoder configuration data generator 1142.

[0241] The loss data 1140 may be based on time difference data. The time difference data is indicative, for each of the one or more further times, t(, of a time difference, between the given time, t, and the respective further time, tj, of the one or more further times. This may be written mathematically

[0242] In some examples, the time difference, 6tt t., is an absolute value of a difference between the given time, t, and the respective further time, tj. This may be written mathematically as Stt t. = |t — t or 8tt t.=\ i ~fl- Such a time difference, 6tt t., may be referred to as an "absolute time difference",

[0243] The time difference data may be provided as further input, in addition to the encoded event data 1126, to the decoder network 1110 to generate the decoded event data 1128. The decoded event data 1128 generated by the decoder network 1110 may, therefore, be time-difference-dependent.

[0244] The time difference data may be provided as input to one or more weight functions. A weight function generates weighting data based on time difference data. The weighting data is indicative of one or more weightings to be applied to the loss data 1140.

[0245] Each of the one or more weightings may be associated with a respective further time, tj, of the one or more further times, ti. Each of the one or more weightings may be dependent on a time difference, 6tt t., between the given time, t, and the respective further time, tj.

[0246] Each of the one or more weightings may decrease monotonically as an absolute time difference, 6tt t., between the given time, t, and the respective further time, t(, increases. As such, time decay may be provided, such that events closest to the given time, t, are weighted more heavily than those further away from the given time, t.

[0247] Generating the encoder configuration data 1132 using the loss data 1140 may comprise (i) providing the loss data 1140 and the weighting data as inputs to one or more weighted loss data generators (not shown) to generate weighted loss data (not shown); and (ii) generating the encoder configuration data 1132 using the weighted loss data.

[0248] The weighted loss data may comprise a plurality of weighted loss data components.

[0249] Each of the weighted loss data components may be associated with a respective further time, tj, of the one or more further times, tj.

[0250] Generating the encoder configuration data 1132 using the weighted loss data may comprise: (i) providing the plurality of weighted loss data components as inputs to one or more weighted loss data component combiners to generate combined weighted cost data; and (ii) generating the encoder configuration data 1132 using the combined weighted cost data.

[0251] Stochastic gradient descent may be used to minimise the combined weighted cost data.

[0252] The decoder network 1110 may comprise a plurality of decoders.

[0253] The one or more further events may comprise a plurality of further events.

[0254] The decoder network 1110 may comprise one or more past event decoders. A past event decoder may be operable to generate decoded past event data. Decoded past event data may represent one or more past events relative to the given event. The decoder network 1110 may comprises one or more future event decoders. A future event decoder may be operable to generate decoded future event data. Decoded future event data may represent one or more future events relative to the given event.

[0255] The encoder network 1108 may comprise a pre-processor. The pre-processor may be operable to generate the encoded event data 1126 by pre-processing the event data 1124 to generate event feature data. The event feature data may represent one or more features of the given event.

[0256] The event feature data may comprise numerical feature data representing one or more numerical features of the given event. Numerical feature data may represent the one or more numerical features using one or more scalar quantities.

[0257] The event feature data may comprise categorical feature data representing one or more categorical features of the given event. Categorical feature data may represent the one or more categorical features using one or more vector quantities.

[0258] The decoded event data may comprise further event feature data. The further event feature data may represent one or more features of the one or more further events.

[0259] The further event feature data may comprise numerical feature data representing one or more numerical features of the one or more further events. Numerical feature data may represent the one or more numerical features using one or more scalar quantities.

[0260] The loss function 1132 may use squared error processing to generate the loss data 1140 when the decoded event data 1126 and the reference event data 1130 comprise numerical feature data.

[0261] The further event feature data may comprise categorical feature data representing one or more categorical features of the one or more further events. Categorical feature data may represent the one or more categorical features using one or more vector quantities.

[0262] The loss function 1132 may use cross entropy processing to generate the loss data 1140 when the decoded event data 1126 and the reference event data 1130 comprise categorical feature data.

[0263] The encoder network 1108 may comprise one or more sequential neural networks.

[0264] The one or more sequential neural networks may comprise one or more recurrent neural networks.

[0265] The one or more recurrent neural networks may comprise one or more commutative state update recurrent neural networks. A commutative state update neural network is especially effective for realtime serving at scale. This, in turn, is especially effective for providing embeddings to downstream models. However, a commutative state update neural network may still be used to train a model.

[0266] The one or more sequential neural networks may comprise one or more attention-mechanism-based neural networks. The encoded event data 1126 may comprise an embedding vector.

[0267] The event data 1124 and the reference event data 1130 may collectively comprise event data from a consortium of event data sources.

[0268] The event data 1124 may be based on sensor data captured by one or more sensors. Various types of sensor may be used. For example, a radio frequency (RF) sensor may capture RF sensor data. A radiofrequency identification (RFID) reader may capture RFID tag data.

[0269] The event data 1124 may be based on one or more messages encoding one or more instructions. The instructions may take various different forms. For example, the instructions may request a recipient of the message(s) to perform one or more actions.

[0270] At least some of the reference event data 1130 may be provided as further input to the encoder network 1108 to generate the encoded event data 1126.

[0271] Past event data may be provided as further input to the encoder network 1108 to generate the encoded event data 1126. The past event data may represent one or more past events relative to the given event.

[0272] The event data 1124 may comprise multi-variate event data. Multi-variate data may also be referred to as "multivariate" data.

[0273] The reference event data 1130 may comprise multi-variate event data.

[0274] The one or more encoder parameters may comprise one or more neural network weights. However, the one or more encoder parameters may comprise one or more other parameters in other examples. For example, for an ADBN encoder architecture, the one or more encoder parameters may comprise time-constants for the exponential time-decay. For attention-mechanism-based encoders, the one or more encoder parameters may comprise window size. Further examples of encoder parameters include, but are not limited to, a number of neural network nodes in each layer and a number of layers in the network.

[0275] The encoder network 1108 may be trained using the encoder configuration data 1132.

[0276] Decoder configuration data may be generated using the loss data 1140.

[0277] The decoder configuration data may comprise one or more decoder parameters.

[0278] The decoder network 1110 may be trained using the decoder configuration data.

[0279] The decoder network 1110 may be discarded after the decoder network 1110 has been trained.

[0280] The one or more decoder parameters may comprise one or more neural network weights. The encoded event data 1126 may comprise an encoded representation of the given event that occurred at the given time, t, and of one or more events that occurred before the given time, t. For example, an embedding can take in state built up over previous events if an RNN encoder is used. An embedding can take in a sequence of previous events if an attention-based encoder, such as a Transformer, is used.

[0281] Referring to Figure 12, there is shown another example system 1200.

[0282] The example system 1200 includes elements that are the same as or otherwise correspond to elements of the example systems 1000, 1100 described above with reference to Figures 10 and 11 respectively. Such features are indicated in Figure 12 using the same reference signs as in Figures 10 and 11 but incremented by 200 and 100 respectively.

[0283] Input data 1234 is provided to an encoder network 1214. The input data 1234 comprises input event data 1234.

[0284] The encoder network 1214 has been configured based on the encoder configuration data 1222 generated, in this example, using the example system 1100 described above with reference to Figure 11.

[0285] Encoded data 1236 is obtained from the encoder network 1214. The encoded data 1236 comprises encoded event data 1236.

[0286] The encoded event data 1236 is based on the input event data 1234.

[0287] The input event data 1234 may comprise event data representing only a single event. In other words, the input event data 1234 may comprise event data representing the single event and not representing any other events.

[0288] The encoded event data 1236 may comprise encoded event data representing only the single event. In other words, the encoded event data 1236 may comprise event data representing the single event and not representing any other events.

[0289] The input event data 1234 may comprise event data representing an event. The input event data 1234 may comprise one or more values of state resulting from processing event data representing one or more past events relative to the event.

[0290] The encoded event data 1236 may comprise encoded event data representing only the event. In other words, the encoded event data 1236 may comprise event data representing the single event and not representing any other events.

[0291] The input event data 1234 may comprise event data representing a plurality of events. The encoded event data 1234 may comprise encoded event data representing the plurality of events. The encoded event data 1236 may be useable by a first decoder 1212-1 to generate first decoded event data 1222-1 and may be usable by a second, different decoder 1212-2 to generate second, different decoded event data 1222-2.

[0292] The encoded event data 1236 may be used to generate a control signal to control operation of an element of a computer network.

[0293] The encoded event data 1236 being based on the input event data 1234 may comprise the encoded event data 1236 being based on an average of at least one feature represented by the input event data 1234.

[0294] Measures (such as methods, systems and computer programs) are also provided that use generative autoregression to generate output event data based on input event data. The input event data comprises time-ordered, event-driven data.

[0295] Measures (such as methods, systems and computer programs) are also provided that use generative autoregression to generate expected event data based on past event data. The expected event data comprises first expected event data representing a first expected event and second expected event data representing a second expected event. Such measures generate a composite loss based on: (i) a difference between the first expected event data and first actual event data; and (ii) a difference between the second expected event data and second actual event data. The first actual event data represents a first actual event. The second actual event data represents a second actual event.

[0296] Measures (such as methods, systems and computer programs) are also provided that use generative autoregression to generate expected transaction data based on past transaction data. The past transaction data represents one or more past transactions and the expected transaction data represents one or more expected transactions.

[0297] Measures (such as methods, systems and computer programs) are also provided that use generative autoregression to generate expected event data based on past event data. The past event data represents one or more past events and the expected event data represents one or more expected events.

[0298] As such, and in contrast to known systems, some examples described herein apply self-supervision to a consortium of transaction datasets to extract features automatically.

[0299] In addition, in contrast to known systems, some examples described herein adapt autoregressive generative modelling to model transactions.

[0300] The application of generative autoregression to transactions described herein differs from known uses of generative autoregression in various ways.

[0301] For example, known generative autoregressive processes train on next-categorical-token prediction. This is, for example, the case with LLMs such as GPT-3 and GPT-4. Some examples described herein differ from such processes in that they predict multiple fields in an event, for example predicting many next tokens or next values. In examples, such prediction uses a composite loss or cost. This type of prediction is especially effective for the types of data described herein.

[0302] In addition, examples described herein reconstruct and / or predict event data for a number of past and / or future events. The loss is weighted by how far in the past or future the events are. This helps to smooth embeddings and improves performance of the embeddings on downstream tasks.

[0303] As explained above, an existing system uses self-supervision in connection with transaction data. However, this uses a different form of self-supervision. Additionally, this system does not use data transferred from different data sources, such as different banks. This existing system uses contrastive learning, in which different sub-sequences of events produce an embedding. Embeddings computed by sub-sequences from the same account are penalized in a loss function for being different. Embeddings from sub-sequences from different accounts are penalized in the loss function for being similar. This does not allow for evolution of embeddings in response to changes in behavior, for example such as that seen during a fraud attack. It also does not encourage accounts that share very similar behavioral state to be similarly embedded. Instead, it encourages characteristic embeddings that are unique to each account. This is less generally effective for downstream ML models.

[0304] As explained above, various specific technologies may be leveraged in connection with the examples described herein. For example, ADBN and Interaction Modules improve the ability of an encoder network ability to model transactions. A Fragmentation Engine may be used is used to manage state updates at high throughput and low latency for real-time serving. Fragmentation is highly effective for real-time serving of embeddings at scale. An ADBN is one of the few RNN architectures to enable fragmented entity processing. This is because state updates of the ADBN are commutative.

[0305] Examples have been described above in which event data comprises transactional data.

[0306] However, as explained above, examples described herein can be used in connection with other types of event data.

[0307] For example, a door-monitoring system may monitor how and when a given door is opened, for example using one or more door sensors. Event data may be generated each time the door is opened. The event data may comprise fields such as "opening direction", "opening time", "open duration", "opening angle", "opening person", and so on. Such event data may be used to predict future dooropening events. Such prediction may be useful for various different purposes. For example, an alert may be triggered if the door is open for an unexpectedly long amount of time at an unexpected time of day.

[0308] Event data may represent irregularly spaced events. Irregularly spaced events are not spaced at regular time intervals and, thus, are different from regularly spaced events which occur at regular time intervals.

[0309] Such event data may also differ from "time stream" data, in which occurrences are at regular intervals. The event data may be considered to be event-driven and the systems described herein may have no control over when the events occur.

[0310] The event data may therefore comprise or be comprised in time-ordered event data.

[0311] As explained above, ML models underpin many current financial systems for use cases such as, but not limited to, fraud detection and churn prediction. Most existing systems are based on supervised learning with hand-engineered features. Large self-supervised generative models have shown significant success in NLP and computer vision. However, to date, they have not been adapted to multi-variate time series of financial transactions. Examples described above provide generative pretraining methods and systems that can be used to obtain contextualised embeddings of financial transactions or other event data. As will be described in more detail below, benchmarks on public datasets demonstrate that such methods and systems outperform state-of-the-art self-supervised methods on a range of downstream tasks. Results of performing large-scale pretraining of an embedding model using a consortium of datasets from 180 issuing banks containing 5.7 billion transactions and applying such to a card fraud detection problem on hold-out datasets are also provided below. The embedding model improves value detection rate at high precision thresholds and transfers well to significantly out-of-domain distributions.

[0312] Results of experiments on public datasets are now provided.

[0313] The above-described methods in accordance with the present disclosure are referred to, in connection with the experimental results below, as NPPR, which stands for next event prediction (NP) and past events reconstruction (PR).

[0314] The performance of entity-level embeddings generated using NPPR are evaluated when used in classification and regression tasks. The evaluations are not focused on fine-tuning the pretrained models on these tasks, but on using embeddings as input feature vectors to downstream models. This evaluation setup can test systems in which an embeddings server and downstream modelling setup are decoupled. Since publicly available transactional datasets are very limited and their schemas are often incompatible, pretraining and evaluation was performed separately on each dataset. Pretraining on a consortium of different datasets and out-of-domain evaluation on hold-out datasets is discussed below using private datasets.

[0315] In terms of datasets, the evaluations use publicly available datasets from various data science competitions comprising debit and credit card financial transactions. These datasets include unlabelled and labelled transactions for four different tasks, namely age group prediction, churn prediction, future expense forecasting, and credit default prediction. Some pertinent statistics of each dataset are show in Table 1 below.

[0316] Table 1: Characteristics of four publicly available datasets of financial transactions.

[0317] Labels for expense forecasting task were created by splitting all transactions in that dataset into a three-month-long training period and a subsequent one-month-long labelling period which was used to create labels as a total amount of spendings by an entity during that month.

[0318] Training and test sets for each dataset were generated by splitting labelled entity histories. 80% of entity histories were in the training set and 20% were in the evaluation set. The pretraining set was constructed by concatenating the unlabelled entity histories and the training set.

[0319] In terms of hyperparameters, the same encoder architecture was on each dataset for pretraining. The encoder architecture used was MLP with 2 hidden layers (having 512 neurons each) and Rectified Linear Unit (ReLU) activation followed by GRU with a hidden state size of 512. This was followed by and a dense projection layer (with sigmoid activation) to the embedding space of size 512. Both the NP and PR decoders were MLPs with 2 hidden layers (512 neurons each) and ReLU activations.

[0320] For training, ADAM was used with a learning rate of le — 3 and early stopping.

[0321] For pretraining, the decay length, A, was chosen to be two months.

[0322] Hyperparameter a controlling the proportion of past events reconstruction task was tuned on each dataset separately, using 5-fold cross validation, giving best performing values: 0.1 (churn), 0.001 (age), 0.005 (expense), and 0.001 (credit default).

[0323] For downstream model training, MLPs with 3 hidden layers (512 neurons each on churn and age prediction, and 1024 neurons each on expense and credit default prediction) and ReLU activations together with dropout and weight decay regularisation were used. Dropout rate, weight decay, and learning rate were tuned using Optuna framework on 5-fold cross validation for each baseline separately.

[0324] In terms of baselines, four baselines were considered to compare against.

[0325] A first baseline is feature engineering (FeatEng). This a hand-crafted feature set. For numerical features, aggregate functions (sum, count, mean, min, max, variance) were applied over all transactions in an entity history. For categorical features, the above aggregates of numerical features were computed within groups of transactions, grouped by every unique value of each categorical feature. A second baseline is SimCSE. This baseline uses sequence embeddings pretrained with contrastive learning, where dropout was used as the data augmentation strategy for generating positive pairs. Dropout value was tuned on each dataset.

[0326] A third baseline is replaced event detection (RED). This involves adaptation of the above-described replaced token detection task, in which randomly sampled transactions (events) were swapped for random events from other sequences in a batch. The decoder was tasked with predicting the sampling mask. 30% sampling probability performed best on downstream tasks.

[0327] A fourth baseline is "CoLES", as described in Babaev, D., Ovsov, N., Kireev, I., Ivanova, M., Gusev, G., Nazarov, I., and Tuzhilin, A. "Coles: Contrastive Learning for Event Sequences with Self-supervision". SIGMOD '22, Association for Computing Machinery. CoLES uses a contrastive learning method in which positive samples were random sub-sequences from the same sequences, and negative samples were sub-sequences from two different sequences. This baseline is sensitive to the choice of minimum and maximum sub-sequence sampling length.

[0328] For each of the above self-supervised methods the embedding of the most recent event was used as the sequence embedding.

[0329] Results show evaluations of the chosen baselines using best-performing parameters found during hyperparameter tuning. They include mean and standard deviation on test sets of five training runs on different folds of the training set.

[0330] In terms of comparisons with baseline methods, Table 2 below compares NPPR against the different baseline methods described above:

[0331] Table 2: Evaluation of self-supervised embeddings on downstream tasks. Average test set performance and standard deviation values from five runs on different training set folds.

[0332] In Table 2, AUC stands for area under the receiver operating characteristic curve. In Table 2, MSLE stands for Mean Squared Logarithmic Error and is used for expense prediction, which is a regression task rather classification. Higher AUC and lower MSLE are better results. NPPR outperforms with statistical significance all other methods on all datasets. NPPR significantly outperforms FeatEng, which has been identified as a strong baseline that outperforms CoLES on two datasets. NPPR also outperforms RED on three datasets and the SimCSE on all datasets.

[0333] NPPR offers significant performance improvements on churn and credit default prediction problems, both of which aim to predict future behaviour of an entity. This confirms that generative modelling is particularly suitable for learning behavioural features that are predictive of future events. On tasks involving predicting a static attribute of an entity, such as age group prediction, the CoLES, which uses a contrastive method, is competitive. However, NPPR shows superior performance even in this case. This is achieved owing to transaction embeddings averaging, which is described in more detail below.

[0334] In terms of ablating past events reconstruction tasks, to identify the impact of including a past events reconstruction task in NPPR, Table 3 below shows results of ablating the two constituent tasks from NPPR.

[0335] Table 3: Ablation evaluation comparing NPPR to next event prediction (NP) and past events reconstruction (PR) tasks used in isolation. Average test set performance and standard deviation values from five runs on different training set folds.

[0336] For both NP and PR, entity embeddings were constructed by averaging transaction embeddings from the whole entity history, similar to that performed for NPPR. In general, embeddings pretrained with the next event prediction task perform significantly better than those using just the past event reconstruction task, with the exception of the churn prediction dataset. Indeed, using only the next event prediction task outperforms other baselines from Table 2 on three out of four problems, which demonstrates the strength of vanilla generative modelling for learning behavioural features.

[0337] Even though the performance gap between the two tasks can be significant, as in the case of the age group prediction dataset, using a combination of both tasks outperforms pretraining with either of the two tasks in isolation on all datasets. Adding even a small amount of past events reconstruction loss to the total loss has a positive effect on the performance on all downstream problems. This indicates that the past reconstruction task encourages each transaction embedding to encode longer term behavioural patterns, which next event prediction task does not explicitly do.

[0338] The past events reconstruction task performed better than next event prediction on churn prediction dataset. Reconstructing past events may help embeddings encode information from further back in time, which, in turn, allows a churn prediction model to model decline in transactional behaviour more accurately. Consequently, the best-performing value of a, which controls the contribution of the past events reconstruction task to the total loss, was larger on churn prediction compared to other datasets.

[0339] In terms of the effect of averaging transaction embeddings, the impact of using averaged transaction embedding as an entity embedding was evaluated by comparing to baseline methods in which an embedding of the most recent transaction is used. Table 4 shows the results of this evaluation.

[0340] Table 4: Relative performance difference between using averaged transaction embedding versus embedding of a most recent transaction.

[0341] Embedding averaging can improve performance on downstream problems, especially in cases where the task involves predicting a static entity attribute, such as age. However, embedding averaging can also have a detrimental effect in problems such as churn prediction. This may be because averaging can over-smooth the features encoded in the more recent embeddings, which capture a decline in the rate of transacting. By contrast, embedding averaging does not improve embeddings in the fourth baseline method, which are designed to be similar if they correspond to transactions of the same sequence.

[0342] Results of experiments on private datasets are now provided.

[0343] The above-described self-supervised NPPR technique was applied to pretrain a Foundation Purchasing Model on transaction data from a large consortium of issuing banks. This was used to produce transaction embeddings for unseen data; specifically transactions from hold-out issuing banks that operate in different countries from the issuers in the pretraining dataset. A separate fraud classifier was then trained on each of the hold-out issuers. The results demonstrate that the pretrained embedding model transfers well to significantly out-of-domain data and that self-supervised fine- tuning on target data can improve fraud detection classifiers in case of large domain shifts. Visualisations of the embedding space further demonstrate that the model learns semantic similarity between different merchant categories.

[0344] In terms of the pretrained consortium embedding model, a single embedding model was pretrained on a consortium of card transaction datasets from 180 European issuing banks. The consortium contains over 5.7 billion transactions, which provide complete transaction histories covering a period of twelve months for 61 million cardholders.

[0345] The architecture of the embedding model is as described above. In particular, the model comprises an MLP with 2 hidden layers (2048 neurons each) with ReLU activations, followed by a GRU layer with state size 1024 and a final projection layer to an embedding space of size 768. The decay length, A, in the NPPR method was 2 months, and the proportion of past events reconstruction task, a, was 0.001. In terms of fraud detection model datasets, labelled datasets from three European issuing banks were used for the downstream fraud detection task. These datasets were not part of the consortium used for pretraining and correspond to issuers that operate in different countries from any of the issuers from the consortium. This presents an opportunity for testing transfer of the embedding model to significantly out-of-domain data. The three datasets contain twelve months of transactions from 17 million, 3.5 million, and 1.8 million cardholders. Each dataset was split into training, validation, and test sets. Such splitting was carried out both temporally and on an entity level. More specifically, the split sets comprise transactions from different cardholders and non-overlapping consecutive time periods. The fraud rate in each dataset is 0.04%, 0.027%, and 0.11% respectively. Owing to the high class imbalance, genuine transactions were downsampled before classification models were trained. However, no downsampling was performed for pretraining.

[0346] In terms of fraud detection model baseline features, a representative traditional fraud prevention model as described in Jha, S., Guillen, M., and Christopher Westland, J. "Employing transaction aggregation strategy to detect credit card fraud", Expert Systems with Applications 39, 16 (2012), 12650-12657 was used as a baseline. This model comprises fourteen hand-engineered behavioural features in the form of aggregations over windows of past transactions by the same entity at different timescales. Examples of such features include the average amount spent per transaction over the last month, and the total number of transactions with the same merchant during the last month.

[0347] In terms of fraud detection models, downstream classification models comprised MLPs with 3 hidden layers (1024 neurons each) and ReLU activations. These were trained with learning rate 10“3, batch size 1024 and early stopping. Dropout was used as regularisation with dropout rate 0.2.

[0348] In terms of fraud detection evaluation metrics, production fraud prevention systems are often judged by their performance in reducing fraud losses while operating at high precision score thresholds. False positive predictions lead to declined transactions. An appropriate metric should measure the value of fraudulent transactions that have been prevented at a certain threshold of declined genuine transactions. A typical metric is the "VDR @ FP-ratio" where value detection rate (VDR) is the true positive rate weighed by transaction value and the threshold metric false positive ratio (FP-ratio) is the number of false positives divided by the number of true positives. The classification score threshold is adjusted to reach a typical value of FP-ratio between 1:1 and 20:1.

[0349] In terms of quantitative results, three different classification models were evaluated. The evaluations compare baseline features, baseline features with NPPR embeddings pretrained on the consortium, and baseline features with NPPR embeddings from models that have been first pretrained on the consortium and subsequently finetuned on downstream datasets.

[0350] Finetuning was performed in a self-supervised manner using NPPR and was therefore applicable to a scenario in which an embeddings server and a downstream modelling setup were decoupled.

[0351] Referring to Figure 13, results 1300 are shown in which VDR is plotted against different FP-ratio thresholds. In more detail, the results 1300 relate to evaluations of fraud detection models trained on a dataset from three different issuers. Three models are shown: baseline hand-engineered features from Jha et al. (the lower line for all three datasets), baseline features with NPPR embeddings pretrained on the consortium (the middle line for all three datasets), and baseline features with NPPR embeddings finetuned on downstream datasets (the upper line for all three datasets). Error bars were computed from five independent training runs.

[0352] The addition of NPPR embeddings to the baseline features provides significant improvements in the value detection rate on all FP-ratio thresholds. At 5:1 FP-ratio, the NPPR embeddings can provide up to 140% uplift over the hand-engineered features.

[0353] On hold-out datasets 1 and 3, embeddings generated by the pretrained model showed comparable performance to embeddings generated by finetuned models. This demonstrates the effectiveness of pretraining on a large consortium of diverse datasets. On dataset 2, a greater degree of domain shift with respect to the pretraining consortium was observed. As a result, the fine-tuning step significantly improved performance.

[0354] In terms of visualising the embedding space, to provide insights into the information encoded by embeddings, visualisations of embeddings of mcc are provided. These codes classify merchants and businesses by the type of goods or services provided. Large merchants classified as airlines, car rental companies and lodging providers typically have their own mcc. Each mcc embedding was calculated as the average embedding of all transactions corresponding to that mcc.

[0355] Referring to Figure 14, there is shown a t-distributed stochastic neighbour embedding (t-SNE) projection 1400 of the mcc embedding space together with three selected mccs and their nearest neighbours. Each mcc embedding was obtained by averaging transaction embeddings corresponding to those mccs.

[0356] Table 5 lists the five nearest neighbours and their distances, measured by cosine distance, to each selected mcc.

[0357] Table 5: Nearest neighbours in the embedding space of three mcc embeddings. For each nearest neighbour mcc, cosine distance in the original space between the mcc in the top row is shown. As can be seen from Table 5, the nearest neighbours of Lufthansa are all airline companies, while those of Hilton Hotels belong to the lodging industry. This demonstrates that embeddings of merchants in the same industry are located close to each other in the embedding space. The cluster of embeddings for hotels is in close proximity to the cluster for airlines. This is expected since purchases from merchants in these two clusters are often correlated.

[0358] This illustrates that generative modelling allows the embedding space to encode meaningful similarity between different mccs akin to semantic similarity captured by word embeddings from LLMs. This observation is consistent with results from Bruss, C. B., Khazane, A., Rider, J., Serpe, R., Gogoglou, A., and Hines, K. E. "DeepTrax: Embedding Graphs of Financial Transactions", 2019, in which merchant node embeddings from a graph neural network were used.

[0359] In a similar fashion, aggregations of transaction embeddings could be used to obtain embeddings of other entities, such as merchants, cardholders, transaction types, and geographical locations. They can potentially be used as features to support decision-making in recommendation engines and other financial systems.

[0360] By way of a summary, examples described above provide self-supervised generative methods and systems for obtaining contextualised embeddings of events, such as financial transactions, by combining two pretraining tasks: next event prediction and past events reconstruction.

[0361] Evaluations on publicly available datasets show that embeddings produced in this manner outperform embeddings from other self-supervised methods and hand-engineered features on a range of downstream tasks.

[0362] The above-described examples have been applied to a card fraud detection problem. The results show that they significantly improve the value detection rate at high-precision thresholds. By pretraining on a large consortium of transactions from multiple issuing banks, it has been demonstrated that embedding models trained in accordance with the above-described examples generalise well to out- of-distribution data.

[0363] Further, visualisations of the merchant category embedding space have been provided. These demonstrate that semantic similarities between industries are encoded. This motivates the use of aggregated embeddings of various entities in financial models.

[0364] While specific embodiments have been described in detail, it will be appreciated by those skilled in the art that the embodiments described in detail are not limiting on the scope of the claimed invention.

[0365] It will also be appreciated that optional features described hereinabove in respect of aspects, embodiments and examples apply equally, where technically appropriate, to other aspects, embodiments and examples. Where technically appropriate, aspects, embodiments and examples may be combined. Methods described herein may be computer-implemented.

[0366] The aspects, embodiments and examples described herein each extend to a non-transitory computer- readable medium comprising instructions that, when executed by a processor, cause the processor to carry out any method described herein.

[0367] The aspects, embodiments and examples described herein each further extend to a computer software product comprising instructions that, when executed by a processor, cause the processor to carry out any method described herein.

[0368] In the context of this specification "comprising" is to be interpreted as "including".

Claims

Claims1. A method of controlling a system having an encoder-decoder architecture, the encoderdecoder architecture having an encoder network and a decoder network, the encoder network comprising at least one encoder, and the decoder network comprising at least one decoder, wherein the method comprises: obtaining event data, the event data representing a given event that occurred at a given time; providing the event data as input to the encoder network to generate encoded event data, the encoded event data comprising an encoded representation of the given event that occurred at the given time; providing the encoded event data as input to the decoder network to generate decoded event data, the decoded event data representing one or more further events, wherein each of the one or more further events occurs at a respective further time; providing the decoded event data and reference event data as inputs to one or more loss functions to generate loss data, wherein the reference event data represents the one or more further events, and wherein the loss data represents one or more losses of the decoded event data relative to the reference event data; and generating encoder configuration data using the loss data, the encoder configuration data comprising one or more encoder parameters.

2. A method according to claim 1, wherein the loss data is based on time difference data, the time difference data being indicative, for each of the one or more further times, of a time difference between the given time and the respective further time of the one or more further times.

3. A method according to claim 2, comprising providing the time difference data as further input to the decoder network to generate the decoded event data.

4. A method according to claim 2 or 3, comprising providing the time difference data as input to one or more weight functions to generate weighting data, the weighting data being indicative of one or more weightings to be applied to the loss data.

5. A method according to claim 4, wherein each of the one or more weightings is associated with a respective further time of the one or more further times, and wherein each of the one or more weightings is dependent on a time difference between the given time and the respective further time.

6. A method according to claim 5, wherein each of the one or more weightings decreases monotonically as an absolute time difference between the given time and the respective further time increases.

7. A method according to any of claims 4 to 6, wherein generating the encoder configuration data using the loss data comprises: providing the loss data and the weighting data as inputs to one or more weighted loss data generators to generate weighted loss data; and generating the encoder configuration data using the weighted loss data.

8. A method according to claim 7, wherein the weighted loss data comprises a plurality of weighted loss data components, wherein each of the weighted loss data components is associated with a respective further time of the one or more further times, and wherein generating the encoder configuration data using the weighted loss data comprises: providing the plurality of weighted loss data components as inputs to one or more weighted loss data component combiners to generate combined weighted cost data; and generating the encoder configuration data using the combined weighted cost data.

9. A method according to claim 8, comprising using stochastic gradient descent to minimise the combined weighted cost data.

10. A method according to any of claims 1 to 9, wherein the decoder network comprises a plurality of decoders.

11. A method according to any of claims 1 to 10, wherein the one or more further events comprises a plurality of further events.

12. A method according to any of claims 1 to 11, wherein the decoder network comprises one or more past event decoders, wherein a past event decoder is operable to generate decoded past event data, and wherein decoded past event data represents one or more past events relative to the given event.

13. A method according to any of claims 1 to 12, wherein the decoder network comprises one or more future event decoders, wherein a future event decoder is operable to generate decoded future event data, and wherein decoded future event data represents one or more future events relative to the given event.

14. A method according to any of claims 1 to 13, wherein the encoder network comprises a preprocessor, wherein the pre-processor is operable to generate the encoded event data by preprocessing the event data to generate event feature data, and wherein the event feature data represents one or more features of the given event.

15. A method according to claim 14, wherein the event feature data comprises numerical feature data representing one or more numerical features of the given event.

16. A method according to claim 14 or 15, wherein the event feature data comprises categorical feature data representing one or more categorical features of the given event.

17. A method according to any of claims 14 to 16, wherein the decoded event data comprises further event feature data, the further event feature data representing one or more features of the one or more further events.

18. A method according to claim 17, wherein the further event feature data comprises numerical feature data representing one or more numerical features of the one or more further events.

19. A method according to claim 18, wherein the loss generator uses squared error processing to generate the loss data when the decoded event data and the reference event data comprise numerical feature data.

20. A method according to any of claims 17 to 19, wherein the further event feature data comprises categorical feature data representing one or more categorical features of the one or more further events.

21. A method according to claim 20, wherein the loss function uses cross entropy processing to generate the loss data when the decoded event data and the reference event data comprise categorical feature data.

22. A method according to any of claims 1 to 21, wherein the encoder network comprises one or more sequential neural networks.

23. A method according to claim 22, wherein the one or more sequential neural networks comprise one or more recurrent neural networks.

24. A method according to claim 23, wherein the one or more recurrent neural networks comprise one or more commutative state update recurrent neural networks.

25. A method according to any of claims 22 to 24, wherein the one or more sequential neural networks comprise one or more attention-mechanism-based neural networks.

26. A method according to any of claims 1 to 25, wherein the encoded event data comprises an embedding vector.

27. A method according to any of claims 1 to 26, wherein the event data and the reference event data collectively comprise event data from a consortium of event data sources.

28. A method according to any of claims 1 to 27, wherein the event data is based on sensor data captured by one or more sensors.

29. A method according to any of claims 1 to 28, wherein the event data is based on one or more messages encoding one or more instructions.

30. A method according to any of claims 1 to 29, comprising providing at least some of the reference event data as further input to the encoder network to generate the encoded event data.

31. A method according to any of claims 1 to 30, comprising providing past event data as further input to the encoder network to generate the encoded event data, the past event data representing one or more past events relative to the given event.

32. A method according to any of claims 1 to 31, wherein the event data comprises multi-variate event data.

33. A method according to any of claims 1 to 32, wherein the reference event data comprises multi-variate event data.

34. A method according to any of claims 1 to 33, wherein the one or more encoder parameters comprise one or more neural network weights.

35. A method according to any of claims 1 to 34, comprising training the encoder network using the encoder configuration data.

36. A method according to any of claims 1 to 35, comprising generating decoder configuration data using the loss data, the decoder configuration data comprising one or more decoder parameters.

37. A method according to claim 36, comprising training the decoder network using the decoder configuration data.

38. A method according to claim 37, comprising discarding the decoder network after the decoder network has been trained.

39. A method according to any of claims claim 36 to 38, wherein the one or more decoder parameters comprise one or more neural network weights.

40. A method according to any of claims 1 to 39, wherein the encoded event data comprising an encoded representation of the given event that occurred at the given time and one or more events that occurred before the given time.

41. A method comprising: providing input event data to an encoder network, the encoder network having been configured based on encoder configuration data generated in accordance with a method according to any of claims 1 to 40; and obtaining encoded event data from the encoder network, the encoded event data being based on the input event data.

42. A method according to claim 41, wherein the input event data comprises event data representing only a single event and wherein the encoded event data comprises encoded event data representing only the single event.

43. A method according to claim 41, wherein the input event data comprises event data representing an event, wherein the input event data comprises one or more values of state resulting from processing event data representing one or more past events relative to the event, and wherein the encoded event data comprises encoded event data representing only the event.

44. A method according to claim 41, wherein the input event data comprises event data representing a plurality of events and wherein the encoded event data comprises encoded event data representing the plurality of events.

45. A method according to any of claims 41 to 44, wherein the encoded event data is useable by a first decoder to generate first decoded event data and is usable by a second, different decoder to generate second, different decoded event data.

46. A method according to any of claims 41 to 45, comprising using the encoded event data to generate a control signal to control operation of an element of a computer network.

47. A method according to any of claims 41 to 46, wherein the encoded event data being based on the input event data comprises the encoded event data being based on an average of at least one feature represented by the input event data.

48. A method comprising using generative autoregression to generate output event data based on input event data, the input event data comprising time-ordered, event-driven data.

49. A method comprising: using generative autoregression to generate expected event data based on past event data, the expected event data comprising first expected event data representing a first expected event and second expected event data representing a second expected event; and generating a composite loss based on: a difference between the first expected event data and first actual event data, the first actual event data representing a first actual event; and a difference between the second expected event data and second actual event data, the second actual event data representing a second actual event.

50. A method comprising using generative autoregression to generate expected transaction data based on past transaction data, the past transaction data representing one or more past transactions and the expected transaction data representing one or more expected transactions.

51. A method comprising using generative autoregression to generate expected event data based on past event data, the past event data representing one or more past events and the expected event data representing one or more expected events.

52. A system configured to perform a method according to any of claims 1 to 51.

Images