Processing sequences of multi-modal entity features using convolutional neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing multi-modal entity data sequences using convolutional neural networks. One of the methods includes receiving an input sequence of multi-modal feature vectors characterizing entities on a time window, wherein each multi-modal feature vector in the input sequence corresponds to a different time interval during the time window; processing the input sequence of the multi-modal feature vectors using a convolutional neural network to generate a potential sequence comprising a plurality of potential feature vectors; processing the potential sequence of potential feature vectors using an aggregated neural network to generate aggregated feature vectors; and processing the aggregated feature vector using an output neural network to generate a prediction characterizing the entity after the time window.

Description

Background Art

[0001] The present specification relates to using neural networks (more specifically, convolutional neural networks) to process time series data to generate predicted outputs representing entities.

[0002] A neural network is a machine learning model that includes an output layer and one or more hidden layers, at least some of which apply non-linear transformations to received inputs to generate outputs. Summary of the invention

[0003] This specification describes a system implemented as a computer program on one or more computers in one or more locations that generates predictions regarding future behavior of an entity by processing a sequence of multimodal feature vectors characterizing the entity using a neural network.

[0004] Each feature vector in the sequence corresponds to a different time interval during the time window and characterizes the entity during the corresponding time interval. Because each feature vector is generated from multiple different types of features that characterize the entity during the corresponding time interval, the feature vector is called "multi-modal".

[0005] The neural network comprises a convolutional neural network, i.e., a neural network comprising one or more convolutional layers with a one-dimensional kernel, which processes a sequence of feature vectors to generate a latent sequence of latent feature vectors as output.

[0006] The neural network also includes an aggregation neural network (e.g., a recurrent neural network) that receives the output of the convolutional neural network (i.e., the latent sequence) as input and processes the output of the aggregation neural network to generate an aggregate feature vector.

[0007] The neural network also includes an output neural network that processes the aggregated feature vector to generate a prediction related to the entity.

[0008] In some aspects, a method includes: receiving an input sequence of multimodal feature vectors representing an entity over a time window, wherein each multimodal feature vector in the input sequence corresponds to a different time interval during the time window; processing the input sequence of multimodal feature vectors using a convolutional neural network to generate a latent sequence including a plurality of latent feature vectors; processing the latent sequence of latent feature vectors using an aggregate neural network to generate an aggregate feature vector; and processing the aggregate feature vector using an output neural network to generate a prediction representing the entity after the time window.

[0009] Some aspects of the described subject matter include one, some, or all of the following features.

[0010] The convolutional neural network includes a plurality of convolutional layers, each of which has a corresponding one-dimensional kernel.

[0011] The aggregate neural network is a recurrent neural network.

[0012] The output neural network includes one or more fully connected layers, and the one or more fully connected layers are followed by an output layer.

[0013] The method also includes: obtaining data representing the entity from a plurality of different data streams each corresponding to a respective non-standardized format; and generating the multimodal feature vector in the input sequence by converting the data representing the entity into a standardized format.

[0014] Generating each multimodal feature vector may include: identifying corresponding features of each feature type among multiple feature types that characterize an entity during a corresponding time interval for the multimodal feature vector; and for each feature type, adding the identified corresponding features of the feature type to one or more entries in the multimodal feature vector corresponding to the feature type.

[0015] The convolutional neural network, the aggregation neural network, and the output neural network have been jointly trained on training data, the training data comprising a plurality of training input sequences and corresponding ground truth results for each training input sequence.

[0016] The entities are financial assets, and each multimodal feature vector includes a technical analysis feature and a sentiment analysis feature.

[0017] Each multimodal feature vector also includes basic analysis features.

[0018] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0019] By using the described deep neural network for prediction, the system can accurately account for complex temporal dependencies between features of different modalities to make accurate predictions related to the future behavior of an entity. In particular, the deep neural network can use one-dimensional convolutions to combine features of different modalities across time. The deep neural network can then use an aggregate neural network (e.g., RNN) to aggregate these features into a single vector representing features from multiple modalities related to the future behavior of the entity, and then make predictions based on the aggregated features. The neural network architecture results in a deep neural network that can make accurate predictions. In addition, the neural network architecture is suitable for being fine-tuned to make predictions about different entities, different time horizons, or based on dedicated data. That is, once a neural network with the described architecture is pre-trained, the neural network can be fine-tuned in a data-efficient and computationally efficient manner to perform any of a variety of prediction tasks.

[0020] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1A An example prediction system is shown.

[0022] Figure 1B An example user interface is shown.

[0023] Figure 2 An example data ingestion system is shown.

[0024] Figure 3 is a flow diagram of an example process for generating predictions that characterize an entity.

[0025] Figure 4 is a flow chart of an example process for training a predictive neural network.

[0026] Like reference numbers and numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0027] Figure 1A An example forecasting system 100 is shown. Forecasting system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below may be implemented.

[0028] The system 100 uses a predictive neural network 108 to process an input sequence 102 including a feature vector characterizing an entity to generate a prediction 132 regarding the future behavior of the entity.

[0029] Typically, the input sequence 102 corresponds to a fixed-size time window, and the prediction 132 is a prediction regarding the future behavior of the entity after the time window has passed.

[0030] More specifically, each feature vector in the input sequence 102 corresponds to a corresponding time interval during the time window and includes data characterizing an entity during the corresponding time interval.

[0031] Typically, each feature vector in the input sequence 102 includes features of multiple modalities representing information related to the entity. That is, each feature vector is a multimodal feature vector generated from multiple different types of features each representing different aspects of the entity during the corresponding time interval.

[0032] The data ingestion system 200 collects data related to an entity and generates an input sequence 102 from the collected data for processing by the prediction neural network 108. Generally, the data ingestion system 200 repeatedly combines multiple different types of features into a single feature vector to generate the input sequence 102. Figure 2 The operation of the data import system 200 is described in more detail.

[0033] As a specific example, the entity may be a financial asset, such as a stock, a commodity, a cryptocurrency, or a different type of financial asset.

[0034] The feature vectors in the input sequence 102 may characterize a financial asset within a time window, and the prediction 132 may be a prediction characterizing the predicted future trading behavior of the financial asset after the end of the time window. As a specific example, the prediction 132 may characterize the predicted trading behavior of the financial asset at the end of the next time interval after the end of the time window.

[0035] When the entity is a financial asset, each feature vector may be generated from a plurality of different types of features that each characterize the asset in a different way. As a specific example, the features may include technical analysis features, sentiment analysis features, and optionally fundamental analysis features. Figure 2 The generation of feature vectors when the entity is a financial asset is described in more detail.

[0036] In some cases, the time window may be a relatively long time window. For example, the time window may cover thirty days or sixty days or ninety days, and the time intervals may each be one day long. Thus, in this example, when the time window is sixty days, the input sequence 102 may include sixty feature vectors, each of which corresponds to a different day in the sixty days and includes information characterizing the financial asset during the corresponding day. In addition, the prediction 132 may characterize the asset price for the day after the sixty-day time window ends.

[0037] In some other cases, the time window may be a relatively short time window. For example, the time window may cover a few seconds or a few minutes, and the time intervals may each be a predetermined fraction of the time window.

[0038] Forecasts 132 may include any of a variety of predictions regarding the future trading behavior of a financial asset.

[0039] For example, the prediction may characterize the future price of a financial asset. As an example, prediction 132 may include a probability distribution for assigning a corresponding probability to each of a plurality of outcomes. For example, each outcome may correspond to a different range of percentage changes in the price of an asset (e.g., the percentage change in the price of an asset from the end of a time window ("current price") to the end of a next time interval after the end of the time window). For example, one outcome may correspond to the price staying within 5% of the current price, another outcome may correspond to the price increasing by more than 5% of the current price, and a third outcome may correspond to the price decreasing by more than 5% of the current price.

[0040] As another example, the prediction 132 may include a regression value representing a predicted change (ie, a percentage change or an absolute change) in the price of the asset from the end of the time window to the end of the next time interval after the end of the time window.

[0041] As another example, the prediction 132 may also include an uncertainty metric, i.e., in addition to the probability distribution, the regression value, or both. In some cases, the neural network 108 generates the uncertainty metric independently, i.e., using a different set of one or more output layers than the set used to generate the rest of the prediction. In some other cases, the system 100 generates the uncertainty metric based on the prediction of the neural network 108 for one or more recent time windows immediately preceding the current time window, e.g., by calculating a variance or other divergence measure between the recent predictions.

[0042] As another example, forecast 132 may characterize the trading volume of the financial asset during the next time interval after the end of the time window.

[0043] In some cases, the system 100 maintains multiple instances of the neural network 108, where different instances of the neural network 108 have been trained to generate different corresponding predictions 132 that characterize the future training behavior of the financial asset in different ways.

[0044] For example, different instances of the neural network 108 may be configured to generate predictions over different time horizons after the end of a current time window, e.g., one instance makes predictions regarding asset trading behavior after a relatively short time interval after the end of the current time window, and another instance makes predictions regarding asset trading behavior after a relatively long time interval after the end of the current time window.

[0045] As another example, different instances of neural network 108 may correspond to different venues where financial assets can be traded (e.g., different regulated markets where financial assets can be traded (e.g., stock exchanges), multilateral trading facilities (MTFs), and organized trading facilities (OTFs)). Thus, each instance may make predictions regarding future trading behavior if trading is conducted at the corresponding trading venue.

[0046] As yet another example, for each trading venue, the system 100 may maintain multiple different instances corresponding to the trading venue but making forecasts over different time horizons.

[0047] In some cases, as will be described below, each instance may be generated by fine-tuning the same pre-trained neural network 108 on a different training dataset.

[0048] The prediction neural network 108 includes a convolutional neural network 110 , an aggregation neural network 120 , and an output neural network 130 .

[0049] The system 100 processes the input sequence 102 using a convolutional neural network 110 to generate a latent sequence 112 including a plurality of latent vectors. Each latent vector generally corresponds to a corresponding latent time interval within a time window.

[0050] In some implementations, the latent time interval is the same length as the input time interval, and each latent feature vector has a corresponding input vector in the input sequence 102 .

[0051] In some other implementations, the latent time interval is longer than the input time interval. In these implementations, the latent sequence 102 has fewer vectors than the input sequence 102, and each latent feature vector corresponds to a plurality of the input feature vectors.

[0052] Additionally, depending on the configuration of the convolutional neural network 110, the dimension of the latent feature vector may be the same as or larger than the dimension of the input feature vector.

[0053] Typically, the convolutional neural network 110 is a neural network that includes one or more convolutional neural network layers with a one-dimensional kernel.

[0054] The convolutional neural network 110 may also include other types of neural network layers, such as a batch normalization layer, a layer normalization layer, a pooling layer, and a dropout layer.

[0055] As a specific example, the neural network 110 may include a stack of convolutional layers with one-dimensional kernels, where each convolutional layer except the last convolutional layer in the stack is followed by a corresponding normalization layer, such as batch normalization and layer normalization, and followed by a dropout layer during training.

[0056] By using the convolutional neural network 110, the system 100 can generate a latent vector that incorporates temporal context from adjacent time intervals. The system can scale how much temporal context is incorporated by each given convolutional layer by increasing the width of the kernel of the convolutional layer, increasing the number of convolutional layers, or both.

[0057] The system 100 then processes the latent sequence 112 using an aggregate neural network 120. The aggregate neural network 120 processes the latent vectors in the latent sequence 112 to generate an aggregate feature vector 122 that characterizes the entity over the entire time window.

[0058] For example, aggregate neural network 120 may be a recurrent neural network, such as a long short-term memory (LSTM) neural network or a gated recurrent unit (GRU) neural network, that initializes a hidden state before processing the first latent feature vector in the latent sequence and then updates the hidden state as each latent feature vector in latent sequence 112 is processed. In this example, aggregate feature vector 122 may be the hidden state of the last recurrent layer in the recurrent neural network after processing the last feature vector in latent sequence 112.

[0059] The system 100 processes the aggregated feature vector 122 using the output neural network 130 to generate a prediction 132 for the entity.

[0060] For example, the output neural network 130 may include one or more fully connected neural network layers followed by an output layer appropriate for the type of prediction 132 being generated by the system 100 .

[0061] For example, when the prediction 132 is a probability distribution over multiple possible classes, the output layer may be a softmax layer.

[0062] As another example, when prediction 132 includes one or more than one regressed value, the output layer may be a linear layer having a number of nodes equal to the number of values ​​that need to be regressed.

[0063] Although Figure 1A The description shows the predictive neural network 108 processing a single input sequence 102 to generate a single prediction 132 for a single entity based on features for the entity during a single time window, but in practice the system 100 can be configured to make many different predictions with low latency.

[0064] In particular, the system 100 can be configured to generate a new prediction 132 for an entity whenever a new time interval passes. That is, whenever a new time interval passes, the system 100 discards the feature vector for the oldest time interval in the time window and adds the new feature vector for the new time interval to the end of the input sequence to generate a new input sequence. The system 100 then processes the new input sequence using the prediction neural network 108 to generate a new prediction.

[0065] As another example, the system 100 can be configured to generate predictions 132 for many different entities in parallel. That is, each time a new time interval passes, the system 100 can generate a corresponding input sequence for each entity in a set of multiple entities, and then process each input sequence using the predictive neural network 108 to generate a corresponding prediction 132 for each entity in the set.

[0066] To reduce the latency of these predictions, system 100 may be co-located with downstream systems that utilize the predictions 132 generated by the system.

[0067] As another example, the system 100 may deploy a respective instance of the neural network 108 for each entity of interest and maintain a separate data stream for each entity to ensure that predictions for each entity can be made quickly.

[0068] As yet another example, instances of neural network 108 may be deployed on respective hardware accelerators, such as graphics processing units (GPUs) or dedicated ASICs designed to accelerate neural network computations.

[0069] Once the system has generated the prediction 132 , the system 100 or another system may use the prediction 132 for any of a variety of purposes.

[0070] As an example, when the entity is a financial asset, the system 100 may apply one or more rules to the prediction 132 to determine whether to initiate an automatic transaction of the financial asset. For example, the system may determine that an automatic purchase or sale of an asset should be initiated if the probability assigned to a particular percentage range exceeds a threshold.

[0071] As another example, the system 100 may generate a corresponding prediction 132 for each asset in a portfolio that the user has identified. The system 100 may then use the corresponding prediction 132 to identify which assets in the portfolio should be recommended to the user for further analysis.

[0072] For example, the system 100 may determine to recommend any asset that is predicted to have a sufficient probability of having a price change of at least a threshold percentage. The recommended assets may then be identified in a user interface presented to the user on the user's computer. Optionally, data derived from the input sequence for the identified assets, such as technical analysis data for the assets and sentiment analysis data for the assets, may also be presented to the user. For example, the user interface may include a "dashboard" that presents a summary of the recommended features for each identified asset and allows the user to browse between the recommended assets.

[0073] Figure 1B An example user interface 150 is shown presenting data generated as a result of predictions made by the neural network 108 .

[0074] In particular, Figure 1B In the context of a financial statement, an asset of interest is an asset that is the subject of an ongoing or completed transaction.

[0075] The user interface 150 includes a portion 160 that identifies transactions that are in progress or completed.

[0076] For each transaction identified in the first portion 160, the first portion 160 includes an "RPM" column 162 that displays a score ("Relative Performance Measure") that is generated using the predictions made by the neural network 108 and characterizes the extent to which the transaction is consistent with the predictions of the neural network 108 for the corresponding asset, i.e., higher scores indicate that the transaction is more strongly consistent with the predictions of the neural network 108.

[0077] As from Figure 1B As can be seen in the example of , the transaction 164 to sell 10,000 shares of AAPL within 8 hours has a relatively low RPM of 35, which indicates that the prediction generated by the neural network 108 for the asset indicates that the asset price will likely increase after the end of the 8-hour window, making the decision to sell the shares of the asset a poor decision. As a result, the user interface 150 displays a warning 166 indicating that the RPM for the transaction 164 is low, thereby giving the user an opportunity to further analyze the transaction or cancel the transaction before it is completed, for example.

[0078] While user interface 150 is one example of the manner in which predictions made by neural network 108 may be used, other examples are possible.

[0079] As another example, the system may use neural network 108 to determine the routing of an order. For example, as described above, the system may maintain multiple instances of neural network 108 corresponding to different time horizons, different venues, or both. When system 100 receives data specifying an order for a particular asset, system 100 may determine the time when the order must be completed and the trading venue at which the order can be completed. The system may then determine a routing for the order specifying when to place the order, at which venue to place the order, or both based on the predictions made by the instances of neural network 108 for each trading venue (and optionally for each of a plurality of time horizons), and then provide information indicating the routing to the user or automatically execute the order according to the routing.

[0080] As another example, the system 100 can generate the forecast 132 as part of testing or evaluating an investment strategy or portfolio allocation. That is, the system 100 or another system can use the forecast 132 for multiple different assets to automatically adjust the investment strategy or portfolio allocation, i.e., automatically adjust the investment strategy or portfolio allocation by calculating the overall predicted returns of multiple different investment strategies and selecting the strategy with the best overall predicted return as the final strategy.

[0081] Before using neural network 108 to generate predictions, system 100 or another training system trains neural network 108 on training data.

[0082] The training data includes a set of training input sequences each corresponding to a corresponding entity and a ground truth result for each training input sequence, where the ground truth result is used to characterize what actually occurs to the corresponding entity after the time window covered by the training input sequence ends.

[0083] The system can then use any suitable supervised learning technique to train the neural network 108 on the training data. For example, the system can use a suitable optimizer such as Adam, rmsProp, or Adafactor to train the neural network 108 to minimize a suitable loss function such as a classification loss, a regression loss, or a loss that includes both a classification loss term and a regression loss term, which measures the error between the predictions for the training input sequence and the corresponding ground truth results for the training input sequence.

[0084] In some cases, the training system may first train neural network 108 on training data that includes training input sequences corresponding to a large number of entities, and then the system may fine-tune the neural network on the training input sequences specific to a particular user before deploying neural network 108 for making predictions for the particular user of system 100. For example, the particular user may be interested in some small subset of a larger set of entities or a set of entities that includes entities that are not represented in the training data used to train neural network 108.

[0085] Reference below Figure 4 Explain fine-tuning neural networks in more detail.

[0086] Figure 2 An example data ingestion system 200 is shown.

[0087] The data ingestion system 200 is an example of a system for processing data representing an entity to generate an input sequence 102 for processing by the predictive neural network 108 .

[0088] In particular, the data ingestion system 200 receives data from a plurality of different data streams 210A-210N of various different types and in various different formats and combines the data to generate data in a standardized format that the neural network 108 can process.

[0089] In particular, each feature vector 202 needs to be in a standardized format to be processed by the neural network 108. More specifically, each feature vector 202 includes the same number of entries, i.e., the same number of values, and each entry in a given feature vector 202 must correspond to the same type of feature as the corresponding entries in other feature vectors in the input sequence.

[0090] Thus, during operation of the prediction system 100, the feature generation engine 230 within the system 200 repeatedly converts data related to a given entity received from the data streams 210A-210N into a standardized format by collecting the data corresponding to each time interval into a single feature vector in a consistent order.

[0091] In particular, the neural network 108 is configured to receive features in which each value falls within a certain normalized range, such as between 0 and 1 or from 0 to 255.

[0092] However, each data stream 210A-210N may provide data having values ​​that do not fall within the range, and different data streams may provide data from different ranges. Therefore, the system normalizes the values ​​received from each data stream to generate features in a standardized format required for processing using the neural network 108. In particular, the system may generate a feature vector for a given time interval by rescaling each value received from a given data stream corresponding to the given time interval so that the value falls within the normalized range and inserting it into an entry of the feature vector corresponding to the data stream. The system may rescale the values ​​from a given data stream using data stream-specific rescaling data (i.e., using rescaling data that specifies a mapping from raw values ​​to values ​​in a normalized range based on minimum and maximum values ​​in historical values ​​received from the data stream).

[0093] The data flows 210A-210N may be, for example, an automated programming interface (API) or other interface that allows the system 200 to repeatedly obtain data for each entity in a set of entities (eg, each financial asset in a plurality of financial assets).

[0094] In some cases, each data stream 210A-210N corresponds to a different type of feature. In some other cases, different data streams 210A-210N may correspond to the same feature type.

[0095] As described above, when the entity is a financial asset, each feature vector includes technical analysis features, sentiment analysis features, and optionally fundamental analysis features. Thus, in these cases, at least one of the data streams 210A-210N provides technical analysis data, at least one of the data streams 210A-210N provides sentiment data, and when fundamental analysis features are included, at least one of the data streams 210A-210N provides fundamental analysis data.

[0096] The technical analysis data for a given time interval characterizes the price and optional trading volume of the financial asset during the time interval and optionally at the previous time interval. For example, the system can obtain features such as moving averages of financial asset prices, momentum of financial asset prices, crossovers of financial assets, and probability bands of financial assets from one or more of the data streams 210A-210N. Each of these features can be calculated at various granularities (i.e., across various time scales). The engine 230 can then generate corresponding entries in the corresponding feature vector by including the technical analysis features of different granularities in the corresponding entries of the feature vector.

[0097] Sentiment data may characterize the sentiment of news articles, social media posts, and other communications related to an entity during a given time period.

[0098] In some implementations, the system 200 obtains the calculated sentiment metric for communications generated during a given time window from one or more of the data streams 210A-210N. The engine 230 can then generate corresponding entries in the corresponding feature vector by calculating various statistics on the received sentiment metric, such as the mean, median, and mode of the sentiment metric, and including the statistics in the feature vector. In some implementations, the engine 230 calculates statistics separately for different types of communications, such as separate statistics for news articles and social media posts.

[0099] In some other implementations, the system 200 obtains the text of the communication from the data streams 210A-210N, and then applies sentiment analysis techniques, such as machine learning-based sentiment analysis, on each communication to generate a corresponding sentiment score for each communication. The engine 230 can then generate corresponding entries in the corresponding feature vector by calculating various statistics on the generated sentiment metric, such as the mean, median, and mode of the sentiment metric, and including the statistics in the feature vector. In some implementations, the engine 230 calculates statistics separately for different types of communications, such as separate statistics for news articles and social media posts.

[0100] The fundamental analytical data includes data that characterizes the inherent value of the asset. For example, the system 200 can obtain features extracted from the financial report of the asset from one or more of the data streams 210A-210N. Examples of such features include income, expenses, assets, and liabilities. In some other implementations, the system 200 obtains the financial report and then extracts the fundamental analytical features from the financial report using natural language processing techniques.

[0101] Figure 3 3 is a flow chart of an example process 300 for generating predictions for an input sequence. For convenience, process 300 will be described as being performed by a system of one or more computers located at one or more locations. For example, a suitably programmed system such as Figure 1A The prediction system 100 and the prediction system 300 may be processed.

[0102] The system may perform an iteration of process 300 each time a new time interval ends (ie, as soon as features for the new time interval become available).

[0103] The system receives an input sequence of multimodal feature vectors representing entities over a time window (step 302). Each multimodal feature vector in the input sequence corresponds to a different time interval during the time window. In other words, each feature vector corresponds to a different proper subset of the time window.

[0104] The system processes the input sequence of multimodal feature vectors using a convolutional neural network to generate a latent sequence including multiple latent feature vectors (step 304). The convolutional neural network includes one or more one-dimensional convolutional layers, i.e., a convolutional layer with a one-dimensional kernel. In some implementations, the latent sequence has a reduced frequency relative to the input sequence but includes a greater number of features.

[0105] The system processes the latent sequence of latent feature vectors using a converged neural network to generate an aggregated feature vector (step 306). For example, the converged neural network can be a recurrent neural network that updates a hidden state as part of processing each latent feature vector in the latent sequence. As a specific example, the aggregated feature vector can be the hidden state of the last recurrent layer in the converged neural network after processing the last latent feature vector in the latent sequence.

[0106] The system processes the aggregated feature vector using an output neural network to generate a prediction representing the entity after the time window (step 308). For example, the output neural network may include one or more fully connected layers followed by an output layer.

[0107] Figure 4 is a flow chart of an example process 400 for training a predictive neural network. For convenience, process 400 will be described as being performed by a system of one or more computers located at one or more locations. For example, a suitably programmed system such as Figure 1A The prediction system 100 and the prediction system 400 may be processed.

[0108] The system trains a predictive neural network on an initial training data set (step 402).

[0109] The training data includes a set of training input sequences each corresponding to a corresponding entity and a ground truth result for each training input sequence, where the ground truth result is used to characterize what actually occurred to the corresponding entity after a time window covered by the training input sequence.

[0110] The system can use any suitable supervised learning technique to train the neural network on the training data. For example, the system can use a suitable optimizer such as Adam, rmsProp, or Adafactor to train the neural network to minimize a suitable loss function such as a classification loss, a regression loss, or a loss that includes both a classification loss term and a regression loss term, which measures the error between the prediction for the training input sequence and the corresponding ground truth result for the training input sequence.

[0111] The system obtains user-specific training data for a specific user of the system (step 404).

[0112] For example, a user may identify a proper subset of entities represented in the training data used to train the neural network in step 402 that are of interest to the user, and the system may generate user-specific training data that includes only entities in the proper subset.

[0113] As another example, a user may provide a training dataset that includes training input sequences for entities of interest to the user.

[0114] As yet another example, a user may provide historical data characterizing entities of interest to the user, and the system may generate user-specific training data by generating a set of training input sequences from the historical data.

[0115] The system fine-tunes the predictive neural network by training the predictive neural network on user-specific training data (step 406).

[0116] In particular, because the system trains the predictive neural network starting from the values ​​of the parameters of the predictive neural network determined by the training of step 402, this process is referred to as "fine-tuning".

[0117] In some cases, the system trains the neural network on the user-specific training data on the same objective function and using the same supervised learning technique used for training in step 402 .

[0118] In some other cases, the system modifies the supervised learning technique for the fine-tuning in step 406. As an example, the system can train the neural network at a reduced learning rate when performing the fine-tuning step 406 compared to when performing the training in step 402. As another example, the system can keep values ​​of certain parameters of the prediction neural network fixed during the fine-tuning step 406. For example, the system can keep parameters of the convolutional neural network fixed and adjust parameters of the aggregation neural network and the output neural network. As another example, the system can keep parameters of the convolutional neural network and the aggregation neural network fixed and only adjust parameters of the output neural network. Modifying the supervised learning technique can assist in preventing the prediction neural network from overfitting to the user-specific training data.

[0119] In some cases, the system can fine-tune multiple different instances of the neural network for the same user. For example, the user can provide or otherwise identify different training data sets, where each training data set corresponds to a corresponding time horizon, i.e., such that for each instance, a different time horizon after the end of the corresponding training sequence produces a ground truth result. As another example, the user can provide or otherwise identify different training data sets, where each training data set includes only data from a corresponding trading venue. Thus, each instance of the neural network is trained to make predictions specific to that trading venue.

[0120] During fine-tuning of a given instance of a neural network, the system can implement any of a variety of techniques to improve the quality of the fine-tuned model. For example, during fine-tuning, the system can periodically resample the training data, e.g., based on the performance of the model, to favor inputs having certain characteristics that the system has determined to improve the performance of the neural network. That is, the system can periodically measure performance during fine-tuning and use the performance metric to resample the training data during fine-tuning for use in future training iterations.

[0121] After fine-tuning the prediction neural network, the system deploys (one or more) fine-tuned prediction neural networks for use by users (step 408). That is, the system deploys the fine-tuned prediction neural network by allowing users to use the fine-tuned prediction neural network to generate predictions for new input sequences.

[0122] In some cases, the system deploys the fine-tuned predictive neural network by instantiating one or more instances of the fine-tuned predictive neural network on one or more user computers controlled by the user. Optionally, in these cases, the initial training step 402 can be performed on a set of computers remote from the user and inaccessible to the user, such as in a data center, while the fine-tuning step 406 can be performed on one or more user computers. This ensures that the user does not have access to the larger training data set used to perform the training step 402, for example, if it is proprietary or cannot be shared with the user for other reasons.

[0123] In some other cases, the system deploys the fine-tuned predictive neural network for use by the user by allowing the user to provide a new input sequence, such as through an API, and obtain the corresponding prediction generated by the predictive neural network, such as through an API. In these cases, steps 402 to 408 can be performed in a data center far away from the user.

[0124] The system can independently perform steps 404 to 408 of process 400 for multiple different users, each with a different set of user-specific training data, while performing only once the initial training step 402. Because the set of training data 402 will typically be much larger than the user-specific training data set, the system can generate "customized" predictive neural networks for multiple different users in a computationally efficient manner, i.e., because generating additional "customized" predictive neural networks for additional users only requires performing a relatively computationally inexpensive fine-tuning step.

[0125] This specification uses the term "configuration" in conjunction with system and computer program components. To configure a system of one or more computers to perform a particular operation or action means that software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action when in operation has been installed on the system. To configure one or more computer programs to perform a particular operation or action means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0126] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0127] The term "data processing equipment" refers to data processing hardware and covers all kinds of equipment, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The equipment may also be or further include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the equipment may optionally include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0128] A computer program (which may also be referred to or described as a program, software, software application, app, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and the computer program may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed over multiple sites and interconnected by a data communications network.

[0129] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way, or at all, and it may be stored on a storage device in one or more than one location. Thus, for example, an index database may include multiple data collections, each of which may be organized and accessed differently.

[0130] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally speaking, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0131] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits (e.g., FPGAs or ASICs), or by a combination of special purpose logic circuits and one or more programmed computers.

[0132] A computer suitable for executing a computer program can be based on a general-purpose microprocessor or a special-purpose microprocessor or both, or a central processing unit of any other kind. In general, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit system or incorporated into the special logic circuit system. In general, the computer will also include one or more mass storage devices (such as disks, magneto-optical disks or optical disks) for storing data, or be operably coupled to receive data from the one or more mass storage devices or to transmit data to them or to perform both. However, the computer does not need to have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device (such as a universal serial bus (USB) flash drive), just to name a few.

[0133] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD ROM and DVD-ROM disks.

[0134] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic, voice, or tactile input). In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, the computer may interact with the user by sending a text message or other form of message to a personal device (e.g., a smart phone running a messaging application) and receiving a response message from the user in return.

[0135] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface, web browser, or application through which a user can interact with an implementation of the subject matter described in this specification), or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs) (e.g., the Internet).

[0136] A computing system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship between the client and the server is generated by means of computer programs that run on respective computers and have a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a user device, for example, for the purpose of displaying data to a user interacting with a device used as a client and receiving user input from it. Data generated at the user device (e.g., the result of a user interaction) can be received from the device at the server.

[0137] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or the scope of what may be claimed, but should be interpreted as descriptions of features that may be specific to a particular embodiment of a particular invention. Specific features described in the context of a separate embodiment in this specification may also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although the features may be described above as working in a specific combination and even initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may involve a sub-combination or a deformation of a sub-combination.

[0138] Similarly, although operations are depicted in the drawings and described in the claims in a particular order, this should not be interpreted as requiring that these operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed to achieve the desired result. In certain circumstances, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described can generally be integrated together in a single software product or packaged into multiple software products.

[0139] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A system comprising: one or more computers; as well as One or more storage devices for storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations including: Receiving an input sequence of multimodal feature vectors representing entities over a time window, wherein each multimodal feature vector in the input sequence corresponds to a different time interval during the time window; Processing the input sequence of the multimodal feature vectors using a convolutional neural network to generate a latent sequence comprising a plurality of latent feature vectors; processing the latent sequence of latent feature vectors using an aggregate neural network to generate an aggregate feature vector; and The aggregated feature vector is processed using an output neural network to generate a prediction characterizing the entity after the time window.

2. The system according to claim 1, wherein: The convolutional neural network includes a plurality of convolutional layers, each of which has a corresponding one-dimensional kernel.

3. The system according to claim 1 or 2, wherein: The aggregate neural network is a recurrent neural network.

4. The system according to any one of claims 1 to 3, wherein: The output neural network includes one or more fully connected layers, and the one or more fully connected layers are followed by an output layer.

5. The system according to any one of claims 1 to 4, wherein the operation further comprises: obtaining data representing the entity from a plurality of different data streams; as well as The multimodal feature vector in the input sequence is generated by converting data representing the entity into a standardized format.

6. The system according to claim 5, wherein: Generating each multimodal feature vector includes: identifying a corresponding feature for each of a plurality of feature types characterizing an entity during a corresponding time interval for the multimodal feature vector; and For each feature type, the identified corresponding features of the feature type are added to one or more entries in the multimodal feature vector corresponding to the feature type.

7. A system according to any one of claims 1 to 6, wherein: The convolutional neural network, the aggregation neural network, and the output neural network have been jointly trained on training data, the training data comprising a plurality of training input sequences and corresponding ground truth results for each training input sequence.

8. The system according to any one of claims 1 to 7, wherein: The entity is a financial asset, and wherein each multimodal feature vector includes a technical analysis feature and a sentiment analysis feature.

9. The system according to claim 8, wherein: Each multimodal feature vector also includes basic analysis features.

10. The system according to claim 8, wherein: The forecast represents the predicted trading behavior of the financial asset at the end of a next time interval after the end of the time window.

11. A method performed by one or more computers, the method comprising: Receiving an input sequence of multimodal feature vectors representing entities over a time window, wherein each multimodal feature vector in the input sequence corresponds to a different time interval during the time window; Processing the input sequence of the multimodal feature vectors using a convolutional neural network to generate a latent sequence comprising a plurality of latent feature vectors; processing the latent sequence of latent feature vectors using an aggregate neural network to generate an aggregate feature vector; and The aggregated feature vector is processed using an output neural network to generate a prediction characterizing the entity after the time window.

12. The method according to claim 11, wherein: The convolutional neural network includes a plurality of convolutional layers, each of which has a corresponding one-dimensional kernel.

13. The method according to claim 11 or 12, wherein: The aggregate neural network is a recurrent neural network.

14. The method according to any one of claims 11 to 13, wherein: The output neural network includes one or more fully connected layers, and the one or more fully connected layers are followed by an output layer.

15. The method according to any one of claims 11 to 14, further comprising: obtaining data representing the entity from a plurality of different data streams; as well as The multimodal feature vector in the input sequence is generated by converting data representing the entity into a standardized format.

16. The method according to claim 15, wherein: Generating each multimodal feature vector includes: identifying a corresponding feature for each of a plurality of feature types characterizing an entity during a corresponding time interval for the multimodal feature vector; and For each feature type, the identified corresponding features of the feature type are added to one or more entries in the multimodal feature vector corresponding to the feature type.

17. The method according to any one of claims 11 to 16, wherein: The convolutional neural network, the aggregation neural network, and the output neural network have been jointly trained on training data, the training data comprising a plurality of training input sequences and corresponding ground truth results for each training input sequence.

18. The method according to any one of claims 11 to 17, wherein: The entity is a financial asset, and wherein each multimodal feature vector includes a technical analysis feature and a sentiment analysis feature.

19. The method according to claim 18, wherein: Each multimodal feature vector also includes basic analysis features.

20. One or more computer-readable storage media storing instructions which, when executed by one or more computers, cause the one or more computers to perform corresponding operations of the method according to any one of claims 11 to 19.

Images