Information processing system, information processing method, and program

Abstract

To carry out market simulation in consideration of market impact.SOLUTION: A system according to the present invention has a first vector calculation unit for calculating a feature vector of an expert based on a time-serial change of past board information, an activity probability estimation unit for specifying a transaction-related activity probability by enhanced learning using a transaction-related initial reward function with initialized coefficient vectors, a second vector calculation unit for calculating a feature vector of an agent corresponding to the feature vector of the expert based on the activity probability, and a reward function generation unit for generating a reward function by updating the coefficient vectors by reverse enhanced learning on the basis of the feature vector of an agent.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] Market simulations that mimic market movements are known. Patent Document 1 discloses a technology for building a simulation system using past transaction data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-022690 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the logic behind price fluctuations in the market was unclear, making it difficult to build a simulation system that took into account the impact of users' trading behavior on the market (market impact).

[0005] The present invention has been made in consideration of such problems, and has as its object to perform a market simulation that takes into account market impact. [Means for solving the problem]

[0006] In order to achieve the above object, the information processing system of the present invention includes a first vector calculation unit that calculates a feature vector of an expert based on time-series changes in past order book information; an action probability estimation unit that identifies action probabilities related to transactions through reinforcement learning using an initial reward function related to transactions in which a coefficient vector has been initialized; a second vector calculation unit that calculates a feature vector of an agent that corresponds to the feature vector of the expert based on the action probabilities; and a reward function generation unit that generates a reward function by updating the coefficient vector through inverse reinforcement learning based on the feature vector of the expert and the feature vector of the agent.

[0007] Another aspect of the present invention is an information processing method including: a first vector calculation step of calculating a feature vector of an expert based on time-series changes in past order book information; an action probability estimation step of identifying action probabilities related to a transaction by reinforcement learning using an initial reward function related to the transaction in which a coefficient vector has been initialized; a second vector calculation step of calculating a feature vector of an agent corresponding to the feature vector of the expert based on the action probabilities; and a reward function generation step of generating a reward function by updating the coefficient vector by inverse reinforcement learning based on the feature vector of the expert and the feature vector of the agent.

[0008] Another aspect of the present invention is a program for causing a computer to execute the following steps: a first vector calculation step of calculating a feature vector of an expert based on time-series changes in past order book information; an action probability estimation step of identifying action probabilities related to a transaction by reinforcement learning using an initial reward function related to the transaction in which a coefficient vector has been initialized; a second vector calculation step of calculating a feature vector of an agent corresponding to the feature vector of the expert based on the action probabilities; and a reward function generation step of generating a reward function by updating the coefficient vector by inverse reinforcement learning based on the feature vector of the expert and the feature vector of the agent. [Effects of the Invention]

[0009] According to the present invention, it is possible to perform a market simulation that takes into account market impact. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating a hardware configuration of an information processing device. [Figure 2] FIG. 2 is a diagram illustrating a functional configuration of an information processing device. [Figure 3] 10 is a flowchart showing a learning process. [Figure 4] FIG. 1 is a diagram schematically illustrating board information. [Figure 5] FIG. 10 is an explanatory diagram of a learning process. [Figure 6] This is an explanatory diagram of actions and states. [Figure 7] 1 is a flowchart of a simulation. DETAILED DESCRIPTION OF THE INVENTION

[0011] 1 is a hardware configuration diagram of an information processing device 10 according to this embodiment. The information processing device 10 is a device that performs a market simulation that takes into account market impact, and also a device that learns a model for the simulation. More specifically, the information processing device 10 uses inverse reinforcement learning to estimate reward functions for market participants and performs a market simulation that takes into account the impact of users' stock trading behavior on the market.

[0012] The information processing device 10 includes a processor 101, a main memory device 102, an auxiliary memory device 103, a communication unit 104, and a user interface (UI) unit 105. The processor 101 is a control unit that controls the information processing device 10. The main memory device 102 is a storage device such as a random access memory (RAM) that is used for temporarily storing information and developing programs. The auxiliary memory device 103 is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD) that stores various information and various programs. The communication unit 104 communicates with external devices via a network 30. The UI unit 105 includes an input unit that accepts user operations and a display unit that displays various information.

[0013] 2 is a diagram showing the functional configuration of the information processing device 10. The processor 101 of the information processing device 10 executes programs stored in the auxiliary storage device 103 or the like, thereby functioning as a first vector calculation unit 201, a behavior probability estimation unit 202, a second vector calculation unit 203, a reward function generation unit 204, and a simulation unit 205. Below, the processes described as being executed by the first vector calculation unit 201, the behavior probability estimation unit 202, the second vector calculation unit 203, the reward function generation unit 204, and the simulation unit 205 are actually processes executed by the processor 101.

[0014] The first vector calculation unit 201 calculates the feature vector of an expert (teacher). In this embodiment, all market participants are considered experts. That is, a single-agent machine learning method is applied by regarding trading behavior in the market as the trading behavior of a single expert. The action probability estimation unit 202 estimates the action probability of the agent. The second vector calculation unit 203 calculates the feature vector of the agent. The reward function generation unit 204 generates a reward function by inverse reinforcement learning. The simulation unit 205 performs a simulation using a market simulator obtained by inverse reinforcement learning. Details of the processing of each unit will be described later.

[0015] FIG. 3 is a flowchart showing the learning process of the market simulator. FIG. 4 is a diagram schematically showing the depth of market information used in the learning process. FIG. 5 is an explanatory diagram of the learning process. First, in step S100, the first vector calculation unit 201 reads the history of past stock transactions in the stock market. In the trading history, the actions taken by the expert in response to the state of the depth of market information at a certain time and the sequence of state transitions of the depth of market information due to the actions are called the expert's trajectory. The depth of market information is assumed to be acquired from an external server device via the communication unit 104, for example.

[0016] Next, in step S102, the first vector calculation unit 201 calculates a feature vector of the expert based on time-series changes in the depth of market. At this time, depth of market for a predetermined period of one or more stocks to be processed is referenced. In this embodiment, as shown in FIG. 4, a model is adopted in which only the number of orders at the best bid / ask price is taken into consideration, and the state of depth of market and the order quantity are limited to a maximum of two units. In other words, the depth of market includes only the number of orders at the best bid / ask price, and does not include price information.

[0017] In this embodiment, for convenience of explanation, as shown in FIG. 6 , the state of the depth information is grouped into nine states, with three possible values ​​for the buy and sell quantities at the best bid / ask price (0, 1, 2). Furthermore, actions (stock orders) are grouped into nine possible actions: no order, (sell, buy) × quantity (1, 2) × buy and sell at the best bid / ask price (two possible actions). In this embodiment, the total total quantity is limited to a maximum of two units, so actions that cannot actually occur are not possible. For example, action 3 in state 0 and action 2 in state 1 are not possible, so the cells are left blank. Furthermore, in this embodiment, the state of the depth information is limited to a maximum of two units, and state 5, state 7, and state 8 cannot actually occur, so there are six possible states.

[0018] For the sake of convenience, the maximum number of orders is set to 2 here, but when using actual order numbers, the order numbers may be classified into multiple groups to reduce the amount of calculation. For example, the order numbers may be grouped into four groups: 0, greater than 0 but not exceeding 25% of the maximum number, greater than 25% but not exceeding 75%, and greater than 75%.

[0019] The process of calculating the feature vector of an expert will be described below. As shown in Fig. 5, the first vector calculation unit 201 first obtains steps corresponding to the state change from t=1, 2, ..., N up to t=N based on the order book information for the initial step at time t=0. Here, the case of N=4 will be described as an example. As shown in Fig. 5, for example, if a buy order for quantity 1 is placed at the initial step, the step changes to the first step. The steps change in the same way up to the fourth step.

[0020] Next, the first vector calculation unit 201 calculates one state visit frequency based on the state of each of the five steps. As shown in Fig. 5, the state visit frequency is information indicating the number of times each of nine states (0 to 8) was visited from the initial step to the fourth step. For example, assume that of the five states taken up until t=4, state 1 was visited twice and state 4 was visited three times. In this case, the number of visits to state 1 is two, state 4 is visited three times, and the number of visits to the other states is zero.

[0021] The first vector calculation unit 201 obtains the state visit frequency up to pattern L by referring to different depth of market information. Here, the case where L=20 is taken as an example. That is, in this case, 20 state visit frequencies of patterns 1 to 20 are obtained. Then, the first vector calculation unit 201 calculates the average of the 20 state visit frequencies of patterns 1 to 20 as the expected value of the state visit frequency of the expert, i.e., the feature vector of the expert. In this embodiment, the first vector calculation unit 201 calculates the feature vector of the expert by referring to the state of depth of market information out of the state of depth of market information and the expert's actions (orders) included in the expert's trajectory.

[0022] Returning to FIG. 3, after step S102, the processor 101 advances the process to step S104. The process from step S104 onward is a process for finding a reward function through inverse reinforcement learning. First, in step S104, the action probability estimation unit 202 initializes the coefficient vector θ of the reward function related to stock trading, thereby initializing the reward function. In other words, an initial reward function is set. Next, in step S106, the action probability estimation unit 202 estimates the agent's action probability (strategy) through reinforcement learning, with the coefficient vector θ given. Specifically, the action probability estimation unit 202 finds an action probability that maximizes the state value using the maximum entropy method. In this embodiment, the state value is the expected value of the total reward obtained by an agent taking a given action probability.

[0023] Next, in step S108, the second vector calculation unit 203 calculates the feature vector of the agent. Specifically, the second vector calculation unit 203 first calculates the probability distribution μ t Calculate (s).

number

[0024] Probability μ of state s' t-1 Multiplying (s') by the action probability π(a|s'), then multiplying it by the transition probability P(s|a,s'), and summing a and s', we get the probability distribution μ t (s) is obtained.

[0025] μ0(s) indicates the probability of being in state s at the initial step (t=0), that is, the distribution of the probability of taking each state at the initial step (probability distribution of the initial state). Here, μ0(s) is equal to the proportion of L trajectories at the initial step (t=0) among the expert trajectories read in step S100 that take state s. Then, by calculating this formula sequentially from t=1 to t=T (T=4), the probability distribution μ at each step can be calculated. t The second vector calculation unit 203 then calculates the probability distributions μ from t=0 to t=T (T=4 in this embodiment) obtained. t The agent's feature vector is calculated by summing (s).

[0026] Next, in step S110, the reward function generation unit 204 updates the coefficient vector θ so that the difference between the expert's feature vector obtained in step S102 and the agent's feature vector obtained in step S108 becomes smaller. Specifically, the reward function generation unit 204 updates the coefficient vector θ by using a gradient descent method that performs maximum likelihood estimation from the gradient of the likelihood. Assuming the principle of maximum entropy, the difference between the expert's feature vector and the agent's feature vector becomes the gradient with respect to the coefficient vector θ.

[0027] Next, in step S112, the reward function generation unit 204 determines whether the update of the coefficient vector θ has been repeated the number of epochs. Here, the number of epochs is a predetermined number. If the update of the coefficient vector θ has not been repeated the number of epochs (N in step S112), the reward function generation unit 204 proceeds to step S106. In this case, by repeating the processes of steps S106 to S110, the agent's action probability is estimated again using the updated coefficient vector θ, the agent's feature vector is calculated, and the coefficient vector θ is updated. Note that the learning rate multiplied by the gradient related to the coefficient vector θ may be a fixed value, or may be gradually decreased as the coefficient vector θ is updated.

[0028] When the update of the coefficient vector θ has been repeated the number of epochs (Y in step S112), the reward function generation unit 204 sets the coefficient vector θ at the time of processing as the coefficient vector of the reward function, and ends the processing. As described above, the reward function generation unit 204 can obtain the coefficient vector of the reward function from the action probability of the agent by inverse reinforcement learning.

[0029] In this embodiment, the reward function generation unit 204 repeats updating of the coefficient vector θ the number of epochs. However, as another example, the update may be repeated until the gradient becomes equal to or less than a threshold.

[0030] FIG. 7 is a flowchart showing the inference process. The inference process is a process of performing a simulation using the reward function generated by the learning process. First, in step S200, the simulation unit 205 sets an order selection method (inference method) in accordance with a user operation. The order selection method includes a method that follows the action probability (strategy) and a method that always follows the maximum probability in the strategy. As another example, the order selection method may be preset to one of the methods.

[0031] Next, in step S202, the simulation unit 205 reproduces (simulates) the order book state that reproduces the time series of past stock prices and orders. Specifically, the simulation unit 205 reproduces past stock prices by sequentially applying orders from that time to the order book state at any time in chronological order. Next, in step S204, the simulation unit 205 accepts orders in response to user operations. Next, in step S206, the simulation unit 205 simulates orders in response to orders. Specifically, the simulation unit 205 uses a strategy to place orders in response to the order book state. This changes the order book state. The simulation unit 205 may continue to place orders in response to changes in the order book state, or may accept orders from the user multiple times. However, the accuracy of this depends on the initial state distribution and the number of steps during learning. This completes the processing. Furthermore, by using the reward function generated by the learning process, the user can learn how the market will change depending on their ordering behavior.

[0032] As described above, the information processing system 1 according to this embodiment can generate a market simulator that takes market impact into account by using inverse reinforcement learning, and can further perform simulations by using the market simulator.

[0033] It should be noted that the present invention is not limited to the specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims, such as combining a modified version of one embodiment with another modified version.

[0034] As a modification, the information used as the depth of market information is not limited to that of the embodiment, and the depth of market information may include information on the best bid / ask price. Furthermore, the depth of market information may include prices other than the best bid / ask price and the number of orders. Even in these cases, the depth of market information is grouped in accordance with predetermined conditions based on the price and order number information. This prevents the amount of calculation from increasing. Furthermore, the information on the number of orders at each price may be expressed as a relative value rather than an absolute value, such as a percentage of the total volume. Furthermore, the trading object is not limited to stocks, and may be anything (such as foreign exchange) whose buying and selling transactions are aggregated as depth of market information at an exchange or the like. Furthermore, although gradient descent is used for inverse reinforcement learning in this embodiment, the present invention is not limited to this, and a neural network such as deep learning may also be used.

[0035] Furthermore, the depth record used to calculate the feature vector of an agent may be limited to depth record for a predetermined period. For example, a trading period may be specified by the user, and depth record for the specified period may be used. This allows for the generation of a model that takes into account transactions during the period desired by the user.

[0036] As another example, in calculating the feature vector of an agent, only order book information for a trading period (such as an intraday session) excluding a certain period from the opening of the stock market and a certain period until the closing of the stock market may be used. This is in response to the fact that price movements during the certain period from the opening of the stock market and the certain period until the closing of the stock market are often different from those during the period excluding these periods. In this way, by excluding these periods, a model that is more in line with reality can be generated.

[0037] Furthermore, the above-described devices, programs, and methods may be realized as a single device or as a processing system using components shared by multiple devices, and include various aspects. For example, they may be provided as an information processing system in which the functions of the information processing device described in this embodiment are realized by multiple devices. Furthermore, the functions described in this embodiment as software may be realized by hardware, and modifications may be made as appropriate, such as partly being software and partly being hardware. Furthermore, the program for controlling the device may be provided as a recording medium. Of course, the recording medium for the program may be a magnetic recording medium or a semiconductor memory. Furthermore, the same can be considered for any recording medium developed in the future. [Explanation of symbols]

[0038] 10...information processing system, 101...processor, 102...main storage device, 103...auxiliary storage device, 104...communication unit, 105...UI unit, 201...first vector calculation unit, 202...action probability estimation unit, 203...second vector calculation unit, 204...reward function generation unit, 205...simulation unit

Claims

1. a first vector calculation unit that calculates a feature vector of an expert based on a time-series change in past market depth information; a behavior probability estimation unit that determines behavior probabilities related to a transaction through reinforcement learning using an initial reward function related to the transaction, the coefficient vector of which has been initialized; a second vector calculation unit that calculates a feature vector of an agent corresponding to the feature vector of the expert based on the action probability; a reward function generation unit that generates a reward function by updating the coefficient vector through inverse reinforcement learning based on the feature vector of the expert and the feature vector of the agent; An information processing system comprising:

2. The information processing system according to claim 1 , wherein the behavior probability estimation unit groups the behaviors related to the transaction into a plurality of behaviors in accordance with a predetermined condition, and identifies a behavior probability related to the transaction based on the behaviors.

3. 2. The information processing system according to claim 1, wherein the first vector calculation unit groups the order book information into a plurality of states in accordance with a predetermined condition, and calculates the feature vector of the expert based on the states.

4. 4. The information processing system according to claim 3, wherein the feature vector of the expert is an expected value of a state visit frequency indicating the number of times each state of the board information has been visited.

5. The information processing system according to claim 1 , wherein the first vector calculation unit calculates the feature vector of the expert based on a change in the order book information related to a best bid / ask price.

6. 2. The information processing system according to claim 1, wherein the first vector calculation unit calculates the feature vector of the expert based on changes in the order book information during a trading period excluding a certain period from the opening of the stock market and a certain period until the closing of the stock market.

7. The information processing system according to claim 1 , wherein the first vector calculation unit calculates the feature vector of the expert based on a change in the order book information during a trading period designated by a user.

8. The information processing system according to claim 1 , wherein the first vector calculation unit calculates the feature vector of the expert based on a change in the depth information of one or more predetermined stocks.

9. The information processing system according to claim 1 , further comprising a simulation unit that, when a stock order is received from a user, performs a simulation of stock trading using the reward function.

10. An information processing method executed by an information processing device, a first vector calculation step of calculating a feature vector of an expert based on a time-series change in past market depth information; an action probability estimation step of identifying an action probability related to the transaction by reinforcement learning using an initial reward function related to the transaction in which the coefficient vector has been initialized; a second vector calculation step of calculating a feature vector of an agent corresponding to the feature vector of the expert based on the action probability; a reward function generation step of generating a reward function by updating the coefficient vector through inverse reinforcement learning based on the feature vector of the expert and the feature vector of the agent; An information processing method including:

11. On the computer, a first vector calculation step of calculating a feature vector of an expert based on a time-series change in past market depth information; an action probability estimation step of identifying an action probability related to the transaction by reinforcement learning using an initial reward function related to the transaction in which the coefficient vector has been initialized; a second vector calculation step of calculating a feature vector of an agent corresponding to the feature vector of the expert based on the action probability; a reward function generation step of generating a reward function by updating the coefficient vector through inverse reinforcement learning based on the feature vector of the expert and the feature vector of the agent; A program to execute.

Images