Method for generating nonlinear collaborative financial product alpha factor by using reinforcement learning

Abstract

The invention discloses a method for generating a nonlinear collaborative financial product alpha factor by using reinforcement learning, and relates to the field of financial quantification and reinforcement learning, and the method comprises the steps: 1, outputting a token of BEG, inputting a current environment state as a feature into a strategy gradient network, and outputting a next token; step 2, updating a current environment state, inputting the current environment state to the strategy gradient network, and outputting a next token; step 3, repeating the step 2 until a complete alpha factor formula is generated; step 4, putting the complete alpha factor expression into an alpha factor pool, and distributing random weights; inputting all factors in the alpha factor pool into a nonlinear neural network for regression training, and determining the weight of each alpha factor; 5, forming a total alpha factor with a synergistic effect, and taking a mutual information coefficient as a reward function; step 6, updating the strategy gradient network by using a Monte Carlo method according to the reward function; and repeating the steps 1-5 until an effective prediction result is obtained.

Description

Technical Field

[0001] The present invention relates to the fields of financial quantification and reinforcement learning, and in particular to a method for generating an alpha factor of a nonlinear collaborative financial product by using reinforcement learning. Background Art

[0002] The alpha factor refers to a characteristic or indicator used to evaluate and predict excess returns in the financial product market. Finding the alpha factor can help traders find investment opportunities in the financial product market that exceed the market benchmark return rate, thereby obtaining additional returns.

[0003] Reinforcement Learning (RL) is a machine learning method that aims to learn how to take actions to maximize cumulative rewards through interaction with the environment. The core idea of ​​RL is that the agent learns the best policy through trial and error in a given environment so that it can obtain the highest possible reward in the long run.

[0004] There are currently two mainstream methods for generating alpha factors. The first is a method based on machine learning deep networks. However, deep neural networks require a large amount of data, and the models are too complex and have low transparency. Deep learning models are usually considered "black box" models. Their internal mechanisms are complex and difficult to explain. The transparency and explainability of trading strategies are particularly important in the financial field, especially in the case of a strict regulatory environment. Lack of transparency may lead to reduced trust in model results and decisions, and it is difficult to explain the behavior of the model and the basis for decision-making to investors and regulators. The second is a formula-based method. This method has the advantage of being more interpretable than the machine learning-based method, but when multiple independently mined formulaic alpha factors are combined, the effect is not greatly improved, and there is not much consideration for the synergy between factors; and the formula-based alpha factor is simple in form, and its performance may deteriorate rapidly when used in the financial market, and the search space for effective factor formulas is very large and difficult to navigate. The most intuitive way to solve this problem is to use genetic programming (GP) to mutate the expression tree to generate new alpha factors. However, due to the complexity of maintaining and mutating a large population, genetic programming algorithms often have poor scalability. It is important to note that mining a set of synergistic alphas at once is a much harder problem as it has a much larger search space, the size of which makes it unaffordable for most existing frameworks.

[0005] Therefore, technicians in this field are committed to developing a new method for generating alpha factors of financial products to solve the above-mentioned defects in the prior art. Summary of the invention

[0006] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to generate an interpretable synergistic nonlinear financial product alpha factor based on reinforcement learning technology.

[0007] To achieve the above object, the present invention provides a method for generating an alpha factor of a nonlinear collaborative financial product using reinforcement learning, wherein the method comprises the following steps: Step 1: Output the token in the sequence indicator indicating the start of generating the alpha factor, so that the current environment state becomes the alpha factor containing a BEG token, and input the current environment state as a feature to the policy gradient network, and the policy gradient network will output the next token; Step 2: Update the current environment state to the alpha factor containing two tokens, and then input the current environment state to the policy gradient network, which will output the next token. Step 3: Repeat step 2 until a complete alpha factor expression is generated, and the policy gradient network outputs a token indicating the end of the SEP, which represents the generation of a complete alpha factor expression; Step 4: Put the generated complete alpha factor expression into the alpha factor pool and assign a randomly initialized weight; then use all factors in the alpha factor pool as features and input them into a nonlinear neural network for regression training, where the loss function is the mean square error between the output of the nonlinear neural network and the future price trend; after the training is completed, determine the weight of each alpha factor in the alpha factor pool; Step 5: After the nonlinear neural network completes training, each alpha factor is multiplied by its own weight to form a total alpha factor with a synergistic effect, and the total alpha factor and the mutual information coefficient of the future price trend are used as the reward function of the PPO algorithm, that is, the objective function; if the mutual information coefficient of the two is close to 1, it means that the policy gradient network generates a good alpha factor, and the probability of the policy gradient network taking corresponding actions is increased; if the mutual information coefficient of the two is close to 0, it means that the policy gradient network generates a bad alpha factor, and the probability of the policy gradient network taking corresponding actions is reduced; Step 6: According to the reward function, the policy gradient network is updated using the Monte Carlo method; after the update, the policy gradient network will repeat steps 1 to 5, repeatedly, until a result that meets a preset threshold is obtained, thereby obtaining a policy gradient network and a synergistic alpha factor pool.

[0008] Furthermore, before outputting a new token, the policy gradient network uses a mask function to remove illegal tokens in the middle, allowing only legal and valid tokens to be output.

[0009] Furthermore, the legal and valid token output by the policy gradient network will be placed in the token pool.

[0010] Furthermore, the token pool includes an operator, original characteristics of a financial product, a constant, a time span, and a sequence indicator; wherein the operator includes a unary operator or a binary operator, and the sequence indicator indicates the start and end of the alpha factor formula.

[0011] Furthermore, the policy gradient network is a multi-layer perceptron.

[0012] Furthermore, the current environment state is the part of the alpha factor formula that has been generated so far.

[0013] Furthermore, if the number of alpha factors in the alpha pool exceeds a limit, the alpha factor with the smallest weight is eliminated.

[0014] Furthermore, the nonlinear neural network is composed of a fully connected layer and a ReLU activation function, wherein the weight of the fully connected layer corresponds to the weight of each alpha factor in the alpha pool.

[0015] Furthermore, the mutual information coefficient is the Pearson correlation coefficient between the total alpha factor and the future price trend. , specifically: in, The factor model is The predicted value of a financial product on a certain day; It is the mean of the predicted values ​​given by the factor model for all financial products on a certain day; It is the actual price trend of financial products, especially the The actual rate of return of a financial product on a certain day; It is the average of the actual returns of all financial products on a certain day; is the number of all financial products; It is the index of a certain financial product.

[0016] Furthermore, in step 6, the alpha factor is calculated With yield The average mutual information coefficient between To measure the effectiveness of the alpha factor in predicting the price trend of financial products, specifically: in, is the value of the alpha factor; is the actual rate of return.

[0017] The method for generating the alpha factor of nonlinear collaborative financial products by using reinforcement learning provided by the present invention has at least the following technical effects: The technical solution proposed in the present invention is to generate an alpha factor expression based on the PPO algorithm, so that the generated factor is interpretable, and to utilize the advantage of the large-scale action space of PPO to efficiently search for a suitable action space and improve the generation efficiency. The alpha factor pool and the corresponding weights are used to generate a synergistic nonlinear alpha factor, which can take into account the synergy between a single factor and multiple factors.

[0018] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a token pool example of a preferred embodiment of the present invention; Figure 2 It is a schematic diagram of the working process of a preferred embodiment of the present invention; Figure 3 This is a specific factor pool pattern of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0021] In order to generate an interpretable synergistic nonlinear financial product alpha factor, the technical solution provided by the present invention uses the proximal policy optimization (PPO) algorithm in reinforcement learning. The algorithm includes the following components: token pool, current environment state, policy gradient network, mask function, alpha factor pool, nonlinear neural network, and objective function.

[0022] in, Token pool: The pool of components necessary to generate the alpha factor formula, including operators (divided into unary operators and binary operators), original features of financial products (Features), constants (Constants), time (TimeDelta), and sequence indicators (Sequence Indicator, indicating the start and end of the formula). All operators used are in Figure 1 Listed in.

[0023] Current environment state: In reinforcement learning, the current environment state is used as a feature input to the neural network. The current environment state is the partial alpha factor formula that has been generated so far. The alpha factor formula in the embodiment of the present invention is composed of twenty tokens, and the current environment state is the partial alpha factor formula composed of zero to nineteen tokens that have been generated.

[0024] Policy gradient network: The input of the policy gradient network is the current environment state, and the output is the corresponding action, that is, taking the next token from the token pool. The policy gradient network is specifically a multi-layer perceptron.

[0025] Mask function: According to the specific form of the previous token, some illegal token selections are blocked.

[0026] Alpha factor pool: The factors finally generated by the policy gradient network are placed into the factor pool to form the alpha factor pool. Each factor in the factor pool corresponds to a weight. If the factor pool is full, the factor with the smallest weight is removed.

[0027] Nonlinear network: It consists of a fully connected layer and a ReLU activation function. The weight of the fully connected layer corresponds to the weight of each factor in the alpha pool.

[0028] Objective function: The goal of the PPO algorithm is to maximize the objective function. An indicator to measure the quality of the alpha factor is the mutual information coefficient. The goal of the PPO algorithm is to maximize the mutual information coefficient between the synergy factor of the alpha factor pool and the price trend of financial products. Therefore, the objective function is the mutual information coefficient between the synergy factor of the alpha factor pool and the price trend of financial products.

[0029] Example 1 The embodiment of the present invention provides a method for generating an alpha factor of a nonlinear collaborative financial product by using reinforcement learning, comprising the following steps: Step 1: Output the token in the sequence indicator that indicates the start of generating the alpha factor, so that the current environment state becomes the alpha factor containing a BEG token, and input the current environment state as a feature to the policy gradient network of the PPO algorithm. The policy gradient network will output the next action, that is, the next token. Step 2: Update the current environment state to an alpha factor containing two tokens, and then input the current environment state to the policy gradient network. The policy gradient network will output the next token again based on the input of the environment features. In the middle, a mask function is used to eliminate illegal token selections, allowing the network to output only legal and valid token symbols, and then continue to update the environment state. Step 3, repeat step 2, repeatedly, until a complete alpha factor expression is generated, then the policy gradient network outputs a token indicating the end of the SEP, representing the generation of a complete alpha factor expression; in this embodiment of the present invention, twenty tokens are used to form a complete alpha factor expression; Step 4. After generating a complete alpha factor formula, the embodiment of the present invention will not immediately evaluate the quality of this factor, but will put the generated complete alpha factor expression into the alpha factor pool. The alpha factor pool stores some alpha factors generated by the previous policy gradient network, and each factor corresponds to a randomly initialized weight. After the newly generated complete alpha factor expression enters the pool, it will be assigned a randomly initialized weight; then all the factors in the alpha factor pool are used as features and input into the nonlinear neural network for regression training. The loss function is the mean square error of the output of the nonlinear neural network and the future price trend, that is, the synergy of all alpha factors in the training factor pool can indicate the future price trend to a certain extent; after the training is completed, the weight of each alpha factor in the alpha factor pool is determined, and the weight measures the importance of each alpha factor; Step 5. After the nonlinear neural network is trained, each alpha factor is multiplied by its own weight to form a total alpha factor with a synergistic effect, and the total alpha factor and the mutual information coefficient of the future price trend are used as the reward function of the PPO algorithm, that is, the objective function; if the mutual information coefficient of the two is close to 1, it means that the policy gradient network generates a good alpha factor, and according to the gradient ascent back propagation, the probability of the policy gradient network taking corresponding actions is increased; if the mutual information coefficient of the two is close to 0, it means that the policy gradient network generates a bad alpha factor, and according to the gradient ascent back propagation, the probability of the policy gradient network taking corresponding actions is reduced; only after the nonlinear neural network is trained can a reward function (the Pearson correlation coefficient of the synergistic alpha factor and the future price trend) be obtained, so that a Monte Carlo episode is completed; Step 6. According to the final reward function, the Monte Carlo method is used to update the policy gradient network. After the update, the policy gradient network will continue to generate the next alpha factor, and then put it into the alpha factor pool to form a new Monte Carlo episode. The nonlinear neural network is trained again to obtain the reward function, and then the policy gradient network is updated. That is, steps 1 to 5 are repeated repeatedly until a result that meets the preset threshold is obtained, thereby obtaining a policy gradient network that can generate an alpha factor with better effect and a synergistic alpha factor pool.

[0030] The specific flow chart is as follows: Figure 2 The alpha factor pool is shown in the attached Figure 3 shown.

[0031] Example 2 Based on Example 1, the legal and valid token output by the policy gradient network will be put into the token pool.

[0032] In particular, the token pool includes operators, original features of financial products, constants, time spans, and sequence indicators; wherein the operators include unary operators or binary operators, and the sequence indicators indicate the start and end of the alpha factor formula.

[0033] In particular, the policy gradient network is a multi-layer perceptron.

[0034] Example 3 Based on the embodiment 1 or 2, the current environment state is the partial alpha factor formula that has been generated so far.

[0035] In particular, if the number of alpha factors in the alpha pool exceeds a limit, the alpha factor with the smallest weight is discarded.

[0036] In particular, the nonlinear neural network consists of a fully connected layer and a ReLU activation function, where the weights of the fully connected layer correspond to the weights of each alpha factor in the alpha pool.

[0037] Specifically, the mutual information coefficient is the Pearson correlation coefficient between the total alpha factor and the future price trend. , specifically: in, The factor model is The predicted value of a financial product on a certain day; It is the mean of the predicted values ​​given by the factor model for all financial products on a certain day; It is the actual price trend of financial products, especially the The actual rate of return of a financial product on a certain day; It is the average of the actual returns of all financial products on a certain day; is the number of all financial products; It is the index of a certain financial product.

[0038] Specifically, step 6 is done by calculating the alpha factor With yield The average mutual information coefficient between To measure the effectiveness of the alpha factor in predicting the price trend of financial products, specifically: in, is the value of the alpha factor; is the actual rate of return.

[0039] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for generating alpha factors of nonlinear collaborative financial products using reinforcement learning, characterized in that: The method comprises the following steps: Step 1: Output the token in the sequence indicator indicating the start of generating the alpha factor, so that the current environment state becomes the alpha factor containing a BEG token, and input the current environment state as a feature to the policy gradient network, and the policy gradient network will output the next token; Step 2: Update the current environment state to the alpha factor containing two tokens, and then input the current environment state to the policy gradient network, which will output the next token. Step 3: Repeat step 2 until a complete alpha factor expression is generated, and the policy gradient network outputs a token indicating the end of the SEP, which represents the generation of a complete alpha factor expression; Step 4: Put the generated complete alpha factor expression into the alpha factor pool and assign a randomly initialized weight; then use all factors in the alpha factor pool as features and input them into a nonlinear neural network for regression training, where the loss function is the mean square error between the output of the nonlinear neural network and the future price trend; after the training is completed, determine the weight of each alpha factor in the alpha factor pool; Step 5: After the nonlinear neural network completes training, each alpha factor is multiplied by its own weight to form a total alpha factor with a synergistic effect, and the total alpha factor and the mutual information coefficient of the future price trend are used as the reward function of the PPO algorithm, that is, the objective function; if the mutual information coefficient of the two is close to 1, it means that the policy gradient network generates a good alpha factor, and the probability of the policy gradient network taking corresponding actions is increased; if the mutual information coefficient of the two is close to 0, it means that the policy gradient network generates a bad alpha factor, and the probability of the policy gradient network taking corresponding actions is reduced; Step 6: According to the reward function, the policy gradient network is updated using the Monte Carlo method; after the update, the policy gradient network will repeat steps 1 to 5, repeatedly, until a result that meets a preset threshold is obtained, thereby obtaining a policy gradient network and a synergistic alpha factor pool.

2. The method for generating alpha factors of nonlinear collaborative financial products using reinforcement learning as claimed in claim 1, characterized in that: Before outputting a new token, the policy gradient network will use a mask function to remove illegal tokens in the middle, allowing only legal and valid tokens to be output.

3. The method for generating alpha factors of nonlinear collaborative financial products using reinforcement learning as claimed in claim 1, characterized in that: The legal and valid tokens output by the policy gradient network will be placed in the token pool.

4. The method for generating alpha factors of nonlinear collaborative financial products using reinforcement learning as claimed in claim 3, characterized in that: The token pool includes an operator, original features of a financial product, a constant, a time span, and a sequence indicator; wherein the operator includes a unary operator or a binary operator, and the sequence indicator indicates the start and end of the alpha factor formula.

5. The method for generating alpha factors of nonlinear collaborative financial products using reinforcement learning as claimed in claim 3, characterized in that: The policy gradient network is a multi-layer perceptron.

6. The method for generating alpha factors of nonlinear collaborative financial products using reinforcement learning as claimed in claim 1, characterized in that: The current environment state is the part of the alpha factor formula that has been generated so far.

7. The method for generating alpha factors of nonlinear collaborative financial products using reinforcement learning as claimed in claim 1, characterized in that: If the number of alpha factors in the alpha pool exceeds the limit, the alpha factor with the smallest weight is discarded.

8. The method for generating alpha factors of nonlinear collaborative financial products using reinforcement learning as claimed in claim 1, characterized in that: The nonlinear neural network is composed of a fully connected layer and a ReLU activation function, wherein the weight of the fully connected layer corresponds to the weight of each alpha factor in the alpha pool.

9. The method for generating alpha factors of nonlinear collaborative financial products using reinforcement learning as claimed in claim 1, characterized in that: The mutual information coefficient is the Pearson correlation coefficient between the total alpha factor and the future price trend. , specifically: in, The factor model is The predicted value of a financial product on a certain day; It is the mean of the predicted values ​​given by the factor model for all financial products on a certain day; It is the actual price trend of financial products, especially the The actual rate of return of a financial product on a certain day; It is the average of the actual returns of all financial products on a certain day; is the number of all financial products; It is the index of a certain financial product.

10. The method for generating alpha factors of nonlinear collaborative financial products using reinforcement learning according to claim 1, characterized in that: Step 6 is to calculate the alpha factor With yield The average mutual information coefficient between To measure the effectiveness of the alpha factor in predicting the price trend of financial products, specifically: in, is the value of the alpha factor; is the actual rate of return.

Images