Method for generating nonlinear collaborative financial product alpha factor using reinforcement learning

Abstract

The application discloses a method for generating nonlinear collaborative financial product alpha factors by using reinforcement learning, relates to the fields of financial quantification and reinforcement learning, and comprises the following steps: step 1, outputting a token of BEG, inputting a current environment state as a feature into a policy gradient network, and outputting a next token; step 2, updating the current environment state, inputting the current environment state into the policy gradient network again, and outputting a next token; step 3, repeating step 2 until a complete alpha factor formula is generated; step 4, placing the complete alpha factor formula into an alpha factor pool, assigning a random weight, inputting all factors in the alpha factor pool into a nonlinear neural network for regression training, and determining the weight of each alpha factor; step 5, composing a total alpha factor with a collaborative effect, and taking a mutual information coefficient as a reward function; step 6, updating the policy gradient network according to the reward function by using a Monte Carlo method; and repeating steps 1 to 5 until an effective prediction result is obtained.

Description

Technical Field

[0001] This invention relates to the fields of financial quantitative analysis and reinforcement learning, and more particularly to a method for generating alpha factors for nonlinear collaborative financial products using reinforcement learning. Background Technology

[0002] An alpha factor is a characteristic or indicator used to assess and predict excess returns in the financial product market. Identifying alpha factors can help traders find investment opportunities in the financial product market that outperform market benchmark returns, thereby generating additional profits.

[0003] Reinforcement learning (RL) is a machine learning method that aims to learn how to take actions to maximize cumulative rewards through interaction with the environment. The core idea of ​​reinforcement learning is that an agent learns the optimal policy in a given environment through a trial-and-error process, thereby maximizing its reward in the long run.

[0004] Currently, there are two main methods for generating alpha factors. The first is based on deep neural networks from machine learning. However, deep neural networks require large amounts of data, are overly complex, and have low transparency. Deep learning models are often considered "black box" models, with complex internal mechanisms that are difficult to explain. Transparency and interpretability of trading strategies are particularly important in the financial field, especially under strict regulatory environments. A lack of transparency can lead to reduced trust in model results and decisions, making it difficult to explain the model's behavior and decision-making basis to investors and regulators. The second method is based on formulas. This method has the advantage of being more interpretable than machine learning-based methods, but when multiple independently mined formulaic alpha factors are combined, the performance improvement is not significant, and the synergistic effect between factors is not considered much. Furthermore, formula-based alpha factors are simple in form, and their performance may deteriorate rapidly when applied to financial markets. The search space for effective factor formulas is also very large and difficult to navigate. The most intuitive way to solve this problem is to use genetic programming (GP) to mutate the expression tree to generate new alpha factors. However, due to the complexity of maintaining and mutating a large population, genetic programming algorithms often have poor scalability. It is important to note that mining a set of collaborative alphas at a time is a more difficult problem because it has a much larger search space, the size of which makes it unsolvable by most existing frameworks.

[0005] Therefore, those skilled in the art are dedicated to developing a new method for generating alpha factors for financial products, addressing the aforementioned shortcomings in existing technologies. Summary of the Invention

[0006] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is how to generate interpretable alpha factors for nonlinear financial products with synergistic effects based on reinforcement learning technology.

[0007] To achieve the above objectives, this invention provides a method for generating alpha factors for nonlinear collaborative financial products using reinforcement learning, characterized in that the method includes the following steps:

[0008] Step 1: The output sequence indicator indicates the start of generating an alpha factor token, so that the current environment state becomes the alpha factor token containing a BEG. The current environment state is then used as a feature input to the policy gradient network, which outputs the next token.

[0009] Step 2: Update the current environment state to the alpha factor formula containing two tokens, and then input the current environment state into the policy gradient network. The policy gradient network will then output the next token.

[0010] Step 3: Repeat step 2 until a complete alpha factor expression is generated. Then, the policy gradient network outputs a token representing the end of the SEP, indicating the generation of a complete alpha factor expression.

[0011] Step 4: Place the generated complete alpha factor expression into the alpha factor pool and assign a randomly initialized weight; then use all factors in the alpha factor pool as features and input them into a nonlinear neural network for regression training. The loss function is the mean squared error of the output of the nonlinear neural network and the future price trend; after training, determine the weight of each alpha factor in the alpha factor pool.

[0012] Step 5: After the nonlinear neural network completes training, each alpha factor is multiplied by its respective weight to form a total alpha factor with synergistic effect. The mutual information coefficient between the total alpha factor and the future price trend is used as the reward function of the PPO algorithm, which is also the objective function. If the mutual information coefficient is close to 1, it means that the policy gradient network generates a good alpha factor, thus increasing the probability of the policy gradient network taking the corresponding action. If the mutual information coefficient is close to 0, it means that the policy gradient network generates a bad alpha factor, thus decreasing the probability of the policy gradient network taking the corresponding action.

[0013] Step 6: Update the policy gradient network using the Monte Carlo method according to the reward function; after the update, the policy gradient network will repeat steps 1 to 5 in a loop until a result that meets a preset threshold is obtained, thereby obtaining the policy gradient network and the alpha factor pool with synergistic effect.

[0014] Furthermore, before outputting a new token, the policy gradient network uses a masking function to remove invalid tokens, allowing only valid tokens to be output.

[0015] Furthermore, the valid tokens output by the policy gradient network will be placed into the token pool.

[0016] Furthermore, the token pool includes operators, original characteristics of financial products, constants, time spans, and sequence indicators; wherein, the operators include unary operators or binary operators, and the sequence indicators indicate the start and end of the alpha factor expression.

[0017] Furthermore, the policy gradient network is a multilayer perceptron.

[0018] Furthermore, the current environment state is the portion of the alpha factor formula that has already been generated.

[0019] Furthermore, if the number of alpha factors in the alpha pool exceeds the limit, the alpha factor with the smallest weight is removed.

[0020] Furthermore, the nonlinear neural network consists of a fully connected layer and a ReLU activation function, wherein the weights of the fully connected layer correspond to the weights of each alpha factor in the alpha pool.

[0021] Furthermore, the mutual information coefficient is the Pearson correlation coefficient between the total alpha factor and future price trends. Specifically:

[0022]

[0023] in, It is a factor model for the first The predicted value given for a financial product on a certain day; It is the mean of the predicted values ​​of all financial products on a given day given by the factor model; It refers to the actual price trend of financial products, specifically the... The actual rate of return of a financial product on a certain day; It is the average of the actual returns of all financial products on a given day; This is the total number of all financial products; It is an index of a certain financial product.

[0024] Furthermore, step 6 involves calculating the alpha factor. With yield Average mutual information coefficient between To measure the effectiveness of the alpha factor in predicting the price trends of financial products, specifically:

[0025]

[0026] in, It is the value of the alpha factor; It is the actual rate of return.

[0027] The method for generating alpha factors for nonlinear collaborative financial products using reinforcement learning provided by this invention has at least the following technical effects:

[0028] The technical solution proposed in this invention generates alpha factor expressions based on the PPO algorithm, making the generated factors interpretable. It also leverages the advantage of PPO's large action space to efficiently search for suitable action spaces, thereby improving generation efficiency. By using an alpha factor pool and corresponding weights to generate nonlinear alpha factors with synergistic effects, it can take into account both the synergistic effects between single factors and multiple factors.

[0029] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0030] Figure 1 This is a token pool example of a preferred embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of the workflow of a preferred embodiment of the present invention;

[0032] Figure 3 This is a specific factor pool pattern of a preferred embodiment of the present invention. Detailed Implementation

[0033] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0034] To generate interpretable alpha factors for synergistic nonlinear financial products, the technical solution provided by this invention uses the Proximal Policy Optimization (PPO) algorithm in reinforcement learning. The algorithm includes the following components: a token pool, the current environment state, a policy gradient network, a mask function, an alpha factor pool, a nonlinear neural network, and an objective function.

[0035] in,

[0036] Token pool: The pool of symbols necessary for generating the alpha factor formula, including operators (divided into unary and binary operators), original features of the financial product, constants, time (TimeDelta), and sequence indicators (indicating the start and end of the formula). All operators used are listed in the table below. Figure 1 Listed in.

[0037] Current environment state: In reinforcement learning, the current environment state is used as a feature input to the neural network. The current environment state is the partially generated alpha factor formula. In this embodiment of the invention, the alpha factor formula consists of twenty tokens, and the current environment state is the partially generated alpha factor formula consisting of zero to nineteen tokens.

[0038] Policy gradient network: The input of the policy gradient network is the current environment state, and the output is the corresponding action, that is, to take the next token from the token pool. The policy gradient network is specifically a multilayer perceptron.

[0039] Masking function: Based on the specific form of the previous token, it filters out some invalid tokens.

[0040] Alpha factor pool: The factors generated by the policy gradient network are placed into the factor pool to form the alpha factor pool. Each factor in the factor pool corresponds to a weight. If the factor pool is full, the factor with the smallest weight is removed.

[0041] Nonlinear network: It consists of a fully connected layer and a ReLU activation function. The weights of the fully connected layer correspond to the weights of each factor in the alpha pool.

[0042] Objective function: The goal of the PPO algorithm is to maximize the objective function. One indicator for measuring the quality of the alpha factor is the mutual information coefficient. The goal of the PPO algorithm is to maximize the mutual information coefficient between the synergistic factors in the alpha factor pool and the price trend of financial products. Therefore, the objective function is the mutual information coefficient between the synergistic factors in the alpha factor pool and the price trend of financial products.

[0043] Example 1

[0044] This invention provides a method for generating alpha factors for nonlinear collaborative financial products using reinforcement learning, comprising the following steps:

[0045] Step 1: The output sequence indicator indicates the start of generating alpha factor tokens, making the current environment state an alpha factor token containing a BEG. The current environment state is then used as a feature input to the policy gradient network of the PPO algorithm. The policy gradient network will output the next action, i.e., the next token.

[0046] Step 2: Update the current environment state to an alpha factor with two tokens, and then input the current environment state into the policy gradient network. The policy gradient network will output the next token based on the input environment features. In the middle, a mask function is used to eliminate illegal token selections and only allow the network to output legal and valid token symbols. Then, the environment state is updated again.

[0047] Step 3: Repeat step 2 in a loop until a complete alpha factor expression is generated. The policy gradient network outputs a token representing the end of the SEP, indicating the generation of a complete alpha factor expression. In this embodiment of the invention, twenty tokens are used to form a complete alpha factor expression.

[0048] Step 4: After generating a complete alpha factor expression, this embodiment of the invention does not immediately evaluate the quality of this factor. Instead, it puts the generated complete alpha factor expression into the alpha factor pool. The alpha factor pool stores some alpha factors generated by the previous policy gradient network, each factor corresponding to a randomly initialized weight. After the newly generated complete alpha factor expression is put into the pool, it is assigned a randomly initialized weight. Then, all the factors in the alpha factor pool are used as features and input into a nonlinear neural network for regression training. The loss function is the mean squared error between the output of the nonlinear neural network and the future price trend. That is, the synergistic effect of all the alpha factors in the training factor pool can indicate the future price trend to a certain extent. After training is completed, the weight of each alpha factor in the alpha factor pool is determined. The weight measures the importance of each alpha factor.

[0049] Step 5: After the nonlinear neural network completes training, each alpha factor is multiplied by its respective weight to form a total alpha factor with synergistic effects. The mutual information coefficient between the total alpha factor and the future price trend is used as the reward function of the PPO algorithm, which is also the objective function. If the mutual information coefficient is close to 1, it means that the policy gradient network has generated a good alpha factor, and the probability of the policy gradient network taking corresponding actions is increased according to gradient ascent and backpropagation. If the mutual information coefficient is close to 0, it means that the policy gradient network has generated a bad alpha factor, and the probability of the policy gradient network taking corresponding actions is reduced according to gradient ascent and backpropagation. Only after the nonlinear neural network is trained can a reward function (the Pearson correlation coefficient between the synergistic alpha factor and the future price trend) be obtained, thus completing one Monte Carlo episode.

[0050] Step 6: Update the policy gradient network using the Monte Carlo method based on the final reward function. After the update, the policy gradient network will continue to generate the next alpha factor. After generation, it will be put into the alpha factor pool to form a new Monte Carlo episode. The nonlinear neural network will be trained again to obtain the reward function, and then the policy gradient network will be updated. That is, steps 1 to 5 are repeated until a result that meets the preset threshold is obtained, thus obtaining a policy gradient network that can generate an alpha factor with good performance and an alpha factor pool with synergistic effect.

[0051] The specific flowchart is attached. Figure 2 As shown in the attached diagram, the alpha factor pool has the following format. Figure 3 As shown.

[0052] Example 2

[0053] Based on Example 1, the valid tokens output by the policy gradient network will be placed into the token pool.

[0054] Specifically, the token pool includes operators, original characteristics of financial products, constants, time spans, and sequence indicators; among them, operators include unary operators or binary operators, and sequence indicators indicate the start and end of the alpha factor expression.

[0055] Specifically, the policy gradient network is a multilayer perceptron.

[0056] Example 3

[0057] Based on Example 1 or 2, the current environment state is the partial alpha factor formula that has been generated so far.

[0058] Specifically, if the number of alpha factors in the alpha pool exceeds the limit, the alpha factor with the smallest weight is removed.

[0059] Specifically, a nonlinear neural network consists of a fully connected layer and a ReLU activation function, where the weights of the fully connected layer correspond to the weights of each alpha factor in the alpha pool.

[0060] Specifically, the mutual information coefficient is the Pearson correlation coefficient between the total alpha factor and future price trends. Specifically:

[0061]

[0062] in, It is a factor model for the first The predicted value given for a financial product on a certain day; It is the mean of the predicted values ​​of all financial products on a given day given by the factor model; It refers to the actual price trend of financial products, specifically the... The actual rate of return of a financial product on a certain day; It is the average of the actual returns of all financial products on a given day; This is the total number of all financial products; It is an index of a certain financial product.

[0063] Specifically, step 6 involves calculating the alpha factor. With yield Average mutual information coefficient between To measure the effectiveness of the alpha factor in predicting the price trends of financial products, specifically:

[0064]

[0065] in, It is the value of the alpha factor; It is the actual rate of return.

[0066] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for generating alpha factors for nonlinear collaborative financial products using reinforcement learning, characterized in that, The method includes the following steps: Step 1: The output sequence indicator indicates the start of generating an alpha factor token, so that the current environment state becomes the alpha factor token containing a BEG. The current environment state is then used as a feature input to the policy gradient network, which outputs the next token. Step 2: Update the current environment state to the alpha factor formula containing two tokens, and then input the current environment state into the policy gradient network. The policy gradient network will then output the next token. Step 3: Repeat step 2 until a complete alpha factor expression is generated. Then, the policy gradient network outputs a token representing the end of the SEP, indicating the generation of a complete alpha factor expression. Step 4: Place the generated complete alpha factor expression into the alpha factor pool and assign a randomly initialized weight; then use all factors in the alpha factor pool as features and input them into a nonlinear neural network for regression training. The loss function is the mean squared error of the output of the nonlinear neural network and the future price trend; after training, determine the weight of each alpha factor in the alpha factor pool. Step 5: After the nonlinear neural network completes training, each alpha factor is multiplied by its respective weight to form a total alpha factor with synergistic effect. The mutual information coefficient between the total alpha factor and the future price trend is used as the reward function of the PPO algorithm, which is also the objective function. If the mutual information coefficient is close to 1, it means that the policy gradient network generates a good alpha factor, thus increasing the probability of the policy gradient network taking the corresponding action. If the mutual information coefficient is close to 0, it means that the policy gradient network generates a bad alpha factor, thus decreasing the probability of the policy gradient network taking the corresponding action. Step 6: Update the policy gradient network using the Monte Carlo method according to the reward function; after the update, the policy gradient network will repeat steps 1 to 5 in a loop until a result that meets a preset threshold is obtained, thereby obtaining the policy gradient network and the alpha factor pool with synergistic effect.

2. The method for generating alpha factors for nonlinear collaborative financial products using reinforcement learning as described in claim 1, characterized in that, Before outputting a new token, the policy gradient network uses a masking function to remove invalid tokens, allowing only valid tokens to be output.

3. The method for generating alpha factors for nonlinear collaborative financial products using reinforcement learning as described in claim 1, characterized in that, The valid tokens output by the policy gradient network will be placed into the token pool.

4. The method for generating alpha factors for nonlinear collaborative financial products using reinforcement learning as described in claim 3, characterized in that, The token pool includes operators, original characteristics of financial products, constants, time spans, and sequence indicators; wherein, the operators include unary operators or binary operators, and the sequence indicators indicate the start and end of the alpha factor expression.

5. The method for generating alpha factors for nonlinear collaborative financial products using reinforcement learning as described in claim 3, characterized in that, The policy gradient network is a multilayer perceptron.

6. The method for generating alpha factors for nonlinear collaborative financial products using reinforcement learning as described in claim 1, characterized in that, The current environment state refers to the portion of the alpha factor formula that has already been generated.

7. The method for generating alpha factors for nonlinear collaborative financial products using reinforcement learning as described in claim 1, characterized in that, If the number of alpha factors in the alpha pool exceeds the limit, the alpha factor with the smallest weight is removed.

8. The method for generating alpha factors for nonlinear collaborative financial products using reinforcement learning as described in claim 1, characterized in that, The nonlinear neural network consists of a fully connected layer and a ReLU activation function, wherein the weights of the fully connected layer correspond to the weights of each alpha factor in the alpha pool.

9. The method for generating alpha factors for nonlinear collaborative financial products using reinforcement learning as described in claim 1, characterized in that, The mutual information coefficient is the Pearson correlation coefficient between the total alpha factor and future price trends. Specifically: in, It is a factor model for the first The predicted value given for a financial product on a certain day; It is the mean of the predicted values ​​of all financial products on a given day given by the factor model; It refers to the actual price trend of financial products, specifically the... The actual rate of return of a financial product on a certain day; It is the average of the actual returns of all financial products on a given day; This is the total number of all financial products; It is an index of a certain financial product.

10. The method for generating alpha factors for nonlinear collaborative financial products using reinforcement learning as described in claim 1, characterized in that, Step 6 involves calculating the alpha factor. With yield Average mutual information coefficient between To measure the effectiveness of the alpha factor in predicting the price trends of financial products, specifically: in, It is the value of the alpha factor; It is the actual rate of return.

Images