A DQN Database Index Recommendation Method Based on Tree-Structured Invalidation Action Masking

Abstract

The present invention discloses a DQN database index recommendation method based on tree-shaped invalid action masking, which can recommend an optimal index design scheme for database design and operation and maintenance personnel given the database table structure, data, and query load. Based on the traditional DQN, first, Double DQN and Dueling DQN technologies are used for improvement. Secondly, according to the leftmost principle of composite indexes, the recommended indexes are selected, new columns are added on their right to generate new composite indexes, and a tree structure is used to count the candidate indexes. When training DQN, the method of invalid action masking is adopted to enable the reinforcement learning agent to only select valid indexes, so as to achieve automatic optimal index configuration and improve the efficiency of the reinforcement learning model.

Description

Technical Field

[0001] The present invention belongs to the technical field of databases, and particularly relates to a method for recommending database indexes. Background Art

[0002] An index is a necessary data structure created to improve the query speed of a database. Through an index, the specified data can be directly located without scanning the entire table. Traditional database administrators often manually design and adjust indexes based on relevant theoretical knowledge and experience, consuming time and manpower, and it is also difficult to ensure the establishment of optimal indexes. The algorithms for automatic index selection first appeared in the 1970s, and the implementation methods and complexities of various methods are different. Many commercial database management systems are applying these methods, such as DTA, DB2Advis, and AutoAdmin.

[0003] In traditional automatic index recommendation technologies, according to the different implementation methods of the algorithms, they can be roughly divided into the following categories:

[0004] 1) Starting from an empty index set, gradually add indexes (AutoAdmin, DB2Advis, DTA, Extend). Traditional methods such as Extend and DTA can often find index configurations close to the optimal ones.

[0005] 2) Starting from a large index set, gradually reduce indexes.

[0006] 3) Linear programming (Cophy).

[0007] 4) Transform it into a 0-1 knapsack problem for solution.

[0008] In recent years, technologies such as genetic algorithms and machine learning have also been gradually applied in the field of database index recommendation. Sharma et al. first applied machine learning to index recommendation in 2018. Subsequently, Sadri et al. used a reinforcement learning-based method to recommend optimal indexes in the database replication scenario. Basu et al. proposed a general reinforcement learning method for database tuning. Welborn et al. trained a reinforcement learning agent using the Sinkhorn policy gradient algorithm in a structured action space to perform the task of index selection.

[0009] Lan et al. used heuristic rules to generate some candidate indexes and adopted the method of prioritized experience replay to train a DQN (Deep Q-network) to find the optimal index combination in the candidate set. However, the heuristic rules pre-narrowed the search space and excluded some potentially better indexes before the search. Licks et al. proposed SMARTIX, also based on reinforcement learning, for index recommendation, but it can only recommend single-column indexes. Perera et al. used multi-armed bandits for online index recommendation, which has the advantages of fast convergence, easy implementation, and a mathematical guarantee of the worst-case scenario compared to the optimal choice, compared with traditional reinforcement learning methods. There are also studies on index recommendation using deep learning other than reinforcement learning. For example, Ding et al. transformed the index selection problem into a classification task of choosing a better one from two index settings, represented the query plan as features and input them into a network, and trained a deep neural network for classification.

[0010] When using reinforcement learning for index recommendation, it can be regarded as a phased reinforcement learning task, and the interaction between the agent and the environment is divided into individual phases (episodes), such as Figure 1 . In each step of a phase, the agent (index recommendation tool) interacts with the environment (database DBMS). The agent selects an action (recommends an index) based on the current state and policy, and then the environment (database DBMS) returns a reward value (the reduction in the load query execution time under the index configuration of the current state compared to the previous step) to the agent as the value estimate of the action selection. When reaching the termination state of the phase (such as the number of indexes reaching the upper limit or exceeding the space constraint), the phase ends, the environment is reset (all recommended indexes are deleted, and the state is set to the initial state), and a new iteration starts. Such an iterative loop continues until the policy is stable and the reward function for each state converges. Generally speaking, in each iteration step, the agent selects an action, the environment executes the action, transfers to a new state, and returns a reward. The data (state, reward, action) of each transfer is stored in the replay buffer, and the agent periodically updates the value function of the action using mini-batch stochastic gradient descent.

[0011] However, when using reinforcement learning to recommend indexes, too many candidate indexes will lead to poor recommendation performance. Summary of the Invention

[0012] To overcome the deficiencies of the prior art, the present invention provides a DQN database index recommendation method based on tree-shaped invalid action masking, which can recommend the optimal index design scheme for database design and operation and maintenance personnel given the database table structure, data, and query load. Based on the traditional DQN, firstly, the Double DQN and Dueling DQN technologies are used for improvement. Secondly, according to the leftmost principle of the composite index, the recommended indexes are selected, new columns are added on their right to generate new composite indexes, and the candidate indexes are counted using a tree structure. When training the DQN, the method of invalid action masking is adopted to enable the reinforcement learning agent to only select valid indexes, so as to achieve automatic optimal index configuration and improve the efficiency of the reinforcement learning model.

[0013] The technical solution adopted by the present invention to solve its technical problems includes the following steps:

[0014] Step 1: Define the state, action, reward, and neural network structure of the DQN;

[0015] Step 1-1: State: For the recommendation of single-column indexes, count the columns involved in all queries in the query load, and each column is used as a candidate index, arranged in an array. The one-hot code representation method is adopted, where 0 at the corresponding position indicates that the index is not established, and 1 indicates that the index is established. The indexes that have been established currently are used as the state; for the recommendation of multi-column indexes, similar to the single-column indexes, each position in the array represents a multi-column composite index, and the one-hot code method is also used to represent the establishment situation of the indexes in a state.

[0016] Step 1-2: Action: The agent selects the process of establishing an index in a state, which is represented by a one-dimensional array. The value at the corresponding position in the array represents the Q value of selecting this action, that is, the action value function of establishing the corresponding index.

[0017] The epsilon-greedy method is used to select actions. In the greedy strategy, the action corresponding to the maximum Q value is selected;

[0018] Step 1-3: Reward: The relative reduction in the query execution time of each step of the index configuration compared to the previous step of the index configuration is used to represent the size of the reward; Let X0 represent the initial index configuration, and X t represent the index configuration after the t-th step;

[0019] The following formula is used as the reward function to encourage the agent to select actions that maximize the reduction of costs:

[0020]

[0021] X t+1 = X t ∪ π(X, X t , W)

[0022] Among them, R(X t ) represents the revenue function at the t-th step, and Cost(W, X t ) represents the execution cost of the load at index X at the t-th step t ; π(X, X t , W) represents the newly selected index from the candidate index group X for the load W at the t-th step; the index X t+1 at the (t + 1)-th step is the union of X t and the newly selected index;

[0023] Steps 1 - 4: Neural network structure: A three-layer fully connected neural network structure is adopted; the input of this neural network is the current index configuration state, and the dimension is equal to the number of candidate indexes. The first layer converts the state space dimension number to 1024, the input and output dimensions of the second layer are the same and still 1024, and the output dimension of the last layer is the dimension of the action space, that is, the number of candidate indexes, representing the action value function corresponding to selecting an action under the input state; each layer normalizes the parameters with a Gaussian distribution with a mean of 0 and a variance of 1;

[0024] Step 2: Construct the database environment for which index recommendations are to be made: Define the database, table structure, prepare the data in the table, and the query loads to be executed;

[0025] Step 3: Initialize the neural network: Construct two neural networks eval-net and target-net in the form of the neural network structure defined in Steps 1 - 4, and randomly initialize the parameter values in the two neural networks;

[0026] Step 4: Generate a tree-shaped candidate index statistical structure: For each table, generate all candidate indexes with different widths that conform to the leftmost principle in a full permutation manner, and use a tree structure to store the information of all candidate indexes; where the root node is the table name, and the nodes other than the root node are a combination of each attribute in the table; starting from a single-column index as the basis for expansion, expand one column to the right to form a new composite index as its child node, and so on.

[0027] Step 5: Use a one-dimensional array A to describe the tree-shaped candidate index statistical structure: Convert the index statistical tree into a one-dimensional structure array A through pre-order traversal. Each element in the array A records the columns included in the index, the position of its parent node and all child nodes, and the masking information, that is, whether the index is selected in the next step; valid indicates whether the index can be selected in the next step, and Yes or No in Index indicates whether the index is currently established;

[0028] Step 6: Construct the input array S of the neural network: Since the input of the neural network is one-dimensional data, it is represented by a one-dimensional array S, and this input represents the current state in the reinforcement learning process; extract the information of Index from the array A to form an integer array, which counts the currently established indexes. The corresponding position 0 indicates that the index at that position is not established, and 1 indicates that the index is established. The initial state is no index, that is, all elements of S0 are 0;

[0029] Step 7: Construct the masking array M: Represent the masking mask with a one-dimensional array M, that is, extract the Valid part from the array A to form an integer array. True indicates that the corresponding index can be selected to be established in the current state, and False indicates that the corresponding index cannot be selected; the initial mask is that the single column corresponding positions are True and the others are False;

[0030] Step 8: Train the neural network for each episode:

[0031] Step 8-1: Initialize the structure array A, the state array S, and the masking array M;

[0032] Step 8-2: Input the current state S into the neural network, and the output obtained is a one-dimensional array S' of the same length as the input. Each element in it represents the Q value, that is, the reward, for establishing an index at the corresponding position, denoted as Q(s,a), that is, the Q value of selecting action a in state s; then, mask it with the masking array M, that is, set the Q values of the positions corresponding to False in M, that is, the invalid actions, to a negative number approaching negative infinity, so that the invalid actions will not be selected; randomly select an action from the valid index selection actions with a probability of ε, and select the action a with the largest Q value in the masked neural network output with a probability of 1-ε t = argmaxQ(s,a);

[0033] Step 8-3: Execute the selected action to obtain the reward and the next state;

[0034] In this process, it is necessary to update the tree-shaped statistical structure and the masking information, and store <state, action, reward, next state, mask of the next state> in the replay cache Cache;

[0035] The update steps of the tree structure and the mask are as follows: After the agent selects an action and executes it, set this action to invalid and it cannot be recommended repeatedly later; at this time, delete the parent node index of this action from the current index set, and retrieve using the left part of the action index according to the leftmost principle; mark the actions corresponding to the child nodes of this action as valid for the next selection, and so on;

[0036] Step 8-4: Sample data from the replay cache Cache. Using the method of stochastic gradient descent, keep the target-net fixed. Taking the target-net as the target, use the square of the difference between the Q-values of the target-net and the eval-net as the loss function to update the eval-net; After a set number of rounds, update the parameters of the target-net to the parameters of the eval-net;

[0037] Step 8-5: Repeat Step 8-2 to Step 8-4 until the space constraint is reached;

[0038] Step 8-6: After this episode ends, record the total reward and the finally recommended index combination, reset the environment, and reset the structure array A, the state array S, and the masking array M to their initial values;

[0039] Step 9: Repeat Step 8 until the set maximum number of rounds is reached. Among all episodes, select the index combination with the maximum total reward as the final index recommended for the given database, query load, and data volume.

[0040] Preferably, in the said Step 1-2, the epsilon-greedy method is used to select actions. In the greedy strategy, select the action corresponding to the maximum Q-value:

[0041]

[0042] In the epsilon-greedy strategy, randomly select actions with a certain probability, and in other cases, use the neural network to select actions:

[0043]

[0044] The beneficial effects of the present invention are as follows:

[0045] The present invention models the index selection problem as a reinforcement learning process. According to the leftmost principle of the composite index, recommend columns from left to right, which is realized by using a tree structure to count candidate indexes. When training the traditional DQN improved by Double DQN and Dueling DQN, use the method of masking invalid actions to narrow the action space, enabling the intelligent agent to recommend better indexes, which is superior to the compared mainstream methods and has practical application value in production practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of DQN interaction for index recommendation of the present invention.

[0047] Figure 2 It is a schematic diagram of the tree-shaped action space in the embodiment of the present invention.

[0048] Figure 3 Schematic diagram of the index recommendation status in the embodiments of the present invention; (a) Schematic diagram of recommending a single-column index under the initial state (no index); (b) Schematic diagram of the status after selecting the candidate index r_regionkey; (c) Schematic diagram of the status after successively selecting r_name and (r_name, r_regionkey).

[0049] Figure 4 The load execution time corresponding to the indexes recommended by 7 algorithms under different experimental configurations in the embodiments of the present invention. Detailed implementation manners

[0050] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0051] The present invention provides a method DQN-AMTAS (DQN with Action Mask in Tree-Structured Action Space) for automatically recommending indexes in the case of a given database table structure, data, and query load. This method is based on the deep reinforcement learning network DQN for index recommendation, focuses on using a tree structure to count candidate indexes, and effectively solves the problem of excessive candidate indexes by using invalid action masking during training.

[0052] The state, action, reward, and neural network structure design of DQN in the present invention are as follows:

[0053] 1) State: For the recommendation of single-column indexes, count the columns involved in all queries in the query load. Each column is used as a candidate index and arranged in an array. The one-hot code representation method is adopted. At the corresponding position, 0 indicates that this index is not built, and 1 indicates that this index is built. In this way, the indexes that have been built currently can be represented, and this is used as the state. For the recommendation of multi-column indexes, it is similar to the single-column index, but each position in the state array represents a multi-column composite index, and the one-hot code method is also used to represent the index establishment situation in a state.

[0054] 2) Action: The action is the process of the agent selecting to build an index in a state, which is represented by a one-dimensional array. The value at the corresponding position represents the Q value (action value function) of selecting this action (i.e., building the corresponding index). In DQN, the one-dimensional array representing the current state is input into the neural network, and an array of the same dimension is output. Each value in the output array represents the value function of selecting the index corresponding to this position, that is, the Q value. The present invention adopts the epsilon-greedy method to select actions. In the greedy strategy, select the action corresponding to the maximum Q value:

[0055]

[0056] In the epsilon-greedy strategy, an action is randomly selected with a certain probability, and in other cases, the neural network is used to greedily select the best action:

[0057]

[0058] 3) Reward function: The reward magnitude is represented by the relative reduction in query execution time of each step's index configuration compared to the previous step's index configuration. The objective reward function needs to be maximized in each iteration of each stage. When making each decision step, consider selecting an index from the candidate index set and adding it to the current index configuration to reduce the query execution time without violating the index count or space constraints. Let X0 represent the initial index configuration, and Xi represent the index configuration after the i-th step.

[0059] The present invention uses the following formula as the reward function to encourage the agent to select actions that maximize cost reduction:

[0060]

[0061] where

[0062] X t+1 = X t ∪ π(X, X t , W)

[0063] 4) Neural network structure: A three-layer fully connected neural network structure is adopted. The input of this neural network is the current index configuration state, and the dimension is equal to the number of candidate indexes. The first layer converts the state space dimension number to 1024, the input and output dimensions of the second layer are the same and still 1024, and the output dimension of the last layer is the dimension of the action space, that is, the number of candidate indexes, representing the action value function corresponding to selecting an action in the input state. Each layer normalizes the parameters with a Gaussian distribution with a mean of 0 and a variance of 1.

[0064] Compared with the traditional DQN-based index recommendation method, the main improvement points in the present invention include:

[0065] 1) Based on the traditional DQN, Double DQN and Dueling DQN are used to optimize the reinforcement learning from two aspects: loss function calculation and network structure.

[0066] 2) Based on ordinary experience replay, prioritized experience replay is adopted. When selecting experiences for parameter update, experiences with the optimal value, that is, experiences with a larger TD-error, are preferentially selected to accelerate the learning efficiency.

[0067] 3) In terms of optimizing the candidate index range when recommending composite indexes, since the number of full permutations of all columns is huge, when all possible ones are used as candidate indexes, there will be too many actions for the agent to find a better index configuration. The present invention counts all candidate indexes through a tree structure and effectively narrows the optional range using the method of invalid action masking, enabling the reinforcement learning agent to effectively find a better index configuration. At the same time, since DQN adopts a greedy strategy and randomly selects from all actions, it avoids the situation of being unable to select a multi-column composite index that cannot be used in the left part but is very effective as a whole.

[0068] When counting all candidate indexes based on the tree structure, according to the leftmost principle of composite indexes, only when the first column is recommended can a two-column index with this column as the first column be recommended, and only when the first two columns are recommended can a three-column index starting with these two columns be recommended, and so on. Taking Figure 2 the TPC-H table structure shown as an example, its root node strings together the 8 nodes in the first layer (representing 8 tables in TPC-H). The child nodes of each table are all single columns in this table. For example, the lower-level nodes of the region table have three children: r_regionkey, r_name, and r_comment. For each node among these three children, the next layer forms a new composite candidate index by adding the columns that have not appeared from this child to the root node in the same table. For example, the lower layer of r_regionkey is (r_regionkey, r_name), (r_regionkey, r_comment), the lower layer of r_name is (r_name, r_regionkey), (r_name, r_comment), etc. Continuing further down, the child of (r_regionkey, r_name) is (r_regionkey, r_name, r_comment), and so on. The i-th layer of this tree is the permutation of (i - 2) indexes in the corresponding table, and the nodes in the second layer and below are all candidate indexes.

[0069] The specific steps of using the DQN-AMTAS method of the present invention for index recommendation are as follows:

[0070] 1) Construct the database environment for which index recommendation is to be performed: Define the database, table structure, prepare the data in the table, and the query workload to be executed.

[0071] 2) Define and initialize the neural network: According to the design of the DQN-AMTAS neural network structure in the invention content part, define two neural networks eval-net and target-net with the same three-layer fully connected structure, and randomly initialize the parameter values in the two neural networks.

[0072] 3) Generate a tree-like candidate index statistical structure: For each table, use a full permutation method to generate all candidate indexes of different widths that comply with the leftmost principle, and use a tree structure to store the information of all candidate indexes. The root node is the table name, and the nodes other than the root node are a combination of the attributes in the table. Figure 2 , expanding from a single-column index as the basis, expanding a column on its right to become a new composite index as its child node, and so on.

[0073] 4) Use a one-dimensional array A to describe the tree-shaped candidate index statistics structure: This index statistics tree is converted into a one-dimensional structure array A through pre-order traversal. Each element in array A records the columns contained in the index, the positions of its parent node and all child nodes, and masking information (i.e., whether the index can be selected in the next step). The position is the subscript of the node in the array, which is convenient for updating effective actions when state migration and action selection. The initial state is as follows Figure 3 As shown in (a), valid indicates whether the index can be selected next, and Yes or No of Index indicates whether the index is currently established.

[0074] 5) Construct the input array S of the neural network: Since the input of the neural network is one-dimensional data, it is represented by a one-dimensional array S, which represents the current state in the reinforcement learning process. Extract the Index information from array A to form an integer array, and count the currently established indexes. The corresponding position 0 means that the index at that position is not established, and 1 means that the index is established. The initial state is no index, such as Figure 3 As shown in (a), S0 is all 0.

[0075] 6) Construct mask array M: Use one-dimensional array M to represent mask, that is, extract the Valid part from array A to form an integer array. True means that the corresponding index can be selected in the current state, and False means that the corresponding index cannot be selected. The initial mask is True for the corresponding position of a single column, and False for the others. Figure 3 (a) as shown:

[0076] M0={True,False,False,False,False,True,False,False,False,False,True,False,False,Fa

[0077] lse,False}.

[0078] 7) Train the neural network, for each episode (until the predetermined maximum number of rounds is reached):

[0079] ① Initialize the structure array A, state array S, and mask array M.

[0080] ②For each step (until the space constraint is reached):

[0081] a. First, input the current state S into the neural network. The output is a one-dimensional array S' of the same length as the input. Each element represents the Q-value (i.e., the reward) for the index established at the corresponding position, also denoted as Q(s,a), which is the Q-value of choosing action a in state s. Then, mask it with the masking array M, that is, set the Q-values (invalid actions) at the positions corresponding to False in M to a very large negative absolute value so that invalid actions will not be selected. Randomly select an action from the valid index selection actions with probability ε, and select the action a with the largest Q-value in the masked neural network output with probability 1 - ε t = argmaxQ(s,a).

[0082] b. Execute the selected action to obtain the reward (the reward function is measured by the relative reduction in the load query execution time of each step index configuration compared to the previous step index configuration) and the next state. During this process, the tree statistical structure and masking information need to be updated, and <state, action, reward, next state, mask of the next state> are stored in the replay cache Cache. The update steps of the tree structure and mask are as follows: After the agent selects an action and executes it, set the action to invalid and it cannot be recommended repeatedly. At this time, delete the parent node index of this action from the current index set, and the left part of the action index can be retrieved according to the leftmost principle; mark the actions corresponding to the child nodes of this action as valid and can be selected next, and so on.

[0083] If the first selection is to create an index on r_regionkey, then r_regionkey itself becomes invalid, and its children (r_regionkey,r_name),(r_regionkey,r_comment) become valid. See the state diagram in Figure 3 (b). At this time, the valid candidate indexes are (r_regionkey,r_name),(r_regionkey,r_comment),r_name,r_comment. Suppose the next two steps select r_name and (r_name,r_regionkey) respectively, and the state becomes Figure 3 (c) as shown. At this time, the valid actions include (r_regionkey,r_name),(r_regionkey,r_comment),(r_name,r_regionkey,r_comment),(r_name,r_comment),(r_comment).

[0084] c. Sample data from the replay cache Cache. Using the method of stochastic gradient descent, keep the target-net fixed. Taking the target-net as the target, use the square of the difference between the Q-values of the two networks as the loss function to update the eval-net. After a certain number of rounds, update the parameters of the target-net to the parameters of the eval-net.

[0085] ③ After this episode ends, record the total reward and the finally recommended index combination, and reset the environment (reset the structure array A, the state array S, and the masking array M to their initial values).

[0086] 8) Select the index combination with the largest total reward among all episodes as the final index recommended for the given database, query load, and data volume.

[0087] To prove the effectiveness of the DQN-AMTAS of the present invention, the method of the present invention is implemented based on the index evaluation framework provided by Kossmann et al., and experiments are carried out on the TPC-H and TPC-DS data sets. Compare DQN-AMTAS with Extend, the DB2Advis method, the Relaxation method, the DTA method of Microsoft SQL Server, and the DQN method based on heuristic rules by Lan et al. of CIKM2020. In the method of Lan et al., heuristic rules are used to pre-analyze several candidate indexes from SQL statements for the reinforcement learning agent to select, and it is called DQN with heuristics. At the same time, the results of the method of taking all permutations of all columns as candidate indexes for the agent to select are also given, and it is called DQN with permutation.

[0088] When using the TPC-H and TPC-DS data sets, a certain amount of data needs to be generated. The scale factor determines the amount of data generated. When scale factor = 0.1, it means generating 0.1 GB of data, and when scale factor = 1, it means generating 1 GB of data. The cost in the experiment is calculated by the execution time of the query load, and the overall cost is represented by the total cost in the query plan obtained by the explain statement. The experimental results under different experimental parameters are as follows:

[0089] Table 1 TPC-H, scale factor = 0.1, memory_budget = 50MB

[0090]

[0091] Table 2 TPC-H scale factor = 1, memory_budget = 50MB

[0092]

[0093] Table 3 TPC-H scale factor = 1, memory_budget = 100MB

[0094]

[0095] Table 4 TPC-DS scale factor = 1, memory_budget = 50MB

[0096]

[0097] In the above experiments, after establishing the index recommended by DQN-AMTAS, the execution time of the load is lower than that of the index recommended by other methods, which proves the effectiveness of DQN-AMTAS.

[0098] Figure 4 The execution time of the load corresponding to the index recommended by various methods under different experimental configurations is shown. The shorter the execution time, the better the algorithm effect. It can be seen from the figure that the extend method usually obtains better results. DQN with permutation often has poor results because there are too many candidate actions and it is unable to find actions with high benefits. The DQN-AMTAS proposed in the present invention combines the advantages of heuristic rules and the full permutation candidate set and achieves the best results.

Claims

1. A DQN database index recommendation method based on tree-shaped invalid action masking, characterized in that, The steps are as follows: Step 1: Define the state, action, reward, and neural network structure of DQN; Step 1-1: State: For single-column index recommendation, count the columns involved in all queries in the query workload. Each column is used as a candidate index and arranged into an array. The one-hot encoding representation is adopted, where 0 at the corresponding position indicates that the index is not created, and 1 indicates that the index is created. The currently created indexes are used as the state. For multi-column index recommendation, similar to single-column index, each position in the array represents a multi-column composite index, and the one-hot encoding method is also used to represent the creation situation of indexes in a state; Step 1-2: Action: The agent selects the process of creating an index in a state, which is represented by a one-dimensional array. The value at the corresponding position in the array represents the Q value, that is, the action value function, of selecting this action, namely creating the corresponding index; The epsilon-greedy method is used to select actions. In the greedy strategy, the action corresponding to the maximum Q value is selected; Step 1-3: Benefit: The benefit is represented by the relative reduction in query execution time of the index configuration at each step compared to the previous step; let X0 represent the initial index configuration, and X t represent the index configuration after the t-th step; The following formula is used as the reward function to encourage the agent to select actions that maximize the reduction of cost: X t+1 = X t ∪ π(X, X t , W) where, R(X t ) represents the revenue function at the t-th step, and Cost(W, X t ) represents the execution cost of the load at the t-th step with index X t ; π(X, X t , W) represents the newly selected index from the candidate index group X for the load W at the t-th step; the index X t+1 at the (t + 1)-th step is the union of X t and the newly selected index; Step 1-4: Neural network structure: A three-layer fully connected neural network structure is adopted. The input of this neural network is the current index configuration state, and the dimension is equal to the number of candidate indexes. The first layer converts the state space dimension number to 1024. The input and output dimensions of the second layer are the same, still 1024. The output dimension of the last layer is the dimension of the action space, that is, the number of candidate indexes, representing the action value function corresponding to selecting an action in the input state. The parameters of each layer are normalized using a Gaussian distribution with a mean of 0 and a variance of 1; Step 2: Construct the database environment for index recommendation: Define the database, table structure, prepare the data in the table, and the query workload to be executed; Step 3: Initialize the neural network: Construct two neural networks, eval-net and target-net, with the neural network structure defined in Step 1-4, and randomly initialize the parameter values in the two neural networks; Step 4: Generate a tree-shaped candidate index statistical structure: For each table, generate all candidate indexes with different widths that conform to the leftmost principle in a full permutation manner, and use a tree structure to store the information of all candidate indexes. The root node is the table name, and the nodes other than the root node are a combination of each attribute in the table. Starting from the single-column index as the basis, expand a column on its right to form a new composite index as its child node, and so on; Step 5: Use a one-dimensional array A to describe the tree-shaped candidate index statistical structure: Convert the index statistical tree into a one-dimensional structure array A through pre-order traversal. Each element in the array A records the columns included in the index, the position of its parent node and all child nodes, and the masking information, that is, whether the index is selected in the next step. valid indicates whether the index can be selected in the next step, and Yes or No in Index indicates whether the index is currently created; Step 6: Construct the input array S of the neural network: Since the input of the neural network is one-dimensional data, it is represented by a one-dimensional array S, and this input represents the current state in the reinforcement learning process; Extract the information of Index from the array A to form an integer array, which counts the currently established indexes. The corresponding position 0 indicates that the index at that position is not established, and 1 indicates that the index is established. The initial state is no index, that is, all of S0 are 0; Step 7: Construct the masking array M: It is represented by a one-dimensional array M for the masking mask, that is, extract the Valid part from the array A to form an integer array. True indicates that the corresponding index can be selected to be established in the current state, and False indicates that the corresponding index cannot be selected; The initial mask is that the single column corresponding positions are True and the others are False; Step 8: Train the neural network for each episode: Step 8-1: Initialize the structure array A, the state array S, and the masking array M; Step 8-2: Input the current state S into the neural network, and the output is a one-dimensional array S' of the same length as the input. Each element represents the Q-value, i.e., the return, for establishing an index at the corresponding position, denoted as Q(s,a), which is the Q-value of selecting action a in state s. Then, mask it with the masking array M, that is, set the Q-values at the positions corresponding to False in M, i.e., the invalid actions, to a negative number approaching negative infinity, so that the invalid actions will not be selected. Randomly select an action from the valid index selection actions with a probability of ε, and select the action a with the maximum Q-value in the masked neural network output with a probability of 1-ε t = argmaxQ(s,a); Step 8-3: Execute the selected action to obtain the reward and the next state; In this process, it is necessary to update the tree statistical structure and the masking information, and store <state, action, reward, next state, mask of the next state> in the replay cache Cache; The update steps of the tree structure and the mask are as follows: After the agent selects and executes an action, set this action to invalid and it cannot be recommended repeatedly later; At this time, delete the parent node index of this action from the current index set, and retrieve according to the leftmost principle using the left part of the action index; Mark all the actions corresponding to the child nodes of this action as valid for the next selection, and so on; Step 8-4: Sample data from the replay cache Cache. Using the method of stochastic gradient descent, keep the target-net fixed. Taking the target-net as the target, use the square of the difference between the Q values of the two networks target-net and eval-net as the loss function to update the eval-net; After a set number of rounds, update the parameters of the target-net to the parameters of the eval-net; Step 8-5: Repeat Step 8-2 to Step 8-4 until the space constraint is reached; Step 8-6: After this episode ends, record the total reward and the finally recommended index combination, reset the environment, and reset the structure array A, the state array S, and the masking array M to their initial values; Step 9: Repeat Step 8 until the set maximum number of rounds is reached, and select the index combination with the largest total reward among all episodes as the final index recommended for the given database, query load, and data volume.

2. The DQN database index recommendation method based on tree-shaped invalid action masking according to claim 1, characterized in that, In Step 1-2, the epsilon-greedy method is used to select actions. In the greedy strategy, select the action corresponding to the maximum Q value: In the epsilon-greedy strategy, randomly select actions with a certain probability, and in other cases, use the neural network to select actions:

Images