Reinforcement learning knob tuning method based on self-designed knob evaluator

Abstract

The invention relates to a reinforcement learning knob tuning method based on a self-designed knob evaluator, and relates to the field of database tuning. The invention is to solve the problem that the existing database knob tuning method still has low tuning accuracy and efficiency, which leads to low database throughput and high latency. The invention comprises the following steps: step one, stratified sampling of database knob configurations, generating a knob evaluation set to be tested, and applying the database knob configuration to be tested in the test database group to obtain performance parameter indicators corresponding to the knob configuration; step two, obtaining a state vector matrix using the knob evaluation set to be tested; step three, training a random forest-based regressor using the state vector matrix and performance parameter indicators to obtain a database knob evaluator; step four, training a reinforcement learning agent based on the knob evaluator obtained in step three to obtain a database knob tuner. Step five, obtaining the optimal configuration of the database using the database knob evaluator. The present invention is used for database tuning.

Description

Technical Field

[0001] The present invention relates to the field of database tuning, and in particular to a reinforcement learning knob tuning method based on a self-designed knob evaluator. Background Art

[0002] As one of the most widely used software systems today, database management systems (DBMS) have become deeply embedded in the core operations of various businesses and organizations, storing and managing large amounts of critical data. During the operation of a database system, knob design has a significant impact on system performance, making it crucial to find the right knob configuration.

[0003] To help database administrators (DBAs) effectively tune knobs, numerous methods and technologies have emerged in recent years. Among them, leveraging reinforcement learning (RL) for knob tuning is the most prominent. RL has achieved remarkable results in database knob tuning and exhibits numerous unique advantages. However, existing RL is equivalent to an ergodic learning task. Each iteration requires executing test tasks in the actual database system to obtain performance metrics. This characteristic results in low efficiency and accuracy during the training and evaluation phases. Consequently, current database knob tuners still suffer from low tuning efficiency and accuracy, which in turn leads to low throughput and high latency in the tuned database. Summary of the Invention

[0004] The purpose of the present invention is to solve the problems of low tuning accuracy and efficiency in existing database knob tuning methods, which leads to low database throughput and high latency, and proposes a reinforcement learning knob tuning method based on a self-designed knob evaluator.

[0005] The specific process of the reinforcement learning knob tuning method based on the self-designed knob evaluator is as follows:

[0006] Input the current database knob configuration into the knob tuner to obtain the tuned database knob configuration;

[0007] The knob tuner is obtained in the following way:

[0008] Step 1: Perform stratified sampling on the database knob configuration to generate a knob evaluation set to be tested, and apply the database knob configuration in the knob evaluation set to be tested in the test database group to obtain the performance indicators corresponding to the knob configuration;

[0009] Step 2: Obtain the state vector matrix using the knob evaluation set to be tested;

[0010] Step 3: Use the state vector matrix obtained in step 2 and the performance index obtained in step 1 to train a random forest-based regressor to obtain a database knob evaluator;

[0011] Step 4: Based on the knob evaluator obtained in step 3, train the reinforcement learning agent to obtain the database knob tuner.

[0012] Furthermore, the stratified sampling of the database knob configuration in step 1 is implemented using a Latin hypercube sampling method.

[0013] Furthermore, the performance indicators corresponding to the knob configuration include: database throughput and database delay indicators.

[0014] Furthermore, the elements in the state vector matrix are knob configurations in the knob evaluation set to be tested.

[0015] Furthermore, the step 4 of training the reinforcement learning agent based on the knob evaluator obtained in step 3 to obtain the database knob tuner includes the following steps:

[0016] Step 41: Use the reward function to train the knob evaluator obtained in step 3, and introduce a preset probability factor. In each round of training, the reward function obtains the current performance indicator corresponding to the current knob configuration through the knob evaluator, and at the same time obtains the probability of actually executing the test in the database. If the probability of actually executing the test in the database hits the preset probability factor, the database test is executed, and the actual performance indicator is returned as the reward value. Then, the accuracy of the current knob evaluator is tested. If the probability of actually executing the test in the database does not hit the preset probability factor, the performance indicator given by the current knob evaluator is used as the reward value.

[0017] Step 42: The intelligent agent using reinforcement learning selects a corresponding action based on the reward value obtained in step 41, constructs an action vector matrix, and uses the action vector matrix to update the database state;

[0018] Step 43: After a preset number of training times, a database knob tuner is obtained.

[0019] Furthermore, the reward function is as follows:

[0020] reward=α*ΔT0+β*ΔTt

[0021] ΔT0=(throughput t -throughput0) / throughput0

[0022] ΔT t =(throughput t -throughput windows ) / throughput windows

[0023]

[0024] Among them, α and β are weight parameters, i∈[1,size windows ], size windows is the window size on the performance indicator throughput data, throughput windows It is the average throughput in the sliding window of the performance indicator throughput data. t is the throughput at time t, throughput0 is the throughput at the initial time, ΔT0 is the throughput change ratio at the current time t relative to the initial time, ΔT t It is the throughput change ratio at the current time t relative to the time x, where the value of x is equal to the window size of the performance indicator throughput data.

[0025] Furthermore, the accuracy of the current knob evaluator is tested as follows:

[0026] If the accuracy of the performance indicator obtained by the knob evaluator is lower than the preset accuracy threshold, the probability factor is updated to factor*1.1; if the accuracy of the performance indicator obtained by the knob evaluator is higher than the preset accuracy threshold, the probability factor is updated to factor*0.9;

[0027] Among them, factor is the preset probability factor.

[0028] Furthermore, the step 42 wherein the agent selects a corresponding action based on the reward value obtained in step 41, forms an action vector matrix, and updates the database state using the action vector matrix includes the following steps:

[0029] Step 421: The reinforcement learning agent selects the corresponding action based on the reward value obtained in step 41. All actions constitute the action vector matrix action;

[0030] Step 422: Use the motion vector matrix obtained in step 421 to obtain the knob configuration parameter increment, and use the knob configuration parameter increment to update the current state, as shown in the following formula:

[0031] new_state=state+delta_knob*λ

[0032] Where λ is the scaling factor, state is the current state, new_state is the updated current state, and delta_knob is the knob configuration parameter increment.

[0033] The intelligent agent of the reinforcement learning is a trained D4PG model;

[0034] Among them, the training set for training the D4PG model is<state,action,new_state,reward> ;

[0035] Among them, state is the current state vector matrix, new_state is the updated state vector matrix, and action is the action vector matrix.

[0036] Furthermore, the knob configuration parameter increment is as follows:

[0037] delta_knob=(max_knob_value-min_knob_value)*action

[0038] Among them, max_knob_value is the maximum value of the knob configuration parameter, and min_knob_value is the minimum value of the knob parameter configuration.

[0039] The beneficial effects of the present invention are:

[0040] The present invention proposes an offline trained database knob evaluator to accelerate reinforcement learning tasks. The present invention combines the traditional machine learning technology with reinforcement learning with the proposed evaluator, making full use of the advantages of both, and obtains the database knob evaluator through offline training. Then the present invention makes full use of the ability of the distributed deep deterministic policy gradient (D4PG) algorithm to process continuous action space, combining the advantages of deep learning and reinforcement learning, thereby obtaining a knob tuner based on the database knob evaluator, bringing a more accurate and efficient solution to the database knob tuning task, making database knob tuning more efficient. The present invention automatically adjusts the database knob parameters through the reinforcement learning algorithm, significantly improving the efficiency and accuracy of database knob tuning, thereby improving the throughput of the tuned database and reducing the latency of the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0042] Specific implementation method 1: Figure 1 As shown, the specific process of the reinforcement learning knob tuning method based on the self-designed knob evaluator in this embodiment is as follows: inputting the current knob configuration of the database into the knob tuner to obtain the tuned database knob configuration;

[0043] The knob tuner is obtained in the following way:

[0044] Step 1: Use the Latin Hypercube Sampling (LHS) algorithm to perform stratified sampling on the database knob configurations to generate a set of knob evaluations to be tested, covering the entire database evaluation knob space. These knob configurations are then applied to the test database cluster (usually the backup database) to collect performance indicator data corresponding to the knob configurations.

[0045] The database knobs are configured as database configuration parameters, which are used to adjust the database's memory allocation, cache size, concurrency control, query optimization, and other aspects;

[0046] In actual applications, database administrators adjust the data knob configuration according to the database load and performance requirements to achieve more efficient data storage, retrieval, and management.

[0047] The performance indicator data includes: database throughput and database delay indicator;

[0048] Step 2: Encode the knob configurations in the knob evaluation set to be tested obtained in Step 1 and construct a state vector matrix;

[0049] The state vector of each dimension in the state vector matrix represents the knob value of the corresponding dimension;

[0050] In reinforcement learning tasks, state is a formal representation of the environment information obtained by the agent during the learning process. It provides the agent with key information needed to interact with the environment. In the database knob tuning task, state is used to represent the value of the knob currently used for tuning. Assuming that there are currently m knobs for tuning, the state vector state = [s1, s2, ..., s m ], where s i Indicates the knob configuration of the i-th knob.

[0051] Step 3: Use the state vector matrix and performance indicator data obtained in step 2 to train a regressor based on the random forest model to obtain a database knob evaluator, that is, a mapping model of the mapping relationship between the knob configuration in the knob evaluation set and the performance indicator;

[0052] The database knob evaluation model creates a mapping from database knob configurations to database performance metrics. Specifically, the evaluation model takes the current database knob configuration as input and generates performance estimates corresponding to that knob configuration. This approach provides real-time, quantitative reference information for optimizing database performance.

[0053] Step 4: Based on the knob evaluator obtained in step 3, train the reinforcement learning agent to obtain the database knob tuner. Specifically:

[0054] Step 41: Use the reward function to iteratively train the knob evaluator obtained in step 3, and introduce a preset probability factor. In each round of iteration, the reward function calculates the performance index under the current knob configuration state through the knob evaluator. At the same time, the database outputs the probability of actually executing the test in the database. If the current probability hits the preset probability factor, the database test is executed, and the actual performance index is used as the reward value. At the same time, the accuracy of the knob evaluator is tested, and the probability factor of actually executing the test in the database is adjusted according to the accuracy test result of the knob evaluator. If the preset probability factor is not hit, the performance parameter index given by the knob evaluator is used as the reward value.

[0055] The accuracy of the knob evaluator is tested. If the accuracy of the performance indicator obtained by the knob evaluator is lower than the preset accuracy threshold, the probability factor of the actual execution of the database test is increased and the probability factor is updated to factor*1.1; if the accuracy of the performance indicator obtained by the knob evaluator is higher than the preset threshold, the probability factor of the actual execution of the database test is reduced and the probability factor is updated to factor*0.9. At the same time, a reasonable upper and lower limit will be set for the factor, such as [0.1, 0.9].

[0056] The reward function is as follows:

[0057] ΔT0=(Throughput t -Throughputt0) / throughput0

[0058] ΔT t =(throughput t -throughput windows ) / throughput windows

[0059]

[0060] Among them, size windows is the window size on the performance indicator throughput data, throughpit windows It is the average throughput in the sliding window of the performance indicator throughput data. t is the throughput at time t, throughput0 is the throughput at the initial time, ΔT0 is the throughput change ratio at the current time t relative to the initial time, ΔT t It is the throughput change ratio at time t relative to time x. The value of x is equal to the window size of the performance indicator throughput data.

[0061] Finally, we combine the above two performance change factors to obtain the reward function as follows:

[0062] reward=α*ΔT0+β*ΔT t

[0063] Here, α and β are weight parameters that adjust the impact of various performance changes on the reward function. In other words, they adjust the database's bias toward global and immediate rewards. Using this reward function, we can effectively guide the reinforcement learning algorithm to find the optimal database knob adjustment strategy.

[0064] Step 42: Use the agent to select the corresponding action based on the reward value obtained in each iteration to form an action vector action matrix, and use the action vector matrix to tune the database. At the same time, tune the knob configuration based on the reward value and the prediction results of the database knob evaluator;

[0065] Step 421. In each round of iteration, the agent selects the corresponding action based on the defined performance parameter indicator reward value. The action vector here is named 'action'. In view of the continuity, high dimensionality and nonlinear characteristics of the database knob tuning task, we designed a continuous action space for the environment so that we can effectively explore and select appropriate knob parameters. The action space is defined as a continuous interval and is described by the action vector action = [a_1, a_2, ..., a_m], where m represents the number of knobs and the value range of each action element a_i is [-1, 1]; the agent is a trained deep deterministic policy gradient (D4PG) model;

[0066] Among them, the training set of the agent is a four-tuple:<state,action,new_state,reward> ;

[0067] During the training phase, as iterations proceed, the model will continuously try and generate such quadruple data, and use this data to update the training network.

[0068] D4PG uses target network technology to reduce instability during the learning process. By using a set of slowly updated target networks instead of directly using online networks for bootstrap estimation, the noise in training can be reduced. When applying the D4PG algorithm, both Actor and Critic use deep neural networks as function approximators. This enables D4PG to handle high-dimensional state spaces and complex action spaces. The actor network is responsible for mapping the input state to the action space to generate the optimal action. The critic network estimates the distributed Q function based on the actions generated by the actor to evaluate the action value;

[0069] Step 422: Use the motion vector obtained in step 421 and the knob configuration parameter range to obtain the knob configuration parameter increment, and use the knob parameter increment to update the current state:

[0070] delta_knob=(max_knob_value-min_knob_value)*action

[0071] new_state=state+delta_knob*λ

[0072] Where max_knob_value is the maximum value of the knob configuration parameter, min_knob_value is the minimum value of the knob parameter configuration, action is the action vector, delta_knob is the knob configuration parameter increment, λ is the scaling factor, state is the current state, and new_state is the updated current state;

[0073] Each dimension of the action vector matrix represents the knob tuning ratio of one round of iterative training, and its value range is [-1, 1];

[0074] The knob parameter range is obtained by searching the database manual.

[0075] Step 43: After a preset number of iterative trainings, a database knob tuner is obtained.

[0076] Based on the above specific implementation methods, the following advantages of the present invention are verified:

[0077] First, regarding strategy selection, the tuner of this invention employs a Bayesian optimization algorithm based on an interpretable model to enhance its global search capabilities. This strategy allows the tuner to more quickly determine the search range that potentially contains the global optimal solution during the global optimization process. Once this range is determined, the present invention leverages the powerful exploration capabilities of reinforcement learning to accurately find the optimal solution, thereby achieving efficient solutions for large-scale and complex database knob tuning problems.

[0078] Secondly, in terms of reward function design, this invention utilizes knob evaluator technology, eliminating the need for actual execution testing in the database for parts of the training process. The accuracy of the knob evaluator is dynamically monitored during training and adjusted accordingly. This design significantly reduces the execution time of the longest phase of training, further improving the algorithm's operational efficiency. Furthermore, a carefully designed reward function with a sliding window allows for better evaluation of local performance, helping to identify appropriate knob adjustment strategies.

[0079] In addition, the present invention also adopts an improved deep deterministic policy gradient algorithm - distributed deep deterministic policy gradient (D4PG), which makes full use of the ability of the distributed deep deterministic policy gradient (D4PG) algorithm to process continuous action space, and combines the advantages of deep learning and reinforcement learning, so that the database knob tuning can be performed more efficiently. The whole process is to automatically adjust the database knob parameters through the reinforcement learning algorithm to achieve a significant improvement in the performance of the entire database system. Compared with the traditional deep deterministic policy gradient (DDPG) algorithm, D4PG performs better in predicting Q value distribution and is more suitable for application in database knob tuning scenarios. At the same time, D4PG has the ability of distributed training, which can significantly improve the training speed and knowledge learning effect, and bring more accurate and efficient solutions to database knob tuning tasks.

Claims

1. A reinforcement learning knob tuning method based on a self-designed knob evaluator, characterized by The specific process of the method is: inputting the current knob configuration of the database into the knob tuner to obtain the tuned database knob configuration; The knob tuner is obtained in the following way: Step 1: Perform stratified sampling on the database knob configuration to generate a knob evaluation set to be tested, and apply the database knob configuration in the knob evaluation set to be tested in the test database group to obtain the performance indicators corresponding to the knob configuration; Step 2: Obtain the state vector matrix using the knob evaluation set to be tested; Step 3: Use the state vector matrix obtained in step 2 and the performance index obtained in step 1 to train a random forest-based regressor to obtain a database knob evaluator; Step 4: Based on the knob evaluator obtained in step 3, train the reinforcement learning agent to obtain the database knob tuner, including the following steps: Step 41: Use the reward function to train the knob evaluator obtained in step 3, and introduce a preset probability factor. In each round of training, the reward function obtains the current performance indicator corresponding to the current knob configuration through the knob evaluator, and at the same time obtains the probability of actually executing the test in the database. If the probability of actually executing the test in the database hits the preset probability factor, the database test is executed, and the actual performance indicator is returned as the reward value. Then, the accuracy of the current knob evaluator is tested. If the probability of actually executing the test in the database does not hit the preset probability factor, the performance indicator given by the current knob evaluator is used as the reward value. Step 42: The intelligent agent using reinforcement learning selects the corresponding action based on the reward value obtained in step 41, forms an action vector matrix, and uses the action vector matrix to update the database state, including the following steps: Step 421: The reinforcement learning agent selects the corresponding action based on the reward value obtained in step 41. All actions constitute the action vector matrix action; Step 422: Use the motion vector matrix obtained in step 421 to obtain the knob configuration parameter increment, and use the knob configuration parameter increment to update the current state, as shown in the following formula: new_state=state+delta_knob*λ Where λ is the scaling factor, state is the current state, new_state is the updated current state, and delta_knob is the knob configuration parameter increment; Step 43: After a preset number of training times, a database knob tuner is obtained.

2. The method for knob tuning based on reinforcement learning and a self-designed knob evaluator according to claim 1, characterized in that: The stratified sampling of the database knob configuration in step 1 is implemented using the Latin hypercube sampling method.

3. The method for knob tuning based on reinforcement learning and a self-designed knob evaluator according to claim 2, characterized in that: The performance indicators corresponding to the knob configuration include: database throughput and database delay indicators.

4. The method for knob tuning based on reinforcement learning and a self-designed knob evaluator according to claim 3, characterized in that: The elements in the state vector matrix are the knob configurations in the knob evaluation set to be tested.

5. The method for knob tuning based on reinforcement learning and a self-designed knob evaluator according to claim 4, characterized in that: The reward function is as follows: reward=α*ΔT0+β*ΔT t ΔT0=(throughput t -throughput0) / throughput0 ΔT t =(throughput t -throughput windows ) / throughput windows Among them, α and β are weight parameters, i∈[1,size windows ], size windows is the window size on the performance indicator throughput data, throughput windows It is the average throughput in the sliding window of the performance indicator throughput data. t is the throughput at time t, throughput0 is the throughput at the initial time, ΔT0 is the throughput change ratio at the current time t relative to the initial time, ΔT t It is the throughput change ratio at the current time t relative to the time x, where the value of x is equal to the window size of the performance indicator throughput data.

6. The method for knob tuning based on reinforcement learning and a self-designed knob evaluator according to claim 5, characterized in that: The accuracy of the current knob evaluator is tested as follows: If the accuracy of the performance indicator obtained by the knob evaluator is lower than the preset accuracy threshold, the probability factor is updated to factor*1.1; if the accuracy of the performance indicator obtained by the knob evaluator is higher than the preset accuracy threshold, the probability factor is updated to factor*0.9; Among them, factor is the preset probability factor.

7. The method for knob tuning based on reinforcement learning and a self-designed knob evaluator according to claim 6, characterized in that: The intelligent agent of the reinforcement learning is a trained D4PG model; Among them, the training set for training the D4PG model is<state,action,new_state,reward> ; Among them, state is the current state vector matrix, new_state is the updated state vector matrix, and action is the action vector matrix.

8. The method for knob tuning based on reinforcement learning and a self-designed knob evaluator according to claim 7, characterized in that: The knob configuration parameter increment is as follows: delta_knob=(max_knob_value-min_knob_value)*action Among them, max_knob_value is the maximum value of the knob configuration parameter, and min_knob_value is the minimum value of the knob parameter configuration.

Images