Intelligent parameter tuning method and system for end-to-end cloud native database
Through parameter space optimization and Q-learning reinforcement learning algorithm, combined with transfer learning, poor performance problems caused by cloud native database parameter setting are solved, automatic parameter adjustment and rapid adaptation are achieved, and resource waste and manual tuning are reduced.
Patent Information
- Application Number
- CN202410924904.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-26
AI Technical Summary
The poor performance problems caused by parameter settings of cloud native databases mainly include complexity, insufficient samples and dynamic variation, resulting in difficulty in manual tuning and waste of resources.
The end-to-end cloud-native database intelligent parameter tuning method is adopted to filter important parameters through the parameter space optimizer, and the parameters are automatically adjusted using the Q-learning reinforcement learning algorithm, and the parameter configuration is automatically adjusted and optimized through transfer learning when the environment changes.
The parameter space is reduced, the demand for high-quality sample collection is reduced, and the dynamically changing workloads and hardware resources are quickly adapted to improve database performance.
Smart Images

Figure CN119025536A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of database performance optimization, and specifically to an end-to-end cloud native database intelligent parameter tuning method and system. Background Art
[0002] There are a large number of configuration parameters in cloud-native databases. These parameters control many aspects of the system, such as memory allocation, I / O optimization, backup and recovery, and have a great impact on the performance of the database. Appropriate parameter configuration can improve the performance of the database by orders of magnitude, while inappropriate parameter configuration will cause its performance to be greatly reduced. Therefore, these parameters usually require the DBA (Database Administrator) to adjust and set them based on their own experience to improve system performance. However, due to the large number of configuration parameters of cloud-native databases and the fact that there are often a large number of database instances in user business scenarios, the workload and hardware configuration of each database instance are constantly changing elastically, making it very difficult to manually configure system parameters to meet the needs of applications. Therefore, automatic tuning of database parameters through intelligent methods is one of the main options for solving this problem. At present, intelligent parameter tuning of cloud-native databases has the following challenges:
[0003] (1) Complexity. Cloud-native databases usually have hundreds of adjustable parameters, which are of various types, such as continuous, discrete, and enumerated. There are also complex implicit dependencies between parameters. Finding the optimal parameter configuration in a complex parameter space is an NP-hard problem.
[0004] (2) Insufficient samples. Cloud-native database parameter optimization usually requires a large number of data samples to obtain good results. In actual scenarios, these samples are difficult to obtain.
[0005] (3) Dynamic variability. The huge scale of cloud-native databases and the flexible and changeable elastic resource allocation often lead to changes in the environment, including changes in workloads and hardware configurations. In this case, the existing optimal parameter configuration will no longer be applicable and the parameters need to be re-optimized, resulting in a waste of resources and time. Summary of the invention
[0006] In view of the needs and shortcomings of current technological development, the present invention provides an end-to-end cloud native database intelligent parameter tuning method and system to solve the problem of poor system performance of cloud native databases due to parameter settings.
[0007] In a first aspect, the present invention provides an end-to-end cloud native database intelligent parameter tuning method, and the technical solution adopted to solve the above technical problems is as follows:
[0008] An end-to-end cloud native database intelligent parameter tuning method includes the following steps:
[0009] S1. Use the parameter space optimizer to filter and simplify the database parameters to obtain a set of important parameters that affect database performance.
[0010] S2, based on the important parameter set, uses Q-learning reinforcement learning algorithm to automatically adjust and recommend optimal parameter settings;
[0011] S3. When the environment changes, automatic adjustment and optimization of database parameter configuration is achieved based on transfer learning.
[0012] Optionally, step S1 is performed to use a parameter space optimizer to filter and simplify the database parameters to obtain an important parameter set. The specific operations taken include:
[0013] S1.1. Automatically clone an equivalent test environment and corresponding data based on the current user configuration, and collect the executed historical SQL statements based on the user's logs within the specified time period; the cloned environment is automatically deleted after the parameter recommendation model training is completed, and the user is not aware of it;
[0014] S1.2. Filter and filter historical SQL query records to form a Top-SQL list, and build a benchmark test load based on the Top-SQL list. The filtering criteria include: 1) sorting the top n records in descending order by execution frequency, where n is manually specified by the DBA; 2) sorting by execution time in descending order, and records with a time exceeding m, where m is manually executed by the DBA;
[0015] S1.3. For all parameters of the cloud native database, the DBA determines whether a certain parameter affects the SQL execution efficiency according to the cloud native database user manual. If so, the parameter data type is classified into enumeration, integer, or floating point type.
[0016] S1.4. For parameters of enumeration type, the initial parameter space is the set of all enumeration values; for integer or floating point parameters, the parameter value interval is divided by the range partitioning method, and a parameter value is selected in each parameter interval by a random algorithm. The initial parameter space is composed of the set of random parameter values in all intervals;
[0017] S1.5, based on the initial parameter space of each parameter, cyclically randomly generate x groups of different parameter combination configurations; based on each parameter combination, execute the benchmark test generated in step S1.2, and record the execution time overhead of the test corresponding to each parameter combination, store the parameter combination and the execution time overhead in a data structure, and construct a training data set;
[0018] S1.6. Use the logistic regression model based on the L1 regular term to perform model training based on the training data set. After the model training is completed, filter out the parameter items corresponding to all non-zero weights, and form an important parameter set based on the filtered parameter items.
[0019] Further optionally, step S2 is performed to automatically adjust and recommend optimal parameter settings based on the important parameter set using a Q-learning reinforcement learning algorithm, and the specific implementation operations include:
[0020] S2.1. Divide the state space according to the value range of important parameters. Each state represents a set of possible parameter configurations. For integer or enumeration type parameters, their state space directly contains all possible values. For floating point parameters, the continuous state space discretization strategy is directly used to divide them into multiple intervals. Each interval is mapped to a discrete state value, thus forming a state space.
[0021] S2.2, all adjustable parameters and their discretized values form an action space, each action is a combination of a parameter-value pair, representing the parameter to be adjusted under a given state and its target value; possible actions include adjusting the value of a parameter or keeping the value of a parameter unchanged;
[0022] S2.3. Define a reward function based on the load execution time performance indicator. The defined reward function is used to evaluate the effect of the database executing an action in a certain state: a) If an action reduces the database execution time, the reward function returns a positive value; b) If an action increases the database execution time, the reward function returns a negative value; c) If an action causes the database execution time to remain unchanged, the reward function returns 0 or a value infinitely close to 0;
[0023] S2.4. In the Q-learning reinforcement learning algorithm, the parameter recommendation model first selects an action A from the action space according to the current state, then executes action A and observes the execution time of the database system load under the new state, thereby obtaining the return value of the reward function, and then uses the Bellman equation to update the Q value table in the parameter recommendation model; repeat this process until the maximum number of iterations is reached or the Q value converges. At this time, the Q value table of the parameter recommendation model contains the best Q value for each action in each state, and the optimal strategy can be selected based on these best Q values;
[0024] S2.5. When using the parameter recommendation model, the parameter recommendation model randomly selects an action for exploration with a probability of ε, and selects the currently known best action for utilization with a probability of 1-ε.
[0025] Optionally, when the environment changes, automatic adjustment and optimization of database parameter configuration can be achieved based on transfer learning. The specific process includes:
[0026] Maintain and save all database instance feature vectors of authorized users in the cloud. The original attributes of the feature vector include: number of instance nodes, instance hardware configuration information, SQL query type ratio and data volume. All original attributes are converted into feature vectors of uniform length through a unified Embedding model. The feature vector is used as the key and the parameter combination is configured as the value, which is saved in the cloud KV storage.
[0027] When the hardware configuration of a database instance of a user changes, the current environment feature vector is recalculated in the cloud, and the vector similarity calculation method is used to query whether there is a feature vector above a certain similarity threshold in the cloud KV storage. a) If it exists, the feature vector with the highest similarity is selected, its parameter combination configuration is obtained, and the parameter combination configuration is updated to the current environment. b) If it does not exist, steps S1 and S2 are executed again to obtain the parameter recommendation model. At the same time, the current environment characterization vector and the latest parameter configuration are inserted into the cloud KV storage.
[0028] In a second aspect, the present invention provides an end-to-end cloud native database intelligent parameter tuning system. The technical solution adopted to solve the above technical problems is as follows:
[0029] An end-to-end cloud native database intelligent parameter tuning system, comprising:
[0030] The screening and simplification module is used to screen and simplify database parameters using the parameter space optimizer to obtain a set of important parameters that affect database performance;
[0031] Automatic setting module, which is used to automatically adjust and recommend optimal parameter settings based on a set of important parameters using the Q-learning reinforcement learning algorithm;
[0032] The learning optimization module is used to automatically adjust and optimize database parameter configuration based on transfer learning when the environment changes.
[0033] Optionally, the screening and simplification module uses a parameter space optimizer to screen and simplify database parameters to obtain a set of important parameters that affect database performance, specifically including the following process:
[0034] Automatically clone an equivalent test environment and corresponding data based on the current user configuration, and collect the executed historical SQL statements based on the user's logs within the specified time period; the cloned environment is automatically deleted after the parameter recommendation model training is completed, and the user is unaware of it;
[0035] The Top-SQL list is formed based on the historical SQL query records, and the benchmark load is built based on the Top-SQL list. The screening criteria include: 1) sorting the top n records in descending order by execution frequency, where n is manually specified by the DBA; 2) sorting the records in descending order by execution time, where the execution time exceeds m, where m is manually executed by the DBA;
[0036] For all parameters of the cloud native database, the DBA determines whether a certain parameter affects the SQL execution efficiency according to the cloud native database user manual. If so, the parameter is classified into enumeration, integer, or floating point type according to the parameter data type;
[0037] For parameters of enumeration type, the initial parameter space is the set of all enumeration values; for integer or floating point parameters, the range partitioning method is used to divide the parameter value interval, and a parameter value is selected in each parameter interval using a random algorithm. The initial parameter space consists of the set of random parameter values in all intervals.
[0038] Based on the initial parameter space of each parameter, randomly generate x groups of different parameter combination configurations in a loop; execute the benchmark test generated in step S1.2 based on each parameter combination, and record the execution time overhead of the test corresponding to each parameter combination, store the parameter combination and the execution time overhead in a data structure, and construct a training data set;
[0039] The model training is performed based on the training data set using a logistic regression model based on the L1 regular term. After the model training is completed, the parameter items corresponding to all non-zero weights are screened, and an important parameter set is formed based on the screened parameter items.
[0040] Further optionally, the automatic setting module involved uses a Q-learning reinforcement learning algorithm to automatically adjust and recommend optimal parameter settings based on the important parameter set, specifically including the following operations:
[0041] The state space is divided according to the value range of important parameters. Each state represents a set of possible parameter configurations. For integer or enumeration type parameters, their state space directly contains all possible values. For floating point parameters, the continuous state space discretization strategy is directly used to divide them into multiple intervals. Each interval is mapped to a discrete state value, thus forming a state space.
[0042] All adjustable parameters and their discretized values form an action space. Each action is a combination of parameter-value pairs, which represents the parameter to be adjusted and its target value under a given state. Possible actions include adjusting the value of a parameter or keeping the value of a parameter unchanged.
[0043] A reward function is defined based on the load execution time performance indicator. The defined reward function is used to evaluate the effect of the database executing an action in a certain state: a) If an action reduces the database execution time, the reward function returns a positive value; b) If an action increases the database execution time, the reward function returns a negative value; c) If an action causes the database execution time to remain unchanged, the reward function returns 0 or a value infinitely close to 0;
[0044] In the Q-learning reinforcement learning algorithm, the parameter recommendation model first selects an action A from the action space according to the current state, then executes action A and observes the execution time of the database system load under the new state, thereby obtaining the return value of the reward function, and then uses the Bellman equation to update the Q value table in the parameter recommendation model; repeat this process until the maximum number of iterations is reached or the Q value converges. At this time, the Q value table of the parameter recommendation model contains the best Q value for each action in each state, and the optimal strategy can be selected based on these best Q values;
[0045] When using the parameter recommendation model, the parameter recommendation model randomly selects an action for exploration with a probability of ε, and selects the currently known best action for utilization with a probability of 1-ε.
[0046] Further optionally, the learning optimization module involved realizes automatic adjustment and optimization of database parameter configuration based on transfer learning when the environment changes, which specifically includes the following process:
[0047] Maintain and save all database instance feature vectors of authorized users in the cloud. The original attributes of the feature vector include: number of instance nodes, instance hardware configuration information, SQL query type ratio and data volume. All original attributes are converted into feature vectors of uniform length through a unified Embedding model. The feature vector is used as the key and the parameter combination is configured as the value, which is saved in the cloud KV storage.
[0048] When the hardware configuration of a database instance of a user changes, the current environment feature vector is recalculated in the cloud, and the vector similarity calculation method is used to query whether there is a feature vector above a certain similarity threshold in the cloud KV storage. a) If it exists, the feature vector with the highest similarity is selected, its parameter combination configuration is obtained, and the parameter combination configuration is updated to the current environment. b) If it does not exist, steps S1 and S2 are executed again to obtain the parameter recommendation model. At the same time, the current environment characterization vector and the latest parameter configuration are inserted into the cloud KV storage.
[0049] The end-to-end cloud native database intelligent parameter tuning method and system of the present invention have the following beneficial effects compared with the prior art:
[0050] 1. The present invention uses a parameter space optimizer to reduce the dimension and screen the huge parameter space, focusing only on the parameters that are critical to database performance, thereby reducing the parameter space; in the reduced parameter space, the Q-learning reinforcement learning algorithm is used to build a parameter recommendation model, and the optimal parameters are recommended through limited samples, reducing the need for collecting a large number of high-quality samples; in response to dynamically changing workloads and hardware resources, transfer learning is introduced to quickly complete parameter configuration updates caused by elastic changes in workloads and hardware resources, thereby reducing model training overhead;
[0051] 2. The present invention can solve the problem of poor system performance of cloud native databases caused by parameter settings. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Attached Figure 1 This is an implementation process architecture diagram of the first embodiment of the present invention;
[0053] Attached Figure 2 This is a module connection block diagram of the second embodiment of the present invention. DETAILED DESCRIPTION
[0054] In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.
[0055] Embodiment 1:
[0056] Combined with Figure 1 , this embodiment proposes an end-to-end cloud native database intelligent parameter tuning method, which includes the following steps:
[0057] S1. Use the parameter space optimizer to filter and streamline database parameters to obtain a set of important parameters that affect database performance. The specific operations taken include:
[0058] S1.1. Automatically clone an equivalent test environment and corresponding data based on the current user configuration, and collect the executed historical SQL statements based on the user's logs within the specified time period; the cloned environment is automatically deleted after the parameter recommendation model training is completed, and the user is not aware of it;
[0059] S1.2. Filter and filter historical SQL query records to form a Top-SQL list, and build a benchmark test load based on the Top-SQL list. The filtering criteria include: 1) sorting the top n records in descending order by execution frequency, where n is manually specified by the DBA; 2) sorting by execution time in descending order, and records with a time exceeding m, where m is manually executed by the DBA;
[0060] S1.3. For all parameters of the cloud native database, the DBA determines whether a certain parameter affects the SQL execution efficiency according to the cloud native database user manual. If so, the parameter data type is classified into enumeration, integer, or floating point type.
[0061] S1.4. For parameters of enumeration type, the initial parameter space is the set of all enumeration values; for integer or floating point parameters, the parameter value interval is divided by the range partitioning method, and a parameter value is selected in each parameter interval by a random algorithm. The initial parameter space is composed of the set of random parameter values in all intervals;
[0062] S1.5, based on the initial parameter space of each parameter, cyclically randomly generate x groups of different parameter combination configurations; based on each parameter combination, execute the benchmark test generated in step S1.2, and record the execution time overhead of the test corresponding to each parameter combination, store the parameter combination and the execution time overhead in a data structure, and construct a training data set;
[0063] S1.6. Use the logistic regression model based on the L1 regular term to perform model training based on the training data set. After the model training is completed, filter out the parameter items corresponding to all non-zero weights, and form an important parameter set based on the filtered parameter items.
[0064] S2: Based on the important parameter set, the Q-learning reinforcement learning algorithm is used to automatically adjust and recommend the optimal parameter settings. The specific implementation operations include:
[0065] S2.1. Divide the state space according to the value range of important parameters. Each state represents a set of possible parameter configurations. For integer or enumeration type parameters, their state space directly contains all possible values. For floating point parameters, the continuous state space discretization strategy is directly used to divide them into multiple intervals. Each interval is mapped to a discrete state value, thus forming a state space.
[0066] S2.2. All adjustable parameters and their discretized values form an action space. Each action is a combination of a parameter-value pair, which represents the parameter to be adjusted under a given state and its target value. Possible actions include adjusting the value of a parameter or keeping the value of a parameter unchanged. For example, there are two adjustable parameters, cache size and number of concurrent connections. A possible action space will include: 1) increase the cache size to X GB and keep the number of concurrent connections unchanged; 2) increase the cache size to Y GB and keep the number of concurrent connections unchanged; 3) keep the cache size unchanged and increase the number of concurrent connections to P; 4) keep the cache size unchanged and increase the number of concurrent connections to Q; and so on, to form an action space set;
[0067] S2.3. Define a reward function based on the load execution time performance indicator. The defined reward function is used to evaluate the effect of the database executing an action in a certain state: a) If an action reduces the database execution time, the reward function returns a positive value; b) If an action increases the database execution time, the reward function returns a negative value; c) If an action causes the database execution time to remain unchanged, the reward function returns 0 or a value infinitely close to 0;
[0068] S2.4. In the Q-learning reinforcement learning algorithm, the parameter recommendation model first selects an action A from the action space according to the current state, then executes action A and observes the execution time of the database system load under the new state, thereby obtaining the return value of the reward function, and then uses the Bellman equation to update the Q value table in the parameter recommendation model; repeat this process until the maximum number of iterations is reached or the Q value converges. At this time, the Q value table of the parameter recommendation model contains the best Q value for each action in each state, and the optimal strategy can be selected based on these best Q values;
[0069] S2.5. When using the parameter recommendation model, the parameter recommendation model randomly selects an action for exploration with a probability of ε, and selects the currently known best action for utilization with a probability of 1-ε.
[0070] S3. When the environment changes, automatic adjustment and optimization of database parameter configuration is achieved based on transfer learning. The specific process includes:
[0071] Maintain and save all database instance feature vectors of authorized users in the cloud. The original attributes of the feature vector include: number of instance nodes, instance hardware configuration information (such as memory, persistent storage, network bandwidth, etc.), SQL query type (such as query, insert, update, delete) ratio and data volume. All original attributes are converted into feature vectors of uniform length through a unified Embedding model. The feature vector is used as the key and the parameter combination is configured as the value, which is saved in the cloud KV storage.
[0072] When the hardware configuration of a database instance of a user changes, the current environment feature vector is recalculated in the cloud, and the vector similarity calculation method is used to query whether there is a feature vector above a certain similarity threshold in the cloud KV storage. a) If it exists, the feature vector with the highest similarity is selected, its parameter combination configuration is obtained, and the parameter combination configuration is updated to the current environment. b) If it does not exist, steps S1 and S2 are executed again to obtain the parameter recommendation model. At the same time, the current environment characterization vector and the latest parameter configuration are inserted into the cloud KV storage.
[0073] Embodiment 2:
[0074] Combined with Figure 2 This embodiment proposes an end-to-end cloud native database intelligent parameter tuning system, which includes:
[0075] The screening and simplification module is used to screen and simplify database parameters using the parameter space optimizer to obtain a set of important parameters that affect database performance;
[0076] Automatic setting module, which is used to automatically adjust and recommend optimal parameter settings based on a set of important parameters using the Q-learning reinforcement learning algorithm;
[0077] The learning optimization module is used to automatically adjust and optimize database parameter configuration based on transfer learning when the environment changes.
[0078] In this embodiment, the screening and simplification module uses a parameter space optimizer to screen and simplify database parameters to obtain a set of important parameters that affect database performance, which specifically includes the following process:
[0079] Automatically clone an equivalent test environment and corresponding data based on the current user configuration, and collect the executed historical SQL statements based on the user's logs within the specified time period; the cloned environment is automatically deleted after the parameter recommendation model training is completed, and the user is unaware of it;
[0080] The Top-SQL list is formed based on the historical SQL query records, and the benchmark load is built based on the Top-SQL list. The screening criteria include: 1) sorting the top n records in descending order by execution frequency, where n is manually specified by the DBA; 2) sorting the records in descending order by execution time, where the execution time exceeds m, where m is manually executed by the DBA;
[0081] For all parameters of the cloud native database, the DBA determines whether a certain parameter affects the SQL execution efficiency according to the cloud native database user manual. If so, the parameter is classified into enumeration, integer, or floating point type according to the parameter data type;
[0082] For parameters of enumeration type, the initial parameter space is the set of all enumeration values; for integer or floating point parameters, the range partitioning method is used to divide the parameter value interval, and a parameter value is selected in each parameter interval using a random algorithm. The initial parameter space consists of the set of random parameter values in all intervals.
[0083] Based on the initial parameter space of each parameter, randomly generate x groups of different parameter combination configurations in a loop; execute the benchmark test generated in step S1.2 based on each parameter combination, and record the execution time overhead of the test corresponding to each parameter combination, store the parameter combination and the execution time overhead in a data structure, and construct a training data set;
[0084] The model training is performed based on the training data set using a logistic regression model based on the L1 regular term. After the model training is completed, the parameter items corresponding to all non-zero weights are screened, and an important parameter set is formed based on the screened parameter items.
[0085] In this embodiment, the automatic setting module involved uses the Q-learning reinforcement learning algorithm to automatically adjust and recommend optimal parameter settings based on the important parameter set, specifically including the following operations:
[0086] The state space is divided according to the value range of important parameters. Each state represents a set of possible parameter configurations. For integer or enumeration type parameters, their state space directly contains all possible values. For floating point parameters, the continuous state space discretization strategy is directly used to divide them into multiple intervals. Each interval is mapped to a discrete state value, thus forming a state space.
[0087] All adjustable parameters and their discretized values form an action space. Each action is a combination of a parameter-value pair, which represents the parameter that needs to be adjusted in a given state and its target value. Possible actions include adjusting the value of a parameter or keeping the value of a parameter unchanged. For example, there are two adjustable parameters, cache size and number of concurrent connections. A possible action space will include: 1) increase the cache size to X GB and keep the number of concurrent connections unchanged; 2) increase the cache size to Y GB and keep the number of concurrent connections unchanged; 3) keep the cache size unchanged and increase the number of concurrent connections to P; 4) keep the cache size unchanged and increase the number of concurrent connections to Q; and so on, to form an action space set;
[0088] A reward function is defined based on the load execution time performance indicator. The defined reward function is used to evaluate the effect of the database executing an action in a certain state: a) If an action reduces the database execution time, the reward function returns a positive value; b) If an action increases the database execution time, the reward function returns a negative value; c) If an action causes the database execution time to remain unchanged, the reward function returns 0 or a value infinitely close to 0;
[0089] In the Q-learning reinforcement learning algorithm, the parameter recommendation model first selects an action A from the action space according to the current state, then executes action A and observes the execution time of the database system load under the new state, thereby obtaining the return value of the reward function, and then uses the Bellman equation to update the Q value table in the parameter recommendation model; repeat this process until the maximum number of iterations is reached or the Q value converges. At this time, the Q value table of the parameter recommendation model contains the best Q value for each action in each state, and the optimal strategy can be selected based on these best Q values;
[0090] When using the parameter recommendation model, the parameter recommendation model randomly selects an action for exploration with a probability of ε, and selects the currently known best action for utilization with a probability of 1-ε.
[0091] In this embodiment, the learning optimization module involved realizes automatic adjustment and optimization of database parameter configuration based on transfer learning when the environment changes, which specifically includes the following process:
[0092] Maintain and save all database instance feature vectors of authorized users in the cloud. The original attributes of the feature vector include: number of instance nodes, instance hardware configuration information (such as memory, persistent storage, network bandwidth, etc.), SQL query type (such as query, insert, update, delete) ratio and data volume. All original attributes are converted into feature vectors of uniform length through a unified Embedding model. The feature vector is used as the key and the parameter combination is configured as the value, which is saved in the cloud KV storage.
[0093] When the hardware configuration of a database instance of a user changes, the current environment feature vector is recalculated in the cloud, and the vector similarity calculation method is used to query whether there is a feature vector above a certain similarity threshold in the cloud KV storage. a) If it exists, the feature vector with the highest similarity is selected, its parameter combination configuration is obtained, and the parameter combination configuration is updated to the current environment. b) If it does not exist, steps S1 and S2 are executed again to obtain the parameter recommendation model. At the same time, the current environment characterization vector and the latest parameter configuration are inserted into the cloud KV storage.
[0094] In summary, the end-to-end cloud native database intelligent parameter tuning method and system of the present invention can solve the problem of poor system performance of the cloud native database due to parameter settings, and realize the rapid adaptation and application of the optimal parameter combination in an environment where hardware elasticity changes.
[0095] The above specific examples are used to explain the principles and implementation methods of the present invention in detail. These examples are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by technicians in this technical field without departing from the principles of the present invention should fall within the scope of patent protection of the present invention.
Claims
1. An end-to-end cloud native database intelligent parameter tuning method, characterized in that: The steps include: S1. Use the parameter space optimizer to filter and streamline database parameters to obtain a set of important parameters that affect database performance. S2, based on a set of important parameters, uses Q-learning reinforcement learning algorithm to automatically adjust and recommend optimal parameter settings; S3. When the environment changes, automatic adjustment and optimization of database parameter configuration is achieved based on transfer learning.
2. The end-to-end cloud native database intelligent parameter tuning method according to claim 1 is characterized in that: Execute step S1 and use the parameter space optimizer to filter and streamline the database parameters to obtain a set of important parameters. The specific operations taken include: S1.
1. Automatically clone an equivalent test environment and corresponding data based on the current user configuration, and collect historical SQL statements executed based on the user's logs within a specified time period; S1.
2. Filter historical SQL query records to form a Top-SQL list. Build a benchmark workload based on the Top-SQL list. The filtering criteria include: 1) sorting the top n records in descending order by execution frequency, where n is manually specified by the DBA; 2) sorting by execution time in descending order, and only looking for records with execution times exceeding m, where m is manually executed by the DBA; S1.
3. For all parameters of the cloud-native database, the DBA determines whether a parameter affects SQL execution efficiency according to the cloud-native database user manual. If so, the DBA classifies the parameter into enumeration, integer, or floating-point types based on the parameter data type. S1.
4. For enumeration type parameters, the initial parameter space is the set of all enumeration values. For integer or floating point parameters, the parameter value interval is divided using a range partitioning method. A parameter value is randomly selected within each parameter interval. The initial parameter space consists of the set of random parameter values within all intervals. S1.
5. Based on the initial parameter space of each parameter, randomly generate x sets of different parameter combination configurations in a loop; execute the benchmark test generated in step S1.2 for each parameter combination, and record the execution time cost of each parameter combination corresponding to the test. Store the parameter combination and execution time cost in a data structure to construct a training dataset; S1.
6. Use the logistic regression model based on the L1 regularization term to perform model training based on the training data set. After completing the model training, filter out all parameter items corresponding to non-zero weights, and form an important parameter set based on the filtered parameter items.
3. The end-to-end cloud native database intelligent parameter tuning method according to claim 2 is characterized in that: Execute step S2, based on the set of important parameters, use the Q-learning reinforcement learning algorithm to automatically adjust and recommend optimal parameter settings. The specific implementation operations include: S2.
1. Divide the state space according to the value range of important parameters, and each state represents a set of possible parameter configurations; S2.
2. All adjustable parameters and their discretized values form an action space. Each action is a parameter-value pair, representing the parameter to be adjusted and its target value in a given state. S2.
3. Define a reward function based on the load execution time performance indicator. The defined reward function is used to evaluate the effect of executing an action on the database in a certain state: a) If an action reduces the database execution time, the reward function returns a positive value; b) If an action increases the database execution time, the reward function returns a negative value; c) If an action causes the database execution time to remain unchanged, the reward function returns 0 or a value as close to 0 as possible; S2.
4. In the Q-learning reinforcement learning algorithm, the parameter recommendation model first selects an action A from the action space based on the current state. It then executes action A and observes the execution time of the database system load under the new state to obtain the return value of the reward function. It then uses the Bellman equation to update the Q-value table in the parameter recommendation model. This process is repeated until the maximum number of iterations is reached or the Q-value converges. At this point, the Q-value table of the parameter recommendation model contains the optimal Q-value for each action in each state. Based on these optimal Q-values, the optimal strategy can be selected. S2.
5. When using the parameter recommendation model, the parameter recommendation model randomly selects an action for exploration with a probability of ε, and selects the currently known best action for utilization with a probability of 1-ε.
4. The end-to-end cloud native database intelligent parameter tuning method according to claim 3 is characterized in that: When executing step S2.1, for parameters of integer or enumeration type, their state space directly contains all possible values; for floating-point parameters, the continuous state space discretization strategy is directly used to divide them into multiple intervals, and each interval is mapped to a discrete state value, thereby forming a state space.
5. The end-to-end cloud native database intelligent parameter tuning method according to claim 3 is characterized in that: Execute step S3 to automatically adjust and optimize database parameter configuration based on transfer learning when the environment changes. The specific process includes: The cloud maintains and saves the feature vectors of all authorized users' database instances. The original attributes of these feature vectors include the number of instance nodes, instance hardware configuration information, SQL query type ratio, and data volume. All original attributes are converted into feature vectors of uniform length through a unified embedding model. The feature vector is used as the key and the parameter combination is configured as the value. The vectors are then stored in the cloud's KV storage. When the hardware configuration of a user's database instance changes, the current environment feature vector is recalculated in the cloud, and the vector similarity calculation method is used to query whether there is a feature vector with a similarity threshold higher than a certain threshold in the cloud KV storage. a) If it exists, the feature vector with the highest similarity is selected, its parameter combination configuration is obtained, and the parameter combination configuration is updated to the current environment. b) If it does not exist, steps S1 and S2 are executed again to obtain the parameter recommendation model. At the same time, the current environment characterization vector and the latest parameter configuration are inserted into the cloud KV storage.
6. An end-to-end cloud native database intelligent parameter tuning system, characterized by: It includes: The screening and simplification module is used to screen and simplify database parameters using the parameter space optimizer to obtain a set of important parameters that affect database performance; Automatic setting module, which is used to automatically adjust and recommend optimal parameter settings based on a set of important parameters using the Q-learning reinforcement learning algorithm; The learning optimization module is used to automatically adjust and optimize database parameter configurations based on transfer learning when the environment changes.
7. The end-to-end cloud native database intelligent parameter tuning system according to claim 6, characterized in that: The screening and simplification module uses the parameter space optimizer to screen and simplify the database parameters to obtain a set of important parameters that affect database performance. Specifically, the process includes the following: Automatically clone an equivalent test environment and corresponding data based on the current user configuration, and collect historical SQL statements executed based on the user's logs within a specified time period; A top-SQL list is generated based on historical SQL query records. A benchmark load is then built based on the top-SQL list. The filtering criteria include: 1) sorting the top n records in descending order by execution frequency, where n is manually specified by the DBA; 2) sorting the top n records in descending order by execution time, with records whose execution time exceeds m, where m is manually executed by the DBA; For all parameters of the cloud-native database, the DBA determines whether a parameter affects SQL execution efficiency based on the cloud-native database user manual. If so, the DBA classifies the parameter into enumeration, integer, or floating-point types based on the parameter data type. For enumeration type parameters, the initial parameter space is the set of all enumeration values. For integer or floating point parameters, the range partitioning method is used to divide the parameter value interval. A parameter value is selected from each parameter interval using a random algorithm. The initial parameter space is composed of the set of random parameter values in all intervals. Based on the initial parameter space of each parameter, randomly generate x sets of different parameter combination configurations in a loop; execute the benchmark test generated in step S1.2 for each parameter combination, and record the execution time overhead of the corresponding test for each parameter combination. Store the parameter combination and execution time overhead in a data structure to construct a training dataset; The model is trained based on the training data set using a logistic regression model based on the L1 regularization term. After the model training is completed, all parameter items corresponding to non-zero weights are screened and an important parameter set is formed based on the screened parameter items.
8. The end-to-end cloud native database intelligent parameter tuning system according to claim 7, characterized in that: The automatic setting module uses the Q-learning reinforcement learning algorithm to automatically adjust and recommend optimal parameter settings based on the important parameter set. Specifically, it performs the following operations: The state space is divided according to the value range of important parameters, and each state represents a set of possible parameter configurations; All adjustable parameters and their discretized values form an action space. Each action is a combination of parameter-value pairs, representing the parameter that needs to be adjusted and its target value in a given state. Define a reward function based on the load execution time performance indicator. The reward function is used to evaluate the effect of executing an action in a certain state: a) If an action reduces the database execution time, the reward function returns a positive value; b) If an action increases the database execution time, the reward function returns a negative value; c) If an action causes the database execution time to remain unchanged, the reward function returns 0 or a value as close to 0 as possible; In the Q-learning reinforcement learning algorithm, the parameter recommendation model first selects an action A from the action space based on the current state. It then executes action A and observes the execution time of the database system load under the new state to obtain the return value of the reward function. The Bellman equation is then used to update the Q-value table in the parameter recommendation model. This process is repeated until the maximum number of iterations is reached or the Q-value converges. At this point, the Q-value table of the parameter recommendation model contains the optimal Q-value for each action in each state. The optimal strategy can be selected based on these optimal Q-values. When using the parameter recommendation model, the parameter recommendation model randomly selects an action for exploration with a probability of ε, and selects the currently known best action for utilization with a probability of 1-ε.
9. The end-to-end cloud native database intelligent parameter tuning system according to claim 8, characterized in that: When dividing the state space according to the value range of important parameters: For parameters of integer or enumeration type, the state space directly contains all possible values; For floating-point parameters, the continuous state space discretization strategy is directly used to divide them into multiple intervals, and each interval is mapped to a discrete state value, thereby forming a state space.
10. The end-to-end cloud native database intelligent parameter tuning system according to claim 8, characterized in that: The learning optimization module automatically adjusts and optimizes database parameter configuration based on transfer learning when the environment changes, specifically including the following process: The cloud maintains and saves the feature vectors of all authorized users' database instances. The original attributes of these feature vectors include the number of instance nodes, instance hardware configuration information, SQL query type ratio, and data volume. All original attributes are converted into feature vectors of uniform length through a unified embedding model. The feature vector is used as the key and the parameter combination is configured as the value. The vectors are then stored in the cloud's KV storage. When the hardware configuration of a user's database instance changes, the current environment feature vector is recalculated in the cloud, and the vector similarity calculation method is used to query whether there is a feature vector with a similarity threshold higher than a certain threshold in the cloud KV storage. a) If it exists, the feature vector with the highest similarity is selected, its parameter combination configuration is obtained, and the parameter combination configuration is updated to the current environment. b) If it does not exist, steps S1 and S2 are executed again to obtain the parameter recommendation model. At the same time, the current environment characterization vector and the latest parameter configuration are inserted into the cloud KV storage.
Citation Information
Cited By
Database configuration parameter tuning method based on load characterization and large model exploration
CN120821715A
Database configuration parameter tuning methods based on load characterization and large model exploration
CN120821715B
Database configuration optimization method based on parameter effective mechanism
CN121350003A
A database configuration optimization method based on a parameter taking effect mechanism
CN121350003B
