Distributed database parameter adaptive tuning method based on multi-agent deep reinforcement learning

Abstract

The invention relates to a distributed database parameter adaptive tuning method based on multi-agent deep reinforcement learning, and belongs to the technical field of information. According to the method, the thought of a multi-agent deep reinforcement learning algorithm is introduced into the field of database parameter tuning, the cooperation and competition relation between nodes in a distributed database environment is explored, and a deep deterministic strategy gradient reinforcement learning algorithm C-MADDPG based on a centralized strategy gradient estimator is provided. A distributed database parameter tuning problem is modeled based on a partially observable Markov decision process. Problem definition takes expansion of a parameter search space as a cost, takes a complex competition effect between database nodes as a black box, carries out mathematical modeling on an observable cooperation effect, improves the universality and scalability of an algorithm, solves a relative generalization problem of the algorithm, and can be expanded to a general distributed database architecture. The C-MADDPG provided by the invention shows good performance in the parameter tuning of the distributed database.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of information technology, and particularly relates to a distributed database parameter self-adaptive tuning method based on multi-agent deep reinforcement learning. BACKGROUND

[0002] In recent years, with the rapid development of mobile application technology, the high-concurrency load problem of database service is becoming more and more serious. A common solution is to build a distributed environment to balance the user request load to multiple nodes. Under the condition of limited number of servers, another common solution is to optimize the performance of the database service to improve the upper limit of the load that each database node can withstand. The parameter tuning of the database is a relatively important research direction in the field of database tuning. The performance optimization of the database not only depends on the reasonable physical design, such as index design and data storage method, but also depends on hundreds of Knobs parameters that can be tuned. However, the value space of the database Knobs parameters is divided into continuous and discrete types, and there are very complex nonlinear relationships between these parameters. Therefore, it is particularly difficult to reasonably tune the combination of these Knobs parameters in high-dimensional space. Especially in a distributed environment, there are complex interactions between each database node, which makes the nonlinear relationship between the parameters of the database nodes more complex. How to effectively tune the parameters of each database node is a difficulty in the field of database parameter tuning. If the parameter tuning is performed separately for each database node, the tuning results are often locally optimal for the cluster performance, and there is still no effective global optimal solution. In addition, there are many kinds of existing distributed database architectures. It is very difficult for a professional DBA to quickly adapt to the working principle of the database architecture, analyze the internal state of each database node, and perform parameter tuning for server clusters with different hardware configurations. It is a huge workload and requires a lot of time for testing and tuning.

[0003] Current research on database parameter tuning mainly falls into two categories: query-level parameter tuning and workload-level parameter tuning. The purpose of query-level parameter tuning research is mainly to handle fine-grained workload parameter tuning, optimize performance for a single user query, and the performance test benchmark is a single SQL statement, which can make the database better adapt to the special load environment. This type of research takes QTune proposed by Guoliang Li as an example. QTune uses a deep reinforcement learning model to train the model for SQL statements and workloads respectively.

[0004] While workload-level parameter tuning research is mainly to solve the overall parameter tuning of the database in the general high concurrency environment, the training efficiency is higher, and the tuning model has universality, which can be extended and deployed in other database architectures and different workload environments. For example, CDBTune proposed by Ji Zhang uses Sysbench and TPC benchmark tools to simulate the workload in a high concurrency environment, and as a tool for evaluating the final model effect, CDBTune first applies deep reinforcement learning model to the field of database parameter tuning, which has a pioneering significance. Existing researches mostly focus on database parameter tuning in single machine or cloud single machine environment, while the research on database parameter tuning in distributed environment considers the distributed application scenario, but does not consider the interaction between nodes, such as transaction strong consistency, load balancing and other distributed database node characteristics. When tuning the distributed database cluster, we actually focus on the performance indicators of the entire cluster, not just the processing speed of a single SQL or Data node. After changing the configuration file of a certain node, it will actually affect the transaction processing speed, load distribution, lock competition and other nodes, thereby affecting the performance indicators of the entire cluster.

[0005] In addition, most of the existing related researches focus on database parameter tuning in single machine or cloud single machine environment, while the research on database parameter tuning in distributed environment considers the distributed application scenario, but does not consider the interaction between nodes, such as transaction strong consistency, load balancing and other distributed database node characteristics. When tuning the distributed database cluster, we actually focus on the performance indicators of the entire cluster, not just the processing speed of a single SQL or Data node. After changing the configuration file of a certain node, it will actually affect the transaction processing speed, load distribution, lock competition and other nodes, thereby affecting the performance indicators of the entire cluster. In the distributed database environment, taking MMDTune proposed by FengYe as an example, MMDTune trains multiple Actor and Critic networks for each modal database based on the Actor-Critic deep reinforcement learning architecture in the multi-modal database environment. However, since reinforcement learning needs to meet the condition that the environment is stable in Markov decision process, that is, the behavior strategy observed by an agent cannot be affected by other agents. MMDTune actually trains neural networks independently for each node in the distributed database. If each node of the multi-modal database belongs to a different database type, MMDTune meets the standard Markov decision process. However, the MMDTune tuning model cannot be migrated to a general distributed database architecture, because the environment observed by the database nodes is non-stationary at this time, and can only be applied to a distributed environment where there is no interaction between nodes. SUMMARY

[0006] The research focus of the present application is based on OLTP workload level parameter tuning. The ultimate goal of distributed database parameter tuning is to improve the transaction processing speed, delay and other performance indicators of the entire database cluster, and different workloads of Sysbench and TPC-C are used to evaluate the effect of the algorithm model, so a distributed database parameter self-adaptive tuning method based on multi-agent deep reinforcement learning is provided, which cooperates with parameter tuning for each database node.

[0007] To achieve the above object, the technical scheme of the present application is: a distributed database parameter self-adaptive optimization method based on multi-agent deep reinforcement learning, which introduces the multi-agent deep reinforcement learning algorithm into the field of database parameter optimization, explores the cooperation and competition relationship between nodes in the distributed database environment, proposes a deep deterministic policy gradient reinforcement learning algorithm C-MADDPG based on centralized strategy gradient estimator, models the database cluster parameter optimization model based on the partially observable Markov decision process of the distributed database parameter optimization problem, defines the problem to expand the parameter search space, regards the complex competition between database nodes as a black box, and models the mathematical model of the database cluster parameter optimization model for the observable cooperation, and finally realizes the distributed database parameter optimization.

[0008] In an embodiment of the present application, the database cluster parameter optimization model is modeled based on the partially observable Markov decision process of the distributed database parameter optimization problem, and is specifically implemented as follows:

[0009] Agent: defined as a distributed database node, one agent corresponds to one database node to be optimized; all agents to be trained share communication to meet the assumption premise of constructing a centralized Critic network and a centralized strategy gradient estimator;

[0010] Environment: defined as a workload, in training the database cluster parameter optimization model, the Sysbench workload simulation tool is selected, and in evaluating the optimization effect of the database cluster parameter optimization model, the TPC-C standard OLTP load simulation tool is used for performance test;

[0011] State: defined as the internal state of the database node, in the database parameter optimization problem, the State calculation method is used to reflect the current resource consumption and work situation of the database node;

[0012] Action: defined as the combination of heterogeneous parameters available for optimization, wherein the continuous parameters need to dynamically define the optimization range, and the discrete parameters dynamically specify the possible specific values;

[0013] Reward: defined as the difference between the performance indicators of the cluster, the observable cooperation is specified to maximize the TPS, QPS and 95% Latency performance indicators, wherein the TPS and QPS change trend are similar, and the given Reward calculation weight is the same;

[0014] Observation: defined as the observable State-Action of other database nodes to be optimized, ensures that all database nodes to be optimized share communication, that is, meets the partially observable Markov decision process assumption premise, so that when training the optimization model, the State-Action of all database nodes can be obtained.

[0015] In an embodiment of the present application, the method comprises the following steps:

[0016] Step S1, the database cluster explores high concurrency workloads using a behavior policy Behavior Policy;

[0017] Step S2, persisting historical experience data and constructing a database cluster state matrix and an action matrix;

[0018] Step S3, a central training server updates a database cluster Actor-Critic network model;

[0019] Step S4, a database cluster parameter optimization model evaluation.

[0020] In an embodiment of the present application, step S1 is implemented as follows:

[0021] Step S11, the Actor Main network adds NoiseLinear random noise disturbance;

[0022] Step S12, database node heterogeneous parameter mapping;

[0023] Step S13, database cluster stress test start and data preheating;

[0024] Step S14, data preprocessing;

[0025] Step S15, dynamically predefining heterogeneous parameter optimization range;

[0026] Step S16, obtaining a database cluster training data five-tuple.

[0027] In an embodiment of the present application, the database cluster training data five-tuple D is:

[0028] D=(S t , A t , R t+1 , S t+1 , A t+1 ) (1)

[0029] C-MADDPG, based on heterogeneous strategy Off-Policy, divides the agent execution policy into behavior policy and target policy, and uses the historical experience replay buffer cluster database to train the data; in formula (1), S t represents the cluster state matrix obtained by the multi-agent interacting with the workload environment at time step t; if it is the first training round and t = 1, A t represents the default cluster action matrix, that is, the initial parameters of the database system, otherwise, it represents the cluster action matrix obtained by the multi-agent exploring the workload environment using the ActorMain network with an increased NoiseLinear noise layer, that is, Behavior Policy, at the last time step, and at time step t, the multi-agent obtains the cluster state matrix S t As the input parameter of the ActorMain network, the forward propagation result output by the Tanh layer of the ActorMain network is A t+1 ; S t+1 represents the cluster state matrix obtained by the multi-agent interacting with the workload environment after the multi-agent disperses and executes A t+1 , that is, the database cluster application configures A t+1 parameter matrix and restarts; R t+1 represents the cluster state matrix obtained by the multi-agent dispersing and executing A t and A t+1 , and the difference between the weighted average of the performance index data obtained by the two environment explorations.

[0030] To map the Tanh layer output result of the ActorMain network as Behavior Policy to the actual available database heterogeneous parameter combination, the cluster action matrix is processed using the following formula:

[0031]

[0032] Among them, formula (2) processes continuous parameters based on linear scaling method, and formula (3) processes discrete parameters based on bucket mapping method; and respectively represent the continuous parameters and discrete parameters in the cluster action matrix recommended by Behavior Policy at time step t; in formula (3), L (L > 0) represents the number of values of the discrete parameters; and represent the maximum and minimum matrices of the continuous parameters in the recommended cluster action matrix:

[0033]

[0034] m represents the number of continuous parameters to be optimized in a single database node, and n represents the total number of database nodes to be optimized, With represents the maximum value of the jth continuous parameter to be optimized in the ith database node to be optimized.

[0035] In an embodiment of the present application, step S2 is implemented as follows:

[0036] Step S21, persist the database node parameter optimization historical experience data;

[0037] Step S22, aggregate and construct the database cluster state matrix and the action matrix.

[0038] In an embodiment of the present application, the database cluster state matrix and the action matrix formula are as follows:

[0039]

[0040] Where S is the cluster state matrix, m represents the number of internal state indicators of a single database node, n represents the total number of database nodes to be optimized, s ij represents the jth internal state indicator data value in the ith database node to be optimized in the target database cluster; A is the cluster action matrix, m represents the number of heterogeneous parameters to be optimized in a single database node, n represents the total number of database nodes to be optimized, a ij represents the jth heterogeneous parameter value to be optimized in the ith database node to be optimized in the target database cluster; the central training server needs to aggregate the persisted state and action experience data in the historical experience replay buffer pool of all database nodes to be optimized before updating the database cluster Actor-Critic network model, construct the cluster state matrix and the action matrix, define as historical experience aggregation samples, and use as Actor-Critic network training data.

[0041] In an embodiment of the present application, step S3 is implemented as follows:

[0042] Step S31, remove the NoiseLinear random noise of the Actor Main network;

[0043] Step S32, update the Critic Main network model parameters;

[0044] Step S33, construct a centralized policy gradient estimator;

[0045] Step S34, update the Actor Main network model parameters;

[0046] Step S35, soft update Actor Target and Critic Target network model parameters;

[0047] Step S36, the center training server publishes the Actor Main network model.

[0048] In an embodiment of the present application, the C-MADDPG divides the Actor-Critic network into the main network Main and the target network Target based on the Bellman optimal formula, including the Actor Main network, the Critic Main network, the Actor Target network and the Critic Target network, wherein the Critic Main network loss function formula is defined as follows:

[0049]

[0050] The loss function of the Critic Main network of the i-th agent is calculated by aggregating the TDError error squares of the historical experience samples, wherein θ Q represents the fitting parameters of the Critic Main network, Q represents the action value, i.e. the Q value, which represents the fitting parameters for value updating; S represents the number of batch historical experience aggregation samples; y j represents the fitting parameters of the Critic Main network, Q represents the action value, i.e. the Q value, which represents the fitting parameters for value updating; S represents the number of batch historical experience aggregation samples; y t and A t+1 as network input and calculate the true value of Q value using the j-th historical experience aggregation sample; Q i represents the TD Target of the j-th historical experience aggregation sample calculated by the i-th agent using the Actor Target network and the Critic Target network, wherein as the input of all agent Actor Target networks, S t+1 and the output of the Actor Target network together as the input of the Critic Target network of the i-th agent, and the output of the Critic Target network is the TD Target, which represents the maximum Q value of the current environment exploration trajectory, and N is the number of agents;

[0051] Before updating the Actor network model parameters, a centralized policy gradient estimator is constructed for the cluster action matrix, and in the center training server, the forward propagation aggregation results of the Actor Main networks of other agents are used to replace the historical experience aggregation samples; therefore, the Actor Main network loss function formula is defined as follows:

[0052]

[0053] J(θ i μ ) is the loss function of the Actor Main network of the i-th agent, and the reciprocal of the Q value is calculated, wherein θ μ represents the fitting parameter of the Actor Main network and a = μ(s), and represents the fitting function mapping relationship between the action and the state in the policy update; in formula (9), the centralized policy gradient estimator aggregates all the Actor Main networks of the agents using S t as an input parameter, and the forward propagation output result is taken as an input parameter The centralized estimation cluster action matrix true value is taken as an input parameter together with S t , and the true value of the Q value is estimated; in addition, the central training server needs to synchronize the updated Actor Main network model parameters to the database node for further environment exploration;

[0054] The soft update is adopted to update the Target network model parameters using the Main network model parameters after a fixed time step is executed, and the soft update parameter formulas of the Actor Target and Critic Target network models are as follows:

[0055]

[0056] θ Q′ and θ μ′ are the parameter results of the Critic Target and Actor Target networks after the soft update is executed, respectively; τ is used to control the network rate of the Target network to update the Main network, and the value range is [0, 1];

[0057] In an embodiment of the present application, step S4 is specifically implemented as follows:

[0058] Step S41, the multi-agent is executed in a decentralized manner A t ;

[0059] Step S42, (S t , R t , A t+1 ) is obtained;

[0060] Step S43, the multi-agent is executed in a decentralized manner A t+1 ;

[0061] Step S44, R t+1 is calculated;

[0062] Step S45, the performance tuning effect improvement ratio is calculated, and the performance tuning effect improvement ratio calculation formula is as follows:

[0063]

[0064] (S t ,R t ,A t+1 ) by a plurality of agents Default cluster parameter matrix A d , the performance index data weighted average sum R is obtained by cluster performance stress test, wherein R t represents the performance index data weighted average sum obtained by the performance stress test; after the plurality of agents disperses and executes the cluster parameter matrix A t+1 , the cluster performance stress test operation is performed again, and the performance index data weighted average sum R t+1 is obtained; r is the performance tuning effect improvement ratio, the value range is [-1, 1], and R t+1 and R t are calculated.

[0065] The application further provides a database cluster monitoring system design system, comprising a memory, a processor and computer program instructions stored on the memory and capable of being executed by the processor, when the processor executes the computer program instructions, the method steps as described above can be realized.

[0066] The application further provides a computer readable storage medium, which stores computer program instructions capable of being executed by a processor, when the processor executes the computer program instructions, the method steps as described above can be realized.

[0067] Compared with the prior art, the application has the following beneficial effects:

[0068] (1) The application is aimed at a distributed database environment, and points out that if the interaction between nodes is ignored and each database node parameter is optimized alone, it is often trapped in a local optimal solution for the entire cluster, leading to an unstable environment, so that the multi-agent reinforcement learning idea is introduced into the database parameter optimization field, and the distributed database parameter optimization problem is modeled based on a partially observable Markov decision process, and the algorithm model has good universality and scalability, and is suitable for parameter optimization problems of general distributed database architectures.

[0069] (2) The application proposes a multi-agent deep deterministic policy gradient reinforcement learning algorithm (C-MADDPG) based on a centralized strategy gradient estimator, which is used to solve the relative generalization phenomenon.

[0070] (3) The application compares and evaluates experiments from four angles under four high-concurrency workloads with DDPG and MADDPG algorithms, and proves that the C-MADDPG algorithm has good performance and generalization ability in distributed database parameter optimization, and has obvious advantages in a database cluster with a large number of nodes.

[0071] (4) The application is based on the edge computing idea, and the work load environment exploration and network model parameter updating steps are executed in parallel by the database nodes and the central training server, each database node maintains a historical experience replay buffer pool, so that the calculation efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 The C-MADDPG algorithm working principle diagram of the application.

[0073] Figure 2 The C-MADDPG algorithm working flow diagram of the application.

[0074] Figure 3 The Actor-Critic network architecture diagram of the application.

[0075] Figure 4 The DDPG, MADDPG and C-MADDPG loss function value trend diagram of the application.

[0076] Figure 5 The performance index change diagram of the tuning process under the oltp_read_write work load of the application.

[0077] Figure 6 The tuning model effect comparison diagram under different work loads of the application. DETAILED DESCRIPTION

[0078] The technical solutions of the application will be specifically described below with reference to the drawings.

[0079] The application provides a distributed database parameter adaptive tuning method based on multi-agent deep reinforcement learning, introduces the multi-agent deep reinforcement learning algorithm idea into the database parameter tuning field, explores the cooperation and competition relationship between nodes in the distributed database environment, proposes a deep deterministic policy gradient reinforcement learning algorithm C-MADDPG based on centralized policy gradient estimator, models the database cluster parameter tuning model based on the partially observable Markov decision process for the distributed database parameter tuning problem, defines the problem to expand the parameter search space, regards the complex competition between database nodes as a black box, and performs mathematical modeling on the observable cooperation for the database cluster parameter tuning model, and finally realizes the distributed database parameter tuning.

[0080] The specific implementation process of the application is as follows.

[0081] The application provides a distributed database parameter adaptive tuning method based on multi-agent deep reinforcement learning, introduces the multi-agent deep reinforcement learning algorithm idea into the database parameter tuning field, explores the cooperation and competition relationship between nodes in the distributed database environment, proposes a deep deterministic policy gradient reinforcement learning algorithm C-MADDPG based on centralized policy gradient estimator, models the database cluster parameter tuning model based on the partially observable Markov decision process for the distributed database parameter tuning problem, defines the problem to expand the parameter search space, regards the complex competition between database nodes as a black box, and performs mathematical modeling on the observable cooperation for the database cluster parameter tuning model, and finally realizes the distributed database parameter tuning. Figure 1 andFigure 2 The application comprises the following steps:

[0082] Step S1, the database cluster explores high concurrency workloads using a behavior policy;

[0083] Step S2, historical experience data is persisted and a cluster state matrix and an action matrix are constructed;

[0084] Step S3, a central training server updates a database cluster Actor Critic network model;

[0085] Step S4, a database cluster parameter tuning model is evaluated.

[0086] To facilitate a detailed description of the application, first, key concepts of reinforcement learning under the distributed database parameter tuning problem need to be defined based on a partially observable Markov decision process:

[0087] Agent. Defined as a distributed database node, one agent corresponds to one database node to be tuned. When implementing the algorithm model, it is necessary to ensure that all agents to be trained share communication, thereby meeting the assumption premise of constructing a centralized Critic network and a centralized policy gradient estimator.

[0088] Environment. Defined as a work load. When training the tuning model, the application selects the Sysbench workload simulation tool which has strong applicability and flexibility, so that the algorithm model can better adapt to the general high-load high-concurrency environment and improve the generalization ability of the model. When evaluating the tuning effect of the algorithm model, the application uses the TPC-C standard OLTP load simulation tool for performance testing. The TPC-C simulates the order system database table in the real production environment, and the association between the database tables is strong.

[0089] State. Defined as the internal state of the database node. In the database parameter tuning problem, the State needs to reflect the current resource consumption and work situation of the database node as much as possible. In order to reduce the number of connections established with each database node and the calculation cost, the application uses a low sampling frequency State calculation method.

[0090] Action. Defined as a heterogeneous parameter combination available for tuning. Among them, the continuous type parameters need to dynamically define the tuning range, and the discrete type parameters dynamically specify possible specific values.

[0091] Reward. Defined as the difference between the performance indicators of the cluster. The present application specifies the observable cooperative relationship as maximizing the performance indicators of the cluster TPS, QPS and 95% Latency. Among them, the change trend of TPS and QPS is similar, and the given Reward calculation weight is the same.

[0092] Observation. Defined as the observable State-Action of other database nodes to be optimized. Ensure that all database nodes to be optimized share communication, that is, meet the partially observable Markov decision process assumption premise, so that the State-Action of all database nodes can be obtained when training the optimization model.

[0093] According to the definition of the partially observable Markov decision process and the C-MADDPG algorithm workflow of the distributed database parameter optimization problem, the distributed database parameter optimization problem can be divided into four steps, mainly including workload environment interaction, persistent historical experience data, Actor Critic network model update, parameter optimization model evaluation, and the present application takes MySQLNDB Cluster 8.0 as the optimization example.

[0094] (1) Workload environment interaction

[0095] When training the C-MADDPG algorithm optimization model, the present application uses the Sysbench workload simulation tool with strong applicability and flexibility, so that the algorithm model can better adapt to the general high-load high-concurrency environment and improve the generalization ability of the algorithm model. When evaluating the effect of the algorithm model, the present application uses the TPC-C standard OLTP load tool for performance testing, which simulates the order system database table in the real production environment, and the correlation between the database tables is strong, which is used for cross-validation test the generalization ability of the C-MADDPG algorithm model. In addition, the performance test time of the database cluster of the present application is set to 60s, wherein the data warm-up time of TPC-C is set to 10s, that is, the total duration of TPC-C performance test is 70s, and the performance test report is not generated in the first 10s of the performance test period.

[0096] The performance test report provided by Sysbench has TPS, QPS and 95% Latency. Among them, TPS represents the number of transactions processed per second, QPS represents the number of queries executed per second, including various add, delete, modify and query SQL statements, and 95% Latency represents that the processing time delay of 95% requests is less than or equal to the Latency. In addition, the present application uses three workload scripts provided by Sysbench, namely oltp_only_read.lua, oltp_read_write.lua and only_write.lua. Among them, the oltp_read_write.lua script is used as the training data set of the tuning algorithm. The performance report provided by TPC-C has tpmC and various order transaction processing performance, and tpmC represents the number of order transactions that can be processed per minute. In this experiment, only tpmC is selected as the performance reference standard. When initializing TPC-C, the present application specifies the number of warehouses as 10, about 200K data; and when initializing the Sysbench test data, the present application specifies the number of tables as 8, the table-size as 100K, and the number of threads as 256. The database workload used by the present application is shown in Table 1.

[0097] Step 1: Add NoiseLinear random noise disturbance to Actor Main network.

[0098] C-MADDPG is based on off-policy and makes full use of historical experience data of database parameter tuning. The execution policy of the agent is divided into behavior policy and target policy. The behavior policy based on Actor Main network is used for agent load environment exploration. The noise network of A3C is added to the output layer of the Actor network as a NoiseLinear noise layer to realize the random disturbance of Action, thereby improving the exploration degree of the policy.

[0099] Step 2: Database node heterogeneous parameter mapping.

[0100] The heterogeneous parameter combination predicted by the Actor Main network through the Tanh layer is converted into a cluster parameter set suitable for the production environment based on the pre-defined parameter value range. Among them, the continuous parameters are converted by linear scaling, and the discrete parameters are obtained by bucket mapping to get the index of the discrete value array. The specific mapping formula is as follows:

[0101]

[0102] The recommended cluster action matrix is mapped to heterogeneous parameters and can be directly distributed to each database node for execution and restart of the target database cluster.

[0103] Step 3: Cluster stress test is started and data is preheated.

[0104] The present application takes the high-concurrency load script oltp_read_write provided by Sysbench as the training environment of the tuning model, the data volume is 800K, that is, 8.5GB, and the simulated thread number is 256. Considering that there is data preheating under high-concurrency load after the database cluster is restarted, that is, the cache page and index cache have not been completely established at this time, the database internal state data and performance index data obtained at this time cannot well reflect the current database working state, therefore, the present application specifies the NDB cluster data preheating time as 15s, and after the data preheating is completed, more accurate State and Reward are obtained.

[0105] Step 4: Data preprocessing.

[0106] In order to reduce the number of connections established with the database and the calculation cost, the present application uses a State calculation mode with a low sampling frequency. The present application first obtains the internal state of all nodes as the database node internal state under low load; after the NDB cluster starts stress testing and the data completes cache preheating, the internal state of the database node is obtained once every stress testing time interval T and finally the average value of each internal state index in State is obtained to obtain the State expectation of the i-th node The calculation mode of the Reward adopted by the present application is that at the beginning of each training round, the performance index r0 of the current stress testing is recorded, and when calculating the difference between the performance indexes in all time steps in the training round, the performance index of each time step is subtracted from r0 based on r0 to obtain R t+1 .

[0107] Step 5: Dynamically predefining the tuning range of heterogeneous parameters.

[0108] After the database internal state State is calculated in step 4, the tunable range of the heterogeneous parameters is dynamically predefined according to the resource occupation of the database node in State, for example, the present application dynamically specifies that the search range of the innodb_buffer_pool_size parameter is 1MB to 75% of the memory according to the memory occupation of the database node. The upper and lower boundary values need to be specified for continuous parameters, and the possible discrete values need to be specified for discrete parameters.

[0109] Step 6: Obtain the five-tuple of database cluster training data.

[0110] S t With R t+1 By performing step 4, the Actor Main network with increased NoiseLinear random noise disturbance obtains S t as an input parameter of the neural network, and obtains A t by forward propagation, and maps A t into a combination of available heterogeneous parameters, which is applied to the database cluster, and the database cluster is subjected to a high-concurrency workload stress test again to obtain S t+1 .

[0111] At this point, the database cluster training data five-tuple D=(S t , A t , R t+1 , S t+1 , A t+1 ) is obtained.

[0112] (2) Persistent historical experience data

[0113] Since the C-MADDPG model designed by the present application is implemented based on Off-Policy, the present application uses a historical experience replay mechanism to persist the database cluster training data five-tuple. At the same time, based on the idea of edge computing, each database node maintains a historical experience replay buffer pool B i , so that the database cluster environment exploration and network model parameter update steps are executed in parallel, and the historical experience replay buffer pool maintained by each database node to be optimized saves the training data five-tuple of the node to explore the workload environment.

[0114] Step 1: Persistent database node parameter optimization historical experience data.

[0115] The present application takes into account the marginal diminishing effect of historical experience replay samples on Reward, and when initializing the historical experience replay buffer pool B i of the database node, the storage upper limit of the historical experience replay buffer pool needs to be specified. The optimization embodiment of the present application specifies that the upper limit is 1000, and when the historical experience replay buffer pool reaches the storage upper limit, the parameter optimization historical experience data that is newly obtained and has more training value will replace the old experience data, so that the algorithm model can be continuously trained and iterated according to the historical experience samples with more optimization value. Since only the four-tuple (S t , R t+1 , S t+1 , A t+1 ) is used as training data in the network model update in step S3, in order to reduce the storage pressure of the node, the historical experience replay buffer pool of each database node only needs to persist the four-tuple training data.

[0116] Step 2: Aggregate the cluster state matrix and action matrix.

[0117] Before training the tuning model, each database node sharing the communication randomly extracts a batch of training data quadruples stored in step 1 from the historical experience replay buffer pool with uniform distribution. In order to satisfy the unique fixed point theorem of Bellman's optimal formula under the random approximation theory as much as possible, the number of traversals of the historical experience replay sample needs to be as large as possible, so the number of samples extracted by the present application increases with the increase of the historical experience replay sample, and the upper limit of the extraction specified by the tuning embodiment of the present application is 128. The database node transmits the part of the observable Observation set persisted in the historical experience replay buffer pool to the central training server, constructs the cluster state matrix and the action matrix, and trains the database cluster tuning model.

[0118] The database cluster state matrix and action matrix formula is as follows:

[0119]

[0120] (3) Actor-Critic network model update.

[0121] The central training server aggregates the part of the observable Observation transmitted by each database node, and constructs the cluster state matrix and the action matrix, that is, the historical experience aggregation sample, as the input parameter of the Actor-Critic network model, and uses the Adam optimizer based on the stochastic gradient descent algorithm to update the network model parameter by back propagation. The Actor-Critic network architecture designed by the present application is shown in Figure 3 Since the database cluster parameter tuning training process is relatively slow, it is difficult to obtain training data, so the present application uses the idea of incremental training. After the end of a round of incremental training, the parameters of all networks and the experience replay buffer pool are automatically saved in the C-MADDPG algorithm process designed and implemented by the present application, and are used in the next incremental training.

[0122] Step 1: Remove the NoiseLinear random noise of the Actor Main network.

[0123] The present application adds a noise network based on A3C to the output layer of the Actor network to realize the random disturbance of the Action, thereby improving the exploration degree of the policy. However, when training and evaluating the model, the noise network needs to be removed from the output layer to improve the stability of the model training.

[0124] Step 2: Update the Critic Main network model parameter.

[0125] Since the Critic Main network needs to be used after updating the model parameters of the Actor Main network, the model parameters of the Critic Main network are first updated. The loss function formula of the Critic Main network is defined as follows:

[0126]

[0127] Since the present application accelerates the network model training speed by using the batch processing historical experience aggregation sample method, the BatchNorm1d batch processing layer is added to the Main network and the Target network.

[0128] Step 3: Construct a centralized policy gradient estimator.

[0129] The centralized policy gradient estimator aggregates all the Actor Main networks of the agents to use the S t Forward propagation output result as an input parameter Centralized estimation of the real value of the cluster action matrix.

[0130] Step 4: Update the model parameters of the Actor Main network.

[0131] The loss function formula of the Actor Main network is defined as follows:

[0132]

[0133] In order to facilitate the mapping of the subsequent network model forward propagation result to the actual available database heterogeneous parameter combination, the output layer of the Actor network is added with Tanh, so that the estimation result is scaled to [-1, 1].

[0134] Step 5: Softly update the model parameters of the Actor Target and Critic Target networks.

[0135] In order to make the Actor-Critic network model training more stable, the present application uses the soft update method to update the Target network model parameters using the Main network model parameters after a fixed time step, and the soft update parameter formula of the Actor Target and Critic Target network model is as follows:

[0136]

[0137] θ Q′ And θ μ′The parameter results after soft updating of Critic Target and Actor Target networks, respectively; τ is used to control the network speed of Target network updating replication Main, and the value range is [0, 1].

[0138] Step 6: The central training server publishes the Actor Main network model

[0139] Since the environment exploration and model updating steps are independent and parallel, the central training server needs to synchronize the updated Actor network model parameters to each database node for further environment exploration.

[0140] Tuning effect comparison experiment

[0141] In order to better analyze and evaluate the feasibility of multi-agent reinforcement learning algorithm to solve the problem of distributed database parameter tuning, this section designs and implements the deep reinforcement learning algorithm DDPG suitable for single agent and the deep reinforcement learning algorithm MADDPG suitable for multi-agent, and conducts parameter tuning comparison experiments with the C-MADDPG tuning algorithm designed in this paper under different workloads, where the performance test tools of workloads are Sysbench and TPC-C.

[0142] The central training server of this experiment is completed on a computer with Windows 10 system, with 11th generation Intel Core i7-11800 processor, RTX 3060 graphics card, and 16GB memory capacity. The MySQL NDB Cluster distributed database cluster is built with a virtual machine with CentOS 7.9 system, and each database node to be tuned is configured with 4-core CPU and 4GB memory capacity.

[0143] This paper uses two kinds of mainstream database workload performance test tools, namely Sysbench and TPC-C, and the simulation of all types of database workload purposes and detailed description are shown in Table 1.

[0144] Table 1 Database workloads

[0145]

[0146] Comparison and analysis of training process

[0147] The DDPG algorithm suitable for single-agent environment in CDBTune, the classic MADDPG algorithm and the C-MADDPG tuning algorithm designed in the present application are compared and analyzed by experiments. After 40 rounds of incremental training iteration, the loss function value change curves of the Actor network and the Critic network of the three tuning algorithm models are respectively as shown in Figure 4 (a) and Figure 4 (b).

[0148] The loss function value change amplitude of the Actor network and the Critic network of the MADDPG and C-MADDPG tuning algorithms is large at the beginning of iteration, and shows an obvious decreasing trend. It shows that in the early stage of training iteration, the behavior policy with added noise network can continuously explore some cluster parameter combinations with good tuning effect, and the algorithm model can learn useful parameter features from the historical experience replay buffer. After about 30 rounds of algorithm model training iteration, it can be found that the loss function value of the Critic network tends to be stable and remains at about 0.5, indicating that the error between the true Q value and the estimated value tends to be stable, and the calculated value of the Q value of each experience replay training sample State-Action tends to converge. At this time, the Critic network can better predict the Q value of each historical experience aggregation sample; the loss function value of the Actor network is almost unchanged between about 20 rounds and 25 rounds, indicating that at this time the model finds fewer cluster parameter combinations with better tuning effect than the current parameter configuration, and the parameter features learned by the model for historical experience samples have tended to converge. After 25 rounds, it can be found that the loss function value of the Actor network decreases gradually, indicating that the Q value is increasing, which indicates that the tuning effect of the cluster parameter combination predicted by the current Actor network is continuously optimized, and it also indicates that the behavior policy at this time explores some parameter combinations with better tuning effect, thereby jumping out of the local optimal solution. This also reflects a significant advantage of deep reinforcement learning algorithm, which can continue to randomly explore some unknown knowledge according to the current learning experience, thereby continuously approaching the global optimal solution.

[0149] Similar to the C-MADDPG algorithm, the loss function values of the Actor network and the Critic network of the DDPG algorithm change greatly at the beginning of the training iteration. Between 15 and 20 rounds, it can be found that both the Actor network and the Critic network show a convergent trend, indicating that the behavior policy has not been able to explore some parameter combinations with better tuning effects at this time. Between 20 and 25 rounds, the loss function value of the Actor network of the DDPG algorithm model can continue to show a downward trend, and the change range of the loss function value of the Critic network is also large. This phenomenon can also explain the advantage of deep reinforcement learning in finding the global optimal solution. Between 30 and 35 rounds, the loss function values of the Actor network and the Critic network gradually show a convergent trend.

[0150] In addition, the loss function value change curve of the Actor network and the Critic network of C-MADDPG is smoother than that of DDPG and MADDPG, which indicates that the DDPG algorithm model may be affected by many factors during network training. This paper speculates that if only single-node parameter tuning is performed in a distributed environment, the load imbalance and distributed lock competition of other NDB nodes have a more obvious impact on the internal interaction of the nodes, resulting in more complex features that the network needs to learn. The C-MADDPG algorithm model takes into account the cooperation and competition between nodes, and simultaneously trains the parameter combinations of the entire cluster, enabling NDB nodes to be optimized collaboratively. Both algorithms have certain global optimization capabilities and can jump out of the local optimal convergent trend.

[0151] Comparison of tuning processes

[0152] During the iteration process of DDPG, MADDPG, and C-MADDPG tuning algorithms, this paper records the performance indicators obtained by the behavior policy executing the exploration strategy at each step to obtain a series of tuning process performance indicator trend graphs. Since the behavior policy adds a noise network for random exploration, rather than noise with an arbitrary distribution, the tuning performance of the behavior policy will improve synchronously with the iteration of the Actor master network parameters. Therefore, the tuning process performance indicators obtained by the behavior policy exploration can reflect the tuning performance changes of the entire algorithm model with training iteration to some extent.

[0153] This section finally gives the performance indicator trend graphs of TPS, QPS, and 95% Latency of the three tuning algorithms under read-write load workloads. The change trend of TPS is actually almost the same as that of QPS. Therefore, this section only analyzes the tuning change trend of QPS and 95% Latency.

[0154] Figure 5 (a) and Figure 5(b) represents the change trend of the QPS performance index of the three tuning algorithms under read-write load workloads, respectively, where the abscissa is the number of rounds, that is, the training rounds; the ordinate represents the average value of the QPS performance index obtained by the database cluster using the behavior strategy to explore the environment in each training round. From Figure 4 (b) It can be seen that the DDPG tuning model is difficult to converge to the optimal tuning effect under the incremental training of 40 rounds, and is stable at about 400 ms, but the single-node 95% Latency minimum value is about 360 ms; the MADDPG and C-MADDPG algorithm models have gradually tended to converge in the tuning iteration process of 40 rounds. Compared with the loss function value change trend described earlier Figure 1 , the performance index of the two algorithm models fluctuates greatly at the beginning of the training iteration, at which time the historical experience replay sample is small and the model fitting effect is poor. With the continuous training of the model, the tuning effect of the model gradually stabilizes, and the change gradually decreases. Since the Actor main network parameters in the behavior strategy contain random noise parameters generated by the noise network, the performance outliers in the training process represent that the agent explores and learns new strategies in unknown parameter space.

[0155] Algorithm time-consuming comparison

[0156] This paper records the network backpropagation update time, recommended and configured cluster parameter combination time, database restart time, Sysbench pressure test cluster total time, and incremental training total time in the iterative training process of the three tuning algorithms. The recommended cluster parameter combination time in the DDPG algorithm training process is to apply the recommended parameter combination to a certain database node, and the database restart time refers to restarting a certain database node service; the MADDPG and C-MADDPG are to apply the recommended parameter combination to the NDB nodes of the entire cluster, which requires restarting the NDB cluster service. This paper will compare and analyze the time-consuming of the four algorithm steps, and summarize the five time indicators recorded above in Table 2, with the time unit being seconds. Although MADDPG and C-MADDPG are more time-consuming in one incremental iteration training process, they can optimize all database nodes to be tuned, while DDPG needs to train a tuning algorithm model for each node individually, and the total training time is NT, where T is the total training time of the DDPG model.

[0157] Table 2 Time-consuming of each step of tuning algorithm (seconds)

[0158]

[0159] Tuning effect comparison and evaluation

[0160] The NDB database cluster is subjected to read-write load, read-only load and write-only load high concurrency stress testing provided by the Sysbench tool. The cluster is subjected to performance tuning using NDB default parameter configuration, random parameter sampling strategy based on uniform distribution, DDPG parameter tuning configuration, MADDPG parameter tuning configuration and C-MADDPG parameter tuning configuration. The tuning effect comparison chart of TPS, 95% Latency and QPS performance indicators is Figure 6 (a), Figure 6 (b) and Figure 5 (c), wherein the abscissa is the Sysbench workload type, and the ordinate is the specific value of the performance indicator; in addition, the TPC-C stress testing tool is used to cross-verify and evaluate the tuning algorithm effect, and the tuning performance indicator is tpmC. The final tuning result comparison bar Figure 6 (d). In order to more finely obtain the tuning effect of C-MADDPG and DDPG, the parameter tuning model evaluation algorithm according to algorithm 5 is used to calculate the tuning effect proportion of the three tuning algorithm models under four workloads. The tuning evaluation experiment is performed for 10 times in succession for each of the four workloads to obtain the average value. Finally, the adaptive tuning result comparison table 3 is obtained.

[0161] Table 3 Comparison of distributed database parameter tuning results

[0162]

[0163] Under four different database workloads, DDPG, MADDPG and C-MADDPG can better optimize the performance of the database cluster, while the random parameter strategy performs poorly, showing negative tuning. The performance of the MADDPG tuning algorithm is lower than that of the DDPG and C-MADDPG tuning algorithms under most load environments, while the performance effect of the C-MADDPG tuning algorithm is better than that of the DDPG. However, under read-write load, the tuning performance of C-MADDPG is slightly lower than that of DDPG. This shows that C-MADDPG cannot learn the internal state characteristics of the database nodes well under high read and write load, and cannot accurately predict the cluster parameter combination.

[0164] Table 4 Comparison of cluster recommended parameter combinations under oltp_read_write workload

[0165]

[0166] To find out the reason why MADDPG and C-MADDPG tuning algorithms perform poorly under high read and write workloads, this paper records the actual recommended cluster parameter combinations of DDPG, MADDPG, and C-MADDPG tuning algorithms under high read and write workloads, and selects the knob parameters that have a greater impact on database read performance. The comparison results are shown in Table 3, where the innodb_random_read_ahead and innodb_adaptive_hash_index parameters represent the database pre-read page and adaptive hash index, respectively. In the case of limited server resources, closing these two parameters can effectively improve the performance of the cluster, because the database nodes cannot pre-read enough pages to improve performance and establish enough hash partitions. MADDPG and C-MADDPG tuning algorithms do not learn this knowledge under read-write mixed workloads, and blindly turn on these two parameters, thereby reducing the performance of the cluster. This shows that due to the aggregation of the cluster state matrix and the action matrix, the parameter space combination that the algorithm needs to explore becomes larger, and it cannot better predict some parameters. Therefore, how to compress and reduce the dimensionality of the cluster state matrix and the action matrix and reduce the complexity of the Actor-Critic network parameters is the further research direction of this paper.

[0167] Experimental conclusions

[0168] Through the detailed analysis of the above details of the training process and tuning effect of DDPG and C-MADDPG, the following algorithm experimental conclusions are summarized in this section:

[0169] (1) From the algorithm training process, all three tuning algorithms have certain global optimization ability and can jump out of the local optimal solution trap. Among them, the DDPG algorithm does not take into account the complex mutual internal influence between nodes, resulting in a relatively steep change curve of Actor Loss and Critic Loss in the training process, and the training process is not as stable as the C-MADDPG algorithm.

[0170] (2) From the comparative analysis results of the tuning process, MADDPG and C-MADDPG have tended to converge in the incremental training of 40 rounds, while the DDPG tuning algorithm still needs further training.

[0171] (3) From the time-consuming comparison analysis results of the algorithm, since DDPG needs to pre-train the algorithm model for each database node in the cluster, while C-MADDPG can cooperatively optimize the parameter space of the entire cluster. Therefore, C-MADDPG has certain advantages in database clusters with a large number of training nodes.

[0172] (4) From the comparison results of the tuning effects, the C-MADDPG pre-training algorithm model has better tuning performance than CDBTune and MADDPG under the read-only load, the write-only load and the TPC-C work load provided by Sysbench, but under the read-write mixed load, the C-MADDPG tuning performance is slightly lower than CDBTune. Through the comparative analysis of the cluster parameter recommendation results, it is speculated that due to the high dimension of the cluster state matrix and the action matrix of C-MADDPG, the Actor-Critic network is relatively complex, which leads to C-MADDPG falling into a local optimal solution, and the compression and dimension reduction of the cluster state matrix and the cluster action matrix are the work worth studying in the future.

[0173] The application further provides a database cluster monitoring system design system, comprising a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor, and when the processor executes the computer program instructions, the method steps described above can be realized.

[0174] The application further provides a computer readable storage medium, which stores computer program instructions capable of being executed by a processor, and when the processor executes the computer program instructions, the method steps described above can be realized.

[0175] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0176] The present application is described with reference to flowcharts and / or block diagrams according to the method, device (system) and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks.

[0177] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 of the flow or flows and / or blocks Figure 1 of the block or blocks specified in the flow.

[0178] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 of the flow or flows and / or blocks Figure 1 of the block or blocks specified in the flow.

[0179] The foregoing is considered as illustrative only of the principles of the application. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the application to the exact construction and operation described. Accordingly, all such variations are intended to be included within the scope of present application as defined in the claims below.

Claims

1. A distributed database parameter adaptive tuning method based on multi-agent deep reinforcement learning, characterized in that, This paper introduces the concept of multi-agent deep reinforcement learning algorithms into the field of database parameter tuning, explores the cooperative and competitive relationships among nodes in a distributed database environment, and proposes a deep deterministic policy gradient reinforcement learning algorithm C-MADDPG based on a centralized policy gradient estimator. The distributed database parameter tuning problem is modeled based on a partially observable Markov decision process. The problem definition, at the cost of expanding the parameter search space, treats the complex competitive interactions between database nodes as a black box, and mathematically models the observable cooperative interactions for database cluster parameter tuning, ultimately achieving distributed database parameter tuning.

2. The method for adaptive tuning of distributed database parameters based on multi-agent deep reinforcement learning according to claim 1, characterized in that, The specific implementation of modeling the distributed database parameter tuning problem based on a partially observable Markov decision process for database cluster parameter tuning is as follows: Agent: Defined as a distributed database node, with one agent corresponding to one database node to be tuned; all agents to be trained share communication to meet the assumptions of building a centralized Critic network and a centralized policy gradient estimator. Environment: Defined as workload. When training the database cluster parameter tuning model, the Sysbench workload simulation tool is selected. When evaluating the tuning effect of the database cluster parameter tuning model, the TPC-C standard OLTP load simulation tool is used for performance testing. State: Defined as the internal state of a database node. In database parameter tuning, the State calculation method is used to reflect the current resource consumption and working status of the database node. Action: Defined as a combination of heterogeneous parameters that can be tuned in the database. Continuous parameters need to be dynamically predefined in terms of tuning range, while discrete parameters have their possible values ​​dynamically specified. Reward is defined as the difference in the cluster's stress test performance metrics. It specifies that the observable cooperative relationship is to maximize the cluster's TPS, QPS, and 95% Latency performance metrics, where TPS and QPS have similar trends and the given Reward calculation weights are the same. Observation: Defined as the observable State-Action of other database nodes to be tuned, ensuring that all database nodes to be tuned share communication, that is, satisfying the assumption of partially observable Markov decision process, so that the State-Action of all database nodes can be obtained when training and tuning the model.

3. A distributed database parameter adaptive tuning method based on multi-agent deep reinforcement learning according to claim 1 or 2, characterized in that, Includes the following steps: Step S1: The database cluster uses Behavior Policy to explore high-concurrency workloads; Step S2: Persist historical experience data and construct the database cluster state matrix and action matrix; Step S3: The central training server updates the Actor-Critic network model of the database cluster. Step S4: Database cluster parameter tuning model evaluation.

4. The method for adaptive tuning of distributed database parameters based on multi-agent deep reinforcement learning according to claim 3, characterized in that, Step S1 is implemented as follows: Step S11: Add NoiseLinear random noise perturbation to the Actor Main network; Step S12: Mapping heterogeneous parameters of database nodes; Step S13: Start the database cluster stress test and perform data preheating; Step S14: Data preprocessing; Step S15: Dynamically predefine the tuning range of heterogeneous parameters; Step S16: Obtain the quintuple of training data for the database cluster.

5. The distributed database parameter adaptive tuning method based on multi-agent deep reinforcement learning according to claim 4, characterized in that, The quintuple D of the database cluster training data is: D=(S t ,A t ,R t+1 ,S t+1 ,A t+1 ) (1) C-MADDPG, based on off-policy, divides the agent's execution policy into behavior policy and target policy, and uses historical experience replay buffer pool to persist training data in the database cluster; in equation (1), S t This represents the cluster state matrix obtained by the multi-agent interaction with the workload environment at time step t in the database cluster parameter tuning model; if this is the first training round and t=1, then A t This represents the default cluster action matrix, i.e., the initial parameters of the database system. Otherwise, it means that at the previous time step, the multi-agent network used the ActorMain network with an added NoiseLinear noise layer, i.e., the Behavior Policy, to explore the cluster action matrix obtained from the workload environment, and at time step t, the multi-agent will... t As the input parameters of the Actor Main network, the forward propagation result of the Actor Main network through the Tanh layer is A. t+1 S t+1 This indicates that multiple agents are executing A in a distributed manner. t+1 That is, database cluster application configuration A t+1 After parameter matrix generation and restart, the cluster state matrix obtained through interaction between the multi-agent system and the workload environment; R t+1 This indicates that at time step t, multiple agents execute A in a distributed manner. t With A t+1 Then, the difference between the weighted average of the performance index data obtained from the two environmental explorations. To map the Tanh layer output of the Actor Main network, which serves as the Behavior Policy, to a combination of heterogeneous parameters from a usable database, the cluster action matrix is ​​processed using the following formula: Equation (2) is based on the linear scaling method to process continuous parameters, and Equation (3) is based on the bucket mapping method to process discrete parameters. and These represent the continuous and discrete parameters in the cluster action matrix recommended by the Behavior Policy at time step t, respectively; in equation (3), L (L>0) represents the number of values ​​for the discrete parameter. and This represents the matrix of maximum and minimum values ​​of continuous parameters in the recommendation cluster action matrix: m represents the number of continuous parameters to be tuned for a single database node, and n represents the total number of database nodes to be tuned. and This represents the maximum value of the j-th continuous parameter to be tuned in the i-th database node to be tuned.

6. The distributed database parameter adaptive tuning method based on multi-agent deep reinforcement learning according to claim 3, characterized in that, Step S2 is implemented as follows: Step S21: Persist the historical experience data for database node parameter tuning; Step S22: Aggregate and construct the database cluster state matrix and action matrix.

7. The method for adaptive tuning of distributed database parameters based on multi-agent deep reinforcement learning according to claim 6, characterized in that, The formulas for the database cluster state matrix and action matrix are as follows: Where S is the cluster state matrix, m represents the number of internal state indicators for a single database node, n represents the total number of database nodes to be tuned, and s ij This represents the j-th internal status metric data value of the i-th database node to be tuned in the target database cluster; A is the cluster action matrix, m represents the number of heterogeneous parameters to be tuned for a single database node, n represents the total number of database nodes to be tuned, and a ij This represents the value of the j-th heterogeneous parameter to be tuned in the i-th database node to be tuned in the target database cluster. Before executing the update of the Actor-Critic network model for the database cluster, the central training server needs to aggregate the persistent state and action experience data in the historical experience replay buffer pool of all database nodes to be tuned, construct the cluster state matrix and action matrix, and define them as historical experience aggregated samples, which are used as Actor-Critic network training data.

8. The distributed database parameter adaptive tuning method based on multi-agent deep reinforcement learning according to claim 3, characterized in that, Step S3 is implemented as follows: Step S31: Remove the NoiseLinear random noise from the Actor Main network; Step S32: Update the parameters of the Critic Main network model; Step S33: Construct a centralized policy gradient estimator; Step S34: Update the parameters of the Actor Main network model; Step S35: Soft update the network model parameters of Actor Target and Critic Target; Step S36: The central training server publishes the Actor Main network model.

9. The method for adaptive tuning of distributed database parameters based on multi-agent deep reinforcement learning according to claim 8, characterized in that, The C-MADDPG divides the Actor-Critic network into a Main network and a Target network based on the Bellman optimal formula, including the Actor Main network, Critic Main network, Actor Target network, and Critic Target network. The loss function formula for the Critic Main network is defined as follows: Let θ be the loss function of the Critic Main network for the i-th agent, and calculate the sum of squared errors of the historical experience aggregated samples TDError, where θ Q The parameters for fitting the Critic Main network are: Q represents the action value, i.e., the Q-value, which indicates that the fitted parameter is used for value updates; S represents the number of aggregated historical experience samples in the batch; y j This indicates that the Critic Main network will S t With A t+1 The true value of Q is calculated using the j-th historical experience aggregated sample as network input; Q i This represents the TD Target of the j-th historical experience aggregated sample calculated by the i-th agent using the Actor Target network and the Critic Target network, where As the input to all Actor-Target networks, S t+1 With the output of the Actor Target network The inputs of the Critic Target network for the i-th agent are used together, and the output of the Critic Target network is the TD Target, which represents the maximum Q value of the current environment exploration trajectory. N is the number of agents. Before updating the Actor network model parameters, a centralized policy gradient estimator is constructed for the cluster action matrix. In the central training server, the forward propagation aggregation results of other agents' Actor Main networks replace the historical experience aggregation samples. Therefore, the loss function formula for the Actor Main network is defined as follows: J(θ i μ Let θ be the loss function of the Actor Main network for the i-th agent, and calculate the negative of the Q-value, where θ μ Let a = μ(s) represent the fitting parameters of the Actor Main network, and let a = μ(s) represent the fitting function mapping relationship between actions and states in policy updates; in formula (9), the centralized policy gradient estimator aggregates the Actor Main network of all agents using S t Forward propagation output as input parameters Centrally estimate the true value of the cluster action matrix and compare it with S t Together, they serve as input to the i-th Critic Main network to estimate the true value of Q; in addition, the central training server needs to synchronize the updated Actor Main network model parameters to the database nodes for further environment exploration. A soft update approach is used to update the parameters of the Target network model using the parameters of the Main network model after a fixed time step. The formulas for the soft update of the parameters of the Actor Target and Critic Target network models are as follows: θ Q′ With θ μ′ The parameters are the result of soft updates performed on the Critic Target and Actor Target networks, respectively; τ is used to control the network rate at which the Target network updates and replicates the Main network, and its value ranges from [0, 1].

10. The method for adaptive tuning of distributed database parameters based on multi-agent deep reinforcement learning according to claim 3, characterized in that, Step S4 is implemented as follows: Step S41: Multi-agent distributed execution of A t ; Step S42, obtain (S t R t A t+1 ); Step S43: Multi-agent distributed execution of A t+1 ; Step S44: Calculate R t+1 ; Step S45: Calculate the performance improvement ratio. The formula for calculating the performance improvement ratio is as follows: (S t R t A t+1 The default cluster parameter matrix A is executed by multiple agents in a distributed manner. d Subsequently, the results were obtained through cluster performance stress testing, where R... t This represents the weighted average of the performance metrics obtained from this performance stress test; the parameter matrix A of the multi-agent distributed execution cluster. t+1 Afterwards, perform cluster performance stress testing again to obtain the weighted average of performance metrics and R. t+1 ; r represents the performance improvement percentage, with a value range of [-1, 1]. (The last part, "R", appears to be a typo and can be omitted.) t+1 With R t Calculated.

Images