A server parameter optimization method and device

Abstract

The present disclosure provides a server parameter optimization method and device, wherein the method comprises: obtaining a state parameter set and an execution action set of a server cluster, the state parameter set and the execution action set having a time label; based on the state parameter set and the execution action set, determining a local evaluation prediction parameter by using a multi-agent reinforcement learning algorithm; based on the state parameter set and the execution action set, determining a global evaluation prediction parameter by using an adaptive dynamic programming model; and based on the local evaluation prediction parameter and the global evaluation prediction parameter, optimizing the execution action set. The method of the present disclosure determines the local evaluation prediction parameter and the global evaluation prediction parameter corresponding to the server cluster by using the QMIX model and the ADP model, thereby realizing the automatic optimization of the execution action set of the server cluster, so as to improve the stability and service efficiency of the server cluster.

Description

Technical Field

[0001] This disclosure relates to the field of dynamic programming, and in particular to a method and apparatus for optimizing server parameters. Background Technology

[0002] Kafka server clusters, as a partitioned distributed server cluster, have high message queue throughput, but the data volume fluctuates significantly over time. Continuously using the same parameters in the same task cannot adapt to these fluctuations, easily leading to significant resource idleness or abnormally high system load, which is not efficient for task execution. Current technology requires parameter configuration based on the operational scenario and necessitates manual configuration, resulting in low efficiency. Currently, more and more big data operators are seeking automated adjustments to significantly improve operational efficiency and minimize resource waste. Summary of the Invention

[0003] This disclosure provides a server parameter optimization method and apparatus to solve problems in related technologies, realize automatic optimization of the execution action set of Kafka server cluster, and improve the stability and service efficiency of server cluster.

[0004] A first aspect of this disclosure proposes a server parameter optimization method, comprising: obtaining a set of state parameters and a set of execution actions of a server cluster, wherein the set of state parameters and the set of execution actions have time labels; determining local evaluation prediction parameters based on the set of state parameters and the set of execution actions using a multi-agent reinforcement learning algorithm; determining global evaluation prediction parameters based on the set of state parameters and the set of execution actions using an adaptive dynamic programming model; and optimizing the set of execution actions based on the local evaluation prediction parameters and the global evaluation prediction parameters.

[0005] In some embodiments of this disclosure, obtaining the state parameter set and execution action set of the server cluster includes: using the monitoring component of the server cluster to obtain the state parameters and execution actions of the server cluster; and storing the state parameters and execution actions in a time series database to obtain the state parameter set and execution action set.

[0006] In some embodiments of this disclosure, determining local evaluation prediction parameters based on a set of state parameters and a set of execution actions using a multi-agent reinforcement learning algorithm includes: determining a local state vector based on the set of state parameters; determining single-agent prediction parameters based on the local state vector and the set of execution actions using a pre-trained deep Q-learning network included in the single agent of the multi-agent reinforcement learning algorithm; and weighted summing of the single-agent prediction parameters to obtain local prediction parameters.

[0007] In some embodiments of this disclosure, the method further includes: determining empirical evaluation parameters based on local state vectors and a set of executed actions, using a deep Q-learning network to be trained within a deep Q-learning network; extracting empirical evaluation parameters based on time parameters to obtain training evaluation parameters using the deep Q-learning network to be trained; determining target evaluation parameters based on the extracted empirical evaluation parameters, using a target deep Q-learning network within a multi-agent reinforcement learning algorithm; and training the deep Q-learning network to be trained with a preset objective function and a preset step size based on the training evaluation parameters and the target evaluation parameters to obtain a pre-trained deep Q-learning network.

[0008] In some embodiments of this disclosure, determining global evaluation prediction parameters based on a set of state parameters and a set of execution actions using an adaptive dynamic programming model includes: determining a global state vector based on the set of state parameters and using time labels; and determining global estimation prediction parameters based on the global state vector and the set of execution actions using a value function estimator in the adaptive dynamic programming model.

[0009] In some embodiments of this disclosure, optimizing the set of execution actions based on local evaluation prediction parameters and global evaluation prediction parameters includes: performing a weighted summation of the local evaluation prediction parameters and global prediction parameters to determine the fused evaluation prediction parameters; and optimizing the set of action parameters based on the execution actions corresponding to the fused evaluation prediction parameters.

[0010] In some embodiments of this disclosure, the set of state parameters includes at least one of the following: producer throughput, consumer throughput, server latency, health status of server partitions, and health status of server backups; the set of execution actions includes at least one of the following: batch size, batch wait time, consumer pull size, consumer pull interval, and number of storage partitions.

[0011] A second aspect of this disclosure provides a server parameter optimization apparatus, comprising: an acquisition unit for acquiring a set of state parameters and a set of execution actions of a server cluster, the set of state parameters and the set of execution actions having time labels; a first evaluation unit for determining local evaluation prediction parameters based on the set of state parameters and the set of execution actions using a multi-agent reinforcement learning algorithm; a second evaluation unit for determining global evaluation prediction parameters based on the set of state parameters and the set of execution actions using an adaptive dynamic programming model; and an optimization unit for optimizing the set of execution actions based on the local evaluation prediction parameters and the global evaluation prediction parameters.

[0012] A third aspect of this disclosure provides an electronic device including: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor, when running the computer program, performs the method described in the first aspect of this disclosure.

[0013] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described in the first aspect of this disclosure.

[0014] A fourth aspect of this disclosure provides a computer program product including a computer program that, when executed by a processor, implements the methods described in the first aspect of this disclosure.

[0015] In summary, the server parameter optimization method proposed in this disclosure includes: obtaining a set of state parameters and a set of execution actions for a server cluster, wherein the state parameter set and the set of execution actions have time labels; determining local evaluation prediction parameters based on the state parameter set and the set of execution actions using a multi-agent reinforcement learning algorithm; determining global evaluation prediction parameters based on the state parameter set and the set of execution actions using an adaptive dynamic programming model; and optimizing the set of execution actions based on the local evaluation prediction parameters and the global evaluation prediction parameters. This method, by utilizing the QMIX model and the ADP model to determine the local evaluation prediction parameters and global evaluation prediction parameters corresponding to the server cluster, achieves automatic optimization of the set of execution actions for the server cluster, thereby improving the stability and service efficiency of the server cluster.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0018] Figure 1 A flowchart of a server parameter optimization method provided in this disclosure embodiment;

[0019] Figure 2 A flowchart illustrating yet another server parameter optimization method provided in this disclosure embodiment;

[0020] Figure 3 A flowchart illustrating another server parameter optimization method provided in this disclosure embodiment;

[0021] Figure 4 A flowchart illustrating another server parameter optimization method provided in this disclosure embodiment;

[0022] Figure 5 This is a schematic diagram of the structure of a server parameter optimization device provided in an embodiment of the present disclosure;

[0023] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0024] Embodiments of this disclosure are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0025] Kafka server clusters, as a partitioned distributed server cluster, have high message queue throughput, but the data volume fluctuates significantly over time. Continuously using the same parameters in the same task cannot adapt to these fluctuations, easily leading to significant resource idleness or abnormally high system load, which is not efficient for task execution. Current technology requires parameter configuration based on the operational scenario and necessitates manual configuration, resulting in low efficiency. Currently, more and more big data operators are seeking automated adjustments to significantly improve operational efficiency and minimize resource waste.

[0026] Therefore, the relevant solutions have the following drawbacks:

[0027] 1. Business scenarios are somewhat unpredictable; if the business scenario provided by operations personnel deviates from the optimal configuration, it will be impossible to obtain the best configuration. If this incorrect configuration is used, it cannot be corrected during the entire task execution process; only the expected performance will be observed. In this case, it is necessary to manually stop the business, re-enter the business scenario into the model to obtain the parameter configuration, and then restart the task.

[0028] 2. Kafka, as a commonly used message queue, may experience fluctuations in data volume over time. Therefore, using the same parameters for the entire task executed through a Kafka-deployed server cluster is somewhat unreasonable. Existing technologies often lack automated decision-making mechanisms and cannot intelligently adjust parameters based on actual needs and operational conditions.

[0029] 3. Current technologies do not provide a way to evaluate the performance of Kafka distributed clusters; they are mostly based on single performance results and do not take a comprehensive approach.

[0030] The present disclosure will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0031] Figure 1 This is a flowchart illustrating a server parameter optimization method provided in an embodiment of this disclosure. Figure 1 As shown, this includes steps 101-104.

[0032] Step 101: Obtain the set of status parameters and the set of actions to be executed for the server cluster.

[0033] In some embodiments, the acquired set of state parameters and set of execution actions of the server cluster are time-stamped, so that the parameters under non-continuous time can be used to train the model and improve the prediction accuracy of the model.

[0034] In some optional embodiments, the monitoring component of the server cluster can be used to obtain the status parameters and actions of the server cluster; and the status parameters and actions can be stored in a time-series database, thereby assigning time tags to the status parameters and actions to achieve the acquisition of the set of status parameters and the set of actions.

[0035] For example, the Kafka Exporter component can be installed on each server in the cluster, enabling it to monitor the Kafka cluster's status and execute actions. It then converts these actions into a format recognizable by Prometheus and exposes them to Prometheus via an HTTP interface.

[0036] Prometheus stores the acquired metric data in its own time-series database. Using Prometheus's query language (PromQL), Kafka's status metrics can be queried and analyzed. This allows devices implementing this solution to access Prometheus to obtain sets of status parameters and action sets.

[0037] In some embodiments, the set of status parameters includes at least one of the following: producer throughput, consumer throughput, server latency, health status of server partitions, and health status of server backups; the set of execution actions includes at least one of the following: batch size, batch wait time, consumer pull size, consumer pull interval, and number of storage partitions.

[0038] It should be understood that the state parameter set may include local state parameters and global state parameters, or it may only include global state parameters, and local state parameters may be obtained by processing the global state parameters.

[0039] Step 102: Based on the set of state parameters and the set of actions to be executed, a multi-agent reinforcement learning algorithm is used to determine the local evaluation and prediction parameters.

[0040] In some embodiments, a local state vector of the server cluster can be determined based on a set of state parameters. This local state vector and the set of actions to be executed are then input into each single agent in the multi-agent reinforcement learning algorithm to obtain single agent prediction parameters. These single agent prediction parameters are then used to determine local evaluation prediction parameters.

[0041] It should be understood that the local prediction parameter is the predicted score when the server cluster performs a certain action in the action parameter set under the local state vector. The local prediction parameter can be used to indicate the rationality of the server cluster performing a certain action under the local state vector.

[0042] Step 103: Based on the set of state parameters and the set of execution actions, determine the global evaluation and prediction parameters using an adaptive dynamic programming model.

[0043] In some embodiments, the global state vector of the server cluster can be determined based on the set of state parameters, and then the global state vector and the set of execution actions can be input into the adaptive dynamic programming model to determine the global evaluation prediction parameters.

[0044] It should be understood that the global prediction parameter is the predicted score for a server cluster to perform a certain action in the action parameter set under the global state vector. The global prediction parameter can be used to indicate the rationality of the server cluster performing a certain action under the global state vector.

[0045] Step 104: Optimize the set of actions to be executed based on the local evaluation prediction parameters and the global evaluation prediction parameters.

[0046] In some embodiments, the local evaluation prediction parameters and the global evaluation prediction parameters can be weighted and summed, or the local evaluation prediction parameters and the global evaluation prediction parameters can be summed directly to obtain the fused evaluation prediction parameters. The execution action corresponding to the parameter with the largest value in the fused evaluation prediction parameters is determined as the optimized execution action, thereby achieving optimization of the execution action set.

[0047] In summary, the server parameter optimization method proposed in this disclosure includes: obtaining a set of state parameters and a set of execution actions for the server cluster, both of which have time labels; determining local evaluation prediction parameters based on the state parameter set and the execution action set using a multi-agent reinforcement learning algorithm; determining global evaluation prediction parameters based on the state parameter set and the execution action set using an adaptive dynamic programming model; and optimizing the execution action set based on the local and global evaluation prediction parameters. This method, by utilizing the QMIX model and the ADP model to determine the local and global evaluation prediction parameters corresponding to the server cluster, achieves automatic optimization of the execution action set of the server cluster, thereby improving the stability and service efficiency of the server cluster.

[0048] Figure 2 A flowchart of a server parameter optimization method proposed in this disclosure is further shown. Figure 1 The embodiment shown, Figure 2 It includes the following steps.

[0049] Step 201: Use the server cluster's listening component to obtain the server cluster's status parameters and execution actions.

[0050] Step 202: Store the state parameters and execution actions in a time series database to obtain the state parameter set and execution action set.

[0051] In some embodiments, the implementation of steps 201 and 202 may refer to Figure 1 The embodiment of step 101 shown will not be described again here.

[0052] Step 203: Determine the local state vector based on the set of state parameters.

[0053] In some embodiments, features can be extracted from the local state parameters in the state parameter set to obtain multiple local state features, and then at least two local state features can be selected from the multiple local state features to form a local state vector.

[0054] Step 204: Based on the local state vector and the set of actions to be executed, the prediction parameters of the single agent are determined by utilizing the pre-trained deep Q-learning network contained in the single agent in the multi-agent reinforcement learning algorithm.

[0055] In some embodiments, by inputting the local state vector and the set of actions to a pre-trained deep Q-learning network, a score is obtained when the server cluster performs a certain action under the local state vector. This score is the single agent prediction parameter.

[0056] It should be understood that multi-agent reinforcement learning algorithms contain multiple single-agent networks, each of which comprises a pre-trained deep Q-learning network (Q-learning with Multi-Agent Systems, QMIX) and a target deep Q-learning network. The target deep Q-learning network is used to obtain the pre-trained deep Q-learning network. For details on how to obtain this network, please refer to [link to relevant documentation]. Figure 3 The illustrated embodiment.

[0057] It should be understood that Deep Q-Learning Networks are algorithms for multi-agent reinforcement learning and collaborative decision-making, suitable for problems involving multiple agents where their decisions influence each other. The goal of QMIX is to enable agents to cooperate and achieve globally optimal decisions by learning local state-action value functions.

[0058] The core idea of ​​QMIX is to control the local state-action value function through a centralized value function to solve the decision-making dilemma among agents. Traditional multi-agent reinforcement learning algorithms often learn local state-action value functions independently, leading to a lack of cooperation. QMIX, on the other hand, introduces a related network (Qatten network) to calculate the action weights of each agent, thereby synthesizing the local state-action value functions.

[0059] Step 205: Weighted summation of the prediction parameters of a single agent to obtain local prediction parameters.

[0060] In some embodiments, the prediction parameters of each single agent in the multi-agent reinforcement learning algorithm can be weighted and summed using a mixing network in the multi-agent reinforcement learning algorithm to obtain a comprehensive single agent prediction parameter, i.e., a local prediction parameter.

[0061] In some embodiments, the determination of the weights of the prediction parameters for a single agent is not limited; for example, they can be directly preset or determined using global state parameters from the state parameter set.

[0062] For example, each single agent takes the global state parameters as input and generates weights for a single layer of hybrid network. Each single agent consists of a single linear layer followed by an absolute value activation function to ensure the weights of the hybrid network are non-negative. Biases are generated in the same way, but the bias generation network does not have an absolute value activation function. The final bias is generated by a two-layer single agent with ReLU nonlinearity. The loss function is, for example, [missing information].

[0063] Step 206: Based on the set of state parameters, use time tags to determine the global state vector.

[0064] In some embodiments, time stamps are used to select global state parameters over continuous time from the set of state parameters, so that the adaptive dynamic programming (ADP) model can more accurately determine global prediction parameters.

[0065] In some embodiments, features can be extracted from the global state parameters in the state parameter set to obtain multiple global state features, and then at least two global state features can be selected from the multiple global state features to form a global state vector.

[0066] It should be understood that ADP generally consists of three parts: a dynamic system, a critical performance index function (CPI) stage, and an action / control stage. The dynamic system (or controlled object) corresponds to the established model. The action / control stage approximates the optimal control strategy. The CPI stage updates parameters based on the Bellman optimality principle. The evaluation network and the action network combine to form an agent. Action / control acts on the dynamic system, and the CPI is influenced by the rewards or penalties generated by the dynamic system. The action / control stage outputs control actions, and the output of the CPI is a cost function value based on the Bellman optimality principle. That is, the action / control stage is adjusted to minimize the output cost function value to make its output action approximately optimal.

[0067] The cost function of the ADP model can be as follows:

[0068] Step 207: Based on the global state vector and the set of actions to be executed, the global estimated prediction parameters are determined using the value function estimator in the adaptive dynamic programming model.

[0069] In some embodiments, by inputting the global state vector and the set of execution actions into an adaptive dynamic programming model, and using the value function estimator in the adaptive dynamic programming model, a score is obtained when the server cluster performs a certain execution action under the global state vector. This score is the global estimated prediction parameter.

[0070] Step 208: Perform a weighted summation of the local evaluation prediction parameters and the global prediction parameters to determine the fusion evaluation prediction parameters.

[0071] In some embodiments, a fusion evaluation prediction parameter is determined by weighted summation of local evaluation prediction parameters and global prediction parameters, thereby achieving the acquisition of comprehensive prediction parameters.

[0072] Step 209: Optimize the action parameter set based on the execution actions corresponding to the fusion evaluation prediction parameters.

[0073] In some embodiments, the execution action corresponding to the fusion evaluation prediction parameter with the highest value or greater than a preset value is selected from the fusion evaluation prediction parameters and determined as the optimized execution action. The determined optimized execution action replaces the execution action in the action parameter set, thereby achieving optimization of the action parameter set.

[0074] In summary, the server parameter optimization method proposed in this disclosure includes: acquiring the state parameters and execution actions of the server cluster using a monitoring component; storing the state parameters and execution actions in a time-series database to obtain a set of state parameters and a set of execution actions; determining local state vectors based on the set of state parameters; determining single-agent prediction parameters using a pre-trained deep Q-learning network included in a single agent within a multi-agent reinforcement learning algorithm based on the local state vectors and the set of execution actions; and weighted summing of the single-agent prediction parameters to obtain local prediction parameters. Based on the set of state parameters, a global state vector is determined using time labels; globally estimated prediction parameters are determined using a value function estimator in an adaptive dynamic programming model based on the global state vector and the set of execution actions; weighted summing of the local and global prediction parameters is performed to determine fused evaluation prediction parameters; and the action parameter set is optimized based on the execution actions corresponding to the fused evaluation prediction parameters. This method utilizes the QMIX and ADP models to determine the local and global evaluation prediction parameters corresponding to the server cluster, thereby achieving automatic optimization of the execution action set of the server cluster to improve its stability and service efficiency.

[0075] Figure 3 A flowchart of a server parameter optimization method proposed in this disclosure is further shown. Figure 2 The embodiment shown may include the following steps for obtaining a pre-trained deep Q-learning network.

[0076] Step 301: Based on the local state vector and the set of executed actions, determine the empirical evaluation parameters using the deep Q-learning network to be trained in the deep Q-learning network.

[0077] In some embodiments, the local state vector and the set of actions are input into the deep Q-learning network to be trained, and the output of the deep Q-learning network to be trained is the empirical evaluation parameter.

[0078] For example, a local state vector s composed of cluster state information. t The DRQN ​​to be trained is passed in, and the evaluation parameters for each set of parameters (i.e., each action) in this state are obtained through the DRQN ​​to be trained. At time t, a is executed. t The experience generated after the action is (s) t ,a t ,r t ,a t+1 ), which is the empirical evaluation parameter mentioned above, and this experience represents the state at the current time t.

[0079] Step 302: Based on the time parameters, extract empirical evaluation parameters to obtain training evaluation parameters using the deep Q-learning network to be trained.

[0080] In some embodiments, empirical evaluation parameters are extracted discontinuously over time based on time parameters. The local state vector and action set corresponding to the extracted empirical evaluation parameters are then input back into the deep Q-learning network to be trained. At this time, the output of the deep Q-learning network to be trained is the training evaluation parameter.

[0081] It should be understood that in each iteration (i.e., all actions in the action set are trained once), the experience is saved to the experience replay pool. When the algorithm needs to learn, a sequence is randomly selected from the experience replay pool, and n time steps are selected backward from a random point in the sequence for learning, depending on the situation. Simultaneously, an ε-greedy strategy is used as the action strategy in iterative learning. Before each update, the state of the GRU hidden layers of the deep Q-learning network to be trained needs to be set to 0. Randomly selecting sequences and time steps disrupts the continuity between samples, making training more effective. Furthermore, once the experience sequence in the experience pool reaches its capacity limit, newly generated experience sequences will overwrite existing experience sequences.

[0082] Step 303: Based on the extracted empirical evaluation parameters, the target evaluation parameters are determined using the target deep Q-learning network in the multi-agent reinforcement learning algorithm.

[0083] In some embodiments, the local state vector and action set used as input to the deep Q-learning network to be trained are input to the target deep Q-learning network, and the output of the target deep Q-learning network is the target evaluation parameter.

[0084] Step 304: Based on the training evaluation parameters and the target evaluation parameters, use a preset objective function and a preset step size to train the deep Q-learning network to be trained, so as to obtain a pre-trained deep Q-learning network.

[0085] For example, multi-agent reinforcement learning algorithms employ two neural networks with identical structures but different parameters to update network parameters. These are the training deep Q-learning network (evaluation network) and the target deep Q-learning network (target network). The network model structure is as follows: Figure 4 As shown. The evaluation network is the same as the training network. After training every N time steps, the learned parameters of the evaluation network are copied to the target network. The objective function (i.e., the loss function) of the entire network is y, where y is the Q calculated by the target network. target (s t+1 ,a t+1 Target value. Q eval (s t ,a t ;θ) represents the predicted value obtained by the evaluation network, and θ represents the model parameters of the evaluation network. -The target network parameters are defined as follows. The preset objective function can be a loss function defined as the mean squared error between the predicted and target values, with the root mean square backpropagation algorithm used to optimize the loss.

[0086] loss(0)=E[(y-Qeval(S,a;0))]

[0087]

[0088] In summary, according to the server parameter optimization method proposed in this disclosure, empirical evaluation parameters are determined based on the local state vector and the set of executed actions, using the deep Q-learning network to be trained within the deep Q-learning network. Empirical evaluation parameters are extracted based on time parameters to obtain training evaluation parameters using the deep Q-learning network to be trained. Based on the extracted empirical evaluation parameters, target evaluation parameters are determined using the target deep Q-learning network in a multi-agent reinforcement learning algorithm. Based on the training evaluation parameters and the target evaluation parameters, the deep Q-learning network to be trained is trained using a preset objective function and a preset step size to obtain a pre-trained deep Q-learning network. The method of this disclosure, by obtaining training evaluation parameters and target evaluation parameters and using a preset objective function, thereby trains the deep Q-learning network to be trained to obtain a pre-trained deep Q-learning network.

[0089] Therefore, this solution has the following beneficial effects:

[0090] 1. This solution can be continuously modified during task execution to compensate for changes in the amount of data in the business scenario.

[0091] 2. A multi-agent reinforcement learning model is used to fully consider that the Kafka distributed cluster has multiple servers representing performance characteristics. A single result is no longer used to represent the overall cluster's operational status.

[0092] 3. Adaptive decision-making mechanism: Combining QMI reinforcement learning and ADP dynamic programming model, a new algorithm model is proposed, which makes more accurate predictions.

[0093] 4. Instead of setting parameters once and applying them throughout the entire task execution, this proposal suggests dynamically adjusting parameters based on the running conditions during the entire task execution process, thereby achieving high efficiency and improving the utilization of cluster resources.

[0094] Figure 5 This is a schematic diagram of a server parameter optimization device 500 provided in an embodiment of this disclosure. Figure 5 As shown, the server parameter optimization device includes:

[0095] The acquisition unit 510 is used to acquire the status parameter set and execution action set of the server cluster. The status parameter set and execution action set have time stamps.

[0096] The first evaluation unit 520 is used to determine local evaluation prediction parameters based on the set of state parameters and the set of executed actions using a multi-agent reinforcement learning algorithm.

[0097] The second evaluation unit 530 is used to determine global evaluation prediction parameters based on the set of state parameters and the set of execution actions using an adaptive dynamic programming model.

[0098] The optimization unit 540 is used to optimize the set of execution actions based on local evaluation prediction parameters and global evaluation prediction parameters.

[0099] In some embodiments of this disclosure, the acquisition unit 510 is further configured to acquire the status parameters and execution actions of the server cluster using the monitoring component of the server cluster; and store the status parameters and execution actions in a time series database to obtain a set of status parameters and a set of execution actions.

[0100] In some embodiments of this disclosure, the first evaluation unit 520 is further configured to determine a local state vector based on a set of state parameters; determine single agent prediction parameters based on the local state vector and a set of actions to be performed, using a pre-trained deep Q-learning network included in a single agent in a multi-agent reinforcement learning algorithm; and perform a weighted summation of the single agent prediction parameters to obtain local prediction parameters.

[0101] In some embodiments of this disclosure, the first evaluation unit 520 is further configured to: determine empirical evaluation parameters based on local state vectors and a set of executed actions using the deep Q-learning network to be trained in the deep Q-learning network; extract empirical evaluation parameters based on time parameters to obtain training evaluation parameters using the deep Q-learning network to be trained; determine target evaluation parameters based on the extracted empirical evaluation parameters using the target deep Q-learning network in the multi-agent reinforcement learning algorithm; and train the deep Q-learning network to be trained using a preset objective function with a preset step size based on the training evaluation parameters and the target evaluation parameters to obtain a pre-trained deep Q-learning network.

[0102] In some embodiments of this disclosure, the second evaluation unit 530 is further configured to determine a global state vector based on a set of state parameters and using time labels; and to determine global estimation prediction parameters based on the global state vector and a set of execution actions using a value function estimator in an adaptive dynamic programming model.

[0103] In some embodiments of this disclosure, the optimization unit 540 is further configured to perform a weighted summation of the local evaluation prediction parameters and the global prediction parameters to determine the fusion evaluation prediction parameters; and to optimize the action parameter set based on the execution actions corresponding to the fusion evaluation prediction parameters.

[0104] In some embodiments of this disclosure, the set of state parameters includes at least one of the following: producer throughput, consumer throughput, server latency, health status of server partitions, and health status of server backups; the set of execution actions includes at least one of the following: batch size, batch wait time, consumer pull size, consumer pull interval, and number of storage partitions.

[0105] In summary, the server parameter optimization apparatus proposed in this disclosure includes: an acquisition unit for acquiring a set of state parameters and a set of execution actions of a server cluster, wherein the state parameter set and the set of execution actions have time labels; a first evaluation unit for determining local evaluation prediction parameters based on the state parameter set and the set of execution actions using a multi-agent reinforcement learning algorithm; a second evaluation unit for determining global evaluation prediction parameters based on the state parameter set and the set of execution actions using an adaptive dynamic programming model; and an optimization unit for optimizing the set of execution actions based on the local evaluation prediction parameters and the global evaluation prediction parameters. This apparatus, by utilizing the QMIX model and the ADP model to determine the local evaluation prediction parameters and the global evaluation prediction parameters corresponding to the server cluster, achieves automatic optimization of the set of execution actions of the server cluster, thereby improving the stability and service efficiency of the server cluster.

[0106] It should be noted that the server parameter optimization device provided in the above embodiments is only illustrated by the division of the above program modules when optimizing server parameters. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the server parameter optimization device can be divided into different program modules to complete all or part of the processing described above.

[0107] Since the apparatus provided in this embodiment corresponds to the methods provided in the above embodiments, the implementation of the methods is also applicable to the apparatus provided in this embodiment, and will not be described in detail in this embodiment.

[0108] The methods and apparatus provided in the embodiments of this application have been described above. To implement the functions of the methods provided in the embodiments of this application, the electronic device may include a hardware structure and software modules, and may implement the above functions in the form of a hardware structure, software modules, or a hardware structure plus software modules. One of the above functions may be executed in the form of a hardware structure, software modules, or a hardware structure plus software modules.

[0109] Figure 6 This is a schematic diagram of the hardware composition structure of the electronic device provided in the embodiments of this disclosure, such as... Figure 6As shown, the electronic device 600 includes at least one processor 602; and a memory 601 communicatively connected to the at least one processor 602; wherein the memory 601 stores instructions executable by the at least one processor 602, the instructions being executed by the at least one processor 602 to implement the steps of the server parameter optimization method described in the embodiments of this disclosure; or, the instructions being executed by the at least one processor 602 to implement the steps of the server parameter optimization method described in the embodiments of this disclosure.

[0110] It is understood that the electronic device also includes a communication interface 603. Various components in the electronic device are coupled together via a bus system 604. It is understood that the bus system 604 is used to implement communication between these components. In addition to a data bus, the bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 6 The general designated all buses as Bus System 604.

[0111] It is understood that memory 601 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CDROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 601 described in this embodiment of the invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0112] The methods disclosed in the above embodiments can be applied to, or implemented by, processor 602. Processor 602 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed through integrated logic circuits in the hardware of processor 602 or through software instructions.

[0113] The processor 602 described above can be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 602 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium, which is located in memory 601. The processor 602 reads information from memory 601 and, in conjunction with its hardware, completes the steps of the aforementioned methods.

[0114] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned method.

[0115] This disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, which, when executed, implement the steps of the server parameter optimization method described in this embodiment of the invention.

[0116] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0117] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0118] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0119] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (control method), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic device, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0120] It should be understood that various parts of the embodiments of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0121] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0122] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc.

[0123] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for optimizing server parameters, characterized in that, The method includes: Obtain the status parameter set and execution action set of the server cluster, wherein the status parameter set and the execution action set have timestamps; Based on the set of state parameters, determine the local state vector; Based on the local state vector and the set of actions to be executed, the prediction parameters of a single agent are determined by utilizing the pre-trained deep Q-learning network contained in the single agent in the multi-agent reinforcement learning algorithm. The prediction parameters of the single agent are weighted and summed to obtain the local evaluation prediction parameters; Based on the set of state parameters, the global state vector is determined using the time stamp. Based on the global state vector and the set of actions to be executed, the global evaluation and prediction parameters are determined using the value function estimator in the adaptive dynamic programming model. The set of execution actions is optimized based on the local evaluation prediction parameters and the global evaluation prediction parameters.

2. The method according to claim 1, characterized in that, The acquisition of the server cluster's status parameter set and execution action set includes: The monitoring component of the server cluster is used to obtain the status parameters and actions of the server cluster. The state parameters and the execution actions are stored in a time series database to obtain the set of state parameters and the set of execution actions.

3. The method according to claim 1, characterized in that, The method further includes: Based on the local state vector and the set of actions to be executed, empirical evaluation parameters are determined using the deep Q-learning network to be trained in the deep Q-learning network. Based on the time parameters, the empirical evaluation parameters are extracted to obtain training evaluation parameters using the deep Q-learning network to be trained. Based on the extracted empirical evaluation parameters, the target evaluation parameters are determined using the target deep Q-learning network in the multi-agent reinforcement learning algorithm. Based on the training evaluation parameters and the target evaluation parameters, the deep Q-learning network to be trained is trained using a preset objective function and a preset step size to obtain the pre-trained deep Q-learning network.

4. The method according to claim 1, characterized in that, The optimization of the set of execution actions based on the local evaluation prediction parameters and the global evaluation prediction parameters includes: The local evaluation prediction parameters and the global prediction parameters are weighted and summed to determine the fusion evaluation prediction parameters; Based on the execution actions corresponding to the fusion evaluation and prediction parameters, the action parameter set is optimized.

5. The method according to claim 1, characterized in that, The set of status parameters includes at least one of the following: producer throughput, consumer throughput, server latency, health status of server partitions, and health status of server backups; The set of execution actions includes at least one of the following: batch size, batch waiting time, consumer pull size, consumer pull interval, and number of storage partitions.

6. A server parameter optimization device, characterized in that, The device includes: The acquisition unit is used to acquire the status parameter set and the execution action set of the server cluster, wherein the status parameter set and the execution action set have time stamps; The first determining unit is used to determine a local state vector based on the set of state parameters; The second determining unit is used to determine the single agent prediction parameters based on the local state vector and the set of executed actions, using the pre-trained deep Q-learning network contained in the single agent in the multi-agent reinforcement learning algorithm. The first evaluation unit is used to perform a weighted summation of the prediction parameters of the single agent to obtain local evaluation prediction parameters. The second evaluation unit is used to determine the global state vector based on the state parameter set and the time tag. The third determining unit is used to determine the global evaluation and prediction parameters based on the global state vector and the set of execution actions, using the value function estimator in the adaptive dynamic programming model. An optimization unit is used to optimize the set of execution actions based on the local evaluation prediction parameters and the global evaluation prediction parameters.

7. An electronic device, characterized in that, include: A processor and a memory for storing a computer program capable of running on the processor, wherein the processor, when running the computer program, performs the method of any one of claims 1-5.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-5.

Images