Server parameter optimization method and device

Abstract

The invention provides a server parameter optimization method and device, and the method comprises the steps: obtaining a state parameter set and an execution action set of a server cluster, and the state parameter set and the execution action set are provided with time labels; based on the state parameter set and the execution action set, a multi-agent reinforcement learning algorithm is utilized to determine local evaluation prediction parameters; based on the state parameter set and the execution action set, global evaluation prediction parameters are determined by using an adaptive dynamic programming model; and optimizing the execution action set based on the local evaluation prediction parameters and the global evaluation prediction parameters. According to the method disclosed by the invention, the local evaluation prediction parameters and the global evaluation prediction parameters corresponding to the server cluster are determined by using the QMIX model and the ADP model, so that automatic optimization of the execution action set of the server cluster is realized, and the stability and the service efficiency of the server cluster are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of dynamic programming, and in particular to a server parameter optimization method and device. Background Art

[0002] As a distributed server cluster that supports partitioning, the Kafka server cluster has a large message queue data throughput, and the data volume fluctuates greatly over time. If a parameter is used continuously in the same task and cannot adapt to the fluctuation of data volume, it is easy to cause a large amount of idle resources or abnormally high system load, which is not scientific for the operation of the task. The existing technology configuration parameters need to be configured according to the operation scenario, and manual configuration is required, which is inefficient. At present, more and more big data operators are seeking automated adjustments in the hope of greatly improving operation and maintenance efficiency and avoiding waste of resources to a greater extent. Summary of the invention

[0003] The present disclosure provides a server parameter optimization method and device to solve the problems in the related art and realize automatic optimization of the execution action set of a Kafka server cluster to improve the stability and service efficiency of the server cluster.

[0004] The first aspect embodiment of the present disclosure proposes a server parameter optimization method, which includes: obtaining a state parameter set and an execution action set of a server cluster, wherein the state parameter set and the execution action set have time tags; based on the state parameter set and the execution action set, using a multi-agent reinforcement learning algorithm, determining local evaluation prediction parameters; based on the state parameter set and the execution action set, using an adaptive dynamic programming model, determining global evaluation prediction parameters; based on the local evaluation prediction parameters and the global evaluation prediction parameters, optimizing the execution action set.

[0005] In some embodiments of the present disclosure, obtaining a state parameter set and an execution action set of a server cluster includes: utilizing a listening component of the server cluster to obtain state parameters and execution actions of the server cluster; storing the state parameters and execution actions in a time series database to obtain a state parameter set and an execution action set.

[0006] In some embodiments of the present disclosure, based on a state parameter set and an execution action set, a multi-agent reinforcement learning algorithm is used to determine local evaluation prediction parameters, including: determining a local state vector based on the state parameter set; determining single-agent prediction parameters based on the local state vector and the execution action set using a pre-trained deep Q learning network contained in a single agent in the multi-agent reinforcement learning algorithm; and weighted summing the single-agent prediction parameters to obtain local prediction parameters.

[0007] In some embodiments of the present disclosure, the method also includes: based on the local state vector and the execution action set, using the deep Q learning network to be trained in the deep Q learning network, determining the experience evaluation parameters; based on the time parameters, extracting the experience evaluation parameters to obtain the training evaluation parameters using the deep Q learning network to be trained; based on the extracted experience evaluation parameters, using the target deep Q learning network in the multi-agent reinforcement learning algorithm, determining the target evaluation parameters; based on the training evaluation parameters and the target evaluation parameters, using the preset objective function, training the deep Q learning network to be trained with a preset step size to obtain a pre-trained deep Q learning network.

[0008] In some embodiments of the present disclosure, based on a state parameter set and an execution action set, using an adaptive dynamic programming model, determining global evaluation prediction parameters includes: based on the state parameter set, using time tags, determining a global state vector; based on the global state vector and the execution action set, using a value function estimator in the adaptive dynamic programming model, determining global estimation prediction parameters.

[0009] In some embodiments of the present disclosure, optimizing the execution action set based on local evaluation prediction parameters and global evaluation prediction parameters includes: taking a weighted sum of the local evaluation prediction parameters and the global prediction parameters to determine the fused evaluation prediction parameters; and optimizing the action parameter set based on the execution actions corresponding to the fused evaluation prediction parameters.

[0010] In some embodiments of the present disclosure, the status parameter set includes at least one of the following: producer throughput, consumer throughput, server latency, health status of server partitions, and health status of server backups; the execution action set includes at least one of the following: batch size, batch waiting time, consumer pull size, consumer pull interval, and number of storage partitions.

[0011] The second aspect of the embodiment of the present disclosure proposes a server parameter optimization device, which includes: an acquisition unit, used to acquire a state parameter set and an execution action set of a server cluster, the state parameter set and the execution action set having a time tag; a first evaluation unit, used to determine local evaluation prediction parameters based on the state parameter set and the execution action set using a multi-agent reinforcement learning algorithm; a second evaluation unit, used to determine global evaluation prediction parameters based on the state parameter set and the execution action set using an adaptive dynamic programming model; and an optimization unit, used to optimize the execution action set based on the local evaluation prediction parameters and the global evaluation prediction parameters.

[0012] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: a processor and a memory for storing a computer program that can be run on the processor, wherein the processor, when used to run the computer program, executes the method described in the first aspect embodiment of the present disclosure.

[0013] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.

[0014] The fourth aspect embodiment of the present disclosure provides a computer program product, including a computer program, which implements the method described in the first aspect embodiment of the present disclosure when executed by a processor.

[0015] In summary, according to a server parameter optimization method proposed in the present disclosure, the method includes: obtaining a state parameter set and an execution action set of a server cluster, the state parameter set and the execution action set having a time tag; based on the state parameter set and the execution action set, using a multi-agent reinforcement learning algorithm, determining local evaluation prediction parameters; based on the state parameter set and the execution action set, using an adaptive dynamic programming model, determining global evaluation prediction parameters; based on the local evaluation prediction parameters and the global evaluation prediction parameters, optimizing the execution action set. The method disclosed in the present disclosure determines the local evaluation prediction parameters and the global evaluation prediction parameters corresponding to the server cluster by using the QMIX model and the ADP model, thereby realizing automatic optimization of the execution action set of the server cluster, so as to improve the stability and service efficiency of the server cluster.

[0016] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0018] Figure 1 A flow chart of a server parameter optimization method provided by an embodiment of the present disclosure;

[0019] Figure 2 A flowchart of another server parameter optimization method provided by an embodiment of the present disclosure;

[0020] Figure 3 A flowchart of another server parameter optimization method provided by an embodiment of the present disclosure;

[0021] Figure 4 A flowchart of another server parameter optimization method provided in an embodiment of the present disclosure;

[0022] Figure 5 A schematic diagram of the structure of a server parameter optimization device provided in an embodiment of the present disclosure;

[0023] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0025] As a distributed server cluster that supports partitioning, the Kafka server cluster has a large message queue data throughput, and the data volume fluctuates greatly over time. If a parameter is used continuously in the same task and cannot adapt to the fluctuation of data volume, it is easy to cause a large amount of idle resources or abnormally high system load, which is not scientific for the operation of the task. The existing technology configuration parameters need to be configured according to the operation scenario, and manual configuration is required, which is inefficient. At present, more and more big data operators are seeking automated adjustments in the hope of greatly improving operation and maintenance efficiency and avoiding waste of resources to a greater extent.

[0026] Therefore, the related solutions have the following defects:

[0027] 1. Business scenarios are partially unpredictable. If the business scenarios provided by the operation and maintenance personnel deviate from the optimal configuration, the optimal configuration cannot be obtained. If this wrong configuration is used, it cannot be corrected during the entire task operation, and the operation performance will not be as expected. In this case, you need to manually stop the business, re-enter the business scenario in the model to obtain the parameter configuration, and then restart the task.

[0028] 2. Kafka is a commonly used message queue, and the amount of data may fluctuate over time. At this time, it is slightly unreasonable to use the same parameters to execute the entire task through a server cluster deployed with Kafka. Existing technologies often lack an automated decision-making mechanism and cannot intelligently adjust parameters according to actual needs and operating conditions.

[0029] 3. In the existing technology, there is no method to evaluate the performance of Kafka distributed clusters. Basically, only single performance results are given, which is not comprehensive enough.

[0030] The present disclosure is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] Figure 1 A flow chart of a server parameter optimization method provided by an embodiment of the present disclosure. Figure 1 As shown, steps 101-104 are included.

[0032] Step 101: Obtain a state parameter set and an execution action set of a server cluster.

[0033] In some embodiments, the acquired state parameter set and execution action set of the server cluster have time tags, so that the model can be trained using parameters at non-continuous time to improve the prediction accuracy of the model.

[0034] In some optional embodiments, the monitoring component of the server cluster can be used to obtain the state parameters and execution actions of the server cluster; and the state parameters and execution actions are stored in a time series database, thereby giving time tags to the state parameters and execution actions to achieve the acquisition of the state parameter set and execution action set.

[0035] For example, the Kafka Exporter component can be installed in each server in the cluster to start monitoring the status of the Kafka cluster and executing actions. It converts it into a format that Prometheus can recognize and then exposes it to Prometheus through the HTTP interface.

[0036] Prometheus stores the acquired indicator data in its own time series database. Through Prometheus's query language (PromQL), Kafka's status indicators can be queried and analyzed. Thus, the device executing this solution can access Prometheus to obtain the status parameter set and execution action set.

[0037] In some embodiments, the state parameter set includes at least one of the following: producer throughput, consumer throughput, server latency, health status of server partitions, and health status of server backups; the execution action set includes at least one of the following: batch size, batch waiting time, consumer pull size, consumer pull interval, and number of storage partitions.

[0038] It should be understood that the state parameter set may include local state parameters and global state parameters, or may only include global state parameters, and the local state parameters are obtained by processing the global state parameters.

[0039] Step 102, based on the state parameter set and the execution action set, using a multi-agent reinforcement learning algorithm, determine the local evaluation prediction parameters.

[0040] In some embodiments, the local state vector of the server cluster can be determined based on the state parameter set, and then the local state vector and the execution action set can be input into each single agent in the multi-agent reinforcement learning algorithm to obtain the single-agent prediction parameters, and then the single-agent prediction parameters can be used to determine the local evaluation prediction parameters.

[0041] It should be understood that the local prediction parameter is the prediction score for the server cluster executing an action in the action parameter set under the local state vector. The local prediction parameter can be used to indicate the rationality of the server cluster executing an action under the local state vector.

[0042] Step 103: Based on the state parameter set and the execution action set, a global evaluation prediction parameter is determined using an adaptive dynamic programming model.

[0043] In some embodiments, a global state vector of the server cluster may be determined based on a state parameter set, and the global state vector and an execution action set may be input into an adaptive dynamic programming model to determine global evaluation prediction parameters.

[0044] It should be understood that the global prediction parameter is the prediction score for a server cluster executing an action in the action parameter set under the global state vector. The global prediction parameter can be used to indicate the rationality of the server cluster executing an action under the global state vector.

[0045] Step 104: Optimize the execution action set based on the local evaluation prediction parameters and the global evaluation prediction parameters.

[0046] In some embodiments, a weighted summation may be performed on the local evaluation prediction parameters and the global evaluation prediction parameters, or the local evaluation prediction parameters and the global evaluation prediction parameters may be directly summed to obtain fused evaluation prediction parameters, and the execution action corresponding to the parameter with the largest value in the fused evaluation prediction parameters may be determined as the optimized execution action, thereby achieving optimization of the execution action set.

[0047] In summary, according to the server parameter optimization method proposed in the present disclosure, the method includes: obtaining a state parameter set and an execution action set of a server cluster, the state parameter set and the execution action set having a time tag; based on the state parameter set and the execution action set, using a multi-agent reinforcement learning algorithm, determining local evaluation prediction parameters; based on the state parameter set and the execution action set, using an adaptive dynamic programming model, determining global evaluation prediction parameters; based on the local evaluation prediction parameters and the global evaluation prediction parameters, optimizing the execution action set. The method disclosed in the present disclosure determines the local evaluation prediction parameters and the global evaluation prediction parameters corresponding to the server cluster by using the QMIX model and the ADP model, thereby realizing automatic optimization of the execution action set of the server cluster, so as to improve the stability and service efficiency of the server cluster.

[0048] Figure 2 A flow chart of a server parameter optimization method proposed in the present disclosure is further shown. Figure 1 The embodiment shown, Figure 2 The following steps are included.

[0049] Step 201, using the monitoring component of the server cluster, obtain the state parameters and execution actions of the server cluster.

[0050] Step 202: store the state parameters and execution actions into a time series database to obtain a state parameter set and an execution action set.

[0051] In some embodiments, the implementation of step 201 and step 202 can refer to Figure 1 The embodiment of step 101 is shown and will not be described in detail here.

[0052] Step 203: Determine a local state vector based on the state parameter set.

[0053] In some embodiments, feature extraction may be performed on local state parameters in the state parameter set to obtain a plurality of local state features, and then at least two local state features are selected from the plurality of local state features to form a local state vector.

[0054] Step 204, based on the local state vector and the execution action set, using the pre-trained deep Q learning network contained in the single agent in the multi-agent reinforcement learning algorithm, determine the single agent prediction parameters.

[0055] In some embodiments, the local state vector and the execution action set are input into a pre-trained deep Q learning network to obtain a score when the server cluster performs a certain execution action under the local state vector, and the score is the single agent prediction parameter.

[0056] It should be understood that the multi-agent reinforcement learning algorithm contains multiple single agents (Agent Network), each of which contains a pre-trained deep Q learning network (Q-learning with Multi-Agent Systems, QMIX) and a target deep Q learning network, where the target deep Q learning network is used to obtain the pre-trained deep Q learning network. For specific acquisition methods, please refer to Figure 3 The embodiment shown.

[0057] It should be understood that the deep Q-learning network is an algorithm for multi-agent reinforcement learning and collaborative decision-making, which is suitable for problems with multiple agents and where the decisions between agents affect each other. The goal of QMIX is to enable agents to cooperate and achieve global optimal decisions by learning local state-action value functions.

[0058] The core idea of ​​QMIX is to control the local state-action value function through a centralized value function to solve the decision-making dilemma between agents. Traditional multi-agent reinforcement learning algorithms often learn local state-action value functions independently, resulting in the lack of cooperation. QMIX introduces a correlation network (Qatten network) to calculate the action weight of each agent, thereby synthesizing the local state-action value function.

[0059] Step 205, weighted sum of single agent prediction parameters to obtain local prediction parameters.

[0060] In some embodiments, a mixing network (MixingNetwork) in the multi-agent reinforcement learning algorithm can be used to perform weighted summation on each single-agent prediction parameter in the multi-agent reinforcement learning algorithm to obtain a comprehensive single-agent prediction parameter, i.e., a local prediction parameter.

[0061] In some embodiments, there is no limitation on the determination of the weight of the single-agent prediction parameter, which may be directly preset, or determined using the global state parameters in the state parameter set.

[0062] For example, each single agent takes the global state parameters as input and generates a layer of weights for the hybrid network. Each single agent consists of a single linear layer followed by an absolute value activation function to ensure that the weights of the hybrid network are non-negative. The biases are generated in the same way, but the bias generation network does not have an absolute value activation function. The final bias is generated by a 2-layer single agent with a ReLU nonlinearity. The loss function is, for example,

[0063] Step 206, based on the state parameter set and using the time tag, determine the global state vector.

[0064] In some embodiments, time tags are used to select global state parameters under continuous time in a state parameter set, so that an adaptive dynamic programming (ADP) model can more accurately determine global prediction parameters.

[0065] In some embodiments, feature extraction may be performed on the global state parameters in the state parameter set to obtain a plurality of global state features, and then at least two global state features are selected from the plurality of global state features to form a global state vector.

[0066] It should be understood that ADP generally includes three parts: dynamic system, critical performance index function, and action / control. The dynamic system (or controlled object) corresponds to the established model, the action / control link is used to approximate the optimal control strategy, the critical performance index function link is based on the Bellman optimality principle to update parameters, and the combination of the evaluation network and the action / control network forms an intelligent entity. Action / control acts on the dynamic system, and the critical performance index function is influenced by the rewards or penalties generated by the dynamic system. The action / control link outputs the control action, and the output of the critical performance index function is the cost function value based on the Bellman optimality principle, that is, the action / control link is adjusted to make its output action approximate to the optimal value with the goal of minimizing the output cost function value.

[0067] Among them, the cost function of the ADP model can be as follows:

[0068] Step 207, based on the global state vector and the execution action set, using the value function estimator in the adaptive dynamic programming model, determine the global estimation prediction parameters.

[0069] In some embodiments, the global state vector and the execution action set are input into an adaptive dynamic programming model, and the value function estimator in the adaptive dynamic programming model is used to obtain a score when the server cluster performs a certain execution action under the global state vector. The score is the global estimated prediction parameter.

[0070] Step 208: Perform weighted summation on the local evaluation prediction parameters and the global prediction parameters to determine the fusion evaluation prediction parameters.

[0071] In some embodiments, a fusion evaluation prediction parameter is determined by weighted summing of local evaluation prediction parameters and global prediction parameters, so as to obtain comprehensive prediction parameters.

[0072] Step 209: Optimize the action parameter set based on the execution action corresponding to the fusion evaluation prediction parameter.

[0073] In some embodiments, the execution action corresponding to the fusion evaluation prediction parameter with the highest value or greater than a preset value is selected from the fusion evaluation prediction parameters and determined as the optimized execution action. The determined optimized execution action replaces the execution action in the action parameter set, thereby optimizing the action parameter set.

[0074] In summary, the server parameter optimization method proposed in the present disclosure includes: using the monitoring component of the server cluster to obtain the state parameters and execution actions of the server cluster; storing the state parameters and execution actions in a time series database to obtain a state parameter set and an execution action set; determining a local state vector based on the state parameter set; determining the single-agent prediction parameters based on the local state vector and the execution action set using the pre-trained deep Q learning network contained in the single-agent in the multi-agent reinforcement learning algorithm; weighted summing the single-agent prediction parameters to obtain local prediction parameters. Based on the state parameter set, the global state vector is determined using the time label; based on the global state vector and the execution action set, the global estimated prediction parameters are determined using the value function estimator in the adaptive dynamic programming model; weighted summing the local evaluation prediction parameters and the global prediction parameters to determine the fusion evaluation prediction parameters; optimizing the action parameter set based on the execution actions corresponding to the fusion evaluation prediction parameters. The method disclosed in the present disclosure uses the QMIX model and the ADP model to determine the local evaluation prediction parameters and the global evaluation prediction parameters corresponding to the server cluster, thereby realizing automatic optimization of the execution action set of the server cluster to improve the stability and service efficiency of the server cluster.

[0075] Figure 3 A flow chart of a server parameter optimization method proposed in the present disclosure is further shown. Figure 2 In the illustrated embodiment, obtaining a pre-trained deep Q learning network may include the following steps.

[0076] Step 301, based on the local state vector and the execution action set, using the deep Q learning network to be trained in the deep Q learning network, determine the empirical evaluation parameters.

[0077] In some embodiments, the local state vector and the execution action set are input into a deep Q learning network to be trained in the deep Q learning network, and the output of the deep Q learning network to be trained is the empirical evaluation parameter.

[0078] For example, the local state vector s composed of cluster state information t The DRQN ​​to be trained is passed in, and the evaluation parameters of each set of parameters (i.e., each action) in this state are obtained through the DRQN ​​to be trained. At time t, execute a t The experience generated after the action is (s t ,a t ,r t ,a t+1 ), namely the above-mentioned experience evaluation parameter, which represents the current state at time t.

[0079] Step 302: extracting empirical evaluation parameters based on the time parameters to obtain training evaluation parameters using the deep Q learning network to be trained.

[0080] In some embodiments, based on the time parameter, the empirical evaluation parameters are extracted discontinuously in time, and the local state vector and execution action set corresponding to the extracted empirical evaluation parameters are input into the deep Q learning network to be trained again. At this time, the output of the deep Q learning network to be trained is the training evaluation parameters.

[0081] It should be understood that the experience is saved to the experience replay pool for each cycle (i.e., all the execution actions in the execution action set participate in the training once). When the algorithm needs to be trained, a sequence is randomly selected from the experience replay pool and n time steps are selected from a random point in the sequence for reverse learning. The ε-greedy strategy is used as the action strategy in iterative learning. Before each update, the state of the GRU hidden layer of the deep Q learning network to be trained needs to be set to 0. Randomly selecting sequences and time steps destroys the continuity between samples, making training more effective. At the same time, when the experience sequence of the experience pool reaches the capacity limit, the newly generated experience sequence will overwrite the existing experience sequence.

[0082] Step 303, based on the extracted empirical evaluation parameters, the target deep Q learning network in the multi-agent reinforcement learning algorithm is used to determine the target evaluation parameters.

[0083] In some embodiments, the local state vector and the set of execution actions used to input the deep Q learning network to be trained are input into the target deep Q learning network. At this time, the output of the target deep Q learning network is the target evaluation parameter.

[0084] Step 304: Based on the training evaluation parameters and the target evaluation parameters, the deep Q learning network to be trained is trained with a preset step size using a preset objective function to obtain a pre-trained deep Q learning network.

[0085] For example, the multi-agent reinforcement learning algorithm uses two neural networks with the same structure but different parameters to achieve the purpose of updating network parameters, namely the deep Q learning network to be trained (evaluation network) and the target deep Q learning network (target network). The network model structure is as follows: Figure 4 As shown in the figure, the evaluation network is the training network. After training every N time steps, the learning parameters of the evaluation network are copied to the target network. The objective function (i.e. loss function) of the entire network, y is the Q calculated by the target network target (s t+1 ,a t+1 ) target value. eval (s t ,a t ; θ) is the predicted value obtained by the evaluation network, θ is the evaluation network model parameter, θ -is the target network parameter. The preset objective function may be the following loss function, which is defined as the mean square error between the predicted value and the target value, and the root mean square back propagation algorithm is used to optimize the loss.

[0086] loss(0)=E[(y-Qeval(S,a;0))]

[0087]

[0088] In summary, according to the server parameter optimization method proposed in the present disclosure, based on the local state vector and the execution action set, the deep Q learning network to be trained in the deep Q learning network is used to determine the experience evaluation parameters; based on the time parameters, the experience evaluation parameters are extracted to obtain the training evaluation parameters using the deep Q learning network to be trained; based on the extracted experience evaluation parameters, the target deep Q learning network in the multi-agent reinforcement learning algorithm is used to determine the target evaluation parameters; based on the training evaluation parameters and the target evaluation parameters, the deep Q learning network to be trained is trained with a preset step size using a preset objective function to obtain a pre-trained deep Q learning network. The method disclosed in the present disclosure obtains the training evaluation parameters and the target evaluation parameters and uses the preset objective function to achieve the training of the deep Q learning network to be trained to obtain the pre-trained deep Q learning network.

[0089] Therefore, this solution has the following beneficial effects:

[0090] 1. This solution can be continuously modified during the task operation to compensate for changes in the amount of data in the business scenario.

[0091] 2. Use a multi-agent reinforcement learning model, taking into account that the Kafka distributed cluster has multiple server performance representations. Do not use a single result to represent the operating status of the entire cluster.

[0092] 3. Adaptive decision-making mechanism: QMI reinforcement learning and ADP dynamic programming model are combined to propose a new algorithm model for more accurate prediction.

[0093] 4. Instead of setting parameters once throughout the entire task execution, this proposal proposes to dynamically adjust parameters according to the operating conditions during the entire task execution process to achieve high efficiency and improve cluster resource utilization.

[0094] Figure 5 FIG. 5 is a schematic diagram of a server parameter optimization device 500 provided in an embodiment of the present disclosure. Figure 5 As shown, the server parameter optimization device includes:

[0095] An acquisition unit 510 is used to acquire a state parameter set and an execution action set of a server cluster, where the state parameter set and the execution action set have a time tag;

[0096] A first evaluation unit 520 is used to determine a local evaluation prediction parameter based on a state parameter set and an execution action set using a multi-agent reinforcement learning algorithm;

[0097] A second evaluation unit 530 is used to determine a global evaluation prediction parameter based on the state parameter set and the execution action set using an adaptive dynamic programming model;

[0098] The optimization unit 540 is used to optimize the execution action set based on the local evaluation prediction parameters and the global evaluation prediction parameters.

[0099] In some embodiments of the present disclosure, the acquisition unit 510 is further used to utilize the monitoring component of the server cluster to acquire the state parameters and execution actions of the server cluster; store the state parameters and execution actions in a time series database to obtain a state parameter set and an execution action set.

[0100] In some embodiments of the present disclosure, the first evaluation unit 520 is also used to determine a local state vector based on a state parameter set; based on the local state vector and the execution action set, determine the single-agent prediction parameters using a pre-trained deep Q learning network included in the single agent in the multi-agent reinforcement learning algorithm; and perform weighted summation of the single-agent prediction parameters to obtain local prediction parameters.

[0101] In some embodiments of the present disclosure, the first evaluation unit 520 is further used to determine empirical evaluation parameters based on the local state vector and the execution action set using the deep Q learning network to be trained in the deep Q learning network; extract empirical evaluation parameters based on the time parameters to obtain training evaluation parameters using the deep Q learning network to be trained; determine target evaluation parameters using the target deep Q learning network in the multi-agent reinforcement learning algorithm based on the extracted empirical evaluation parameters; and train the deep Q learning network to be trained with a preset step size using a preset objective function based on the training evaluation parameters and the target evaluation parameters to obtain a pre-trained deep Q learning network.

[0102] In some embodiments of the present disclosure, the second evaluation unit 530 is also used to determine the global state vector based on the state parameter set using the time tag; based on the global state vector and the execution action set, determine the global estimation prediction parameters using the value function estimator in the adaptive dynamic programming model.

[0103] In some embodiments of the present disclosure, the optimization unit 540 is further used to perform weighted summation of local evaluation prediction parameters and global prediction parameters to determine fusion evaluation prediction parameters; and optimize the action parameter set based on the execution actions corresponding to the fusion evaluation prediction parameters.

[0104] In some embodiments of the present disclosure, the status parameter set includes at least one of the following: producer throughput, consumer throughput, server latency, health status of server partitions, and health status of server backups; the execution action set includes at least one of the following: batch size, batch waiting time, consumer pull size, consumer pull interval, and number of storage partitions.

[0105] In summary, the server parameter optimization device proposed in the present disclosure includes: an acquisition unit, which is used to acquire the state parameter set and execution action set of the server cluster, and the state parameter set and execution action set have time tags; a first evaluation unit, which is used to determine the local evaluation prediction parameters based on the state parameter set and the execution action set by using a multi-agent reinforcement learning algorithm; a second evaluation unit, which is used to determine the global evaluation prediction parameters based on the state parameter set and the execution action set by using an adaptive dynamic programming model; an optimization unit, which is used to optimize the execution action set based on the local evaluation prediction parameters and the global evaluation prediction parameters. The device of the present disclosure determines the local evaluation prediction parameters and the global evaluation prediction parameters corresponding to the server cluster by using the QMIX model and the ADP model, thereby realizing automatic optimization of the execution action set of the server cluster to improve the stability and service efficiency of the server cluster.

[0106] It should be noted that: the server parameter optimization device provided in the above embodiment only uses the division of the above-mentioned program modules as an example when performing server parameter optimization. In actual applications, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the server parameter optimization device is divided into different program modules to complete all or part of the processing described above.

[0107] Since the device provided in the embodiment of the present disclosure corresponds to the methods provided in the above-mentioned embodiments, the implementation of the method is also applicable to the device provided in the embodiment and will not be described in detail in this embodiment.

[0108] In the embodiments provided in the present application, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the functions in the methods provided in the embodiments of the present application, the electronic device may include a hardware structure and a software module, and implement the functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. A function of the functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.

[0109] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure is shown in FIG. Figure 6As shown, the electronic device 600 includes at least one processor 602; and a memory 601 communicatively connected to the at least one processor 602; wherein the memory 601 stores instructions executable by the at least one processor 602, and the instructions are executed by the at least one processor 602 to implement the steps of the server parameter optimization method described in the embodiment of the present disclosure; or, the instructions are executed by the at least one processor 602 to implement the steps of the server parameter optimization method described in the embodiment of the present disclosure.

[0110] It is understood that the electronic device also includes a communication interface 603. The various components in the electronic device are coupled together through a bus system 604. It is understood that the bus system 604 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 6 Various buses are labeled as bus system 604 .

[0111] It can be understood that the memory 601 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CDROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 601 described in the embodiment of the present invention is intended to include but is not limited to these and any other suitable types of memories.

[0112] The method disclosed in the above embodiment of the present disclosure may be applied to the processor 602, or implemented by the processor 602. The processor 602 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in the processor 602 or an instruction in the form of software.

[0113] The processor 602 described above may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 602 may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. A general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiments of the present invention, it may be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a storage medium, which is located in the memory 601, and the processor 602 reads the information in the memory 601 and completes the steps of the aforementioned method in combination with its hardware.

[0114] In an exemplary embodiment, the electronic device may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), FPGA, general purpose processor, controller, MCU, microprocessor, or other electronic components to execute the aforementioned method.

[0115] The embodiment of the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to implement the steps of the server parameter optimization method described in the embodiment of the present invention when the computer instructions are executed.

[0116] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0117] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0118] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0119] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0120] It should be understood that the various parts of the embodiments of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (P GA), a field programmable gate array (FPGA), etc.

[0121] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0122] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0123] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A server parameter optimization method, characterized in that: The method comprises: Acquire a state parameter set and an execution action set of a server cluster, wherein the state parameter set and the execution action set have a time tag; Based on the state parameter set and the execution action set, using a multi-agent reinforcement learning algorithm, determining local evaluation prediction parameters; Based on the state parameter set and the execution action set, using an adaptive dynamic programming model, determining a global evaluation prediction parameter; The execution action set is optimized based on the local evaluation prediction parameters and the global evaluation prediction parameters.

2. The method according to claim 1, characterized in that The acquisition of the state parameter set and execution action set of the server cluster includes: Utilizing the monitoring component of the server cluster, obtaining the state parameters of the server cluster and executing actions; The state parameters and the execution actions are stored in a time series database to obtain the state parameter set and the execution action set.

3. The method according to claim 1, characterized in that The determining of the local evaluation prediction parameters based on the state parameter set and the execution action set by using a multi-agent reinforcement learning algorithm comprises: Based on the state parameter set, determining a local state vector; Based on the local state vector and the set of executed actions, determining single-agent prediction parameters using a pre-trained deep Q-learning network included in a single-agent in the multi-agent reinforcement learning algorithm; The single agent prediction parameters are weighted and summed to obtain local prediction parameters.

4. The method according to claim 3, characterized in that The method further comprises: Based on the local state vector and the set of executed actions, using a deep Q learning network to be trained in the deep Q learning network, determining an empirical evaluation parameter; Based on the time parameter, extracting the experience evaluation parameter to obtain the training evaluation parameter using the deep Q learning network to be trained; Based on the extracted empirical evaluation parameters, the target evaluation parameters are determined using the target deep Q learning network in the multi-agent reinforcement learning algorithm; Based on the training evaluation parameters and the target evaluation parameters, the deep Q learning network to be trained is trained with a preset step size using a preset objective function to obtain the pre-trained deep Q learning network.

5. The method according to claim 1, characterized in that Determining the global evaluation prediction parameters based on the state parameter set and the execution action set by using an adaptive dynamic programming model includes: Determining a global state vector based on the state parameter set and using the time tag; Based on the global state vector and the set of execution actions, the global estimation prediction parameters are determined using a value function estimator in the adaptive dynamic programming model.

6. The method according to claim 1, characterized in that The optimizing the execution action set based on the local evaluation prediction parameter and the global evaluation prediction parameter comprises: Performing weighted summation on the local evaluation prediction parameters and the global prediction parameters to determine a fusion evaluation prediction parameter; The action parameter set is optimized based on the execution action corresponding to the fusion evaluation prediction parameter.

7. The method according to claim 1, characterized in that The set of status parameters includes at least one of the following: producer throughput, consumer throughput, server latency, health status of server partitions, and health status of server backups; The execution action set includes at least one of the following: batch size, batch waiting time, consumer pull size, consumer pull interval and storage partition quantity.

8. A server parameter optimization device, characterized in that: The device comprises: An acquisition unit, used for acquiring a state parameter set and an execution action set of a server cluster, wherein the state parameter set and the execution action set have a time tag; A first evaluation unit is used to determine a local evaluation prediction parameter based on the state parameter set and the execution action set using a multi-agent reinforcement learning algorithm; A second evaluation unit is used to determine a global evaluation prediction parameter based on the state parameter set and the execution action set by using an adaptive dynamic programming model; An optimization unit is used to optimize the execution action set based on the local evaluation prediction parameters and the global evaluation prediction parameters.

9. An electronic device, characterized in that: include: A processor and a memory for storing a computer program that can be run on the processor, wherein the processor is configured to execute the method according to any one of claims 1 to 7 when running the computer program.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

11. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images