Micro-service elastic scaling method and system based on deep Monte Carlo tree search

Abstract

The invention discloses a micro-service elastic scaling method and system based on deep Monte Carlo tree search, and the method comprises the steps: inputting the current observation state information into an elastic scaling agent, and outputting a next micro-service optimal elastic scaling strategy, the elastic telescopic agent is constructed based on a Monte Carlo tree search method and deep reinforcement learning combination; determining strategy distribution and state value estimation according to a next micro-service optimal elastic scaling strategy, and performing performance evaluation on the micro-service system; calculating a joint loss value of the elastic telescopic agent through a joint loss function according to a performance evaluation result; and based on the joint loss value, optimizing parameters of the elastic telescoping intelligent agent, and performing micro-service elastic telescoping through the updated elastic telescoping intelligent agent. According to the method, the decision-making efficiency and the response capability of the elastic scaling strategy of the micro-service system are remarkably improved, resource waste is reduced, and the problems of response lagging and strategy instability of a traditional scaling method are solved.

Description

Technical Field

[0001] This invention relates to the field of microservice resource management technology, and in particular to a microservice elastic scaling method and system based on deep Monte Carlo tree search. Background Technology

[0002] In recent years, microservice architecture has been widely used in large-scale internet applications and enterprise-level systems. It divides functional modules at the service level, offering excellent scalability and deployment flexibility. In microservice systems, the number of service components is enormous, and the operating environment changes frequently. Therefore, the key to ensuring service quality and reducing operational costs lies in how to adjust resources in real time according to system load through elastic scaling mechanisms.

[0003] Traditional microservice scaling strategies mostly rely on static thresholds or linear models, such as scaling up when CPU utilization exceeds 80% and scaling down when it falls below 20%. This approach is simple but lacks flexibility, and it tends to lag in response to sudden loads or business peaks, easily leading to overscaling or resource waste.

[0004] In recent years, some studies have introduced predictive models (such as the statistical model (Autoregressive Integrated Moving Average ARIMA) and the Long Short-Term Memory (LSTM) model, which are widely used in time series analysis) to predict future loads and perform feedforward scaling. However, there are still problems such as weak generalization ability, parameter sensitivity, and difficulty in adapting to sudden events.

[0005] Therefore, how to solve the problems of slow response and slow policy convergence in existing scaling methods, so as to achieve intelligent, efficient and stable elastic scaling control of microservice systems, has become an urgent problem to be solved.

[0006] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention The main objective of this invention is to provide a microservice elastic scaling method and system based on deep Monte Carlo tree search, aiming to solve the problems of slow response and slow policy convergence in existing scaling methods, so as to achieve intelligent, efficient and stable elastic scaling control of microservice systems.

[0007] To achieve the above objectives, this invention provides a microservice elastic scaling method based on depth Monte Carlo tree search, the microservice elastic scaling method based on depth Monte Carlo tree search comprising: S1, obtain the current observation status information of the microservice system; S2, input the current observation state information into the elastic scaling agent, and output the next microservice optimal elastic scaling strategy. The elastic scaling agent is constructed based on a combination of Monte Carlo tree search and deep reinforcement learning. S3, determine the policy distribution and state value estimation according to the next microservice optimal elastic scaling policy, and perform performance evaluation on the microservice system based on the policy distribution and the state value estimation; S4. Calculate the joint loss value of the elastically scalable agent based on the performance evaluation results using the joint loss function. S5, optimize the parameters of the elastic scaling agent based on the joint loss value, and perform microservice elastic scaling through the updated elastic scaling agent.

[0008] Optionally, S1 includes: S1.1, Obtain the current operating metrics information of the microservice cluster and the future workload information of the microservice system; S1.2, determine the current observation status information based on the current operating index information and the future workload information.

[0009] Optionally, S2 includes: S2.1, Input the current observation state information into the elastic scaling agent, the elastic scaling agent including a representation layer, a mechanism layer and a prediction layer; S2.2, The current observation state information is encoded through the representation layer, and the encoded potential state is used as the root node of the search tree and the current potential state; S2.3, Based on the Monte Carlo tree search method, the mechanism layer outputs the next round potential state and the corresponding predicted instant reward according to the current potential state and the average value of the current node, and adds the next round potential state as a leaf node to the search tree; S2.4, backtrack from the leaf node to the root node to obtain the number of visits to each node in the path, and update the average value of each node based on the predicted instant reward and the number of visits to each node; S2.5, based on the potential state of the next round, the prediction layer outputs the strategy distribution and state value estimation, and determines whether the current round has reached the preset round; S2.6 If not, return to S2.3 until the current round reaches the preset round; S2.7, Select the optimal path from the multiple paths within the search tree; S2.8, Based on the policy distribution and state value estimation of each node corresponding to the optimal path, the elastic scaling agent outputs the optimal elastic scaling strategy for the next microservice.

[0010] Optionally, S2.3 includes: S2.3.1, Based on the current potential state and multiple preliminary actions, determine the next round action using an action selection formula; S2.3.2, determine whether the action in the next round is consistent with the action of each sub-node corresponding to the current potential state; S2.3.3, if not, then based on the current potential state and the next round action, the mechanism layer outputs the next round potential state and the corresponding instant reward, and adds the next round potential state as a leaf node to the search tree; S2.3.4 If so, then take the child node corresponding to the consistent action as the current potential state and return to S2.3.1 for iterative execution.

[0011] Optionally, the action selection formula is:

[0012] In the formula, The average value of the current node. This represents the predicted probability of the current node's action. This represents the number of visits to the current node. This represents the number of times the initial action b is accessed on the current node. and It is a constant. The action chosen for step k.

[0013] Optionally, the performance evaluation of the microservice system based on the policy distribution and the state value estimation includes: The microservice system is controlled to execute based on the policy distribution and the state value estimation, and the observation instant reward is calculated through the instant reward function. The instant reward function is:

[0014] In the formula, For observation of immediate rewards, Scaling factor As a regulating factor, The reward corresponding to the request response time. Rewards corresponding to resource utilization; The performance of the microservice system is evaluated based on the immediate rewards observed.

[0015] Optionally, the joint loss function is:

[0016] In the formula, For strategic losses, For the loss of value, To reward losses, For the action sequence in the k-th round at current time t, Let be the probability distribution of taking an action in the potential state at time t, round k. The long-term value of the k-th round at current time t. Estimate the state value for the k-th round at current time t. The instant reward for observation in the k-th round at current time t. Let be the predicted instant reward for the k-th round at the current time t, and c be the regularization coefficient.

[0017] Furthermore, to achieve the above objectives, this invention also proposes a microservice elastic scaling system based on deep Monte Carlo tree search, wherein the microservice elastic scaling system based on deep Monte Carlo tree search includes: The data acquisition module is used to acquire the current observation status information of the microservice system; The elastic scaling module is used to input the current observation state information into the elastic scaling agent and output the optimal elastic scaling strategy for the next microservice. The elastic scaling agent is constructed based on a combination of Monte Carlo tree search and deep reinforcement learning. The performance evaluation module is used to determine the policy distribution and state value estimation based on the next optimal elastic scaling strategy for the microservice, and to perform performance evaluation on the microservice system based on the policy distribution and the state value estimation. The iterative optimization module is used to calculate the joint loss value of the elastically scalable agent based on the performance evaluation results using a joint loss function. The iterative optimization module is also used to optimize the parameters of the elastic scaling agent based on the joint loss value, and to perform microservice elastic scaling through the updated elastic scaling agent.

[0018] Furthermore, to achieve the above objectives, the present invention also proposes a microservice elastic scaling device based on deep Monte Carlo tree search. The device includes: a memory, a processor, and a microservice elastic scaling program based on deep Monte Carlo tree search stored in the memory and executable on the processor. The microservice elastic scaling program based on deep Monte Carlo tree search is configured to implement the steps of the microservice elastic scaling method based on deep Monte Carlo tree search as described above.

[0019] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a microservice elastic scaling program based on deep Monte Carlo tree search, wherein when the microservice elastic scaling program based on deep Monte Carlo tree search is executed by a processor, it implements the steps of the microservice elastic scaling method based on deep Monte Carlo tree search as described above.

[0020] This invention first acquires the current observation state information of the microservice system, then inputs this information into an elastic scaling agent to output the optimal elastic scaling strategy for the next microservice. The elastic scaling agent is constructed based on a combination of Monte Carlo tree search and deep reinforcement learning. Next, it determines the policy distribution and state value estimation based on the optimal elastic scaling strategy for the next microservice, and performs performance evaluation on the microservice system based on these estimations. Finally, based on the performance evaluation results, it calculates the joint loss value of the elastic scaling agent using a joint loss function, optimizes the parameters of the elastic scaling agent based on the joint loss value, and performs microservice elastic scaling using the updated agent. This invention organically combines the foresight of heuristic search with the adaptability of reinforcement learning, improving the training efficiency and decision accuracy of the scaling strategy while enhancing its adaptability to highly dynamic service scenarios. It also solves the problems of slow response, slow policy convergence, and large service quality fluctuations in existing scaling methods, achieving intelligent, efficient, and stable elastic scaling control of the system. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the structure of a microservice elastic scaling device based on deep Monte Carlo tree search in the hardware operating environment involved in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the first embodiment of the microservice elastic scaling method based on deep Monte Carlo tree search of the present invention. Figure 3 This is a schematic diagram of the elastic scaling agent structure of the first embodiment of the microservice elastic scaling method based on deep Monte Carlo tree search of the present invention. Figure 4 This is a diagram of the MuScaler algorithm framework of the first embodiment of the microservice elastic scaling method based on deep Monte Carlo tree search of the present invention. Figure 5 This is a structural block diagram of the first embodiment of the microservice elastic scaling system based on deep Monte Carlo tree search of the present invention.

[0022] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0024] Reference Figure 1 , Figure 1 This is a schematic diagram of the microservice elastic scaling device structure based on deep Monte Carlo tree search in the hardware operating environment of the embodiment of the present invention.

[0025] like Figure 1 As shown, the microservice elastic scaling device based on deep Monte Carlo tree search may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to implement communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage system independent of the aforementioned processor 1001.

[0026] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on microservice elastic scaling devices based on deep Monte Carlo tree search and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0027] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a microservice elastic scaling program based on deep Monte Carlo tree search.

[0028] exist Figure 1 In the microservice elastic scaling device based on deep Monte Carlo tree search shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the microservice elastic scaling device based on deep Monte Carlo tree search of the present invention can be set in the microservice elastic scaling device based on deep Monte Carlo tree search. The microservice elastic scaling device based on deep Monte Carlo tree search calls the microservice elastic scaling program based on deep Monte Carlo tree search stored in memory 1005 through processor 1001 and executes the microservice elastic scaling method based on deep Monte Carlo tree search provided in the embodiment of the present invention.

[0029] This invention provides a microservice elastic scaling method based on depth Monte Carlo tree search, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the microservice elastic scaling method based on deep Monte Carlo tree search of the present invention.

[0030] In this embodiment, the microservice elastic scaling method based on deep Monte Carlo tree search includes the following steps: S1, obtain the current observation status information of the microservice system.

[0031] It is easy to understand that the execution subject of this embodiment can be a microservice elastic scaling system based on deep Monte Carlo tree search with functions such as data processing, network communication and program execution, or other computer devices with similar functions. This embodiment does not limit it.

[0032] Furthermore, the processing method for obtaining the current observation status information of the microservice system is as follows: obtain the current operation index information and the future workload information of the microservice system collected in real time by the microservice cluster; determine the current observation status information based on the current operation index information and the future workload information.

[0033] It should be noted that the current operational metrics include the number of instances of each microservice, average CPU utilization, average memory utilization, average received traffic, and average sent traffic.

[0034] The future workload information of the microservice system is predicted by the load prediction module. The future load information is the request volume data of multiple time steps predicted by the load prediction module based on the historical request rate.

[0035] Observation status information This describes the operational status of the entire microservice deployment system, consisting of service metric data from the monitoring module. The load prediction data output by the load prediction module is a prediction load window of length H. It consists of two parts:

[0036] Among them, service indicator data This describes the runtime status of each microservice instance in the microservice cluster:

[0037] in, microservices The service operation metrics, where N represents the total number of deployed microservices:

[0038] S2, input the current observation state information into the elastic scaling agent, and output the optimal elastic scaling strategy for the next microservice. The elastic scaling agent is constructed based on a combination of Monte Carlo tree search and deep reinforcement learning.

[0039] It should also be noted that the elastically scalable agent includes a representation layer, a mechanism layer, and a prediction layer, as referenced. Figure 3 , Figure 3 This is a schematic diagram of the elastic scaling agent structure in the first embodiment of the microservice elastic scaling method based on deep Monte Carlo tree search of the present invention. The representation layer, mechanism layer, and prediction layer solve the microservice elastic scaling task by learning the latent representation of the environment and predicting the dynamic changes of the environment.

[0040] The representation layer is used to learn the internal representation of the environment, transforming the original observation state into the representation required by the neural network.

[0041] Characterization layer Represented as:

[0042] in, To observe the situation, the network will encode it as a latent state, initialized with the state information of the root node. Used for Monte Carlo tree search; The mechanism layer is used to simulate the dynamic characteristics of the environment, thus enabling environmental modeling. The mechanism layer is represented as follows:

[0043] in, For the k-th round, calculate the current potential state. In this case, adopt the hypothetical action The next potential state after the transition and the instant rewards received ; The prediction layer is represented as:

[0044] in, Indicates that in a given potential state The probability distribution of taking an action. Assess the long-term value of a given potential state.

[0045] Furthermore, the processing flow for inputting the current observation state information into the elastic scaling agent and outputting the microservice elastic scaling strategy based on the current potential state is as follows: S2.1, Input the current observation state information into the elastic scaling agent, which includes a representation layer, a mechanism layer, and a prediction layer; S2.2, Encode the current observation state information through the representation layer, and use the encoded potential state as the root node of the search tree and the current potential state; S2.3, Based on the Monte Carlo tree search method, output the next round potential state and the corresponding predicted instant reward through the mechanism layer according to the current potential state and the average value of the current node, and use the next round potential state as the leaf node. S2.4 Add nodes to the search tree; S2.5 Backtrack from leaf nodes to root nodes to obtain the number of visits to each node in the path, and update the average value of each node based on the predicted instant reward and the number of visits to each node; S2.6 Based on the potential state of the next round, output the policy distribution and state value estimate through the prediction layer, and determine whether the current round has reached the preset round; S2.7 If not, return to S2.8 until the current round reaches the preset round; S2.8 Select the optimal path from multiple paths in the search tree; S2.9 Based on the policy distribution and state value estimate of each node corresponding to the optimal path, output the optimal elastic scaling strategy for the next microservice through the elastic scaling agent.

[0046] It should also be noted that when constructing the initial search tree, the encoded latent state is used as the root node of the search tree, and then the encoded latent state is used as the current latent state to construct the leaf nodes.

[0047] S2.3 includes: S2.3.1, determining the next round action based on the current potential state and multiple preliminary actions using an action selection formula; S2.3.2, determining whether the next round action is consistent with the actions of each child node corresponding to the current potential state; S2.3.3, if not, outputting the next round potential state and corresponding immediate reward through the mechanism layer based on the current potential state and the next round action, and adding the next round potential state as a leaf node to the search tree; S2.3.4, if yes, taking the child node corresponding to the consistent action as the current potential state, and returning to S2.3.1 for iterative execution.

[0048] Suppose the root node is S0, and the child nodes of S0 are S1 and S2, where the actions corresponding to S1 and S2 are a1 and a2, respectively. The current potential state is S0, and the next action is a3. We determine whether a3 is consistent with a1 or a2. If a3 is consistent with a2, then S2 is taken as the current potential state. Based on the current potential state and multiple initial actions, the action a3 is re-determined using the action selection formula. If S2 has no child nodes or the action corresponding to the child node of S2 is inconsistent with the re-determined a3, then the next potential state S3 is added to the search tree as a child node of S2. All child nodes in the search tree are leaf nodes.

[0049] In the specific implementation, refer to Figure 4 , Figure 4 This is a diagram of the MuScaler algorithm framework of the first embodiment of the microservice elastic scaling method based on deep Monte Carlo tree search of the present invention. First, the three deep neural networks of the elastic scaling agent are initialized, namely the representation layers. Mechanism level and prediction layer The parameters are set, and the experience replay pool D is initialized. The microservice deployment environment is initialized. At each time step of the training phase, service metric data is first obtained through the monitoring module. Call the trained load prediction module to obtain load prediction data. By splicing the two, the environmental observation status can be obtained. Then, three deep neural networks are used to simulate the planning process of MCTS selection, expansion, and backtracking, updating the MCTS search tree node information and obtaining the strategy. After interacting with the microservice system environment, rewards are obtained through the performance evaluation module. Finally, the experience gained from this interaction is stored in the experience replay pool. A small batch is randomly selected from the experience replay pool, and the three deep neural networks of the agent are trained using the joint loss function and their parameters are updated.

[0050] MuScaler's MCTS planning process: The MCTS planning process uses a representation network (i.e., representation layer), a mechanism network (i.e., mechanism layer), and a prediction network (i.e., prediction layer) to simulate the environmental interaction process, increasing empirical sampling and accelerating model convergence through planning. Its input is the observed state sequence. The output is a strategy, used for action selection in a real microservice deployment environment.

[0051] The detailed process is as follows: 1. Initialize the Monte Carlo search tree. Representation layer. Receive the state sequence of the current time step Encode it as a latent state and will As the root node of the Monte Carlo tree search, initialize the search tree structure.

[0052] 2. Selection phase starts from the root node Initially, based on information such as the number of visits to each edge (i.e., the number of visits to the current node), the policy prior (i.e., the predicted probability of the action at the current node), and the current average value (i.e., the average value of the current node), the leaf node action is iteratively selected using the action selection formula, with the goal of maximizing the upper limit of the confidence interval.

[0053] In the formula, This represents the current average value of the current node. This represents the predicted probability of the current node's action. This represents the number of visits on the current node. This represents the number of times the initial action b is accessed on the current node. and It is a constant, a constant and Used for adjustment strategies Relative value The impact, The action chosen for step k.

[0054] It should also be noted that the initial action b is one of several possible actions, and one of these initial actions is selected as the target action.

[0055] Expand the search tree downwards along the chosen path until an unexpanded node is encountered.

[0056] It should also be noted that during the selection phase, the simulation starts from the root node. Initially, at each time step k=1,2,...,l in the simulation, based on the potential state... Actions are selected by maximizing the upper limit of the confidence interval in order to balance exploration and the pursuit of value.

[0057] 3. Expansion and Simulation Phase. For the selected action... Mechanism network Receive the current potential state With action Output the potential state for the next round. With corresponding instant rewards :

[0058] Newly expanded state nodes Added to the search tree and through the prediction network Output the policy distribution under this node. With state value estimation :

[0059] Repeat the above expansion process until the round limit is reached. Record the state and reward information on the simulated path completely. Select the optimal path from the search tree to obtain the optimal elastic scaling strategy for the next microservice. The optimal elastic scaling strategy for the next microservice includes the strategy distribution and state value estimation of each node on the optimal path.

[0060] In the specific implementation, the number of visits to the child nodes under the root node is determined, the child node with the maximum number of visits is selected, and then the node with the maximum number of visits under that child node is selected until the end of the node is reached. This path is then taken as the optimal path.

[0061] 4. Backtracking Phase. After each round, backtrack from the leaf nodes (i.e., newly added nodes) to the root node, updating the statistics of each path edge: Among them, the number of node visits: → ; Reward accumulation value update: Use the search value from the backtracking path With future discount value Estimate long-term value :

[0062] And update the average value estimate for each node. And retain the policy distribution in all backtracking processes for subsequent action sampling and policy training.

[0063] Assume the backtracking path is S3, S2, S0, with S3 visited 1 time, S2 visited 2 times, and S3 visited 3 times. The immediate reward for S3 is r3, the immediate reward for S2 is r2, and the immediate reward for S0 is r0. The average value estimate for S3 is... =r³ / 1, the average value estimate of S2 is... = (r2 + r3) / 2, the average value estimate of S0 is: = (r0 + r2 + r3) / 3.

[0064] S3, determine the policy distribution and state value estimation according to the next microservice optimal elastic scaling policy, and perform performance evaluation on the microservice system based on the policy distribution and the state value estimation.

[0065] The approach to performance evaluation of microservice systems based on policy distribution and state value estimation is as follows: control the execution of microservice systems based on policy distribution and state value estimation, calculate the observed instant reward through an instant reward function, and evaluate the performance of microservice systems based on the observed instant reward.

[0066] The instant reward function is:

[0067] In the formula, For observation of immediate rewards, The scaling factor is a constant value. As a regulating factor, ∈[0,1], The reward corresponding to the request response time. Rewards are given based on the amount of resources utilized.

[0068] Response time reward Defined as:

[0069] It is the maximum request response time specified by SLO. This is the penalty factor, a negative constant. It applies when the request response time... satisfy When the request response time is shorter, the reward value is higher; when the request response time violates the rule of law, the reward value is higher. At that time, a negative penalty factor is obtained. As a reward; Resource utilization rewards Defined as:

[0070] in, This represents the normalized average CPU utilization across all microservices. The normalized average memory utilization for all microservices:

[0071]

[0072] in, Indicates the total number of microservices. and These are two constants used for CPU and memory normalization to eliminate the dimensional differences between CPU and memory.

[0073] S4. Calculate the joint loss value of the elastically scalable agent using the joint loss function based on the performance evaluation results.

[0074] It should be noted that the immediate reward for observation can be used as a performance evaluation result.

[0075] The joint loss function is:

[0076] In the formula, For strategic losses, For the loss of value, To reward losses, For the action sequence in the k-th round at current time t, Let be the probability distribution of taking an action in the potential state at time t, round k. The long-term value of the k-th round at current time t. Estimate the state value for the k-th round at current time t. The instant reward for observation in the k-th round at current time t. Let be the predicted instant reward for the k-th round at the current time t, and c be the regularization coefficient.

[0077] S5, optimize the parameters of the elastic scaling agent based on the joint loss value, and perform microservice elastic scaling through the updated elastic scaling agent.

[0078] It should also be noted that the parameters of the elastic agent can be optimized using a step descent algorithm based on the joint loss value, so that the updated elastic agent can predict the optimal policy for the next cycle.

[0079] In this embodiment, the current observation state information of the microservice system is first obtained. This information is then input into the elastic scaling agent, which outputs the optimal elastic scaling strategy for the next microservice. The elastic scaling agent is constructed based on a combination of Monte Carlo tree search and deep reinforcement learning. Next, the strategy distribution and state value estimation are determined according to the optimal elastic scaling strategy for the next microservice. Based on the strategy distribution and state value estimation, the performance of the microservice system is evaluated. Finally, based on the performance evaluation results, the joint loss value of the elastic scaling agent is calculated using a joint loss function. The parameters of the elastic scaling agent are optimized based on the joint loss value, and the microservice is elastically scaled using the updated elastic scaling agent. This embodiment organically combines the foresight of heuristic search with the adaptability of reinforcement learning, improving the training efficiency and decision accuracy of the scaling strategy while enhancing its adaptability to highly dynamic service scenarios. It also solves the problems of slow response, slow strategy convergence, and large service quality fluctuations in existing scaling methods, achieving intelligent, efficient, and stable elastic scaling control of the system.

[0080] Reference Figure 5 , Figure 5This is a structural block diagram of the first embodiment of the microservice elastic scaling system based on deep Monte Carlo tree search of the present invention.

[0081] like Figure 5 As shown, the microservice elastic scaling system based on deep Monte Carlo tree search proposed in this embodiment of the invention includes: Data acquisition module 5001 is used to acquire the current observation status information of the microservice system; The elastic scaling module 5002 is used to input the current observation state information into the elastic scaling agent and output the optimal elastic scaling strategy for the next microservice. The elastic scaling agent is constructed based on a combination of Monte Carlo tree search and deep reinforcement learning. The performance evaluation module 5003 is used to determine the policy distribution and state value estimation according to the next microservice optimal elastic scaling strategy, and to perform performance evaluation on the microservice system based on the policy distribution and the state value estimation. The iterative optimization module 5004 is used to calculate the joint loss value of the elastically scalable agent based on the performance evaluation results and the joint loss function. The iterative optimization module 5004 is also used to optimize the parameters of the elastic scaling agent based on the joint loss value, and to perform microservice elastic scaling through the updated elastic scaling agent.

[0082] Other embodiments or specific implementations of the microservice elastic scaling system based on deep Monte Carlo tree search of the present invention can be found in the above-described method embodiments, and will not be repeated here.

[0083] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0084] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0086] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A microservice elastic scaling method based on depth Monte Carlo tree search, characterized in that, The method includes the following steps: S1, obtain the current observation status information of the microservice system; S2, input the current observation state information into the elastic scaling agent, and output the next microservice optimal elastic scaling strategy. The elastic scaling agent is constructed based on a combination of Monte Carlo tree search and deep reinforcement learning. S3, determine the policy distribution and state value estimation according to the next microservice optimal elastic scaling policy, and perform performance evaluation on the microservice system based on the policy distribution and the state value estimation; S4, Calculate the joint loss value of the elastically scalable agent based on the performance evaluation results using the joint loss function; S5, optimize the parameters of the elastic scaling agent based on the joint loss value, and perform microservice elastic scaling through the updated elastic scaling agent.

2. The method as described in claim 1, characterized in that, S1 includes: S1.1, Obtain the current operating metrics information of the microservice cluster and the future workload information of the microservice system; S1.2, determine the current observation status information based on the current operating index information and the future workload information.

3. The method as described in claim 1, characterized in that, S2 includes: S2.1, Input the current observation state information into the elastic scaling agent, the elastic scaling agent including a representation layer, a mechanism layer and a prediction layer; S2.2, The current observation state information is encoded through the representation layer, and the encoded potential state is used as the root node of the search tree and the current potential state; S2.3, Based on the Monte Carlo tree search method, the mechanism layer outputs the next round potential state and the corresponding predicted instant reward according to the current potential state and the average value of the current node, and adds the next round potential state as a leaf node to the search tree; S2.4, backtrack from the leaf node to the root node to obtain the number of visits to each node in the path, and update the average value of each node based on the predicted instant reward and the number of visits to each node; S2.5, based on the potential state of the next round, the prediction layer outputs the strategy distribution and state value estimation, and determines whether the current round has reached the preset round; S2.6 If not, return to S2.3 until the current round reaches the preset round; S2.7, Select the optimal path from the multiple paths within the search tree; S2.8, Based on the policy distribution and state value estimation of each node corresponding to the optimal path, the elastic scaling agent outputs the optimal elastic scaling strategy for the next microservice.

4. The method as described in claim 3, characterized in that, S2.3 includes: S2.3.1, Based on the current potential state and multiple preliminary actions, determine the next round action using an action selection formula; S2.3.2, determine whether the action in the next round is consistent with the action of each sub-node corresponding to the current potential state; S2.3.3, if not, then based on the current potential state and the next round action, the mechanism layer outputs the next round potential state and the corresponding instant reward, and adds the next round potential state as a leaf node to the search tree; S2.3.4 If so, then take the child node corresponding to the consistent action as the current potential state and return to S2.3.1 for iterative execution.

5. The method as described in claim 4, characterized in that, The action selection formula is as follows: In the formula, The average value of the current node. This represents the predicted probability of the current node's action. This represents the number of visits to the current node. This represents the number of times the initial action b is accessed on the current node. and It is a constant. The action chosen for step k.

6. The method as described in claim 1, characterized in that, The performance evaluation of the microservice system based on the policy distribution and the state value estimation includes: The microservice system is controlled to execute based on the policy distribution and the state value estimation, and the observation instant reward is calculated through the instant reward function. The instant reward function is: In the formula, For observation of immediate rewards, Scaling factor As a regulating factor, The reward corresponding to the request response time. Rewards corresponding to resource utilization; The performance of the microservice system is evaluated based on the immediate rewards for the observed data.

7. The method as described in claim 1, characterized in that, The joint loss function is: In the formula, For strategic losses, For the loss of value, To reward losses, For the action sequence in the k-th round at current time t, Let be the probability distribution of taking an action in the potential state at time t, round k. The long-term value of the k-th round at current time t. Estimate the state value for the k-th round at current time t. The instant reward for observation in the k-th round at current time t. Let be the predicted instant reward for the k-th round at the current time t, and c be the regularization coefficient.

8. A microservice elastic scaling system based on depth Monte Carlo tree search, characterized in that, The system includes: The data acquisition module is used to acquire the current observation status information of the microservice system; The elastic scaling module is used to input the current observation state information into the elastic scaling agent and output the optimal elastic scaling strategy for the next microservice. The elastic scaling agent is constructed based on a combination of Monte Carlo tree search and deep reinforcement learning. The performance evaluation module is used to determine the policy distribution and state value estimation based on the next optimal elastic scaling strategy for the microservice, and to perform performance evaluation on the microservice system based on the policy distribution and the state value estimation. The iterative optimization module is used to calculate the joint loss value of the elastically scalable agent based on the performance evaluation results using a joint loss function. The iterative optimization module is also used to optimize the parameters of the elastic scaling agent based on the joint loss value, and to perform microservice elastic scaling through the updated elastic scaling agent.

Images