A kubernetes microservice optimal deployment method and device based on reward accumulation deep reinforcement learning and a medium

Abstract

The application discloses a Kubernetes micro-service optimal deployment method and device based on reward accumulation deep reinforcement learning and a medium, relates to the field of electric digital data processing, and comprises the following steps: S1, constructing a multi-target micro-service deployment optimization problem; S2, constructing a micro-service resource preference function combined with an optimization target; S3, constructing a reinforcement learning model; S4, constructing a reward function and training the model; and S5, obtaining an optimal deployment strategy online according to a real state through the trained model. The application considers both the lowest service response delay and resource load balancing by constructing a multi-target micro-service deployment optimization problem, effectively converts the Kubernetes deployment requirement into a computable and solvable optimal problem, designs a micro-service resource preference function, combines the function with the optimization target, reduces the calculation complexity during model training, guarantees the real-time performance of the Kubernetes deployment, and thus meets the development requirement of the micro-service architecture.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric digital data processing, in particular to cloud computing resource optimization, and specifically to a Kubernetes microservice optimal deployment method based on reward accumulation deep reinforcement learning, a device and a medium. BACKGROUND

[0002] With the rapid development of cloud computing, traditional service development methods are gradually being replaced by microservice development methods due to poor scalability and maintenance difficulties. Microservice architecture splits a single application into multiple independent microservices, which call each other through interfaces without affecting each other's running results. This method not only improves the flexibility and maintainability of the system, but also reduces resource redundancy caused by resource occupation of a single application. At the same time, with the rapid development of virtualization technology and edge computing, container-based microservice development is becoming the mainstream of microservice design. Containers on different edge nodes can provide virtualized isolation space for a single microservice to ensure that they do not interfere with each other.

[0003] However, the computing resources required by each microservice are irregular, and the total amount of resources of the node is limited. Therefore, how to efficiently utilize computing resources and reasonably allocate them within limited resources has become an important problem in microservice deployment. Traditional microservice deployment mostly involves coarse-grained resource allocation, which is easy to cause resource waste. In addition, while reasonably allocating resources, microservice deployment should also consider quality of service (QOS). In particular, when multiple user requests occur simultaneously, it is necessary to ensure fast service response. Therefore, an efficient microservice deployment method should ensure faster user request response and fine resource utilization.

[0004] In recent years, there have been many studies focusing on microservice deployment. Some methods aim to effectively shorten service response time, which includes processing time and data transmission time of each service, but ignore the limitation of resources. Some methods consider optimizing resource load balancing and service response delay, but ignore the elastic expansion and user access under high concurrency. Some methods achieve microservice traffic routing under high concurrency and high traffic, but do not deploy microservices. Kubernetes is a comprehensive tool for microservice orchestration and deployment. It makes each microservice establish a container, and containers interact with each other through network communication. Its flexibility, open source and stability make it gradually become one of the mainstream tools to realize microservice architecture. However, although deploying microservices on Kubernetes has strong convenience, existing Kubernetes deployment strategies mostly rely on static rules or single target optimization (such as only focusing on resource utilization or response delay), which is difficult to balance the low delay demand under high concurrency scenario and resource load balancing, leading to service interruption risk or resource waste.

[0005] Therefore, there is an urgent need for a Kubernetes optimal deployment method that balances resource load and low service response delay to solve the above problems. SUMMARY

[0006] The present application overcomes the deficiencies of the prior art and provides a Kubernetes microservice optimal deployment method based on reward accumulation deep reinforcement learning, equipment and medium.

[0007] To achieve the above purpose, the technical solution adopted by the present application is as follows: First aspect: provide a Kubernetes microservice optimal deployment method based on reward accumulation deep reinforcement learning, comprising the following steps:

[0008] S1, construct a multi-objective microservice deployment optimization problem model;

[0009] S2, construct a microservice resource preference function, and combine the optimization target to reduce the computational complexity in the optimization problem model training;

[0010] S3, construct a reinforcement learning model;

[0011] S4, construct a reward function, and use an offline deep reinforcement learning method to train the reinforcement learning model;

[0012] S5, obtain the optimal deployment strategy according to the real state online through the trained reinforcement learning model.

[0013] In one preferred embodiment of the present application, in the step of S1, the multi-objective microservice deployment optimization model includes a user request response model, a resource load balancing model, a constraint condition, and two optimization targets of minimum user request service response delay and minimum cluster resource variance.

[0014] In one preferred embodiment of the present application, the obtaining of the user request response model includes the following steps:

[0015] S11, split the user request response delay into service running delay, request queuing delay and data transmission delay, and obtain the corresponding calculation method according to the corresponding calculation strategy respectively;

[0016] S12, the total user request response delay is obtained by adding the corresponding three delays.

[0017] In one preferred embodiment of the present application, the obtaining of the resource load balancing model includes the following steps:

[0018] S111, calculate the resource usage rate of CPU, memory, IO and network transmission rate for one user request respectively, and obtain the variance of four types of computing resources;

[0019] S112, design resource weight, then the total resource load balancing can be calculated according to variance and weight.

[0020] In a preferred embodiment of the application, the obtaining of the multi-target micro-service deployment optimization problem comprises the following steps:

[0021] S1111, to meet the target demand, set multiple optimization targets, respectively, the user request service response delay is lowest, and the resource load is most balanced;

[0022] S1112, to meet the target demand, the constraint condition in the micro-service deployment process needs to be met, including the sum of all computing resources running on each node needs to be less than the amount of computing resources available on the node, and each micro-service guarantees that at least one corresponding container is deployed on one node.

[0023] In a preferred embodiment of the application, in the step of S2, according to different computing resource requirements of different micro-services, normalize each type of computing resource, construct the micro-service resource preference function, and combine the optimization target, the following steps are taken to obtain:

[0024] S21, design the use degree of the first l class resource required by the first j micro-service compared with the total available resource on the node i , compare the use degree of each resource with the CPU usage, obtain the resource requirement preference weight of the micro-service j , which can be calculated by the following micro-service resource preference function: l

[0025] ;

[0026] Among them, R indicates the available resource on the node; indicates the preference degree of the micro-service l to memory; indicates the preference degree of the micro-service l to CPU; indicates the preference degree of the micro-service l to IO; indicates the preference degree of the micro-service l to network transmission rate; the use degree of the memory resource required by the first l micro-service compared with the total available memory resource on the node i ; the use degree of the CPU resource required by the first l micro-service compared with the total available CPU resource on the node i ​​a degree of use of total available CPU resources on the node; a degree of use of IO resources required by the first l microservice compared to total available IO resources on the node i ; a degree of use of network transmission rate resources required by the first l microservice compared to total available network transmission rate on the node i ;

[0027] S22, combining the microservice resource preference function and the optimization target, a load balancing calculation method before each step of training of different microservices can be obtained:

[0028] ;

[0029] wherein, total resource variance is represented; a weight of the first j type of resource is represented; V CPU CPU resource variance in the cluster is represented; similarly, the smaller the minimum cluster total resource variance is, the more balanced the cluster resource consumption is for the microservice l .

[0030] In a preferred embodiment of the present application, in the step of S4, the process of training the model by the offline deep reinforcement learning method includes the following steps:

[0031] S41, at the beginning of each step of training, the agent calculates the deployment operation to be performed in the current state by using the model according to the current input network state, and at each deployment, in order to reduce the number of action spaces, the microservices in the microservice chain are deployed one by one, so that each action can be represented as ; wherein, n the edge node is represented;

[0032] S42, according to the current state, an evaluation value of the performance of the action is obtained after the reward function is calculated; at the same time, the reward is regarded as the feedback of the model to constrain the model to find the deployment action that maximizes the objective function;

[0033] S43, the cumulative reward estimation function is calculated by using the deployment trajectory path that completes the entire user request to evaluate the advantages and disadvantages of the state; and the entire model is updated after the agent completes the entire user request deployment scheme.

[0034] In a preferred embodiment of the present application, the calculation method of the reward function in each step of reinforcement learning is:

[0035] ;

[0036] wherein, in the function represents that there is only 1 worker node used in the training process; otherwise represents that the number of deployed worker nodes exceeds the number of microservices; and represents the weight of the reward function; represents the positive bias reward when the agent successfully deploys a microservice instance; represents the request queuing delay; represents the data transmission delay; represents the total service response delay; represents the traffic on the worker node i ; represents the total resource variance; UM represents the microservice.

[0037] The second aspect: the application provides a Kubernetes deployment system device cluster, comprising: a master node for Kubernetes orchestration, and a worker node for running each microservice; wherein, the Kubernetes microservice optimal deployment method in any one of the above is executed in the master node, and after the optimal deployment strategy is obtained, the microservice is deployed and run in the worker node;

[0038] The master node comprises: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the Kubernetes microservice optimal deployment method in any one of the above.

[0039] The third aspect: the application provides a computer readable storage medium, the computer readable storage medium stores computer instructions, the computer instructions are used to make the processor execute to realize the Kubernetes microservice optimal deployment method in any one of the above.

[0040] The application solves the defects in the background art, and has the following beneficial effects:

[0041] (1) The application provides a Kubernetes microservice optimal deployment method and device based on reward accumulation deep reinforcement learning and a medium, a multi-target microservice deployment optimization model is constructed, service response delay and resource load balancing are converted into a mathematically solvable problem, and constraint conditions such as node resource total quantity limitation and container deployment quantity are set, the reward accumulation deep reinforcement learning method adjusts the reward function weight dynamically, comprehensively evaluates the delay and load balancing indexes in each step of deployment, optimizes the model parameters by using the cumulative reward mechanism, realizes multi-target collaborative optimization, and thus avoids resource waste or excessively high service delay caused by single-target optimization, compared with the traditional static scheduling strategy, the application solves the optimal microservice deployment demand on Kubernetes in the prior art, can simultaneously meet the low delay and load balancing demand, and further improves the service stability and resource utilization rate of Kubernetes in a high concurrency scenario.

[0042] (2) In the application, a multi-target microservice deployment optimization model combining a microservice resource preference function is designed for the complexity of multi-resource dimension variance calculation, the use degree of memory, IO and other resources is compared with the CPU utilization rate, the resource demand preference weight is calculated, and the load balancing calculation is simplified by combining the resource weight, the multi-resource dimension variance calculation is converted into single-target optimization based on the preference weight, the function reduces the simultaneous calculation amount of the four types of resources in each step of training through normalization processing, high-dimensional matrix operation is reduced, and thus the calculation complexity of model training is reduced, compared with the traditional multi-target optimization mode which needs to traverse all resource combinations, the calculation resource consumption can be significantly reduced, the problem that microservices need low service response and high resource utilization rate when deploying on Kubernetes in a high concurrency is effectively solved, the real-time requirement of Kubernetes deployment is further met, and the stability and efficiency of microservice architecture development on Kubernetes can be improved.

[0043] (3) In the application, a reward feedback strategy based on reward accumulation is designed in the offline deep reinforcement learning method, and a microservice chain step-by-step deployment strategy is adopted, the deployment of the entire user request chain is disassembled into step-by-step decision of single microservice instance, the exponential state space is decoupled into a linear decision sequence, each action only selects a deployment node of a single microservice, and simultaneously, the reward accumulation mechanism accumulates multi-step rewards (instead of single-step feedback) through a trajectory path, drives the agent to learn a long-term optimal strategy, and suppresses the invalidity of training caused by action combination explosion, so that the convergence difficulty of the reinforcement learning algorithm caused by state space explosion is effectively avoided, and the real-time performance and reliability of the deployment strategy are further ensured. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only represent some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0045] Figure 1 is a flow chart of the Kubernetes microservice optimal deployment method based on reward accumulation deep reinforcement learning of the embodiment 1 of the present application;

[0046] Figure 2 is a system parameter association relationship diagram of the optimization problem model of the embodiment 1 of the present application;

[0047] Figure 3 is a deep reinforcement learning method implementation architecture diagram of the embodiment 1 of the present application;

[0048] Figure 4 is a Kubernetes implementation architecture diagram of the embodiment 1 of the present application;

[0049] Figure 5 is a cluster load balancing comparison diagram of the embodiment 1 of the present application;

[0050] Figure 6 is a service response comparison diagram of the embodiment 1 of the present application;

[0051] Figure 7 shows a master node structure schematic diagram that can be used to implement the embodiment 1 of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only represent some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0053] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, therefore, the scope of protection of the present application is not limited by the specific embodiments disclosed below.

[0054] SUMMARY

[0055] The core features of Kubernetes include declarative API, self-healing capability, service discovery and load balancing, and plug-in-based extensible scheduling mechanism, but the default scheduling algorithm has limitations in complex microservice scenarios: on the one hand, the traditional scheduling strategy uses coarse-grained indicators (such as the average CPU usage of the node) to evaluate resource load balancing, ignoring the coordinated optimization of multiple dimensions of resources such as memory and IO; on the other hand, the calculation of service response delay does not fully decompose the superimposed effect of running delay, queuing delay and transmission delay, resulting in insufficient accuracy of QoS guarantee.

[0056] To achieve low-delay response and high resource utilization at the same time, multi-objective optimization modeling is a feasible path. However, the applicant found that if multi-objective optimization (such as minimizing response delay and resource variance at the same time) is directly applied to Kubernetes scheduling, there may be potential problems, specifically:

[0057] 1. High computational complexity: due to high-dimensional calculation, the optimization target needs to calculate the variance of four types of resources (CPU / memory / IO / network) and three layers of delay (running, queuing, and transmission) at the same time. Assuming that the cluster contains N nodes, the computational complexity of single-class resource variance is O(N), and the superposition of four types of resources requires O(4N) node state scans. In a scenario of 100 nodes and 1000 containers, a single load balancing calculation requires 400 resource state traversals, and the convergence speed of real-time calculation of resource variance and delay may not meet the response requirements of Kubernetes.

[0058] 2. State space explosion: the state space needs to cover multiple variables such as node resources, container states, and user requests. Taking 10 nodes, 100 containers, and 5 microservice chains as an example (each variable is discretized into 10 states), the theoretical state space exceeds 10 15 combinations, making it difficult for traditional optimization methods to converge or impossible to solve.

[0059] To solve the above problems, the present application proposes a Kubernetes microservice optimal deployment method based on reward accumulation deep reinforcement learning, which simplifies the four-dimensional resource variance to a single-dimensional indicator by constructing a resource preference function, reduces the computational complexity, and designs a dynamic reward mechanism to decompose the deployment action into step-by-step decisions for microservice chains, avoiding the problem of state space explosion, optimizing the state space exploration efficiency, and thus realizing the coordinated optimization of low delay and load balancing, improving the deployment timeliness and resource utilization of Kubernetes in high-concurrency scenarios.

[0060] Embodiment 1: as shown in the following table, a Kubernetes microservice optimal deployment method based on reward accumulation deep reinforcement learning includes the following steps: Figure 1

[0061] ​S1. Construct a multi-objective microservice deployment optimization problem model;

[0062] In this implementation, considering the limited resources of each edge node in the microservice architecture and the need for low latency in user request response under high concurrency, deploying microservices on Kubernetes requires achieving the goals of load balancing and low service response latency. Therefore, a multi-objective microservice deployment optimization problem is constructed, and the optimal deployment strategy is obtained by solving this optimization problem.

[0063] Specifically, based on the Kubernetes deployment process, a system model is built, including parameters for the deployment process, such as:

[0064] User Request ;

[0065] microservices ;

[0066] node ;

[0067] Available resources on the node ;

[0068] container ;

[0069] and microservice resource requirements ;

[0070] in, i Indicates the first in the cluster i 1 node s Indicates the first s One user request, j Indicates the first node j Class resources, l Indicates the corresponding number l Microservices k Indicates the first k Each container. The relationships between the parameters are as follows: Figure 2 As shown.

[0071] Furthermore, considering the above parameters, the multi-objective optimization problem should satisfy the following constraints:

[0072] a : That is, the sum of the corresponding computing resources of all containers running on each node does not exceed the total amount of corresponding computing resources available on the node;

[0073] in, i Indicates the first i One working node; j Indicates the first working node j Class computing resources; Represents a container k The first of the runtime requirements j The amount of resources for this type of computing resource; Indicates the first i The working node is for the first l Each microservice runs in its corresponding container. k The number of; Indicates the first i On the first working node j The maximum available capacity of the resource class;

[0074] b : That is, each microservice guarantees that at least one corresponding container is deployed on at least one node;

[0075] in, , indicating the first i The corresponding user request was deployed on each node. s At least one corresponding container; , indicating the first s The microservice corresponding to each user request l Deployed on nodes i superior.

[0076] Furthermore, optimizing the target load balancing can be transformed into minimizing the variance of various resources in the cluster. Therefore, we first calculate the variance of each node. i Upper j Utilization of computing resources (CPU, memory, I / O, network transmission rate):

[0077] , obtain the first in the cluster j Average resource utilization rate of class resources ;

[0078] Among them, the resource utilization rate is for the first s If a user makes a request, then This represents calculating one request in the user request chain at the same time; thus, the first request can be obtained. j Variance of resource class:

[0079] ;

[0080] Because different resources have different priorities when running in a microservice, resource weights are designed accordingly. ,in j Indicates the first node j For computational resources, the total resource variance can be obtained as follows:

[0081] ;

[0082] in, Indicates the firstj Transposing the variance of the resource, the formula is the j Sum of the product of the weight of the resource and its variance, we can know V useage The smaller, the smaller the overall variance of the cluster, the more balanced the resources.

[0083] At the same time, the optimization goal of low service response delay can be divided into service running delay , request queuing delay and data transmission delay .

[0084] Specifically, for the s th user request, its corresponding service running delay is:

[0085] ;

[0086] Where, represents the single container running time of the microservice; represents the corresponding container of the i th user request deployed on the s th node; represents the number of corresponding containers of the i th worker node running for the l th microservice. k

[0087] For the s th user request, its corresponding data transmission delay is:

[0088] ;

[0089] Where, represents the data transmission time between different containers within the same node; is to represent the data transmission time between nodes.

[0090] For the th user request, its corresponding request queuing delay is calculated according to the M / M / 1 queuing theory, first represents the service rate of the microservice instance running on different edge nodes, ;

[0091] Where, i represents the computing rate of the physical edge node ; l represents the computing intensity of the microservice

[0092] For a microservice s in the microservice chain of user request l , its service rate in the worker node​i The total traffic on the cluster is represented as The total traffic on the cluster is represented as l The total traffic on the cluster is represented as The total traffic on the cluster is represented as The total traffic on the cluster is represented as l The total traffic on the cluster is represented as i The total traffic on the cluster is represented as The total traffic on the cluster is represented as l The total traffic on the cluster is represented as The total traffic on the cluster is represented as The total traffic on the cluster is represented as

[0093] .

[0094] In summary, the calculation method of the total service response delay is:

[0095] The total traffic on the cluster is represented as T The smaller the user request service response delay is.

[0096] Further, the multi-objective microservice deployment optimization problem includes the above two optimization objectives and two constraint conditions, and the optimization problem can be represented as:

[0097] ;

[0098] Wherein, min V useage Indicates the minimum cluster resource variance, and the load is balanced; min T Indicates the minimum user request service response delay, a And b Indicates the above constraint condition.

[0099] S2, construct a microservice resource preference function, and combine the optimization objective to reduce the calculation complexity in the optimization problem model training;

[0100] In this embodiment, for a user request microservice chain, the demand degree of different microservices for each type of computing resource is inconsistent, and the influence degree of different resources on the service is different. According to the total amount of available computing resources of the edge node, the different resource types are normalized, and the Indicates the demand of the i-th microservice for the j-th resource l The use degree of the total available resource on the node j The use degree of the total available resource on the node i The use degree of the total available resource on the node j The use degree of the total available resource on the node

[0101] ;

[0102] Wherein, Indicates the containerk the first j resource quantity of the computing resource; represents the maximum available capacity of the first i resource on the first j worker node.

[0103] Optionally, the normalization is one of linear normalization or min-max normalization.

[0104] Linear normalization refers to mapping data to a specific interval through linear transformation, and the calculation formula is: ; wherein, X represents the original data value (such as the CPU usage, memory occupation, etc. of a certain node, which is the index to be normalized); X max represents the maximum value of the index in all samples; represents the normalized value;

[0105] The min-max normalization formula is ; wherein, X represents the original data value; X max represents the maximum value of the index in all samples; X min represents the minimum value of the index in all samples; represents the normalized value;

[0106] Further, when the CPU usage on the edge node is too high, the service call delay in the system will significantly increase, which may cause service interruption or complete unavailability, therefore, the usage degree of each resource is compared with the CPU usage to obtain the resource demand preference weight l of the microservice , j = cpu, mem, IO, web which can be calculated by the following microservice resource preference function:

[0107] ;

[0108] wherein, represents the preference degree of the microservice l to memory; represents the preference degree of the microservice l to CPU; represents the preference degree of the microservice l to IO; represents the preference degree of the microservice l to network transmission rate; represents the first la degree of usage of memory resources required by the i-th microservice compared to total available memory resources on the node i ; a degree of usage of CPU resources required by the i-th microservice compared to total available CPU resources on the node l ; i a degree of usage of IO resources required by the i-th microservice compared to total available IO resources on the node ; l a degree of usage of network transmission rate resources required by the i-th microservice compared to total available network transmission rate on the node i ; l ; i

[0109] Further, by combining the microservice resource preference function and the optimization target, a calculation method of a new total resource variance of different microservices before each training step can be obtained:

[0110] ;

[0111] wherein, V CPU represents a variance of CPU resources in the cluster.

[0112] It can be seen that the microservice resource preference is calculated before each training step, and accordingly, the calculation complexity in each training step is reduced; similarly, the smaller the minimum cluster total resource variance , the more balanced the cluster resource consumption is for the microservice l .

[0113] S3, constructing a reinforcement learning model;

[0114] In this embodiment, in order to solve the above multi-objective microservice optimization problem, a deep reinforcement learning method based on reward accumulation is proposed; the method mainly includes two parts, one is to use the deep reinforcement learning method for model training offline, and the other is to perform online optimization and solving according to the trained model. The main factors of reinforcement learning theory include state, agent, action and reward. The reinforcement learning model is constructed by setting these factors, and the following can be obtained:

[0115] According to the previous system model, the state variables mainly include node state, container state and user request state, and the state at each training step is set as S , then

[0116] ;

[0117] wherein, ​​, respectively, represent the number of nodes, the available CPU resources on the nodes, the available memory resources, the available IO resources, and the available network transmission rate resources NTR on the nodes; , respectively, represent the number of containers on the nodes, and the computing resources required by the containers; , respectively, represent user requests, microservices corresponding to the user requests, and maximum request response delays.

[0118] Further, the action is an operation of deploying a container on an edge node under the condition that the agent performs the operation satisfying the constraint a and b In addition, a user request can require multiple microservices to provide a service response, and the microservices can be deployed on multiple corresponding containers on multiple edge nodes.

[0119] S4, build a reward function, and train the reinforcement learning model by using an offline deep reinforcement learning method, and an architecture diagram of the algorithm is shown in Figure 3 ;

[0120] In this embodiment, at the beginning of each step of training, the agent calculates, according to a current input network state, a deployment operation to be performed in the current state by using the model, and obtains a reward value to evaluate the performance of the action after the calculation according to the reward function based on the current state; meanwhile, the reward is regarded as feedback of the model to constrain the model to find a deployment action that maximizes the objective function.

[0121] Further, in order to reduce the number of action spaces, at each deployment, the microservices in a microservice chain corresponding to a user request are deployed one by one, that is, for each user request UR s , the corresponding microservice chain is , each step of training starts from and is deployed to ;

[0122] So that at each step of deployment, the model only meets part of the requirements of the user request; meanwhile, each step of deployment is performed on the corresponding edge node, so that the action space of the microservice deployment is the number of edge network nodes that can be selected when deploying a microservice instance, and each action can be represented as:

[0123] ;

[0124] By designing each step of action, the action space caused by a large number of user request chains and network nodes in reinforcement learning is greatly reduced, and the invalidity of the reinforcement learning algorithm is avoided.

[0125] Further, the reward function is designed as follows:

[0126] ​ ;

[0127] wherein, the function indicates that there is only 1 worker node used in the training process; otherwise indicates that the number of deployed worker nodes exceeds the number of microservices; and indicate the weight of the reward function; indicates the positive bias reward when the agent successfully deploys a microservice instance; indicates the request queuing delay; indicates the data transmission delay; indicates the total service response delay; indicates the traffic on the worker node i ; indicates the total resource variance.

[0128] In each training step, the reward is calculated according to the reward function and the current state, and the reward value obtained is used as feedback to constrain the model to obtain the optimal target function deployment action. When the constraint condition is violated, a penalty Loss is executed.

[0129] wherein, the cumulative reward is calculated using the deployment trajectory path that completes the entire user request to evaluate the advantages and disadvantages of the state, and then the entire model is updated after the agent completes the entire user request deployment scheme.

[0130] S5, obtaining the optimal deployment strategy online according to the real state through the trained reinforcement learning model.

[0131] In this embodiment, through continuous iterative training, the goal of the agent is to obtain the optimal action and deployment strategy with the maximum reward in the current environment. After obtaining the real data and state, the trained agent obtains the optimal deployment strategy for the current state in the real environment; the entire deep reinforcement learning method implementation architecture is as shown in Figure 3 .

[0132] Specifically, the hardware environment of the Kubernetes optimal microservice deployment method based on reward accumulation deep reinforcement learning is shown in Table 1, and the required computing resources of each microservice in Example 1 are shown in Table 2.

[0133] Table 1: Hardware environment of Kubernetes cluster in Example 1

[0134]

[0135] Table 2: Required resources of each microservice in Example 1

[0136]

[0137] As the Kubernetes scheduling algorithm becomes more flexible with the development of microservices, the scheduling method can be replaced by plugins. In the experimental process, models and methods were developed as scheduling tools to replace the default scheduling method as a plugin to achieve real-time container scheduling. Combined with Prometheus, real-time scheduling and management of the platform were achieved.

[0138] where Prometheus is an open-source system monitoring tool. The open-source nature and plug-in deployment make it easy to implement in Kubernetes. As shown in Figure 4 , Master represents the master node, and the above algorithms are implemented on the master node; Node represents the worker node; Pod is the smallest running unit in Kubernetes, which can run more than one container; In the process of microservice deployment in Kubernetes, Kubelet executes the result of the optimal deployment strategy, that is, deploying containers to nodes, which contains cAdvisor, which stores the resource usage of pods in the cluster, and the Node-Exporter component is responsible for collecting monitoring data on the node and transmitting it to Prometheus for display; Controller Manager is the cluster manager of Kubernetes, and Kube-proxy records the traffic of each container; Etcd saves the running results; Scheduler records the method of the optimal deployment strategy, and API Server is the communication interface of Kubernetes. The above components communicate through the API server. According to the same user request, the optimal deployment method is implemented on the master node and deployed to the node. The results are compared in terms of response time and load balancing.

[0139] Figure 5 and Figure 6 respectively show that the microservice optimal deployment method based on reward accumulation deep reinforcement learning (DRFLM) has a total resource variance and average service response time compared with the Kubernetes default scheduling method (Default), deep Q learning method (DQL) and first fit algorithm (FFD) under the same number of user requests (Number of user requests).

[0140] From Figure 5In the middle, in order to clearly show the difference of variance value, the corresponding order of magnitude is increased, and the results after the deployment of the above four methods can be obtained through the monitoring and calculation of the cluster resources; it can be seen that with the increase of the number of user requests, the variance of the above four methods increases, which shows that the more the number of user requests, the more difficult it is to control the resource use balance, but when the number of user requests is the same, the variance of the microservice optimal deployment method based on reward accumulation deep reinforcement learning is the lowest, which shows that the method can effectively ensure the more balanced use of cluster resources.

[0141] From Figure 6 In the middle, through the above four methods, the average service response time under different user request quantities is collected; it can be seen that with the increase of the number of user requests, the average service response time also increases, because the queuing time and algorithm running time are increasing, but when the number of user requests is the same, the average service response time of the microservice optimal deployment method based on reward accumulation deep reinforcement learning is the lowest, which shows that the method can effectively reduce the service response delay.

[0142] Through the above comparison, it shows that the microservice deployment of the method on Kubernetes has high efficiency and stability, and can meet the needs of low service response, high resource utilization, stability and real-time of the system developed on the microservice architecture and Kubernetes.

[0143] Specifically, the present application considers the service response delay and the limited computing resources of the nodes in the Kubernetes deployment process, effectively converts the Kubernetes deployment requirements into a computable and solvable optimal problem by constructing a multi-objective microservice deployment optimization problem, taking into account the two objectives of the lowest service response delay and resource load balancing; at the same time, a microservice resource preference function is designed, which is combined with the optimization target to reduce the computational complexity during model training and ensure the real-time of Kubernetes deployment, thereby meeting the development needs of the microservice architecture.

[0144] And the multi-objective microservice deployment optimization model is a step-by-step serialization of actions taken under dynamic constraints and states to maximize expected returns, so the learning ability of the reinforcement learning method is suitable for solving the above microservice deployment optimization problem; by designing the action space of each training step, combining the feedback mechanism based on reward accumulation, and obtaining the best agent through offline deep reinforcement learning training, and then solving online according to the real state, not only avoids the too large action space of reinforcement learning, but also meets the real-time requirement of the deployment strategy, and also obtains the optimal deployment strategy that meets the needs.

[0145] Embodiment 2: The Kubernetes deployment system device representing the cluster edge node in Embodiment 1 of the present application is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers, and can also be cloud servers, also known as cloud computing servers or cloud hosts, which are a host product in the cloud computing service system. The components shown in this paper, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the application described and / or claimed herein.

[0146] The Kubernetes deployment system device cluster includes: a master node for Kubernetes orchestration, and a worker node for running microservices; wherein the method provided by the present application is executed in the master node, and after the optimal deployment strategy is obtained, the microservices are deployed and run in the worker node;

[0147] As shown in Figure 7 The master node 10 includes at least one processor 11 and a memory, such as a read-only memory (Read-Only Memory, ROM) 12, a random access memory (Random Access Memory, RAM) 13, etc., which are in communication connection with the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor 11, and the computer program is executed by the at least one processor 11 to enable the at least one processor 11 to execute the method provided by the present application.

[0148] Further, the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the master node 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0149] The plurality of components in the master node 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the master node 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunications networks.

[0150] Further, the processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Process (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, e.g., the method of resource management for a database.

[0151] In some specific embodiments, the method of resource management for a database can be implemented as a computer program tangibly embodied in a computer readable storage medium, e.g., the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the master node 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the method of resource management for a database described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the method of resource management for a database by other any appropriate means, e.g., by means of firmware.

[0152] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a System on Chip (SOC), a Complex Programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0153] Computer programs used to implement the present methods can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the machine, implements the functions / acts specified in the flow diagrams and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as part of a standalone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0154] In the context of the present application, the computer readable storage medium stores computer instructions for causing a processor to implement the method of resource management of a database provided by the present application when executed. The computer readable storage medium can be a tangible medium which can contain or store the computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium can be a machine readable signal medium. More specific examples of the machine readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0155] To provide for interaction with a user, the systems and techniques described here can be implemented on a host node having a display device (e.g., a Cathode Ray Tube (CRT) or a Liquid Crystal Display (LCD) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the host node. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0156] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0157] Optionally, the computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private server (VPS) services.

[0158] The above is described according to the ideal embodiments of the present application, and for those skilled in the art, it is obvious that the present application is not limited to the details of the above exemplary embodiments, and the present application can be realized in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended to include all changes falling within the meaning and scope of the equivalent elements of the claims. Any reference signs in the claims should not be regarded as limiting the claims involved.

[0159] In addition, it should be understood that, although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be properly combined to form other embodiments that those skilled in the art can understand.

Claims

1. A Kubernetes microservice optimal deployment method based on reward accumulation deep reinforcement learning, characterized in that, The method comprises the following steps: S1, constructing a multi-objective micro-service deployment optimization problem model; S2, constructing a micro-service resource preference function, and combining the optimization target to reduce the calculation complexity in the optimization problem model training; S3, constructing a reinforcement learning model; S4, constructing a reward function, and training the reinforcement learning model by using an offline deep reinforcement learning method; S5, obtaining an optimal deployment strategy online according to a real state by using the trained reinforcement learning model; In the step S1, the multi-objective micro-service deployment optimization problem model comprises a user request response model, a resource load balancing model, a constraint condition, and two optimization targets of minimum user request service response delay and minimum cluster resource variance; In the step S2, different types of computing resources are normalized according to different computing resource requirements of different micro-services, the micro-service resource preference function is constructed, and the optimization target is combined, The method comprises the following steps: S21、design representing the first l class resource demand of the j microservice requirement i Compared with the total available resources on the node j , the usage degree of the above each resource is compared with the CPU usage, and the resource demand preference weight of the microservice l is obtained , which can be calculated by the following microservice resource preference function: ； wherein, R represents the available resources on the node; represents the microservice l the preference degree for memory; represents the microservice l the preference degree for CPU; represents the microservice l the preference degree for IO; represents the microservice l the preference degree for network transmission rate; represents the usage degree of the memory resource required by the l thmicroservice compared to the total available memory resource on the node i ; represents the usage degree of the CPU resource required by the l thmicroservice compared to the total available CPU resource on the node i ; represents the usage degree of the IO resource required by the l thmicroservice compared to the total available IO resource on the node i ; represents the usage degree of the network transmission rate resource required by the l thmicroservice compared to the total available network transmission rate on the node i ; S22, the micro-service resource preference function and the optimization target are combined, and a load balancing calculation method of different micro-services before each training step is obtained: ； wherein, represents the total resource variance; represents the weight of the j class resource; V CPU represents the variance of CPU resources in the cluster; the smaller the minimum cluster total resource variance, the more balanced the cluster resource consumption for the microservice l . In the step S4, the calculation method of the reward function in each step of reinforcement learning is as follows: ； wherein the function indicates that there is only 1 worker node used in the training process; otherwise indicates that the number of deployed worker nodes exceeds the number of microservices; and indicate the weight of the reward function; indicates the positive bias reward when the agent successfully deploys a microservice instance; indicates the request queuing delay; indicates the data transmission delay; T indicates the total service response delay; indicates the traffic on the worker node i ; indicates the total resource variance; UM indicates the microservice.

2. The Kubernetes microservice optimal deployment method based on reward accumulation deep reinforcement learning according to claim 1, characterized in that: The user request response model is obtained by the following steps: S11, the user request response delay is split into service running delay, request queuing delay and data transmission delay, and the corresponding calculation methods are obtained according to the corresponding calculation strategies; S12, the total user request response delay is obtained by adding the corresponding three delays.

3. The Kubernetes microservice optimal deployment method based on reward accumulation deep reinforcement learning of claim 1, characterized in that: The resource load balancing model is obtained by the following steps: S111, the resource usage rates of CPU, memory, IO and network transmission rate for one user request are calculated respectively, and the variances of the four types of computing resources are obtained; S112, the resource weight is designed, and the total resource load balancing can be calculated according to the variance and the weight.

4. The Kubernetes microservice optimal deployment method based on reward accumulation deep reinforcement learning of claim 1, characterized in that: The multi-objective micro-service deployment optimization problem is obtained by the following steps: S1111, in order to meet the target demand, multiple optimization targets are set, which are respectively minimum user request service response delay and most balanced resource load; S1112, in order to meet the target demand, the constraint conditions in the micro-service deployment process need to be met, including that the sum of all computing resources running on each node needs to be less than the amount of available computing resources on the node, and each micro-service needs to be guaranteed to be deployed on at least one corresponding container.

5. The Kubernetes microservice optimal deployment method based on reward accumulation deep reinforcement learning of claim 1, wherein: In the step S4, the process of training the model by using the offline deep reinforcement learning method comprises the following steps: S41, at the beginning of each step of training, the agent calculates the deployment operation to be performed in the current state according to the current input network state with the model, and at each deployment, in order to reduce the number of action spaces, the microservices in the microservice chain are deployed one by one, so that each action can be represented as ; wherein, n is an edge node; S42, according to the current state, a reward value is obtained after the reward function is calculated to evaluate the performance of the action; at the same time, the reward is regarded as the feedback of the model to constrain the model to find the deployment action that maximizes the objective function; S43, the cumulative reward estimation function is calculated by using the deployment trajectory path of completing the whole user request to evaluate the advantages and disadvantages of the state; and the whole model is updated after the agent completes the whole user request deployment scheme.

6. A cluster of Kubernetes deployment system devices, characterized in that, The method comprises: A master node for Kubernetes orchestration, and a worker node for running microservices; wherein the Kubernetes microservice optimal deployment method in any one of claims 1-5 is executed in the master node, and after the optimal deployment strategy is obtained, the microservices are deployed and run in the worker node; The master node comprises at least one processor and a memory connected in communication with the at least one processor; wherein the memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the Kubernetes microservice optimal deployment method in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling the processor to implement the Kubernetes microservice optimal deployment method in any one of claims 1-5 when executed.

Images