A Kubernetes cluster resource dynamic adjustment method and electronic device
Through the LSTM-Kalman filter prediction model, the Kubernetes cluster resources are dynamically adjusted, which solves the problem of insufficient real-time and compatibility of resource management in the existing technology, and realizes resource stability and efficient utilization under traffic fluctuations.
Patent Information
- Application Number
- CN202010709708.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-22
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-07-22
AI Technical Summary
The existing Kubernetes cluster resource management technology is difficult to achieve real-time dynamic adjustment when facing traffic fluctuations, resulting in insufficient resources or redundancy, affecting service stability and resource utilization. In addition, existing solutions such as HPA and VPA are in real-time and There is a lack of compatibility.
The LSTM-Kalman filter is used to pre-train with historical monitoring index data, build a prediction model, dynamically adjust the Pod resource limit value, and handle noise through sliding windows and Kalman filters to achieve peak resource consumption prediction and update.
Real-time resource adjustment during traffic fluctuations is realized, service stability and resource utilization are ensured, application version differences and cluster heterogeneity problems are avoided, large-scale pressure testing is avoided, and resource utilization and adaptability are improved.
Smart Images

Figure CN113971066B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computers, and in particular to a Kubernetes cluster resource dynamic adjustment method and electronic device. Background Art
[0002] With the rise of container-as-a-service technology supported by Docker, Kubernetes (a container cluster management technology that can realize container orchestration management) has been sought after by the industry for its powerful container orchestration technology. It has become the "de facto standard" for container cluster management and is even known as the "Linux of the cloud era". In order to prevent applications from consuming too many resources and affecting the performance of other applications and nodes, it is necessary to implement application resource control isolation through the resource restriction mechanism of Kubernetes and the Cgroup technology of Docker. Usually, when deploying applications, static deployment can only be performed through configuration files, and the demand for resources such as CPU and Memory needs to be manually estimated. Obviously, when the reserved resources are insufficient, the container will cause CPU throttle or Out of memory kill, affecting the service quality. It is necessary to create a new Pod (the most basic operation unit of Kubernetes, containing one or more closely related business logic containers, and the smallest unit of cluster scheduling) and deploy it again. When redundant configuration occurs, the cluster resource utilization rate will be low, affecting the resources that can be allocated to the node, resulting in other Pods being unable to be scheduled to the node.
[0003] Generally, if there is no resource limit for Pod, Pod will use up all the resources on the node without limit, resulting in resource competition and a huge impact on the stability of the service. In the actual production environment, traffic will change regularly, so resource utilization will change accordingly. Static deployment will lead to insufficient or redundant resources. In order to dynamically adjust resources when resources fluctuate, the open source community provides HPA (horizontal Pod autoscaling, https: / / kubernetes.io / docs / tasks / run-application / horizontal-pod-autoscale / ) and VPA (vertical autoscaling, https: / / github.com / kubernetes / community / blob / master / contributors / design-proposals / autoscaling / vertical-pod-autoscaler.md). The principle of HPA is to run a Controller through the Controller Manager component of Kubernetes, periodically monitor the Pod resource usage, and automatically increase the number of Pods when it is higher than the set threshold. When it is lower than a certain threshold, it automatically reduces the number of Pods, and dynamically change resources by adjusting the number of Pods in the cluster. VPA is similar to HPA, but it adjusts the resource request value of a single Pod. By monitoring the data of components, it calculates the recommended resource value of the Pod, allows appropriate scheduling on the node, and provides appropriate resources for each Pod. When the actual resource consumption value of the Pod taken over by VPA differs greatly from the recommended value, the Pod will be evicted and re-established and deployed to the cluster again, thereby achieving the purpose of dynamic resource change.
[0004] For the stability of services, the industry will also conduct full-link stress testing, using the cAdvisor monitoring system integrated in Kubernetes to predict the number of replicas and resources required by the Pod in advance, and adopt redundant configuration when it is impossible to accurately budget resources. In addition, the academic community generally obtains the performance curve of resource changes through experiments, determines a suitable resource reservation value, and reserves the corresponding resources in advance. For example, the document [Xu, G., Xu, C.-Z MEER: Online Estimation of Optimal Reservations for Long Lived Containers in In-Memory Cluster Computing. ICDCS2019] proposes a container-sensitive memory resource reservation mechanism for long life cycles, allowing containers to run with much less memory than the ideal, with only a certain performance loss. The document uses offline training to obtain the model, and then predicts the Pod resource limit online.
[0005] As mentioned above, if the HPA or VPA solution of the open source community is adopted, the real-time performance is not enough for online or real-time computing applications that are sensitive to time, CPU, Memory, network resources, etc. The applications usually deployed are heavyweight Java frameworks or Web applications of related technology stacks. Using HPA to adjust the number of Pods for horizontal expansion can increase the number of Pods in a short time, but it takes a long time from triggering the expansion to scheduling to the appropriate node through Kube-Scheduler (Kubernetes cluster scheduler, responsible for scheduling Pods to the appropriate node) and then pulling the image through the Docker client to start the container and start the service. As for the vertical expansion provided by VPA, it is necessary to expel the old Pod from the node, re-establish the Pod that meets the resource requirements, and then redeploy Kube-Scheduler through the API Server (the only operation entry for Kubernetes resource objects, other components must operate resource data through the API it provides). It is impossible to upgrade in place, which obviously does not meet the real-time requirements. In addition, VPN is still in alpha version and has not been merged into the official Kubernetes release. Moreover, HPA and VPA are currently incompatible. Only one can be used, otherwise there will be interference between the two.
[0006] For the offline and online stress tests of the entire link, there are methods for early prediction, or for example, the literature [Xie Wenzhou, Sun Yanxia. Research on resource prediction model based on Kubernetes load characteristics [J]. Network Security Technology and Applications, 2018 (04): 27-28.] Through a large number of experiments on certain types of applications, the resource reservation value is constantly changed, the performance indicators of the application are measured, and the resource and performance change curve is established, and finally a resource reservation value is obtained by compromise. However, there are thousands of applications in the heterogeneous clusters of the production environment, various application versions are frequently released and updated, the application links are complex, and the physical configurations of each node are different. It is not realistic to exhaustively perform stress tests on all applications to find the optimal or suboptimal resource adjustment solution, which is too expensive, time-consuming and labor-intensive. Even if it is possible, the test data of the old version may not be suitable for the new version, or the same data on different machines may have different performance due to different configurations, and the dynamic adjustment effect is not achieved. Summary of the invention
[0007] The present application provides a Kubernetes cluster resource dynamic adjustment method and electronic device, aiming to solve at least one of the above-mentioned technical problems in the prior art to a certain extent.
[0008] In order to solve the above problems, this application provides the following technical solutions:
[0009] A method for dynamically adjusting Kubernetes cluster resources includes the following steps:
[0010] Step a: Collect historical monitoring indicator data in the application, and obtain resource consumption time series data based on the monitoring indicator data;
[0011] Step b: constructing an LSTM-Kalman filter, inputting the resource consumption time series data into the LSTM-Kalman for pre-training, and obtaining a prediction model; wherein the LSTM-Kalman filter is a Kalman filter modified by using LSTM;
[0012] Step c: Collect new resource consumption time series data, input the new resource consumption time series data into the prediction model to predict the resource consumption peak value within a future preset time period, and use the resource consumption peak value as the resource limit value to update the resources.
[0013] The technical solution adopted by the embodiment of the present application also includes: in the step a, the historical monitoring indicator data collected in the application includes:
[0014] Monitor the containers running in the Pod and collect various monitoring indicator data; the collected monitoring indicator data includes at least one or a combination of CPU utilization and limit, file system read / write utilization and limit, and network message sending / receiving / discarding rate data;
[0015] The collected monitoring indicator data is stored in the time series database.
[0016] The technical solution adopted by the embodiment of the present application also includes: in the step a, obtaining the resource consumption time series data according to the monitoring indicator data includes:
[0017] The time series database is queried regularly to obtain the indicator data of interest, and the indicator data of interest is cached and sorted to form resource consumption time series data that can be directly used for prediction.
[0018] The technical solution adopted by the embodiment of the present application also includes: the caching and sorting of the indicator data of interest includes:
[0019] Use the sliding window to slide each indicator data in turn, and select the maximum value in the window as the new data set.
[0020] The technical solution adopted in the embodiment of the present application also includes: in the step b, the construction of the LSTM-Kalman filter includes:
[0021] Assuming that the observed value is a noisy estimate of the system state value, let the observation matrix H = I:
[0022] X(k)=f(X(k-1))+W(k),W~N(0,Q)
[0023] Z(k)=X(k)+V(k), V~N(0,R)
[0024] In the above formula, X(k) is the system state at time k, Z(k) is the observation value at time k, W(k) and V(k) represent the process and measurement noise respectively, and their covariances are Q and R respectively; f is the function model generated by the LSTMf model;
[0025] The prediction steps are:
[0026] X(k|k-1)=f(X(k-1|k-1))
[0027] P(k|k-1)=FP(k-1|k-1)FT+Q(k)
[0028] In the above formula, F represents the Jacobian matrix of f with respect to X(k-1|k-1), and Q(k) is obtained by LSTM Q given;
[0029] The update steps are:
[0030] Kg(k)=P(k|k-1)((P(k|k-1)+R(k))-1
[0031] X(k|k)=X(k|k-1)+Kg(k)(Z(k)–X(k|k-1)
[0032] P(k|k)=(I-Kg(k))P(k|k-1)
[0033] In the above formula, R(k) is obtained by LSTMr, and Z(k) represents the observation value at time k;
[0034] The request value in the configuration file is taken as X(k-1|k-1), and the indicator data collected at the current time k is taken as the observation value Z(k|k). The predicted value X(k|k-1) and the observation noise R(k) at the time k are obtained through the LSTMf and LSTMr networks respectively; X(k|k-1) passes through the LSTM Q The state noise Q(k) at time k is obtained; and the Kalman filter is recursively updated using R(k), Q(k), and X(k-1|k-1).
[0035] The technical solution adopted by the embodiment of the present application also includes: in the step c, the inputting the resource consumption time series data into the LSTM-Kalman for pre-training, and obtaining the prediction model also includes:
[0036] Model evaluation: The prediction model is iteratively trained using new monitoring indicator data until the prediction model loss converges or stops iterative training when it only oscillates within a preset range.
[0037] The technical solution adopted by the embodiment of the present application also includes: the model evaluation also includes:
[0038] Obtain new monitoring indicator data within a preset time period, compare the prediction result of the prediction model with the new monitoring indicator data, and determine whether model training is needed again: if the time consumption value greater than the set threshold is below the prediction result of the prediction model, the model will no longer be updated; otherwise, the new monitoring indicator data will be used to update the parameters of the prediction model and correct the prediction model.
[0039] The technical solution adopted by the embodiment of the present application also includes: in the step c, the prediction of the resource consumption peak value within the future preset time period also includes:
[0040] Wait for the LSTMf, LSTMr, LSTM QWhen the network losses of the three sub-cycles converge or decrease to a preset threshold, recursive estimation begins. The data is reported to the prediction model through the metric collector. The prediction model makes an optimal estimate of the resource consumption peak within a preset time period in the future, and uses the optimal estimate to update the container resources in the Pod.
[0041] Another technical solution adopted by the embodiment of the present application is: an electronic device, comprising:
[0042] at least one processor; and
[0043] a memory communicatively connected to the at least one processor; wherein,
[0044] The memory stores instructions that can be executed by the one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the following operations of the above-mentioned Kubernetes cluster resource dynamic adjustment method:
[0045] Step a: Collect historical monitoring indicator data in the application, and obtain resource consumption time series data based on the monitoring indicator data;
[0046] Step b: constructing an LSTM-Kalman filter, inputting the resource consumption time series data into the LSTM-Kalman for pre-training, and obtaining a prediction model; wherein the LSTM-Kalman filter is a Kalman filter modified by using LSTM;
[0047] Step c: Collect new resource consumption time series data, input the new resource consumption time series data into the prediction model to predict the resource consumption peak value within a future preset time period, and use the resource consumption peak value as the resource limit value to update the resources.
[0048] Compared with the prior art, the beneficial effects produced by the embodiments of the present application are as follows: the Kubernetes cluster resource dynamic adjustment method and electronic device of the embodiments of the present application first perform static redundant configuration deployment on the application, collect historical monitoring indicator data of the application under the real load environment, and perform online pre-training to obtain a more reliable prediction model; then the prediction model is iteratively trained until the model loss converges or oscillates only within the preset range, and new resource consumption time series data is continuously collected and input into the prediction model to predict the peak value of resource consumption in a short period of time in the future, and the value is updated as the resource limit value to realize dynamic Pod resource limit adjustment. Compared with the prior art, the present application has at least the following beneficial effects:
[0049] 1. In the face of traffic "peaks", it can play a good buffering role in a short period of time to ensure the real-time and stability of the service; in the face of traffic "troughs", it can return idle resources to the kernel, effectively improving the utilization of system resources, solving the problem of resource fragmentation, and at the same time having a certain degree of adaptive ability.
[0050] 2. Perform noise filtering while predicting to ensure data accuracy.
[0051] 3. It avoids the problem of model inapplicability caused by differences in application versions and cluster heterogeneity. There is no need to conduct a large number of stress tests to establish performance and resource relationship curves.
[0052] 4. By predicting the peak resource consumption in the future as the resource limit value, resources are dynamically restricted, and the configuration idea of using the local maximum value instead of the global maximum value is adopted, which improves resource utilization and achieves an adaptive effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 is a flow chart of a method for dynamically adjusting Kubernetes cluster resources according to an embodiment of the present application;
[0054] Figure 2 This is a schematic diagram of the structure of the Kubernetes cluster resource dynamic adjustment system according to an embodiment of the present application;
[0055] Figure 3 This is a schematic diagram of the Kalman filter after LSTM transformation;
[0056] Figure 4 It is a schematic diagram of the hardware device structure of the Kubernetes cluster resource dynamic adjustment method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0058] In view of the shortcomings of the prior art, the Kubernetes cluster resource dynamic adjustment method of the embodiment of the present application proposes an online prediction and dynamic adjustment solution for Kubernetes Pod resources. The entire solution includes three stages: data collection and collation, online synchronous training, and online prediction execution. First, static redundant configuration and deployment of the application is performed, and historical data such as CPU and Memory of the application under the real load environment are collected, and online pre-training is performed to obtain a more reliable prediction model; and the prediction model is iteratively trained until the model loss converges or oscillates only within the preset range, then the training is stopped, and the online prediction execution stage is entered, and new resource consumption time series data is continuously collected, and the peak value of resource consumption in a short period of time in the future is predicted in the prediction model, and the value is updated as the resource limit value to achieve dynamic Pod resource limit adjustment.
[0059] For details, please refer to Figure 1 , is a flow chart of a method for dynamically adjusting Kubernetes cluster resources in an embodiment of the present application. The method for dynamically adjusting Kubernetes cluster resources in an embodiment of the present application comprises the following steps:
[0060] Step 100: Data collection and collation: First, static redundant configuration deployment is performed on the application, and historical monitoring indicator data such as CPU and Memory of the application under a real load environment is collected;
[0061] In step 100, data collection and collation specifically include:
[0062] Step 101: Monitor the containers running in the Pod and collect various monitoring indicator data through cAdvisor integrated in Kubelet;
[0063] In this step, Kubelet is a process running on each node in the cluster. It is used to handle the tasks sent to this node by the Master node and is responsible for managing Pods and containers in Pods. cAdvisor is integrated into the Kubelet component. Kubelet obtains individual container usage statistics from cAdvisor, and then exposes these aggregated Pod resource usage statistics through the REST API. cAdvisor is an open source agent tool for analyzing container resource utilization and performance characteristics. It automatically searches for containers on the node and collects relevant indicator data for each container. In the embodiment of the present application, the indicator data collected by cAdvisor includes at least one or a combination of more than one of the following data: CPU utilization and limit, file system read / write utilization and limit, network message sending / receiving / discarding rate, etc. For details, please refer to Figure 2 , is a structural diagram of the Kubernetes cluster resource dynamic adjustment system according to an embodiment of the present application.
[0064] Step 102: Store the collected monitoring indicator data into TSDB (time series database) through Prometheus;
[0065] In this step, Prometheus is an open source service monitoring system and time series database, which provides a common data model and convenient data collection, storage and query interface. Prometheus Server, the core component of Prometheus, regularly pulls monitoring indicator data from statically configured monitoring targets or targets automatically configured based on service discovery mechanisms. In an embodiment of the present application, Prometheus Server periodically performs data capture operations through the metrics interface provided by cAdvisor, and then stores the captured data in a time series manner in the server-side memory cache or persists it in a storage device.
[0066] Step 103: Use the custom component Metrics Collector to periodically collect the monitoring indicator data of interest from Prometheus for secondary cache, and organize the monitoring indicator data to form resource consumption time series data that can be directly used for prediction;
[0067] In this step, the custom component Metrics Collector uses the query language PromQL to periodically query the time series database of Prometheus Server to obtain the indicator data of interest, cache and organize the indicators, and form resource consumption time series data that can be directly used for prediction, reducing the pressure on TSDB. At the same time, it also ensures the real-time nature of the time series data, which facilitates the Kalman filter to perform model training and estimation.
[0068] In an embodiment of the present application, when processing time series data, a sliding window is used to slide each piece of data in turn, and the maximum value in the window is selected as a new data set, so that the resource consumption of the value predicted by the filter will not exceed the predicted value in the future.
[0069] Step 200: Online synchronous training: Build an LSTM-Kalman filter, input the resource consumption time series data collected by the Metrics Collector into the LSTM-Kalman filter for pre-training, and obtain the prediction model LSTM-KF;
[0070] In step 200, the Kalman filter is an optimized autoregressive data processing algorithm. Kalman uses the dynamic information of the target to try to remove the influence of noise and obtain an optimal estimate of the target. Considering the inaccuracy of the collected data caused by factors such as system oscillation and mutual interference between applications, the Kalman filter is used to avoid these unstable factors and obtain a better estimate. At the same time, the model can achieve the optimal estimate of the new value through only 5 steps of recursion, but the existing Kalman filter needs to rely on the dynamic model in practical applications.
[0071] Kalman's spatial state equation can be described by a linear stochastic difference equation:
[0072] X(k)=AX(k-1)+W(k),W~N(0,Q) (1)
[0073] Z(k)=HX(k)+V(k),V~N(0,R) (2)
[0074] Formula (1) represents the state equation, and formula (2) represents the measurement equation. Where X(k) is the system state at time k, Z(k) is the observation value at time k, A is the state transfer matrix, H is the observation matrix, W(k) and V(k) represent the process and measurement noise respectively, assuming that they obey Gaussian white noise, and their covariances are Q and R respectively.
[0075] The existing Kalman filtering algorithm is as follows:
[0076] Use the current state model of the system to predict the next state of the system; assuming the current state of the system is k, predict the current state based on the previous state of the system:
[0077] X(k|k-1)=AX(k-1|k-1) (3)
[0078] In formula (3), X(k|k-1) is the result of prediction using the previous state, and X(k-1|k-1) is the best estimate of the previous state. The covariance corresponding to X(k|k-1) is:
[0079] P(k|k-1)=AP(k-1|k-1)AT+Q (4)
[0080] Equations (3) and (4) correspond to the system prediction process, and the predicted value under state k is obtained. Then, combined with the measured value Z(t), the optimal estimated value X(k|k) under state k is obtained:
[0081] X(k|k) = X(k|k-1) + Kg(k)(Z(k) – HX(k|k-1) (5)
[0082] In formula (5), Kg is Kalman Gain:
[0083] Kg(k) = P(k|k-1)HT / ((H*P(k|k-1)H'+ R) (6)
[0084] In order to make the filter continue to iterate, it is also necessary to update the covariance under the X(k|k) state:
[0085] P(k|k)=(I-Kg(k)H)P(k|k-1) (7)
[0086] In formula (7), I is a matrix of all 1s. When entering the k+1 state, P(k|k) is equal to P(k|k-1) in formula (4).
[0087] The above algorithms are very dependent on the dynamic model. However, for practical problems, the dynamic characteristics are unknown, and the state transfer matrix A and the observation matrix H cannot be given a priori. The noise distribution is also often assumed to obey the Gaussian distribution. There are many subjective factors in the assumptions of these parameters, which makes the model and the actual data not match well, resulting in large prediction errors.
[0088] Therefore, this application uses LSTM to transform the Kalman filter so that it can get rid of its excessive dependence on the dynamic model, so that the model parameters do not need to be given a priori. The specific transformation process is as follows:
[0089] This gives the new state space equation:
[0090] Assuming that the observed value is a noisy estimate of the system state value, let the observation matrix H = I:
[0091] X(K)=f(X(k-1))+W(k),W~N(0,Q) (8)
[0092] Z(k)=X(k)+V(k),V~N(0,R) (9)
[0093] In formula (9), f is the function model generated by the LSTMf model.
[0094] The new prediction step is:
[0095] X(k|k-1)=f(X(k-1|k-1)) (10)
[0096] P(k|k-1)=FP(k-1|k-1)FT+Q(k) (11)
[0097] In formula (11), F represents the Jacobian matrix of f with respect to X(k-1|k-1), and Q(k) is obtained by LSTM Q Given.
[0098] The new update steps are:
[0099] Kg(k)=P(k|k-1)((P(k|k-1)+R(k))-1 (12)
[0100] X(k|k) = X(k|k-1) + Kg(k)(Z(k) – X(k|k-1) (13)
[0101] P(k|k)=(I-Kg(k))P(k|k-1) (14)
[0102] In equations (12) and (13), R(k) is obtained by LSTMr, and Z(k) represents the observation value at time k.
[0103] like Figure 3 The figure shows a schematic diagram of the Kalman filter after LSTM transformation. During initialization, the request value in the configuration file can be used as X(k-1|k-1), and the indicator data collected at the current time k can be used as the observation value Z(k|k). The predicted value X(k|k-1) and the observation noise R(k) at time k are obtained through the LSTMf and LSTMr networks respectively. Q The state noise Q(k) at time k is obtained. Using R(k), Q(k), and X(k-1|k-1), the Kalman filter is recursively updated through equations (12), (13), and (14).
[0104] Based on the above, the Kalman filter after LSTM transformation takes into account both prediction and noise filtering, while getting rid of excessive dependence on the dynamic model. The model parameters do not need to be given a priori, and can be learned from the data to give full play to the correlation between the time series before and after; and the long-term prediction ability is enhanced, rather than just relying on the value of the previous moment, and the prediction accuracy can be significantly improved.
[0105] Step 300: Model evaluation: Use the new monitoring indicator data to iteratively train the prediction model to update and correct the prediction model until the model loss converges or oscillates only within a preset range, and obtain a trained prediction model LSTM-KF;
[0106] In step 300, the model evaluation process is specifically as follows: the Controller pulls the monitoring indicator data within the time period set by the Metrics Collector, and compares the prediction results of the prediction model with the monitoring indicator data reported from the Metrics Collector to determine whether the model needs to be trained again: if the time consumption value of the set threshold is below the optimal estimate of the prediction model, the model will no longer be updated; otherwise, the new monitoring indicator data will be used again to update the prediction model parameters and correct the model. The Controller also backs up a historical data peak value in case the filter is trained again and is temporarily used as a container resource limit.
[0107] Step 400: Enter the online prediction execution phase: continuously collect new resource consumption time series data, input the trained prediction model to predict the resource consumption peak value within the future preset time period, and update the value as the resource limit value;
[0108] In step 400, wait for LSTMf, LSTMr, LSTM Q When the three sub-loop network losses converge or drop to the preset threshold, recursive estimation begins. Metrics Collector reports data to the prediction model LSTM-KF. The prediction model LSTM-KF makes the best estimate of the resource consumption peak value in the next short period of time. The Executor directly uses the best estimate to update the limit value of the Container Cgroup where the container in the Pod is located. This application uses the best estimate to represent the peak value in the next short period of time, that is, using the local peak value instead of the global peak value, which can avoid redundant configuration, ensure the stability of the service, improve resource utilization, and achieve an adaptive effect.
[0109] Cgroup is a module in the Linux kernel used to implement resource usage and statistics. Docker uses Cgroup to isolate and limit resources. Kubelet, as the node agent of Kubernetes, implements all operations on Cgroup through the internal Container Manager module. Container Manager will use Cgroup to restrict resource usage. When the container resource request and limit are specified through configuration files such as yaml, Docker will set runtime indicators such as cpu.share, cpu.quota, cpu.period, mem.limit, etc. for the container. Executor updates resources by modifying these indicator files, thereby realizing dynamic Pod resource limit adjustment.
[0110] In the embodiment of the present application, since there are frequent calling relationships between the components, for the convenience of deployment, the components can be containerized with the help of the Sidecar idea (a way to separate application functions from the application itself as a separate process. It can add multiple functions to the application without intrusion) and deployed in the same Pod to facilitate operation and maintenance management.
[0111] Figure 4 Schematic diagram of the hardware device structure of the Kubernetes cluster resource dynamic adjustment method provided in the embodiment of the present application. Figure 4 As shown, the device includes one or more processors and a memory. Taking a processor as an example, the device may also include: an input system and an output system.
[0112] The processor, memory, input system and output system can be connected through a bus or other means. Figure 4 The example of connecting through bus is taken in the following.
[0113] The memory is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules. The processor executes various functional applications and data processing of the electronic device by running the non-transitory software programs, instructions and modules stored in the memory, that is, the processing method of the above method embodiment is implemented.
[0114] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processing system via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0115] The input system can receive input digital or character information and generate signal input. The output system can include display devices such as display screens.
[0116] The one or more modules are stored in the memory, and when executed by the one or more processors, perform the following operations of any of the above method embodiments:
[0117] Step a: Collect historical monitoring indicator data in the application, and obtain resource consumption time series data based on the monitoring indicator data;
[0118] Step b: constructing an LSTM-Kalman filter, inputting the resource consumption time series data into the LSTM-Kalman for pre-training, and obtaining a prediction model; wherein the LSTM-Kalman filter is a Kalman filter modified by using LSTM;
[0119] Step c: Collect new resource consumption time series data, input the new resource consumption time series data into the prediction model to predict the resource consumption peak value within a future preset time period, and use the resource consumption peak value as the resource limit value to update the resources.
[0120] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided in the embodiment of the present application.
[0121] The embodiment of the present application provides a non-transitory (non-volatile) computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions can perform the following operations:
[0122] Step a: Collect historical monitoring indicator data in the application, and obtain resource consumption time series data based on the monitoring indicator data;
[0123] Step b: constructing an LSTM-Kalman filter, inputting the resource consumption time series data into the LSTM-Kalman for pre-training, and obtaining a prediction model; wherein the LSTM-Kalman filter is a Kalman filter modified by using LSTM;
[0124] Step c: Collect new resource consumption time series data, input the new resource consumption time series data into the prediction model to predict the resource consumption peak value within a future preset time period, and use the resource consumption peak value as the resource limit value to update the resources.
[0125] An embodiment of the present application provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the computer performs the following operations:
[0126] Step a: Collect historical monitoring indicator data in the application, and obtain resource consumption time series data based on the monitoring indicator data;
[0127] Step b: constructing an LSTM-Kalman filter, inputting the resource consumption time series data into the LSTM-Kalman for pre-training, and obtaining a prediction model; wherein the LSTM-Kalman filter is a Kalman filter modified by using LSTM;
[0128] Step c: Collect new resource consumption time series data, input the new resource consumption time series data into the prediction model to predict the resource consumption peak value within a future preset time period, and use the resource consumption peak value as the resource limit value to update the resources.
[0129] The Kubernetes cluster resource dynamic adjustment method and electronic device of the embodiment of the present application first deploy static redundant configuration for the application, collect historical monitoring indicator data of the application under the real load environment, and perform online pre-training to obtain a more reliable prediction model; then iteratively train the prediction model until the model loss converges or oscillates only within the preset range, continuously collect new resource consumption time series data and input it into the prediction model to predict the peak value of resource consumption in a short period of time in the future, and update this value as the resource limit value to achieve dynamic Pod resource limit adjustment. Compared with the prior art, this application has at least the following beneficial effects:
[0130] 1. In the face of traffic "peaks", it can play a good buffering role in a short period of time to ensure the real-time and stability of the service; in the face of traffic "troughs", it can return idle resources to the kernel, effectively improving the utilization of system resources, solving the problem of resource fragmentation, and at the same time having a certain degree of adaptive ability.
[0131] 2. Perform noise filtering while predicting to ensure data accuracy.
[0132] 3. It avoids the problem of model inapplicability caused by differences in application versions and cluster heterogeneity. There is no need to conduct a large number of stress tests to establish performance and resource relationship curves.
[0133] 4. By predicting the peak resource consumption in the future as the resource limit value, resources are dynamically restricted, and the configuration idea of using the local maximum value instead of the global maximum value is adopted, which improves resource utilization and achieves an adaptive effect.
[0134] The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the present application. Various modifications to these embodiments will be apparent to professionals and technicians in the field, and the general principles defined in this application can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown in the present application, but will conform to the widest range consistent with the principles and novel features disclosed in the present application.
Claims
1. A method for dynamically adjusting Kubernetes cluster resources, characterized in that: The following steps are involved: Step a: Collect historical monitoring indicator data in the application, and obtain resource consumption time series data based on the monitoring indicator data; Step b: constructing an LSTM-Kalman filter, inputting the resource consumption time series data into the LSTM-Kalman for pre-training, and obtaining a prediction model; wherein the LSTM-Kalman filter is a Kalman filter modified by using LSTM; Step c: Collect new resource consumption time series data, input the new resource consumption time series data into the prediction model to predict the resource consumption peak value within a future preset time period, and use the resource consumption peak value as the resource limit value to update the resource; In the step b, the construction of the LSTM-Kalman filter includes: Assuming that the observed value is a noisy estimate of the system state value, let the observation matrix H = I: X(k)=f(X(k-1))+W(k),W~N(0,Q) Z(k)=X(k)+V(k), V~N(0,R) In the above formula, X(k) is the system state at time k, Z(k) is the observation value at time k, W(k) and V(k) represent the process and measurement noise respectively, and their covariances are Q and R respectively; f is the function model generated by the LSTMf model; The prediction steps are: X(k|k-1)=f(X(k-1|k-1)) P(k|k-1)=FP(k-1|k-1)FT+Q(k) In the above formula, F represents the Jacobian matrix of f with respect to X(k-1|k-1), and Q(k) is obtained by LSTM Q given; The update steps are: Kg(k)=P(k|k-1)((P(k|k-1)+R(k))-1 X(k|k)=X(k|k-1)+Kg(k)(Z(k)-X(k|k-1) P(k|k)=(I-Kg(k))P(k|k-1) In the above formula, R(k) is obtained by LSTMr, and Z(k) represents the observation value at time k; The request value in the configuration file is taken as X(k-1|k-1), and the indicator data collected at the current time k is taken as the observation value Z(k|k). The predicted value X(k|k-1) and the observation noise R(k) at the time k are obtained through the LSTMf and LSTMr networks respectively; X(k|k-1) passes through the LSTM Q The state noise Q(k) at time k is obtained; and the Kalman filter is recursively updated using R(k), Q(k), and X(k-1|k-1).
2. The method for dynamically adjusting Kubernetes cluster resources according to claim 1, characterized in that: In step a, the historical monitoring indicator data collected in the application includes: Monitor the containers running in the Pod and collect various monitoring indicator data; the collected monitoring indicator data includes at least one or a combination of CPU utilization and limit, file system read / write utilization and limit, and network message sending / receiving / discarding rate data; The collected monitoring indicator data is stored in the time series database.
3. The Kubernetes cluster resource dynamic adjustment method according to claim 2, characterized in that: In the step a, obtaining resource consumption time series data according to the monitoring indicator data includes: The time series database is queried regularly to obtain the indicator data of interest, and the indicator data of interest is cached and sorted to form resource consumption time series data that can be directly used for prediction.
4. The method for dynamically adjusting Kubernetes cluster resources according to claim 3, characterized in that: The caching and sorting of the indicator data of interest includes: Use the sliding window to slide each indicator data in turn, and select the maximum value in the window as the new data set.
5. The method for dynamically adjusting Kubernetes cluster resources according to any one of claims 1 to 4, characterized in that: In the step b, the step of inputting the resource consumption time series data into the LSTM-Kalman for pre-training to obtain a prediction model further comprises: Model evaluation: The prediction model is iteratively trained using new monitoring indicator data until the prediction model loss converges or stops iterative training when it only oscillates within a preset range.
6. The method for dynamically adjusting Kubernetes cluster resources according to claim 5, characterized in that: The model evaluation also includes: Obtain new monitoring indicator data within a set time period, compare the prediction result of the prediction model with the new monitoring indicator data, and determine whether model training is needed again: if the time consumption value greater than the set threshold is below the prediction result of the prediction model, the model will no longer be updated; otherwise, the new monitoring indicator data will be used to update the parameters of the prediction model and correct the prediction model.
7. The method for dynamically adjusting Kubernetes cluster resources according to claim 6, characterized in that: In step c, predicting the resource consumption peak value within a future preset time period also includes: Wait for the LSTMf, LSTMr, LSTM Q When the network losses of the three sub-cycles converge or decrease to a preset threshold, recursive estimation begins. The data is reported to the prediction model through the metric collector. The prediction model makes an optimal estimate of the resource consumption peak within a preset time period in the future, and uses the optimal estimate to update the container resources in the Pod.
8. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the Kubernetes cluster resource dynamic adjustment method described in any one of claims 1 to 7 above.
