Reserved resource configuration method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a reserved resource configuration method and device, equipment, a storage medium and a program product, and relates to the field of cloud computing. The method comprises the following steps: detecting a first state of a node server in a Kubernetes cluster, wherein the first state is a hardware resource which can be allocated to a container by the node server; determining a target action in actions corresponding to the first state by a pre-trained reinforcement learning model according to the first state, the actions including increasing or decreasing hardware resources reserved for a core process of an operating system of the node server and increasing or decreasing hardware resources reserved for a core service component of the Kubernetes system; and controlling the node server to execute the target action. According to the method provided by the invention, the hardware resources reserved by the node server for the core process of the operating system and the core service component of the Kubernetes system are dynamically adjusted according to the hardware resources which can be allocated to the container by the node server through the pre-trained reinforcement learning model.

Description

Technical Field

[0001] The present application relates to the field of cloud computing, and in particular to a reserved resource configuration method, device, equipment, storage medium and program product. Background Art

[0002] Container technology is a virtualization method in the field of cloud computing that allows developers to package and run applications and all their dependencies in an isolated environment. Unlike traditional virtual machines, containers do not require a complete operating system, but share the kernel of the host operating system, which makes them more lightweight and efficient. The core components of container technology include container orchestration tools, which are used to manage large-scale container deployments and provide automated container scheduling, load balancing, expansion, and management capabilities. Kubernetes (K8S) is the most popular container orchestration tool.

[0003] Building a node server cluster based on K8S is a common containerization solution in the industry. K8S allows users to specify the operating system of the node server to reserve certain hardware resources for the operating system process and the K8S system's own process to avoid system crashes caused by excessive resource usage by the container. Therefore, the usual practice is to specify the reserved resources for each node server when building the cluster and not change them later.

[0004] However, the fixed reserved resource configuration method in the prior art has the following defects: first, if the reserved resources are too few, the container will occupy too many hardware resources, which will cause the system to crash and reduce the system stability; second, if the reserved resources are too many, the resources allowed to be allocated to the container will be reduced, resulting in a decrease in the utilization rate of the server hardware. Summary of the invention

[0005] The present application provides a reserved resource configuration method, apparatus, device, storage medium and program product, which are used to solve the technical problems caused by the fixed reserved resource configuration method in the prior art that if the reserved resources are too few, the container will occupy too many hardware resources, which will cause system crash and reduce system stability; and if the reserved resources are too many, the resources allowed to be allocated to the container will be reduced, and the server hardware utilization rate will be reduced.

[0006] In a first aspect, the present application provides a reserved resource configuration method, comprising:

[0007] Detecting a first state of a node server in a Kubernetes cluster, where the first state is hardware resources that the node server can allocate to a container;

[0008] Determining, by the pre-trained reinforcement learning model, a target action among the actions corresponding to the first state according to the first state, the actions comprising: increasing or decreasing hardware resources reserved for the core process of the operating system of the node server, and increasing or decreasing hardware resources reserved for the core service components of the Kubernetes system;

[0009] Control the node server to execute the target action.

[0010] In a second aspect, the present application provides a reserved resource configuration device, including:

[0011] A detection module is used to: detect a first state of a node server in a Kubernetes cluster, where the first state is hardware resources that the node server can allocate to a container;

[0012] A decision module is used to: determine a target action among actions corresponding to the first state according to the first state by a pre-trained reinforcement learning model, wherein the action includes: increasing or decreasing hardware resources reserved for a core process of an operating system of the node server, and increasing or decreasing hardware resources reserved for a core service component of a Kubernetes system;

[0013] The control module is used to control the node server to execute the target action.

[0014] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0015] The memory stores computer-executable instructions;

[0016] The processor executes the computer-executable instructions stored in the memory to implement the method as described in the first aspect.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0018] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which implements the method described in the first aspect when executed by a processor.

[0019] The reserved resource configuration method, device, equipment, storage medium and program product provided in the present application have the following technical effects: the reinforcement learning model can be trained to determine the current optimal action according to the current state of the node server in the Kubernetes cluster, so that it can be realized that according to the hardware resources that the node server can allocate to the container, the hardware resources reserved by the node server for the core process of the operating system and the core service components of the Kubernetes system can be dynamically adjusted, so as to allocate as many hardware resources as possible to the containers in the node server while satisfying the normal operation of the operating system and the Kubernetes system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0021] Figure 1 A schematic diagram of a reserved resource configuration method provided in an embodiment of the present application;

[0022] Figure 2 A schematic diagram of hardware resources that can be allocated to a container by a computing node server provided in an embodiment of the present application;

[0023] Figure 3 Schematic diagram of the training process of the deep Q network model;

[0024] Figure 4 A schematic diagram of execution actions provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of a deep Q network model training method provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of the structure of a reserved resource configuration device provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of the structure of the electronic device provided in this application.

[0028] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0030] It should be noted that the reserved resource configuration method, device, equipment, storage medium and program product provided in the present application can be used in the field of cloud computing, and can also be used in any field other than cloud computing. The application field of the reserved resource configuration method, device, equipment, storage medium and program product in the present application is not limited.

[0031] Containers are a lightweight virtualization technology that allows developers to package an application and all its dependencies into a single unit. This unit can run in any environment that supports the container runtime without having to worry about differences in the underlying operating system. Container technology is widely used in the field of cloud computing.

[0032] Kubernetes (K8S) is an open source container orchestration platform for automating the deployment, scaling, and management of containerized applications. It has the following core components:

[0033] 1. Cluster: A Kubernetes cluster consists of a group of nodes (servers) that work together to run containerized applications and services. The cluster is the basic management unit of Kubernetes, responsible for coordinating and managing resources and workloads on all nodes.

[0034] 2. Node: A node is a single server in a cluster, which can be a physical machine or a virtual machine. Each node runs the key components of Kubernetes. There are two types of nodes: master nodes and worker nodes.

[0035] 3. Pod: The smallest deployment unit in Kubernetes. A Pod can contain one or more containers. Pods are scheduled and run on worker nodes, and Kubernetes is responsible for managing the life cycle of Pods.

[0036] In a Kubernetes cluster, the master node and the worker nodes communicate through the network, and the master node is responsible for scheduling and managing the resources and workloads on the worker nodes. Through this architecture, Kubernetes achieves high availability, scalability, and flexibility of applications.

[0037] Building a node server cluster based on K8S is a common containerization solution in the industry. K8S allows users to specify the operating system of the node server to reserve certain hardware resources for the operating system process and the K8S system's own process to avoid system crashes when the container occupies too many resources. Therefore, the usual practice is to specify the reserved resources for each node server when the cluster is built, and no longer change them later. However, the fixed reserved resource configuration method in the prior art has the following defects: First, if the reserved resources are too few, the container will occupy too many hardware resources, which will cause the system to crash, resulting in reduced system stability; second, if the reserved resources are too many, the resources allowed to be allocated to the container will be reduced, resulting in reduced server hardware utilization.

[0038] Reinforcement Learning (RL) is a machine learning paradigm that learns how to take actions to maximize accumulated rewards through the interaction between the agent and the environment. Reinforcement learning has the following core concepts:

[0039] 1. Agent: In reinforcement learning, the agent is the entity that performs actions. It learns strategies by interacting with the environment.

[0040] 2. Environment: The external world in which the agent exists, which reacts to the agent's actions and provides feedback.

[0041] 3. State: The specific situation of the environment at a certain moment, usually described by a set of variables.

[0042] 4. Actions: The operations that the agent can perform in each state.

[0043] 5. Reward: The feedback signal returned by the environment after the agent performs an action, usually a scalar value, used to guide the agent's learning.

[0044] 6. Policy: The rule or function by which an agent chooses an action in a given state, which can be deterministic or stochastic.

[0045] 7. Value function: used to evaluate the quality of a state or state-action pair.

[0046] 8. Discount factor: used to weigh the importance of current rewards and future rewards, with a value range of [0,1].

[0047] Reinforcement learning enables intelligent agents to learn optimal strategies in complex and dynamic environments through trial and error learning and reward mechanisms during training. Therefore, based on reinforcement learning, the technical concept of this application to solve the defects of the fixed reserved resource configuration method in the prior art is: detect the current state of the node server in the Kubernetes cluster, and the current state of the node server is the hardware resources that the node server can allocate to the container. The pre-trained reinforcement learning model determines the optimal action corresponding to the current state according to the current state of the node server. The optimal action includes increasing or decreasing the hardware resources reserved for the core process of the node server operating system and increasing or decreasing the hardware resources reserved for the Kubernetes core service components. Finally, the node server is controlled to execute the optimal action.

[0048] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0049] Figure 1 This is a schematic diagram of a reserved resource configuration method provided in an embodiment of the present application, which can be applied to a server. Figure 1 As shown, the method includes:

[0050] S101, detecting a first state of a node server in a Kubernetes cluster, where the first state is hardware resources that the node server can allocate to a container;

[0051] Specifically, the first state is the current state of the node server. When a new node joins the Kubernetes cluster, the node reports its available hardware resources, which may include CPU, memory, storage, and other allocatable resources. Kubelet periodically sends the node's capacity (total resources) and allocatable resources (minus the resources already allocated to the Pod) to the Kubernetes API Server to obtain the first state of the node server. In addition, you can also use resource monitoring tools to detect the resource usage of the node in real time.

[0052] S102, determining a target action from the actions corresponding to the first state according to the first state by the pre-trained reinforcement learning model, the actions comprising: increasing or decreasing hardware resources reserved for the core process of the operating system of the node server, and increasing or decreasing hardware resources reserved for the core service components of the Kubernetes system;

[0053] Specifically, it is first necessary to collect historical data of the node server status in multiple historical cycles. The node server status is the hardware resources that the node server can allocate to the container. The collected historical data of the first state is used to train the reinforcement learning model. The goal of the reinforcement learning model is to learn how to select the optimal action in different first states. The reinforcement learning model can be a Q-learning model, a deep Q network model, a policy gradient model, etc. The pre-trained reinforcement learning model is deployed in the actual environment. The pre-trained reinforcement learning model can determine the optimal target action in the action corresponding to the first state based on the current first state of the node server.

[0054] S103: Control the node server to execute the target action.

[0055] Specifically, the hardware resource allocation of the node server can be dynamically adjusted through an API or script to execute the target action.

[0056] The reserved resource configuration method provided in this embodiment has the following technical effects:

[0057] Through training, the reinforcement learning model can determine the current optimal action based on the current status of the node server in the Kubernetes cluster, so that it can dynamically adjust the hardware resources reserved by the node server for the core processes of the operating system and the core service components of the Kubernetes system based on the hardware resources that the node server can allocate to the container, so as to allocate as many hardware resources as possible to the containers in the node server while ensuring the normal operation of the operating system and the Kubernetes system.

[0058] Optionally, the first state may include memory resources that the node server can allocate to the container and computing resources that the node server's processor can allocate to the container. By detecting the memory resources that the node server can allocate to the container and the computing resources that the processor can allocate to the container, the optimal target action determined by the reinforcement learning model can meet the memory requirements and computing power requirements of the container operation. The first state of the node server, i.e., the memory resources that the node server can allocate to the container and the computing resources that the node server's processor can allocate to the container, can be detected by the following method:

[0059] Detect the memory space occupied by the core processes of the operating system and the memory space occupied by the core service components of the Kubernetes system. Calculate the memory resources that the node server can allocate to the container based on the total physical memory of the node server, the memory space occupied by the core processes of the operating system, the memory space occupied by the core service components of the Kubernetes system, and the memory threshold for the node server to evict pods. Among them, pod is the smallest deployable unit of the Kubernetes system.

[0060] Detect the processor usage of the operating system's core processes and the Kubernetes system's core service components. Based on the total processor capacity, the processor usage of the operating system's core processes, the processor usage of the Kubernetes system's core service components, and the processor usage threshold for the node server to evict pods, calculate the node server's processors that can be allocated to the container's computing resources.

[0061] Specifically, the core process components of the operating system refer to the key parts of the operating system that are responsible for providing basic functions and processes. These components are the core of the operating system, and they directly manage hardware resources and provide support for applications. Examples of core process components of the operating system include the kernel, device driver, memory manager, process manager, file system manager, etc. The core service components of the Kubernetes system are the basis of the Kubernetes cluster, and they work together to manage and coordinate the deployment, expansion, and operation of containerized applications. Examples of core service components of the Kubernetes system include the Kubernetes API Server, Kubelet, Kube-proxy, etc. The core service components of the Kubernetes system work together to provide the basic functions of Kubernetes, including container orchestration, automated deployment, expansion, maintenance, and fault recovery. Each component has specific roles and responsibilities in the cluster to ensure the efficient and stable operation of the entire system.

[0062] Figure 2 A schematic diagram of hardware resources that can be allocated to containers by a computing node server provided in an embodiment of the present application, such as Figure 2As shown in the figure, when calculating the memory resources that can be allocated to the container by the node server, "total node capacity" refers to the total physical memory of the node server, "system reserve" refers to the memory space occupied by the core process of the operating system, "K8S reserve" refers to the memory space occupied by the core service components of the Kubernetes system, "eviction threshold" refers to the memory threshold for the node server to evict pods, and "capacity that can be allocated to containers" refers to the memory resources that can be allocated to containers by the node server; when calculating the computing resources that can be allocated to containers by the processor of the node server, "total node capacity" refers to the total capacity of the processor, "system reserve" refers to the usage rate of the processor by the core process of the operating system, "K8S reserve" refers to the usage rate of the processor by the core service components of the Kubernetes system, "eviction threshold" refers to the usage rate threshold of the processor for the node server to evict pods, and "capacity that can be allocated to containers" refers to the computing resources that the processor can allocate to the container, which can be specifically the proportion of the idle time that can be allocated to the container by the processor within a certain CPU time to the total running time. Among them, the memory threshold for the node server to evict pods is the threshold set by the K8S system to avoid causing system crashes. If the threshold is exceeded, the node will start to evict pods, which is the highest tolerable memory usage. Similarly, the processor usage threshold for the node server to evict pods refers to the threshold set by the K8S system to avoid causing system crashes. If the threshold is exceeded, the node will start to evict pods, which is the highest tolerable CPU usage.

[0063] The memory resources that can actually be allocated to containers by the node server can be accurately calculated based on the total physical memory of the node server, the memory space occupied by the core processes of the operating system, the memory space occupied by the core service components of the Kubernetes system, and the memory threshold for the node server to evict pods. The computing resources that can actually be allocated to containers by the processor can be accurately calculated based on the total capacity of the processor, the usage rate of the core processes of the operating system, the usage rate of the core service components of the Kubernetes system, and the processor usage threshold for the node server to evict pods.

[0064] Therefore, optional, pre-trained reinforcement learning model selectable actions include:

[0065] Increase or decrease the memory resources reserved for the core processes of the operating system;

[0066] Increase or decrease the memory resources reserved for core service components of the Kubernetes system;

[0067] Increase or decrease the processor's computing resources reserved for the operating system's core processes;

[0068] Increase or decrease the computing resources of the processors reserved for the core service components of the Kubernetes system.

[0069] By dynamically adjusting the memory resources and processor computing resources reserved for the core processes of the operating system, as well as the memory resources and processor computing resources reserved for the core service components of the Kubernetes system, the memory resources and processor computing resources that the node server can allocate to the container can be adjusted.

[0070] Optionally, a pre-trained deep Q network model can be used in the embodiment of the present application to learn and make decisions in the state space of the K8S cluster scenario. The training steps of the deep Q network model are described in detail below.

[0071] First, define the nouns and corresponding symbols involved:

[0072] Operating system core process resource statistics time window: The statistical period of resources occupied by the operating system core process, denoted as T s1 、T s2 ……T sn .

[0073] K8S core service component resource statistics time window: The statistical period of resources occupied by K8S core service components, denoted as T k1 、T k2 ……T kn .

[0074] The total physical memory of the node server: M total .

[0075] Memory space occupied by the core process of the operating system: The memory space reserved by the K8S system for the core process of the node server operating system. This part of the space will not be allocated to the container and is recorded as M rs .

[0076] Memory space occupied by the core service components of the Kubernetes system: The memory space reserved by the K8S system for the K8S core service components. This part of the space will not be allocated to the container and is recorded as M rk .

[0077] Node server pod eviction memory threshold: The threshold set by the K8S system to avoid system crashes. If the threshold is exceeded, the node will start to evict pods. In this scenario, it is the highest tolerable memory usage, denoted as M evic .

[0078] Completely free memory: The physical memory space that is completely unused in the node server operating system, denoted as M free .

[0079] Kernel buffer memory: In the node server operating system, the block device memory cache area is released when the application in the system requires memory, recorded as M buffer .

[0080] Page buffer memory: In the node server operating system, the file system cache area is released when the application in the system requires memory, recorded as M cache .

[0081] Memory resources that can be allocated to containers by the node server: memory that can be allocated to containers by the K8S system, denoted as M a .

[0082] Number of CPU physical cores: The number of physical cores of the node server CPU, recorded as C.

[0083] CPU reserved for the core process of the operating system: The CPU resources reserved by the K8S system for the core process of the node server operating system will not be allocated to the container, denoted as C rs .

[0084] Kubernetes system core service component reserved CPU: K8S system reserves CPU resources for K8S core service components. These resources will not be allocated to containers and are recorded as C rk .

[0085] Processor usage threshold for node server to evict pods: This is the threshold set by the K8S system to avoid system crashes. If the threshold is exceeded, the node will begin to evict pods. In this scenario, this is the highest tolerable CPU usage, denoted as C evic .

[0086] CPU idle time ratio: The ratio of idle time to total time within a certain CPU time, denoted as R free .

[0087] Specifically, Figure 3 This is a schematic diagram of the training process of the deep Q network model, such as Figure 3 As shown, the method includes:

[0088] S301, collecting historical data of node server status in multiple historical periods;

[0089] Specifically, in a time period T s1 The average available memory size during the collection period is M avs1 =M frees1 +M buffers1 +M caches1 Collect the CPU idle time ratio R in this cycle frees1 After installing the K8S system, in the time period T k1Collect the average available memory size M during this period avk1 =M freek1 +M bufferk1 +M cachek1 Collect the proportion of idle time in the CPU time to the total time R in this cycle freek1 .

[0090] If this collection cycle is the first collection cycle, the memory space occupied by the core process of the operating system is M rs1 =M total -M avs1 , the core process of the operating system reserves CPU for C rs1 =(1-R frees1 )*C. The memory space occupied by the core service components of the Kubernetes system is M rk1 =M avs1 -M avk1 Kubernetes system core service components reserve CPU for C rk1 =(1-R freek1 )*C-(1-R frees1 )*C is C rk1 =(R frees1 -R freek1 )*C. Collect the above data for n cycles, through Figure 2 The method shown forms the state space of the node server state <M a ,R freeR >, R freeR It is the ratio of the idle time of the node server that can be allocated to the container to the total running time of the node server processor, that is, the computing resources that the node server can allocate to the container.

[0091] S302: Use historical data of node server status in multiple historical periods to train the deep Q network model.

[0092] Specifically, the executable actions include: increasing M rs , increase M rk , increase C rs , increase C rk , reduce M rs , reduce M rk , reduce C rs , reduce C rk In the embodiment of the present application, a quantified execution action is applied - increasing and decreasing reserved resources in proportion, that is, executing an increase Mrs action once represents increasing the memory resources reserved by K8S for the core process of the operating system in a certain proportion, and so on. Figure 4 The following is a schematic diagram of the execution actions provided in the embodiment of the present application, such as Figure 4As shown, state a is the current state. If there is no need to change the resource reservation scheme, for example, when the current resource utilization and system performance requirements are within an acceptable range, the system will continue to remain in state a. If additional reserved resources need to be added to handle higher demands, it can execute increase Mrs and transition from state a to state b; if the actual system resource utilization is low and it is desired to reduce the reserved resources, it can execute decrease Mrs and transition from state a to state c.

[0093] The goal of the deep Q network model is to monitor the resource reservation and actual resource usage of the K8S cluster node server in the historical period, analyze and predict the optimal resource reservation plan for the system in the next time period through the model, and achieve the optimal resource reservation plan under the premise of meeting performance requirements (memory and CPU usage do not exceed the threshold M evic and C evic , the reference initial value is set to 15%), the operating system core process and K8S core service component reserve resources (M rs +M rk and C rs +C rk ) is minimized, and the resources that can be allocated to the container (M a and R free )maximize.

[0094] The deep Q-network model employs delayed reward computation in order to properly capture the impact of changes in the system.

[0095] NewQ(s,a)=Q(s,a)+α[R(s,a)+γmax Q'(s',a')-Q(s,a)](1)

[0096] The deep Q network model algorithm solves the optimal strategy based on the Bellman equation, where R(s,a) is the reward function, maxQ'(s',a') refers to the Q obtained by the action that maximizes the reward for the next state, and γmaxQ'(s',a') refers to the future long-term reward. α is the learning rate and γ is the decay value. In this scenario, the reward calculation of the reward function includes three factors: the proportion of memory that can be allocated to the container (M a / M total ) and CPU idle time ratio (R free ), and the Pod operation status (i.e., whether the Pod is evicted). The percentage of memory that can be allocated to the container and the percentage of CPU idle time are both system monitoring data. The larger the value, the higher the calculated reward value. The Pod operation status is set to 1 if the Pod is not evicted, otherwise -2.

[0097] In the initial collection of n cycles, the K8S cluster is in the current state (s t ) and randomly execute actions (at ) adjusts resource configuration and then returns to the next state (s t+1 ) and rewards (r t ), then (s t ,a t ,r t ,s t+1 ) tuples are stored in the memory pool. After collecting n cycles, the memory pool is full. Figure 5 A schematic diagram of a deep Q network model training method provided in an embodiment of the present application, such as Figure 5 As shown, training is done through the following steps:

[0098] Step 1: Initialize parameters α, γ (reference values ​​are 0.1 and 0.9 respectively), ε, the number of samples m selected for each iteration, the number of iterations T, the initial current Q and the target Q;

[0099] Step 2: In each iteration (for i from 1 to T):

[0100] 2.1 Randomly select an initial state S;

[0101] 2.2 Among all possible actions in the current state S, use the ε greedy method to select an action a in the current Q value;

[0102] 2.3 After executing action a, the new state S' is obtained and the corresponding r is calculated;

[0103] 2.4 The newly obtained (s t ,a t ,r t ,s t+1 ) into the playback memory pool. If the same state data (s t ,a old ,r old ,s old ), replace the data, otherwise replace a random data;

[0104] 2.5 Take m samples from the playback pool and calculate the current Q value (y) according to the above formula 1;

[0105] 2.6 Using the mean square error function Calculate the difference between the current network Q value and the target network Q value, and calculate the gradient of the error function relative to the deep Q network model parameters through the back propagation algorithm. The deep Q network model parameters can be the neural network weights and bias parameters;

[0106] 2.7 After a certain number of iterations, Q 当前 The updated deep Q network model parameters are copied to Q 目标 ;

[0107] 2.8 If convergence occurs, stop; otherwise, jump to 2.2.

[0108] In each computing cycle, the deep Q network model predicts the optimal resource reservation adjustment plan for the current state. After the computing cycle ends, all resource reservation data in the memory pool that caused the Pod to be evicted are counted, and the smallest CPU usage C is taken. min and memory usage M min , adjust the threshold M evic and C evic 1-C min and 1-M min . Further help analyze and dynamically adjust resource reservation adjustment plans.

[0109] After the above training steps, the trained deep Q network model is deployed to the actual environment. The pre-trained deep Q network model determines the first state-action pair with the highest value as the first target state-action pair in the first state-action pair consisting of the first state and the action corresponding to the first state, and determines the action in the first target state-action pair as the target action. Then, the node server of the Kubernetes cluster is controlled to execute the target action. It should be noted that in the Kubernetes (K8S) cluster scenario, the state space is continuous and infinite, which makes it difficult to effectively apply the traditional Q-Learning method, because Q-Learning relies on Q-table to store the value of each state-action pair, which is not feasible in high-dimensional and continuous state spaces. In this case, the use of a deep Q network (DQN) has the following specific advantages:

[0110] 1. Handling high-dimensional and continuous state spaces: DQN uses a deep neural network to approximate the Q-value function, which enables it to handle high-dimensional and continuous state spaces without explicitly storing the Q-value for each state-action pair.

[0111] 2. Automatic feature extraction: Through deep learning, DQN can automatically extract features from complex input data, reducing the reliance on manual feature engineering. This is especially important when processing complex state information in a K8S cluster.

[0112] 3. Experience replay: DQN uses the experience replay mechanism to break data correlation and improve sample efficiency by randomly sampling from past experience. This is beneficial for stable training in dynamic and complex K8S environments.

[0113] 4. Target network stability: DQN introduces a target network to stabilize the training process and reduce oscillations during the Q value update process. This helps improve the stability and reliability of the model in the rapidly changing environment of the K8S cluster.

[0114] 5. Scalability and flexibility: DQN can be combined with other reinforcement learning techniques, such as prioritized experience replay, dual DQN, etc., to further improve performance. This flexibility allows DQN to adapt to different optimization objectives and constraints.

[0115] 6. Model-free learning: DQN is a model-free method that does not require explicit modeling of the environment, which is an important advantage in the complex and difficult to accurately model K8S cluster environment.

[0116] 7. Adapt to dynamic environments: The load and resource requirements in the K8S cluster may change dynamically. DQN is able to adapt to these changes through continuous learning and updating strategies, thereby optimizing resource allocation and task scheduling.

[0117] Based on the pre-trained deep Q network model, after controlling the node server to execute the target action, the second state of the node server can be detected, and the second state is the hardware resources that the node server can allocate to the container. It should be noted that the aforementioned method for determining the first state can be applied to determining the second state.

[0118] After obtaining the second state of the node server, the deep Q network model can update the value corresponding to the first target state-action pair according to the second state. It should be noted that the deep Q network model can update the value corresponding to the first target state-action pair according to the second state, which is the aforementioned delayed reward mechanism. The delayed reward mechanism allows the deep Q network model to consider the long-term impact of an action, not just the immediate result. In the K8S cluster, an action may not show its full effect immediately. Through delayed rewards, the deep Q network model can better evaluate the long-term impact of these actions on the overall performance of the system. By considering the impact of subsequent states (such as the second state) on the current action, DQN can update the Q value more accurately. This mechanism helps to improve the strategy so that the deep Q network model can choose better actions in future decisions, thereby improving the overall efficiency and performance of the system to optimize the hardware resource allocation of the node server. In addition, the delayed reward mechanism reduces the sharp fluctuations in the Q value caused by a single action by gradually updating the Q value. This smooth update process helps to improve the stability of learning and make the deep Q network model more robust in the dynamic environment of actual production.

[0119] Optionally, updating the value corresponding to the first target state-action pair according to the second state includes:

[0120] According to the reward function, determine the reward value of the target action;

[0121] The pre-trained deep Q network model determines, according to the second state, a second state-action pair having the highest corresponding value among second state-action pairs consisting of the second state and the action corresponding to the second state as a second target state-action pair;

[0122] According to the reward value and the value corresponding to the second target state-action pair, the value corresponding to the first target state-action pair is updated through the Bellman equation.

[0123] Specifically, the Bellman equation can refer to the aforementioned training steps. It should be noted that by designing a suitable reward function, the system can be guided to learn in the direction of a specific goal or task. For example, in the embodiment of the present application, the more hardware resources that can be allocated to the container, the greater the reward value, and rewards will also be provided when no POD eviction occurs, thereby meeting the goal of the deep Q network model: allocating as many hardware resources as possible to the container while reducing the number of POD evictions.

[0124] The Bellman equation converts future rewards to the present through a discount factor, so that the system not only focuses on immediate rewards but also considers long-term returns, thereby achieving optimal decision-making.

[0125] Optionally, the reward value of the target action is determined according to the reward function, including:

[0126] After the control node server executes the target action, the reward function calculates the reward value based on the hardware resources that the node server can allocate to the container and whether the node server has a pod expulsion. The pod is the smallest deployable unit of the Kubernetes system, where: the more hardware resources the node server can allocate to the container, the higher the reward value. For example, if the pod is not expelled, a positive reward value is given; if the pod is expelled, a negative reward value is given. For example, if the pod is not expelled, a reward value of 1 is given; if the pod is expelled, a reward value of -2 is given. The reward function calculates two conditions, namely the hardware resources that the node server can allocate to the container and whether the pod is expelled. Thus, the reward function calculates the corresponding reward value to change the Q value of the first state-action pair, so that the reinforcement learning model can choose to allocate as many hardware resources as possible to the container while avoiding the action of pod expulsion.

[0127] Figure 6 A schematic diagram of the structure of the reserved resource configuration device provided in the embodiment of the present application, such as Figure 6 As shown, the device 60 includes:

[0128] The detection module 601 is used to: detect a first state of a node server in a Kubernetes cluster, where the first state is hardware resources that the node server can allocate to a container;

[0129] The decision module 602 is used to: determine a target action from the actions corresponding to the first state according to the first state by a pre-trained reinforcement learning model, where the actions include: increasing or decreasing hardware resources reserved for the core process of the operating system of the node server, and increasing or decreasing hardware resources reserved for the core service components of the Kubernetes system;

[0130] The control module 603 is used to control the node server to execute the target action.

[0131] The reserved resource configuration device provided in the embodiment of the present application can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0132] Figure 7 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 7 As shown, the electronic device 70 provided in this embodiment includes: at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, the memory 702 and the communication component 703 are connected via a bus 704.

[0133] In a specific implementation process, at least one processor 701 executes the computer-executable instructions stored in the memory 702, so that at least one processor 701 executes the above method.

[0134] The specific implementation process of the processor 701 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0135] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0136] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0137] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0138] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0139] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0140] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0141] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0142] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0143] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0145] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0146] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0147] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A reserved resource configuration method, characterized in that: The method comprises: Detecting a first state of a node server in a Kubernetes cluster, where the first state is hardware resources that the node server can allocate to a container; Determining, by the pre-trained reinforcement learning model, a target action among the actions corresponding to the first state according to the first state, the actions comprising: increasing or decreasing hardware resources reserved for the core process of the operating system of the node server, and increasing or decreasing hardware resources reserved for the core service components of the Kubernetes system; Control the node server to execute the target action.

2. The method according to claim 1, characterized in that: The detecting a first state of a node server in a Kubernetes cluster includes: Detect memory resources that the node server can allocate to the container and computing resources that the processor of the node server can allocate to the container, wherein the first state includes memory resources that the node server can allocate to the container and computing resources that the processor of the node server can allocate to the container.

3. The method according to claim 2, characterized in that The detecting that the memory resources of the node server can be allocated to the container and the computing resources of the processor of the node server can be allocated to the container includes: Detect the memory space occupied by the core process of the operating system and the memory space occupied by the core service components of the Kubernetes system, and calculate the memory resources that can be allocated to the container by the node server according to the total physical memory of the node server, the memory space occupied by the core process of the operating system, the memory space occupied by the core service components of the Kubernetes system, and the memory threshold for the node server to evict the pod, wherein the pod is the smallest deployable unit of the Kubernetes system; Detect the usage rate of the processor by the core process of the operating system and the usage rate of the processor by the core service component of the Kubernetes system, and calculate the computing resources of the processor of the node server that can be allocated to the container based on the total capacity of the processor, the usage rate of the processor by the core process of the operating system, the usage rate of the processor by the core service component of the Kubernetes system, and the usage threshold of the processor at which the node server expels the pod.

4. The method according to any one of claims 1 to 3, characterized in that The actions include: increasing or decreasing memory resources reserved for core processes of the operating system; Increase or decrease the memory resources reserved for the core service components of the Kubernetes system; increasing or decreasing computing resources of the processor reserved for a core process of the operating system; Increase or decrease the computing resources of the processor reserved for the core service components of the Kubernetes system.

5. The method according to claim 1, characterized in that The pre-trained reinforcement learning model determines, according to the first state, an optimal action corresponding to the first state, including: The pre-trained deep Q network model determines, according to the first state, among first state-action pairs consisting of the first state and the action corresponding to the first state, the first state-action pair with the highest corresponding value as a first target state-action pair; The action in the first target state-action pair is determined as the target action.

6. The method according to claim 5, characterized in that After controlling the node server to perform the target action, the method further includes: detecting a second state of the node server, where the second state is hardware resources that the node server can allocate to a container; According to the second state, the value corresponding to the first target state-action pair is updated.

7. The system according to claim 6, characterized in that The updating of the value corresponding to the first target state-action pair according to the second state includes: Determining a reward value for the target action according to a reward function; The pre-trained deep Q network model determines, according to the second state, among second state-action pairs consisting of the second state and the action corresponding to the second state, the second state-action pair with the highest corresponding value as a second target state-action pair; According to the reward value and the value corresponding to the second target state-action pair, the value corresponding to the first target state-action pair is updated through the Bellman equation.

8. The system according to claim 7, characterized in that Determining the reward value of the target action according to the reward function includes: After controlling the node server to perform the target action, the reward function calculates the reward value according to the hardware resources that can be allocated to the container by the node server and whether the node server has expelled a pod, where the pod is the smallest deployable unit of the Kubernetes system, wherein: the more hardware resources that can be allocated to the container by the node server, the higher the reward value; if the pod is not expelled, a positive reward value is given; if the pod is expelled, a negative reward value is given.

9. A reserved resource configuration device, comprising: A detection module is used to: detect a first state of a node server in a Kubernetes cluster, where the first state is hardware resources that the node server can allocate to a container; A decision module is used to: determine a target action among actions corresponding to the first state according to the first state by a pre-trained reinforcement learning model, wherein the action includes: increasing or decreasing hardware resources reserved for a core process of an operating system of the node server, and increasing or decreasing hardware resources reserved for a core service component of a Kubernetes system; The control module is used to control the node server to execute the target action.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method as claimed in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

12. A computer program product, characterized in that The invention comprises a computer program, which implements the method described in any one of claims 1 to 8 when being executed by a processor.

Images