Target node determination method, container deployment method, container deployment decision model updating method and electronic equipment

Abstract

The invention provides a target node determination method, a container deployment method, a container deployment decision model updating method and electronic equipment. According to the method, the container deployment decision is performed by introducing the deep reinforcement learning agent and based on the Markov decision process framework, the load state change of the cluster can be dynamically adapted, and the scheduling problem of multi-dimensional resource requirements is fully considered, so that the resource allocation and load balance of the cluster are optimized. Specifically, by minimizing the load balancing deviation of the node cluster, the intelligent agent can realize global load balancing of resources in the container deployment process instead of single-dimensional local optimization, so that the phenomenon of resource waste or node overload possibly caused by a traditional method is effectively avoided. Finally, the method can improve the utilization rate of cluster resources, improve the efficiency and stability of container deployment, reduce the risk of system downtime, and optimize the long-term effect of container scheduling decision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of container deployment, and in particular to a target node determination method, a container deployment method, a container deployment decision model updating method and an electronic device. BACKGROUND

[0002] Container technology is an emerging virtualization solution, which can package an application and its required resources (such as operating system, library, configuration file, etc.) together, so as to run on different operating systems without the need to reconfigure the environment, which can greatly improve the efficiency and availability of virtualization, faster deployment of applications, improve the portability of applications, and reduce resource waste. In addition, compared with virtual machines, containers have faster startup speed. Container technology provides a simple method to deploy and manage containerized applications in a cloud environment, allowing users to break free from the limitations of traditional software deployment methods and better utilize the advantages of cloud computing platforms. Therefore, container technology has become a lightweight solution for software development of cloud computing applications and services, and is widely used in cloud environments.

[0003] The existing container deployment method cannot adaptively achieve global load balancing optimization of multi-dimensional resources in a dynamic load environment. SUMMARY

[0004] The present application aims to overcome the above technical deficiencies and provide a target node determination method, a container deployment method, a container deployment decision model updating method and an electronic device to solve the technical problem that the existing container deployment method cannot adaptively achieve global load balancing optimization of multi-dimensional resources in a dynamic load environment in related technologies.

[0005] To achieve the above technical purposes, the present application adopts the following technical solutions: In a first aspect, the present application provides a target node determination method, comprising: In response to receiving a user request, obtaining state data; wherein the user request represents a request indicating to perform deployment on at least one to-be-deployed container, and the user request carries resource requirement data of the to-be-deployed container; the state data is data used to represent the load state of a node cluster; wherein the resource requirement indicates at least two resource requirements, and the load state indicates the load state of at least two resources; deploying the at least one container to be deployed to the target node, wherein the container deployment decision model is a deep reinforcement learning agent configured to perform container deployment decisions based on a Markov decision process framework and with a load balancing bias of the node cluster as an optimization objective, and wherein the load balancing bias is a load balancing bias of at least two resources.

[0006] Further, the step of obtaining state data in response to receiving the user request comprises: sending a request for obtaining state data to a preset data collection module; receiving state data sent by the data collection module, wherein the state data at least includes utilization rate data of a target resource of any single node in the node cluster, mean data of utilization rate of the target resource in the node cluster, or standard deviation data of utilization rate of the target resource in the node cluster, and wherein the target resource at least includes CPU resource and memory resource.

[0007] Further, the container deployment decision model is a deep reinforcement learning agent based on a Dueling DQN algorithm, and the Markov decision process framework at least includes a state space, an action space, a reward function, and a state transition function, wherein the state space is constructed based on the resource requirement data and the state data, the action space is used to represent a set of node numbers in the node cluster, the action output is a target node number in the node cluster, the reward function is modeled based on the optimization objective and used to calculate a reward value according to a deployment result to guide training, and the state transition function is used to update a load state of the node cluster to generate a next state in response to deploying the container to be deployed to the target node. The step of deploying the at least one container to be deployed to the target node using the preset container deployment decision model based on the resource requirement data and the state data comprises: constructing a state space based on the resource requirement data and the state data; inputting the state space into the deep reinforcement learning agent based on the Dueling DQN algorithm to obtain a target node number output by the action space.

[0008] Further, the optimization objective of the deep reinforcement learning agent based on the Dueling DQN algorithm is modeled to include an optimization objective function, a first constraint, a second constraint and a third constraint; wherein the optimization objective function is represented as minimizing a load balancing deviation metric by optimizing the deployment decision variables, the load balancing deviation metric being a total deviation of the load states of the node cluster from the average load state of the node cluster; The first constraint is used to represent the constraint that the CPU resources on each node will not be overused. The second constraint is used to represent the constraint that the memory resources on each node will not be overused. The third constraint is used to represent the constraint that each to-be-deployed container can only be deployed on one node.

[0009] Further, the model structure of the deep reinforcement learning agent based on the Dueling DQN algorithm includes: An input layer for receiving data in the state space; A shared feature extraction layer including at least one fully connected hidden layer, the activation function of the fully connected hidden layer being a ReLU function, for extracting shared features of the node cluster load state; A branch processing layer including a state value function branch and an action advantage function branch; wherein the state value function branch is connected to the shared feature extraction layer and outputs a state value function representing the current overall cluster state value, and the action advantage function branch is connected to the shared feature extraction layer and outputs an advantage value function representing the action advantage value of each node; An output merging layer for receiving the state value function and the action advantage value, and outputting the Q value of each node based on the state value function and the action advantage value, and then normalizing the Q value through a softmax function to generate a scheduling action probability distribution of all nodes; A decision layer for selecting a target node number based on the scheduling action probability distribution, wherein the node number corresponding to the highest probability is the output action.

[0010] In a second aspect, the present application provides a container deployment method, comprising: Determining the target node corresponding to the to-be-deployed container; wherein the target node corresponding to the to-be-deployed container is determined by using the target node determination method described above; Deploying the to-be-deployed container to the target node.

[0011] In a third aspect, the present application provides an update method of a container deployment decision model, comprising: deploy the container to be deployed to a target node; wherein a target node determination method is used to determine the target node corresponding to the container to be deployed; obtain a deployment result; in response to the deployment result, calculate a reward value through a reward function; update the network parameters of the deep reinforcement learning agent based on the reward value.

[0012] Further, the step of calculating a reward value through a reward function in response to the deployment result comprises: if the container deployment fails, the reward value is a first preset value; wherein the first preset value is a negative value; if the deployment is successful and the cluster load balancing deviation after deployment is less than that before deployment, the reward value is a second preset value; wherein the second preset value is a positive value greater than 0 and less than 1; if the deployment is successful and the cluster load balancing deviation after deployment is greater than or equal to that before deployment, the reward value is the load balancing deviation before deployment minus the load balancing deviation after deployment; wherein the load balancing deviation is calculated by the sum of the Euclidean distances of node resource utilization and cluster average utilization; wherein the load balancing deviation is constrained in the numerical interval of 0 to 1.

[0013] Further, the step of updating the network parameters of the deep reinforcement learning agent based on the reward value comprises: select a deployment node through a directional exploration strategy in the training phase; wherein the step of selecting a deployment node through a directional exploration strategy in the training phase comprises: in response to entering an action exploration phase, obtaining the CPU available utilization and the memory available utilization of each node in the node cluster; based on the CPU available utilization and the memory available utilization, calculating the comprehensive resource available utilization of each node; based on a preset directional greed rule, selecting the node with the maximum comprehensive resource available utilization as the deployment action.

[0014] In a fourth aspect, the present application provides an electronic device comprising: a memory, and one or more processors in communication connection with the memory; The memory stores instructions executable by the one or more processors, and the one or more processors execute the instructions to make the one or more processors implement a target node determination method, or implement a container deployment method, or implement a container deployment decision model updating method.

[0015] Advantages: The application can dynamically adapt to the load state change of the cluster and fully consider the scheduling problem of multi-dimensional resource demand (for example, CPU, memory, etc.) by introducing a deep reinforcement learning agent and making container deployment decisions based on a Markov decision process framework, thereby optimizing resource allocation and load balancing of the cluster. Specifically, by minimizing the load balancing deviation of the node cluster, the agent can achieve global load balancing of resources in the container deployment process, rather than single-dimensional local optimization, which effectively avoids the resource waste or node overload phenomenon that may be caused by traditional methods. Ultimately, the method can improve the utilization of cluster resources, improve the efficiency and stability of container deployment, reduce the risk of system downtime, and optimize the long-term effect of container scheduling decisions. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a flowchart of a target node determination method provided by an embodiment of the application; Figure 2 is a timing diagram of a container deployment process in the application scenario of Kubernetes provided by an embodiment of the application; Figure 3 is a container deployment process diagram of a deep reinforcement learning agent based on a Dueling DQN algorithm provided by an embodiment of the application; Figure 4 is a network structure diagram of a deep reinforcement learning agent based on a Dueling DQN algorithm provided by an embodiment of the application; Figure 5 is a Markov decision process diagram provided by an embodiment of the application; Figure 6 is one of the experimental verification result diagrams provided by an embodiment of the application, which is a DCLI index comparison diagram; Figure 7 is another experimental verification result diagram provided by an embodiment of the application, which is a DI index comparison diagram. DETAILED DESCRIPTION

[0017] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0018] In the related art, in a cloud computing architecture, container orchestration has become one of the core infrastructures supporting cloud native computing. Kubernetes, as a container orchestration platform, plays a crucial role in the automatic management, resource scheduling, expansion and recovery of large-scale containerized applications.

[0019] In the Kubernetes cluster architecture, there is a hardware layer composed of multiple heterogeneous physical nodes, each node having different resource configurations such as CPU, memory, storage, etc. The resource differences between these nodes (e.g. CPU, memory size, disk I / O capability, etc.) are often very significant, resulting in the need to consider how to efficiently utilize these heterogeneous nodes when scheduling containers, avoiding overloading or resource idling of certain nodes.

[0020] In the Kubernetes cluster architecture, there is an orchestration layer. In the orchestration layer, the Kubernetes scheduler is responsible for deciding which node is most suitable to host a container instance based on the user's container deployment request (e.g. CPU, memory requirements, etc.) and the current resource state of the cluster. The scheduler relies on the current load information of the cluster to select the target node when making scheduling decisions.

[0021] It can be understood that the life cycle of a container is often short and its resource requirements fluctuate greatly, making container scheduling a dynamic and complex problem. In particular, in scenarios such as AI training tasks, the resource requirements of a container can suddenly increase, causing rapid consumption of node resources (e.g. sudden increase in memory requirements), which in turn affects the load balancing of the entire cluster.

[0022] In this highly dynamic load environment, existing container deployment methods are difficult to effectively respond to sudden changes in resource requirements, especially in the case of constantly changing cluster resource states. How to intelligently schedule containers and make decisions based on multi-dimensional resource load balancing optimization strategies has become a problem that needs to be solved.

[0023] In one possible implementation, scheduling can be performed based on static rules. Specifically, static rule-based scheduling methods, such as the default algorithm of Kubernetes, mainly schedule cluster resources based on static rules. These rules are often fixed and single, such as deciding the deployment node of a container based on the free CPU number of a node. Such algorithms often only focus on the load of a single dimension such as CPU or memory, ignoring the multi-dimensional characteristics of container resource requirements. For example, when scheduling memory-intensive containers, the algorithm may prefer to select a node with free CPU, but due to insufficient memory on the node, the node becomes heavily loaded, and even system crashes may occur. This resource skew phenomenon poses a serious threat to the stability of the cluster, causing a chain reaction.

[0024] In a possible implementation, a heuristic search algorithm can perform the scheduling. Specifically, a heuristic search algorithm, for example, a genetic algorithm, can try to find the optimal solution in a larger search space through selection, crossover, mutation, and the like. Its advantage is that it can explore the problem space through various strategies, but it also has some defects that cannot be ignored. The heuristic algorithm often needs to set parameters such as crossover rate and mutation rate in advance. However, these parameters cannot be dynamically adjusted with the cluster load. When the cluster load changes suddenly, the response speed of the heuristic algorithm often lags behind, and it cannot make effective decisions under real-time load changes. Moreover, the optimization goal of such algorithms is limited to the current deployment, ignoring future load changes and long-term balance of the cluster. This short-sighted optimization method can lead to inefficient use of cluster resources, or even over-allocation or over-waste of resources.

[0025] Therefore, through the above analysis, it can be seen that the existing scheduling method often optimizes resources such as CPU and memory independently, and lacks a global load balancing perspective. This single-dimensional optimization method leads to serious resource skew, with some nodes overloaded and others idle, causing inefficient operation of the cluster. The parameters of the static rule type scheduling method lag behind the changes in the cluster load, and especially in the case of load bursts, it cannot make real-time adaptive adjustments. The existing method does not fully consider the resource differences of the heterogeneous cluster, so the homogeneous processing method cannot effectively manage different types of nodes. For example, CPU-intensive tasks are incorrectly scheduled to memory-intensive nodes, causing node overload or downtime.

[0026] In summary, the existing container deployment method cannot adaptively achieve global load balancing optimization of multi-dimensional resources in a dynamic load environment. The present embodiment provides a method for determining a target node, and the execution subject of the determining method can be a container orchestration scheduler (for example, Kubernetes Scheduler). The scheduler can listen to Pod creation requests (user requests) through the Watch API, and obtain node resource states (load state data) through the Metrics Server. The scheduler can have a built-in decision engine (container deployment decision model) to output a Binding object to bind the container to the target node (Bind stage of kube-scheduler). Specifically, the scheduler can be deployed on the management node or control plane node of the cluster, responsible for the scheduling task of the entire cluster. The execution subject of the determining method can also be an intelligent scheduling agent (reinforcement learning service), that is, a separately deployed reinforcement learning model service. Specifically, the intelligent scheduling agent can be deployed on a dedicated training node or a hybrid architecture node.

[0027] As Figure 1 , Figure 2 andFigure 3 As shown, the embodiment provides a method for determining a target node, which can include: Step S12: In response to receiving a user request, obtaining state data; wherein the user request is a request for instructing to perform deployment on at least one to-be-deployed container, and the user request carries resource requirement data of the to-be-deployed container; the state data is data for characterizing a load state of a node cluster; wherein the resource requirement indicates at least two resource requirements, and the load state indicates a load state of at least two resources.

[0028] In the embodiment, the user request carries resource requirement data of the to-be-deployed container, which can be data of a CPU resource requirement of the to-be-deployed container, data of a memory resource requirement of the to-be-deployed container, data of a disk I / O capability resource requirement of the to-be-deployed container, data of a network bandwidth resource requirement of the to-be-deployed container, data of a GPU resource requirement of the to-be-deployed container, or data of a container security requirement of the to-be-deployed container.

[0029] In the embodiment, the state data can include: utilization rate data of a target resource of any single node in the node cluster; mean value data of the utilization rate of the target resource in the node cluster; and standard deviation data of the utilization rate of the target resource in the node cluster.

[0030] In the embodiment, the state data can also include: maximum / minimum utilization rate data of the target resource. Specifically, the maximum utilization rate of the target resource in the node cluster reflects the most busy node condition of the resource in the cluster. The minimum utilization rate of the target resource in the node cluster reflects the most idle node condition of the resource in the cluster.

[0031] In the embodiment, the state data can also include: a change rate of the resource utilization rate. Specifically, the change rate or the amplitude of increase / decrease of the resource utilization rate within a certain time interval. This can reflect a sudden change of the cluster load, such as a sudden rise or fall of the load, helping the scheduling system to quickly adapt to the load fluctuation.

[0032] In the embodiment, the resource can include a CPU resource, a memory resource, a disk I / O capability resource, a network bandwidth resource, a GPU resource, and a container security requirement, etc.

[0033] Step S14: using a preset container deployment decision model, performing a decision on container deployment for the resource requirement data and the state data to obtain a target node; wherein the target node is a node for deploying the at least one to-be-deployed container, the container deployment decision model is a deep reinforcement learning agent, the deep reinforcement learning agent is configured to perform a container deployment decision based on a Markov decision process framework, and has an optimization objective of minimizing a load balance deviation of a node cluster; wherein the load balance deviation is a load balance deviation of at least two resources.

[0034] In the embodiment, the deep reinforcement learning agent can be a deep reinforcement learning agent based on a Dueling DQN algorithm. In the embodiment, the deep reinforcement learning agent can be a deep reinforcement learning agent based on a DQN (Deep Q-Network) algorithm. In the embodiment, the deep reinforcement learning agent can be a deep reinforcement learning agent based on a Double DQN. In the embodiment, the deep reinforcement learning agent can be a deep reinforcement learning agent based on a DDPG (Deep Deterministic Policy Gradient) algorithm. In the embodiment, the deep reinforcement learning agent can be a deep reinforcement learning agent based on a Rainbow DQN algorithm.

[0035] In the embodiment, the optimization objective of the preset container deployment decision model can be to minimize the load balance deviation of the node cluster, and the load balance deviation is a load balance deviation of at least two resources. The load balance deviation can be modeled as a global imbalance metric in multiple resource dimensions. It can be understood that the load balance deviation metric is a key indicator in the optimization process. The load balance deviation quantifies the deviation between the load state of the node cluster and the average load state of the node cluster. The goal is to optimize the deployment decision variable so that the resource load distribution of the entire cluster is more balanced, avoiding overloading of some nodes or resource idling.

[0036] The load balancing bias is calculated by comparing the load of each node in the cluster with the average load of the cluster, and the overall deviation of the load state. The load state can refer to the use of CPU, memory or other resources (disk I / O capability resources, network bandwidth resources, GPU resources, and container security requirements, etc.). For example, assuming that there is a node cluster, the resource usage of each node in the cluster can be represented by a vector (for example, the usage of CPU and memory). The load balancing bias is the difference between the load state of these nodes and the average load state of the cluster. By minimizing this bias, the deep reinforcement learning agent can adjust the container scheduling strategy so that the resource usage state of each node is as close as possible to the average load of the cluster, avoiding uneven resource usage.

[0037] In one specific embodiment, the optimization objective of the deep reinforcement learning agent can be modeled to include an optimization objective function, a first constraint, a second constraint, and a third constraint; wherein the optimization objective function is represented as minimizing a load balancing bias metric by optimizing deployment decision variables, the load balancing bias metric being the overall deviation of the load state of the node cluster from the average load state of the node cluster; the first constraint is used to represent the constraint that the CPU resources on each node will not be overused; the second constraint is used to represent the constraint that the memory resources on each node will not be overused; and the third constraint is used to represent the constraint that each to-be-deployed container can only be deployed on one node.

[0038] The embodiment has the beneficial effect that by introducing a deep reinforcement learning agent and making container deployment decisions based on a Markov decision process framework, the agent can dynamically adapt to changes in the load state of the cluster and fully consider the scheduling problem of multi-dimensional resource requirements (such as CPU, memory, etc.), thereby optimizing the resource allocation and load balancing of the cluster. Specifically, by minimizing the load balancing bias of the node cluster, the agent can achieve global load balancing of resources during container deployment, rather than single-dimensional local optimization, which effectively avoids the resource waste or node overload phenomenon that may be caused by traditional methods. Ultimately, this method can improve the resource utilization of the cluster, improve the efficiency and stability of container deployment, reduce the risk of system downtime, and optimize the long-term effect of container scheduling decisions.

[0039] In some embodiments, the step of obtaining state data in response to receiving the user request comprises: Step S122: Send a request for obtaining state data to a preset data collection module.

[0040] In this embodiment, the data collection module can be a cluster monitoring component for collecting and monitoring the resource utilization of each node in the cluster in real time.

[0041] In particular, the cluster monitoring component can be a Kubernetes Metrics Server. It can be appreciated that in a Kubernetes environment, the Metrics Server is a core component for collecting and aggregating cluster node and container resource utilization data (e.g., CPU, memory, etc.).

[0042] The cluster monitoring component can be a Prometheus data collection module.

[0043] The cluster monitoring component can be a custom collection agent. That is, a lightweight agent program is deployed to collect data through operating system calls.

[0044] Step S124: receiving state data sent by the data collection module; wherein the state data at least includes utilization rate data of a target resource of any single node in the node cluster, mean data of the utilization rate of the target resource in the node cluster, or standard deviation data of the utilization rate of the target resource in the node cluster; wherein the target resource at least includes CPU resources and memory resources.

[0045] In the embodiment, the target resource further includes disk I / O capability resources, network bandwidth resources, GPU resources, and container security requirements, etc.

[0046] In one specific embodiment, a container deployment method based on deep reinforcement learning in Kubernetes is provided.

[0047] It should be noted that compared with traditional virtual machines, container technology has the following characteristics: container technology can provide fine-grained resource allocation based on operating system-level virtualization technology, and users can specify the resource requirements of container instances. Compared with predefined virtual machine types, container instances have more diverse types and quantities of resource requirements; containers can be deployed on physical hosts or virtual machines, and their diverse configurations result in different computing resource capacities, which makes the physical host nodes in the container cluster heterogeneous in terms of resource capacity; the characteristics of fast start and stop and easy expansion of containers make cloud environments based on container technology have high scalability and elasticity, so the workloads of cloud environments based on container technology have high dynamics compared with traditional cloud environments based on virtual machine technology.

[0048] Container technology itself has the advantages of lightness and fast deployment, but if it is to be used in a large-scale production environment, it is not enough to just use containers, and container orchestration tools are needed to solve the deployment and management problems of containers in production environments. Currently, common container orchestration tools include Kubernetes, Docker Swarm, etc.

[0049] Container Deployment refers to a process in a container orchestration platform, in which the container orchestration platform creates a container instance according to a service request and deploys the container instance to a node, and a host node provides resources required by the container instance running time. The process of deploying a container to a node is often modeled as a bin packing problem, which is an NP-Hard problem, and many algorithms have been proposed by the industry and scholars for application in the container deployment problem.

[0050] The Kubernetes, a commonly used container orchestration platform in the industry, provides default preferred scheduling algorithms such as LeastRequestsPriority to select appropriate nodes for deployment of a to-be-deployed container, so as to achieve load balancing. The LeastRequestPriority algorithm selects the optimal deployment node for the container according to the percentage of the resources required by the to-be-scheduled resource object to the current node available resources, and the Balanced ResourceAllocation algorithm calculates the usage rate of CPU and memory on the node, and the node with the most balanced usage rate is the optimal node. The Docker Swarm provides three scheduling strategies, Binpack, Spread, and Random, which are based on fixed rules for scheduling. Although they are simple, they do not achieve good load balancing effect.

[0051] Although the container orchestration platform provides a set of scheduling mechanisms, there are still some deficiencies. Many scholars have proposed container deployment algorithms based on heuristic methods, mainly focusing on finding the optimal or near-optimal strategy of the problem. In the research of using heuristic algorithms for container deployment, in some related technologies, by analyzing the network traffic characteristics between containers, it is found that the network traffic distribution between containers conforms to Zipf distribution. Based on this, a container deployment algorithm Blender is proposed, which divides the container application into different container groups through subdivision algorithm and segmentation algorithm, and deploys the groups on virtual machines. Experimental results show that the Blender algorithm can effectively improve the resource utilization rate of physical cluster nodes. However, this implementation does not consider the load balancing problem of container cluster resource utilization.

[0052] In some related technologies, a multi-objective optimization method on a heterogeneous cluster is proposed, which converts the container deployment problem into a bin packing problem and solves it using a heuristic algorithm. Compared with the scheduling method of Kubernetes, this method can make the cluster achieve higher resource utilization, but ignores the overall load balancing of the cluster.

[0053] In some related technologies, the concurrent multi-resource container scheduling problem is simulated as a minimum cost flow model. Unlike traditional queue-based scheduling methods, this scheduling method can handle concurrent requests and deploy a batch of containers.

[0054] In some related technologies, a Kubernetes resource scheduling algorithm based on genetic algorithm is proposed to consider only CPU and memory resource dimensions for container scheduling. The algorithm considers network bandwidth and disk IO as two resource dimensions. Experimental results show that the algorithm has better effect on load balancing of the cluster. Static scheduling algorithms such as random scheduling algorithm are based on fixed rules for scheduling, and the load balancing effect is not good in the face of the diversity of container types and the heterogeneity of container cluster nodes in complex cloud environment. Secondly, the resource scheduling process of various cluster management systems is a very complex process, and it is usually difficult to design a general heuristic scheduling algorithm suitable for all scheduling scenarios. The cluster environment is usually heterogeneous, and many parameters in the heuristic method need to be configured for different clusters and different optimization targets, which often requires professional personnel to perform specific analysis. Finally, the heuristic algorithm is easy to fall into local optimum and is sensitive to parameters. In the case of workload changes, the administrator needs to manually adjust the algorithm parameters to achieve optimal results. In the container deployment scenario, a more intelligent algorithm is needed to adapt to the dynamically changing load environment and achieve the goal of cluster load balancing.

[0055] Load balancing methods based on deep reinforcement learning have been applied in computer networks and cloud task scheduling fields: in the field of computer network load balancing, in some related technologies, a mobility load balancing (MLB) algorithm based on deep reinforcement learning is proposed, and a two-layer structure is used to solve the large-scale load balancing problem of ultra-dense networks (UDN). In some related technologies, a MLB algorithm based on deep reinforcement learning off-orbit strategy is proposed, which autonomously learns the optimal MLB strategy under the asynchronous parallel learning framework without any prior knowledge assumption of the underlying UDN environment. In the cloud computing task scheduling scenario, in some related technologies, a task scheduling strategy based on DDPG algorithm is proposed under the condition of considering the SLA service agreement restrictions of tasks in the cloud environment, aiming to reduce the response time of tasks. The task and real-time load information of the cluster are input into the deep reinforcement learning agent, and the deep reinforcement learning agent makes a decision to assign the task to a virtual machine. Experiments show that in the face of dynamically changing loads, the algorithm can effectively reduce the task response time and improve the load balancing level of the cluster. In some related technologies, a cloud task allocation and scheduling algorithm DDMTS is proposed using the DQN algorithm, aiming to improve the load balancing level between virtual machines in the cluster and reduce the task rejection rate under the restriction of SLA. Experiments show that compared with the round-robin scheduling, random scheduling and other algorithms, the DDMTS algorithm can better achieve the load balancing effect of the virtual machine cluster. However, in this research, only the CPU resource utilization standard deviation is used as the load balancing judgment index, and the multi-resource dimension load balancing is not considered.

[0056] The above algorithms focus on the load balancing problem in the network load and task scheduling scenario in the cloud environment, and there are few studies on the load balancing problem of container clusters using deep reinforcement learning algorithm. However, the above research proves that deep reinforcement learning algorithm can make reasonable decisions in the face of dynamically changing loads in large-scale and complex cloud environments, achieve the goal of load balancing, and have good reference value for the research of container scheduling algorithm using deep reinforcement learning.

[0057] In one specific embodiment, a method for deploying containers in Kubernetes based on deep reinforcement learning is provided.

[0058] As Figure 2As shown, the basic flow of deploying containers is shown, when a container is deployed to a node, first a container to be deployed is popped out from the queue of containers to be deployed and input into the scheduler, the container to be deployed contains resource requests for CPU and memory. The resource capacity of the nodes in the node list is often heterogeneous, with different resource capacities, the scheduler needs to allocate the required CPU and memory for the container on the node to meet the resource requirements of the container according to the resource requirements of the container and the resource usage of the node. An important goal in this process is to achieve load balancing of node resource usage, the deployment algorithm should evenly distribute container instances to multiple physical nodes to avoid overloading of container instances on a node and cause the system to degrade or crash.

[0059] In addition, the deployment algorithm should intelligently allocate container instances to nodes with sufficient resources based on the resource status of the physical nodes and the resource requirements of the container instances, thereby achieving optimal utilization of resources. By reasonably allocating container instances, performance bottlenecks caused by node resource imbalance can be avoided, and the performance and resource utilization of the system can be improved.

[0060] The system model of container deployment can be summarized as follows: first, a container to be deployed is popped out from the queue of containers to be deployed, and its resource requirements are input into the scheduler. The scheduler includes three core modules, namely the load monitoring module, the load balancing determination module and the DRLCD agent. The load monitoring module is responsible for monitoring the load of the nodes, the load balancing determination module is responsible for determining the overall load balancing status of the current nodes, and the DRLCD agent selects the deployment node for the container according to the current state of the environment.

[0061] The output of the scheduler is the deployment node of the container to be deployed, and finally the container is bound to the selected node. Specifically, when the scheduler needs to deploy the container, the CPU and memory resource requirements of the container to be deployed are input into the scheduler; the monitoring module first obtains the current running state of the nodes, including the usage rate of CPU and memory resources of each node; CPU and memory resources are selected because in Kubernetes and other container orchestration platforms, the two resource dimensions are considered when deploying containers. The two states are combined as a state as the input of the DRLC agent, which is input into the DRLCD agent; the DRLCD agent selects the appropriate node for the container to be deployed according to the state, with the goal of load balancing of the container cluster in multiple resource dimensions. When the container is bound to the node, the monitoring module obtains the node load state again, and inputs the state into the load balancing determination module, which returns a reward to the RLCD agent according to the current node load state, so that the DRLCD agent is updated and learned.

[0062] The above container deployment problem can be formally represented as follows: a set of containers to be deployed: , , for container , its CPU and memory resource request amount is denoted as , . for physical machine node cluster set , node 's CPU and memory resource capacity . In order to represent whether container is deployed to node , a binary variable is defined , when container is deployed to node , otherwise :

[0063] The optimization goal of DRLCD algorithm is the load balancing of multi-resource dimensions of container cluster. Next, the load balancing optimization problem of container cluster is formally represented. The load condition of container cluster node can be represented as the ratio of the sum of allocated resources on the node to the node resource capacity, and represent the capacity of CPU and memory resources on node :

[0064] In the formula, the integral of the numerator is actually the integral of the resources currently occupied by each node, in other words, it is the resources already occupied in the node cluster. The denominator is the total resources, so this ratio actually represents the resource abundance or not. Specifically, denotes the CPU utilization rate of node , denotes the memory utilization rate of node , is the capacity of CPU on node , is the capacity of memory on node , is the CPU resource request amount of container , is the memory resource request amount of container .

[0065] The load state of node in the cluster can be represented as:

[0066] In the formula, denotes the CPU utilization rate of node the load status (CPU utilization and memory utilization) of each node in the container cluster.

[0067] The average load status of CPU and memory for the whole container cluster is:

[0068] In the formula, represents the average utilization of CPU in the cluster, represents the average utilization of memory in the cluster, represents the total number of nodes.

[0069] The average load status of the container cluster is defined as:

[0070] In order to reflect the load balancing of the container cluster on a single resource, the standard deviation of the utilization of the single resource in the cluster can be used to determine the load balancing status of the container cluster. For example, the standard deviation of the CPU utilization of the container cluster can be represented as follows:

[0071] In the formula, is the standard deviation of the CPU utilization of the container cluster.

[0072] The standard deviation of the resource utilization reflects the current load balancing status of the container cluster. The goal of the present embodiment is to achieve load balancing in multiple resource dimensions of the container cluster. Inspired by the standard deviation of the utilization of a single resource, the load balancing optimization target is defined as follows (optimization objective function):

[0073] In the formula, is the sum of the Euclidean distances between the load status of all nodes in the cluster and the average load status of the cluster, which represents the overall deviation of the load status of the nodes in the cluster from the average load status of the cluster. Using the Euclidean distance can reflect the load balancing level of the container cluster in two resource dimensions of CPU and memory.

[0074] The embodiment obtains state data through a standardized data acquisition module, and significantly improves the real-time performance and global optimization capability of container deployment decision. First, the integrity of decision information is enhanced, and through accurate acquisition of node-level resource utilization (CPU, memory, and other core indicators) and cluster-level statistics (for example, mean, standard deviation), a multi-dimensional and multi-granularity load state image is provided for a deep reinforcement learning agent, thereby solving the one-sidedness problem of traditional methods relying only on local information. Secondly, the adaptability of the dynamic environment is improved, and the mean data reveals the overall load trend of the cluster, and the standard deviation data quantifies the degree of resource deviation between nodes. The dynamic indicators of these two dimensions work together to enhance the decision robustness of the agent in the load fluctuation scenario.

[0075] In some embodiments, the container deployment decision model is a deep reinforcement learning agent based on a Dueling DQN algorithm, and the Markov decision process framework includes at least a state space, an action space, a reward function, and a state transition function; wherein the state space is constructed based on the resource requirement data and the state data; the action space is used to represent a set of node numbers in the node cluster, and the action output is a target node number in the node cluster, the reward function is modeled based on the optimization target, and is used to calculate a reward value according to a deployment result to guide training, and the state transition function is used to update the load state of the node cluster in response to deploying the to-be-deployed container to the target node, and generate a next state.

[0076] In this embodiment, the Markov decision process framework can further include a discount factor. The discount factor is introduced in the Markov decision process to balance the influence of long-term rewards and short-term rewards obtained in the deployment process on the agent.

[0077] The step of using a preset container deployment decision model to perform a decision on container deployment for the resource requirement data and the state data to obtain a target node includes: Step S142: constructing a state space based on the resource requirement data and the state data; Step S144: inputting the state space into a deep reinforcement learning agent based on a Dueling DQN algorithm to obtain a target node number output by the action space.

[0078] The beneficial effects of the present embodiment are that by adopting a deep reinforcement learning agent based on the Dueling DQN algorithm and combining the Markov decision process framework, the intelligentization and global optimization of container deployment decisions are achieved. First, the construction of the state space is based on the resource demand data of the container and the cluster state data, ensuring that the model can accurately capture various resource demands and node load states in the container deployment process, thereby making the scheduling decision more accurate. Second, through the design of the action space, the agent can select the target node in the cluster that is most suitable for deploying the container, which helps to avoid the problem of resource overload or imbalance. Third, the reward function is designed according to the optimization target (load balancing), which can guide the agent to continuously optimize its decision strategy and improve the effect of container scheduling. Finally, the state transition function updates the load state of the node cluster after the container is deployed and provides new input for the next decision, so that the agent can continuously learn and adapt in a dynamic load environment, thereby improving the decision efficiency and load balancing ability of the system. This approach significantly improves the intelligent level of the container scheduling process, enabling more efficient and balanced allocation of cluster resources.

[0079] In some embodiments, the optimization target of the deep reinforcement learning agent based on the Dueling DQN algorithm is modeled to include an optimization objective function, a first constraint, a second constraint, and a third constraint; wherein the optimization objective function is represented as minimizing a load balancing deviation metric by optimizing deployment decision variables, the load balancing deviation metric being the overall deviation of the load state of the node cluster from the average load state of the node cluster; The first constraint is used to represent the constraint that the CPU resources on each node will not be overused. The second constraint is used to represent the constraint that the memory resources on each node will not be overused. The third constraint is used to represent the constraint that each to-be-deployed container can only be deployed on one node.

[0080] The beneficial effects of the present embodiment are that by using a deep reinforcement learning agent based on the Dueling DQN algorithm to optimize container deployment decisions, more accurate and efficient resource scheduling can be achieved. First, the optimization objective function minimizes the load balancing deviation metric, ensuring that the resource usage of each node in the cluster is more balanced, avoiding the phenomenon of node overload or resource waste, thereby improving the overall resource utilization and stability of the cluster. Second, the introduction of constraints ensures the rationality of resource usage when the system performs container scheduling, such as preventing excessive use of node resources through CPU resource constraints and memory resource constraints to avoid performance degradation or system crashes due to resource overload; container deployment constraints ensure that each container is deployed on only one node, avoiding resource redundancy and scheduling errors. Through the organic combination of these optimization objectives and constraints, the agent can ensure load balancing and resource utilization while maintaining the efficiency, robustness, and reliability of the system, effectively dealing with dynamic load changes and complex cluster environments.

[0081] In a specific embodiment, the optimization problem of the deployment algorithm can be formalized as follows:

[0082] In the formula, represents the set of containers to be deployed, containing M containers in total; represents the set of physical nodes in the container cluster, containing J nodes in total; represents the CPU resource request of container ; represents the memory resource request of container ; represents the available CPU resource capacity of node ; represents the available memory resource capacity of node ; represents the decision variable of whether container is deployed to node ; if , it means that container is deployed to node ; if , it means that container is not deployed to node . The first constraint ensures that the CPU resources on each node will not be overused; the second constraint ensures that the memory resources on each node will not be overused; the third constraint ensures that each container must be and can only be deployed on one node; the optimization objective function which means to minimize the load balancing deviation metric by optimizing the deployment decision variable d , i.e. to improve the resource allocation balance of the cluster.

[0083] From the above formula, we can know that the result is affected by the container deployment decision . When the container is deployed to the cluster node according to the result output by the scheduling algorithm, the container cluster state changes from to , and the DRLCD algorithm acts as a scheduling algorithm to interact with the container cluster. Therefore, the container deployment problem can be modeled as a Markov decision process (MDP). The Markov decision process is represented by the tuple . For the container deployment problem, the meaning of the elements in the tuple is as follows: State space S: In the container deployment problem, the elements in the state set S are the running states of the container cluster after each deployment . The state is a set composed of the current running states of each node in the cluster. .

[0084] Action space A: For the container deployment problem, the action set is a set composed of the nodes in the container cluster: .

[0085] Discount factor : In the Markov decision process, the discount factor is introduced to balance the influence of long-term rewards and short-term rewards obtained during the deployment process on the agent.

[0086] Reward function : When deploying a container , the reward depends on both the current container cluster state and the action taken, i.e. the node to which the container is deployed.

[0087] The above formula presents the goal of container cluster load balancing, so the design of the reward function should be based on .

[0088] State transition function : represents the probability of reaching state after performing action a in state S.

[0089] The DRLCD algorithm uses a deep reinforcement learning algorithm to solve the optimization problem, and the solution strategy is: , which represents the probability of taking action a in state S. The goal of the DRLCD algorithm is to design the state space, action space and reward function based on the framework of Dueling DQN algorithm for the container deployment problem, and solve the optimal strategy , so that the DRLCD agent can obtain the maximum cumulative reward value from the beginning of deployment to the end of deployment according to the optimization goal .

[0090] In some embodiments, the model structure of the deep reinforcement learning agent based on the Dueling DQN algorithm includes: An input layer for receiving data in the state space.

[0091] In this embodiment, the input layer is the first layer of the deep reinforcement learning agent, mainly responsible for receiving state space data. The state space data can include resource usage of each node in the cluster (CPU, memory, network bandwidth, etc.) and resource requirement data of the container. The task of the input layer is to pass these state data to the next layer for further feature extraction and processing.

[0092] A shared feature extraction layer including at least one fully connected hidden layer with a ReLU function as the activation function, for extracting shared features of the node cluster load state.

[0093] In this embodiment, the shared feature extraction layer is responsible for feature extraction of the input data (resource requirements, node load state) and converts it into an abstract representation that can be used for subsequent decision-making. The shared feature extraction layer can include multiple fully connected hidden layers (FC Layers) with ReLU (Rectified Linear Unit) as the activation function.

[0094] In this embodiment, the shared feature extraction layer is responsible for feature extraction of the input data (resource requirements, node load state) and converts it into an abstract representation that can be used for subsequent decision-making.

[0095] A branch processing layer including a state value function branch and an action advantage function branch; wherein the state value function branch is connected to the shared feature extraction layer and outputs a state value function representing the current overall cluster state value, and the action advantage function branch is connected to the shared feature extraction layer and outputs an advantage value function representing the action advantage value of each node.

[0096] In this embodiment, the branch processing layer divides the network into two independent branches: the state value function branch and the action advantage function branch, which respectively calculate the value of the overall cluster state and the action advantage of each node.

[0097] In this embodiment, the state value function branch calculates the overall value of the current state of the cluster, i.e., the overall quality of the current environment given the state of the cluster resources. The output of this part reflects the overall health of the cluster, helping the agent to understand the performance of the current system.

[0098] In this embodiment, the action advantage function branch evaluates the action advantage of each node, i.e., the relative advantage of choosing a particular node to deploy containers in a certain state. It helps the agent to focus on evaluating whether the selection of this node is better than other nodes when evaluating each node.

[0099] The output merging layer is used to receive the state value function and the action advantage value, and output the Q value of each node based on the state value function and the action advantage value, and then normalize the Q value through the softmax function to generate the scheduling action probability distribution of all nodes.

[0100] In this embodiment, the output merging layer can output the Q value of each node based on the state value function and the action advantage value through the following function:

[0101] In the formula, Q (S, a) represents the Q value, and represents the expected cumulative reward when choosing action in state S. V (S) represents the state value function, which is used to evaluate the value of the overall state of the current cluster, A (S, a) represents the action advantage function, which is used to evaluate the relative advantage value of choosing node to deploy in state S, is the network parameter shared by the state value function and the advantage function, which is used to extract the first few layers of features in the neural network, and are the parameters of the state value function and the advantage function, respectively.

[0102] In this embodiment, the output merging layer can output the Q value of each node based on the state value function and the action advantage value through the following function:

[0103] In the formula, a is any action in the action space, is the deployment action.

[0104] In this embodiment, the output merging layer can output the Q value of each node based on the state value function and the action advantage value through the following function:

[0105] In the formula, The action space size.

[0106] A decision layer for selecting a target node number based on the scheduling action probability distribution, wherein the node number corresponding to the highest probability is the output action.

[0107] In this embodiment, the decision layer can select a target node number according to the scheduling action probability distribution. This layer uses the probability distribution from the output merging layer.

[0108] The decision layer can select the target node number with the highest probability from the generated probability distribution as the final scheduling action by using the maximum probability method (greedy strategy). That is, the agent can select the node with the highest probability to deploy the container, ensuring the optimality of the scheduling decision.

[0109] In a specific embodiment, deep reinforcement learning is a machine learning method that organically combines the capabilities of deep learning and reinforcement learning. Deep reinforcement learning, by using the capabilities of deep learning, can learn various high-level data representations from raw data, discover complex structural patterns from large data sets, and effectively approximate complex functions; by using the capabilities of reinforcement learning, it can learn optimization strategies to solve complex decision and control problems. Through the fusion of using the achievements of deep learning to support the reinforcement learning process and using the reinforcement learning process to supervise the deep learning process, the learning ability of deep reinforcement learning has been significantly improved. Reinforcement learning is a branch of machine learning, and its background can be traced back to the artificial intelligence research in the 1950s and 1960s. At that time, researchers hoped to be able to let machines learn and make decisions like humans, but traditional machine learning methods could not solve problems involving unknown environments. In order to solve this problem, researchers began to explore a new learning method, i.e., reinforcement learning. Reinforcement learning allows machines to learn and improve their decision-making abilities by interacting with the environment. This learning method is similar to human trial-and-error learning, which adjusts its behavior through continuous trial and feedback to gradually adapt to the environment.

[0110] In traditional reinforcement learning algorithms, hand-designed features are usually used to represent the state, but these features may not be optimal. In this context, deep learning technology provides a new way to solve these problems. Deep neural networks can solve high-dimensional state space problems by learning high-level features that represent data; and deep reinforcement learning algorithms can solve continuous action space problems by learning deep representations of strategies or value functions.

[0111] The state space refers to the current container cluster environment information perceived by the deep reinforcement learning (DRL) CD agent. When a container needs to be deployed, the deployment process is triggered. The deep reinforcement learning agent needs to obtain two aspects of state: one is the state of the container, including the resource information requested by the container; the other is the running state of the cluster, including the CPU resource availability and memory resource availability of the nodes in the cluster. In a cloud environment, the node resource capacity is often heterogeneous, and different nodes have different resource quantities, and the resource quantities are inconsistent in order of magnitude. In order to cope with the challenge of node heterogeneity, the DRLCD algorithm uses the resource utilization rate of the node as the state input to limit the value of the node state to the interval, improving the accuracy and stability of the model.

[0112] The action space refers to the action space of the deep reinforcement learning agent, which refers to the output action set. In the container deployment scenario, for a cluster with J nodes, the output action is the number of nodes in the cluster, and the action space of the Dueling DQN agent is the set of node numbers:

[0113] In the formula, represents the action space, represents the action at time t, and J is the total number of cluster nodes.

[0114] Suppose that after the DRLCD agent judges, the output action is This action represents that the container needs to be deployed to node .

[0115] In reinforcement learning, the reward function is used to measure whether the agent's behavior in the environment is effective. The reward function is usually a function that converts from the state of the environment to a scalar value, indicating the good or bad degree of the agent's behavior in that state. The reward function defines the goal of the problem and provides feedback signals to guide the agent's learning. Therefore, the reward function plays a crucial role in reinforcement learning, determining the learning goal, learning speed, and stability of the agent. Therefore, when designing the reward function of reinforcement learning, careful consideration is needed to ensure that the agent can learn useful behaviors in the environment.

[0116] The goal of this embodiment is to balance the load of the two resource dimensions of CPU and memory in the cluster, and the reward function must be designed around this goal. When the agent successfully schedules, it can bind the container to be scheduled to the target node. The reward function will evaluate the good or bad of this action according to the changes in the container cluster load balancing.

[0117] The following formula defines the optimization goal of load balancing:

[0118] In the formula, is the sum of the Euclidean distance between the load state of all nodes in the cluster and the average load state of the cluster, which represents the overall deviation of the load state of the nodes in the cluster from the average load state of the cluster. Using the Euclidean distance can reflect the load balancing level of the container cluster in two resource dimensions of CPU and memory.

[0119] When designing the reward function, the optimization goal of load balancing should be around. At time t, the load balancing degree of the container cluster is :

[0120] When the container is deployed to the node , at time t+1 :

[0121] In the formula, is the load balancing degree at time t+1.

[0122] It comprehensively reflects the deviation of each node in the current cluster from the average load, and reflects the load balancing state of CPU and memory resources. According to the values of and before and after deployment, the embodiment determines whether the action is beneficial to the load balancing state of the container cluster: if , it means that the container deployment process improves the load balancing degree of the cluster, and the DRLCD agent needs to return a positive reward; if , it means that the overall load balancing degree of the cluster does not improve, and the reward is set to a negative value.

[0123] When the agent selects the target node for the container to be scheduled , and the container cannot be bound to the corresponding node due to insufficient resources, it is directly considered as a scheduling failure. and represent the available CPU and memory on the node at time t, that is,

[0124] In the formula, and represent the available CPU and memory on the node at time t.

[0125] At this time The agent should be given the most severe punishment. In combination with the above situation, the DRLCD is set as follows: The reward function of the agent can eventually make the agent converge and obtain good experimental results:

[0126] In the formula, is the reward function of the agent.

[0127] The neural network part of the DRLCD agent adopts a fully connected structure, connecting the input state to the network structure of the fully connected layer. The activation function of each hidden layer is the ReLU function. The input of the network is the global resource state information of the current container cluster, including the CPU and memory utilization of each node, and the resource request demand of the to-be-deployed container, etc. The DRLCD agent uses a network with a duel (Dueling) structure. The output layer is divided into two branches, representing the state value function and the action advantage function respectively. The state value branch estimates the value of the current overall system resource state, while the action advantage branch measures the relative merits of deploying the container to different nodes. This structure helps to improve the stability of training, especially in scenarios where certain nodes have little influence on the final scheduling action, which can reduce the interference of invalid information on policy learning. The first few layers of the network are standard fully connected structures, which are used to extract shared resource state features. Then it enters the state value and advantage two sub-network branches, which are also composed of several fully connected layers, respectively completing the modeling tasks of state value and each action (i.e. node) scoring. The output layer uses the softmax function to normalize the scores of all nodes after merging the two branches, forming the final scheduling action probability distribution. The agent selects actions according to the distribution, i.e. determines which target node the to-be-deployed container should be scheduled to. The overall network structure is gradually converged through the training of the integration group resource scheduling data samples in the experiment, showing good scheduling effect and policy generalization ability.

[0128] The embodiment provides a container deployment method, comprising: Step S22: determining the target node corresponding to the to-be-deployed container; wherein a target node determination method provided in the above embodiment is used to determine the target node corresponding to the to-be-deployed container; Step S24: deploying the to-be-deployed container to the target node.

[0129] In the present embodiment, the scheduler can generate a binding instruction of the target node, which can include a target node identifier and resource requirement data of the to-be-deployed container. The scheduler can submit the binding instruction to a local agent component on the target node. The local agent component can execute instantiation of the container.

[0130] The embodiment provides a container deployment decision model updating method, including: Step S42: deploying the to-be-deployed container to the target node; wherein a target node determination method provided in the above embodiment is used to determine the target node corresponding to the to-be-deployed container; Step S44: obtaining a deployment result.

[0131] In the embodiment, the deployment result can be in two cases: 1. successful deployment; 2. unsuccessful deployment.

[0132] Step S46: calculating a reward value through a reward function in response to the deployment result.

[0133] In the embodiment, the reward function can be:

[0134] Step S48: updating network parameters of the deep reinforcement learning agent based on the reward value.

[0135] In some embodiments, the step of calculating the reward value through the reward function includes: If the container deployment fails, the reward value is a first preset value; wherein the first preset value is a negative value; If the deployment is successful and a cluster load balancing deviation after the deployment is smaller than that before the deployment, the reward value is a second preset value; wherein the second preset value is a positive value greater than 0 and smaller than 1; If the deployment is successful and the cluster load balancing deviation after the deployment is greater than or equal to that before the deployment, the reward value is the load balancing deviation before the deployment minus the load balancing deviation after the deployment; wherein the load balancing deviation is calculated through the sum of Euclidean distances of node resource utilization rates and cluster average utilization rates; and wherein the load balancing deviation is constrained in a numerical interval of 0 to 1.

[0136] In the embodiment, the first preset value can be -1, -0.99, -0.98, etc.

[0137] In the embodiment, the second preset value can be 0.4, 0.45, 0.55, etc.

[0138] In some embodiments, the step of updating the network parameters of the deep reinforcement learning agent based on the reward value includes: The deployment node is selected by a directional exploration strategy in the training stage; wherein the step of selecting the deployment node by the directional exploration strategy in the training stage comprises: in response to entering the action exploration stage, obtaining the CPU available utilization and the memory available utilization of each node in the node cluster; based on the CPU available utilization and the memory available utilization, calculating the comprehensive resource available rate of each node; based on a preset directional greed rule, selecting the node with the largest comprehensive resource available rate as the deployment action.

[0139] The present embodiment significantly improves the training efficiency and decision quality of the deep reinforcement learning agent by introducing the directional exploration strategy. First, the convergence speed is accelerated, and by preferentially selecting nodes with high resource availability as exploration actions, the number of invalid deployment attempts is reduced, thereby avoiding the agent from falling into inefficient random exploration in the early training stage and shortening the time for the model to converge to the optimal strategy. Second, resource waste is inhibited, and the directional greed rule avoids nodes with insufficient resources, reducing the frequency of triggering punitive rewards due to deployment failures and ensuring the stability of the training process.

[0140] In a specific embodiment, there are two very important concepts in reinforcement learning tasks: exploration and exploitation.

[0141] Exploration: refers to the agent selecting other unknown actions outside the known (state, action) pair distribution.

[0142] Exploitation: In reinforcement learning, exploitation refers to the agent selecting the optimal action among all known (state, action) pair distributions based on the principle of maximizing action value.

[0143] Reinforcement learning algorithms often use a policy to exploit and explore, selecting actions from the action space .

[0144]

[0145] In the formula, argmax is a mathematical operator that finds the parameter value that maximizes the function value, is the estimated value of the action value function, is the action space, is the exploration probability.

[0146] When the agent performs random action sampling, i.e., the action exploration stage, the agent randomly selects actions from the entire action space with equal probability This selection strategy is not optimized for container deployment load balancing scenarios, which will lead to invalid exploration of load balancing targets in the early exploration stage, reducing the learning efficiency of the DRLCD agent. Therefore, unlike trial-and-error learning, in order to improve the efficiency of action exploration and enable the DRLCD agent to learn the load balancing deployment strategy more quickly, the present embodiment uses heuristic scheduling rules to stimulate the agent to perform directional exploratory learning in the early stage of DRLCD agent training, and proposes a DRLCD action exploration strategy.

[0147] At deployment, the node set , unlike randomly selecting nodes from the set with equal probability, for the load balancing scenario in the container deployment stage, the present embodiment introduces a greedy rule, which deploys containers to nodes with more available resources to improve the load balancing of the cluster, so the node with the most available resources is preferred in the exploration stage. In addition, due to the dynamic nature of container cluster load changes, the load of the current cluster needs to be judged when exploring randomly, and the deployment node is selected according to the load of the current container cluster.

[0148] For , the node has an available CPU utilization rate of and a memory utilization rate of .

[0149] The resource comprehensive availability of the node is: In the formula, is the resource comprehensive availability of the node (the average of the available CPU utilization rate and the memory utilization rate of the node ).

[0150] The set of resource comprehensive availabilities of the current cluster is: ; According to the greedy rule, the node corresponding to the node with the maximum resource comprehensive availability of the current cluster is selected as the deployment node in the exploration stage. In summary, the pseudo code of the DRLCD action exploration strategy is as follows: Input: node set; Output: action; 1. Get the node cluster state ; 2. For the node in the node set do; 3. Calculate the node comprehensive resource availability ; 4. End for; 5. Selecting the current cluster resource comprehensive availability rate maximum from the cluster resource comprehensive availability rate Corresponding node ; 6. Output .

[0151] The container deployment and load determination are performed.

[0152] Load balancing of the container cluster needs to consider multiple resource dimensions of indicators, because different indicators can reflect different aspects of performance and state of the cluster, and selection of a single indicator can cause important features of the cluster to be ignored, thereby reducing the final effect of load balancing. In the embodiment, the indicators of CPU and memory resources are selected to determine the state of load balancing of the cluster.

[0153] In order to measure the load balancing state in the cluster, the runtime CPU and memory utilization standard deviation in the cluster are used to measure the load balancing degree of the cluster. Assuming that the total runtime of the cluster is T, at time t, the average CPU utilization and memory utilization in the cluster are:

[0154] In the formula, is the total number of cluster nodes, is the CPU utilization of node j at time t, is the ram utilization of node j at time t, is the average CPU utilization of the cluster at time t, is the average ram utilization of the cluster at time t.

[0155] According to the above formula, the standard deviation of CPU and memory utilization is:

[0156] In the related art, an indicator DCLI (Degree of Cumulative Load Imbalance) for measuring the load imbalance degree of the cluster is proposed, that is, the cumulative value of the standard deviation of CPU and memory utilization of the cluster at each time within time T:

[0157] In the formula, DCLI measures the load difference between each node in the cluster, and the lower the value of DCLI, the higher the load balancing degree of the cluster, is the CPU cumulative imbalance degree (total amount of CPU load fluctuation within time T), The accumulated imbalance degree of memory (total amount of RAM load fluctuation within time T), The standard deviation of CPU utilization at time t, The standard deviation of RAM utilization at time t.

[0158] Another index DI (Degree of Imbalanca) is proposed in the related art to measure the imbalance degree of cluster load, which is the difference between the highest load and the lowest load of the cluster divided by the average load of the cluster. DI is defined as follows at time t:

[0159] In the formula, DI is an index for measuring the imbalance degree of cluster load, is the maximum node load (the peak value of the load of all nodes in the cluster at time t), is the minimum node load (the valley value of the load of all nodes in the cluster at time t), is the average load of the cluster.

[0160] The embodiment measures the load balancing degree of two resource dimensions of CPU and memory, i.e.:

[0161] In the formula, the DI index measures the deviation of the maximum load and the minimum load in the cluster compared to the average load, reflecting the overall load balancing state of the cluster. The lower the value of DI, the higher the load balancing degree of the cluster.

[0162] Performing update training Dueling DQN agent.

[0163] As shown in Figure 5 , the Markov property means that the next state depends only on the current state of the environment and action , and is independent of earlier states or actions. A Markov process is a binary tuple , where S is a set of finite states, and P is a state transition probability, denotes the probability of transitioning from state to state . Once the initial state is given, a complete Markov process can be determined. A Markov Decision Process (MDP) is a discrete-time stochastic control process consisting of 5 elements . Among them, S is the state space; A is the action space; T is the transition function (a set of conditional transition probabilities between states); R is the reward function, and has a definite range; is the discount factor. The transition probabilities of the Markov Decision Process follow the definition:

[0164] Reinforcement learning is usually modeled using Markov Decision Processes. At each time node, the agent makes a decision by choosing an action from the set of actions , which leads to a state of the environment with a certain probability given by the transition function , and a reward given by the reward function . This series of action-dependent decisions is called a policy (policy) in reinforcement learning, denoted as :

[0165] , which refers to the distribution over the set of actions given the state . Through multiple iterations of the loop, the agent will learn the optimal policy .

[0166] The Q-learning algorithm cannot be applied to problems with a large state or action space due to the dimensionality of the Q-table. The core idea of the DQN algorithm is to use a deep neural network to approximate the Q-value function, which to some extent solves the problem of space explosion.

[0167] Suppose the parameters w of the neural network used to fit the function Q, that is, the Q value of all possible actions a under each state s can be expressed as , the neural network used to fit the function Q is called the Q network. According to the update rule of the Q-learning algorithm for the Q value function:

[0168] In the formula, is the state-action value function, is the next state, is the learning rate, is the immediate reward, is the discount factor, is the parameter update operation.

[0169] The above formula uses the time difference learning goal to incrementally update for a set of data . The loss function of the Q network can be constructed in the form of mean square error

[0170] In the formula, is the weight parameter of the Q network, is the optimal network parameter after optimization, represents the number of training samples, represents the state of the i-th sample, represents the action of the i-th sample, represents the immediate reward of the i-th sample, represents the next state of the i-th sample, represents the predicted Q value, the expected long-term return of the agent for action and the next state . represents the target Q value, the expected long-term return of the optimal action for the new state .

[0171] The gradient of the loss function is calculated according to the above formula and the neural network is updated.

[0172] In addition to the Q network, the DQN algorithm has two important modules: experience replay and target network, which can help DQN achieve more stable and excellent performance. The DQN algorithm uses the experience replay method, which maintains a replay buffer to store the four-tuple data (state, action, reward, next state) sampled from the environment each time, and randomly samples some data from the replay buffer for training when training the Q network. This can make the samples meet the independent assumption and improve sample efficiency.

[0173] To solve the instability of neural network training, DQN uses the idea of target network. Two sets of Q networks are used: (1) the original training network , which is used to calculate the term in the loss function and is updated by normal gradient descent method. (2) the target network , which is used to calculate the term in the loss function, where represents the parameters in the target network. To make the update target more stable, the target network is not updated every step. Specifically, the target network uses a relatively old set of parameters of the training network, and the training network is updated at every step while the parameters of the target network are updated once every C steps, i.e. . This makes the target network more stable than the training network. In summary, the specific process of the DQN algorithm is as follows: 1, initialize the network with random network parameters W; 2, copy the same parameters ; 3, initialize the experience replay pool R; 4, for sequence ; 5, get the initial state of the environment ; 6, for time step 7. Based on the current network ,by Greedy strategy selects action 8. Perform the action to receive a reward The environmental state becomes 9. Store in playback pool R; 10. If there is enough data in R, sample N data points from r. 11. For each data point, compute using the target network. Minimize target loss This updates the current network. 12. Update the target network; 13. End for; 14. End for.

[0174] Dueling DQN is an improved algorithm that, in reinforcement learning, defines the advantage function A as the result of subtracting the state value function V from the action value function Q. In a unified state, the sum of the advantage values ​​of all actions is 0, because the expected value of all actions is equal to the state value of that state. Therefore, in Dueling DQN, the Q-network is modeled as follows:

[0175] The network structure of Dueling DQN is as follows: Figure 4 As shown, in Figure 4 middle, For the state value function, and This is the advantage function for taking different actions in this state, representing the difference in taking different actions; These are network parameters shared by the state-value function and the advantage function, used in neural networks to extract features in the first few layers; while... and These are the parameters for the state value function and the advantage function, respectively. In this model, the neural network no longer directly outputs the Q-value. Instead, the last few layers of the neural network are trained to output the state value function and the advantage function, respectively, and then the Q-value is obtained by summing them.

[0176] The advantage of modeling the state value function and the advantage function separately is that in some environments, the agent only focuses on the state value and does not care about the differences caused by different actions. In this case, modeling the two separately can enable the agent to better handle states that are less related to actions.

[0177] Regarding this formula , it has the problem of non-uniqueness of V value and A value modeling. For example, for the same Q value, if the V value is added by an arbitrary constant C, and all A values are reduced by C, the resulting Q value remains unchanged, which leads to instability of training.

[0178] To solve this problem, Dueling DQN forces the actual output of the advantage function of the optimal action to be 0, that is:

[0179] At this time , the uniqueness of V value modeling can be ensured. In the implementation process, the average operation can also be replaced by the maximum operation, that is:

[0180] Dueling DQN can more efficiently learn the state value function. Each time the function V is updated, which will also affect the Q value of other actions. While DQN will only update the Q value of a certain action, and the Q value of other actions will not be updated. Therefore, Dueling DQN can more frequently and accurately learn the state value function.

[0181] In this embodiment, an experiment is performed to verify.

[0182] 1. Experimental environment configuration All experiments in this embodiment are carried out in the Windows10 operating system, Python3.6 environment, and the Pytorch version is 1.9.0. The experiment is carried out in a simulation environment. The simulation environment used or referred to in related research is the widely used cloud computing environment simulation tool CloudSim. CloudSim can provide effective simulation of real cloud computing environments. This embodiment refers to the specific implementation of CloudSim and develops a simulation platform suitable for Python environment based on it, which can simulate the running process of container clusters in a real environment.

[0183] 2. Experimental data set and experimental parameter setting This embodiment uses the data set published in Alibaba Cluster Trace Program as the experimental data set. This paper uses data from 2018. The 2018 data set includes load data of 4000 machines in 8 days, and many studies have used the 2018 data set Container configuration According to different resource requests, the data set provides containers with different resource demand configurations of CPU resources and memory resources. This configuration simulates the diversity of resource demand of containers during deployment.

[0184] 3. Load balancing evaluation index Load balancing of a container cluster needs to consider indexes of multiple resource dimensions, because different indexes can reflect different aspects of performance and state of the cluster, and selection of a single index can cause important features of the cluster to be ignored, thereby reducing the final effect of load balancing. In the present embodiment, indexes of two resources, CPU and memory, are selected to determine the state of load balancing of the cluster. To measure the load balancing state in the cluster, the present embodiment uses the runtime cluster internal difference to measure the degree of load balancing of the cluster. Assuming that the total running time of the cluster is T, at time t, the average CPU utilization and memory utilization in the cluster are:

[0185] wherein, represents the CPU utilization of the node at the t time, represents the memory utilization of the node at the t time. According to the above formula, the standard deviation of the CPU and memory utilization can be calculated as:

[0186] A related art proposes an index for measuring the degree of load imbalance of the cluster, DCLI (Degree of Cumulative Load Imbalance), that is, the cumulative value of the standard deviation of the CPU and memory utilization of the cluster at each time within the time:

[0187] DCLI measures the difference in load between each node in the cluster, and the lower the value of DCLI, the higher the degree of load balancing of the cluster.

[0188] Another related art proposes an index DI (Degree of Imbalanca) for measuring the degree of load imbalance of the cluster, which is the difference between the highest load and the lowest load of the cluster divided by the average load of the cluster, and DI is defined as follows, at time t:

[0189] The present embodiment measures the degree of load balancing of two resource dimensions, CPU and memory, that is:

[0190] The DI index measures the deviation of the maximum load and the minimum load in the cluster compared to the average load, reflecting the overall load balancing state of the cluster. The lower the value of DI, the higher the degree of load balancing of the cluster.

[0191] 4. Comparison algorithms In terms of comparison algorithms, in order to fully compare with existing different types of methods, five comparison algorithms are selected in this paper. The five comparison algorithms include the load balancing method based on deep reinforcement learning, the widely cited resource allocation method based on heuristic method, the heuristic container deployment algorithm and the container scheduling method used in actual tools, which are respectively: DDMTS algorithm: load balancing method based on deep reinforcement learning, which uses DQN agent to select appropriate nodes for task deployment according to the current state of the system.

[0192] Tetris algorithm: classic heuristic resource allocation method, which selects appropriate nodes for container deployment according to the resources required by the container and the current resource usage state of the system. BACP algorithm: heuristic container deployment algorithm, which selects nodes with lower average load and enables the cluster load to be balanced after container deployment as container deployment nodes.

[0193] Least Requested algorithm (LR): one of the default preferred strategies of container orchestration tool Kubernetes, which selects the node with the most idle CPU and memory resources to deploy containers.

[0194] Round Robin algorithm (RR): rule-based scheduling method, which selects available nodes to deploy containers in turn. This method is used as the benchmark method in the experiment.

[0195] 5. Load balancing experimental results Figure 6 The DCLI index of CPU and memory is shown. The meaning of DCLI index is the cumulative value of standard deviation of resource utilization in a deployment period, so the lower the value of DCLI index, the better the load balancing performance of container cluster in a deployment period. The value of DCLI of DRLCD method is the lowest among these methods, which means that the best load balancing effect can be achieved in the container deployment stage. When compared with the Tetris algorithm which is the best heuristic method, the DRLCD algorithm improves the DCLI value of CPU and memory by 12% and 11% respectively. Compared with the DDMTS algorithm which is also based on deep reinforcement learning algorithm, the DRLCD algorithm improves the DCLI value of CPU and memory by 7.2% and 7.8% respectively. It shows that the DRLCD algorithm based on DuelingDQN proposed in this paper is more effective than the DDMTS method using DQN algorithm in achieving the goal of cluster load balancing. Figure 7Another DI index for evaluating the load balancing degree of the cluster is shown. The DI index measures the load imbalance level of the whole cluster, and the lower the value of DI, the more balanced the load of the cluster. Similar to the DCLI index, the RR based on the fixed rule scheduling algorithm cannot achieve good load balancing effect, and the DI value is significantly higher than that of other methods. The DRLCD algorithm based on deep reinforcement learning is still better than the heuristic-based algorithm in the DI value. In particular, the effect of the DRLCD index is still the best, and the DI index of the CPU and memory of the DRLCD algorithm is improved by 4.9% and 3% compared with the DDMTS algorithm. In summary, under this scenario, the DRLCD algorithm can achieve good load balancing effect in container scheduling and deployment. According to an embodiment of the present application, an electronic device is provided. The electronic device in the embodiment can include one or more of the following components: a processor, a network interface, a memory, a non-volatile memory, and one or more application programs, wherein the one or more application programs can be stored in the non-volatile memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform the method described in the foregoing method embodiments. According to an embodiment of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. When the computer program is executed by a computer, the computer performs the method described in any of the foregoing embodiments. In the foregoing embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments. The above is only the preferred embodiment of the present application. It should be noted that, for those skilled in the art, without departing from the principle of the present application, a number of improvements and refinements can be made, which should also be regarded as the protection scope of the present application.

Claims

1. A method of determining a target node, characterized by, The method comprises: in response to receiving a user request, obtaining state data; wherein the user request represents a request for instructing to perform deployment on at least one to-be-deployed container, and the user request carries resource requirement data of the to-be-deployed container; the state data is data for characterizing a load state of a node cluster; wherein the resource requirement indicates at least two resource requirements, and the load state indicates load states of at least two resources; using a preset container deployment decision model, performing a decision on container deployment for the resource requirement data and the state data to obtain a target node; wherein the target node is a node for deploying the at least one to-be-deployed container, the container deployment decision model is a deep reinforcement learning agent, the deep reinforcement learning agent is configured to perform a container deployment decision based on a Markov decision process framework, and minimization of load balancing deviation of the node cluster is taken as an optimization target; wherein the load balancing deviation is a load balancing deviation of at least two resources.

2. The determination method according to claim 1, characterized in that, The step of obtaining the state data in response to receiving the user request comprises: sending a request for obtaining the state data to a preset data collection module; receiving the state data sent by the data collection module; wherein the state data at least includes utilization rate data of a target resource of any single node in the node cluster, mean data of utilization rate of the target resource in the node cluster, or standard deviation data of utilization rate of the target resource in the node cluster; wherein the target resource at least includes CPU resource and memory resource.

3. The determination method according to claim 1, characterized in that, The container deployment decision model is a deep reinforcement learning agent based on a Dueling DQN algorithm, and the Markov decision process framework at least includes a state space, an action space, a reward function, and a state transition function; wherein the state space is constructed based on the resource requirement data and the state data; the action space is used to characterize a set of node numbers in the node cluster, and an action output is a target node number in the node cluster; the reward function is modeled based on the optimization target and is used to calculate a reward value according to a deployment result to guide training; and the state transition function is used to update the load state of the node cluster to generate a next state in response to deploying the to-be-deployed container to the target node. The step of using the preset container deployment decision model to perform the decision on container deployment for the resource requirement data and the state data to obtain the target node comprises: constructing the state space based on the resource requirement data and the state data; inputting the state space into the deep reinforcement learning agent based on the Dueling DQN algorithm to obtain the target node number output by the action space.

4. The determination method according to claim 3, characterized in that, The optimization objective of the deep reinforcement learning agent based on the Dueling DQN algorithm is modeled to include an optimization objective function, a first constraint, a second constraint, and a third constraint; wherein the optimization objective function is represented as minimizing a load balancing deviation metric by optimizing deployment decision variables, the load balancing deviation metric being a total deviation of load states of a node cluster from an average load state of the node cluster; The first constraint is used to represent a constraint that CPU resources on each node are not overused; The second constraint is used to represent a constraint that memory resources on each node are not overused; The third constraint is used to represent a constraint that each to-be-deployed container can only be deployed on one node.

5. The determination method according to claim 3, characterized in that, The model structure of the deep reinforcement learning agent based on the Dueling DQN algorithm includes: An input layer for receiving data in the state space; A shared feature extraction layer including at least one fully connected hidden layer, the activation function of the fully connected hidden layer being a ReLU function, for extracting shared features of node cluster load states; A branch processing layer including a state value function branch and an action advantage function branch; wherein the state value function branch is connected to the shared feature extraction layer and outputs a state value function representing the current overall cluster state value, and the action advantage function branch is connected to the shared feature extraction layer and outputs an advantage value function representing the action advantage value of each node; An output merging layer for receiving the state value function and the action advantage value, and outputting Q values of each node based on the state value function and the action advantage value, and then normalizing the Q values through a softmax function to generate a scheduling action probability distribution of all nodes; A decision layer for selecting a target node number based on the scheduling action probability distribution, wherein the node number corresponding to the highest probability is the output action.

6. A method of deploying a container, characterized by, It includes: Determining a target node corresponding to a to-be-deployed container; wherein a target node determination method according to any one of claims 1-5 is used to determine the target node corresponding to the to-be-deployed container; Deploying the to-be-deployed container to the target node.

7. An update method of a container deployment decision model characterized by comprising: It includes: Deploying the to-be-deployed container to the target node; wherein a target node determination method according to claim 6 is used to determine the target node corresponding to the to-be-deployed container; Obtaining a deployment result; Calculating a reward value through a reward function in response to the deployment result; Updating network parameters of the deep reinforcement learning agent based on the reward value.

8. The updating method according to claim 7, characterized by, The step of calculating a reward value through a reward function in response to the deployment result includes: If the container deployment fails, the reward value is a first preset value; wherein the first preset value is a negative value; If the deployment is successful and the load balancing deviation of the cluster after deployment is smaller than that before deployment, the reward value is a second preset value; wherein the second preset value is a positive value greater than 0 and less than 1; If the deployment is successful and the post-deployment cluster load balancing deviation is greater than or equal to the pre-deployment, the reward value is the pre-deployment load balancing deviation minus the post-deployment load balancing deviation; wherein the load balancing deviation is calculated by the sum of the Euclidean distance of node resource utilization and cluster average utilization; wherein the load balancing deviation is constrained in the numerical interval of 0 to 1.

9. The updating method according to claim 8, characterized in that, The step of updating the network parameters of the deep reinforcement learning agent based on the reward value comprises: In the training phase, the deployment node is selected by a directional exploration strategy; wherein the step of selecting the deployment node by the directional exploration strategy in the training phase comprises: in response to entering an action exploration phase, obtaining the CPU available utilization and the memory available utilization of each node in the node cluster; based on the CPU available utilization and the memory available utilization, calculating the comprehensive resource available rate of each node; based on a preset directional greedy rule, selecting the node with the maximum comprehensive resource available rate as the deployment action.

10. An electronic device, comprising: Comprise: a memory, and one or more processors in communication connection with the memory; The memory has instructions executable by the one or more processors, and the instructions are executed by the one or more processors to make the one or more processors implement the determination method of a target node according to any one of claims 1 to 5, or implement the container deployment method according to claim 6, or implement the update method of the container deployment decision model according to any one of claims 7 to 9.

Images