A container orchestration method and system based on reinforcement learning

Abstract

The application relates to a container orchestration method and system based on reinforcement learning, which comprises the following steps: periodically collecting multi-dimensional running data of cluster nodes and container services to generate time sequence observation sequences; after data normalization, calculating resource utilization, resource balance and service quality indexes to construct three-level cluster state vectors of cluster level, node level and task level; inputting the state vectors into a pre-trained reinforcement learning scheduling model, filtering invalid scheduling actions through a dynamic action mask mechanism, outputting an optimal scheduling strategy in the valid action subspace, and completing container life cycle orchestration. Through the cooperative design of multi-dimensional state representation, marginal change reward and embedded action mask, the application solves the problems of poor dynamic adaptability, low decision accuracy and excessive invalid exploration of existing scheduling schemes, can significantly improve the cluster resource utilization and service quality stability, and is suitable for heterogeneous dynamic container cluster scenarios.

Description

Technical Field

[0001] This invention relates to the field of container orchestration and intelligent scheduling technology, specifically to a container orchestration method and system based on reinforcement learning. Background Technology

[0002] Container technology, with its lightweight, portable, and fast startup characteristics, has become the mainstream carrier for application deployment in cloud-native scenarios. Kubernetes, as the most widely used container orchestration system, undertakes core functions such as cluster resource management, container scheduling and deployment, and service operation and maintenance. The quality of its scheduling strategy directly determines the cluster resource utilization and application service quality.

[0003] Existing container scheduling technologies can be mainly divided into the following three categories, all of which have significant limitations in dynamic cluster environments: Kubernetes native static scheduling employs a two-stage architecture of "pre-selection + optimization." The pre-selection stage filters out nodes that do not meet the criteria based on fixed rules, while the optimization stage selects the optimal node using preset scoring rules. This solution relies on predefined static rules and cannot adapt to dynamic scenarios such as sudden traffic surges, service link reconstruction, and node failures. The scheduling strategy response is lagging, making it difficult to cope with business environments with frequent load fluctuations.

[0004] Mathematical programming and heuristic scheduling: Mathematical models such as integer linear programming and analytic hierarchy process can obtain better placement schemes in static cluster scenarios, but they cannot adapt to the dynamic changes in container load, and the scheduling accuracy drops significantly with environmental fluctuations; Heuristic methods such as ant colony algorithm and genetic algorithm can handle multi-objective optimization problems, but they have drawbacks such as high computational complexity, easy to get trapped in local optima, difficulty in modeling inter-container dependencies and performance interference, and easy to cause problems such as resource fragmentation and uneven node load.

[0005] Conventional reinforcement learning scheduling schemes generally suffer from three shortcomings: First, the granularity of state representation is coarse, often using only the instantaneous utilization of node CPU and memory as state inputs, failing to characterize dynamic information such as resource balance, service dependency, and load trends. Second, the reward mechanism is simply designed, often directly using the absolute values ​​of resource utilization and response time as reward signals, which is greatly affected by historical states and cannot accurately quantify the actual contribution of a single scheduling action. Third, there is no effective constraint on the action space, causing the model to blindly explore the entire action space, easily resulting in illegal scheduling actions such as insufficient resources and overload, which reduces training convergence efficiency and threatens the stability of cluster operation.

[0006] In summary, existing container scheduling technologies generally suffer from technical defects in dynamic heterogeneous cluster scenarios, such as insufficient state awareness accuracy, poor scheduling adaptability, low decision-making efficiency, and difficulty in guaranteeing service quality, and cannot achieve a dynamic optimal balance between resource efficiency and service performance. Summary of the Invention

[0007] A primary objective of this invention is to overcome at least one of the aforementioned deficiencies by providing a container orchestration method and system based on reinforcement learning. This method is applicable to container cluster scenarios with heterogeneous resource constraints and dynamic load fluctuations. It can improve cluster resource utilization and load balancing while ensuring container service quality, thereby achieving intelligent and adaptive container lifecycle orchestration.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: This invention provides a container orchestration method based on reinforcement learning, applied to a container cluster consisting of multiple compute nodes. The method includes: Step S1: Cluster running status time series collection: Periodically collect the resource status of each computing node and the running status of each container service in the container cluster, obtain real-time CPU and memory usage data of nodes, container resource consumption time series data, container service end-to-end response time data and inter-container dependency data, in order to generate a time series observation sequence. Step S2, Multi-dimensional Cluster State Modeling and Reward Calculation: Normalize the original data in the time-series observation sequence, and calculate the resource utilization rate, node resource balance, and container service quality index of each computing node; Based on the resource utilization rate, resource balance, and service quality index, construct a cluster state vector containing cluster-level features, node-level features, and task-level features, as the state input of the reinforcement learning model; Step S3, Reinforcement Learning Scheduling Decision with Action Mask: Input the cluster state vector into the pre-trained reinforcement learning scheduling model, and filter the original action space output by the model in real time through a dynamic action mask mechanism to remove invalid scheduling actions that do not meet resource constraints, load balancing constraints and service quality constraints. Select the optimal scheduling strategy from the filtered valid action subspace, and complete the deployment and operation adjustment of container instances on the computing nodes according to the scheduling strategy.

[0009] According to one embodiment of this application, the data collected in step S1 further includes: CPU and memory requirement thresholds for container services, gradient of container resource usage rate changes, amount of allocable resource fragments for computing nodes, and node resource competition index.

[0010] According to one embodiment of this application, in step S2, the calculation method for the resource utilization rate, node resource balance, and container service quality index is as follows: Resource utilization rate: Calculate the ratio of the usage of each type of resource on a single computing node to the total amount of the corresponding resources, and take the arithmetic mean of the ratios of each type of resource to obtain the node resource utilization rate; Resource balance: Calculate the standard deviation of the utilization rate of various resources on a single computing node, and subtract the standard deviation from 1 to obtain the node resource balance. Container service quality index: The container service quality index is obtained by calculating the ratio of the actual end-to-end response time of the container service to the preset target response time.

[0011] According to one embodiment of this application, in step S2, the features of each level of the cluster state vector are as follows: Cluster-level characteristics include: overall cluster resource utilization, average cluster resource utilization, and standard deviation of cluster resource utilization. Node-level characteristics include: single resource utilization rate of each computing node, average resource utilization rate of nodes, standard deviation of node resource utilization rate, and task occupancy rate of nodes; Task-level characteristics include the resource request rate of container services and the task dependency rate of container services.

[0012] According to one embodiment of this application, the reinforcement learning scheduling model used in step S3 includes the following training steps: The training environment is constructed by inputting the sample cluster state vector and outputting the scheduling action after filtering through the action masking mechanism. The instant reward value is calculated based on the changes in cluster state before and after the scheduling action is executed; The network parameters of the reinforcement learning scheduling model are updated using the instant reward value as a feedback signal, and the model is iteratively trained until it converges. The instant reward value is obtained by weighting the changes in the average resource balance of the cluster, the average resource utilization of the cluster, and the average number of replicas of the container service by a preset weighting coefficient. The reciprocal of the change in the average number of replicas of the container service is used in the weighting calculation.

[0013] According to one embodiment of this application, the change in the average resource balance of the cluster is the difference between the average resource balance of all computing nodes after the scheduling action is executed and the average resource balance of all computing nodes before the execution. The change in the average resource utilization of the cluster is the difference between the average resource utilization of all computing nodes after the scheduling action is executed and the average resource utilization of all computing nodes before the execution. The change in the average number of container service replicas is the difference between the average number of all container service replicas after the scheduling action is executed and the average number of all container service replicas before the execution.

[0014] According to one embodiment of this application, in step S3, the execution process of the dynamic action masking mechanism filtering the original action space output by the model in real time is as follows: A masking process is inserted between the policy network output layer and the probability normalization layer of the reinforcement learning scheduling model; Based on the current cluster's node remaining resource threshold, node load limit, and cross-node communication overhead constraints, identify scheduling actions that do not meet the constraints; The output weights corresponding to the scheduling actions that do not meet the constraints are compressed to a preset minimum value, so that the selection probability of the scheduling actions that do not meet the constraints after probability normalization approaches zero.

[0015] According to one embodiment of this application, in step S3, the reinforcement learning scheduling model adopts a near-end policy optimization algorithm, including an Actor policy network and a Critic value network. The Actor policy network is used to receive the cluster state vector and output the action probability distribution of container scheduling. The Critic value network is used to output the advantage function of the current cluster state to guide the parameter updates of the Actor policy network.

[0016] According to one embodiment of this application, in step S3, the deployment and operation adjustment of container instances on compute nodes are completed according to the scheduling policy, covering the entire lifecycle of the container service, including the following execution process: During the initial container creation phase, the initial deployment nodes are determined based on the container startup resource requirements and node resource distribution. During the container runtime phase, a rescheduling strategy is generated based on container resource usage time-series data, real-time service quality data, and container dependencies. By combining the horizontal and vertical scaling mechanisms of the container orchestration platform, the resource quotas and number of replicas of containers can be adjusted.

[0017] In particular, the present invention also provides a reinforcement learning-based container orchestration system applied to a container cluster containing multiple compute nodes, the system comprising: The data acquisition module is used to periodically collect the resource status of each computing node and the running status of each container service, and generate time-series observation sequences. The state calculation module is used to normalize the time-series observation sequence, calculate the resource utilization rate, node resource balance and container service quality indicators of each computing node, and construct a cluster state vector containing cluster-level, node-level and task-level features. The reinforcement learning scheduling module has a built-in pre-trained reinforcement learning scheduling model and action masking unit. It is used to receive the cluster state vector, filter invalid scheduling actions in the original action space through the action masking unit, and output the scheduling strategy in the effective action subspace. The scheduling and execution module is used to perform deployment, migration, and scaling operations of container instances according to the scheduling policy.

[0018] In particular, the present invention also provides a computing device including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor executes the computer program to implement the method described above.

[0019] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above. The computer-readable storage medium is preferably a non-volatile readable storage medium.

[0020] Compared with existing technologies, the advantages and beneficial effects of the reinforcement learning-based container orchestration method and container orchestration system proposed in this patent application are as follows: This application constructs a three-level multi-dimensional state vector at the cluster, node, and task levels, incorporating features such as resource balance, task dependency rate, and load time-series trends. This comprehensively characterizes the global distribution of the cluster, the node load structure, and the characteristics of task requirements, providing a high signal-to-noise ratio environment input for reinforcement learning decision-making and effectively improving the accuracy of scheduling decisions in dynamic scenarios. In addition, by embedding dynamic action masks between the policy network output and probability normalization, illegal actions are automatically eliminated based on real-time cluster constraints. This achieves deep linkage between constraint logic and the decision-making process, reducing ineffective exploration in reinforcement learning, lowering training costs, and avoiding risks such as resource overload and service degradation from a mechanism perspective, thereby improving the stability and controllability of the scheduling system.

[0021] Furthermore, this application uses the marginal change in state before and after scheduling to calculate the immediate reward, rather than the traditional absolute value reward. This can accurately quantify the actual optimization contribution of a single scheduling action, eliminate the interference of historical state on the reward signal, and cover multiple objectives such as resource utilization, load balancing, and replica cost. It can be flexibly adapted to different business scenarios through weight coefficients, effectively accelerate the model convergence speed, and improve the accuracy of strategy optimization. Attached Figure Description

[0022] The following sections will describe some specific embodiments of the invention in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings: Figure 1 This is a schematic diagram of the main process of a container orchestration method based on reinforcement learning according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the workflow of a container orchestration method based on reinforcement learning according to another embodiment of the present invention; Figure 3 This is a structural diagram of a computing device according to another embodiment of this application; Figure 4 This is a computer-readable storage medium structure diagram according to another embodiment of this application. Detailed Implementation

[0023] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0025] Example 1: This embodiment describes a container orchestration method based on reinforcement learning, applied to a container cluster consisting of multiple compute nodes, such as... Figure 1 As shown, the method includes: S1. Cluster running status timing data collection The system periodically collects data on the resource status of each compute node in the container cluster and the running status of each container service, obtaining real-time CPU and memory usage data, container resource consumption time-series data, container service end-to-end response time data, and inter-container dependency data to generate a time-series observation sequence.

[0026] Furthermore, the data collected in step S1 also includes: CPU and memory requirement thresholds for container services, gradient of container resource usage rate changes, amount of allocable resource fragments for computing nodes, and node resource contention indicators, providing data support for fine-grained scheduling decisions.

[0027] S2, Multi-dimensional Cluster State Modeling and Reward Calculation The original data in the time-series observation sequence are normalized, and the resource utilization rate, node resource balance, and container service quality index of each computing node are calculated respectively. Based on the resource utilization rate, resource balance, and service quality index, a cluster state vector containing cluster-level features, node-level features, and task-level features is constructed as the state input of the reinforcement learning model.

[0028] The calculation methods for each core indicator are as follows: Resource utilization rate: Calculate the ratio of the usage of each type of resource on a single computing node to the total amount of the corresponding resources, and take the arithmetic mean of the ratios of each type of resource to obtain the node resource utilization rate, which is used to characterize the overall resource stress of the node; Resource balance: Calculate the standard deviation of the utilization rate of various resources on a single computing node, and subtract the standard deviation from 1 to obtain the node resource balance, which is used to measure the degree of balance of the utilization of various resources within the node. The larger the value, the more balanced the resource utilization. Container service quality metrics: The ratio of the actual end-to-end response time of a container service to the preset target response time is used to quantify application performance.

[0029] Furthermore, the features at each level of the cluster state vector are specifically as follows: Cluster-level features include overall cluster resource utilization, average cluster resource utilization, and standard deviation of cluster resource utilization, which are used to characterize the global resource distribution of the cluster. Node-level features include the single resource utilization rate of each computing node, the average resource utilization rate of the node, the standard deviation of the node resource utilization rate, and the node task occupancy rate, which are used to characterize the resource load structure of a single node. Task-level characteristics: These include the resource request rate and task dependency rate of container services, used to characterize the resource requirements and related features of container services to be scheduled.

[0030] Furthermore, the reinforcement learning scheduling model is trained through the following steps: A training environment is constructed, and the input sample cluster state vector is filtered by an action masking mechanism to output the scheduling action. The instant reward value is calculated based on the change in cluster state before and after the execution of the scheduling action. The network parameters of the reinforcement learning scheduling model are updated with the instant reward value as a feedback signal, and the model is iteratively trained until it converges.

[0031] The instant reward value is obtained by weighting the changes in the average resource balance of the cluster, the average resource utilization of the cluster, and the average number of replicas of the container service by a preset weighting coefficient. The reciprocal of the change in the average number of replicas of the container service is used in the weighting calculation.

[0032] The calculation rules for the three types of changes are as follows: Cluster average resource balance change: The difference between the average resource balance of all computing nodes after the scheduling action is executed and the average resource balance of all computing nodes before the execution. Change in average cluster resource utilization: The difference between the average resource utilization of all computing nodes after the scheduling action is executed and the average resource utilization of all computing nodes before the execution. Change in average number of container service replicas: The difference between the average number of replicas of all container services after the scheduling action is executed and the average number of replicas of all container services before the execution.

[0033] S3, Reinforcement Learning Scheduling Decision with Action Mask The cluster state vector is input into the pre-trained reinforcement learning scheduling model. The original action space output by the model is filtered in real time through a dynamic action masking mechanism to remove invalid scheduling actions that do not meet resource constraints, load balancing constraints, and service quality constraints. The optimal scheduling strategy is selected from the filtered valid action subspace, and the deployment and operation adjustment of container instances on the computing nodes are completed according to the scheduling strategy.

[0034] Furthermore, the execution process of the dynamic action masking mechanism is as follows: A masking process is inserted between the policy network output layer and the probability normalization layer of the reinforcement learning scheduling model; based on the current cluster's node remaining resource threshold, node load limit, and cross-node communication overhead constraints, scheduling actions that do not meet the constraints are identified; the output weights corresponding to the scheduling actions that do not meet the constraints are compressed to a preset minimum value, so that the selection probability of the scheduling actions that do not meet the constraints after probability normalization approaches zero.

[0035] Furthermore, the reinforcement learning scheduling model employs the Proximal Policy Optimization (PPO) algorithm, which includes an Actor policy network and a Critic value network. The Actor policy network receives the cluster state vector and outputs the action probability distribution for container scheduling. The Critic value network outputs the advantage function of the current cluster state to guide the parameter update of the Actor policy network.

[0036] Furthermore, the scheduling policy covers the entire lifecycle of the container service, and the execution process includes: During the initial container creation phase, the initial deployment nodes are determined based on the container startup resource requirements and node resource distribution. During the container operation phase, a rescheduling strategy is generated based on container resource usage time-series data, real-time service quality data, and container dependencies. Combined with the horizontal and vertical scaling mechanisms of the container orchestration platform, the container resource quotas and replica counts are adjusted to form a closed-loop scheduling optimization.

[0037] Example 2: This embodiment presents a reinforcement learning-based container orchestration method applied to a Kubernetes-based container cluster environment. The cluster contains M worker nodes, each configured with K types of computing resources (K = 2 in this embodiment, representing CPU and memory respectively). N container services are deployed within the cluster, each corresponding to several pod container instances. A pod is the smallest execution unit for container scheduling. The method described in this embodiment can be deployed on the Kubernetes control plane as a custom scheduler plugin, replacing or collaborating with the native scheduler.

[0038] Similarly, as in Example 1, three processing steps are employed, which may include: Step S1, time-series acquisition of cluster running status; Step S2, multi-dimensional cluster state modeling and reward calculation; Step S3, reinforcement learning scheduling decision with action masking. The specific working process in this example is as follows: Figure 2 As shown, it includes: Step S1: Cluster running status time series collection This step involves data collection through integration with cluster monitoring components. Node-side data is obtained via kubelet and MetricsServer, container runtime data is obtained via cAdvisor, and service dependency and response time data is obtained via service mesh (such as Istio) or call chain monitoring components.

[0039] Acquisition period: set to 15 seconds / time. All acquired data are aligned with a unified timestamp to form a continuous time series observation sequence, which is stored in the time series database.

[0040] Basic data collection metrics: Node dimension: Real-time CPU utilization, real-time memory utilization, and available resource balance of each computing node; Container dimension: CPU and memory resource consumption of each container instance, and time-series curves of resource consumption changes; Service dimension: end-to-end response time of each container service, and call dependencies between container services.

[0041] Extended data collection metrics include: CPU and memory requirement thresholds for container services (request / limit), gradient of container resource usage rate changes (the change in resource consumption per unit time), amount of allocable resource fragments on compute nodes, and node resource contention (the ratio of the number of tasks to be run on a node to available resources), which are used to support fine-grained scheduling decisions.

[0042] The collected raw data is cleaned and timestamped to form a structured time-series observation sequence, which is then output to the state calculation module.

[0043] Step S2: Multi-dimensional cluster state modeling and reward calculation This step is divided into four sub-steps: data preprocessing, core indicator calculation, cluster state vector construction, and reward function calculation.

[0044] During data preprocessing, normalization is performed to map all original indicators to the [0,1] interval, eliminating the dimensional differences between different resource types. The calculation formula is as follows: in, These are the original index values. , These are the historical minimum and maximum values ​​for this indicator.

[0045] When calculating core metrics, node resource utilization rate is generally included. Node resource balance Container service quality indicators The calculation of these three factors includes: Node resource utilization The overall resource utilization rate of the m-th computing node at time t is represented by the arithmetic mean of the utilization rates of various resources within the node. in, Let m be the utilization rate of the k-th type of resource at time t. The formula is as follows: In the formula, Let m be the total capacity of the k-th type of resource. Let be the remaining amount of the k-th type of resource at node m at time t.

[0046] Node resource balance This represents the degree of balance in the utilization of various resources at time t for the m-th computing node, calculated using the standard deviation of resource utilization rates. The resource balance value ranges from [0,1]. The closer the value is to 1, the more balanced the utilization of various resources within the node, and the less likely a single resource bottleneck will occur.

[0047] Container service quality metrics The performance of the nth container service at time t is represented by the following formula: in, The actual end-to-end response time for container n at time t. This is a preset target response time threshold. The smaller the value of this container service quality indicator, the better the service quality.

[0048] When constructing the three-level cluster state vector, the core indicators obtained from the above calculations are integrated into the cluster state vector. , by cluster-level features Node-level features Task-level features It is composed of three parts: in, Cluster-level features It includes three types of features used to characterize the global resource state of the cluster: The overall resource utilization vector of the cluster is composed of the overall utilization of various resources within the cluster, and has a dimension of K. Cluster resource average utilization rate, the arithmetic mean of resource utilization rates of all nodes; The standard deviation of cluster resource utilization, which is the standard deviation of resource utilization across all nodes, reflects the overall load balancing of the cluster.

[0049] Node-level features It includes four types of features to characterize the load state of each computing node. The dimensions are related to the number of nodes M, namely: The utilization rate of a single node's resources, which is the utilization rate of various resources of each node, with a dimension of M×K; Average resource utilization rate of nodes, overall resource utilization rate of each node. 、; Standard deviation of node resource utilization rate; standard deviation of various resource utilization rates within each node. Node task utilization rate is the proportion of Pods running on each node to the total number of Pods in the cluster.

[0050] Task-level features It includes two types of features used to characterize the attributes of the container service to be scheduled: Resource request rate is the ratio of the number of requests for various types of resources by the container services to be scheduled to the total remaining resources in the cluster. Task dependency ratio is the ratio of the number of other services that the container service to be scheduled depends on to the number of dependent services that have been deployed on the same node. It is used to measure the risk of cross-node communication overhead.

[0051] The training of the reinforcement learning scheduling model incorporates the calculation of the input reward function and adopts a mode combining offline training and online inference. During the training phase, the model parameters are iteratively optimized through interaction with the simulation cluster environment. Specifically, in this embodiment, the process of training the reinforcement learning scheduling model includes calculating state changes, calculating the immediate reward function, setting training constraints, and training iterations.

[0052] (1) When calculating the state change, the difference in cluster state before and after a single scheduling action is used as the change, which is divided into three categories: Cluster average resource balance change : Change in average resource utilization of the cluster : Change in the average number of replicas for container services : in The number of Pod replicas serving the nth container at time t.

[0053] (2) The formula for the instant reward function is as follows: In the formula , , The weight hyperparameter, in this embodiment, typically takes the value of [value]. =0.4、 =0.4、 =0.2, which can be adjusted according to business scenarios; the change in the number of replicas is taken as the reciprocal, so as to constrain the scale of replicas and reduce resource costs while ensuring service capabilities.

[0054] (3) Training constraints During training, three types of constraints must be met. Scheduling actions that do not meet these constraints will be masked by the action masking mechanism: Resource constraints: The remaining resources on a node are not less than the resource requests of the container service, i.e. ≥ ,in The number of requests for the k-th type of resource to serve n containers; Quality of Service Constraints: QoS metrics for container services must meet [the specified requirements]. ,in The preset minimum service quality threshold is set to 0.6 in this embodiment; Replica count constraint: Each container service must deploy at least one Pod replica, i.e. .

[0055] (4) Training iteration The training iteration process involves: inputting the sample cluster state vector into the model, filtering it through an action mask, and then outputting the scheduled action; executing the action and calculating the immediate reward; updating the model network parameters in reverse using the reward signal; repeating the above process until the reward value converges and stabilizes, thus completing model training.

[0056] Step S3: Reinforcement learning scheduling decision with action mask This step includes establishing the reinforcement learning model architecture, embedding and executing the dynamic action masking mechanism, and scheduling execution throughout the entire lifecycle, as detailed below: The reinforcement learning model architecture in this embodiment adopts the proximal policy optimization (PPO) algorithm and is implemented based on the Actor-Critic dual-network architecture, wherein: The input to the Actor policy network is the cluster state vector D. The network structure employs three fully connected layers, using ReLU activation, and has an output layer dimension of N×M, corresponding to the deployment decision logits values ​​of N container services across M nodes. The implemented function is to output a probability distribution of scheduling actions, i.e., a strategy, based on the input state. , For network parameters; The input to the Critic value network is the cluster state vector consistent with the Actor policy network. The network structure employs three fully connected layers, uses ReLU as the activation function, and outputs a single-valued state value. Its function is to evaluate the long-term value of the current state, calculating the generalized advantage estimate (GAE) as the advantage function. This guides the parameter updates of the Actor network. The advantage function is calculated using the following formula: This is the reward discount factor, which is set to 0.99 in this embodiment; For the GAE parameter, this example uses 0.95 to balance the bias and variance; The total loss function of the Proximal Policy Optimization (PPO) algorithm consists of three parts: pruning loss, value loss, and entropy loss. Among them, pruning loss limits the update range of the strategy to ensure training stability; value loss fits the value of the state; and entropy loss encourages the model to explore and avoid getting trapped in local optima.

[0057] The action mask unit is embedded between the Actor network output layer and the Softmax normalization layer, and the execution flow is as follows: Constraint Decision: Based on the current cluster state, the legality of each candidate scheduling action is verified one by one. The decision rules include: Resource constraints: if the remaining resources of a node are less than the resource request amount of the container to be scheduled, the action is deemed invalid. Load constraints: if the overall resource utilization rate of a node exceeds the preset upper limit (85% in this embodiment), it is determined to be an invalid action; Communication overhead constraints: if the container to be scheduled and its strongly dependent services are deployed across nodes and the estimated communication latency exceeds the threshold, the action is deemed invalid.

[0058] Masking operation: For actions deemed invalid, their corresponding logits values ​​are compressed to a preset minimum value (-10^9 in this embodiment); the logits values ​​of valid actions remain unchanged.

[0059] Probability normalization: Softmax operation is performed on the masked logits, and the output probability of invalid actions approaches 0, ensuring that the model makes decisions only in the valid action subspace.

[0060] The trained model is deployed as a custom Kubernetes scheduler, covering the entire lifecycle scheduling of container services. Initial deployment phase: When a Pod creation request is received, the scheduler collects the current cluster state and constructs a state vector, inputs it into the reinforcement learning model, and selects the node with the highest probability as the deployment node after filtering by action mask, and issues scheduling instructions to complete the Pod creation.

[0061] Runtime rescheduling phase: A global status check is triggered every 5 minutes, or an emergency rescheduling is triggered when the node load exceeds the threshold or the service QoS is lower than the threshold; a rescheduling strategy is generated based on the real-time status vector to migrate overloaded nodes and Pods with low QoS services to the optimal node to achieve dynamic load balancing.

[0062] Elastic scaling phase: Combining Kubernetes' horizontal Pod auto-scaling (HPA) and vertical Pod auto-scaling (VPA) mechanisms, the number of container service replicas and single Pod resource quotas are dynamically adjusted based on load time-series changes and service quality feedback. The scheduling strategy is synchronously adapted to the cluster state after scaling, forming a closed-loop optimization from collection to decision-making, to execution and feedback.

[0063] Example 3: This embodiment describes a container orchestration system based on reinforcement learning, applied to a container cluster containing multiple computing nodes. The system includes a data acquisition module, a state calculation module, a reinforcement learning scheduling module, and a scheduling execution module. The specific implementation and interaction relationships of each module are as follows: The data acquisition module consists of a node acquisition agent and a cluster aggregation unit. The node acquisition agent is deployed on each compute node in the form of a DaemonSet and connects to kubelet and cAdvisor to collect node and container data. The cluster aggregation unit is deployed on the control plane and connects to the service mesh component to collect service call and response time data. After aggregation, it generates a time series observation sequence and sends it to the state calculation module.

[0064] State Calculation Module: Receives time-series observation sequences, performs data normalization, calculates core indicators, and constructs state vectors. The generated cluster state vectors are then sent to the reinforcement learning scheduling module. During the model training phase, this module is also responsible for calculating the immediate reward value corresponding to the scheduling action and feeding it back to the model training unit.

[0065] The reinforcement learning scheduling module includes a model inference unit, an action masking unit, and a model training unit. During online scheduling, the model inference unit receives the state vector and outputs action logits, the action masking unit performs invalid action filtering, and finally outputs the optimal scheduling strategy. During the training phase, the model training unit iteratively updates the network parameters based on environmental feedback and reward values.

[0066] The scheduling and execution module interfaces with the Kubernetes API Server, translating scheduling policies into specific API calls to execute operations such as Pod creation, migration, scaling, and shrinking. After the operation is completed, the execution results and the latest cluster status are fed back to the data acquisition module, completing the closed loop.

[0067] Tests have verified that, in a test cluster containing 10 compute nodes and 30 container services, the scheduling scheme of this invention, compared with the native Kubernetes scheduling, improves the average resource utilization of the cluster by 18%, reduces the standard deviation of node load by 32%, reduces the average service response time by 15%, and improves the training convergence speed of reinforcement learning models by about 40%, effectively verifying the technical effectiveness of the scheme.

[0068] This application also provides a computing device, referring to... Figure 3 The computing device includes a memory 1120, a processor 1110, and a computer program stored in the memory 1120 and executable by the processor 1110. The computer program is stored in a space 1130 for program code in the memory 1120. When executed by the processor 1110, the computer program implements a method step 1131 according to the invention.

[0069] This application also provides a computer-readable storage medium. (See attached embodiments.) Figure 4 The computer-readable storage medium includes a storage unit for program code, the storage unit being provided with a program 1131' for performing the method steps according to the invention, the program being executed by a processor.

[0070] This application also provides a computer program product containing instructions. When the computer program product is run on a computer, it causes the computer to perform the method steps according to the invention.

[0071] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When a computer loads and executes the computer program instructions, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0072] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0073] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0074] In particular, the present invention also provides a computing device including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor executes the computer program to implement the method described above.

[0075] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above. The computer-readable storage medium is preferably a non-volatile readable storage medium.

[0076] The above embodiments are only for illustrating the technical concept and features of the present invention. Their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be used to limit the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A container orchestration method based on reinforcement learning, applied to a container cluster consisting of multiple computing nodes, characterized in that, The methods include: Step S1: Cluster running status time series collection: Periodically collect the resource status of each computing node and the running status of each container service in the container cluster, obtain real-time CPU and memory usage data of nodes, container resource consumption time series data, container service end-to-end response time data and inter-container dependency data, in order to generate a time series observation sequence. Step S2, Multi-dimensional Cluster State Modeling and Reward Calculation: Normalize the original data in the time-series observation sequence, and calculate the resource utilization rate, node resource balance, and container service quality index of each computing node; Based on the resource utilization rate, resource balance, and service quality index, construct a cluster state vector containing cluster-level features, node-level features, and task-level features, as the state input of the reinforcement learning model; Step S3, Reinforcement Learning Scheduling Decision with Action Mask: Input the cluster state vector into the pre-trained reinforcement learning scheduling model, and filter the original action space output by the model in real time through a dynamic action mask mechanism to remove invalid scheduling actions that do not meet resource constraints, load balancing constraints and service quality constraints. Select the optimal scheduling strategy from the filtered valid action subspace, and complete the deployment and operation adjustment of container instances on the computing nodes according to the scheduling strategy.

2. The container orchestration method based on reinforcement learning according to claim 1, characterized in that, The data collected in step S1 also includes: CPU and memory requirement thresholds for container services, gradient of container resource usage rate changes, amount of allocable resource fragments for compute nodes, and node resource competition index.

3. The container orchestration method based on reinforcement learning according to claim 1, characterized in that, In step S2, the calculation methods for the resource utilization rate, node resource balance, and container service quality indicators are as follows: Resource utilization rate: Calculate the ratio of the usage of each type of resource on a single computing node to the total amount of the corresponding resources, and take the arithmetic mean of the ratios of each type of resource to obtain the node resource utilization rate; Resource balance: Calculate the standard deviation of the utilization rate of various resources on a single computing node, and subtract the standard deviation from 1 to obtain the node resource balance. Container service quality index: The container service quality index is obtained by calculating the ratio of the actual end-to-end response time of the container service to the preset target response time.

4. The container orchestration method based on reinforcement learning according to claim 3, characterized in that, In step S2, the features of each level of the cluster state vector are as follows: Cluster-level characteristics include: overall cluster resource utilization, average cluster resource utilization, and standard deviation of cluster resource utilization. Node-level characteristics include: single resource utilization rate of each computing node, average resource utilization rate of nodes, standard deviation of node resource utilization rate, and task occupancy rate of nodes; Task-level characteristics include the resource request rate of container services and the task dependency rate of container services.

5. The container orchestration method based on reinforcement learning according to claim 1, characterized in that, The reinforcement learning scheduling model used in step S3 includes the following training steps: The training environment is constructed by inputting the sample cluster state vector and outputting the scheduling action after filtering through the action masking mechanism. The instant reward value is calculated based on the changes in cluster state before and after the scheduling action is executed; The network parameters of the reinforcement learning scheduling model are updated using the instant reward value as a feedback signal, and the model is iteratively trained until it converges. The instant reward value is obtained by weighting the changes in the average resource balance of the cluster, the average resource utilization of the cluster, and the average number of replicas of the container service by a preset weighting coefficient. The reciprocal of the change in the average number of replicas of the container service is used in the weighting calculation.

6. The container orchestration method based on reinforcement learning according to claim 5, characterized in that, The change in the average resource balance of the cluster is the difference between the average resource balance of all computing nodes after the scheduling action is executed and the average resource balance of all computing nodes before the execution. The change in the average resource utilization of the cluster is the difference between the average resource utilization of all computing nodes after the scheduling action is executed and the average resource utilization of all computing nodes before the execution. The change in the average number of container service replicas is the difference between the average number of all container service replicas after the scheduling action is executed and the average number of all container service replicas before the execution.

7. The container orchestration method based on reinforcement learning according to claim 1, characterized in that, In step S3, the execution process of the dynamic action masking mechanism to filter the original action space output by the model in real time is as follows: A masking process is inserted between the policy network output layer and the probability normalization layer of the reinforcement learning scheduling model; Based on the current cluster's node remaining resource threshold, node load limit, and cross-node communication overhead constraints, identify scheduling actions that do not meet the constraints; The output weights corresponding to the scheduling actions that do not meet the constraints are compressed to a preset minimum value, so that the selection probability of the scheduling actions that do not meet the constraints after probability normalization approaches zero.

8. The container orchestration method based on reinforcement learning according to claim 1, characterized in that, In step S3, the reinforcement learning scheduling model adopts a near-end policy optimization algorithm, which includes an Actor policy network and a Critic value network. The Actor policy network is used to receive the cluster state vector and output the action probability distribution of container scheduling. The Critic value network is used to output the advantage function of the current cluster state to guide the parameter updates of the Actor policy network.

9. The container orchestration method based on reinforcement learning according to claim 1, characterized in that, In step S3, the deployment and operation of container instances on compute nodes are adjusted according to the scheduling policy, covering the entire lifecycle of the container service. This includes the following execution process: During the initial container creation phase, the initial deployment nodes are determined based on the container startup resource requirements and node resource distribution. During the container runtime phase, a rescheduling strategy is generated based on container resource usage time-series data, real-time service quality data, and container dependencies. By combining the horizontal and vertical scaling mechanisms of the container orchestration platform, the resource quotas and number of replicas of containers can be adjusted.

10. A container orchestration system based on reinforcement learning, applied to a container cluster containing multiple computing nodes, characterized in that, The system includes: The data acquisition module is used to periodically collect the resource status of each computing node and the running status of each container service, and generate time-series observation sequences. The state calculation module is used to normalize the time-series observation sequence, calculate the resource utilization rate, node resource balance and container service quality indicators of each computing node, and construct a cluster state vector containing cluster-level, node-level and task-level features. The reinforcement learning scheduling module has a built-in pre-trained reinforcement learning scheduling model and action masking unit. It is used to receive the cluster state vector, filter invalid scheduling actions in the original action space through the action masking unit, and output the scheduling strategy in the effective action subspace. The scheduling and execution module is used to perform deployment, migration, and scaling operations of container instances according to the scheduling policy.

Images