Pod scheduling model construction method and device, equipment and medium

Abstract

The present application relates to the technical field of computer, and discloses a pod scheduling model construction method, device, equipment and medium, the method comprises the following steps: obtaining joint state data and environment data in the current training period in the current cycle; inputting the joint state data and the environment data into the pod scheduling model iterative training; when generating the joint reward corresponding to the current training period, determining whether the current cycle stops the iterative training according to the joint reward and / or the training period statistical value corresponding to the current training period; when determining that the current cycle stops the iterative training, entering the next cycle, and obtaining the joint state data and the environment data from the sample data pool corresponding to the next cycle to perform the iterative training on the pod scheduling model. The method can ensure that the model better allocates resources, reduces resource waste, and reduces scheduling delay.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and specifically to a method, apparatus, device, and medium for constructing a pod scheduling model. Background Technology

[0002] Kubernetes is a mainstream container orchestration technology that simplifies container deployment, scaling, and management. Container scheduling, a key feature of Kubernetes, enables automated deployment of containerized applications, improves program reliability and scalability, and ensures load balancing across the cluster. Therefore, the quality of container scheduling strategies directly impacts cluster availability and resource utilization.

[0003] Current container scheduling methods mostly calculate the status information of Pods and Nodes, and select the most suitable Node to complete the binding according to the scheduling policy. However, they ignore the mutual influence between the various Nodes, resulting in low scheduling efficiency and the problem of low resource utilization due to the imbalance between resource supply and demand. Summary of the Invention

[0004] In view of this, the present invention provides a method, apparatus, device and medium for constructing a pod scheduling model, in order to solve the problems of low scheduling efficiency and low utilization rate caused by the imbalance between resource supply and demand in the container scheduling methods of related technologies.

[0005] In a first aspect, the present invention provides a method for constructing a pod scheduling model, the method comprising:

[0006] Within the current training cycle in the current loop, joint state data and environment data are obtained from the sample data pool corresponding to the current loop. The joint state data consists of the remaining resources of each node in the K8s cluster, and the environment data includes multiple pods currently to be scheduled.

[0007] The joint state data and environment data obtained from the sample data pool corresponding to the current loop are input into the pod scheduling model for iterative training within the current loop.

[0008] When the pod scheduling model generates a joint reward corresponding to the current training cycle, it determines whether to stop iterative training in the current loop based on the joint reward and / or the training cycle statistics corresponding to the current training cycle. The joint reward includes the resource request reward of the target pod, the matching degree reward between the target pod and the candidate node, and the inter-node scheduling similarity reward corresponding to the influence of the candidate node and other nodes in the K8s cluster on the candidate node. The target pod is any one of multiple pods, and the candidate node is any one of multiple nodes.

[0009] When it is determined that the current loop will stop iterative training based on the joint reward and / or the training cycle statistics corresponding to the current training cycle, the next loop will be entered. Joint state data and environment data will be obtained from the sample data pool corresponding to the next loop to iteratively train the pod scheduling model until the number of loops reaches the preset number threshold, at which point iterative training will stop and the final pod scheduling model will be obtained.

[0010] The pod scheduling model construction method provided by this invention has the following advantages:

[0011] By considering the remaining resources of each node in the Kubernetes cluster, the model can allocate Pods more effectively, thereby maximizing the utilization of cluster resources. Through iterative training, the model continuously learns how to make optimal scheduling decisions based on the current cluster state and the needs of the Pods to be scheduled, thus improving Pod scheduling efficiency. Specifically, this application includes two nested loops: one sets different iterative training cycles within each loop, and the other includes multiple loops for iterative training, making the iterative training process richer and the final model more adaptable to different scenarios. The joint reward mechanism considers multiple factors, including the resource request reward of the target Pod, the matching degree reward between the Pod and the candidate node, and the influence reward of the candidate node on other nodes, helping the model make more comprehensive and reasonable scheduling decisions. In particular, the method in this application considers the influence of other nodes on a particular node, maximizing scheduling efficiency and resource utilization. Furthermore, the model can adjust its behavior based on the reward corresponding to the current training cycle and the statistical values ​​of the training cycle, enabling the model to adapt to constantly changing environments and needs. Through iterative training, the model can continuously learn new scheduling strategies from the sample data pool until a preset training count threshold is reached, ensuring that the model fully absorbs information from the data during training. Through multiple iterations of training and the acquisition of the final model, it can be ensured that the trained Pod scheduling model has high accuracy and reliability.

[0012] In summary, this method ensures better resource allocation and reduces resource waste. It quickly finds suitable nodes to schedule Pods, reducing scheduling latency. Through precise matching, it increases the probability of successful Pod startup and operation. The model can adapt to different workloads and cluster states.

[0013] In one optional implementation, the sample data in the sample data pool corresponding to each loop is obtained in the following ways:

[0014] Within the current loop, obtain the initial joint state data and environment data. The initial joint state data is the initial remaining resources of each node in the Kubernetes cluster within the current loop.

[0015] In the initial data generation cycle, the initial joint state data and environment data are input into the initial pod scheduling model to generate the joint action corresponding to the next data generation cycle, as well as the joint reward corresponding to the current data generation cycle.

[0016] The initial joint state data, environment data, joint action corresponding to the next data generation cycle, and joint reward corresponding to the current data generation cycle are combined into a quadruple as a sample data and added to the sample data pool corresponding to the current cycle.

[0017] In the i-th data generation cycle, based on the joint action corresponding to the i-th generation cycle and the joint state data corresponding to the (i-1)-th data generation cycle, the joint state data corresponding to the i-th data generation cycle is generated, where i is a positive integer greater than or equal to 2, and i increments by 1.

[0018] Input the joint state data and environment data corresponding to the i-th data generation cycle into the initial pod scheduling model to obtain the joint action corresponding to the (i+1)-th data generation cycle, as well as the joint reward corresponding to the i-th data generation cycle.

[0019] The quadruple consisting of the joint state data and environment data corresponding to the i-th data generation cycle, the joint reward corresponding to the i-th data generation cycle, and the joint action corresponding to the i+1-th data generation cycle is added as a sample data and added to the sample data pool corresponding to the current cycle.

[0020] The iteration stops when i equals the preset threshold, and all sample data in the sample data pool corresponding to the current loop are obtained.

[0021] Specifically, by dynamically determining the joint state in each data generation cycle, the model can capture the dynamic characteristics of resource allocation and Pod scheduling in a Kubernetes cluster over time. The joint actions and joint rewards generated in each data generation cycle simulate the decisions and outcomes during Pod scheduling, helping to train a scheduling model that better reflects reality. Through continuous data generation and iteration, the collected sample data becomes richer, encompassing various scenarios under different states and decisions, which is beneficial for the model's generalization ability. Training with pre-generated sample data improves model training efficiency because it eliminates the need for real-time interaction with the Kubernetes cluster to acquire data. By using sample data containing multiple states and decisions, the model can learn more accurate and comprehensive scheduling strategies. Because the training data covers various possible scheduling scenarios, it adapts to different workloads and cluster states. In this method, multiple sample data can be generated in the manner described above to avoid situations where model training fails to meet requirements due to insufficient sample data. Moreover, the generated sample data helps the model learn how to utilize cluster resources more effectively, reducing resource waste. Furthermore, pre-generated sample data reduces the need for real-time data collection, thereby shortening training time. This ensures that the final trained model can make scheduling decisions more quickly, improving Pod scheduling efficiency. Sample data helps the Pod scheduling model learn more accurately how to efficiently schedule Pods in a Kubernetes cluster, thereby improving the overall performance and resource utilization of the cluster.

[0022] In one optional implementation, when the pod scheduling model generates a joint reward corresponding to the current training cycle, it determines whether the current loop should stop iterative training based on the joint reward and / or the training cycle statistics corresponding to the current training cycle. Specifically, this includes:

[0023] When the reward value of the combined reward is greater than or equal to the preset reward threshold, the current loop is stopped from iterative training.

[0024] And / or,

[0025] When the statistical value of the training cycle corresponding to the current training cycle reaches the preset number of cycles, the current loop is stopped from iterative training.

[0026] Specifically, when the joint reward value generated by the model reaches or exceeds a preset reward threshold, the model can be considered to have learned an effective scheduling strategy. Stopping iterative training at this point avoids overfitting and saves computational resources. When the training epochs reach a preset number, iterative training stops regardless of the model's performance. This ensures the model completes training within a reasonable timeframe, preventing unnecessary long training periods. When a certain number of training epochs have been reached, the model has accumulated sufficient experience, and continued training may not significantly improve performance; therefore, stopping training ensures model stability. Setting explicit stopping conditions in this way helps understand when the model reaches performance requirements, improving the transparency and interpretability of the training process.

[0027] In one alternative implementation, the resource types of the remaining resources in each node include one or more of the following:

[0028] CPU resources, memory resources, disk resources, network resources, video memory resources, and GPU resources.

[0029] In one alternative implementation, when the resource type includes CPU resources, memory resources, disk resources, network resources, video memory resources, and GPU resources, the expression for the target pod to request resource rewards is as follows:

[0030] R score =w cpu *req cpu +w mem *req mem +w disk *req disk +w net *req net +w vimem *req vimem +w gpu *req gpu (Formula 1)

[0031] w cpu +w mem +w disk +w net +w vimem +w gpu =1 (Formula 2)

[0032] Among them, R score The reward value for requesting resources from the target pod, req cpu For remaining CPU resources, req mem For remaining memory resources, req disk For remaining disk resources, req net For remaining network resources, req vimemFor the remaining video memory resources, req gpu For the remaining GPU resources, w cpu w mem w disk w net w vimem and w gpu These are the weighting coefficients corresponding to each resource.

[0033] In one alternative implementation, when the resource types include CPU resources, memory resources, disk resources, network resources, video memory resources, and GPU resources, the matching degree reward between the target pod and the candidate nodes is represented by the following expression:

[0034]

[0035] Among them, R similar The reward value is awarded based on the degree of matching between the target pod and the candidate nodes. For joint actions, it means Pod i Scheduled to N j Let i be the i-th pod and j be the j-th node. and These represent the remaining CPU resources on the i-th pod and the remaining CPU resources on the j-th node, respectively. and These represent the remaining memory resources on the i-th pod and the remaining memory resources on the i-th node, respectively. and These represent the remaining disk resources on the i-th pod and the remaining disk resources on the i-th node, respectively. and These represent the remaining network resources on the i-th pod and the remaining network resources on the j-th node, respectively. and These represent the remaining video memory resources on the i-th pod and the remaining video memory resources on the j-th node, respectively. and These represent the remaining GPU resources on the i-th pod and the remaining GPU resources on the j-th node, respectively.

[0036] In one alternative implementation, the inter-node scheduling similarity reward is represented by the following expression:

[0037]

[0038] Among them, R attention The scheduling similarity reward between the nodes is given, where for the i-th node, Q is the scheduling similarity query vector, and K... TFor each node, a similarity feature representation is used to calculate the similarity with Q. V scores the actual similarity of each node and is used for the final weighted output. k The dimension of the scheduling similarity query vector.

[0039] In an optional implementation, the joint reward expression is shown below:

[0040] R = μ1R score +μ2R similar +μ3R attention (Formula 6)

[0041] Where R is the reward value of the joint reward, and μ1, μ2, and μ3 are the pre-configured weight coefficients.

[0042] Secondly, the present invention provides a pod scheduling model construction apparatus, the apparatus comprising:

[0043] The acquisition module is used to acquire joint state data and environment data from the sample data pool corresponding to the current cycle within the current training period. The joint state data consists of the remaining resources of each node in the K8s cluster, and the environment data includes multiple pods currently to be scheduled.

[0044] The training module is used to input the joint state data and environment data obtained from the sample data pool corresponding to the current loop into the pod scheduling model, so as to iteratively train the pod scheduling model within the current loop;

[0045] The processing module is used to determine whether to stop iterative training in the current loop when the pod scheduling model generates a joint reward corresponding to the current training cycle, based on the joint reward and / or the training cycle statistics corresponding to the current training cycle. The joint reward includes the resource request reward of the target pod, the matching degree reward between the target pod and the candidate node, and the inter-node scheduling similarity reward corresponding to the influence of the candidate node and other nodes in the K8s cluster on the candidate node. The target pod is any one of multiple pods, and the candidate node is any one of multiple nodes. When it is determined that the current loop should stop iterative training, the next loop is started.

[0046] The acquisition module is also used to acquire joint state data and environmental data from the sample data pool corresponding to the next cycle;

[0047] The training module is also used to iteratively train the pod scheduling model using joint state data and environment data obtained from the sample data pool corresponding to the next cycle, until the number of cycles reaches a preset threshold, at which point the iterative training stops and the final pod scheduling model is obtained.

[0048] The pod scheduling model construction device provided by this invention has the following advantages:

[0049] By considering the remaining resources of each node in the Kubernetes cluster, the model can allocate Pods more effectively, thereby maximizing the utilization of cluster resources. Through iterative training, the model continuously learns how to make optimal scheduling decisions based on the current cluster state and the needs of the Pods to be scheduled, thus improving Pod scheduling efficiency. Specifically, this application includes two nested loops: one sets different iterative training cycles within each loop, and the other includes multiple loops for iterative training, making the iterative training process richer and the final model more adaptable to different scenarios. The joint reward mechanism considers multiple factors, including the resource request reward of the target Pod, the matching degree reward between the Pod and the candidate node, and the influence reward of the candidate node on other nodes, helping the model make more comprehensive and reasonable scheduling decisions. In particular, this application considers the influence of other nodes on a particular node, maximizing scheduling efficiency and resource utilization. Furthermore, the model can adjust its behavior based on the reward corresponding to the current training cycle and the statistical values ​​of the training cycle, enabling the model to adapt to constantly changing environments and needs. Through iterative training, the model can continuously learn new scheduling strategies from the sample data pool until a preset training count threshold is reached, ensuring that the model fully absorbs information from the data during training. Through multiple iterations of training and the acquisition of the final model, it can be ensured that the trained Pod scheduling model has high accuracy and reliability.

[0050] In summary, this method ensures better resource allocation and reduces resource waste. It quickly finds suitable nodes to schedule Pods, reducing scheduling latency. Through precise matching, it increases the probability of successful Pod startup and operation. The model can adapt to different workloads and cluster states.

[0051] Thirdly, the present invention provides a computer device, including: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the pod scheduling model construction method of the first aspect or any corresponding embodiment described above.

[0052] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the pod scheduling model construction method of the first aspect or any corresponding embodiment thereof.

[0053] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the pod scheduling model construction method of the first aspect or any corresponding embodiment described above. Attached Figure Description

[0054] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0055] Figure 1 This is a flowchart illustrating a method for constructing a pod scheduling model according to an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram illustrating the structural principle of a pod scheduling model provided by the present invention;

[0057] Figure 3 This is a schematic diagram of the environmental data provided by the present invention;

[0058] Figure 4 This is a schematic diagram of the joint state data provided by the present invention;

[0059] Figure 5 The present invention is represented as Pod i Scheduled to N j A diagram illustrating the combined actions of running containerized applications;

[0060] Figure 6 This is a flowchart illustrating another method for constructing a pod scheduling model provided in an embodiment of the present invention;

[0061] Figure 7 This is a schematic diagram of the overall process of production sample data provided by the present invention;

[0062] Figure 8 This is a schematic diagram of the overall process of a pod scheduling model construction method provided by the present invention;

[0063] Figure 9 This is a structural block diagram of a pod scheduling model construction device provided in an embodiment of the present invention;

[0064] Figure 10 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] To address the aforementioned problems, this invention provides a pod scheduling embodiment. It should be noted that the steps shown in the flowcharts in the accompanying drawings can be executed in a computer system (computer device) including a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0067] This embodiment provides a method for constructing a pod scheduling model, which can be used in the aforementioned terminal devices, such as mobile phones and tablets. Figure 1 This is a flowchart illustrating a method for constructing a pod scheduling model provided in an embodiment of the present invention.

[0068] Before introducing the method steps of the embodiments of this application, the terms that may be involved in the embodiments of this application will be explained first, as detailed below:

[0069] Reinforcement Learning (RL): An agent interacts with the environment to obtain reward and punishment information to adjust its search strategy to maximize cumulative rewards and ultimately achieve the learning objective.

[0070] Multi-Agent Reinforcement Learning (MARL) is an extension of reinforcement learning in which multiple agents acquire information and optimize policies by observing other agents and environmental dynamics, thereby maximizing the performance of the overall system.

[0071] Container scheduling: The process of allocating containers to appropriate host nodes based on factors such as container resource requirements, scheduling strategies, and node resource status, while ensuring high availability and load balancing of nodes.

[0072] The method steps in the embodiments of this application are described below, for details please refer to: Figure 1 As shown, the process includes the following steps:

[0073] Step S101: In the current training cycle within the current loop, obtain joint state data and environment data from the sample data pool corresponding to the current loop.

[0074] The joint state data consists of the remaining resources of each node in the Kubernetes cluster, while the environment data includes the multiple pods currently awaiting scheduling.

[0075] Specifically, the container scheduling process consists of a list of Pods to be scheduled, P = (Pod1, Pod2, Pod3, ..., Pod4). i ,…,Pod m The cluster consists of a complex, dynamic multi-agent system composed of factors such as the remaining resources of currently available Nodes. Therefore, the process of a "single-agent" Node binding a Pod to run a containerized application can be extended to multi-agent collaboration. In this process, each agent (Node) needs to consider its interaction with the dynamic environment and its mutual influence with other agents, which can generally be described as (N, S, A, P). sa ,R,γ), where N=(N1,N2,N3,…,N i ,…,N n ) represents N nodes; S, A, and R represent the current joint state, action, and reward of the multi-agent system, respectively; P sa The probability of a multi-agent system executing an action from a joint state s to s_ is represented by p(s_|s,a); γ is the reward decay coefficient, which is between 0 and 1, and the larger the value, the more important future rewards are.

[0076] In the embodiments of this application, multiple loops may be included, and multiple cycles of iterative training may be performed within each loop.

[0077] Before performing iterative training, joint state data and environment data need to be obtained from the sample data pool corresponding to the current loop. The environment data consists of multiple pods in the list of pods to be scheduled. The number of pods corresponding to each loop can be the same or different. Specifically, which pods are the same or different can also be different.

[0078] Step S102: The joint state data and environment data obtained from the sample data pool corresponding to the current loop are input into the pod scheduling model for iterative training of the pod scheduling model within the current loop.

[0079] Step S103: When the pod scheduling model generates a joint reward corresponding to the current training cycle, determine whether to stop iterative training in the current loop based on the joint reward and / or the training cycle statistics corresponding to the current training cycle.

[0080] Specifically, in an optional example, the pod scheduling model can be a reinforcement learning model. This reinforcement learning model is built upon the Actor-Critic (AC) algorithm, using the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. In this model, the action-state information of all Node agents is input into the evaluation network, and each Node agent executes its policy through its own policy network, resulting in strong robustness. However, considering that this model cannot allocate the joint reward to each agent according to confidence (i.e., there is a "confidence allocation" problem), an attention mechanism is introduced. This mechanism differentiates the influence between agents, thereby achieving the goal of allocating the joint reward to different agents according to confidence. Therefore, the final model is MADADPG (Multi-Agent Deep Attention Deterministic Policy Gradient).

[0081] For details on the specific pod scheduling model, please refer to [link / reference]. Figure 2 As shown, Figure 2 The diagram illustrates that the scheduling model comprises multiple sub-models. These sub-models interact with each other through an attention mechanism, and this interaction is incorporated into the joint reward value generated by each sub-model. For example... Figure 2 The system defines two sub-models: one for node N1 and another for node Nn. Taking the N1 sub-model as an example, Actor1 generates the action corresponding to the current moment based on the current reward value and updates the state of node N1. Critic1 generates a reward value by calculating the similarity information of all node states and actions to evaluate the merits of the current scheduling strategy. All sub-models update their corresponding joint state and joint action at the current moment based on the above principles. The joint reward includes a resource request reward for the target pod, a matching reward between the target pod and candidate nodes, and an influence reward corresponding to the influence of candidate nodes on other nodes in the Kubernetes cluster. The target pod can be any one of multiple pods, and the candidate node can be any one of multiple nodes.

[0082] For details, see Figure 3 As shown, Figure 3 The diagram illustrates the environment data. In an optional example, the Pod list P = (Pod1, Pod2, Pod3, ..., Pod...). i , ..., Pod mThe types of resources a pod can request include CPU, memory, disk, and network. However, to meet the diverse resource needs of a cluster, GPUs and video memory can also be included in the scheduling criteria. The resource request of the i-th container pod can be described as a resource tuple vector. in This indicates the request status of each resource in the i-th Pod. cpu, mem, disk, net, vimem, and gpu represent the CPU, memory, disk, network, video memory, and GPU metrics, respectively.

[0083] Figure 4 The diagram illustrates the joint state data. The container scheduling state can be described as the current idle Nodes in the cluster, N = (N1, N2, N3, ..., N...). j , ..., N n The joint state of ) such as Figure 4 As shown, the remaining resource status of the j-th Node can be represented by a tuple vector. It means that among them This indicates the remaining status of each resource on the j-th Node.

[0084] The container scheduling multi-agent system selects joint actions based on the available Node N nodes in the current cluster to achieve efficient container resource scheduling, such as... Figure 5 As shown, the action space of a joint action can be represented as a list of Pod containers to be scheduled, such as... Figure 5 As shown, the combined action Where i and j are both integers, representing Pods i Scheduled to N j Run containerized applications on them.

[0085] Step S104: When it is determined that the current loop will stop iterative training based on the joint reward and / or the training cycle statistics corresponding to the current training cycle, the next loop will be entered, and the joint state data and environment data will be obtained from the sample data pool corresponding to the next loop to iteratively train the pod scheduling model until the number of loops reaches the preset number threshold, at which point iterative training will stop and the final pod scheduling model will be obtained.

[0086] Specifically, when the combined reward generated in the current loop and / or the number of training cycles has reached the preset number, which is the training cycle statistics value (maximum number of training steps), the current loop can be stopped and the next loop can begin.

[0087] After entering the next loop, iterative training is performed using a similar method and steps as described above. The sample data used during training is the joint state data and environment data obtained from the sample pool corresponding to the next loop. Iterative training stops when the number of loops reaches a preset threshold, and the final pod scheduling model is obtained.

[0088] The pod scheduling model construction method provided in this embodiment, by considering the remaining resources of each node in the Kubernetes cluster, enables the model to allocate Pods more effectively, thereby maximizing the utilization of cluster resources. Through iterative training, the model continuously learns how to make optimal scheduling decisions based on the current cluster state and the needs of the Pods to be scheduled, thus improving Pod scheduling efficiency. Specifically, this application includes two nested loops: one sets different iterative training cycles within each loop, and the other includes multiple loops for iterative training, making the iterative training process richer and the final model more adaptable to different scenarios. The joint reward mechanism considers multiple factors, including the resource request reward of the target Pod, the matching degree reward between the Pod and the candidate node, and the influence reward of the candidate node on other nodes, helping the model make more comprehensive and reasonable scheduling decisions. In particular, this application method considers the influence of other nodes on a particular node, maximizing scheduling efficiency and resource utilization. Furthermore, the model can adjust its behavior based on the reward corresponding to the current training cycle and the statistical values ​​of the training cycle, enabling the model to adapt to constantly changing environments and needs. Through iterative training, the model continuously learns new scheduling strategies from the sample data pool until a preset training iteration threshold is reached, ensuring that the model fully absorbs information from the data during training. Multiple iterations of training and the acquisition of the final model ensure that the trained Pod scheduling model has high accuracy and reliability.

[0089] In summary, this method ensures better resource allocation and reduces resource waste. It quickly finds suitable nodes to schedule Pods, reducing scheduling latency. Through precise matching, it increases the probability of successful Pod startup and operation. The model can adapt to different workloads and cluster states.

[0090] This embodiment provides a method for constructing a pod scheduling model, which can be used on the aforementioned mobile terminals, such as mobile phones and tablets. Figure 2 This is a flowchart illustrating the pod scheduling model construction method provided in this embodiment of the invention, as shown below. Figure 2 As shown, based on the aforementioned embodiments, the method for obtaining sample data from the sample data pool corresponding to each cycle includes the following steps, see details below. Figure 6 As shown:

[0091] Step S601: In the current loop, obtain the initial joint state data and environment data.

[0092] The initial joint state data refers to the initial remaining resources of each node in the Kubernetes cluster within the current loop.

[0093] Step S602: In the initial data generation cycle, the initial joint state data and environment data are input into the initial pod scheduling model to generate the joint action corresponding to the next data generation cycle and the joint reward corresponding to the current data generation cycle.

[0094] Specifically, as introduced above, the pod scheduling model is the MADADPG model. Specifically, after inputting the initial joint state data and environment data into the initial pod scheduling model, corresponding joint actions and joint rewards will be generated.

[0095] Specifically, for a cluster of n nodes N = (N1, N2, N3, ..., N... j , ..., N n The joint policy function can be expressed as π = (v(θ1), π(θ2), ..., v(θ)). N )), where θ i Let i ∈ [1, n] be the policy function parameters of the i-th agent. During the model training phase, the input to the evaluation network (Critic) is the observed states S = (s1, s2, ..., sn) of all agents. i , ..., s n ) and the action set A = (a1, a2, ..., a i , ..., a n Then, the Q-value of each agent can be expressed as:

[0096]

[0097] in, For the target network, FC1 i FC2 i For the fully connected layer of the i-th agent, V i V is the sum of the value weights of all other agents. i It can be represented as:

[0098]

[0099] Where j∈[1,n] and j≠i, ReLU is the activation function, W is a fixed matrix, and the attention weight α j The embedding e i and e j Comparisons are made, and gradient vanishing is prevented by scaling the matrix dimensions. αj It can be represented as:

[0100]

[0101] in W i e i and e j If the weight coefficients are given, then the loss function for evaluating the network can be expressed as:

[0102]

[0103] in (S, A, R, S_) ~ D represents the sample data in the sample data pool corresponding to the current loop in the current training phase, and E represents the expected value.

[0104] The update of the decision network parameters at this time can be expressed as:

[0105]

[0106] Where, θ μ For the policy network parameters, α u For the policy network learning rate, denoted by gradient descent, and J represents the expected return function.

[0107] After training, the agent only needs to consider a portion of the observed input (the input in the evaluation network itself includes some observed data plus input from other agents, and the partial observed input includes joint rewards), and input it into the decision network to obtain the optimal decision action and the corresponding joint reward.

[0108] Step S603: The initial joint state data, environment data, joint action corresponding to the next data generation cycle, and joint reward corresponding to the current data generation cycle are combined into a quadruple as a sample data and added to the sample data pool corresponding to the current cycle.

[0109] Step S604: In the i-th data generation cycle, based on the joint action corresponding to the i-th generation cycle and the joint state data corresponding to the (i-1)-th data generation cycle, generate the joint state data corresponding to the i-th data generation cycle.

[0110] Where i is a positive integer greater than or equal to 2, and i increments by 1.

[0111] Specifically, a joint action refers to a pod with scheduling being scheduled to a certain node based on the current situation. If scheduled to that node, it will inevitably occupy certain resources. Therefore, joint state data for the current period, i.e., the i-th period, can be generated based on the scheduling action and the joint state data in the (i-1)-th period. Here, i increments with the number of periods, with each increment being 1.

[0112] Step S605: Input the joint state data and environment data corresponding to the i-th data generation cycle into the initial pod scheduling model to obtain the joint action corresponding to the (i+1)-th data generation cycle and the joint reward corresponding to the i-th data generation cycle.

[0113] Step S606: The quadruple consisting of the joint state data and environment data corresponding to the i-th data generation cycle, the joint reward corresponding to the i-th data generation cycle, and the joint action corresponding to the (i+1)-th data generation cycle is added as a sample data to the sample data pool corresponding to the current cycle. The iteration stops when i equals the preset threshold, and all sample data in the sample data pool corresponding to the current cycle are obtained.

[0114] Figure 7 The diagram illustrates the overall process, including how different nodes generate joint scheduling actions A1, A2, ..., An based on the input joint reward R and joint state S, using the aforementioned model. Then, pods in the list of containers to be scheduled are scheduled according to scheduling action A, thereby obtaining the joint reward and the joint state S for the next cycle. Further, based on the joint state and joint reward of the next cycle, new scheduling actions are generated... This process is repeated iteratively, acquiring the aforementioned sample data and adding it to the sample data pool.

[0115] By dynamically determining the joint state in each data generation cycle, the model can capture the dynamic characteristics of resource allocation and Pod scheduling in a Kubernetes cluster over time. The joint actions and joint rewards generated in each data generation cycle simulate the decisions and outcomes during Pod scheduling, helping to train a scheduling model that better reflects reality. Through continuous data generation and iteration, the collected sample data becomes richer, encompassing various scenarios under different states and decisions, which is beneficial for the model's generalization ability. Training with pre-generated sample data improves model training efficiency because it eliminates the need for real-time interaction with the Kubernetes cluster to acquire data. With sample data containing multiple states and decisions, the model can learn more accurate and comprehensive scheduling strategies. Because the training data covers various possible scheduling scenarios, it adapts to different workloads and cluster states. In this method, multiple sample data can be generated in the manner described above to avoid situations where model training fails to meet requirements due to insufficient sample data. Moreover, the generated sample data helps the model learn how to utilize cluster resources more effectively, reducing resource waste. Furthermore, pre-generated sample data reduces the need for real-time data collection, thereby shortening training time. This ensures that the final trained model can make scheduling decisions more quickly, improving Pod scheduling efficiency. Sample data helps the Pod scheduling model learn more accurately how to efficiently schedule Pods in a Kubernetes cluster, thereby improving the overall performance and resource utilization of the cluster.

[0116] In an optional example, when the pod scheduling model generates a joint reward corresponding to the current training cycle, it determines whether to stop iterative training in the current loop based on the joint reward and / or the training cycle statistics corresponding to the current training cycle. Specifically, this includes:

[0117] When the reward value of the combined reward is greater than or equal to the preset reward threshold, the current loop is stopped from iterative training.

[0118] And / or,

[0119] When the statistical value of the training cycle corresponding to the current training cycle reaches the preset number of cycles, the current loop is stopped from iterative training.

[0120] Specifically, when the joint reward value generated by the model reaches or exceeds a preset reward threshold, the model can be considered to have learned an effective scheduling strategy. Stopping iterative training at this point avoids overfitting and saves computational resources. When the training epochs reach a preset number, iterative training stops regardless of the model's performance. This ensures the model completes training within a reasonable timeframe, preventing unnecessary long training periods. When a certain number of training epochs have been reached, the model has accumulated sufficient experience, and continued training may not significantly improve performance; therefore, stopping training ensures model stability. Setting explicit stopping conditions in this way helps understand when the model reaches performance requirements, improving the transparency and interpretability of the training process.

[0121] Based on any of the foregoing embodiments, when the resource type includes CPU resources, memory resources, disk resources, network resources, video memory resources, and GPU resources, the expression for the target pod to request resource rewards is as follows:

[0122] R score =w cpu *req cpu +w mem *req mem +w disk *req disk +w net *req net +w vimem *req vimem +w gpu *req gpu (Formula 12)

[0123] w cpu +w mem +w disk +w net +w vimem +w gpu =1 (Formula 13)

[0124] Among them, R score The reward value for requesting resources from the target pod, req cpu For remaining CPU resources, req mem For remaining memory resources, req disk For remaining disk resources, req net For remaining network resources, req vimem For the remaining video memory resources, req gpu For the remaining GPU resources, w cpu w mem w disk w net w vimem and wgpu These are the weighting coefficients corresponding to each resource.

[0125] Based on any of the foregoing embodiments, when the resource types include CPU resources, memory resources, disk resources, network resources, video memory resources, and GPU resources, the matching degree reward between the target pod and the candidate node is expressed by the following expression:

[0126]

[0127] Among them, R similar The reward value is awarded based on the degree of matching between the target pod and the candidate nodes. For joint actions, it means Pod i Scheduled to N j Let i be the i-th pod and j be the j-th node. and These represent the remaining CPU resources on the i-th pod and the remaining CPU resources on the j-th node, respectively. and These represent the remaining memory resources on the i-th pod and the remaining memory resources on the i-th node, respectively. and These represent the remaining disk resources on the i-th pod and the remaining disk resources on the i-th node, respectively. and These represent the remaining network resources on the i-th pod and the remaining network resources on the j-th node, respectively. and These represent the remaining video memory resources on the i-th pod and the remaining video memory resources on the j-th node, respectively. and These represent the remaining GPU resources on the i-th pod and the remaining GPU resources on the j-th node, respectively.

[0128] The inter-node scheduling similarity reward is represented by the following expression:

[0129]

[0130] Among them, R attention The scheduling similarity reward between the nodes is given, where for the i-th node, Q is the scheduling similarity query vector, and K... T For each node, a similarity feature representation is used to calculate the similarity with Q. V scores the actual similarity of each node and is used for the final weighted output. k The dimension of the scheduling similarity query vector.

[0131] In an optional implementation, the joint reward expression is shown below:

[0132] R = μ1R score +μ2R similar +μ3R attention (Formula 17)

[0133] Where R is the reward value of the joint reward, and μ1, μ2, and μ3 are the pre-configured weight coefficients.

[0134] It should be noted that for all types of resources, All must meet

[0135]

[0136] Figure 8 The diagram above illustrates the overall process of the above method. See below for details. Figure 8 As shown, the specific implementation process has been described in detail above, so it will not be repeated here.

[0137] Furthermore, this application also provides a pod scheduling method. By acquiring the relevant environmental data and joint state data, and then inputting them into the aforementioned trained model, corresponding scheduling actions can be generated.

[0138] This embodiment also provides a pod scheduling model construction apparatus, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0139] This embodiment provides a pod scheduling model construction device, such as... Figure 9 The system includes: an acquisition module 901, a training module 902, and a processing module 903.

[0140] The acquisition module 901 is used to acquire joint state data and environment data from the sample data pool corresponding to the current cycle within the current training period. The joint state data is the remaining resources of each node in the K8s cluster, and the environment data includes multiple pods currently to be scheduled.

[0141] Training module 902 is used to input the joint state data and environment data obtained from the sample data pool corresponding to the current loop into the pod scheduling model, so as to iteratively train the pod scheduling model within the current loop;

[0142] Processing module 903 is used to determine whether to stop iterative training in the current loop when the pod scheduling model generates a joint reward corresponding to the current training cycle, based on the joint reward and / or the training cycle statistics corresponding to the current training cycle. The joint reward includes a resource request reward for the target pod, a matching degree reward between the target pod and the candidate node, and a node scheduling similarity reward corresponding to the influence of the candidate node and other nodes in the K8s cluster on the candidate node. The target pod is any one of multiple pods, and the candidate node is any one of multiple nodes. When it is determined that the current loop should stop iterative training, the next loop is entered.

[0143] The acquisition module 901 is also used to acquire joint state data and environmental data from the sample data pool corresponding to the next cycle;

[0144] The training module 902 is also used to iteratively train the pod scheduling model by obtaining joint state data and environment data from the sample data pool corresponding to the next cycle until the number of cycles reaches a preset threshold, at which point the iterative training stops and the final pod scheduling model is obtained.

[0145] In an optional implementation, the acquisition module 901 is specifically used for:

[0146] Within the current loop, obtain the initial joint state data and environment data. The initial joint state data is the initial remaining resources of each node in the Kubernetes cluster within the current loop.

[0147] In the initial data generation cycle, the initial joint state data and environment data are input into the initial pod scheduling model to generate the joint action corresponding to the next data generation cycle, as well as the joint reward corresponding to the current data generation cycle.

[0148] The initial joint state data, environment data, joint action corresponding to the next data generation cycle, and joint reward corresponding to the current data generation cycle are combined into a quadruple as a sample data and added to the sample data pool corresponding to the current cycle.

[0149] In the i-th data generation cycle, based on the joint action corresponding to the i-th generation cycle and the joint state data corresponding to the (i-1)-th data generation cycle, the joint state data corresponding to the i-th data generation cycle is generated, where i is a positive integer greater than or equal to 2, and i increments by 1.

[0150] Input the joint state data and environment data corresponding to the i-th data generation cycle into the initial pod scheduling model to obtain the joint action corresponding to the (i+1)-th data generation cycle, as well as the joint reward corresponding to the i-th data generation cycle.

[0151] The quadruple consisting of the joint state data and environment data corresponding to the i-th data generation cycle, the joint reward corresponding to the i-th data generation cycle, and the joint action corresponding to the i+1-th data generation cycle is added as a sample data and added to the sample data pool corresponding to the current cycle.

[0152] The iteration stops when i equals the preset threshold, and all sample data in the sample data pool corresponding to the current loop are obtained.

[0153] In an optional implementation, the processing module 903 is specifically used to: determine that the current loop stops iterative training when the reward value of the joint reward is greater than or equal to a preset reward threshold;

[0154] And / or,

[0155] When the statistical value of the training cycle corresponding to the current training cycle reaches the preset number of cycles, the current loop is stopped from iterative training.

[0156] In one optional implementation, the resource types of the remaining resources in each node include one or more of the following:

[0157] CPU resources, memory resources, disk resources, network resources, video memory resources, and GPU resources.

[0158] In an optional implementation, when the resource type includes CPU resources, memory resources, disk resources, network resources, video memory resources, and GPU resources, the expression for the target pod to request resource rewards is as follows:

[0159] R score =w cpu *req cpu +w mem *req mem +w disk *req disk +w net *req net +w vimem *req vimem +w gpu *req gpu (Formula 18)

[0160] w cpu +w mem +w disk +w net +w vimem +w gpu =1 (Formula 19)

[0161] Among them, R scoreThe reward value for requesting resources from the target pod, req cpu For remaining CPU resources, reqme m reqdis is a resource for remaining memory. k For remaining disk resources, request t For remaining network resources, req vimem For the remaining video memory resources, req gpu For the remaining GPU resources, w cpu w mem w disk w net w vimem and w gpu These are the weighting coefficients corresponding to each resource.

[0162] In an optional implementation, when the resource types include CPU resources, memory resources, disk resources, network resources, video memory resources, and GPU resources, the matching degree reward between the target pod and the candidate node is represented by the following expression:

[0163]

[0164] Among them, R similar The reward value is awarded based on the degree of matching between the target pod and the candidate nodes. For joint actions, it means Pod i Scheduled to N j Let i be the i-th pod and j be the j-th node. and These represent the remaining CPU resources on the i-th pod and the remaining CPU resources on the j-th node, respectively. and These represent the remaining memory resources on the i-th pod and the remaining memory resources on the i-th node, respectively. and These represent the remaining disk resources on the i-th pod and the remaining disk resources on the i-th node, respectively. and These represent the remaining network resources on the i-th pod and the remaining network resources on the j-th node, respectively. and These represent the remaining video memory resources on the i-th pod and the remaining video memory resources on the j-th node, respectively. and These represent the remaining GPU resources on the i-th pod and the remaining GPU resources on the j-th node, respectively.

[0165] In an optional implementation, the inter-node scheduling similarity reward is represented by the following expression:

[0166]

[0167] Among them, R attention The scheduling similarity reward between the nodes is given, where for the i-th node, Q is the scheduling similarity query vector, and K... T For each node, a similarity feature representation is used to calculate the similarity with Q. V scores the actual similarity of each node and is used for the final weighted output. k The dimension of the scheduling similarity query vector.

[0168] In an optional implementation, the joint reward expression is shown below:

[0169] R = μ1R score +μ2R similar +μ3R attention (Formula 23)

[0170] Where R is the reward value of the joint reward, and μ1, μ2, and μ3 are the pre-configured weight coefficients.

[0171] In this embodiment, the pod scheduling model construction device is presented in the form of functional modules. Here, a module refers to an application-specific integrated circuit (ASIC), a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0172] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0173] This invention provides a pod scheduling model construction device. By considering the remaining resources of each node in a Kubernetes cluster, the model can allocate Pods more effectively, thereby maximizing the utilization of cluster resources. Through iterative training, the model continuously learns how to make optimal scheduling decisions based on the current cluster state and the needs of the Pods to be scheduled, thus improving Pod scheduling efficiency. Specifically, this application includes two nested loops: one sets different iterative training cycles within each loop, and the other includes multiple loops for iterative training, making the iterative training process richer and the final model more adaptable to different scenarios. The joint reward mechanism considers multiple factors, including resource request rewards for the target Pod, rewards for the matching degree between the Pod and candidate nodes, and rewards for the influence of candidate nodes on other nodes, helping the model make more comprehensive and reasonable scheduling decisions. In particular, this application considers the influence of other nodes on a particular node, maximizing scheduling efficiency and resource utilization. Furthermore, the model can adjust its behavior based on the rewards corresponding to the current training cycle and the statistical values ​​of the training cycle, enabling the model to adapt to constantly changing environments and needs. Through iterative training, the model continuously learns new scheduling strategies from the sample data pool until a preset training iteration threshold is reached, ensuring that the model fully absorbs information from the data during training. Multiple iterations of training and the acquisition of the final model ensure that the trained Pod scheduling model has high accuracy and reliability.

[0174] In summary, this method ensures better resource allocation and reduces resource waste. It quickly finds suitable nodes to schedule Pods, reducing scheduling latency. Through precise matching, it increases the probability of successful Pod startup and operation. The model can adapt to different workloads and cluster states.

[0175] This invention also provides a computer device having the above-described features. Figure 9 The pod scheduling model construction device shown.

[0176] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 10As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 10 Take a processor 10 as an example.

[0177] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include an integrated circuit. The integrated circuit may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPRS), or any combination thereof.

[0178] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0179] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device as shown by a landing page for an app. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0180] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0181] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.

[0182] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0183] This invention also provides a computer-readable storage medium. The methods provided in the above embodiments can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0184] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0185] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for constructing a pod scheduling model, characterized in that, The method comprises: obtaining joint state data and environment data from a sample data pool corresponding to the current cycle in a current training period within the current cycle, wherein the joint state data is the remaining resources of each node in the plurality of nodes in the K8s cluster, and the environment data comprises a plurality of pods to be scheduled currently; inputting the joint state data and the environment data obtained from the sample data pool corresponding to the current cycle into a pod scheduling model, so as to iteratively train the pod scheduling model within the current cycle; when the pod scheduling model generates a joint reward corresponding to the current training period, determining whether the current cycle stops iterative training according to the joint reward and / or a training period statistical value corresponding to the current training period, wherein the joint reward comprises a target pod request resource reward, a matching degree reward between the target pod and a candidate node, and a node-to-node scheduling similarity reward corresponding to an influence of the candidate node and nodes other than the candidate node in the K8s cluster on the candidate node, the target pod being any one of the plurality of pods, and the candidate node being any one of the plurality of nodes; when it is determined according to the joint reward and / or the training period statistical value corresponding to the current training period that the current cycle stops iterative training, entering a next cycle, and obtaining joint state data and environment data from a sample data pool corresponding to the next cycle to iteratively train the pod scheduling model, until the number of cycles reaches a preset number threshold, and stopping iterative training to obtain a final pod scheduling model.

2. The method of claim 1, wherein, The sample data in the sample data pool corresponding to each cycle comprises: in the current cycle, obtaining initial joint state data and environment data, wherein the initial joint state data is the initial remaining resources of each node in the plurality of nodes in the K8s cluster within the current cycle; in an initial data generation period, inputting the initial joint state data and the environment data into an initial pod scheduling model to generate joint actions corresponding to a next data generation period and a joint reward corresponding to the current data generation period; forming a four-tuple of the initial joint state data, the environment data, the joint actions corresponding to the next data generation period, and the joint reward corresponding to the current data generation period as a piece of sample data, and adding the piece of sample data to the sample data pool corresponding to the current cycle; in an i th data generation period, generating joint state data corresponding to the i th data generation period based on joint actions corresponding to the i th generation period and joint state data corresponding to an (i-1) th data generation period, wherein i is a positive integer greater than or equal to 2, and i is incremented by 1; inputting the joint state data corresponding to the i th data generation period and the environment data into the initial pod scheduling model to obtain joint actions corresponding to an (i+1) th data generation period and a joint reward corresponding to the i th data generation period; and the joint action corresponding to the i+1th data generation period as a quadruple of sample data, and add the quadruple to the sample data pool corresponding to the current cycle; stop iteration until i is equal to a preset threshold value, and obtain all sample data in the sample data pool corresponding to the current cycle.

3. The method according to claim 1 or 2, characterized in that, When the pod scheduling model generates a joint reward corresponding to the current training period, whether the current cycle stops iteration training is determined according to the joint reward and / or a training period statistical value corresponding to the current training period, and specifically includes: When the reward value of the joint reward is greater than or equal to a preset reward threshold value, it is determined that the current cycle stops iteration training. and / or, When it is determined that the training period statistical value corresponding to the current training period reaches a preset period number value, it is determined that the current cycle stops iteration training.

4. The method according to claim 1 or 2, characterized in that, The resource types of the remaining resources in each node include one or more of the following: CPU resources, memory resources, disk resources, network resources, graphics memory resources, and GPU resources.

5. The method of claim 4, wherein, When the resource types include CPU resources, memory resources, disk resources, network resources, graphics memory resources, and GPU resources, the expression of the target pod request resource reward is as follows: R score = w cpu * req cpu + w mem * req mem + w disk * req disk + w net * req net +w vimem *req vimem +w gpu *req gpu (Formula 1) w cpu +w mem +w disk +w net +w vimem +w gpu = 1 (Equation 2) wherein R score is a reward value of a reward requested by the target pod for resources, req cpu is a remaining resource of a CPU, req mem is a remaining resource of a memory, req disk is a remaining resource of a disk, req net is a remaining resource of a network, req vimem is a remaining resource of a graphics memory, req gpu is a remaining resource of a GPU, w cpu , w mem , w disk , w net , w vimem , and w gpu are weight coefficients corresponding to each resource, respectively.

6. The method of claim 4, wherein, When the resource types include CPU resources, memory resources, disk resources, network resources, graphics memory resources, and GPU resources, the matching degree reward between the target pod and the candidate node is represented by the following expression: Among them, R similar The reward value is awarded based on the degree of matching between the target pod and the candidate node. For joint actions, it means Pod i Scheduled to N j Let i be the i-th pod and j be the j-th node. and These represent the remaining CPU resources on the i-th pod and the remaining CPU resources on the j-th node, respectively. and These represent the remaining memory resources on the i-th pod and the remaining memory resources on the j-th node, respectively. and These represent the remaining disk resources on the i-th pod and the remaining disk resources on the j-th node, respectively. and These represent the remaining network resources on the i-th pod and the remaining network resources on the j-th node, respectively. and These represent the remaining video memory resources on the i-th pod and the remaining video memory resources on the j-th node, respectively. and These represent the remaining GPU resources on the i-th pod and the remaining GPU resources on the j-th node, respectively.

7. The method of claim 1 or 2, wherein, The inter-node scheduling similarity reward is represented by the following expression: wherein R attention is the inter-node scheduling similarity reward, Q is the scheduling similarity query vector, K T is the per-node scheduling similarity degree feature representation, V is the per-node actual similarity score for final weighted output, d k is the dimension size of the scheduling similarity query vector. 8.A pod scheduling model construction apparatus, characterized by comprising: The apparatus includes: An acquisition module, configured to acquire joint state data and environment data in a current training period in a current cycle from a sample data pool corresponding to the current cycle, wherein the joint state data is remaining resources of each node in a plurality of nodes in a K8s cluster, and the environment data includes a plurality of pods to be scheduled currently; A training module, configured to input the acquired joint state data and environment data in the sample data pool corresponding to the current cycle to a pod scheduling model, so as to perform iteration training on the pod scheduling model in the current cycle; A processing module, configured to, when the pod scheduling model generates a joint reward corresponding to the current training period, determine whether the current cycle stops iteration training according to the joint reward and / or a training period statistical value corresponding to the current training period, wherein the joint reward includes a target pod request resource reward, a matching degree reward between the target pod and a candidate node, and an inter-node scheduling similarity reward corresponding to an influence of the candidate node on the candidate node in the K8s cluster except the candidate node, the target pod is any one of a plurality of pods, and the candidate node is any one of a plurality of nodes; when it is determined that the current cycle stops iteration training, a next cycle is entered. The acquisition module is further configured to acquire joint state data and environment data from a sample data pool corresponding to a next loop. The training module is further configured to perform iterative training on the pod scheduling model by using the joint state data and the environment data acquired from the sample data pool corresponding to the next loop, and stop the iterative training and obtain a final pod scheduling model when a number of loops reaches a preset number threshold.

9. A computer device, comprising: Comprise: A memory and a processor, which are in communication connection with each other, and the memory stores computer instructions, and the processor executes the computer instructions to perform the pod scheduling model construction method in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to make a computer execute the pod scheduling model construction method in any one of claims 1 to 7.

Images