A dynamic scheduling method and system for multi-dimensional resources in off-line mixing

Abstract

The present application relates to the technical field of offline resource scheduling, in particular to a kind of dynamic scheduling method and system of multidimensional resource in offline mixed department;The present application is first made, filtering is carried out to filter out the node meeting the requirement, then each node is according to the value change of respective interference health degree, executes dynamic resource adjustment and offline job re-migration, then obtains the current resource monitoring data index of each node and maintains monitoring historical data, continues every set interval time, obtains the resource monitoring data index of each node to calculate the current interference health degree of each node;Finally, according to the interference health degree of each node, score is carried out, and the node with the highest score is selected as the scheduling node, and the new job is scheduled to the scheduling node;The present application monitors resource data index, adopts dynamic resource adjustment after initial scheduling, and then carries out the strategy of re-migration to ensure that the performance of online job under mixed department scene is not affected.

Description

Technical Field

[0001] This invention relates to the field of offline resource scheduling technology, and in particular to a dynamic scheduling method and system for multi-dimensional resources in offline mixed environments. Background Technology

[0002] Traditional service deployment methods result in low server resource utilization in data centers, typically between 10% and 20%, leading to significant resource waste. Co-location technology, which combines online and offline services, is an effective way to improve resource utilization that has gained significant attention from major cloud vendors in recent years. It greatly improves server resource utilization by co-deploying online services and offline jobs within the same cluster. However, co-location technology still has many problems. For example, offline jobs may compete for shared resources on the server simultaneously, interfering with the performance of online services and impacting user experience.

[0003] Job scheduling in offline and mixed deployment scenarios is a complex multi-objective problem. It must consider not only conventional optimization objectives such as load balancing and job affinity at the cluster level, but also the performance interference of online jobs after mixed deployment. Furthermore, after scheduling, resources must be dynamically adjusted in real time according to the actual operation of the jobs to maximize the guarantee of the SLA (Service Level Agreement) of online services. Therefore, it is a very challenging problem.

[0004] Previous work has proposed several solutions, such as the Paragon scheduling model proposed by Delimitrou et al., which uses a series of monitoring indicators such as memory usage, L1 / L2 / L3 cache, and network bandwidth as input, uses a collaborative filtering algorithm to predict the performance interference of nodes after mixed deployment, and finally uses a greedy algorithm to find the optimal node. Another example is the HySARC scheduling model proposed by Vasile, which clusters the CPU and I / O resources required by the job and then performs resource matching to complete the scheduling. Yet another example is the Medea scheduling model proposed by Garefalakis et al., which treats scheduling as an integer linear programming problem, putting multiple objectives such as maximizing resource utilization and minimizing fragmentation into a single objective function, and then using an optimizer to solve it. However, none of the solutions proposed by these authors can effectively solve the job scheduling problem in offline mixed deployment scenarios. Summary of the Invention

[0005] The main technical problem solved by this invention is to provide a dynamic scheduling method for multi-dimensional resources in offline mixed deployment. This method monitors resource data indicators, uses deep neural networks to construct an accurate performance interference model, and adopts a strategy of dynamic resource adjustment after initial scheduling and then re-migration to ensure that the performance of online operations is not affected in mixed deployment scenarios. It also provides a dynamic scheduling system for multi-dimensional resources in offline mixed deployment.

[0006] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is to provide a dynamic scheduling method for offline mixed-use systems, comprising the following steps:

[0007] Step S1: Submit a new job and filter it to select nodes that meet the requirements;

[0008] Step S2: Each node performs dynamic resource adjustment and offline job re-migration based on the changes in its respective interference health value;

[0009] Step S3: Obtain the current resource monitoring data metrics for each node and maintain historical monitoring data;

[0010] Step S4: At set intervals, obtain the resource monitoring data indicators of each node to calculate the current interference health of each node.

[0011] Step S5: Score each node based on its interference health, select the node with the highest score as the scheduling node, and schedule the new job to that scheduling node.

[0012] As an improvement of the present invention, in step S2, each node is provided with a DaemonSet type Agent for maintaining the interference health of the node. The Agent continuously performs dynamic resource adjustment and offline job re-migration according to the change of the interference health value.

[0013] As a further improvement of the present invention, the step of dynamic resource adjustment is as follows:

[0014] Step S201: If the interference health of the current node is detected to be ≤5 and >2, trigger the offline job resource compression action, limit the CPU usage of all current offline jobs on the node, and adjust the CFS Quota value of the offline jobs to half of the current value.

[0015] Step S202: If the interference health of the current node is detected to be >9, trigger the offline job resource recovery action, restore the CPU usage quota of all current offline jobs on the node and restore the CFS Quota value of the offline jobs to the initial value.

[0016] Step S203: Repeat steps S201 and S202 for 10 seconds.

[0017] As a further improvement of the present invention, in step S202, the initial value is obtained from the Kubernetes API Server.

[0018] As a further improvement of the present invention, the step of offline job re-migration is as follows:

[0019] Step S211: If the interference health of the current node is detected to be ≤2 and the time difference between the last time the node performed the offline job re-migration action and the current time is more than 2 minutes, then the offline job re-migration action is triggered, and the time of the most recent offline job re-migration of the current node is updated; otherwise, step S211 is executed again, with an interval of 10 seconds.

[0020] Step S212: Sort the CPU usage of offline jobs and select the offline job with the highest current CPU usage as the target to be migrated;

[0021] Step S213: Obtain the last migration time of the offline job from the annotation of the object to be migrated. If the difference between the current time and the last migration time is greater than 30 minutes, remove the object to be migrated and update the annotation of the object to be migrated, recording the last migration time; otherwise, select the object to be migrated again.

[0022] Step S214: The Kubernetes ReplicaSet Controller restarts and schedules the objects to be migrated.

[0023] Step S215: Repeat steps S211 to S214, with an interval of 10 seconds.

[0024] As a further improvement of the present invention, in step S3, the resource monitoring data indicators include the overall CPU utilization rate, the overall memory utilization rate, the overall memory bandwidth utilization rate, IPC, and disk usage percentage.

[0025] As a further improvement of the present invention, in step S4, historical data of the resource monitoring data of each node for the past 60 seconds is obtained every 10 seconds.

[0026] As a further improvement of the present invention, the interference health status of the current node is obtained by taking the maintenance monitoring historical data of the past 60 seconds as input.

[0027] As a further improvement of the present invention, in step S4, the interference health status of the current node is pushed to the Agent of each node to update the interference health status at the latest moment.

[0028] A dynamic scheduling system for offline and mixed deployments, comprising:

[0029] The submission filtering module is used to submit new jobs and filter them to select nodes that meet the requirements.

[0030] The execution module is used by each node to perform dynamic resource adjustments and offline job re-migration based on the changes in its own interference health value.

[0031] The monitoring module is used to obtain the current resource monitoring data metrics for each node and maintain historical monitoring data.

[0032] The calculation module is used to obtain the resource monitoring data indicators of each node at set intervals to calculate the current interference health of each node.

[0033] The scheduling module is used to score the interference health of each node and select the node with the highest score as the scheduling node.

[0034] The beneficial effects of this invention are as follows: Compared with the prior art, this invention monitors resource data indicators, uses deep neural networks to construct an accurate performance interference model, and adopts a strategy of dynamic resource adjustment after initial scheduling and then re-migrating to ensure that the performance of online operations in mixed deployment scenarios is not affected. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating the steps of the dynamic scheduling method for online and offline hybrid systems according to the present invention.

[0036] Figure 2 This is a flowchart illustrating the steps of dynamic resource adjustment in this invention.

[0037] Figure 3 This is a flowchart illustrating the steps of offline job re-migration according to the present invention;

[0038] Figure 4 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0040] The two best existing methods are as follows:

[0041] The first approach involves predicting the performance impact of mixed deployment and selecting the node with the least interference for scheduling. However, predicting the performance impact after mixed deployment is not easy because the resource usage data of the new job can only be collected after scheduling and cannot be obtained before scheduling. Therefore, this type of method relies on the current resource usage of the server and the resource request of the new job to predict the degree of performance impact after mixed deployment. However, the resource request of the new job is not necessarily consistent with the resources used after it runs. The actual usage may be much lower than the requested value. Therefore, relying on the resource request of the new job to predict the performance impact that may occur in the future has a low accuracy rate.

[0042] The second approach treats scheduling as an optimization problem and uses linear programming to find the optimal solution. However, while treating scheduling as an optimization problem and using integer linear programming can achieve good results, this type of method has too high computational complexity and takes too long, making it unsuitable for large-scale mixed deployment scenarios.

[0043] Please refer to Figures 1 to 4 The present invention provides a dynamic scheduling method for offline mixed-use systems, comprising the following steps:

[0044] Step S1: Submit a new job and filter it to select nodes that meet the requirements;

[0045] Step S2: Each node performs dynamic resource adjustment and offline job re-migration based on the changes in its respective interference health value;

[0046] Step S3: Obtain the current resource monitoring data metrics for each node and maintain historical monitoring data;

[0047] Step S4: At set intervals, obtain the resource monitoring data indicators of each node to calculate the current interference health of each node.

[0048] Step S5: Score each node based on its interference health, select the node with the highest score as the scheduling node, and schedule the new job to that scheduling node.

[0049] This invention monitors resource data indicators, uses deep neural networks to build an accurate performance interference model, and adopts a strategy of dynamic resource adjustment after initial scheduling and then re-migrating to ensure that the performance of online operations in mixed deployment scenarios is not affected.

[0050] In step S2, each node is equipped with a DaemonSet type Agent for maintaining the node's interference health. The Agent continuously performs dynamic resource adjustments and offline job re-migration based on the changes in the interference health value.

[0051] In step S1, new job submissions are received, and nodes that do not meet the requirements are filtered out based on the requested resource quota, affinity, interference health, and other requirements, thereby selecting nodes that meet the requirements.

[0052] Within this invention, the steps for dynamic resource adjustment are as follows:

[0053] Step S201: If the interference health of the current node is detected to be ≤5 and >2, trigger the offline job resource compression action, limit the CPU usage of all current offline jobs on the node, and adjust the CFS Quota value of the offline jobs to half of the current value.

[0054] Step S202: If the interference health of the current node is detected to be >9, trigger the offline job resource recovery action, restore the CPU usage quota of all current offline jobs on the node and restore the CFS Quota value of the offline jobs to the initial value.

[0055] Step S203: Repeat steps S201 and S202 for 10 seconds.

[0056] In step S202, the initial value is obtained from the Kubernetes API Server (Application Programming Interface Server).

[0057] In this invention, the offline job re-migration step is as follows:

[0058] Step S211: If the interference health of the current node is detected to be ≤2 and the time difference between the last time the node performed the offline job re-migration action and the current time is more than 2 minutes, then the offline job re-migration action is triggered, and the time of the most recent offline job re-migration of the current node is updated; otherwise, step S211 is executed again, with an interval of 10 seconds.

[0059] Step S212: Sort the CPU usage of offline jobs and select the offline job with the highest current CPU usage as the target to be migrated;

[0060] Step S213: Obtain the last migration time of the offline job from the annotation of the object to be migrated. If the difference between the current time and the last migration time is greater than 30 minutes, remove the object to be migrated and update the annotation of the object to be migrated, recording the last migration time; otherwise, select the object to be migrated again.

[0061] Step S214: The Kubernetes ReplicaSet Controller restarts and schedules the objects to be migrated.

[0062] Step S215: Repeat steps S211 to S214, with an interval of 10 seconds.

[0063] Within this invention, in step S3, the resource monitoring data indicators include overall CPU utilization, overall memory utilization, overall memory bandwidth utilization, IPC, and disk usage percentage.

[0064] Within this invention, in step S4, historical data of the resource monitoring data of each node for the past 60 seconds is acquired every 10 seconds; using the historical monitoring data of the past 60 seconds as input, the interference health of the current node is obtained; the interference health of the current node is pushed to the Agent of each node to update the interference health status at the latest moment; specifically, the resource mixed-distribution interference model acquires historical data of the multi-dimensional resource indicators of each node for the past 60 seconds from the monitoring component every 10 seconds; the established resource mixed-distribution interference model uses a Long Short-Term Memory (LSTM) network as a feature extractor, and uses the historical resource monitoring data as input to give the current node's performance interference health score. This is a classification process, and the interference health score is 1-10. The interference health is determined by the classification of the neural network model. For this purpose, a dataset needs to be constructed for the neural network training. In practical applications, cases that affect business indicators can be distinguished into different health levels based on the log records generated by online business; the resource mixed-distribution interference model pushes the interference health to the Agent on each node to maintain the interference health status at the latest moment.

[0065] This invention provides an embodiment of a dynamic scheduling method for offline and mixed-use environments, which includes the following steps:

[0066] (1) When a new job is submitted, nodes that meet the resource and topology requirements are first filtered out by the intelligent scheduling method. Specifically, when a new job is submitted, it requests CPU, memory quota or topology relationship requirements, etc. Nodes that do not meet these requirements are first excluded before proceeding to the next step.

[0067] (2) Each node has an Agent of type DaemonSet, which is responsible for maintaining the interference health of each node and performing dynamic resource adjustment and offline job re-migration according to the change of the interference health value.

[0068] (3) The monitoring component is responsible for obtaining the current resource monitoring metrics of each node and maintaining historical data for a period of time;

[0069] (4) Establish a resource mixed interference model, pull historical resource monitoring data of each node from the monitoring component at fixed intervals, calculate the current interference health of each node based on its own model, and then notify the Agent of each node to update its own interference health.

[0070] (5) Based on the interference health of each node, the node with the highest score is selected as the scheduling node, and the new job is scheduled to that node.

[0071] Each node has an Agent of type DaemonSet, which is responsible for maintaining the interference health level (1-10) of each node and continuously performing dynamic resource adjustments and offline job re-migration based on the changes in the interference health level.

[0072] The specific steps for dynamic resource adjustment are as follows:

[0073] 1) If the interference health of the current node is detected to be less than or equal to 5 and greater than 2, trigger the offline job resource compression action: limit the CPU usage of all current offline jobs on the node, and adjust the CFSQuota value of the offline jobs to half of the current value through Cgroup technology;

[0074] 2) If the interference health of the current node is detected to be greater than 9, trigger the offline job resource recovery action: restore the CPU usage of all current offline jobs on the node, and restore the CFS Quota value of the offline jobs to the initial value through Cgroup technology, where the initial value can be obtained from the Kubernetes API Server;

[0075] 3) Repeat steps 1)-2) above, with an interval of 10 seconds.

[0076] The specific steps for re-migrating offline jobs are as follows:

[0077] 1) If the current node’s interference health is less than or equal to 2 and the time difference between the last time the node performed the offline job re-migration action and the current time is more than 2 minutes, then the offline job re-migration action is triggered and the time of the last offline job re-migration of the current node is updated; otherwise, 1) is executed again, with an interval of 10 seconds.

[0078] 2) Sort the offline jobs by CPU usage and select the offline job with the highest current CPU usage as the target to be migrated;

[0079] 3) Obtain the last migration time of the offline job from the annotation of the object to be migrated. If the difference between the current time and the last migration time is greater than 30 minutes, kill the object to be migrated and update the annotation of the object to be migrated, recording the last migration time; otherwise, select a new object to be migrated.

[0080] 4) The ReplicaSet Controller of Kubernetes is responsible for restarting and scheduling the objects to be migrated, and the scheduling is completed by the scheduling strategy in step 1 of this invention;

[0081] 5) Repeat steps 1)-4) above, with an interval of 10 seconds.

[0082] The present invention also provides a dynamic scheduling system for offline and mixed deployments, comprising:

[0083] The submission filtering module is used to submit new jobs and filter them to select nodes that meet the requirements.

[0084] The execution module is used by each node to perform dynamic resource adjustments and offline job re-migration based on the changes in its own interference health value.

[0085] The monitoring module is used to obtain the current resource monitoring data metrics for each node and maintain historical monitoring data.

[0086] The calculation module is used to obtain the resource monitoring data indicators of each node at set intervals to calculate the current interference health of each node.

[0087] The scheduling module is used to score the interference health of each node and select the node with the highest score as the scheduling node.

[0088] The monitoring module includes a monitoring component, and the calculation module includes a model unit for establishing a resource mixed interference model.

[0089] The present invention has the following advantages:

[0090] 1. Based on Kubernetes scoring rules, interference health score has been added, which is easy to extend and implement, and has low computational complexity, making it suitable for large-scale offline and offline mixed deployment scenarios;

[0091] 2. The dynamic resource adjustment strategy increases or decreases the CPU quota of offline jobs based on the interference health level; the offline job re-migration is carried out based on the interference health level and frequency control to migrate offline jobs on the node; after one-time scheduling, the performance stability of online services on the node can be dynamically guaranteed.

[0092] 3. The following diagnostic indicators were selected: overall CPU utilization, overall memory utilization, overall memory bandwidth utilization, IPC (instructions per cycle, the number of instructions that can be executed per CPU clock cycle), and disk usage percentage. Long Short-Term Memory (LSTM) network was used as the feature extractor to capture the characteristics of resource cycle changes and obtain a more accurate current performance interference health status.

[0093] 4. The node interference health status is used to evaluate the performance interference of mixed deployment. Its level (1-10) is given by the resource mixed deployment interference model. Only one index is needed to quantify the degree of current interference.

[0094] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A dynamic scheduling method for offline and mixed deployments, characterized in that, Includes the following steps: Step S1: Submit a new job and filter it to select nodes that meet the requirements; Step S2: Each node performs dynamic resource adjustment and offline job re-migration based on the changes in its respective interference health value; Step S3: Obtain the current resource monitoring data metrics for each node and maintain historical monitoring data; Step S4: At set intervals, obtain the resource monitoring data indicators of each node to calculate the current interference health of each node. Step S5: Score each node based on its interference health, select the node with the highest score as the scheduling node, and schedule the new job to that scheduling node. The steps for dynamic resource adjustment are as follows: Step S201: If the interference health of the current node is detected to be ≤5 and >2, trigger the offline job resource compression action, limit the CPU usage of all current offline jobs on the node, and adjust the CFSQuota value of the offline jobs to half of the current value through Cgroup technology. Step S202: If the interference health of the current node is detected to be >9, trigger the offline job resource recovery action, restore the CPU usage quota of all current offline jobs on the node, and restore the CFS Quota value of the offline jobs to the initial value through Cgroup technology; Step S203: Repeat steps S201 and S202 for 10 seconds. The steps for re-migrating offline jobs are as follows: Step S211: If the interference health of the current node is detected to be ≤2 and the time difference between the last time the node performed the offline job re-migration action and the current time is more than 2 minutes, then the offline job re-migration action is triggered, and the time of the most recent offline job re-migration of the current node is updated; otherwise, step S211 is executed again, with an interval of 10 seconds. Step S212: Sort the CPU usage of offline jobs and select the offline job with the highest current CPU usage as the target to be migrated; Step S213: Obtain the last migration time of the offline job from the Annotation of the object to be migrated. If the difference between the current time and the last migration time is greater than 30 minutes, remove the object to be migrated and update the Annotation of the object to be migrated, recording the last migration time. Otherwise, select a new object to migrate; Step S214: The Kubernetes ReplicaSet Controller restarts and schedules the objects to be migrated. Step S215: Repeat steps S211 to S214, with an interval of 10 seconds.

2. The dynamic scheduling method for offline and mixed deployments according to claim 1, characterized in that, In step S2, each node is equipped with a DaemonSet type Agent for maintaining the node's interference health. The Agent continuously performs dynamic resource adjustments and offline job re-migration based on the changes in the interference health value.

3. The dynamic scheduling method for offline and mixed deployments according to claim 1, characterized in that, In step S202, the initial value is obtained from the Kubernetes API Server.

4. The dynamic scheduling method for offline and mixed deployments according to claim 1, characterized in that, In step S3, the resource monitoring data indicators include overall CPU utilization, overall memory utilization, overall memory bandwidth utilization, IPC, and disk usage percentage.

5. The dynamic scheduling method for offline mixed-use according to claim 4, characterized in that, In step S4, historical data of the past 60 seconds of resource monitoring data for each node is obtained every 10 seconds.

6. The dynamic scheduling method for offline mixed-use according to claim 5, characterized in that, The established resource mixed-distribution interference model uses a long short-term memory network as a feature extractor and takes historical resource monitoring data as input to obtain the interference health of the current node.

7. The dynamic scheduling method for offline mixed-use according to claim 6, characterized in that, In step S4, the interference health status of the current node is pushed to the Agent of each node to update the interference health status at the latest moment.

8. A dynamic scheduling system for mixed offline and online deployments, characterized in that, Perform a dynamic scheduling method for offline mixed-use as described in any one of claims 1-7; The dynamic scheduling system includes: The submission filtering module is used to submit new jobs and filter them to select nodes that meet the requirements. The execution module is used by each node to perform dynamic resource adjustments and offline job re-migration based on the changes in its own interference health value. The monitoring module is used to obtain the current resource monitoring data metrics for each node and maintain historical monitoring data. The calculation module is used to obtain the resource monitoring data indicators of each node at set intervals to calculate the current interference health of each node. The scheduling module is used to score the interference health of each node and select the node with the highest score as the scheduling node.

Images