A Cluster Scheduling and Control Method Based on a Computing Power Cabin

Abstract

The present invention relates to the technical field of cluster scheduling of computing power cabins. The present invention relates to a cluster scheduling control method based on computing power cabins. It includes the following steps: S1, obtain the performance data of the computing power cabins, and then perform resource virtualization to form a resource pool; S2, collect the historical operation data and real-time operation data of service users, and at the same time predict the resource requirements of service users to obtain the predicted resource requirements; S3, perform prediction volatility analysis by combining the predicted resource requirements with the real-time operation data, set an isolation volatility threshold, and then compare the prediction volatility with the isolation volatility threshold. When the prediction volatility is less than the isolation volatility threshold; by using the historical operation data and real-time operation data for resource requirement prediction, the present invention can allocate resources to users more accurately. When the prediction volatility is less than the isolation volatility threshold, the resource instance is automatically started to isolate the users by nodes, ensuring the stability of resource allocation.

Description

Technical Field

[0001] The present invention relates to the technical field of computing power cabin cluster scheduling, and specifically, to a cluster scheduling control method based on a computing power cabin. Background Art

[0002] By deploying computing power cabins in areas rich in energy, such as the western regions using renewable energy, it is possible to significantly reduce energy transmission losses and improve energy efficiency.

[0003] Currently, when enterprises conduct large-scale data analysis and business processing, they rely on a computing power cabin cluster to improve data processing speed and decision-making efficiency. That is, the resources of the computing power cabin are the resources of a cluster interconnected by multiple computing power cabins (from A to N cabins). However, the resource allocation of the computing power cabin is not flexible enough and often cannot be dynamically adjusted according to actual task requirements and user situations, resulting in idle resources on some nodes while other nodes are overloaded, affecting the overall computing efficiency. Secondly, in terms of task scheduling, there is a lack of intelligent classification and adaptive strategies, which may assign inappropriate tasks to inappropriate nodes, increasing task execution time. When node failures or load imbalances occur, they cannot be adjusted in a timely and effective manner, which may lead to a decline in system performance or even system crashes. Therefore, a cluster scheduling control method based on a computing power cabin is proposed. Summary of the Invention

[0004] The purpose of the present invention is to provide a cluster scheduling control method based on a computing power cabin to solve the problems raised in the above background art.

[0005] To achieve the above purpose, a cluster scheduling control method based on a computing power cabin is provided, including the following steps:

[0006] S1. Obtain the performance data of the computing power cabin, and then perform resource virtualization to form a resource pool;

[0007] S2. Collect the historical operation data and real-time operation data of service users, and at the same time predict the resource requirements of service users to obtain predicted resource requirements;

[0008] S3. Combine the predicted resource requirements with the real-time operation data for prediction volatility analysis, set an isolation volatility threshold, and then compare the predicted volatility with the isolation volatility threshold. When the predicted volatility is less than the isolation volatility threshold, automatically start a resource instance to isolate the user on a node;

[0009] S4. Combine the predicted resource requirements with the corresponding node of the user for load analysis. When it shows that the node cannot handle the load, use the idle resources in the resource pool to increase the computing power of the node;

[0010] S5. Combine the corresponding predicted resource requirements within the node with the node for resource fragmentation, and at the same time combine the real-time operation data with the node to calculate the real-time load occupancy ratio. Then generate healthy nodes based on the priority of the occupancy ratio for the idle resources.

[0011] As a further improvement of this technical solution, in S1, by deploying a monitoring and control terminal in the computing power cabin, the information monitoring permission and computing resource control permission of the computing power cabin are obtained. At the same time, the performance data of the computing power cabin, the computing nodes allocated by the computing power cabin, and the operating status of each node are obtained by using the information monitoring permission and computing resource control permission.

[0012] As a further improvement of this technical solution, in S1, the computing, storage, and network resources in the computing power cabin are virtualized to form a unified resource pool. Then, resource instances are added and adjusted in the resource pool to complete the addition of node configurations and the adjustment of the number of nodes.

[0013] As a further improvement of this technical solution, the steps of S2 are as follows:

[0014] S2.1. Extract historical operation data and real-time operation data in the computing power cabin, and classify the historical operation data and real-time operation data according to service users to obtain the historical operation data and real-time operation data corresponding to each user.

[0015] S2.2. Calculate the fluctuation range based on the historical operation data to obtain the fluctuation range corresponding to each user. Then, use machine learning algorithms to combine the fluctuation range, the historical operation data, and the real-time operation data corresponding to each user to predict the resource requirements and obtain the predicted resource requirements corresponding to the user.

[0016] As a further improvement of this technical solution, in S2.2, the larger the fluctuation range, the more historical operation data is intercepted. On the contrary, when the fluctuation range is smaller, the less historical operation data is intercepted.

[0017] As a further improvement of this technical solution, the steps of S3 are as follows:

[0018] S3.1. Combine the predicted resource requirements with the real-time operation data for predicted volatility analysis to obtain the predicted volatility corresponding to the user.

[0019] S3.2. Set the isolation fluctuation threshold, and then compare the predicted volatility with the isolation fluctuation threshold. When the predicted volatility is less than the isolation fluctuation threshold, automatically start the resource instance to isolate the user to a node. On the contrary, when the predicted volatility is greater than the isolation fluctuation threshold, keep multiple users sharing the node without node isolation.

[0020] As a further improvement of this technical solution, the steps of S4 are as follows:

[0021] S4.1. Obtain the maximum demand value in the predicted resource requirements of each user, and at the same time obtain the maximum load value of each node;

[0022] S4.2. Calculate the total demand value according to the user list and the maximum demand value corresponding to each user, and then combine the total demand value of each node with the maximum load value for load analysis. When it shows that the node cannot bear the load, use the idle resources in the resource pool to increase the calculation of this node. On the contrary, when it shows that the node has sufficient load, continue to monitor.

[0023] As a further improvement of this technical solution, step S4.2 further includes the following steps:

[0024] S4.2.1. Set the configuration upper limit of the node;

[0025] S4.2.2. When the total demand value is greater than the node configuration upper limit, use the idle resources in the resource pool to increase resource instances, and migrate the users with the maximum workload in the original node.

[0026] As a further improvement of this technical solution, the steps of S5 are as follows:

[0027] S5.1. Combine the corresponding total demand value in the node with the node configuration for resource fragmentation. When the node configuration is greater than the total demand value, organize the overflow computing resources into idle resources;

[0028] S5.2. Set the load buffer ratio, and at the same time calculate the real-time load occupancy ratio by combining the real-time running data with the node to obtain the load occupancy ratio of each node. Then generate healthy nodes according to the priority of the load occupancy ratio for the idle resources. The higher the load occupancy ratio, the higher the priority.

[0029] As a further improvement of this technical solution, the healthy nodes in S5.2 are configured and increased by combining the configuration of the original node with the load buffer ratio, so as to generate healthy nodes with higher load performance than the original node.

[0030] Compared with the prior art, the beneficial effects of the present invention are:

[0031] 1. In this cluster scheduling control method based on the computing power cabin, real-time monitoring and dynamic allocation of computing power resources are realized, and the resource allocation strategy is flexibly adjusted according to task requirements, so as to optimize performance and reduce costs.

[0032] 2. In the cluster scheduling control method based on the computing power cabin, by using historical operation data and real-time operation data for resource demand prediction, resources can be allocated to users more accurately. When the prediction volatility is less than the isolation volatility threshold, resource instances are automatically started to isolate users to ensure the stability of resource allocation. When the prediction volatility is greater than the isolation volatility threshold, multiple users share nodes, improving the resource sharing efficiency. At the same time, the total corresponding demand value within the node is combined with the node configuration for resource fragmentation, and the overflow computing resources are sorted into idle resources, further improving the resource utilization efficiency.

[0033] 3. In the cluster scheduling control method based on the computing power cabin, by combining real-time operation data to calculate the real-time load percentage, the load percentage of each node is obtained. At the same time, healthy nodes are generated according to the priority of the load percentage, and healthy nodes with higher load performance than the original nodes can be generated in a timely manner when the system load is high, improving the reliability and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a schematic diagram of the computing power cabin structure of the present invention;

[0035] Figure 2 is a block diagram of the overall process of the present invention;

[0036] Figure 3 is a block diagram of the process of obtaining historical operation data and real-time operation data corresponding to each user of the present invention;

[0037] Figure 4 is a block diagram of the process of obtaining the prediction volatility corresponding to the user of the present invention;

[0038] Figure 5 is a block diagram of the process of obtaining the maximum load value of each node of the present invention.

[0039] Figure 6 is a block diagram of the process of generating healthy nodes with higher load performance than the original nodes of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0041] Please refer to Figures 1 - 6 as shown, the purpose of this embodiment is to provide a cluster scheduling control method based on a computing power cabin, including the following steps:

[0042] S1. Obtain the performance data of the computing power cabin, and then perform resource virtualization to form a resource pool;

[0043] In S1, by deploying a monitoring and control terminal in the computing power cabin, the information monitoring permission and computing resource control permission of the computing power cabin are obtained. At the same time, the performance data of the computing power cabin, the computing nodes allocated by the computing power cabin, and the running status of each node are obtained by using the information monitoring permission and computing resource control permission.

[0044] Deploy a comprehensive monitoring system to track the status of computing resources in real time, including key metrics such as CPU usage, memory occupancy, storage space, and network bandwidth.

[0045] In S1, by virtualizing the computing, storage, and network resources in the computing power cabin to form a unified resource pool, and then adding and adjusting resource instances in the resource pool, the configuration addition and quantity adjustment of nodes are completed. The specific steps are as follows:

[0046] Resource virtualization: Virtualize the computing resources such as CPU, GPU, etc., storage resources such as hard disks, memory, etc., and network resources in the computing power cabin. Use virtualization technologies such as hypervisors and container technologies to abstract physical resources into virtual resources to form a unified resource pool;

[0047] Resource pool management: Establish a resource pool management system to monitor and manage the virtual resources in the resource pool, and real-time monitor the usage, availability, and performance metrics of the resources;

[0048] Adding resource instances: When it is necessary to increase the node configuration, determine the required resource types and quantities, select appropriate virtual resource instances from the resource pool, and allocate them to specific nodes, such as allocating more CPU cores or GPU resources to the nodes;

[0049] Adjusting resource instances: According to the actual needs, adjust the resource instances in the resource pool, increase or decrease the resource instances of specific types to meet the changing needs of different nodes.

[0050] S2. Collect the historical operation data and real-time operation data of service users, and at the same time predict the resource requirements of service users to obtain the predicted resource requirements;

[0051] The steps of S2 are as follows:

[0052] S2.1. Extract the historical operation data and real-time operation data in the computing power cabin, and classify the historical operation data and real-time operation data according to service users to obtain the historical operation data and real-time operation data corresponding to each user;

[0053] The historical operation data includes the task execution situation, resource usage situation, etc. in the past period of time.

[0054] Real-time operation data can be obtained through real-time monitoring tools, such as the progress of the current task, resource occupancy, etc.

[0055] Classify historical operation data and real-time operation data according to the identification information of service users, such as user ID, user name, etc.

[0056] S2.2. Calculate the fluctuation range based on the historical operation data, obtain the fluctuation range corresponding to each user, and then use machine learning algorithms to combine the fluctuation range, the historical operation data and the real-time operation data corresponding to each user to predict the resource requirements, and obtain the predicted resource requirements corresponding to the user;

[0057] In S2.2, the greater the fluctuation range, the more historical operation data is intercepted. On the contrary, when the fluctuation range is smaller, the less historical operation data is intercepted. The specific steps are as follows:

[0058] For the historical operation data of each user, select key indicators, and the coefficient of variation can be used to measure the fluctuation range. The formula is as follows:

[0059]

[0060] Among them, P is the standard deviation and K is the average value;

[0061]

[0062] Among them, N is the number of data points, X i is the value of each data point, and then the length of the intercepted historical data, the formula is as follows:

[0063]

[0064] Among them, a and b are adjustable parameters. When the fluctuation range CV is larger, more historical data is intercepted;

[0065] For each user, according to the determined length of the intercepted historical data, extract the corresponding data segment from the historical operation data. At the same time, obtain the real-time operation data of the user and use machine learning algorithms for prediction. The formula is as follows:

[0066]

[0067] Among them, y is the predicted resource requirement, t i是 each feature (including historical data features, real-time data features, fluctuation range features, etc.), w i are model parameters. Determine the model parameters through training data and then make predictions.

[0068] S3. Combine the predicted resource requirements with the real-time operation data for predicted volatility analysis, set the isolation volatility threshold, and then compare the predicted volatility with the isolation volatility threshold. When the predicted volatility is less than the isolation volatility threshold, automatically start a resource instance to isolate the user to a node;

[0069] The steps of S3 are as follows:

[0070] S3.1. Combine the predicted resource requirements with the real-time operation data for predicted volatility analysis to obtain the predicted volatility corresponding to the user. Using the predicted resource requirements and key metrics in the real-time operation data (such as CPU usage rate, memory usage rate, etc.), the standard deviation method can be used to calculate the predicted volatility;

[0071] S3.2. Set the isolation volatility threshold, and then compare the predicted volatility with the isolation volatility threshold. When the predicted volatility is less than the isolation volatility threshold, automatically start a resource instance to isolate the user to a node. On the contrary, when the predicted volatility is greater than the isolation volatility threshold, keep multiple users sharing the node without node isolation. The specific steps are as follows:

[0072] Set the isolation volatility threshold: It is set by the staff or custom analyzed by the historical data system to set a reasonable isolation volatility threshold;

[0073] Execute the corresponding operation: If the predicted volatility is less than the isolation volatility threshold, automatically start a resource instance to isolate the user to a node, and allocate an independent computing node from the resource pool to the user to ensure the stability of its resources;

[0074] If the predicted volatility is greater than the isolation volatility threshold, keep multiple users sharing the node without node isolation.

[0075] S4. Combine the predicted resource requirements with the nodes corresponding to the users for load analysis. When it shows that the node cannot handle the load, use the idle resources in the resource pool to increase the computing power of the node;

[0076] The steps of S4 are as follows:

[0077] S4.1. Obtain the maximum value of the requirements in the predicted resource requirements of each user, and at the same time obtain the maximum load of each node;

[0078] S4.2. Calculate the total demand according to the user list and the maximum value of the requirements corresponding to each user, and then combine the total demand of each node with the maximum load for load analysis. When it shows that the node cannot handle the load, use the idle resources in the resource pool to increase the computing power of the node. On the contrary, when it shows that the node has enough load, keep continuous monitoring. The specific steps are as follows:

[0079] Obtain the maximum demand value and the maximum node load: For the predicted resource demands of each user, determine the maximum demand value among them. You can traverse the predicted resource demand data of each user to find the maximum value;

[0080] Obtain the maximum load value of each node, which can be obtained by monitoring the resource usage of the node;

[0081] Calculate the total demand value: According to the user list, combined with the maximum demand value corresponding to each user, calculate the total demand value. The formula is as follows:

[0082]

[0083] Among them, the user list is U1, U2, ⋯, U n , and the maximum demand values of each user are M1, M2, ⋯, M n , and the total demand value is T;

[0084] Conduct load analysis: Combine the total demand value of each node with the maximum load value for load analysis. If the sum of its current maximum load value and the total demand value exceeds the maximum carrying capacity of the node, it indicates that the node cannot handle the load. Otherwise, it indicates that the node has sufficient load;

[0085] Execute corresponding operations: When it is shown that the node cannot handle the load, use the idle resources in the resource pool to increase the calculation capacity of the node, and allocate additional calculation resources (such as CPU cores, memory, etc.) from the resource pool to the node to meet the demand. When it is shown that the node has sufficient load, continue to monitor and re-conduct load analysis regularly.

[0086] S4.2 also includes the following steps:

[0087] S4.2.1. Set the configuration upper limit of the node;

[0088] Determine the configuration upper limit of each node, including limitations in aspects such as the number of CPU cores, memory capacity, and storage capacity. These upper limit values can be set according to hardware specifications, performance requirements, and practical experience.

[0089] S4.2.2. When the total demand value is greater than the node configuration upper limit, use the idle resources in the resource pool to increase resource instances and migrate the user with the maximum workload in the original node.

[0090] S5. Combine the corresponding predicted resource demands within the node with the node for resource fragmentation, and at the same time combine the real-time operation data with the node for calculating the real-time load occupancy ratio. Then generate healthy nodes based on the occupancy ratio priority of the idle resources.

[0091] The steps of S5 are as follows:

[0092] S5.1. Combine the total corresponding requirements within the node with the node configuration for resource fragmentation. When the node configuration is greater than the total requirements, organize the overflow computing resources into idle resources.

[0093] When it is found that the node configuration is greater than the total requirements, identify the overflow computing resources, organize and release these overflow resources back to the resource pool to make them idle resources. Resource release can be achieved by adjusting the resource allocation of virtual machines or containers, turning off unnecessary hardware devices, etc.

[0094] S5.2. Set the load buffer ratio, and at the same time calculate the real-time load ratio by combining the real-time running data with the node to obtain the load ratio of each node. Then generate healthy nodes from the idle resources according to the load ratio priority. The higher the load ratio, the higher the priority.

[0095] The healthy nodes in S5.2 are configured and increased by combining the configuration of the original node with the load buffer ratio, so as to generate healthy nodes with higher load performance than the original node. The specific steps are as follows:

[0096] Set the load buffer ratio: Determine a suitable load buffer ratio for considering a certain load margin when generating healthy nodes. This ratio can be set by the staff according to experience, system performance requirements, and resource availability.

[0097] Calculate the real-time load ratio: Combine the real-time running data and the node configuration to calculate the load ratio of each node. For a specific node, select key resource indicators (such as CPU usage rate, memory usage rate, etc.). The load ratio can be calculated by dividing the current resource usage by the total resource capacity of the node.

[0098] Determine the priority of idle resources: Determine the allocation priority of idle resources according to the load ratio of each node. The higher the load ratio of the node, the higher the priority, which means that idle resources are preferentially allocated to these nodes when generating healthy nodes.

[0099] Generate healthy nodes: Use the idle resources to increase the configuration of the nodes according to the priority to generate healthy nodes. The formula is as follows:

[0100]

[0101] Among them, C h is the computing resource of the healthy node, C o is the computing resource of the original node, and B is the load buffer ratio.

[0102] The foregoing has shown and described the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and the descriptions in the specification are only preferred examples of the present invention, and are not used to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. A cluster scheduling and control method based on a computing power cabin, characterized in that: It includes the following steps: S1. Obtain the performance data of the computing power cabin, and then perform resource virtualization to form a resource pool; S2. Collect the historical operation data and real-time operation data of users, and at the same time predict the resource requirements of users to obtain the predicted resource requirements; S3. Combine the predicted resource requirements with the real-time operation data for predicted volatility analysis, set the isolation volatility threshold, and then compare the predicted volatility with the isolation volatility threshold. When the predicted volatility is less than the isolation volatility threshold, automatically start the resource instance to isolate the user to a node; S4. Combine the predicted resource requirements with the corresponding node of the user for load analysis. When it shows that the node cannot bear the load, use the idle resources in the resource pool to increase the calculation of the node; S5. Combine the corresponding predicted resource requirements in the node with the node for resource fragmentation, and at the same time combine the real-time operation data with the node for real-time load ratio calculation, and then generate healthy nodes according to the ratio priority of the idle resources.

2. The cluster scheduling control method based on the computing power cabin according to claim 1, wherein: In S1, by deploying a monitoring and control terminal in the computing power cabin, the information monitoring permission and computing resource control permission of the computing power cabin are obtained. At the same time, the performance data of the computing power cabin, the computing nodes allocated by the computing power cabin, and the operating status of each node are obtained by using the information monitoring permission and computing resource control permission.

3. The cluster scheduling control method based on the computing power cabin according to claim 1, wherein: In S1, by virtualizing the computing, storage, and network resources in the computing power cabin to form a unified resource pool, and then adding and adjusting resource instances in the resource pool, the configuration addition and quantity adjustment of the nodes are completed.

4. The cluster scheduling control method based on a computing power cabin according to claim 1, wherein: The steps of S2 are as follows: S2.

1. Extract the historical operation data and real-time operation data in the computing power cabin, and classify the historical operation data and real-time operation data according to users to obtain the historical operation data and real-time operation data corresponding to each user; S2.

2. Calculate the fluctuation range according to the historical operation data to obtain the fluctuation range corresponding to each user, and then use machine learning algorithms to combine the fluctuation range, the historical operation data and real-time operation data corresponding to each user for resource requirement prediction to obtain the predicted resource requirements corresponding to the users.

5. The cluster scheduling control method based on a computing power cabin according to claim 4, wherein: In S2.2, the larger the fluctuation range, the more historical operation data is intercepted. On the contrary, when the fluctuation range is smaller, the less historical operation data is intercepted.

6. The cluster scheduling control method based on a computing power module according to claim 1, wherein: The steps of S3 are as follows: S3.

1. Combine the predicted resource requirements with the real-time operation data for predicted volatility analysis to obtain the predicted volatility corresponding to the user; S3.

2. Set the isolation volatility threshold, and then compare the predicted volatility with the isolation volatility threshold. When the predicted volatility is less than the isolation volatility threshold, automatically start the resource instance to isolate the user to a node. On the contrary, when the predicted volatility is greater than the isolation volatility threshold, keep multiple users sharing the node without node isolation.

7. The cluster scheduling control method based on the computing power cabin according to claim 1, characterized in that: The steps of S4 are as follows: S4.

1. Obtain the maximum demand value in the predicted resource requirements of each user, and at the same time obtain the maximum load value of each node; S4.

2. Calculate the total demand value according to the user list and the maximum demand value corresponding to each user, and then analyze the total demand value of each node in combination with the maximum load. When it is shown that a node cannot bear the load, use the idle resources in the resource pool to increase the calculation capacity of this node. On the contrary, when it is shown that a node has sufficient load capacity, continue to monitor.

8. The cluster scheduling control method based on a computing power cabin according to claim 7, wherein: The S4.2 further includes the following steps: S4.2.

1. Set the configuration upper limit of the node; S4.2.

2. When the total demand value is greater than the node configuration upper limit, use the idle resources in the resource pool to increase resource instances, and migrate the users with the maximum workload in the original node.

9. The cluster scheduling control method based on a computing power module according to claim 8, wherein: The steps of S5 are as follows: S5.

1. Combine the corresponding total demand value in the node with the node configuration to perform resource fragmentation. When the node configuration is greater than the total demand value, organize the overflow calculation resources into idle resources; S5.

2. Set the load buffer ratio, and at the same time calculate the real-time load occupancy ratio by combining the real-time running data with the node to obtain the load occupancy ratio of each node. Then generate healthy nodes from the idle resources according to the priority of the load occupancy ratio. The higher the load occupancy ratio, the higher the priority.

10. The cluster scheduling control method based on a computing power cabin according to claim 9, characterized in that: The healthy nodes in S5.2 are configured and increased based on the configuration of the original node combined with the load buffer ratio, so as to generate healthy nodes with higher load performance than the original node.

Images