Cluster scheduling control method based on computing power cabin

Abstract

The invention relates to the technical field of computing power cabin cluster scheduling. The invention relates to a cluster scheduling control method based on a computing power cabin. The method comprises the following steps: S1, obtaining performance data of a computing power cabin, and then carrying out resource virtualization to form a resource pool; s2, collecting historical operation data and real-time operation data of the service user, and predicting a resource demand of the service user to obtain a predicted resource demand; s3, combining the predicted resource demand with the real-time operation data to carry out predicted volatility analysis, setting an isolation fluctuation threshold value, then comparing the predicted volatility with the isolation fluctuation threshold value, and when the predicted volatility is smaller than the isolation fluctuation threshold value, judging whether the predicted volatility is smaller than the isolation fluctuation threshold value or not; according to the method, resource demand prediction is carried out by using the historical operation data and the real-time operation data, resources can be allocated to the user more accurately, and when the predicted fluctuation rate is smaller than the isolation fluctuation threshold value, the resource instance is automatically started to carry out node isolation on the user, so that the stability of resource allocation is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of computing power cabin cluster scheduling, and in particular to a cluster scheduling control method based on computing power cabin. Background Art

[0002] By deploying computing cabins in energy-rich areas, such as the western region that uses renewable energy, energy transmission losses can be significantly reduced and energy efficiency can be improved.

[0003] At present, when enterprises conduct large-scale data analysis and business processing, they use computing cabin clusters to improve data processing speed and decision-making efficiency. That is, the resources of the computing cabin are composed of multiple computing cabins (A to N cabins) interconnected to form a cluster. However, the allocation of computing cabin resources is not flexible enough, and often cannot be dynamically adjusted according to actual task requirements and user conditions, resulting in some node resources being idle while other nodes are overloaded, affecting the overall computing efficiency. Secondly, in terms of task scheduling, there is a lack of intelligent classification and adaptive strategies, which may assign inappropriate tasks to inappropriate nodes, increasing task execution time. When node failures or load imbalances occur, adjustments cannot be made in a timely and effective manner, which may lead to system performance degradation or even collapse. Therefore, a cluster scheduling control method based on computing cabins is proposed. Summary of the invention

[0004] The purpose of the present invention is to provide a cluster scheduling control method based on a computing power cabin to solve the problems raised in the above background technology.

[0005] To achieve the above purpose, a cluster scheduling control method based on a computing module is provided, comprising the following steps: S1. Obtain the performance data of the computing power cabin, and then virtualize the resources to form a resource pool; S2. Collect historical and real-time operation data of service users, and predict resource requirements of service users to obtain predicted resource requirements; S3. Combine the predicted resource demand with the real-time operation data to perform predicted volatility analysis, set the isolation volatility threshold, and then compare the predicted volatility with the isolation volatility threshold. When the predicted volatility is less than the isolation volatility threshold, automatically start the resource instance to isolate the user from the node. S4. Perform load analysis on the predicted resource demand and the corresponding node of the user. When the node is shown to be unable to bear the load, use the idle resources in the resource pool to increase the calculation of the node. S5. Combine the predicted resource demand corresponding to the node with the node to perform resource fragmentation, and combine the real-time operation data with the node to calculate the real-time load ratio. Then, generate healthy nodes according to the ratio priority of the idle resources.

[0006] As a further improvement of the present technical solution, S1 obtains the information monitoring authority and computing resource control authority of the computing cabin by deploying a monitoring and control terminal in the computing cabin, and at the same time uses the information monitoring authority and computing resource control authority to obtain the performance data of the computing cabin and the computing nodes allocated to the computing cabin, as well as the operating status of each node.

[0007] As a further improvement of the present technical solution, S1 virtualizes the computing, storage and network resources in the computing power cabin to form a unified resource pool, and then adds and adjusts resource instances in the resource pool to complete the configuration addition and quantity adjustment of the nodes.

[0008] As a further improvement of the technical solution, the steps of S2 are as follows: S2.1. Extract historical operation data and real-time operation data from the computing power cabin, and classify the historical operation data and real-time operation data according to the service users, and obtain the historical operation data and real-time operation data corresponding to each user; S2.2. Calculate the fluctuation range based on the historical operation data to obtain the fluctuation range corresponding to each user. Then use the machine learning algorithm to combine the fluctuation range and the historical operation data and real-time operation data corresponding to each user to predict the resource demand and obtain the predicted resource demand corresponding to the user.

[0009] As a further improvement of the present technical solution, the larger the fluctuation range in S2.2, the more historical operation data is intercepted, and conversely, the smaller the fluctuation range is, the less historical operation data is intercepted.

[0010] As a further improvement of the technical solution, the steps of S3 are as follows: S3.1. Forecast resource demand and combine it with real-time operation data to perform forecast volatility analysis and obtain the forecast volatility corresponding to the user; S3.2. Set the isolation fluctuation threshold, and then compare the predicted volatility with the isolation fluctuation threshold. When the predicted volatility is less than the isolation fluctuation threshold, automatically start the resource instance to isolate the user's node. Conversely, when the predicted volatility is greater than the isolation fluctuation threshold, keep multiple users sharing the node and do not isolate the node.

[0011] As a further improvement of the technical solution, the step of S4 is as follows: S4.1. Obtain the maximum value of the predicted resource demand of each user and the maximum load of each node; S4.2. Calculate the total demand based on the user list and the maximum demand of each user. Then analyze the total demand of each node in combination with the maximum load. When a node is shown to be unable to load, use the idle resources in the resource pool to increase the calculation of the node. Conversely, when a node is shown to have sufficient load, continue monitoring.

[0012] As a further improvement of the technical solution, the S4.2 further includes the following steps: S4.2.1, set the upper limit of node configuration; S4.2.2. When the total demand value is greater than the upper limit of the node configuration, use the idle resources in the resource pool to increase resource instances and migrate the user with the largest workload in the original node.

[0013] As a further improvement of the technical solution, the step of S5 is as follows: S5.1. Defragment the resources by combining the corresponding total demand value in the node with the node configuration. When the node configuration is greater than the total demand value, the overflow computing resources are sorted into idle resources. S5.2. Set the load buffer ratio, and calculate the real-time load share by combining the real-time operation data with the nodes, obtain the load share of each node, and then generate healthy nodes based on the idle resources according to the load share priority. The higher the load share, the higher the priority.

[0014] As a further improvement of the technical solution, the healthy nodes in S5.2 are configured and increased by combining the configuration of the original nodes with the load buffer ratio, thereby generating healthy nodes with higher load performance than the original nodes.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. In this cluster scheduling control method based on computing power cabin, real-time monitoring and dynamic allocation of computing power resources are realized, and resource allocation strategies are flexibly adjusted according to task requirements, thereby optimizing performance and reducing costs.

[0016] 2. In the cluster scheduling control method based on the computing power cabin, by using historical operation data and real-time operation data to predict resource demand, resources can be allocated to users more accurately. When the predicted volatility is less than the isolation volatility threshold, the resource instance is automatically started to isolate the user's node, ensuring the stability of resource allocation. When the predicted volatility is greater than the isolation volatility threshold, multiple users share the node, improving the resource sharing efficiency. At the same time, the corresponding total demand value in the node is combined with the node configuration to defragment the resources, and the overflowed computing resources are sorted into idle resources, further improving the resource utilization efficiency.

[0017] 3. In the cluster scheduling control method based on the computing power cabin, the real-time load share is calculated by combining real-time operation data to obtain the load share of each node. At the same time, healthy nodes are generated according to the load share priority. When the system load is high, healthy nodes with higher load performance than the original nodes can be generated in time, thereby improving the reliability and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a schematic diagram of the structure of the computing module of the present invention; Figure 2 It is the overall flow chart of the present invention; Figure 3 A flowchart of the present invention for obtaining historical operation data and real-time operation data corresponding to each user; Figure 4 A flowchart of the present invention for obtaining the predicted volatility corresponding to the user; Figure 5 The present invention is a flowchart of obtaining the maximum load value of each node.

[0019] Figure 6 A flow chart of generating a healthy node with a higher load performance than the original node for the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] See also Figure 1-Figure 6 As shown, the purpose of this embodiment is to provide a cluster scheduling control method based on a computing power cabin, comprising the following steps: S1. Obtain the performance data of the computing power cabin, and then virtualize the resources to form a resource pool; S1 obtains the information monitoring authority and computing resource control authority of the computing cabin by deploying a monitoring and control terminal in the computing cabin. At the same time, it uses the information monitoring authority and computing resource control authority to obtain the performance data of the computing cabin and the computing nodes allocated to the computing cabin, as well as the operating status of each node.

[0022] Deploy a comprehensive monitoring system to track the status of computing resources in real time, including key indicators such as CPU usage, memory usage, storage space, and network bandwidth.

[0023] S1 virtualizes the computing, storage, and network resources in the computing power cabin to form a unified resource pool, and then adds and adjusts resource instances in the resource pool to complete the configuration increase and quantity adjustment of nodes. The specific steps are as follows: Resource virtualization: Virtualize computing resources such as CPU, GPU, storage resources such as hard disk, memory, and network resources in the computing power cabin. Use virtualization technologies such as virtual machine hypervisor and container technology to abstract physical resources into virtual resources to form a unified resource pool. Resource pool management: Establish a resource pool management system to monitor and manage virtual resources in the resource pool and monitor resource usage, availability, and performance indicators in real time; Adding resource instances: When you need to increase node configuration, determine the required resource type and quantity, select the appropriate virtual resource instance from the resource pool, and assign it to a specific node, such as allocating more CPU cores or GPU resources to the node; Adjust resource instances: Adjust resource instances in the resource pool according to actual needs, increase or decrease resource instances of specific types to meet the changing needs of different nodes.

[0024] S2. Collect historical and real-time operation data of service users, and predict resource requirements of service users to obtain predicted resource requirements; The steps of S2 are as follows: S2.1. Extract historical operation data and real-time operation data from the computing power cabin, and classify the historical operation data and real-time operation data according to the service users, and obtain the historical operation data and real-time operation data corresponding to each user; Historical operation data includes task execution status, resource usage, etc. over a period of time.

[0025] Real-time operation data can be obtained through real-time monitoring tools, such as the progress of the current task, resource usage, etc.

[0026] Historical operation data and real-time operation data are classified according to the identification information of the service user, such as user ID, user name, etc.

[0027] S2.2. Calculate the fluctuation range based on the historical operation data to obtain the fluctuation range corresponding to each user. Then use the machine learning algorithm to combine the fluctuation range and the historical operation data and real-time operation data corresponding to each user to predict resource demand and obtain the predicted resource demand corresponding to the user. The larger the fluctuation range in S2.2, the more historical operation data will be intercepted. Conversely, the smaller the fluctuation range, the less historical operation data will be intercepted. The specific steps are as follows: For each user's historical operation data, select key indicators and use the coefficient of variation to measure the fluctuation range. The formula is as follows:

[0028] Among them, P is the standard deviation and K is the mean value;

[0029] Where N is the number of data points, X i is the value of each data point, and then the historical data is cut off at the length. The formula is as follows:

[0030] Among them, a and b are adjustable parameters. The larger the fluctuation range CV is, the more historical data will be intercepted; For each user, according to the determined historical data interception length, the corresponding data segment is extracted from the historical operation data. At the same time, the real-time operation data of the user is obtained and predicted using the machine learning algorithm. The formula is as follows:

[0031] Where y is the predicted resource demand, t i是 Each feature (including historical data features, real-time data features, fluctuation range features, etc.), w i are model parameters, which are determined by training data and then predicted.

[0032] S3. Combine the predicted resource demand with the real-time operation data to perform predicted volatility analysis, set the isolation volatility threshold, and then compare the predicted volatility with the isolation volatility threshold. When the predicted volatility is less than the isolation volatility threshold, automatically start the resource instance to isolate the user from the node. The steps for S3 are as follows: S3.1. Forecast resource demand and real-time operation data to perform forecast volatility analysis, obtain the forecast volatility corresponding to the user, and use the predicted resource demand and key indicators in the real-time operation data (such as CPU usage, memory usage, etc.) to calculate the forecast volatility using the standard deviation method; S3.2, set the isolation fluctuation threshold, and then compare the predicted fluctuation rate with the isolation fluctuation threshold. When the predicted fluctuation rate is less than the isolation fluctuation threshold, the resource instance is automatically started to isolate the user's node. On the contrary, when the predicted fluctuation rate is greater than the isolation fluctuation threshold, multiple users are kept sharing the node without node isolation. The specific steps are as follows: Setting isolation fluctuation threshold: A reasonable isolation fluctuation threshold is set by staff or based on custom analysis of historical data system; Execute corresponding operations: If the predicted volatility is less than the isolation volatility threshold, automatically start the resource instance to isolate the user's node and allocate an independent computing node from the resource pool to the user to ensure the stability of its resources; If the predicted volatility is greater than the isolation volatility threshold, keep multi-user sharing the node and do not isolate the node.

[0033] S4. Perform load analysis on the predicted resource demand and the corresponding node of the user. When the node is shown to be unable to bear the load, use the idle resources in the resource pool to increase the calculation of the node. The steps of S4 are as follows: S4.1. Obtain the maximum value of the predicted resource demand of each user and the maximum load of each node; S4.2. Calculate the total demand value based on the user list and the maximum demand value of each user. Then analyze the total demand value of each node in combination with the maximum load value. When the node is displayed as unable to load, use the idle resources in the resource pool to calculate and increase the node. On the contrary, when the node is displayed as having sufficient load, continue to monitor. The specific steps are as follows: Obtain the maximum demand and the maximum node load: For each user's predicted resource demand, determine the maximum demand. You can traverse the predicted resource demand data of each user and find the maximum value. Get the maximum load of each node by monitoring the resource usage of the node; Calculate the total demand value: Based on the user list and the maximum demand value of each user, calculate the total demand value using the following formula:

[0034] Among them, the user list is U1,U2,⋯,U n , the maximum demand of each user is M1,M2,⋯,M n , the total demand value is T; Load analysis: The total demand value of each node is combined with the maximum load value to perform load analysis. If the current maximum load value plus the total demand value exceeds the maximum carrying capacity of the node, the node is displayed as unable to carry the load. Otherwise, the node is displayed as having sufficient load. Perform corresponding operations: When the display node cannot be loaded, use the idle resources of the resource pool to increase the computing of the node, and allocate additional computing resources (such as CPU cores, memory, etc.) from the resource pool to the node to meet the demand. When the display node is sufficiently loaded, continue to monitor and re-analyze the load regularly.

[0035] S4.2 also includes the following steps: S4.2.1, set the upper limit of node configuration; Determine the configuration upper limit of each node, including restrictions on the number of CPU cores, memory capacity, storage capacity, etc. These upper limits can be set based on hardware specifications, performance requirements, and actual experience.

[0036] S4.2.2. When the total demand value is greater than the upper limit of the node configuration, use the idle resources in the resource pool to increase resource instances and migrate the user with the largest workload in the original node.

[0037] S5. Combine the predicted resource demand corresponding to the node with the node to perform resource fragmentation, and combine the real-time operation data with the node to calculate the real-time load ratio. Then, generate healthy nodes according to the ratio priority of the idle resources.

[0038] The steps of S5 are as follows: S5.1. Defragment the resources by combining the corresponding total demand value in the node with the node configuration. When the node configuration is greater than the total demand value, the overflow computing resources are sorted into idle resources. When it is found that the node configuration is greater than the total demand value, the overflow computing resources are identified, and these overflow resources are sorted and released back to the resource pool to make them idle resources. The release of resources can be achieved by adjusting the resource allocation of virtual machines or containers, shutting down unnecessary hardware devices, etc.

[0039] S5.2. Set the load buffer ratio, and calculate the real-time load share by combining the real-time operation data with the nodes, obtain the load share of each node, and then generate healthy nodes based on the idle resources according to the load share priority. The higher the load share, the higher the priority.

[0040] The healthy nodes in S5.2 are configured and increased by combining the configuration of the original nodes with the load buffer ratio, thereby generating healthy nodes with higher load performance than the original nodes. The specific steps are as follows: Set the load buffer ratio: Determine an appropriate load buffer ratio to consider a certain load margin when generating healthy nodes. This ratio can be set by staff based on experience, system performance requirements, and resource availability; Real-time load ratio calculation: Combine real-time operation data and node configuration to calculate the load ratio of each node. For a specific node, select key resource indicators (such as CPU usage, memory usage, etc.). The load ratio can be calculated by dividing the current resource usage by the total resource capacity of the node. Determine the priority of idle resources: Determine the priority of idle resource allocation based on the load ratio of each node. The higher the load ratio of the node, the higher the priority, which means that idle resources are allocated to these nodes first when generating healthy nodes. Generate healthy nodes: Use idle resources to configure and increase nodes according to priority to generate healthy nodes. The formula is as follows:

[0041] Among them, C h is the computing resource of healthy nodes, C o is the computing resource of the original node, and B is the load buffer ratio.

[0042] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A cluster scheduling control method based on a computing power cabin, characterized in that: The following steps are involved: S1. Obtain the performance data of the computing power cabin, and then virtualize the resources to form a resource pool; S2. Collect historical and real-time operation data of service users, and predict resource requirements of service users to obtain predicted resource requirements; S3. Combine the predicted resource demand with the real-time operation data to perform predicted volatility analysis, set the isolation volatility threshold, and then compare the predicted volatility with the isolation volatility threshold. When the predicted volatility is less than the isolation volatility threshold, automatically start the resource instance to isolate the user from the node. S4. Perform load analysis on the predicted resource demand and the corresponding node of the user. When the node is shown to be unable to bear the load, use the idle resources in the resource pool to increase the calculation of the node. S5. Combine the predicted resource demand corresponding to the node with the node to perform resource fragmentation, and combine the real-time operation data with the node to calculate the real-time load ratio. Then, generate healthy nodes according to the ratio priority of the idle resources.

2. The cluster scheduling control method based on computing power cabin according to claim 1 is characterized in that: The S1 obtains the information monitoring authority and computing resource control authority of the computing cabin by deploying a monitoring and control terminal in the computing cabin, and uses the information monitoring authority and computing resource control authority to obtain the performance data of the computing cabin and the computing nodes allocated to the computing cabin, as well as the operating status of each node.

3. The cluster scheduling control method based on computing power cabin according to claim 1 is characterized in that: The S1 virtualizes the computing, storage and network resources in the computing power cabin to form a unified resource pool, and then adds and adjusts resource instances in the resource pool to complete the configuration increase and quantity adjustment of the nodes.

4. The cluster scheduling control method based on computing power cabin according to claim 1 is characterized in that: The steps of S2 are as follows: S2.

1. Extract historical operation data and real-time operation data from the computing power cabin, and classify the historical operation data and real-time operation data according to the service users, and obtain the historical operation data and real-time operation data corresponding to each user; S2.

2. Calculate the fluctuation range based on the historical operation data to obtain the fluctuation range corresponding to each user. Then use the machine learning algorithm to combine the fluctuation range and the historical operation data and real-time operation data corresponding to each user to predict the resource demand and obtain the predicted resource demand corresponding to the user.

5. The cluster scheduling control method based on computing power cabin according to claim 4 is characterized in that: The larger the fluctuation range in S2.2, the more historical operation data will be intercepted. Conversely, the smaller the fluctuation range, the less historical operation data will be intercepted.

6. The cluster scheduling control method based on computing power cabin according to claim 1 is characterized in that: The steps of S3 are as follows: S3.

1. Forecast resource demand and combine it with real-time operation data to perform forecast volatility analysis and obtain the forecast volatility corresponding to the user; S3.

2. Set the isolation fluctuation threshold, and then compare the predicted volatility with the isolation fluctuation threshold. When the predicted volatility is less than the isolation fluctuation threshold, automatically start the resource instance to isolate the user's node. Conversely, when the predicted volatility is greater than the isolation fluctuation threshold, keep multiple users sharing the node and do not isolate the node.

7. The cluster scheduling control method based on computing power cabin according to claim 1 is characterized in that: The steps of S4 are as follows: S4.

1. Obtain the maximum value of the predicted resource demand of each user and the maximum load of each node; S4.

2. Calculate the total demand based on the user list and the maximum demand of each user. Then analyze the total demand of each node in combination with the maximum load. When a node is shown to be unable to load, use the idle resources in the resource pool to increase the calculation of the node. Conversely, when a node is shown to have sufficient load, continue monitoring.

8. The cluster scheduling control method based on computing power cabin according to claim 7 is characterized in that: The S4.2 further comprises the following steps: S4.2.1, set the upper limit of node configuration; S4.2.

2. When the total demand value is greater than the upper limit of the node configuration, use the idle resources in the resource pool to increase resource instances and migrate the user with the largest workload in the original node.

9. The cluster scheduling control method based on computing power cabin according to claim 1 is characterized in that: The steps of S5 are as follows: S5.

1. Defragment the resources by combining the corresponding total demand value in the node with the node configuration. When the node configuration is greater than the total demand value, the overflow computing resources are sorted into idle resources. S5.

2. Set the load buffer ratio, and calculate the real-time load share by combining the real-time operation data with the nodes, obtain the load share of each node, and then generate healthy nodes based on the idle resources according to the load share priority. The higher the load share, the higher the priority.

10. The cluster scheduling control method based on computing power cabin according to claim 9 is characterized in that: The healthy nodes in S5.2 are configured and increased by combining the configuration of the original nodes with the load buffer ratio, thereby generating healthy nodes with higher load performance than the original nodes.

Images