An intelligent computing cluster management system with intelligent scheduling
Through the intelligent computing cluster management system of intelligent scheduling, a resource demand prediction model is constructed using hybrid nonlinear regression and adaptive algorithms, combined with reinforcement learning optimization scheduling strategies, the problem of unreasonable resource allocation and load balancing in the existing technology is solved, and the problem of more Efficient resource utilization and system stability.
Patent Information
- Application Number
- CN202411217766.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-09-02
AI Technical Summary
The existing cluster management system has inefficiency in resource allocation and load balancing, and cannot accurately predict resource demand, resulting in unreasonable resource allocation and low utilization rate, and lack of dynamic feature extraction and weight adjustment mechanisms, making it difficult to adapt to task changes.
A smart computing cluster management system for intelligent scheduling is designed, including data acquisition module, resource monitoring and prediction module, intelligent resource allocation module, dynamic load balancing module and adaptive optimization scheduling strategy module, using hybrid nonlinear regression and adaptive algorithms Build a resource demand prediction model, combine reinforcement learning algorithm to optimize the scheduling strategy, and achieve load balancing through feedback control.
It improves the accuracy and load balancing effect of resource demand forecasting, enhances resource utilization, and ensures the stable operation and high reliability of the cluster management system.
Smart Images

Figure CN118760527B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to an intelligently scheduled intelligent computing cluster management system. Background Art
[0002] In existing cluster management systems, resource allocation and load balancing are two key issues. Cluster management systems usually contain multiple nodes, each of which needs to handle various complex tasks, which consume different amounts of resource requirements during execution. Traditional resource allocation and load balancing methods are mostly based on static rules and preset strategies, which are difficult to adapt to dynamically changing task requirements and load conditions, resulting in low resource utilization and system performance degradation. Existing resource demand prediction methods mainly rely on linear regression models and simple statistical methods. When dealing with resource usage patterns of complex tasks, these methods often find it difficult to capture nonlinear relationships in the data, resulting in low accuracy in resource demand prediction. In addition, traditional methods lack effective feature extraction and weight adjustment mechanisms, and cannot dynamically adapt to changes in tasks, further affecting the rationality of resource allocation.
[0003] At the same time, the existing cluster management system still has the problem of inaccurate data processing and is unable to obtain accurate feature data, resulting in low accuracy in resource demand prediction, which further causes technical problems such as unreasonable resource allocation and low utilization. Summary of the invention
[0004] The present invention provides an intelligently scheduled intelligent computing cluster management system to solve the technical problems that in the intelligent computing cluster management system, data processing is not accurate enough to obtain precise feature data, resulting in low accuracy in resource demand prediction, further causing unreasonable resource allocation and low utilization.
[0005] The present invention provides an intelligently scheduled intelligent computing cluster management system, which specifically includes the following technical solutions:
[0006] An intelligent scheduling intelligent computing cluster management system includes the following parts:
[0007] Data acquisition module, resource monitoring and prediction module, intelligent resource allocation module, dynamic load balancing module, adaptive optimization scheduling strategy module;
[0008] The data acquisition module acquires the real-time resource usage data of each node in the cluster management system and transmits the real-time resource usage data to the resource monitoring and prediction module;
[0009] The resource monitoring and prediction module collects resource usage data and load data of historical tasks and real-time resource usage data, and builds a resource demand prediction model based on the resource usage data and load data of historical tasks; predicts the resource demand of each task based on the real-time resource usage data, and transmits the predicted task resource demand to the intelligent resource allocation module;
[0010] The intelligent resource allocation module dynamically adjusts resource allocation based on the predicted task resource requirements and task priorities to obtain an optimized resource allocation plan, and transmits the optimized resource allocation plan to the dynamic load balancing module and the adaptive optimization scheduling strategy module;
[0011] The dynamic load balancing module monitors the real-time load data of each node in the cluster in real time, adjusts the task allocation in combination with the optimized resource allocation scheme, and adopts the feedback control mechanism to dynamically adjust the node load to achieve load balancing;
[0012] The adaptive optimization scheduling strategy module, based on load balancing and based on the optimized resource allocation plan, uses the reinforcement learning algorithm to learn and optimize the scheduling strategy, obtains the adaptively optimized scheduling strategy, and performs intelligent scheduling of the intelligent computing cluster management based on the adaptively optimized scheduling strategy.
[0013] Preferably, the resource monitoring and prediction module constructs a resource demand prediction model based on resource usage data and load condition data of historical tasks by using a hybrid nonlinear regression and adaptive algorithm.
[0014] Preferably, the resource monitoring and prediction module, in the process of implementing the hybrid nonlinear regression and adaptive algorithm, preprocesses the resource usage data of historical tasks and the load conditions of historical tasks, extracts features from the preprocessed resource usage data and load condition data of historical tasks, and adaptively adjusts the feature weights.
[0015] Preferably, the resource monitoring and prediction module generates non-linear combination features based on the adjusted feature weights.
[0016] Preferably, the resource monitoring and prediction module performs prediction processing on the nonlinear combination features through a hybrid model composed of a long short-term memory network and a convolutional neural network to obtain predicted resource requirements.
[0017] Preferably, the dynamic load balancing module performs load balancing through an intelligent adaptive load balancing algorithm; in the process of implementing the intelligent adaptive load balancing algorithm, based on the optimized resource allocation scheme and the real-time load data of the node, a load balancing optimization formula is introduced to balance the load of the node by minimizing the maximum value of the node load;
[0018] The load balancing optimization formula is as follows:
[0019]
[0020] in, It's a task At the node Predicted load on is the total number of tasks; It's a task Assigning to nodes in resource allocation optimization Result; Is a node Real-time load data; Is a node resource capacity; is the adjustment coefficient, used to adjust the influence of the sine function; Indicates the current time; It's a time cycle.
[0021] Preferably, the dynamic load balancing module introduces a feedback control mechanism during the implementation of the intelligent adaptive load balancing algorithm, and generates feedback control by calculating the error between the current load and the expected load.
[0022] Preferably, the dynamic load balancing module introduces a node load adjustment formula according to feedback control, and realizes load balancing by dynamically adjusting the node load; the node load adjustment formula is as follows:
[0023]
[0024] in, Is a node Load adjustment; is the adjustment factor, which is used to control the sensitivity of load adjustment; is the maximum load allowed by the cluster management system; is the adjustment coefficient, used to adjust the influence of the cosine function; Indicates feedback control.
[0025] Preferably, the adaptive optimization scheduling strategy module, when performing intelligent scheduling of intelligent computing cluster management, constructs a multi-level scheduling control architecture, including a global scheduling layer, a regional scheduling layer and a node scheduling layer, to handle scheduling tasks at different levels respectively.
[0026] The beneficial effects of the technical solution of the present invention are:
[0027] 1. By using hybrid nonlinear regression and adaptive algorithms, feature extraction, adaptive adjustment of feature weights and multi-layer nonlinear combination feature generation are performed on the resource usage data and load data of historical tasks, which enhances the expressiveness of the resource demand prediction model, can capture the complex relationships in the data, and improves the prediction accuracy of the resource demand prediction model for complex task resource requirements.
[0028] 2. Through the load balancing optimization formula and feedback control mechanism, the cluster management system can effectively balance the load of each node, avoid overloading or waste of resources of some nodes, and improve the overall resource utilization of the cluster; on the basis of load balancing, the cluster management system continuously optimizes the scheduling strategy through reinforcement learning algorithm to ensure the stable operation and high reliability of the cluster management system. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 This is a structural diagram of an intelligently scheduled intelligent computing cluster management system described in the present invention. DETAILED DESCRIPTION
[0030] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0031] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
[0032] The specific scheme of an intelligently scheduled intelligent computing cluster management system provided by the present invention is described in detail below with reference to the accompanying drawings.
[0033] See attached Figure 1 , which shows a structure diagram of an intelligent cluster management system for intelligent scheduling provided by an embodiment of the present invention, the system includes the following parts:
[0034] Data acquisition module, resource monitoring and prediction module, intelligent resource allocation module, dynamic load balancing module, adaptive optimization scheduling strategy module;
[0035] The data acquisition module acquires the real-time resource usage data of each node of the cluster management system through a monitor such as a resource monitoring agent, including the CPU usage, memory usage, storage usage, and network traffic of each node; and transmits the acquired real-time resource usage data to the resource monitoring and prediction module;
[0036] The resource monitoring and prediction module uses a hybrid nonlinear regression and adaptive algorithm to establish a resource demand prediction model based on the resource usage data and load data of historical tasks, and predicts the resource demand of each task based on the real-time resource usage data; the specific implementation process is as follows:
[0037] Collect resource usage data of historical tasks, including CPU usage, memory usage, storage usage, network traffic, etc. At the same time, collect the load of historical tasks, such as task type, execution time, and task priority.
[0038] Furthermore, data preprocessing is performed. In order to make the resource usage data and load data of historical tasks consistent and comparable, the above data needs to be cleaned and standardized. The process of cleaning data includes removing outliers and noise. This step can be achieved through statistical analysis and outlier detection. Statistical analysis and outlier detection methods are technical means well known to those skilled in the art and will not be described in detail here.
[0039] After data preprocessing is completed, the feature extraction phase begins. Existing feature engineering techniques are used to extract key features from the preprocessed historical task resource usage data and load data, such as task type, historical resource usage pattern, and task load characteristics.
[0040] Furthermore, after feature extraction, the feature weights are adaptively adjusted. According to the historical performance of the task and the current system status, the feature weights are dynamically adjusted to make the features more representative and accurate for prediction. The formula for adaptively adjusting feature weights is:
[0041]
[0042] in, It is The weight after the iteration is expressed in At the iteration, The input data pair The contribution of a feature; the larger the weight, the more important the input data is to the feature, and the input data is the resource usage data and load data of the pre-processed historical tasks; It is The weight after iterations; is the learning rate, which indicates the step size of weight update at each iteration; the learning rate determines the speed of weight adjustment. Too large a learning rate may cause oscillation, while too small a learning rate may cause slow convergence; It is a loss function, which is used to measure the difference between the predicted value and the true value of the resource demand prediction model. The smaller the value of the loss function, the better the prediction performance of the resource demand prediction model. The appropriate loss function is selected according to the specific problem, such as mean square error (MSE), cross entropy loss, etc. is the loss function for The weight after iteration The gradient of the loss function indicates the sensitivity of the loss function to the weight. It is calculated by the back propagation algorithm and is used to guide the update direction and amplitude of the weight. The larger the gradient, the greater the influence of the weight on the loss function. The weight is adjusted by the gradient descent method to gradually reduce the prediction error and improve the accuracy of the resource demand prediction model.
[0043] Furthermore, nonlinear combination features are generated. The expressive power of the resource demand prediction model is enhanced by generating multi-layer nonlinear combination features to capture the complex relationships in the data. The formula for generating nonlinear combination features is:
[0044]
[0045] in, It is The layer generated Features, after Features generated after layer nonlinear transformation; is the number of input features; It is Layer Input features For generating Output features The influence weight of is obtained through adaptive adjustment and optimization; the updated weight When entering the nonlinear combination feature generation stage of the next layer, it will be used as the weight of the current layer use; It is Layer Features It is Layer used to adjust the The bias term of the feature; It is Tier Nonlinear adjustment parameters of features; is an activation function (such as ReLU or Sigmoid). The above formula generates more complex features through multi-level nonlinear combinations, improving the modeling ability of the resource demand prediction model for complex relationships.
[0046] Furthermore, the features generated after the multi-layer nonlinear transformation are predicted and processed through a hybrid model composed of an existing long short-term memory network and a convolutional neural network. The hybrid model is the prediction processing part of the resource demand prediction model. Through the hybrid model, the predicted resource demand is obtained, including CPU usage demand, memory usage demand, storage usage demand, network bandwidth demand, and load demand.
[0047] Based on the real-time resource usage data, the resource demand of the task is predicted through the above resource demand prediction model; the predicted task resource demand is transmitted to the intelligent resource allocation module;
[0048] The intelligent resource allocation module dynamically adjusts resource allocation based on the predicted task resource requirements and combines task priorities using optimization algorithms such as linear programming and genetic algorithms to obtain an optimized resource allocation plan; the optimization algorithm is a technical means well known to technical personnel in this field and will not be described in detail here; the optimized resource allocation plan is transmitted to the dynamic load balancing module and the adaptive optimization scheduling strategy module.
[0049] The dynamic load balancing module monitors the real-time load data of each node in the cluster in real time, uses an intelligent adaptive load balancing algorithm, adjusts task allocation based on an optimized resource allocation scheme combined with the real-time load data of each node in the cluster, and uses a feedback control mechanism to dynamically adjust node loads to ensure load balancing. The specific implementation process is as follows:
[0050] Based on the optimized resource allocation scheme and the real-time load data of the nodes, a load balancing optimization formula is introduced to ensure the load balance of each node by minimizing the maximum value of the node load and taking into account the periodic load fluctuations. Load balancing optimization formula:
[0051]
[0052] in, It's a task At the node Predicted load on is the total number of tasks; It's a task Assigning to nodes in resource allocation optimization The result indicates the allocation status determined after optimization. The result of resource allocation optimization is a binary variable, where 1 indicates allocation and 0 indicates no allocation. Is a node Real-time load data reflects the actual resource usage of the current node; Is a node The resource capacity of the node The maximum amount of resources that can be provided is used to normalize the load of nodes so that load balancing calculations are comparable; is the adjustment factor used to adjust the effect of the sine function to ensure response to periodic load fluctuations; is a time variable, indicating the current time; is the time period, indicating the period of load fluctuation. Indicates all nodes assigned Tasks on The sum of predicted loads is used to determine which tasks are assigned to which nodes after resource allocation optimization. Indicates periodic fluctuations in load.
[0053] Furthermore, a feedback control mechanism is introduced to dynamically adjust the node load by calculating the error between the current load and the expected load to ensure the stability and efficiency of the cluster management system. The feedback control formula is used to achieve:
[0054]
[0055] in, It is feedback control, used to adjust node load; It's time The error value represents the difference between the actual load and the expected load; is the proportionality factor, used to control the direct effect of the error; is the integral coefficient, which is used to control the cumulative effect of the error; is the differential coefficient, which is used to control the effect of the error rate of change; is the quadratic term coefficient, which is used to increase the control accuracy and represents the parameter that adjusts the influence of the quadratic error; is the sinusoidal term coefficient, which is used to consider the periodic variation of the error and represents the parameter for adjusting the periodic variation of the error; is the square error change rate term coefficient, which is used to increase the adjustment of the error change rate; is the integration variable; is the integral of the error and is used to calculate the cumulative effect of the error over time; is the time derivative of the error, which is used to calculate the rate of change of the error; is the sinusoidal function of the error, which is used to take into account the periodic variation of the error; is the time derivative of the squared error and is used to calculate the rate of change of the squared error over time.
[0056] Based on feedback control , introduce the node load adjustment formula to achieve load balancing by dynamically adjusting the node load. The node load adjustment formula is:
[0057]
[0058] in, Is a node Load adjustment; is the adjustment factor, which is used to control the sensitivity of load adjustment; It is the maximum load allowed by the cluster management system and the configured maximum load value, which is used to regulate and constrain load adjustment; is the adjustment coefficient, used to adjust the influence of the cosine function; It's time The periodic cosine function of reflects the periodic change of the load, and the time period 𝑇 is determined by the operation cycle of the cluster management system;
[0059] Furthermore, the node load is dynamically adjusted based on the current load data plus the calculated load adjustment amount to achieve load balancing;
[0060] The adaptive optimization scheduling strategy module is based on the optimized resource allocation plan and load balancing. It uses the existing reinforcement learning algorithm to continuously learn and optimize the scheduling strategy to obtain the adaptively optimized scheduling strategy. Based on the adaptively optimized scheduling strategy, it realizes intelligent scheduling of intelligent computing cluster management.
[0061] For the intelligent computing cluster management system of intelligent scheduling, a multi-level scheduling control architecture is constructed, including the global scheduling layer, the regional scheduling layer and the node scheduling layer, which respectively handle the scheduling tasks at different levels. The global scheduling layer is responsible for the overall resource allocation and global optimization. By analyzing and integrating the data from each module, the optimal resource allocation plan is formulated and passed to the regional scheduling layer and the node scheduling layer. The regional scheduling layer specifically manages the resource allocation and scheduling in each area according to the optimal resource allocation plan provided by the global scheduling layer to ensure the efficient utilization and load balancing of resources in the area. At the same time, the regional scheduling layer also needs to provide real-time feedback on the resource usage and load data in the area to the global scheduling layer for further optimization of the global strategy. The node scheduling layer is responsible for the specific execution of scheduling tasks, and performs task allocation and resource management according to the instructions of the regional scheduling layer to ensure that the resource utilization and performance of each node are in the best state. In each scheduling layer of the multi-level scheduling control architecture, the above implementation steps are implemented in combination with the specific situation of the scheduling layer. Through close cooperation, each level forms a complete closed-loop system, so that resource allocation and scheduling can be optimally configured at each level, and finally ensure the stable operation of the cluster management system and the improvement of the overall performance.
[0062] In summary, an intelligent scheduling and intelligent computing cluster management system has been completed.
[0063] The order of the embodiments of the invention is for description only and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0064] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.
[0065] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.
Claims
1. An intelligent scheduling intelligent computing cluster management system, characterized in that: Includes the following sections: Data acquisition module, resource monitoring and prediction module, intelligent resource allocation module, dynamic load balancing module, adaptive optimization scheduling strategy module; The data acquisition module acquires the real-time resource usage data of each node in the cluster management system and transmits the real-time resource usage data to the resource monitoring and prediction module; The resource monitoring and prediction module collects resource usage data and load data of historical tasks and real-time resource usage data, and builds a resource demand prediction model based on the resource usage data and load data of historical tasks by using a hybrid nonlinear regression and adaptive algorithm. In the process of implementing the hybrid nonlinear regression and adaptive algorithm, the resource usage data and load data of historical tasks are preprocessed, and feature extraction is performed on the preprocessed resource usage data and load data of historical tasks, and the feature weight is adaptively adjusted. The formula for adaptively adjusting the feature weight is: in, It is The weight after the iteration is expressed in At the iteration, The input data pair The contribution of each feature; It is The weight after iterations; is the learning rate; is the loss function; is the loss function for The weight after iteration The gradient of Based on the adjusted feature weights, nonlinear combination features are generated. The specific formula is: in, It is The layer generated Features is the number of input features; It is Layer Input features For generating Output features The weight of influence; It is Layer used to adjust the The bias term of the feature; It is Tier Nonlinear adjustment parameters of features; is the activation function; The nonlinear combination features are predicted and processed through a hybrid model based on long short-term memory network and convolutional neural network to obtain predicted resource requirements, and the predicted task resource requirements are transmitted to the intelligent resource allocation module; The intelligent resource allocation module dynamically adjusts resource allocation based on the predicted task resource requirements and task priorities to obtain an optimized resource allocation plan, and transmits the optimized resource allocation plan to the dynamic load balancing module and the adaptive optimization scheduling strategy module; The dynamic load balancing module monitors the real-time load data of each node in the cluster in real time, adjusts the task allocation in combination with the optimized resource allocation scheme, and adopts the feedback control mechanism to dynamically adjust the node load to achieve load balancing; The adaptive optimization scheduling strategy module, based on load balancing and based on the optimized resource allocation plan, uses the reinforcement learning algorithm to learn and optimize the scheduling strategy, obtains the adaptively optimized scheduling strategy, and performs intelligent scheduling of the intelligent computing cluster management based on the adaptively optimized scheduling strategy.
2. The intelligent computing cluster management system for intelligent scheduling according to claim 1 is characterized in that: The dynamic load balancing module performs load balancing through an intelligent adaptive load balancing algorithm. In the process of implementing the intelligent adaptive load balancing algorithm, a load balancing optimization formula is introduced based on an optimized resource allocation scheme and real-time load data of nodes to balance the load of nodes by minimizing the maximum value of the node load. The load balancing optimization formula is as follows: in, It's a task At the node Predicted load on is the total number of tasks; It's a task Assigning to nodes in resource allocation optimization Result; Is a node Real-time load data; Is a node resource capacity; is the adjustment coefficient, used to adjust the influence of the sine function; Indicates the current time; It's a time cycle.
3. The intelligent computing cluster management system for intelligent scheduling according to claim 1 is characterized in that: The dynamic load balancing module introduces a feedback control mechanism during the implementation of the intelligent adaptive load balancing algorithm, and generates feedback control by calculating the error between the current load and the expected load.
4. The intelligent computing cluster management system for intelligent scheduling according to claim 3 is characterized in that: The dynamic load balancing module introduces a node load adjustment formula according to feedback control, and realizes load balancing by dynamically adjusting the node load; the node load adjustment formula is as follows: in, Is a node Load adjustment; is the adjustment factor, which is used to control the sensitivity of load adjustment; is the maximum load allowed by the cluster management system; is the adjustment coefficient, used to adjust the influence of the cosine function; Indicates feedback control.
5. The intelligent computing cluster management system for intelligent scheduling according to claim 1 is characterized in that: The adaptive optimization scheduling strategy module, when performing intelligent scheduling of intelligent computing cluster management, builds a multi-level scheduling control architecture, including a global scheduling layer, a regional scheduling layer and a node scheduling layer, to handle scheduling tasks at different levels respectively.
