Intelligent task scheduling and load balancing regulation and control method and system for data center

Abstract

The invention discloses an intelligent task scheduling and load balancing regulation and control method for a data center, and the method comprises the steps: 1, the construction of a learning model, 2, the construction of a computing power demand model, 3, the preliminary construction of a balancing regulation and control system, 4, the improvement of the balancing regulation and control system, and 5, the introduction and application of the system. The system comprises a data acquisition system, a data processing system and a distribution scheduling system. According to the method, adaptive learning adjustment and optimization are carried out on the task scheduling strategy of the data center by constructing the adaptive updating deep reinforcement learning model, the task scheduling efficiency is gradually improved, tasks are intelligently allocated on the basis of considering renewable energy source supply and electricity price fluctuation, stable resource utilization is ensured, and the task scheduling efficiency is improved. And meanwhile, the execution time which can be delayed by the tasks is fully considered, and the emergency tasks are preferentially allocated, so that the reduction of the overall energy consumption of the data center and the optimization and improvement of the performance are realized, and a better intelligent regulation and control direction is provided for the balance of resource utilization and energy efficiency management of the data center.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent scheduling of data centers, and in particular to a method and system for intelligent task scheduling and load balancing control of data centers. Background Art

[0002] With the development of cloud computing and big data technology, data centers are facing increasing task processing demands and complex resource management challenges. Traditional task scheduling and load balancing methods often rely on fixed rules or simple algorithms, which are difficult to adapt to dynamically changing workloads and diversified service requirements. At the same time, they also fail to take into account the server load status and energy supply status. Some high-load servers repeatedly perform high-computing power demand tasks, which will greatly reduce the computing performance of the data center and cause a significant increase in energy consumption. Therefore, the present invention proposes a data center intelligent task scheduling and load balancing control method and system to solve the problems existing in the prior art. Summary of the invention

[0003] In view of the above problems, the purpose of the present invention is to propose a method and system for intelligent task scheduling and load balancing control of a data center. The method for intelligent task scheduling and load balancing control of a data center adaptively adjusts and optimizes the task scheduling strategy of the data center by constructing an adaptively updated deep reinforcement learning model, thereby gradually improving the task scheduling efficiency. When the supply of renewable energy is sufficient or the electricity price is low, tasks with high computing power requirements are allocated to run; when the supply of renewable energy is insufficient or the electricity price is peak, tasks with low computing power requirements are allocated to run, thereby ensuring smooth resource utilization. At the same time, full consideration is given to the fact that the execution time of tasks can be postponed, and urgent tasks are allocated first, thereby achieving a reduction in the overall energy consumption of the data center and an optimization and improvement in performance, making full use of renewable resources, and providing a better intelligent control direction for the balance between resource utilization and energy efficiency management of the data center.

[0004] To achieve the purpose of the present invention, the present invention is implemented through the following technical solutions: a data center intelligent task scheduling and load balancing control method, comprising the following steps:

[0005] Step 1: Building a learning model: Building a learning model based on a recurrent neural network and combining it with model adaptive update technology. Then, using the historical data of the data center to build a data set to train the model, we can obtain an adaptively updated deep reinforcement learning model.

[0006] Step 2: Construct a computing power demand model. Construct a task computing power demand model based on BIM technology, and use the data of historical tasks to train the computing power demand model to obtain the task computing power demand model.

[0007] Step 3: Preliminary construction of the balanced control system. Based on the server load monitoring data, the task computing power demand model and the renewable energy supply situation judgment are the core, and the input data processing and judgment block of the balanced control system is preliminarily built;

[0008] Step 4: Improve the balance control system. Introduce the attention mechanism network architecture after the data processing and judgment block, and introduce the adaptively updated deep reinforcement learning model to obtain a perfect balance control system.

[0009] Step 5: Import the system into the application, install the complete balancing control system into the data center main control machine, and connect the external renewable energy supply detection equipment to the system to complete the system application.

[0010] A further improvement is that the data set in step one includes load monitoring data, task characteristic data and renewable energy supply data, and the data set is divided into a training set and a validation set, wherein 70% of the load monitoring data, 70% of the task characteristic data and 70% of the renewable energy supply data in the data set are the training set, and the remaining 30% of the load monitoring data, task characteristic data and renewable energy supply data in the data set are the validation set.

[0011] A further improvement is that the specific steps for constructing the deep reinforcement learning model in step 1 are:

[0012] S1. Using the recurrent neural network as a framework, inserting the adaptive update technology, and taking the overall energy consumption reduction and performance optimization of the data center as the reward goals to initially build the model;

[0013] S2. Import the weight decay coefficient into the initially constructed model to perform regularization optimization on the model;

[0014] S3. Finally, the model is trained using the training set divided by the data set, and verified using the validation set.

[0015] Further improvement lies in: the specific construction process of the task computing power requirement model in step 2 is to first use BIM technology as a framework, and introduce data recognition and extraction technology, and at the same time use the task computing power requirement data as output for preliminary construction, and finally extract the historical task data and train the preliminary constructed model to obtain the task computing power requirement model.

[0016] A data center intelligent task scheduling and load balancing control system includes a data acquisition system, a data processing system and an allocation and scheduling system. The data acquisition system is used to obtain system task computing power requirements and delay time data, server load monitoring data and renewable energy supply monitoring data. The data acquisition system includes a task processing module, a load monitoring module and an energy supply monitoring module. The data processing system is used to identify, divide and process the acquired data. The data processing system includes a data division module and a data sorting module. The allocation and scheduling system intelligently allocates and schedules tasks based on an adaptively updated deep reinforcement learning model. The allocation and scheduling system includes a server scheduling module and an intelligent allocation module.

[0017] Further improvements are: the task processing module extracts task data from all tasks received by the data center based on the task computing power requirement model, processes it and derives the task computing power requirement ranking, and extracts the task delay time data at the same time; the load monitoring module is used to monitor the operating load of each server in real time; the energy supply monitoring module is used to monitor and calculate the renewable energy supply and resource utilization in real time.

[0018] Further improvements are: the data division module is used to classify, divide and organize the acquired data, and the data organization module is used to organize and judge the acquired task computing power demand data, delay time data, load monitoring data and energy supply monitoring data.

[0019] Further improvements are as follows: the server scheduling module is used to perform task allocation and scheduling according to the load conditions of each server and in combination with the task computing power requirements and the delay time. The intelligent allocation module allocates tasks with high computing power requirements and long-term delays to low-load servers or chooses to run them when electricity prices are low based on the given renewable energy supply monitoring data judgment results.

[0020] The beneficial effects of the present invention are as follows: the present invention adaptively learns, adjusts and optimizes the task scheduling strategy of the data center by constructing an adaptively updated deep reinforcement learning model, thereby gradually improving the task scheduling efficiency. When the supply of renewable energy is sufficient or the electricity price is low, tasks with high computing power requirements are assigned to run; when the supply of renewable energy is insufficient or the electricity price is peak, tasks with low computing power requirements are assigned to run, thereby ensuring smooth resource utilization. At the same time, full consideration is given to the fact that the execution time of tasks can be postponed, and urgent tasks are assigned first, thereby achieving a reduction in the overall energy consumption of the data center and an optimization and improvement in performance, achieving full utilization of renewable resources, and providing a better intelligent regulation direction for the balance between resource utilization and energy efficiency management of the data center. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of the method of Example 1 of the present invention.

[0022] Figure 2 This is a system architecture diagram of Example 2 of the present invention. DETAILED DESCRIPTION

[0023] In order to deepen the understanding of the present invention, the present invention will be further described in detail below in conjunction with examples. The examples are only used to explain the present invention and do not constitute a limitation on the protection scope of the present invention.

[0024] Example 1

[0025] according to Figure 1 As shown, this embodiment provides a data center intelligent task scheduling and load balancing control method, including the following steps:

[0026] Step 1: Building a learning model: Building a learning model based on a recurrent neural network and combining it with model adaptive update technology. Then, using the historical data of the data center to build a data set to train the model, we can obtain an adaptively updated deep reinforcement learning model.

[0027] The data set contains load monitoring data, task feature data and renewable energy supply data. The data set is divided into a training set and a validation set. 70% of the load monitoring data, 70% of the task feature data and 70% of the renewable energy supply data in the data set are the training set, and the remaining 30% of the load monitoring data, task feature data and renewable energy supply data in the data set are the validation set.

[0028] The specific steps for building an adaptively updated deep reinforcement learning model are:

[0029] S1. Using the recurrent neural network as a framework, inserting the adaptive update technology, and taking the overall energy consumption reduction and performance optimization of the data center as the reward goals to initially build the model;

[0030] S2. Import the weight decay coefficient into the initially constructed model to perform regularization optimization on the model;

[0031] S3. Finally, the model is trained using the training set divided by the data set, and verified using the validation set to obtain an adaptively updated deep reinforcement learning model.

[0032] Step 2: Construct a computing power demand model. Construct a task computing power demand model based on BIM technology, and use the data of historical tasks to train the computing power demand model to obtain the task computing power demand model. The output of the task computing power demand model not only includes the computing power demand data of the task, but also can automatically sort and output the computing power demand data of all tasks in the data center.

[0033] The specific construction process of the task computing power demand model is to first use BIM technology as a framework, and introduce data recognition and extraction technology, and use the task computing power demand data as output for preliminary construction. Finally, after extracting the historical task data, 70% is used as a training set and 30% is used as a verification set to train and verify the preliminary model, and the task computing power demand model is obtained, which is expressed by the following formula;

[0034] S=φP×F

[0035] Where S is the computing power requirement of the task; φ represents the weight of the extracted data that needs to be calculated; P represents the amount of data that needs to be calculated; and F represents the computing power requirement for running the calculation of a single parameter, which is a fixed value relative to the computing power of the same server.

[0036] The task computing power requirement model uses data recognition and extraction technology to extract data with possible calculations in the task content, and imports this data into the model as the task computing power requirement data. The model determines and outputs the computing power requirement data of the task, and determines the model computing power requirement size based on the computing power requirement data.

[0037] Step 3: Preliminary construction of the balanced control system, based on server load monitoring data. The monitoring data specifically refers to the current time of each server participating in the task load operation. The task computing power demand model and renewable energy supply situation are the center, which will be used as the scheduling basis to preliminarily build the input data processing and judgment block of the balanced control system;

[0038] The process of building the processing and judgment block is to use the XXLJOB scheduling platform, set the server load monitoring data and renewable energy supply conditions as the input, introduce the task computing power demand model and synchronously set the output results of the model as the input, and use it as the system's input pre-processing block to identify and judge the input data.

[0039] Step 4: Improve the balance control system. Introduce the attention mechanism network architecture after the data processing and judgment block, and introduce the adaptively updated deep reinforcement learning model to obtain a perfect balance control system.

[0040] The attention mechanism network architecture includes attention module, self-attention, multi-head attention and encoder-decoder. The attention module defines the real-time change data of server load data, task computing power requirements and renewable resource supply as the information types of attention, and uses self-attention and multi-head attention to capture the intrinsic relationship between the data and assign weights to the real-time change data of server load data, task computing power requirements and renewable resource supply. Finally, the encoder-decoder outputs the output data and imports it into the subsequent adaptively updated deep reinforcement learning model for processing.

[0041] Step 5. Import the application into the system, install the perfect balancing control system into the main control machine of the data center, and connect the external renewable energy supply detection equipment to the system to complete the application of the system. Use the external renewable resource supply situation and electricity price peak and valley data as well as the task computing power demand data and server load situation as input to derive the task scheduling allocation instructions, which are executed by the data center to allocate tasks, so that the task processing efficiency and energy consumption level of the data center reach the optimal state.

[0042] Example 2

[0043] according to Figure 1 As shown, this embodiment provides a data center intelligent task scheduling and load balancing control system, including a data acquisition system, a data processing system and a distribution scheduling system. The data acquisition system is used to obtain system task computing power requirements and delay time data, server load monitoring data and renewable energy supply monitoring data. The renewable energy supply monitoring data includes real-time monitoring data of supply quantity and peak and valley time period data of electricity prices, which serve as the basis for task scheduling;

[0044] The data processing system is used to identify, divide and process the acquired data;

[0045] The allocation and scheduling system intelligently allocates and schedules tasks based on an adaptively updated deep reinforcement learning model. It allocates and runs tasks with high computing power requirements when the supply of renewable energy is sufficient or the electricity price is low, and allocates and runs tasks with low computing power requirements when the supply of renewable energy is insufficient or the electricity price is high.

[0046] The data acquisition system includes a task processing module, a load detection module and an energy supply monitoring module. The task processing module extracts task data from all tasks received by the data center based on the task computing power demand model, processes it, and derives the task computing power demand ranking while extracting the task delay time data. The load detection module is used to monitor the operating load of each server in real time. At the same time, it can divide the server numbers with high and low loads to facilitate the subsequent allocation of tasks with different computing power requirements. The energy supply monitoring module is used to monitor and calculate the supply of renewable energy and resource utilization in real time, and monitor the peak and low periods of electricity prices, and allocate high computing power demand tasks to the low electricity price period to reduce energy consumption.

[0047] The data processing system includes a data division module and a data sorting module. The data division module is used to classify and sort the acquired data. Since different data are imported into the system synchronously, it is necessary to sort the data according to different types of data through the data division module and input them into different processing modules. The data sorting module is used to sort and judge the acquired task computing power demand data, delay time data, load monitoring data and energy supply monitoring data.

[0048] The allocation and scheduling system includes a server scheduling module and an intelligent allocation module. The server scheduling module is used to perform task allocation and scheduling according to the load conditions of each server and in combination with the task computing power requirements and the delay time. Specifically, tasks with low computing power requirements are allocated to servers with higher loads, and tasks with high computing power requirements are allocated to servers with low loads. The intelligent allocation module allocates tasks with high computing power requirements and long delays to low-load servers or chooses to run them when electricity prices are low based on the given renewable energy supply monitoring data. Tasks with low computing power requirements are allocated to servers with higher loads, thereby reducing the energy consumption of the data center and improving the processing efficiency of the data center.

[0049] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A data center intelligent task scheduling and load balancing control method, characterized in that: The following steps are involved: Step 1: Building a learning model: Building a learning model based on a recurrent neural network and combining it with model adaptive update technology. Then, using the historical data of the data center to build a data set to train the model, we can obtain an adaptively updated deep reinforcement learning model. Step 2: Construct a computing power demand model. Construct a task computing power demand model based on BIM technology, and use the data of historical tasks to train the computing power demand model to obtain the task computing power demand model. Step 3: Preliminary construction of the balanced control system. Based on the server load monitoring data, the task computing power demand model and the renewable energy supply situation judgment are the core, and the input data processing and judgment block of the balanced control system is preliminarily built; Step 4: Improve the balance control system. Introduce the attention mechanism network architecture after the data processing and judgment block, and introduce the adaptively updated deep reinforcement learning model to obtain a perfect balance control system. Step 5: Import the system into the application, install the complete balancing control system into the data center main control machine, and connect the external renewable energy supply detection equipment to the system to complete the system application.

2. According to claim 1, a data center intelligent task scheduling and load balancing control method is characterized by: The data set in step one includes load monitoring data, task characteristic data and renewable energy supply data, and the data set is divided into a training set and a validation set, wherein 70% of the load monitoring data, 70% of the task characteristic data and 70% of the renewable energy supply data in the data set are the training set, and the remaining 30% of the load monitoring data, task characteristic data and renewable energy supply data in the data set are the validation set.

3. According to claim 1, a data center intelligent task scheduling and load balancing control method is characterized by: The specific steps of constructing the deep reinforcement learning model in step 1 are: S1. Using the recurrent neural network as a framework, inserting the adaptive update technology, and taking the overall energy consumption reduction and performance optimization of the data center as the reward goals to initially build the model; S2. Import the weight decay coefficient into the initially constructed model to perform regularization optimization on the model; S3. Finally, the model is trained using the training set divided by the data set, and verified using the validation set.

4. A data center intelligent task scheduling and load balancing control method according to claim 1, characterized in that: The specific construction process of the task computing power requirement model in step 2 is to first use BIM technology as a framework, and introduce data recognition and extraction technology, and at the same time use the task computing power requirement data as output for preliminary construction, and finally extract the historical task data and train the preliminary constructed model to obtain the task computing power requirement model.

5. A data center intelligent task scheduling and load balancing control system, characterized by: It includes a data acquisition system, a data processing system and an allocation and scheduling system. The data acquisition system is used to obtain system task computing power requirements and delay time data, server load monitoring data and renewable energy supply monitoring data. The data acquisition system includes a task processing module, a load detection module and an energy supply monitoring module. The data processing system is used to identify, divide and process the acquired data. The data processing system includes a data division module and a data sorting module. The allocation and scheduling system intelligently allocates and schedules tasks based on an adaptively updated deep reinforcement learning model. The allocation and scheduling system includes a server scheduling module and an intelligent allocation module.

6. The data center intelligent task scheduling and load balancing control system according to claim 5 is characterized by: The task processing module extracts task data from all tasks received by the data center based on the task computing power requirement model, processes them, and derives the task computing power requirement ranking while extracting the task delay time data. The load detection module is used to monitor the operating load of each server in real time. The energy supply monitoring module is used to monitor and calculate the renewable energy supply and resource utilization in real time.

7. The data center intelligent task scheduling and load balancing control system according to claim 5 is characterized by: The data division module is used to classify, divide and organize the acquired data, and the data organization module is used to organize and judge the acquired task computing power requirement data, delay time data, load monitoring data and energy supply monitoring data.

8. The data center intelligent task scheduling and load balancing control system according to claim 5 is characterized by: The server scheduling module is used to perform task allocation and scheduling based on the load conditions of each server and in combination with the task computing power requirements and the delay time. The intelligent allocation module allocates tasks with high computing power requirements and long delays to low-load servers or chooses to run them when electricity prices are low based on the given renewable energy supply monitoring data judgment results.

Images