A task load balancing and scheduling method and device based on big data

Abstract

The application provides a task load balancing scheduling method and device based on big data, which acquires the task load state on the big data platform in real time, constructs a task load state sequence based on the real-time acquired task load state, updates the resource competition heat map based on the task load information sequence when reaching the preset update time point, and then makes a dynamic decision based on the current resource competition heat map and the historical resource utilization rate of the to-be-scheduled task when receiving the to-be-scheduled task, obtains the recommended running time period of the to-be-scheduled task, and schedules the to-be-scheduled task based on the recommended running time period, which can effectively identify the resource usage trend, accurately predict the future resource competition state, and make the optimal running time recommendation combined with the historical running data of the to-be-scheduled task, so as to realize the balanced use of system resources and the maximization of scheduling efficiency, can significantly alleviate the task resource conflict problem, reduce the task queuing time, and improve the system throughput capacity and task completion rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of task scheduling, in particular to a task load balancing scheduling method and device based on big data. BACKGROUND

[0002] The current big data processing platform has gradually developed into a complex system supporting heterogeneous computing tasks, and is mostly used for processing mixed deployment job scenarios of multiple types of tasks (data integration, computing analysis, report generation). However, since most platforms use fixed time window batch scheduling strategies, such as executing a large number of data integration and cleaning tasks in the early morning, although it is operable in business, the result is to cause the system resource load to be extremely unbalanced in different time periods. Secondly, the scheduling system relies heavily on human experience for task configuration. When a new task is put online, the operation and maintenance engineer needs to estimate the required resources and execution time of the task based on historical experience, and iteratively adjust the configuration parameters in the test environment. This process not only takes time, but also has low accuracy, affecting business agility and data product development cycle. In addition, the lack of execution of service level agreement (SLA) guarantee mechanism is also a key problem. The scheduling system cannot perceive the chain reaction caused by potential resource conflicts, especially when there are high-priority report or model tasks, execution delays are often caused by low-priority task resource occupation or sudden task insertion.

[0003] In summary, the current task scheduling system lacks dynamicity and is difficult to achieve multiple goals of optimizing resource utilization, shortening task online period, and stabilizing business SLA guarantee. SUMMARY

[0004] The present application provides a task load balancing scheduling method and device based on big data to solve the defects in the prior art that lack dynamicity and are difficult to achieve multiple goals of optimizing resource utilization, shortening task online period, and stabilizing business SLA guarantee.

[0005] The present application provides a task load balancing scheduling method based on big data, comprising:

[0006] Real-time acquisition of task load states on a big data platform, and construction of a task load state sequence based on the real-time acquired task load states; the task load states include the runtime of each task and the resource utilization rate of each task in the running process;

[0007] When a preset update time point is reached, update the resource competition heat map based on the task load information sequence; the resource competition heat map contains the resource utilization rate of each unit time period in a future preset time period;

[0008] When receiving a to-be-scheduled task, a dynamic decision is made based on a current resource competition heat map and historical resource utilization of the to-be-scheduled task, a recommended running time period of the to-be-scheduled task is obtained, and the to-be-scheduled task is scheduled based on the recommended running time period; the historical resource utilization of the to-be-scheduled task includes resource utilization of a same type of scheduled task in a running process.

[0009] According to the method, the recommended running time period of the to-be-scheduled task is obtained based on the current resource competition heat map and the historical resource utilization of the to-be-scheduled task.

[0010] A target cost function is established based on resource utilization of each unit time period in the current resource competition heat map, task submission time of the to-be-scheduled task, and the historical resource utilization of the to-be-scheduled task, and a dynamic decision is made based on the target cost function to determine the recommended running time period of the to-be-scheduled task.

[0011] The independent variable of the target cost function is a candidate scheduling time period, and the candidate scheduling time period is any unit time period.

[0012] The target cost function is established based on a difference between the candidate scheduling time period and the task submission time of the to-be-scheduled task, a difference between resource utilization after the to-be-scheduled task is scheduled in the candidate scheduling time period and resource utilization of other candidate scheduling time periods in the resource competition heat map, and resource utilization of each type of resource in the candidate scheduling time period in the resource competition heat map; the resource utilization after the to-be-scheduled task is scheduled in the candidate scheduling time period is determined based on resource utilization of the candidate scheduling time in the resource competition heat map and the historical resource utilization of the to-be-scheduled task.

[0013] According to the method, the resource competition heat map includes resource utilization of each type of resource in each unit time period; the each type of resource includes network resources, storage resources, and computing resources.

[0014] The target cost function is established based on a difference between the candidate scheduling time period and the task submission time of the to-be-scheduled task, a difference between resource utilization of each type of resource after the to-be-scheduled task is scheduled in the candidate scheduling time period and resource utilization of corresponding resources of other candidate scheduling time periods in the resource competition heat map, and resource utilization of each type of resource in the candidate scheduling time period in the resource competition heat map.

[0015] The application provides a task load balancing scheduling method based on big data, which dynamically decides based on a target cost function, determines a recommended running time period of a to-be-scheduled task, and comprises the following steps:

[0016] Based on the resource utilization rate of each unit time period in the current resource competition heat map, the task submission time of the to-be-scheduled task and the historical resource utilization rate of the to-be-scheduled task, a state space is constructed.

[0017] Based on each of the unit time periods in the future preset time period, an action space is constructed.

[0018] Based on the target cost function, a reward function is determined, and based on the state space, the action space and the reward function, a reinforcement learning algorithm is used to dynamically decide and determine the recommended running time period of the to-be-scheduled task.

[0019] According to the task load balancing scheduling method based on big data, the difference between the resource utilization rate of any type of resource after the to-be-scheduled task is scheduled in the candidate scheduling time period and the resource utilization rate of the corresponding resource in other candidate scheduling time periods in the resource competition heat map is obtained by weighting and summing the variance between the resource utilization rate of any type of resource after the to-be-scheduled task is scheduled in the candidate scheduling time period and the resource utilization rate of the any type of resource in other candidate scheduling time periods in the resource competition heat map and the weight of the any type of resource.

[0020] According to the task load balancing scheduling method based on big data, the resource competition heat map is updated based on the task load information sequence, which comprises the following steps:

[0021] A resource prediction model is constructed based on a long short-term memory network.

[0022] The task load information sequence is input into the resource prediction model to obtain the resource competition heat map output by the resource prediction model.

[0023] According to the task load balancing scheduling method based on big data, the to-be-scheduled task is scheduled based on the recommended running time period, which comprises the following steps:

[0024] Based on the task urgency of the to-be-scheduled task, it is determined whether the to-be-scheduled task needs to be scheduled in advance or delayed.

[0025] If early scheduling or delay scheduling is needed, the preset time is advanced or delayed based on the recommended running time period, and the to-be-scheduled task is scheduled based on the advanced or delayed time; otherwise, the to-be-scheduled task is scheduled at the recommended running time period.

[0026] The application further provides a task load balancing scheduling device based on big data, comprising:

[0027] The resource load acquisition unit is configured to acquire the task load state on the big data platform in real time, and construct a task load state sequence based on the task load state acquired in real time; the task load state comprises the running time length of each task and the resource utilization rate of each task in the running process;

[0028] The resource competition graph updating unit is configured to update the resource competition heat map based on the task load information sequence when reaching a preset updating time point; the resource competition heat map comprises the resource utilization rate of each unit time period in a future preset time period.

[0029] The task scheduling decision unit is configured to, when receiving a to-be-scheduled task, make a dynamic decision based on the current resource competition heat map and the historical resource utilization rate of the to-be-scheduled task, obtain a recommended running time period of the to-be-scheduled task, and schedule the to-be-scheduled task based on the recommended running time period; the historical resource utilization rate of the to-be-scheduled task comprises the resource utilization rate of the same type of scheduled task in the running process.

[0030] The application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the big data-based task load balancing scheduling method according to any one of the above when executing the program.

[0031] The application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the big data-based task load balancing scheduling method according to any one of the above.

[0032] The application further provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the big data-based task load balancing scheduling method according to any one of the above.

[0033] The application provides a task load balancing scheduling method and device based on big data, which acquires the task load state on a big data platform in real time, constructs a task load state sequence based on the real-time acquired task load state, updates a resource competition heat map based on the task load information sequence when reaching a preset update time point, and then makes a dynamic decision based on the current resource competition heat map and the historical resource utilization rate of the to-be-scheduled task when receiving the to-be-scheduled task, obtains the recommended running time period of the to-be-scheduled task, and schedules the to-be-scheduled task based on the recommended running time period, which can effectively identify the resource usage trend, accurately predict the future resource competition state, and make the optimal running time recommendation combined with the historical running data of the to-be-scheduled task, so as to realize the balanced use of system resources and the maximization of scheduling efficiency, can significantly relieve the task resource conflict problem, reduce the task queuing time, and improve the system throughput capacity and task completion rate. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort.

[0035] Figure 1 is a flowchart of the task load balancing scheduling method based on big data provided by the application;

[0036] Figure 2 is a flowchart of the task scheduling dynamic decision method provided by the application;

[0037] Figure 3 is a structural schematic diagram of the task load balancing scheduling device based on big data provided by the application;

[0038] Figure 4 is a structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the application more clear, the technical solutions in the application will be described clearly and completely in the following with reference to the drawings in the application. Obviously, the described embodiments are some embodiments of the application, not all embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative effort belong to the protection scope of the application.

[0040] Figure 1 is a flowchart of the task load balancing scheduling method based on big data provided by the application, as Figure 1As shown, the method comprises:

[0041] In step 110, the task load state on the big data platform is acquired in real time, and a task load state sequence is constructed based on the task load state acquired in real time; the task load state comprises the running time length of each task and the resource utilization rate of each task in the running process;

[0042] In step 120, when a preset update time point is reached, the resource competition heat map is updated based on the task load information sequence; the resource competition heat map comprises the resource utilization rate of each unit time period in a future preset time period;

[0043] In step 130, when a to-be-scheduled task is received, a dynamic decision is made based on the current resource competition heat map and the historical resource utilization rate of the to-be-scheduled task, a recommended running time period of the to-be-scheduled task is obtained, and the to-be-scheduled task is scheduled based on the recommended running time period; the historical resource utilization rate of the to-be-scheduled task comprises the resource utilization rate of the same type of scheduled task in the running process.

[0044] Here, in the running process, the monitoring module continuously collects the task load state from each computing node of the big data platform in real time. The task load state refers to the running time length and resource utilization rate of each task currently running in the big data platform. Specifically, the task load state can include the running time length of each task, the average resource utilization rate of CPU, the average memory occupancy rate, the disk I / O ratio, and the average resource utilization rate of network resources. These information is collected by the monitoring module deployed on the node side at a set time interval (denoted as a unit time period) and synchronized to the central data integration module. The data integration module can arrange and format the data in the task load state received in real time, and splice it with the previously acquired task load state to form a task load state sequence. It can be seen that the task load state sequence is a data set arranged in chronological order, each task load state in the sequence contains the running time length and resource utilization rate of each task in the corresponding time period, which can describe the resource utilization behavior of each task in different time segments. Based on the task load state sequence, the change trend of resource utilization can be captured and quantified, thereby reflecting the dynamic demand of each unit time period for system resources.

[0045] When entering the preset update time point, for example, every ten minutes, the system triggers the update mechanism of the resource competition heat map. The resource competition heat map can be a two-dimensional vector, which contains the resource utilization rate of each unit time period in the future preset time period (for example, the next 6 hours). Further, the resource competition heat map can also be a three-dimensional vector, which contains the resource utilization rate of different types of resources in each unit time period in the future preset time period (for example, the next 6 hours), that is, the first dimension is time, the second dimension is resource type, and the third dimension is resource utilization rate. The resource competition heat map can be obtained based on the above task load information sequence prediction, by capturing the dynamic demand of each unit time period in the task load state sequence for system resources, predicting the demand of each unit time period for system resources in the future preset time period. In one embodiment, a resource prediction model can be constructed based on time series analysis method, for example, a resource prediction model can be constructed based on long short-term memory network, and the resource prediction model is trained in advance using the collected sample data, so as to predict at the update time point using the resource prediction model. Specifically, the task load information sequence can be input into the resource prediction model to obtain the resource competition heat map output by the resource prediction model. In addition, the resource prediction model can also be fine-tuned according to the real-time acquired task load state to ensure its prediction accuracy.

[0046] By constantly refreshing the resource competition heat map, a forward-looking resource utilization view can be obtained, which can perceive in advance which time period the resources are relatively abundant and which time period the resources will face high concurrency pressure, to provide decision reference for subsequent task scheduling.

[0047] When a user submits a new to-be-scheduled task, the most suitable running time period needs to be recommended for the to-be-scheduled task. At this time, the current latest resource competition heat map and the historical resource utilization rate of the to-be-scheduled task can be obtained. Among them, the historical resource utilization rate of the to-be-scheduled task is not directly derived from the task itself, but is a comprehensive portrait obtained by analyzing the same type of scheduled tasks in the big data platform. Specifically, the scheduled tasks with high similarity in task type, data processing mode, execution logic, etc. with the current to-be-scheduled task can be found, and their resource utilization rate in the running process is counted and used as a reference for the resource demand of the current to-be-scheduled task.

[0048] Based on the resource competition heat map and the historical resource utilization rate of the to-be-scheduled task, a dynamic decision can be made to determine a recommended running time period for the to-be-scheduled task, so that the task is scheduled to run in the recommended running time period. In some embodiments, to cope with unexpected situations, whether early scheduling or delayed scheduling is needed can be determined based on the task urgency of the to-be-scheduled task. If early scheduling or delayed scheduling is needed, the recommended running time period is advanced or delayed by a preset time, and the to-be-scheduled task is scheduled based on the time after the advance or delay; otherwise, the to-be-scheduled task is scheduled in the recommended running time period.

[0049] In the dynamic decision-making stage, the insertion of the current task in multiple time periods in the future can be simulated, it can be evaluated whether the system resources in each time period can support the execution of the task, and the resource matching degree, system load risk, and estimated completion time of each time period can be calculated, so that the system resource conflict and task queuing waiting time are minimized to improve the overall resource utilization rate while ensuring the smooth completion of the task.

[0050] In some embodiments, a target cost function can be established based on the resource utilization rate of each unit time period in the current resource competition heat map, the task submission time of the to-be-scheduled task, and the historical resource utilization rate of the to-be-scheduled task, and a dynamic decision can be made by optimizing the target cost function to determine the recommended running time period of the to-be-scheduled task.

[0051] The variables (i.e., independent variables) of the target cost function are all candidate scheduling time periods, i.e., each unit time period in the future preset time period described above. The design of the target cost function mainly considers two objectives: one is the difference between the task submission time and the actual execution time, i.e., scheduling delay; the other is the impact on the system resource state after the task is inserted into a candidate time period. The former reflects the timeliness of task response, and the latter reflects the balance of system load. In addition, a constraint can be added, i.e., the difference between the resource utilization rate of the candidate scheduling time period and the preset threshold, i.e., whether the system is busy, to avoid the system busy period as much as possible.

[0052] Specifically, for a candidate scheduling time period t i , the target cost function can be represented as follows:

[0053] C(t i ) = α×D(t i , t submit ) + β×R(t i )

[0054] Wherein, α and β are weight coefficients for balancing the timeliness of response and resource balance; D(t i , t submit) is the difference between the candidate scheduling time period t i and the task submission time t submit of the task to be scheduled; R(t i ) is the difference between the resource utilization of the candidate scheduling time period t i after the task to be scheduled is scheduled and the resource utilization of other candidate scheduling time periods in the resource competition heat map.

[0055] Here, by superimposing the resource demand of the task to be scheduled (determined by the historical resource utilization of the task to be scheduled) on the existing data in the resource competition heat map (i.e., the resource utilization of the candidate scheduling time period t i in the resource competition heat map), the new resource utilization of the time period after the task to be scheduled is inserted can be determined. Then, the difference between the resource utilization of the candidate scheduling time period t i after the task to be scheduled is scheduled and the resource utilization of other candidate scheduling time periods in the resource competition heat map can be compared.

[0056] In some other embodiments, the resource competition heat map can include the resource utilization of each type of resource in each unit time period, where the types of resources include network resources, storage resources, and computing resources. By subdividing the types of resources, the scheduling decision can be further optimized from the perspective of each type of resource, achieving overall resource balance.

[0057] In this case, the target cost function can be established based on the difference between the candidate scheduling time period and the task submission time of the task to be scheduled, the difference between the resource utilization of each type of resource after the task to be scheduled is scheduled in the candidate scheduling time period and the resource utilization of the corresponding resource of other candidate scheduling time periods in the resource competition heat map, and the resource utilization of each type of resource of the candidate scheduling time period in the resource competition heat map.

[0058] Here, the design of the target cost function takes into account three core factors. The first is the time difference between the submission time of the task and the candidate scheduling time period. Generally, the more timely the scheduling, the better it meets the real-time needs of the business, so the candidate scheduling time period with the smaller time difference should have higher priority. The second part of the target cost function considers the difference in the resource utilization of each type of resource in a certain candidate scheduling time period after the task is inserted in the time period and other time periods. That is, by simulating the possible rise in the pressure of each type of resource when the task is scheduled in each candidate scheduling time period, it is compared with the existing resource occupation in other time periods. Through this comparison, it can be found whether the resource occupation of each type of resource caused by scheduling is balanced, whether it is easy to cause resource competition or bottleneck. The third factor of the target cost function reflects the resource utilization of the current candidate scheduling time period itself, in order to avoid scheduling the task to a time period that is already very busy.

[0059] Finally, the target cost function is a weighted combination of the above three factors, which can be expressed as:

[0060] C(t i ) = α×D(t i , t submit ) + β×R(t i ) + γ×L(t i )

[0061] wherein a, β and γ are weight parameters dynamically adjusted according to different scheduling strategies. For tasks with higher real-time response requirements, a will be significantly amplified; and for tasks with higher system throughput requirements and tolerable scheduling delay, the weights of β and γ will be higher. L(t i ) is the weighted average of the resource utilization of each type of resource in the resource competition heat map in the candidate scheduling time period.

[0062] In some embodiments, the difference R(t i ) between the resource utilization of each type of resource after scheduling the to-be-scheduled task in the candidate scheduling time period t j and the resource utilization of the corresponding resource in other candidate scheduling time periods (t i , j≠i) in the resource competition heat map can be obtained by weighting and summing the variance between the resource utilization of any type of resource after scheduling the to-be-scheduled task in the candidate scheduling time period t i and the resource utilization of the same type of resource in other candidate scheduling time periods t j in the resource competition heat map, and the weight of each type of resource. The weight of each type of resource can be determined based on the historical resource utilization of each type of resource corresponding to the to-be-scheduled task, and the weight of each type of resource is greater when the historical resource utilization of the resource is higher.

[0063] For example, R(t i ) can be expressed as follows:

[0064] R(t i ) = w1×Dif(Res1) +w2×Dif(Res2) +... + wp×Dif(Resp)

[0065] wherein w1, w2,..., wp are the weights of the p types of resources, Dif() represents the variance between the resource utilization of the corresponding type of resource after scheduling the to-be-scheduled task in the candidate scheduling time period t i and the resource utilization of the same type of resource in other candidate scheduling time periods t j in the resource competition heat map, and Resi represents the ith resource.

[0066] After the target cost function is constructed, the target cost function can be evaluated on all candidate scheduling time periods, and then the candidate scheduling time period with the minimum cost is selected as the recommended running time period of the task to be scheduled.

[0067] To further improve the accuracy of dynamic decision-making, a reinforcement learning mechanism can also be introduced. Specifically, as shown in Figure 2 The dynamic decision-making can be performed in the following manner to determine the recommended running time period of the task to be scheduled:

[0068] In step 210, a state space is constructed based on the resource utilization rate of each unit time period in the current resource competition heat map, the task submission time of the task to be scheduled, and the historical resource utilization rate of the task to be scheduled.

[0069] In step 220, an action space is constructed based on each of the unit time periods in the future preset time period.

[0070] In step 230, a reward function is determined based on the target cost function, and a reinforcement learning algorithm is used to make a dynamic decision based on the state space, the action space, and the reward function to determine the recommended running time period of the task to be scheduled.

[0071] The state space can be constructed according to the resource competition heat map, the task submission time of the task to be scheduled, and the historical resource utilization rate. Here, the state space is defined as a multi-dimensional vector composed of the resource utilization rate of each type of resource in each unit time period, the task submission time of the task to be scheduled, and the historical resource utilization rate in the current time period and the previous several time periods. It can be understood that the state space not only contains the objective resource state of the system, but also integrates the individual behavior tendency of the task to be scheduled, thereby providing sufficient information to guide the scheduling decision. When constructing the action space, all selectable unit time periods in the future preset time period can be regarded as an action in the action space. For example, if the preset time range is six hours in the future and the scheduling granularity is ten minutes per unit time period, the action space contains 36 candidate scheduling time periods. Each action represents the system's possible choice to schedule the task to be scheduled to the corresponding time period to start execution.

[0072] After the state space and the action space are established, the system needs a mechanism to measure the pros and cons of each action in the current state, that is, a reward function needs to be designed. Among them, the reward function can be constructed based on the previously defined target cost function. Since the target cost function describes the scheduling delay cost and system resource load difference that may be caused by scheduling tasks in a candidate scheduling time period, the lower the function value of the target cost function, the better. However, in the reinforcement learning framework, the higher the function value of the reward function, the better. Therefore, in this embodiment, the target cost function can be numerically reversed as the basis of the reward function. That is, the lower the target cost function value of a scheduling scheme, the higher the corresponding reward value, thereby guiding the reinforcement learning agent to tend to choose scheduling decisions with lower scheduling delay and less system resource load difference (i.e., more balanced resource allocation).

[0073] Next, based on the reinforcement learning algorithm, learning and decision-making are performed based on the above-defined state space, action space and reward function. In this embodiment, a reinforcement learning network such as a deep Q network (DQN) can be used for dynamic learning and decision-making. During the learning process, the agent receives a state input each time, that is, the resource utilization rate of each unit time period in the current resource competition heat map, the task submission time of the task to be scheduled, and the historical resource utilization rate; then it selects a candidate scheduling time period from the action space as a scheduling action; the reinforcement learning network will evaluate the reward value obtained after this selection based on the reward function, and feed the reward value back to the model for parameter update. With the continuous progress of the learning process, the reinforcement learning network can gradually learn to make the best scheduling choices under different resource distribution states, different types of tasks and different delay pressures. Finally, when a new task to be scheduled arrives, instead of exhaustively calculating the target cost function values of all candidate scheduling time periods, the trained reinforcement learning network can be directly used for state analysis and action prediction, thereby recommending the optimal running time period in a short time.

[0074] As can be seen, this dynamic scheduling mechanism based on reinforcement learning can continuously learn the optimal strategy through actual running data and has self-adaptive ability. In a multi-task high-concurrency environment, the scheduling strategy learned by it can maintain high efficiency of task scheduling and high utilization of system resources under the condition of frequent resource fluctuations.

[0075] In summary, the task load balancing scheduling method provided by the embodiment of the present application can effectively identify resource usage trends, accurately predict future resource competition states, and make optimal running time recommendations in combination with historical running data of the to-be-scheduled task, so as to realize balanced use of system resources and maximization of scheduling efficiency, can significantly alleviate the task resource conflict problem, reduce the task queuing time, and improve the system throughput capacity and task completion rate.

[0076] The task load balancing scheduling device based on big data provided by the present application is described below, and the task load balancing scheduling device based on big data described below can be referred to each other corresponding to the task load balancing scheduling method based on big data described above.

[0077] Based on any of the above embodiments, Figure 3 is a structural schematic diagram of the task load balancing scheduling device based on big data provided by the present application, as Figure 3 shown, the device comprises:

[0078] The resource load acquisition unit 310 is configured to acquire the task load state on the big data platform in real time, and construct a task load state sequence based on the task load state acquired in real time; the task load state comprises the running time length of each task and the resource utilization rate of each task in the running process;

[0079] The resource competition map updating unit 320 is configured to update the resource competition heat map based on the task load information sequence when reaching a preset update time point; the resource competition heat map comprises the resource utilization rate of each unit time period in a future preset time period.

[0080] The task scheduling decision unit 330 is configured to, when receiving a to-be-scheduled task, make a dynamic decision based on the current resource competition heat map and the historical resource utilization rate of the to-be-scheduled task, obtain a recommended running time period of the to-be-scheduled task, and schedule the to-be-scheduled task based on the recommended running time period; the historical resource utilization rate of the to-be-scheduled task comprises the resource utilization rate of the same type of scheduled task in the running process.

[0081] The device provided by the embodiment of the application can effectively identify resource usage trends, accurately predict future resource competition states, and make optimal running time recommendations in combination with historical running data of the to-be-scheduled task, so as to realize balanced use of system resources and maximization of scheduling efficiency, can significantly alleviate the task resource conflict problem, reduce task queuing time, and improve system throughput capacity and task completion rate.

[0082] According to any one of the above embodiments, the dynamic decision making based on the current resource competition heat map and the historical resource utilization rate of the to-be-scheduled task to obtain the recommended running time period of the to-be-scheduled task comprises:

[0083] According to any one of the above embodiments, the dynamic decision making based on the current resource competition heat map and the historical resource utilization rate of the to-be-scheduled task to obtain the recommended running time period of the to-be-scheduled task comprises:

[0084] According to any one of the above embodiments, the dynamic decision making based on the current resource competition heat map and the historical resource utilization rate of the to-be-scheduled task to obtain the recommended running time period of the to-be-scheduled task comprises:

[0085] According to any one of the above embodiments, the dynamic decision making based on the current resource competition heat map and the historical resource utilization rate of the to-be-scheduled task to obtain the recommended running time period of the to-be-scheduled task comprises:

[0086] According to any one of the above embodiments, the resource competition heat map comprises resource utilization rates of various types of resources in each unit time period; the various types of resources include network resources, storage resources, and computing resources.

[0087] The target cost function is established based on a difference between the candidate scheduling time period and a task submission time of the to-be-scheduled task, a difference between resource utilization of each type of resource after the to-be-scheduled task is scheduled in the candidate scheduling time period and resource utilization of corresponding resources of other candidate scheduling time periods in the resource competition heat map, and resource utilization of each type of resource of the candidate scheduling time period in the resource competition heat map.

[0088] Based on any one of the above embodiments, the dynamic decision making based on the target cost function determines a recommended running time period of the to-be-scheduled task, including:

[0089] Based on the resource utilization of each unit time period in the current resource competition heat map, the task submission time of the to-be-scheduled task, and the historical resource utilization of the to-be-scheduled task, a state space is constructed.

[0090] Based on each of the unit time periods in the future preset time period, an action space is constructed.

[0091] Based on the target cost function, a reward function is determined, and based on the state space, the action space, and the reward function, a reinforcement learning algorithm is used for dynamic decision making to determine the recommended running time period of the to-be-scheduled task.

[0092] Based on any one of the above embodiments, the difference between the resource utilization of each type of resource after the to-be-scheduled task is scheduled in the candidate scheduling time period and the resource utilization of corresponding resources of other candidate scheduling time periods in the resource competition heat map is obtained by weighting and summing the variance between the resource utilization of any type of resource after the to-be-scheduled task is scheduled in the candidate scheduling time period and the resource utilization of the any type of resource of other candidate scheduling time periods in the resource competition heat map, and the weight of the any type of resource. The weight of each type of resource is determined based on the historical resource utilization of each type of resource corresponding to the to-be-scheduled task.

[0093] Based on any one of the above embodiments, the updating of the resource competition heat map based on the task load information sequence includes:

[0094] A resource prediction model is constructed based on a long short-term memory network.

[0095] The task load information sequence is input into the resource prediction model to obtain a resource competition heat map output by the resource prediction model.

[0096] Based on any one of the above embodiments, the scheduling of the to-be-scheduled task based on the recommended running time period includes:

[0097] Based on the task urgency of the to-be-scheduled task, it is determined whether early scheduling or delayed scheduling is needed.

[0098] If early scheduling or delay scheduling is needed, the preset time is advanced or delayed based on the recommended running time period, and the to-be-scheduled task is scheduled based on the advanced or delayed time; otherwise, the to-be-scheduled task is scheduled in the recommended running time period.

[0099] Figure 4 is a structural schematic diagram of an electronic device provided by the present application, as shown in Figure 4 The electronic device can include a processor 410, a memory 420, a communications interface 430, and a communications bus 440, wherein the processor 410, the memory 420, and the communications interface 430 complete mutual communication through the communications bus 440. The processor 410 can invoke a logical instruction in the memory 420 to execute a big data-based task load balancing scheduling method, which includes: acquiring a task load state on a big data platform in real time, and constructing a task load state sequence based on the task load state acquired in real time; the task load state includes a running time length of each task and a resource utilization rate of each task in a running process; when a preset update time point is reached, a resource competition heat map is updated based on the task load information sequence; the resource competition heat map contains a resource utilization rate of each unit time period in a future preset time period; when a to-be-scheduled task is received, a dynamic decision is made based on a current resource competition heat map and a historical resource utilization rate of the to-be-scheduled task, a recommended running time period of the to-be-scheduled task is obtained, and the to-be-scheduled task is scheduled based on the recommended running time period; the historical resource utilization rate of the to-be-scheduled task includes a resource utilization rate of a same type of scheduled task in a running process.

[0100] In addition, the logical instruction in the memory 420 described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0101] In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the big data-based task load balancing scheduling method provided by the above method, which comprises: acquiring a task load state on a big data platform in real time, and constructing a task load state sequence based on the task load state acquired in real time; the task load state comprises a runtime of each task and a resource utilization rate of each task in a running process; when a preset update time point is reached, updating a resource competition heat map based on the task load information sequence; the resource competition heat map comprises a resource utilization rate of each unit time period in a future preset time period; when a to-be-scheduled task is received, making a dynamic decision based on the current resource competition heat map and a historical resource utilization rate of the to-be-scheduled task to obtain a recommended running time period of the to-be-scheduled task, and scheduling the to-be-scheduled task based on the recommended running time period; the historical resource utilization rate of the to-be-scheduled task comprises a resource utilization rate of a same type of scheduled task in a running process.

[0102] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, which, when executed by a processor, implements the big data-based task load balancing scheduling method provided by the above method, which comprises: acquiring a task load state on a big data platform in real time, and constructing a task load state sequence based on the task load state acquired in real time; the task load state comprises a runtime of each task and a resource utilization rate of each task in a running process; when a preset update time point is reached, updating a resource competition heat map based on the task load information sequence; the resource competition heat map comprises a resource utilization rate of each unit time period in a future preset time period; when a to-be-scheduled task is received, making a dynamic decision based on the current resource competition heat map and a historical resource utilization rate of the to-be-scheduled task to obtain a recommended running time period of the to-be-scheduled task, and scheduling the to-be-scheduled task based on the recommended running time period; the historical resource utilization rate of the to-be-scheduled task comprises a resource utilization rate of a same type of scheduled task in a running process.

[0103] The apparatus embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0104] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0105] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for task load balancing scheduling based on big data, characterized in that, The method comprises: obtaining a task load state on a big data platform in real time, and constructing a task load state sequence based on the task load state obtained in real time; the task load state comprises a running time length of each task and a resource utilization rate of each task in a running process; when a preset update time point is reached, updating a resource competition heat map based on the task load information sequence; the resource competition heat map is represented as a three-dimensional vector comprising a resource utilization rate of each type of resource in each unit time period in a future preset time period; a first dimension of the three-dimensional vector is time, a second dimension is a resource type, and a third dimension is a resource utilization rate; When receiving a to-be-scheduled task, a target cost function C(t i ) is established based on resource utilization of each unit time period in the current resource competition heat map, task submission time of the to-be-scheduled task, and historical resource utilization of the to-be-scheduled task, and a dynamic decision is made based on the target cost function C(t i ) to determine a recommended running time period of the to-be-scheduled task, and the to-be-scheduled task is scheduled based on the recommended running time period; the historical resource utilization of the to-be-scheduled task includes resource utilization of a same type of scheduled task in a running process; wherein the target cost function C(t i ) is expressed as a weighted sum between a difference D(t i , t submit ) between a candidate scheduling time period t i and the task submission time t submit of the to-be-scheduled task and a difference R(t i ) between resource utilization after the to-be-scheduled task is scheduled in the candidate scheduling time period and resource utilization of other candidate scheduling time periods in the resource competition heat map, that is, C(t i ) = α × D(t i , t submit ) + β × R(t i ), wherein α and β are weight parameters dynamically adjusted according to different scheduling strategies, used to balance response timeliness and resource balance; R(t i ) = w1 × Dif(Res1) + w2 × Dif(Res2) +... + wp × Dif(Resp), wherein w1, w2,..., wp are weights of p types of resources respectively, Dif() represents variance between resource utilization of a corresponding type of resource after the to-be-scheduled task is scheduled in a candidate scheduling time period and resource utilization of the type of resource in other candidate scheduling time periods in the resource competition heat map, Resi represents the i-th resource; the weight of any type of resource is determined based on historical resource utilization of the to-be-scheduled task on any type of resource, and the higher the historical resource utilization of a certain type of resource is, the greater the weight of the type of resource is; the resource utilization after the to-be-scheduled task is scheduled in the candidate scheduling time period is determined based on resource utilization of the candidate scheduling time in the resource competition heat map and historical resource utilization of the to-be-scheduled task. 2.The big data based task load balancing scheduling method according to claim 1, characterized in that, the types of resources comprise network resources, storage resources, and computing resources; The target cost function C(t i ) can also be represented as a weighted sum of the difference D(t i , t submit ) between the candidate scheduling time period t i and the task submission time t submit of the task to be scheduled, the difference R(t i ) between the resource utilization of each type of resource after scheduling the task to be scheduled at the candidate scheduling time period and the resource utilization of the corresponding resource of other candidate scheduling time periods in the resource competition heat map, and the weighted average L(t i ) of the resource utilization of each type of resource of the candidate scheduling time period in the resource competition heat map, i.e. C(t i ) = α × D(t i , t submit ) + β × R(t i ) + γ × L(t i ), where α, β and γ are weight parameters dynamically adjusted according to different scheduling strategies. For tasks with high real-time response requirements, α will be amplified. For tasks with higher system throughput requirements and tolerable scheduling delay, the weights of β and γ will be higher. R(t i ) = w1 × Dif(Res1) + w2 × Dif(Res2) +... + wp × Dif(Resp), where w1, w2,..., wp are the weights of p types of resources, Dif() represents the variance between the resource utilization of the corresponding type of resource after scheduling the task to be scheduled at the candidate scheduling time period and the resource utilization of the type of resource of other candidate scheduling time periods in the resource competition heat map, and Resi represents the i-th resource. The weight of any type of resource is determined based on the historical resource utilization of any type of resource corresponding to the task to be scheduled. The higher the historical resource utilization of a type of resource, the greater the weight of the type of resource. 3.The big data based task load balancing scheduling method of claim 2, wherein, The dynamic decision is made based on the target cost function C(t i ) to determine the recommended running time period of the task to be scheduled. constructing a state space based on a resource utilization rate of each unit time period in the current resource competition heat map, a task submission time of the to-be-scheduled task, and a historical resource utilization rate of the to-be-scheduled task; constructing an action space based on each unit time period in the future preset time period; determine a reward function based on the target cost function C(t i ), and determine a recommended running time period of the task to be scheduled based on the state space, the action space and the reward function by using a reinforcement learning algorithm for dynamic decision-making. 4.The big data based task load balancing scheduling method according to any one of claims 1 to 3, characterized in that, the updating of the resource competition heat map based on the task load information sequence comprises: constructing a resource prediction model based on a long short-term memory network; inputting the task load information sequence into the resource prediction model to obtain a resource competition heat map output by the resource prediction model. 5.The big data based task load balancing scheduling method according to any one of claims 1 to 3, characterized in that, the scheduling of the to-be-scheduled task based on the recommended running time period comprises: determining whether early scheduling or delayed scheduling is needed based on a task urgency of the to-be-scheduled task; if early scheduling or delayed scheduling is needed, scheduling the to-be-scheduled task based on a time that is advanced or delayed by a preset time based on the recommended running time period; otherwise, scheduling the to-be-scheduled task at the recommended running time period. 6.A big data based task load balancing scheduling apparatus, characterized in that, The method comprises: a resource load obtaining unit configured to obtain a task load state on a big data platform in real time, and construct a task load state sequence based on the task load state obtained in real time; the task load state comprises a running time length of each task and a resource utilization rate of each task in a running process; a resource competition map updating unit configured to, when a preset update time point is reached, update a resource competition heat map based on the task load information sequence; the resource competition heat map is represented as a three-dimensional vector comprising a resource utilization rate of each type of resource in each unit time period in a future preset time period; a first dimension of the three-dimensional vector is time, a second dimension is a resource type, and a third dimension is a resource utilization rate; The task scheduling decision unit is configured to, when receiving a task to be scheduled, establish a target cost function C(t i ) based on the resource utilization rate of each unit time period in the current resource competition heat map, the task submission time of the task to be scheduled, and the historical resource utilization rate of the task to be scheduled, and make a dynamic decision based on the target cost function C(t i ) to determine a recommended running time period of the task to be scheduled, and schedule the task to be scheduled based on the recommended running time period; the historical resource utilization rate of the task to be scheduled includes the resource utilization rate of a same type of scheduled task in a running process; wherein the target cost function C(t i ) is expressed as a weighted sum between a difference D(t i , t submit ) between a candidate scheduling time period t i and the task submission time t submit of the task to be scheduled and a difference R(t i ) between the resource utilization rate after scheduling the task to be scheduled at the candidate scheduling time period and the resource utilization rate of other candidate scheduling time periods in the resource competition heat map, that is, C(t i ) = α × D(t i , t submit ) + β × R(t i ), wherein α and β are weight parameters dynamically adjusted according to different scheduling strategies, and are used to balance response timeliness and resource balance; R(t i ) = w1 × Dif(Res1) + w2 × Dif(Res2) +... + wp × Dif(Resp), wherein w1, w2,..., wp are weights of p types of resources, Dif() represents a variance between the resource utilization rate of a corresponding type of resource after scheduling the task to be scheduled at a candidate scheduling time period and the resource utilization rate of the type of resource at other candidate scheduling time periods in the resource competition heat map, Resi represents an i-th resource; the weight of any type of resource is determined based on the historical resource utilization rate of the task to be scheduled corresponding to any type of resource, and the higher the historical resource utilization rate of a certain type of resource, the greater the weight of the type of resource; the resource utilization rate after scheduling the task to be scheduled at the candidate scheduling time period is determined based on the resource utilization rate of the candidate scheduling time in the resource competition heat map and the historical resource utilization rate of the task to be scheduled.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, the processor implements the big data-based task load balancing scheduling method of any one of claims 1 to 5 when executing the program.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, the computer program implements the big data-based task load balancing scheduling method of any one of claims 1 to 5 when executed by the processor.

Images