Task load balancing scheduling method and device based on big data

Abstract

The invention provides a task load balancing scheduling method and device based on big data, and the method comprises the steps: obtaining task load states on a big data platform in real time, and constructing a task load state sequence based on the task load states obtained in real time; the resource competition thermodynamic diagram is updated based on the task load information sequence when a preset updating time point is reached, and then when a to-be-scheduled task is received, a dynamic decision is made based on the current resource competition thermodynamic diagram and the historical resource utilization rate of the to-be-scheduled task to obtain a recommended operation time period of the to-be-scheduled task. And the to-be-scheduled task is scheduled based on the recommended operation time period, so that the resource use trend can be effectively identified, the future resource competition state can be accurately predicted, and the optimal operation time recommendation is made in combination with the historical operation data of the to-be-scheduled task, thereby realizing the balanced use of system resources and the maximization of the scheduling efficiency. The task resource conflict problem can be obviously relieved, the task queuing time is shortened, and the system throughput and the task completion rate are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of task scheduling, and in particular, to a task load balancing scheduling method and device based on big data. Background Art

[0002] The current big data processing platform has gradually evolved into a complex system that supports heterogeneous computing tasks, and is mostly used to process job scenarios with mixed deployment of multiple types of tasks (data integration, computational analysis, report generation). However, since the vast majority of platforms adopt a fixed-time window batch scheduling strategy, for example, a large number of data integration and cleaning tasks are concentrated in the early morning for execution. Although it is operationally feasible in terms of business, the result is that the system resource load is extremely unbalanced in different time periods. Secondly, the scheduling system highly relies on manual experience for task configuration. When a new task goes online, it is necessary for the operation and maintenance engineer to estimate the resources required for the task and the execution time based on historical experience, and iteratively adjust the configuration parameters in the test environment for multiple rounds. This process is not only time-consuming but also has a low accuracy rate, affecting business agility and the data product development cycle. In addition, the lack of enforcement of the service level agreement (SLA) guarantee mechanism is also a key problem. The scheduling system cannot perceive the chain reaction brought by potential resource conflicts. Especially when there are high-priority report or model tasks, execution delays are often caused by low-priority tasks occupying resources or sudden tasks being inserted.

[0003] In summary, the current task scheduling system lacks dynamics and is difficult to achieve multiple goals such as optimizing resource utilization rate, shortening the task online cycle, and stably guaranteeing business SLA. Summary of the Invention

[0004] The present invention provides a task load balancing scheduling method and device based on big data, which are used to solve the defects in the prior art that lack dynamics and are difficult to achieve multiple goals such as optimizing resource utilization rate, shortening the task online cycle, and stably guaranteeing business SLA.

[0005] The present invention provides a task load balancing scheduling method based on big data, including: Obtaining the task load status on the big data platform in real time, and constructing a task load status sequence based on the task load status obtained in real time; the task load status includes the running duration of each task and the resource utilization rate of each task during the running process; When reaching a preset update time point, updating the resource competition heat map based on the task load information sequence; the resource competition heat map contains the resource utilization rate of each unit time period within a preset future time period; When a task to be scheduled is received, dynamic decision-making is performed based on the current resource competition heat map and the historical resource utilization rate of the task to be scheduled, to obtain a recommended running time period for the task to be scheduled, and the task to be scheduled is scheduled based on the recommended running time period; the historical resource utilization rate of the task to be scheduled includes the resource utilization rate during the running of the scheduled tasks of the same type.

[0006] According to a task load balancing and scheduling method based on big data provided by the present invention, the dynamic decision-making based on the current resource competition heat map and the historical resource utilization rate of the task to be scheduled to obtain the recommended running time period for the task to be scheduled includes: Based on the resource utilization rate of each unit time period in the current resource competition heat map, the task submission time of the task to be scheduled, and the historical resource utilization rate of the task to be scheduled, a target cost function is established, and dynamic decision-making is performed based on the target cost function to determine the recommended running time period for the task to be scheduled; wherein, the independent variable of the target cost function is a candidate scheduling time period, and the candidate scheduling time period is any one of the unit time periods; The target cost function is established based on the difference between the candidate scheduling time period and the task submission time of the task to be scheduled, and the difference between the resource utilization rate of each type of resource after scheduling the task to be scheduled in the candidate scheduling time period and the resource utilization rate of other candidate scheduling time periods in the resource competition heat map; the resource utilization rate after scheduling the task to be scheduled in the candidate scheduling time period is determined based on the resource utilization rate of the candidate scheduling time in the resource competition heat map and the historical resource utilization rate of the task to be scheduled.

[0007] According to a task load balancing and scheduling method based on big data provided by the present invention, the resource competition heat map includes the resource utilization rate of various types of resources in each unit time period; the various types of resources include network resources, storage resources, and computing resources; The target cost function is established based on the difference between the candidate scheduling time period and the task submission time of the task to be scheduled, the difference between the resource utilization rate of each type of resource after scheduling the task to be scheduled in the candidate scheduling time period and the corresponding resource utilization rate of other candidate scheduling time periods in the resource competition heat map, and the resource utilization rate of each type of resource in the candidate scheduling time period in the resource competition heat map.

[0008] According to a task load balancing and scheduling method based on big data provided by the present invention, the dynamic decision-making based on the target cost function to determine the recommended running time period for the task to be scheduled includes: Construct a state space based on the resource utilization rate of each unit time period in the current resource competition heat map, the task submission time of the to-be-scheduled task, and the historical resource utilization rate of the to-be-scheduled task. Construct an action space based on each of the unit time periods within a preset future time period. Determine a reward function based on the target cost function, and perform dynamic decision-making using a reinforcement learning algorithm based on the state space, the action space, and the reward function to determine the recommended running time period of the to-be-scheduled task.

[0009] According to a task load balancing scheduling method based on big data provided by the present invention, the difference between the resource utilization rates of various resources after scheduling the to-be-scheduled task in the candidate scheduling time period and the resource utilization rates of corresponding resources in other candidate scheduling time periods in the resource competition heat map is based on the variance between the resource utilization rate of any type of resource after scheduling the to-be-scheduled task in the candidate scheduling time period and the resource utilization rate of the any type of resource in other candidate scheduling time periods in the resource competition heat map, and the weights of the any type of resource are obtained by weighted summation; the weights of various resources are determined based on the historical resource utilization rates of various resources corresponding to the to-be-scheduled task.

[0010] According to a task load balancing scheduling method based on big data provided by the present invention, the updating of the resource competition heat map based on the task load information sequence includes: Construct a resource prediction model based on a long short-term memory network. Input the task load information sequence into the resource prediction model to obtain the resource competition heat map output by the resource prediction model.

[0011] According to a task load balancing scheduling method based on big data provided by the present invention, the scheduling of the to-be-scheduled task based on the recommended running time period includes: Determine whether early scheduling or delayed scheduling is required based on the task urgency of the to-be-scheduled task. If early scheduling or delayed scheduling is required, then advance or delay by a preset time based on the recommended running time period, and schedule the to-be-scheduled task based on the advanced or delayed time; otherwise, schedule the to-be-scheduled task in the recommended running time period.

[0012] The present invention also provides a task load balancing scheduling device based on big data, including: A resource load acquisition unit, configured to acquire the task load status on the big data platform in real time, and construct a task load status sequence based on the task load status acquired in real time; the task load status includes the running duration of each task and the resource utilization rate of each task during the running process. A resource competition graph update unit, configured to update a resource competition heat map based on the task load information sequence when reaching a preset update time point; the resource competition heat map includes the resource utilization rates of each unit time period within a preset future time period. A task scheduling decision-making unit, configured to, when receiving a to-be-scheduled task, make a dynamic decision based on the current resource competition heat map and the historical resource utilization rate of the to-be-scheduled task, obtain a recommended running time period for the to-be-scheduled task, and schedule the to-be-scheduled task based on the recommended running time period; the historical resource utilization rate of the to-be-scheduled task includes the resource utilization rates during the running process of the already-scheduled tasks of the same type.

[0013] The present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the program, it implements the method for task load balancing scheduling based on big data as described in any one of the above.

[0014] The present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the method for task load balancing scheduling based on big data as described in any one of the above.

[0015] The present invention further provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the method for task load balancing scheduling based on big data as described in any one of the above.

[0016] A method and device for task load balancing scheduling based on big data provided by the present invention, by obtaining the task load status on the big data platform in real time, constructing a task load status sequence based on the obtained task load status in real time, thereby updating the resource competition heat map based on the task load information sequence when reaching the preset update time point, and then when receiving a to-be-scheduled task, making a dynamic decision based on the current resource competition heat map and the historical resource utilization rate of the to-be-scheduled task, obtaining a recommended running time period for the to-be-scheduled task, and scheduling the to-be-scheduled task based on the recommended running time period, can effectively identify the resource usage trend, accurately predict the future resource competition status, and make an optimal running time recommendation in combination with the historical running data of the to-be-scheduled task, so as to achieve the balanced use of system resources and the maximization of scheduling efficiency, can significantly alleviate the task resource conflict problem, reduce the task queuing time, and improve the system throughput capacity and task completion rate. Description of the Drawings

[0017] To more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0018] Figure 1 It is a schematic flow chart of the task load balancing scheduling method based on big data provided by the present invention; Figure 2 It is a schematic flow chart of the task scheduling dynamic decision-making method provided by the present invention; Figure 3 It is a schematic structural diagram of the task load balancing scheduling device based on big data provided by the present invention; Figure 4 It is a schematic structural diagram of the electronic device provided by the present invention. Detailed implementation manners

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the following clearly and completely describes the technical solutions in the present invention with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments in the present invention belong to the scope of protection of the present invention.

[0020] Figure 1 It is a schematic flow chart of the task load balancing scheduling method based on big data provided by the present invention. As Figure 1 shown, the method includes: Step 110: Real-time obtain the task load status on the big data platform, and construct a task load status sequence based on the real-time obtained task load status; the task load status includes the running duration of each task and the resource utilization rate of each task during operation; Step 120: When reaching the preset update time point, update the resource competition heat map based on the task load information sequence; the resource competition heat map includes the resource utilization rate of each unit time period within a preset future time period; Step 130: When receiving a task to be scheduled, make a dynamic decision based on the current resource competition heat map and the historical resource utilization rate of the task to be scheduled, obtain the recommended running time period of the task to be scheduled, and schedule the task to be scheduled based on the recommended running time period; the historical resource utilization rate of the task to be scheduled includes the resource utilization rate of the scheduled tasks of the same type during operation.

[0021] Here, during operation, the monitoring module continuously and real-time collects the task load status from each computing node of the big data platform. Among them, the task load status refers to the running duration and resource utilization rate during the running process of each task currently running in the big data platform. Specifically, the task load status may include the running duration of each task, the average resource utilization rate of the CPU, the average memory occupancy rate, the disk I / O ratio, and the average resource utilization rate of network resources, etc. These information are collected by the monitoring module deployed on the node side at a set time interval (denoted as the unit time period), and synchronized to the central data integration module. The data integration module can organize and format the data in the task load status received in real time, and splice it with the previously obtained task load status to form a task load status sequence. It can be seen that this task load status sequence is a data set arranged in chronological order. Each task load status in this sequence contains the running duration and resource utilization rate of each task within the corresponding time period, and can describe the resource utilization behavior of each task in different time segments. Based on the task load status sequence, the changing trend of resource utilization can be captured and quantified, so as to reflect the dynamic demand for system resources in each unit time period.

[0022] When entering the preset update time point, for example, every ten minutes, the system will trigger the update mechanism of the resource competition heat map. The manifestation form of the resource competition heat map can be a two-dimensional vector, which contains the resource utilization rate of each unit time period within the preset future time period (such as the next 6 hours). Further, the manifestation form of the resource competition heat map can also be a three-dimensional vector, which contains the resource utilization rate of different types of resources of each unit time period within the preset future time period (such as the next 6 hours), that is, the first dimension is time, the second dimension is the resource type, and the third dimension is the resource utilization rate. The resource competition heat map can be predicted based on the above task load information sequence. By capturing the dynamic demand for system resources in each unit time period in the task load status sequence, the demand for system resources in each unit time period within the preset future time period can be predicted. In one embodiment, a resource prediction model can be constructed based on the time series analysis method. For example, a resource prediction model can be constructed based on the long short-term memory network, and the resource prediction model can be trained in advance using the collected sample data, so as to use the resource prediction model for prediction at the update time point. Specifically, the task load information sequence can be input into the resource prediction model to obtain the resource competition heat map output by the resource prediction model. In addition, the resource prediction model can also be fine-tuned according to the real-time obtained task load status to ensure its prediction accuracy.

[0023] By continuously refreshing the resource competition heat map, a forward-looking resource utilization view can be obtained, and it can be perceived in advance which time periods the resources are relatively surplus and which time periods the resources will face high concurrency pressure, so as to provide a decision-making reference for subsequent task scheduling.

[0024] When a user submits a new task to be scheduled, it is necessary to recommend the most suitable running time period for the task to be scheduled. At this time, the latest resource competition heat map and the historical resource utilization rate of the task to be scheduled can be obtained. Among them, the historical resource utilization rate of the task to be scheduled does not directly come from the task itself, but is a comprehensive portrait obtained through the analysis of the scheduled tasks of the same type in the big data platform. Specifically, scheduled tasks with high similarity to the current task to be scheduled in terms of task type, data processing mode, execution logic, etc. can be found, and their resource utilization rates during operation can be counted and used as a reference for the resource requirements of the current task to be scheduled.

[0025] Based on the resource competition heat map and the historical resource utilization rate of the task to be scheduled, dynamic decision-making can be carried out accordingly to determine the recommended running time period for the task to be scheduled, so as to schedule and run the task during the recommended running time period. In some embodiments, in order to cope with emergencies, it can be determined whether it is necessary to schedule in advance or delay according to the task urgency of the task to be scheduled. If it is necessary to schedule in advance or delay, the time is advanced or delayed by a preset time based on the recommended running time period, and the task to be scheduled is scheduled based on the advanced or delayed time; otherwise, the task to be scheduled is scheduled during the recommended running time period.

[0026] In the dynamic decision-making stage, the insertion situation of the current task can be simulated in multiple future time periods, evaluate whether the system resources can support the execution of the task in each time period, and calculate indicators such as the resource matching degree, system load risk, and estimated completion time of each time period, so as to minimize the system resource conflict and task queuing waiting time and improve the global resource utilization rate on the premise of ensuring the smooth completion of the task operation.

[0027] In some embodiments, a target cost function can be established based on the resource utilization rate of each unit time period in the current resource competition heat map, the task submission time of the task to be scheduled, and the historical resource utilization rate of the task to be scheduled, and dynamic decision-making can be carried out by optimizing the target cost function, so as to determine the recommended running time period of the task to be scheduled.

[0028] Among them, the variables (i.e., independent variables) of the objective cost function are all candidate scheduling time periods, that is, each unit time period within the future preset time period described above. The design of the objective cost function mainly takes into account two objectives: one is the difference between the task submission time and the actual execution time, that is, the scheduling delay; the other is the impact on the system resource status after the task is inserted into a certain candidate time period. The former reflects the timeliness of task response, and the latter reflects the balance of system load. In addition, a layer of constraint can be added, that is, the difference between the resource utilization rate of the candidate scheduling time period and the preset threshold, that is, whether the system is busy, so as to avoid the busy periods of the system as much as possible.

[0029] Specifically, for the candidate scheduling time period t i , the objective cost function can be expressed as follows: C(t i ) = α×D(t i , t submit ) + β×R(t i ) Among them, α and β are weight coefficients used to balance the response timeliness and resource balance; D(t i , t submit ) is the difference between the candidate scheduling time period t i and the task submission time t submit of the task to be scheduled; R(t i ) is the difference between the resource utilization rate after scheduling the task to be scheduled in the candidate scheduling time period t i and the resource utilization rates of other candidate scheduling time periods in the resource competition heat map.

[0030] Here, by superimposing the resource requirements of the task to be scheduled (determined by the historical resource utilization rate of the task to be scheduled) on the existing data in the resource competition heat map (that is, the resource utilization rate of the candidate scheduling time period t i in the resource competition heat map), the new resource utilization rate of this time period after inserting the task to be scheduled can be determined. Subsequently, the difference between the resource utilization rate after scheduling the task to be scheduled in the candidate scheduling time period t i and the resource utilization rates of other candidate scheduling time periods in the resource competition heat map can be compared and known.

[0031] In some other embodiments, the resource competition heat map may include the resource utilization rates of various types of resources within each unit time period, where the various types of resources include network resources, storage resources, and computing resources. By subdividing the resource types, the scheduling decision can be further optimized from the perspective of various types of resources to achieve all-round resource balance.

[0032] In this case, the objective cost function can be established based on the difference between the candidate scheduling time period and the task submission time of the task to be scheduled, the difference between the resource utilization rates of various resources after scheduling the task to be scheduled in this candidate scheduling time period and the resource utilization rates of the corresponding resources in other candidate scheduling time periods in the resource competition heat map, and the resource utilization rates of various resources in this candidate scheduling time period in the resource competition heat map.

[0033] Here, the design of the objective cost function considers three core factors. The first is the time difference between the task submission time and the candidate scheduling time period. Generally, the more timely the scheduling, the more it can meet the real-time requirements of the service. Therefore, the candidate scheduling time period with a smaller time difference should have a higher priority. The second part of the objective cost function considers the degree of difference between the resource utilization rates of various resources in this time period and those in other time periods after inserting the task in a certain candidate scheduling time period. That is, by simulating the pressure increase of various resources that may be caused when the task is scheduled in each candidate scheduling time period and comparing it with the existing resource occupancy in other time periods. Through this comparison, it can be found whether the resource occupancy of various resources caused by scheduling is balanced and whether it is likely to cause resource competition or bottlenecks. The third factor of the objective cost function reflects the resource utilization situation of the current candidate scheduling time period itself to avoid scheduling the task to a time period that is already very busy.

[0034] Finally, the objective cost function is composed of a weighted combination of the above three factors, and its form can be expressed as: C(t i ) = α×D(t i , t submit ) + β×R(t i ) + γ×L(t i ) Among them, α, β, and γ are weight parameters dynamically adjusted according to different scheduling strategies. For tasks with high real-time response requirements, α will be significantly amplified; while for tasks with higher system throughput requirements and tolerable certain scheduling delays, the weights of β and γ will be higher. L(t i ) is the weighted average of the resource utilization rates of various resources in this candidate scheduling time period in the resource competition heat map.

[0035] In some other embodiments, the difference R(t i ) between the resource utilization rates of various resources after scheduling the task to be scheduled in the candidate scheduling time period t j and the resource utilization rates of the corresponding resources in other candidate scheduling time periods (t i , j≠i) in the resource competition heat map can be based on the candidate scheduling time period t iThe variance between the resource utilization rate of any type of resource after scheduling the to-be-scheduled task and the resource utilization rate of the same type of resource in other candidate scheduling time periods t in the resource competition heat map, and the weights of various types of resources are weighted and summed. Among them, the weights of various types of resources can be determined based on the historical resource utilization rates of various types of resources corresponding to the to-be-scheduled task. The higher the historical resource utilization rate of a certain type of resource, the greater the weight of this type of resource. j For example, R(t

[0036] ) can be expressed as follows: i R(t ) = w1×Dif(Res1) + w2×Dif(Res2) +... + wp×Dif(Resp) i where w1, w2,..., wp are the weights of p types of resources respectively, and Dif() represents the variance between the resource utilization rate of the corresponding type of resource after scheduling the to-be-scheduled task in the candidate scheduling time period t and the resource utilization rate of the same type of resource in other candidate scheduling time periods t in the resource competition heat map, and Resi represents the i-th resource. After the target cost function is constructed, the target cost function can be evaluated for all candidate scheduling time periods, and then the candidate scheduling time period with the minimum cost is selected as the recommended running time period for the to-be-scheduled task. i To further improve the accuracy of dynamic decision-making, a reinforcement learning mechanism can also be introduced. Specifically, as j shown, the following method can be used for dynamic decision-making to determine the recommended running time period of the to-be-scheduled task:

[0037] Step 210, construct a state space based on the resource utilization rate of each unit time period in the current resource competition heat map, the task submission time of the to-be-scheduled task, and the historical resource utilization rate of the to-be-scheduled task;

[0038] Step 220, construct an action space based on each of the unit time periods within the preset future time period; Figure 2 Step 230, determine a reward function based on the target cost function, and perform dynamic decision-making using a reinforcement learning algorithm based on the state space, the action space, and the reward function to determine the recommended running time period of the to-be-scheduled task.

[0039]

[0039] Among them, a state space can be constructed based on three types of data: the resource competition heat map, the task submission time of the task to be scheduled, and the historical resource utilization rate. Here, the state space is defined as a multi-dimensional vector composed of the resource utilization rate of various resources, the task submission time of the task to be scheduled, and the historical resource utilization rate in each unit time period within the current time period and several time periods before it. It can be understood that this state space not only includes the objective resource state of the system but also integrates the individual behavior tendencies of the tasks to be scheduled, thus providing sufficient information for guiding scheduling decisions. When constructing the action space, all optional unit time periods within the previously mentioned future preset time period can be used as an action within the action space. For example, if the preset time range is the next six hours and the scheduling granularity is one unit time period every ten minutes, then the action space contains 36 candidate scheduling time periods. Each action represents that the system may choose to schedule the task to start execution at the corresponding time period.

[0040] After the state space and the action space are established, the system needs a mechanism to measure the quality of each action in the current state, that is, a reward function needs to be designed. Among them, the reward function can be constructed based on the previously defined objective cost function. Since the objective cost function describes the scheduling delay cost and the system resource load difference that may be caused by scheduling tasks on a certain candidate scheduling time period, etc., the lower the function value of the objective cost function, the better. However, in the reinforcement learning framework, the higher the function value of the reward function, the better. Therefore, in this embodiment, the objective cost function can be numerically inverted as the basis of the reward function. That is to say, the lower the objective cost function value of a scheduling scheme, the higher its corresponding reward value, thus guiding the reinforcement learning agent to tend to choose scheduling decisions with lower scheduling delays and less system resource load differences (that is, more balanced resource allocation).

[0041] Next, based on the reinforcement learning algorithm, learning and decision-making are carried out based on the state space, action space, and reward function defined above. In this embodiment, a reinforcement learning network such as a Deep Q-Network (DQN) can be used for dynamic learning and decision-making. During the learning process, the agent receives a state input each time, that is, the resource utilization rate of each unit time period in the current resource competition heat map, the task submission time of the task to be scheduled, and the historical resource utilization rate; subsequently, it selects a candidate scheduling time period from the action space as the scheduling action; the reinforcement learning network will evaluate the reward value obtained after this selection based on the reward function and feedback this reward value to the model for parameter update. As the learning process continues, the reinforcement learning network can gradually learn to make the best scheduling choice under different resource distribution states, different types of tasks, and different latency pressures. Finally, when a new task to be scheduled arrives, instead of enumerating the objective cost function values of all candidate scheduling time periods, the trained reinforcement learning network can be directly used for state analysis and action prediction, so as to recommend the optimal running time period in a short time.

[0042] It can be seen that this dynamic scheduling mechanism based on reinforcement learning can continuously learn the optimal strategy through actual operation data and has the ability of self-adaptation. In a multi-task high-concurrency environment, the scheduling strategy it learns can maintain high efficiency in task scheduling and high utilization of system resources under the condition of frequent resource fluctuations.

[0043] In summary, the task load balancing and scheduling method provided by the embodiment of the present invention can effectively identify the resource usage trend, accurately predict the future resource competition state, and make the optimal running time recommendation by combining the historical operation data of the task to be scheduled, so as to realize the balanced use of system resources and the maximization of scheduling efficiency, significantly alleviate the task resource conflict problem, reduce the task queuing time, and improve the system throughput capacity and task completion rate.

[0044] Next, a task load balancing and scheduling device based on big data provided by the present invention will be described. The task load balancing and scheduling device based on big data described below can be mutually corresponding and referred to the task load balancing and scheduling method described above.

[0045] Based on any of the above embodiments, Figure 3It is a schematic structural diagram of a task load balancing scheduling device based on big data provided by the present invention. As Figure 3 shown, the device includes: A resource load acquisition unit 310, configured to acquire the task load status on the big data platform in real time, and construct a task load status sequence based on the task load status acquired in real time; the task load status includes the running duration of each task and the resource utilization rate of each task during operation. A resource competition graph update unit 320, configured to update the resource competition heat map based on the task load information sequence when reaching a preset update time point; the resource competition heat map includes the resource utilization rate of each unit time period within a preset future time period. A task scheduling decision unit 330, configured to make a dynamic decision based on the current resource competition heat map and the historical resource utilization rate of the to-be-scheduled task when receiving the to-be-scheduled task, obtain the recommended running time period of the to-be-scheduled task, and schedule the to-be-scheduled task based on the recommended running time period; the historical resource utilization rate of the to-be-scheduled task includes the resource utilization rate of the scheduled tasks of the same type during operation.

[0046] The device provided by the embodiment of the present invention acquires the task load status on the big data platform in real time, constructs a task load status sequence based on the task load status acquired in real time, thereby updating the resource competition heat map based on the task load information sequence when reaching the preset update time point, and then, when receiving the to-be-scheduled task, makes a dynamic decision based on the current resource competition heat map and the historical resource utilization rate of the to-be-scheduled task, obtains the recommended running time period of the to-be-scheduled task, and schedules the to-be-scheduled task based on the recommended running time period. It can effectively identify the resource usage trend, accurately predict the future resource competition status, and make the optimal running time recommendation in combination with the historical operation data of the to-be-scheduled task, so as to achieve the balanced use of system resources and the maximization of scheduling efficiency, significantly alleviate the task resource conflict problem, reduce the task queuing time, and improve the system throughput capacity and task completion rate.

[0047] Based on any of the above embodiments, making a dynamic decision based on the current resource competition heat map and the historical resource utilization rate of the to-be-scheduled task, and obtaining the recommended running time period of the to-be-scheduled task includes: Establishing an objective cost function based on the resource utilization rate of each unit time period in the current resource competition heat map, the task submission time of the to-be-scheduled task, and the historical resource utilization rate of the to-be-scheduled task, and making a dynamic decision based on the objective cost function to determine the recommended running time period of the to-be-scheduled task; wherein, the independent variable of the objective cost function is a candidate scheduling time period, and the candidate scheduling time period is any one of the unit time periods; The target cost function is established based on the difference between the candidate scheduling time period and the task submission time of the to-be-scheduled task, and the difference between the resource utilization rate after scheduling the to-be-scheduled task in the candidate scheduling time period and the resource utilization rates of other candidate scheduling time periods in the resource competition heat map; the resource utilization rate after scheduling the to-be-scheduled task in the candidate scheduling time period is determined based on the resource utilization rate of the candidate scheduling time in the resource competition heat map and the historical resource utilization rate of the to-be-scheduled task.

[0048] Based on any of the above embodiments, the resource competition heat map includes the resource utilization rates of various types of resources in each unit time period; the various types of resources include network resources, storage resources, and computing resources; The target cost function is established based on the difference between the candidate scheduling time period and the task submission time of the to-be-scheduled task, the difference between the resource utilization rates of various types of resources after scheduling the to-be-scheduled task in the candidate scheduling time period and the resource utilization rates of the corresponding resources in other candidate scheduling time periods in the resource competition heat map, and the resource utilization rates of various types of resources in the candidate scheduling time period in the resource competition heat map.

[0049] Based on any of the above embodiments, the dynamic decision-making based on the target cost function to determine the recommended running time period of the to-be-scheduled task includes: Construct a state space based on the resource utilization rates of each unit time period in the current resource competition heat map, the task submission time of the to-be-scheduled task, and the historical resource utilization rate of the to-be-scheduled task; Construct an action space based on each of the unit time periods within a preset future time period; Determine a reward function based on the target cost function, and perform dynamic decision-making using a reinforcement learning algorithm based on the state space, the action space, and the reward function to determine the recommended running time period of the to-be-scheduled task.

[0050] Based on any of the above embodiments, the difference between the resource utilization rates of various types of resources after scheduling the to-be-scheduled task in the candidate scheduling time period and the resource utilization rates of the corresponding resources in other candidate scheduling time periods in the resource competition heat map is obtained by weighted summation based on the variance between the resource utilization rate of any type of resource after scheduling the to-be-scheduled task in the candidate scheduling time period and the resource utilization rate of the any type of resource in other candidate scheduling time periods in the resource competition heat map, and the weight of the any type of resource; the weights of various types of resources are determined based on the historical resource utilization rates of the to-be-scheduled task corresponding to various types of resources.

[0051] Based on any of the above embodiments, the updating of the resource competition heat map based on the task load information sequence includes: Construct a resource prediction model based on a long short-term memory network; Input the task load information sequence into the resource prediction model to obtain a resource competition heat map output by the resource prediction model.

[0052] Based on any of the above embodiments, scheduling the to-be-scheduled task based on the recommended running time period includes: Determine whether early scheduling or delayed scheduling is required based on the task urgency of the to-be-scheduled task; If early scheduling or delayed scheduling is required, advance or delay by a preset time based on the recommended running time period, and schedule the to-be-scheduled task based on the advanced or delayed time; otherwise, schedule the to-be-scheduled task during the recommended running time period.

[0053] Figure 4 It is a schematic structural diagram of an electronic device provided by the present invention, as Figure 4 shown. The electronic device may include: a processor 410, a memory 420, a communication interface 430, and a communication bus 440. Among them, the processor 410, the memory 420, and the communication interface 430 communicate with each other through the communication bus 440. The processor 410 can call the logical instructions in the memory 420 to execute a task load balancing scheduling method based on big data. The method includes: obtaining the task load status on the big data platform in real time, and constructing a task load status sequence based on the task load status obtained in real time; the task load status includes the running duration of each task and the resource utilization rate of each task during the running process; when reaching a preset update time point, updating the resource competition heat map based on the task load information sequence; the resource competition heat map contains the resource utilization rate of each unit time period within a preset future time period; when receiving a to-be-scheduled task, making a dynamic decision based on the current resource competition heat map and the historical resource utilization rate of the to-be-scheduled task to obtain the recommended running time period of the to-be-scheduled task, and scheduling the to-be-scheduled task based on the recommended running time period; the historical resource utilization rate of the to-be-scheduled task includes the resource utilization rate of the already scheduled tasks of the same type during the running process.

[0054] In addition, when the logical instructions in the above-mentioned memory 420 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0055] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the big data-based task load balancing scheduling method provided by the above-mentioned various methods. The method includes: obtaining the task load status on the big data platform in real time, and constructing a task load status sequence based on the task load status obtained in real time; the task load status includes the running duration of each task and the resource utilization rate of each task during the running process; when reaching a preset update time point, updating the resource competition heat map based on the task load information sequence; the resource competition heat map includes the resource utilization rate of each unit time period within a preset future time period; when receiving a task to be scheduled, making a dynamic decision based on the current resource competition heat map and the historical resource utilization rate of the task to be scheduled, obtaining the recommended running time period of the task to be scheduled, and scheduling the task to be scheduled based on the recommended running time period; the historical resource utilization rate of the task to be scheduled includes the resource utilization rate of the same type of scheduled tasks during the running process.

[0056] In another aspect, the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the task load balancing scheduling method based on big data provided above. The method includes: obtaining the task load status on the big data platform in real time, and constructing a task load status sequence based on the task load status obtained in real time; the task load status includes the running duration of each task and the resource utilization rate of each task during the running process; when reaching a preset update time point, updating the resource competition heat map based on the task load information sequence; the resource competition heat map includes the resource utilization rate of each unit time period within a preset future time period; when receiving a task to be scheduled, making a dynamic decision based on the current resource competition heat map and the historical resource utilization rate of the task to be scheduled, obtaining the recommended running time period of the task to be scheduled, and scheduling the task to be scheduled based on the recommended running time period; the historical resource utilization rate of the task to be scheduled includes the resource utilization rate of the already scheduled tasks of the same type during the running process.

[0057] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0058] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, also by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of each embodiment of the present invention.

Claims

1. A task load balancing and scheduling method based on big data, characterized in that, Including: Obtain the task load status on the big data platform in real time, and construct a task load status sequence based on the real-time obtained task load status; the task load status includes the running duration of each task and the resource utilization rate of each task during the running process; When reaching the preset update time point, update the resource competition heat map based on the task load information sequence; the resource competition heat map includes the resource utilization rate of each unit time period within a preset future time period; When receiving a to-be-scheduled task, make a dynamic decision based on the current resource competition heat map and the historical resource utilization rate of the to-be-scheduled task to obtain the recommended running time period of the to-be-scheduled task, and schedule the to-be-scheduled task based on the recommended running time period; the historical resource utilization rate of the to-be-scheduled task includes the resource utilization rate of the already-scheduled tasks of the same type during the running process.

2. The method for task load balancing scheduling based on big data according to claim 1, wherein The making a dynamic decision based on the current resource competition heat map and the historical resource utilization rate of the to-be-scheduled task to obtain the recommended running time period of the to-be-scheduled task includes: Based on the resource utilization rate of each unit time period in the current resource competition heat map, the task submission time of the to-be-scheduled task, and the historical resource utilization rate of the to-be-scheduled task, establish an objective cost function, and make a dynamic decision based on the objective cost function to determine the recommended running time period of the to-be-scheduled task; Wherein, the independent variable of the objective cost function is a candidate scheduling time period, and the candidate scheduling time period is any one of the unit time periods; The objective cost function is established based on the difference between the candidate scheduling time period and the task submission time of the to-be-scheduled task, and the difference between the resource utilization rate after scheduling the to-be-scheduled task in the candidate scheduling time period and the resource utilization rates of other candidate scheduling time periods in the resource competition heat map; the resource utilization rate after scheduling the to-be-scheduled task in the candidate scheduling time period is determined based on the resource utilization rate of the candidate scheduling time in the resource competition heat map and the historical resource utilization rate of the to-be-scheduled task.

3. The method for task load balancing scheduling based on big data according to claim 2, wherein The resource competition heat map includes the resource utilization rate of various types of resources within each unit time period; the various types of resources include network resources, storage resources, and computing resources; The objective cost function is established based on the difference between the candidate scheduling time period and the task submission time of the to-be-scheduled task, the difference between the resource utilization rates of various types of resources after scheduling the to-be-scheduled task in the candidate scheduling time period and the resource utilization rates of the corresponding resources of other candidate scheduling time periods in the resource competition heat map, and the resource utilization rates of various types of resources of the candidate scheduling time period in the resource competition heat map.

4. The task workload balancing and scheduling method based on big data according to claim 3, characterized in that The making a dynamic decision based on the objective cost function to determine the recommended running time period of the to-be-scheduled task includes: Based on the resource utilization rate of each unit time period in the current resource competition heat map, the task submission time of the to-be-scheduled task, and the historical resource utilization rate of the to-be-scheduled task, construct a state space; Based on each of the unit time periods within the preset future time period, construct an action space; Determine a reward function based on the target cost function, and perform dynamic decision-making using a reinforcement learning algorithm based on the state space, the action space, and the reward function to determine the recommended running time period of the to-be-scheduled task.

5. The task load balancing scheduling method based on big data according to claim 3, characterized in that The difference between the resource utilization rate of various resources after scheduling the to-be-scheduled task in the candidate scheduling time period and the resource utilization rate of the corresponding resources in other candidate scheduling time periods in the resource competition heat map is obtained by weighted summation based on the variance between the resource utilization rate of any type of resource after scheduling the to-be-scheduled task in the candidate scheduling time period and the resource utilization rate of the any type of resource in other candidate scheduling time periods in the resource competition heat map, and the weight of the any type of resource; the weights of various resources are determined based on the historical resource utilization rates of various resources corresponding to the to-be-scheduled task.

6. The method for task load balancing scheduling based on big data according to any one of claims 1 to 5, characterized in that The updating the resource competition heat map based on the task load information sequence includes: Construct a resource prediction model based on a long short-term memory network; Input the task load information sequence into the resource prediction model to obtain the resource competition heat map output by the resource prediction model.

7. The method for task load balancing scheduling based on big data according to any one of claims 1 to 5, characterized in that, The scheduling the to-be-scheduled task based on the recommended running time period includes: Determine whether early scheduling or late scheduling is required based on the task urgency of the to-be-scheduled task; If early scheduling or late scheduling is required, advance or delay by a preset time based on the recommended running time period, and schedule the to-be-scheduled task based on the advanced or delayed time; otherwise, schedule the to-be-scheduled task in the recommended running time period.

8. A task load balancing and scheduling device based on big data, characterized in that, Includes: A resource load acquisition unit, configured to acquire the task load status on the big data platform in real time, and construct a task load status sequence based on the task load status acquired in real time; the task load status includes the running duration of each task and the resource utilization rate of each task during operation; A resource competition graph updating unit, configured to update the resource competition heat map based on the task load information sequence when reaching a preset update time point; the resource competition heat map includes the resource utilization rates of each unit time period within a future preset time period; A task scheduling decision-making unit, configured to, when receiving a to-be-scheduled task, perform dynamic decision-making based on the current resource competition heat map and the historical resource utilization rate of the to-be-scheduled task to obtain the recommended running time period of the to-be-scheduled task, and schedule the to-be-scheduled task based on the recommended running time period; the historical resource utilization rate of the to-be-scheduled task includes the resource utilization rate of the already scheduled tasks of the same type during operation.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the big data-based task load balancing scheduling method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the big data-based task load balancing scheduling method according to any one of claims 1 to 7.

Images