Self-adaptive scheduling method for data intermediate stations based on QoS (Quality of Service) driving
Through the QoS-driven data middle platform adaptive scheduling method, the priority is calculated using attributes such as task type, waiting time and workload, and dynamically adjusting resource allocation, the problem of insufficient task priority distinction in traditional scheduling methods is solved, and users are improved Experience and resource utilization ensure system stability and resource efficiency.
Patent Information
- Application Number
- CN202411037794.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-12-20
AI Technical Summary
Traditional task scheduling methods cannot effectively distinguish the priority of tasks, resulting in poor user experience, low service quality of data middle platform, and low resource utilization rate.
Adaptive scheduling method of data center-driven based on QoS is used to quantify and normalize properties such as task type, waiting time and workload, calculate task priority in combination with hierarchical analysis method, and schedule tasks according to the minimum completion time, dynamically Adjust resource allocation.
Ensure that high-priority tasks are performed first, improve user experience and system stability, maximize resource utilization, avoid resource waste or overload, and make the decision process clearer and controllable through partial solution problems.
Smart Images
Figure CN119166326A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of task scheduling, and involves a dynamic task scheduling method for a data middle station, specifically a QoS-driven adaptive scheduling method for a data middle station. Background Art
[0002] In the digital age, data middle platforms, as a new type of data management and service mechanism, are gradually recognized and widely used by enterprises. Task scheduling plays a key role in the entire data processing life cycle of data middle platforms. Traditional strategies cannot distinguish the priority of tasks, resulting in poor user experience and reduced service quality of data middle platforms. Task scheduling based on QoS (quality of service) is an important scheduling idea in task scheduling.
[0003] QoS-based task scheduling selects attributes that are more important to users to assign priorities to tasks, ensuring that high-priority tasks are executed first, which means that critical tasks can be completed on time, ensuring the availability and stability of the system while greatly improving user satisfaction and experience quality. In addition, by reasonably allocating system resources, QoS task scheduling can maximize resource utilization. It can dynamically adjust resource allocation to adapt to changes in system load, thereby avoiding resource waste or overload.
[0004] The weight calculation based on the hierarchical analysis method breaks down the decision problem into a hierarchical structure and breaks down large problems into manageable sub-problems, making the decision process clearer and more controllable. At the same time, the hierarchical analysis method considers multiple criteria and factors and quantifies and compares their importance. This allows for a more comprehensive evaluation of various options and avoids the limitations of making decisions based on only a single criterion. Summary of the invention
[0005] In view of the problem that traditional strategies cannot distinguish the priorities of tasks, the purpose of the present invention is to propose a data middle station adaptive scheduling method based on QoS drive to realize the function of task priority calculation.
[0006] In order to achieve the above object, the technical solution of the present invention is as follows:
[0007] A data middle station adaptive scheduling method based on QoS drive, characterized in that the method comprises the following steps:
[0008] (1) After the tasks arrive at the task queue, they are batch processed every period of time: appropriate attributes are selected and quantified, and a new task set is formed based on the quantified attributes;
[0009] (2) Normalize the quantified attributes and calculate the priority of each task in the task queue according to the established priority calculation model;
[0010] (3) Sort the task queue in descending order of priority;
[0011] (4) Initialize and adjust the minimum completion time matrix and schedule tasks to resources;
[0012] (5) Record the results of each task scheduling and provide feedback.
[0013] The appropriate attributes selected in step (1) are: task type, task waiting time, and task workload.
[0014] The attribute quantification processing is as follows: the expected priority of the task type is set as high priority, medium priority and low priority; the task waiting time is described by the difference between the current time when processing the batch of tasks and the task arrival time; the task workload is the number of instructions in the task, representing the workload of the task.
[0015] The attribute normalization process in step (2) is as follows:
[0016]
[0017] Where, X i represents the i-th attribute value in the attribute value set, X min Indicates the minimum value, X max Indicates the maximum value, Y i Indicates attribute value X i Normalized results.
[0018] The priority calculation model is: according to the hierarchical model established by the hierarchical analysis method, the normalized indicators are weighted and the priority quantitative value is obtained.
[0019] The sorting in step (3) is as follows: the calculated priority value is used as a new attribute of the task, and the attribute is used as a reference to readjust the task queue using bubble sort.
[0020] The initialization and adjustment of the minimum completion time matrix in step (4) is as follows: the tasks are scheduled to the resources with the minimum completion time in sequence and the minimum completion time matrix is updated after each task is scheduled to obtain the minimum completion time in the current state until all tasks are scheduled to the corresponding resources.
[0021] The feedback in step (5) is as follows: each time the task is scheduled, the task scheduling time and the final completion time are recorded to observe whether the task is scheduled within the effective time.
[0022] In the present invention, the adaptive scheduling model of the data center is as follows: Figure 1As shown in the figure, the model is mainly divided into three parts: priority calculation module, task sorting and scheduling module and task feedback module. The main process of the model includes four stages: indicator quantification, priority calculation, task sorting and scheduling, and scheduling feedback.
[0023] Indicator quantification stage: The indicators selected in the model need to be quantified before they can serve as the basis for subsequent priority calculations. Quantification needs to be completed at the very beginning of scheduling, that is, reading the data of the tasks, selecting the required attributes, reorganizing the task set, and calculating the quantified values for them.
[0024] Priority calculation stage: After the task set is formed, the priority calculation module compares the relative importance of indicators based on the hierarchical model established by the hierarchical analysis method, and assigns relative weights to them. The quantified indicators are normalized to eliminate the adverse effects of large differences in the dimensions of different attribute values on mixed priorities. Finally, the normalized indicators are weighted and summed to obtain the priority quantization value.
[0025] Task sorting and scheduling phase: In the process of task scheduling, in order to ensure that each task is completed as soon as possible, the minimum completion time of the task is used as the main scheduling method. The minimum completion time matrix is initialized by calculating the completion time of each task on different resources. Then, according to the order of the task queue, the tasks are scheduled to the resources with the minimum completion time. Each time a task is scheduled, the minimum completion time matrix is updated to obtain the minimum completion time of the remaining tasks in the current state until all tasks are scheduled to the corresponding resources.
[0026] Task feedback stage: Each time the task is scheduled, the task scheduling time and the final completion time are recorded to observe whether the task is scheduled within the effective time.
[0027] This method selects three task attributes, namely, expected task priority, task waiting time, and task length, as the measurement indicators of the priority of tasks in the data, and uses the hierarchical analysis method to obtain the relative weights of each indicator. The final priority of the task is calculated according to the weight and the task queue is sorted according to the priority. Finally, the tasks are scheduled in sequence according to the minimum completion time of the task on the resource. This method fully considers the attributes of the tasks in the data, improves the user experience, improves the task completion efficiency and optimizes the resource load problem to a certain extent.
[0028] The beneficial effects of the present invention are:
[0029] Task scheduling based on QoS selects attributes that are more important to users to assign priorities to tasks, ensuring that high-priority tasks are executed first, ensuring system availability and stability, and greatly improving user satisfaction and experience quality. In addition, by reasonably allocating system resources, resource utilization is maximized. It can dynamically adjust resource allocation to adapt to changes in system load, thereby avoiding resource waste or overload.
[0030] The weight calculation based on the hierarchical analysis method breaks down the decision problem into a hierarchical structure and breaks down large problems into manageable sub-problems, making the decision process clearer and more controllable. At the same time, the hierarchical analysis method considers multiple criteria and factors and quantifies and compares their importance. This allows for a more comprehensive evaluation of various options and avoids the limitations of making decisions based on only a single criterion. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 Provides an adaptive scheduling model architecture for the data center;
[0032] Figure 2 The working process of the adaptive scheduling model for the data center. DETAILED DESCRIPTION
[0033] The present invention will be further described below in conjunction with the accompanying drawings.
[0034] A QoS-driven data center adaptive scheduling method, such as Figure 1 As shown in the figure, the framework of the model is mainly divided into three parts: priority calculation module, task sorting and scheduling module and task feedback module. The priority calculation module and task sorting and scheduling module are the core of the framework and are the modules for implementing specific scheduling.
[0035] The workflow of this model is as follows Figure 2 As shown, the specific process is as follows:
[0036] Step 1: When a task arrives at the task queue, the corresponding indicators are selected and quantified, and a new task set is formed based on the quantified attributes.
[0037] Expected priority refers to the prediction of the urgency of tasks waiting to be scheduled. This concept emphasizes the importance of tasks relative to other tasks in task management and scheduling. It is usually set based on the deadline, time sensitivity or other urgency factors of the task. Here, setting expected priority includes high priority, medium priority and low priority.
[0038] Waiting delay is widely used in various queuing systems and scenarios. It provides a quantifiable indicator to measure the time delay experienced by tasks or individuals while waiting in line. Real-time monitoring and calculation of delay time can not only be used to adjust the priority of tasks, but also help the system allocate resources more efficiently to shorten the overall waiting time. The waiting delay TL can be calculated by the current time T when processing the batch of tasks. now and the task arrival time T arrive Calculate the difference between .
[0039] TL=T now -T arrive #
[0040] The task length is the number of instructions in a task, which indicates the workload of the task, especially the number of instructions for loop unrolling. In this definition, the task length is a key measure of the workload and execution complexity of task T. By considering the task length, the execution time and resource requirements of the task can be more accurately evaluated.
[0041] Step 2: The priority calculation module reads the attribute data of the task set.
[0042] The priority calculation module compares the importance of indicators and calculates their relative weights based on the hierarchical model established by the hierarchical analysis method, and then normalizes the quantified indicators to eliminate the adverse effects of large differences in the dimensions of different attribute values on mixed priorities. Finally, the normalized indicators are weighted and summed to obtain the priority quantization value.
[0043] In order to combine multiple attributes into priorities, we normalize the attribute values. Normalization can transform and map data within a certain range, eliminating the adverse effects of large differences in the dimensions of different attribute values on mixed priorities.
[0044]
[0045] Where, X i represents the i-th attribute value in the attribute value set, X min Indicates the minimum value, X max Indicates the maximum value, Y i Indicates attribute value X i Normalized results.
[0046] According to the hierarchical model established by the hierarchical analysis method, the relative weights calculated, and the results of the normalization of the above attribute values, the task priority calculation model is as follows:
[0047] P i =ω1×ep i +ω2×tl i +ω3×wl i#
[0048] Where ep, tl, and wl represent the normalized results of the three attribute values EP, TL, and WL, respectively.
[0049] Step 3: The task sorting and scheduling module re-sorts the task queue according to the task priority, in descending order of priority, to obtain a new task queue.
[0050] Step 4: The task sorting and scheduling module initializes the minimum completion time matrix and sequentially schedules the tasks in the queue to the resources with the minimum completion time until all tasks are scheduled to the corresponding resources.
[0051] The expected completion time (ECT) matrix contains the expected completion time of all tasks on all resources. In the ETC matrix, the element in the i-th row represents the expected completion time of a given task T. i The expected completion time on different resources; the elements in the jth column represent the given resource R j The expected completion time of different tasks on the same task. ECT(i, j) can represent the expected completion time of task T i In the resource R j The expected completion time.
[0052] The start time (ST) matrix contains the earliest time when all resources can start to be used after completing the tasks assigned to them. ST(j) can represent the resource R j The start time of the
[0053] The minimum completion time (MCT) matrix contains the completion time of all tasks on all resources. In the MCT matrix, the elements in a row represent the completion time of a given task on different resources, while the elements in a column represent the completion time of different tasks on a given resource.
[0054] From the above definition, we can get the calculation formula of MCT matrix:
[0055] MCT(i,j)=ECT(i,j)+ST(j)#
[0056]
[0057] Step 5: The task feedback module records the task scheduling time and the final completion time each time to observe whether the task is scheduled within the effective time.
[0058] This method selects three task attributes, namely, expected task priority, task waiting time, and task length, as the measurement indicators of the priority of tasks in the data, and uses the hierarchical analysis method to obtain the relative weights of each indicator. The final priority of the task is calculated according to the weight and the task queue is sorted according to the priority. Finally, the tasks are scheduled in sequence according to the minimum completion time of the task on the resource. This method fully considers the attributes of the tasks in the data, improves the user experience, improves the task completion efficiency and optimizes the resource load problem to a certain extent.
Claims
1. A QoS-driven data middle station adaptive scheduling method, characterized in that The method comprises the following steps: (1) After the tasks arrive at the task queue, they are batch processed every period of time: appropriate attributes are selected and quantified, and a new task set is formed based on the quantified attributes; (2) Normalize the quantified attributes and calculate the priority of each task in the task queue according to the established priority calculation model; (3) Sort the task queue in descending order of priority; (4) Initialize and adjust the minimum completion time matrix and schedule tasks to resources; (5) Record the results of each task scheduling and provide feedback.
2. The QoS-driven data middle station adaptive scheduling method according to claim 1 is characterized in that: The appropriate attributes selected in step (1) are: task type, task waiting time, and task workload.
3. The QoS-driven data middle station adaptive scheduling method according to claim 2 is characterized in that: The attribute quantification processing in step (1) is as follows: the expected priority of the task type is set as high priority, medium priority and low priority; the task waiting time is described by the difference between the current time when processing the batch of tasks and the task arrival time; the task workload is the number of instructions in the task, representing the workload of the task.
4. The QoS-driven data middle station adaptive scheduling method according to claim 1 is characterized in that: The attribute normalization process in step (2) is as follows: Where, X i represents the i-th attribute value in the attribute value set, X min represents the minimum value, X max Indicates the maximum value, Y i Indicates attribute value X i Normalized results.
5. The QoS-driven data middle station adaptive scheduling method according to claim 1 is characterized in that: The priority calculation model in step (2) is: according to the hierarchical model established by the hierarchical analysis method, the normalized indicators are weighted and the priority quantitative value is obtained.
6. The QoS-driven data middle station adaptive scheduling method according to claim 1 is characterized in that: The sorting in step (3) is as follows: the calculated priority value is used as a new attribute of the task, and the attribute is used as a reference to readjust the task queue using bubble sort.
7. The QoS-driven data middle station adaptive scheduling method according to claim 1 is characterized in that: The initialization and adjustment of the minimum completion time matrix in step (4) is as follows: the tasks are scheduled to the resources with the minimum completion time in sequence and the minimum completion time matrix is updated after each task is scheduled to obtain the minimum completion time in the current state until all tasks are scheduled to the corresponding resources.
8. The QoS-driven data middle station adaptive scheduling method according to claim 1 is characterized in that: The feedback in step (5) is as follows: each time the task is scheduled, the task scheduling time and the final completion time are recorded to observe whether the task is scheduled within the effective time.