ETL task scheduling method and device based on weight

Abstract

The invention relates to a weight-based ETL task scheduling method and device, and belongs to the field of medical big data management. The method comprises three treatment stages, in the task preprocessing stage, the importance weight WEj of each ETL task is calculated, and an ETL task set E is divided into a high-level task set H and a low-level task set L through a task grading algorithm; in a task allocation stage, calculating performance weights WNi of different nodes, obtaining source data volumes of ETL tasks, and allocating tasks in a high-level task set H and a low-level task set L to task sets of different nodes through a weighted greedy algorithm to form a set G consisting of a plurality of node task sets; in the task execution stage, the task execution sequence of each node is dynamically adjusted through a hierarchical high response ratio priority algorithm. The task execution duration can be pre-estimated, and the cluster resource utilization rate is improved; and dynamically adjusting the execution sequence of the tasks by calculating the response ratio of the to-be-executed tasks.

Description

Technical Field

[0001] The present invention relates to a weight-based ETL task scheduling method and device, belonging to the technical field of medical big data management. Background Art

[0002] The ETL (Extract Transform Load) task is a key step in data processing, including data extraction, transformation and loading. In this process, the original data is extracted from different sources, cleaned, transformed and mapped, and finally loaded into the data warehouse to provide a basis for analysis and mining. In the medical field, considering the problems of multi-source heterogeneity, low data quality, and complex business, it is usually necessary to use ETL tools with strong data processing capabilities and high flexibility for the collection and management of medical data. However, when most mainstream ETL tools are applied to smart medical construction such as comprehensive operations and clinical research, they are faced with problems such as low data pertinence and low cluster resource utilization, and cannot meet the diverse data governance and cluster scheduling requirements in medical scenarios. The Kettle tool used in the middleware of the present invention, although it has the advantages of rich data processing functions and strong scalability, can be well applied in various big data scenarios, but when aggregating medical big data through cluster scheduling, it will face the following problems: (a) When reading tasks, the cluster cannot distinguish the importance of tasks, resulting in too long waiting time for some key data aggregation tasks. (b) When allocating tasks, the cluster does not consider the impact of data scale and node performance on task execution time, resulting in uneven cluster resource allocation. (c) When executing tasks, the cluster cannot guarantee equal execution opportunities for long and short tasks and may face serious task starvation problems. Summary of the invention

[0003] In order to solve the above-mentioned problems, the present invention provides a weight-based ETL task scheduling method and device, which is used to ensure priority scheduling of important aggregation tasks by dividing the levels of ETL tasks; estimate the task execution time by data scale and node performance to improve cluster resource utilization; and dynamically adjust the execution order of tasks by calculating the response ratio of tasks to be executed.

[0004] The technical solution of the present invention is: in a first aspect, the present invention provides a weight-based ETL task scheduling method, comprising three processing stages;

[0005] In the task preprocessing stage, calculate the importance weight W of each ETL task Ej , the ETL task set E is divided into a high-level task set H and a low-level task set L through a task classification algorithm;

[0006] In the task allocation phase, the performance weights W of different nodes are calculated. Ni, obtain the source data volume of the ETL task, and assign the tasks in the high-level task set H and the low-level task set L to the task sets of different nodes through the weighted greedy algorithm to form a set G composed of multiple node task sets;

[0007] During the task execution phase, the task execution order of each node is dynamically adjusted through a hierarchical high response ratio priority algorithm.

[0008] Furthermore, in the preprocessing stage, the importance weight W Ej The specific calculation formula is as follows:

[0009] W Ej =W Ej1 *25%+W Ej2 *25%+W Ej3 *5%+W Ej4 *5% (1)

[0010] Where j represents the task number; W Ej1 Indicates whether the ETL task is required by the hospital, and the value is {0,1}; W Ej2 Indicates whether the ETL task is related to the core data of the medical system, and the value is {0,1}; W Ej3 represents the number of associations between ETL tasks and other medical systems, with values ​​of {0, 1, 2, 3, 4, 5}; W Ej4 Indicates the number of associations between an ETL task and other ETL tasks. The value is {0,1,2,3,4,5}.

[0011] Furthermore, the preprocessing stage specifically includes the following:

[0012] First, calculate the importance W of each task in the ETL task set E Ej ;

[0013] Then, combined with the "80 / 20 rule", the tasks in the ETL task set E are reordered from the largest to the smallest according to the importance weight. Finally, the tasks ranked in the top 20% of the ETL task set E are taken as high-level tasks that need to be executed first and stored in the high-level task set H; the remaining tasks are regarded as low-level tasks and stored in the low-level task set L, so that more important high-level tasks can be processed first in the task allocation and execution stage.

[0014] Furthermore, the task allocation stage specifically includes the following steps: randomly extracting a set of ETL task sets Ew from the ETL task set E, and recording the total execution time of the task set Ew at the node i as Tw i , performance weights of different nodes W Ni The calculation method is as follows:

[0015]

[0016]

[0017] Among them, Tw m Indicates the maximum total execution time, n is the total number of nodes;

[0018] After calculating the performance weight of each node, the nodes are first reordered from large to small according to the performance weight to form a node information set N = {N1, N2, ..., N n}; Then for each node information N i Create the corresponding task set G i , forming a set G = {G1, G2, ..., G n}; Calculate the total number of tasks S for H and L at the same time H With S L And reorder the tasks in sets H and L according to the source data volume from large to small; finally, the weighted greedy algorithm is used to assign the tasks in sets H and L to the node task set G i middle.

[0019] Furthermore, in the weighted greedy algorithm of the task allocation stage, n tasks with higher importance are extracted from the sets H and L, and tasks with large data volumes are allocated to nodes with better performance; then, according to the estimated execution time S of each node, i , and sequentially allocate the remaining tasks in sets H and L, so as to give priority to high-level tasks and make full use of cluster resources to improve scheduling efficiency; S i The calculation formula is:

[0020]

[0021] Where D i represents the total amount of source data of the tasks assigned to node i, and d represents the current task to be assigned E f The amount of source data.

[0022] Furthermore, the task execution phase includes:

[0023] According to the task levels divided in the task preprocessing stage, the task set G on each node i Divided into high-level task set G Hi With low-level task set G Li ; Then execute the set G in turn through the high response ratio priority algorithm Hi With G Li , comprehensively consider the task importance, task waiting time, and task execution time, so as to give priority to important tasks and ensure the fairness of task execution; the calculation formula of response ratio is:

[0024]

[0025] T hi =Ts hi -Ta hi (6)

[0026] Te hi ≈k*D h (7)

[0027] Among them, R hi represents the response ratio of task h on node i, Tw hi represents the waiting time of task h on node i, Te hi represents the execution time of task h on node i, Ta hi represents the arrival time of task h at node i, Ts hi represents the execution start time of task h on node i, k is a constant, D h Represents the amount of source data for task h.

[0028] In a second aspect, the present invention provides a weight-based ETL task scheduling device, comprising a module for executing the method described in the first aspect.

[0029] The beneficial effects of the present invention are:

[0030] In the pre-processing stage, the ETL tasks are classified and high-priority tasks are selected for priority scheduling, ensuring that important aggregation tasks are scheduled first;

[0031] In the allocation phase, the task execution time is estimated based on the data scale and node performance, so that the execution time of each node is as balanced as possible, the cluster resource utilization rate is improved, and the scheduling time is shortened;

[0032] In the execution phase, the execution order of tasks is dynamically adjusted through a hierarchical high response ratio priority algorithm to avoid serious task starvation problems;

[0033] It is applied to the allocation and execution of ETL management tasks such as desensitization, cleaning, and association, to assist in the efficient management of the entire amount of thoracic surgery data. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a schematic diagram of the Weight-ETL scheduling algorithm framework of the present invention;

[0035] Figure 2 This is a schematic diagram of the effect of the number of tasks on the scheduling duration of the present invention;

[0036] Figure 3 The impact of the number of tasks on resource utilization in the present invention;

[0037] Figure 4The impact of the number of nodes on the scheduling duration of the present invention;

[0038] Figure 5 This is the impact of the number of nodes on resource utilization in the present invention. DETAILED DESCRIPTION

[0039] The embodiments of the present invention are described below in conjunction with the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention.

[0040] Example 1: Figure 1 As shown in the figure, a weight-based ETL task scheduling method (Weight-ETL) is applied to the allocation and execution of ETL governance tasks such as desensitization, cleaning, and association, to achieve efficient governance of the full amount of thoracic surgery data. In the data governance process, the Weight-ETL scheduling algorithm ensures priority scheduling of important aggregation tasks by dividing the levels of ETL tasks; estimates the task execution time by data scale and node performance to improve cluster resource utilization; and dynamically adjusts the execution order of tasks by calculating the response ratio of tasks to be executed.

[0041] like Figure 1 As shown in the figure, the Weight-ETL scheduling method can allocate the ETL task set E related to thoracic surgery designed based on the original medical system data to nodes with different performance through a multi-stage scheduling method. After the governance process of desensitizing privacy data, cleaning complex data, and associating related data, high-quality medical data assets can be obtained, including generalized data such as height and age, as well as specific data related to thoracic surgery such as chest X-ray and chest CT. After further analysis, these data can be applied to smart medical applications such as clinical decision support and clinical pathway management.

[0042] The method comprises 3 processing stages;

[0043] In the task preprocessing stage, calculate the importance weight W of each ETL task Ek , the ETL task set E is divided into a high-level task set H and a low-level task set L through a task classification algorithm;

[0044] In the task allocation phase, the performance weights W of different nodes are calculated. Ni , obtain the source data volume of the ETL task, and assign the tasks in the high-level task set H and the low-level task set L to the task sets of different nodes through the weighted greedy algorithm to form a set G composed of multiple node task sets;

[0045] During the task execution phase, the task execution order of each node is dynamically adjusted through a hierarchical high response ratio priority algorithm.

[0046] Furthermore, in the field of medical data governance, the priority of data governance is usually determined by hospital needs. For example, for hospitals, the effective use of medical resources is directly related to the hospital's operational efficiency and service quality. Therefore, in order to meet the needs of patients, it is necessary to prioritize the aggregation of data such as patient basic information and doctor scheduling information; and in order to meet the needs of doctors, it is necessary to prioritize the aggregation of data such as patient history and medical equipment status. Based on this concept, this article proposes an importance model for medical ETL tasks to prioritize the scheduling of important tasks.

[0047] In the present invention, the importance weight W of the ETL task is Ej Related to hospital needs, system core data, medical systems, and other ETL tasks. For example, tasks related to hospital needs generally involve summarizing system core data and are likely to be related to multiple medical systems or ETL tasks, so they have a larger importance weight and a higher execution priority; some ETL tasks are affected by database design and are related to multiple ETL tasks, but are likely to be unrelated to system core data or hospital needs, so they have a smaller importance weight and a lower execution priority. In the preprocessing stage, the importance weight W Ej The specific calculation formula is as follows:

[0048] W Ej =W Ej1 *25%+W Ej2 *25%+W Ej3 *5%+W Ej4 *5% (1)

[0049] Where j represents the task number; W Ej Indicates whether the ETL task is required by the hospital, and the value is {0,1}; W Ej2 Indicates whether the ETL task is related to the core data of the medical system, and the value is {0,1}; W Ej3 represents the number of associations between ETL tasks and other medical systems, with values ​​of {0, 1, 2, 3, 4, 5}; W Ej Indicates the number of associations between an ETL task and other ETL tasks. The value is {0,1,2,3,4,5}.

[0050] Furthermore, the preprocessing stage specifically includes the following:

[0051] First, calculate the importance W of each task in the ETL task set E Ej ;

[0052] Then, combined with the "80 / 20 rule", the tasks in the ETL task set E are reordered from the largest to the smallest according to the importance weight. Finally, the tasks ranked in the top 20% of the ETL task set E are taken as high-level tasks that need to be executed first and stored in the high-level task set H; the remaining tasks are regarded as low-level tasks and stored in the low-level task set L, so that more important high-level tasks can be processed first in the task allocation and execution stage.

[0053] Furthermore, the task allocation stage specifically includes the following: Ni It is obtained by comparing the execution time of the same set of tasks on different nodes in advance. Therefore, a set of ETL task sets Ew is randomly selected from the ETL task set E, and the total execution time of the task set Ew on node i is recorded as Tw i , performance weights of different nodes W Ni The calculation method is as follows:

[0054]

[0055]

[0056] Among them, Tw m Indicates the maximum total execution time, n is the total number of nodes;

[0057] After calculating the performance weight of each node, the nodes are first reordered from large to small according to the performance weight to form a node information set N = {N1, N2, ..., N n}; Then for each node information N i Create the corresponding task set G i , forming a set G = {G1, G2, ..., G n}; Calculate the total number of tasks S for H and L at the same time H With S L And reorder the tasks in sets H and L according to the source data volume from large to small; finally, the weighted greedy algorithm is used to assign the tasks in sets H and L to the node task set G i middle.

[0058] Furthermore, in the weighted greedy algorithm of the task allocation stage, n tasks with higher importance are extracted from the sets H and L, and tasks with large data volumes are allocated to nodes with better performance; then, according to the estimated execution time S of each node, i , and sequentially allocate the remaining tasks in sets H and L, so as to give priority to high-level tasks and make full use of cluster resources to improve scheduling efficiency; S i The calculation formula is:

[0059]

[0060] Where Di represents the total amount of source data of the tasks assigned to node i, and d represents the current task to be assigned E f The amount of source data.

[0061] The weighted greedy algorithm in this embodiment is specifically shown in Table 1;

[0062] Table 1 Weighted Greedy Algorithm

[0063]

[0064] In actual business scenarios, the configuration and load of each server in a hospital are not necessarily the same, resulting in different processing capabilities of different nodes, which makes the execution time of each node fluctuate greatly when executing the same ETL task. In addition, when different ETL tasks are executed on the same node, the execution time of the ETL tasks is inconsistent due to factors such as the amount of data. Therefore, in the task allocation stage, the performance weight W of the node is calculated by the present invention. Ni ,Through the weighted greedy algorithm, the data scale and node performance are comprehensively considered, and the reasonable allocation of sets H and L is completed.

[0065] Furthermore, the task execution phase includes:

[0066] In order to avoid the starvation problem during the task execution phase, the present invention divides the task set G on each node according to the task level divided in the task preprocessing phase. i Divided into high-level task set G Hi With low-level task set G Li ; Then execute the set G in turn through the high response ratio priority algorithm hi With G Li , comprehensively consider the task importance, task waiting time, and task execution time, so as to give priority to important tasks and ensure the fairness of task execution; the calculation formula of response ratio is:

[0067]

[0068] T hi =Ts hi -Ta hi (6)

[0069] Te hi ≈k*D h (7)

[0070] Among them, R hi represents the response ratio of task h on node i, Tw hi represents the waiting time of task h on node i, Te ho represents the execution time of task h on node i, Ta hirepresents the arrival time of task h at node i, Ts hi represents the execution start time of task h on node i, k is a constant, D h Represents the amount of source data for task h.

[0071] The following is an embodiment of the device of the present invention. The embodiment of the device of the present invention is used to execute the method implemented by the first embodiment of the method of the present invention. For ease of explanation, only the relevant parts of the embodiment of the present invention are shown. For the specific parts not disclosed, please refer to the first embodiment of the present invention.

[0072] The embodiment of the present invention provides a weight-based ETL task scheduling device, comprising:

[0073] Task preprocessing module: used to calculate the importance weight W of each ETL task Ej , the ETL task set E is divided into a high-level task set H and a low-level task set L through a task classification algorithm;

[0074] Task allocation module: used to calculate the performance weight W of different nodes Ni , obtain the source data volume of the ETL task, and assign the tasks in the high-level task set H and the low-level task set L to the task sets of different nodes through the weighted greedy algorithm to form a set G composed of multiple node task sets;

[0075] Task execution module: used to dynamically adjust the task execution order of each node through a hierarchical high response ratio priority algorithm.

[0076] In a feasible implementation, the task preprocessing module is specifically used to calculate the importance W corresponding to each task in the ETL task set E. Ej ;

[0077] Then, combined with the "80 / 20 rule", the tasks in the ETL task set E are reordered from the largest to the smallest according to the importance weight. Finally, the tasks ranked in the top 20% of the ETL task set E are taken as high-level tasks that need to be executed first and stored in the high-level task set H; the remaining tasks are regarded as low-level tasks and stored in the low-level task set L, so that more important high-level tasks can be processed first in the task allocation and execution stage.

[0078] In a feasible implementation, the task allocation module is specifically used to randomly select a set of ETL task sets Ew from the ETL task set E, and record the total execution time of the task set Ew at the node i as Tw i , performance weights of different nodes W Ni The calculation method is as follows:

[0079]

[0080]

[0081] Among them, Tw m Indicates the maximum total execution time, n is the total number of nodes;

[0082] After calculating the performance weight of each node, the nodes are first reordered from large to small according to the performance weight to form a node information set N = {N1, N2, ..., N n}; Then for each node information N i Create the corresponding task set G i , forming a set G = {G1, G2, ..., G n}; Calculate the total number of tasks S for H and L at the same time H With S L And reorder the tasks in sets H and L according to the source data volume from large to small; finally, the weighted greedy algorithm is used to assign the tasks in sets H and L to the node task set G i middle.

[0083] In a feasible implementation, in the weighted greedy algorithm in the task allocation module, n tasks with higher importance are extracted from the sets H and L, and tasks with large data volumes are allocated to nodes with better performance; then, according to the estimated execution time S of each node, i , and sequentially allocate the remaining tasks in sets H and L, so as to give priority to high-level tasks and make full use of cluster resources to improve scheduling efficiency; S i The calculation formula is:

[0084]

[0085] Where D i represents the total amount of source data of the tasks assigned to node i, and d represents the current task to be assigned e f The amount of source data.

[0086] In a feasible implementation manner, the task execution module is specifically used to assign the task set G on each node to the task level divided in the task preprocessing stage. i Divided into high-level task set G Hi With low-level task set G Lo ; Then execute the set G in turn through the high response ratio priority algorithm Ho With G Lo , comprehensively consider the task importance, task waiting time, and task execution time, so as to give priority to important tasks and ensure the fairness of task execution; the calculation formula of response ratio is:

[0087]

[0088] T hi =Ts hi -Ta hi (6)

[0089] Te hi ≈k*D h (7)

[0090] Among them, R hi represents the response ratio of task h on node i, Tw hi represents the waiting time of task h on node i, Te hi represents the execution time of task h on node i, Ta hi represents the arrival time of task h at node i, Ts hi represents the execution start time of task h on node i, k is a constant, D h Represents the amount of source data for task h.

[0091] In the present invention, cluster scheduling is to distribute the ETL governance tasks related to thoracic surgery, such as desensitization, cleaning, and association designed by the platform middleware based on Kettle, to nodes with different performances, so as to realize the convergence of data from multiple medical systems. For this reason, the present invention randomly selects some designed ETL tasks and corresponding parts of medical data for experimental verification. At the same time, computers with different configurations are selected to simulate nodes with different performances, and the node performance weights are calculated in combination with the algorithm in the task allocation stage. The node configuration information and weight values ​​are shown in Table 2.

[0092] Table 2 Node configuration information and weight table

[0093]

[0094] Finally, the performance of the algorithm of the present invention is verified by comparing it with two benchmark algorithms. The specific algorithm is as follows:

[0095] 1) NRF method: No classification is performed in the preprocessing stage, a round-robin algorithm is used in the allocation stage, and a first-come, first-served algorithm is used in the execution stage.

[0096] 2) NGS method: No classification is performed in the preprocessing stage, a greedy algorithm is used in the allocation stage, and a short job first algorithm is used in the execution stage.

[0097] The specific experimental results are as follows:

[0098] (1) Impact of classification

[0099] The present invention prepares 20 ETL governance tasks for experiments to verify the impact of level division, including 4 high-level tasks A = {A1, A2, ..., A4} and 16 low-level tasks B = {B1, B2, ..., B 16}, by showing the execution order of 20 mixed tasks on 3 nodes, it is verified that the algorithm of the present invention can prioritize important tasks.

[0100] As shown in Table 3, in the baseline algorithm, high-level tasks cannot be guaranteed to be executed first. If other tasks take too long, it may affect hospital operation management, clinical research, etc. However, the algorithm of the present invention ensures that more important high-level tasks can be processed by the cluster at the first time by allocating and executing high-level tasks first.

[0101] Table 3 The impact of level classification on the execution order of important tasks

[0102]

[0103] (2) Impact of changing the number of tasks

[0104] The present invention prepares medical ETL tasks with a total number of 40, 60, and 80 tasks respectively to be executed on three nodes, and records the scheduling duration of different algorithms to evaluate the impact of changing the number of tasks on the algorithm performance when other parameters remain unchanged.

[0105] exist Figure 2 In the figure, the vertical axis reflects the scheduling time of different algorithms. As the number of tasks increases, the cluster needs to spend more scheduling time to complete thoracic surgery data management. However, compared with the baseline algorithm, the scheduling time of the invented algorithm is shorter under different numbers of tasks, and the performance is improved by 9.6%.

[0106] And in Figure 3 In the figure, the ordinate reflects the average execution time and variance of each node. Compared with the benchmark algorithm, the variance of the algorithm of the present invention is smaller overall, indicating that the algorithm of the present invention can make more full use of cluster resources under different total number of tasks and avoid long idle time of some nodes.

[0107] (3) Impact of Changing the Number of Nodes

[0108] The present invention prepares 60 medical ETL tasks in total, which are run on 2, 3, and 4 nodes respectively, and records the scheduling duration of different algorithms to evaluate the impact of changing the number of nodes on algorithm performance when other parameters remain unchanged.

[0109] exist Figure 4 In the data set, as the number of nodes increases, the cluster can complete the thoracic surgery data management faster, and the difference in scheduling time between different algorithms is getting smaller and smaller. However, compared with the benchmark algorithm, the algorithm of the present invention has a shorter scheduling time overall and an 8.8% performance improvement.

[0110] And in Figure 5 In the figure, compared with the benchmark algorithm, the variance of the algorithm of the present invention is smaller overall, indicating that the algorithm of the present invention can fully utilize cluster resources under different numbers of nodes and avoid excessive or low load on some nodes.

[0111] In general, compared with the benchmark algorithm, the Weight-ETL scheduling algorithm proposed in this invention can more efficiently utilize cluster resources, shorten scheduling time, and ensure that important tasks are executed first. Therefore, in the process of aggregating medical data such as thoracic surgery, the Weight-ETL scheduling algorithm can efficiently manage multi-source heterogeneous data, build high-quality medical data assets, and provide data support for the construction of disease-specific data sets and medical big data analysis.

[0112] The specific implementation modes of the present invention have been described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A weight-based ETL task scheduling method, characterized by: It includes 3 processing stages; In the task preprocessing stage, the importance weight W of each ETL task is calculated Ej , the ETL task set E is divided into a high-level task set H and a low-level task set L through a task classification algorithm; In the task allocation phase, the performance weights W of different nodes are calculated. Ni , obtain the source data volume of the ETL task, and assign the tasks in the high-level task set H and the low-level task set L to the task sets of different nodes through the weighted greedy algorithm to form a set G composed of multiple node task sets; During the task execution phase, the task execution order of each node is dynamically adjusted through a hierarchical high response ratio priority algorithm.

2. The weight-based ETL task scheduling method according to claim 1, characterized in that: In the preprocessing stage, the importance weight W Ej The specific calculation formula is as follows: IN Ej =In Ej1 *25%+W Ej2 *25%+W Ej3 *5%+W Ej4 *5%(1) Where j represents the task number; W Ej1 Indicates whether the ETL task is required by the hospital, and the value is {0,1}; W Ej2 Indicates whether the ETL task is related to the core data of the medical system, and the value is {0,1}; W Ej3 represents the number of associations between ETL tasks and other medical systems, with values ​​of {0,1,2,3,4,5}; W Ej4 Indicates the number of associations between an ETL task and other ETL tasks. The value is {0,1,2,3,4,5}.

3. The weight-based ETL task scheduling method according to claim 1, characterized in that: The pre-processing stage specifically includes the following: First, calculate the importance W of each task in the ETL task set E Ej ; Then, combined with the "80 / 20 rule", the tasks in the ETL task set E are reordered from the largest to the smallest according to the importance weight. Finally, the tasks ranked in the top 20% of the ETL task set E are taken as high-level tasks that need to be executed first and stored in the high-level task set H; the remaining tasks are regarded as low-level tasks and stored in the low-level task set L, so that more important high-level tasks can be processed first in the task allocation and execution stage.

4. The weight-based ETL task scheduling method and device according to claim 1, characterized in that: The task allocation stage specifically includes the following steps: randomly extract a set of ETL task sets Ew from the ETL task set E, and record the total execution time of the task set Ew at node i as Tw i , performance weights of different nodes W Ni The calculation method is as follows: Among them, Tw m Indicates the maximum total execution time, n is the total number of nodes; After calculating the performance weight of each node, the nodes are first reordered from large to small according to the performance weight to form a node information set N = {N1, N2, ..., N n }; Then for each node information N i Create the corresponding task set G i , forming a set G = {G1, G2, ..., G n }; Calculate the total number of tasks S for H and L at the same time H With S L And reorder the tasks in sets H and L according to the source data volume from large to small; finally, the weighted greedy algorithm is used to assign the tasks in sets H and L to the node task set G i middle.

5. The weight-based ETL task scheduling method and device according to claim 1, characterized in that: In the weighted greedy algorithm of the task allocation stage, n tasks with higher importance are extracted from the sets H and L, and tasks with large data volumes are allocated to nodes with better performance; then, the estimated execution time S of each node is used to calculate the task. i , and sequentially allocate the remaining tasks in sets H and L, so as to give priority to high-level tasks and make full use of cluster resources to improve scheduling efficiency; S i The calculation formula is: Where D i represents the total amount of source data of the tasks assigned to node i, and d represents the current task to be assigned E f The amount of source data.

6. The weight-based ETL task scheduling method and device according to claim 1, characterized in that: The task execution phase includes: According to the task levels divided in the task preprocessing stage, the task set G on each node i Divided into high-level task set G Hi With low-level task set G Li ; Then execute the set G in turn through the high response ratio priority algorithm Hi With G Li , comprehensively consider the task importance, task waiting time, and task execution time, so as to give priority to important tasks and ensure the fairness of task execution; the calculation formula of response ratio is: Tw hi =Ts hi -The hi (6) yourself hi ≈k*D h (7) Among them, R hi represents the response ratio of task h on node i, Tw hi represents the waiting time of task h on node i, Te hi represents the execution time of task h on node i, Ta hi represents the arrival time of task h at node i, Ts hi represents the execution start time of task h on node i, k is a constant, D h Represents the amount of source data for task h.

7. A weight-based ETL task scheduling device, characterized by: The method comprises a module for executing the method according to any one of claims 1 to 6.

Images