A method for task scheduling and data transmission based on hypergraph

Abstract

The application relates to the technical field of task scheduling, in particular to a task scheduling and data transmission method based on a hypergraph, which comprises the following steps: modeling a plurality of tasks, a plurality of data files and a plurality of data centers into hypergraphs respectively to obtain a plurality of original hypergraphs; partitioning the plurality of original hypergraphs to obtain a partitioning result; transferring the plurality of tasks based on the partitioning result to obtain a current executable task; selecting a target data file from the plurality of data files and adding the target data file into a transmission queue of a corresponding data center, and adding the current executable task into a task queue; and iteratively comparing the execution time of the task in the task queue and the transmission time of the data file in the transmission queue to obtain an optimal task and an optimal transmission data file, so that the problem of low data transmission efficiency of the existing task scheduling and data transmission method is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of task scheduling, and particularly relates to a task scheduling and data transmission method based on a hypergraph. BACKGROUND

[0002] In the era of big data, the Internet has to process a large number of tasks every day.

[0003] The traditional way is to place the tasks submitted by users in a data center for processing. However, with the continuous increase of data volume, a single data center cannot meet the demand. Therefore, more and more researches have turned to geographically distributed data centers, which alleviate the computing pressure and speed up the completion time of tasks by placing tasks in different data centers.

[0004] However, the processing of tasks often requires corresponding data files, which are stored in different data centers. At the same time, the cross-data-center transmission consumes a large amount of bandwidth, reducing the efficiency of data transmission. SUMMARY

[0005] The present application aims to provide a task scheduling and data transmission method based on a hypergraph, which aims to solve the problem of low efficiency of data transmission in the existing task scheduling and data transmission method.

[0006] To achieve the above-mentioned purpose, the present application provides a task scheduling and data transmission method based on a hypergraph, comprising the following steps:

[0007] Modeling a plurality of tasks, a plurality of data files and a plurality of data centers as hypergraphs respectively to obtain a plurality of original hypergraphs;

[0008] Partitioning a plurality of the original hypergraphs to obtain a partitioning result;

[0009] Transferring a plurality of the tasks based on the partitioning result to obtain a current executable task;

[0010] Selecting a target data file from a plurality of data files and adding it to the transmission queue of the corresponding data center, and adding the current executable task to the task queue;

[0011] Iteratively comparing the execution time of the tasks in the task queue and the transmission time of the data files in the transmission queue to obtain an optimal task and an optimal transmission data file.

[0012] Among them, the specific way of partitioning a plurality of the original hypergraphs to obtain a partitioning result is:

[0013] Compressing each of the original hypergraphs into a plurality of compressed hypergraphs by using a matching clustering method;

[0014] partitioning the hypergraph using recursive bisection for each of the compressed hypergraphs to obtain partitioned hypergraphs;

[0015] mapping each of the partitioned hypergraphs back to the original hypergraph to obtain a partitioned result.

[0016] The specific manner of mapping each of the partitioned hypergraphs back to the original hypergraph to obtain a partitioned result is:

[0017] After adjusting the nodes between multiple partitioned hypergraphs using the BFM algorithm, each of the partitioned hypergraphs is mapped back to the original hypergraph through multi-level fine-grained partitioning to obtain a partitioned result.

[0018] The specific manner of transferring multiple tasks based on the partitioned result to obtain a current executable task is:

[0019] Based on the partitioned result, the total completion time of tasks on multiple data centers is compared to find a data center with the longest task completion time for traversal to obtain a target task.

[0020] The target task is assigned to a data center with the shortest current task completion time to obtain a current executable task.

[0021] The specific manner of selecting a target data file from multiple data files and adding it to the transmission queue of the corresponding data center, and adding the current executable task to the task queue is:

[0022] Multiple data files are added to a file set, and a target file is found from the file set and added to a frequency queue.

[0023] The sizes of the file data amounts in the frequency queue are compared to find a data file with the smallest data amount to obtain a target data file.

[0024] Multiple data centers are traversed to find a data center corresponding to the target data file.

[0025] The target data file is added to the transmission queue of the corresponding data center.

[0026] The current executable task is added to the task queue.

[0027] The application discloses a task scheduling and data transmission method based on a hypergraph, and the method comprises the following steps: modeling multiple tasks, multiple data files and multiple data centers into hypergraphs respectively to obtain multiple original hypergraphs; partitioning the multiple original hypergraphs to obtain a partition result; transferring the multiple tasks based on the partition result to obtain a current executable task; selecting a target data file from the multiple data files and adding the target data file into a transmission queue of a corresponding data center; adding the current executable task into a task queue; and iteratively comparing execution time of the tasks in the task queue and transmission time of the data files in the transmission queue to obtain an optimal task and an optimal transmission data file. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0029] Figure 1 is a system framework diagram.

[0030] Figure 2 is a principle block diagram of a task scheduling and data transmission method based on a hypergraph provided by the present application.

[0031] Figure 3 is a multi-stage roughing method flow chart.

[0032] Figure 4 is a recursive bisection method flow chart.

[0033] Figure 5 is a module diagram of a task scheduling and data transmission method based on a hypergraph provided by the present application.

[0034] Figure 6 is a schematic diagram of task completion time.

[0035] Figure 7 is a schematic diagram of data transmission amount.

[0036] Figure 8 is a flow chart of a task scheduling and data transmission method based on a hypergraph provided by the present application. Detailed Implementation

[0037] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0038] Please see Figures 1 to 8 This invention provides a task scheduling and data transmission method based on a hypergraph, comprising the following steps:

[0039] S1 models multiple tasks, multiple data files, and multiple data centers as hypergraphs, resulting in multiple original hypergraphs;

[0040] Specifically, the system framework models tasks, data files, and data centers as a hypergraph, with tasks and data centers as vertices and data files as hyperedges. The size of the task and the processing capacity of the data center are assigned weights to the vertices, while the size of the data file is assigned weights to the hyperedges. We set an initial weight. Cl∈C is the weight of the data center. Used to calculate the average weight, the sum of the weights of each data center.

[0041] Divided by the total number of data centers |M|, α is a weighting factor.

[0042]

[0043] Then, the weights on each data center are recalculated as follows:

[0044]

[0045] S2 partitions the multiple original hypergraphs to obtain partitioning results;

[0046] The specific method is as follows:

[0047] S21 uses a matching clustering method to compress each of the original hypergraphs into multiple compressed hypergraphs;

[0048] Specifically, in the multi-level coarsening stage, matching clustering is used to compress the original large hypergraph (original hypergraph) into multiple sufficiently small hypergraphs (compressed hypergraphs).

[0049] Assume that each vertex u forms a single cluster at the beginning of each coarsening level. Then, visit the vertices in a random order. If the vertex... If not, one of its unmatched neighbors is selected according to the weight of the node (i.e. the length of the task it represents or the processing power of the data center). If there is such a vertex v, we merge the pair u and v into a cluster. If u has no unmatched neighbors, the vertex u remains unmatched, i.e. u remains a single cluster.

[0050] S22 partitions each of the compressed hypergraphs using recursive bipartitioning to obtain partitioned hypergraphs;

[0051] Specifically, we first generate a cluster around a randomly selected vertex. In the coarse process of the algorithm, selected and unselected nodes will be divided into two parts. Unselected vertices v us connected to the growing cluster will be inserted into a priority queue according to their FM gain. If a vertex moves to a growing cluster, the gain of the unselected vertex decreases in correspondence to the cutsize of the current bipartition. Then, the vertex with the highest gain is selected from the priority queue. When a vertex moves to a growing cluster, the gains of the currently unselected neighbors in the priority queue are updated, and those vertices not in the priority queue are inserted, and the cluster growing operation continues until a predetermined balancing criterion is reached. At this point, the result of the first hypergraph partitioning is obtained, and a breadth-first search is used on the remaining graph, and the above process is repeated to obtain a sequence of hypergraph partitions.

[0052] S23 maps each of the partitioned hypergraphs back to the original hypergraph to obtain a partitioning result.

[0053] Specifically, each of the partitioned hypergraphs is mapped back to the original hypergraph through multi-level fine-grained partitioning after adjusting the nodes between the partitioned hypergraphs using the BFM algorithm to obtain a partitioning result.

[0054] After multi-level hypergraph partitioning, the obtained subgraphs need to be mapped back to the original hypergraph through multi-level fine-grained partitioning. Before each mapping, the BFM algorithm is used to adjust the nodes between the subgraphs, so as to further optimize the hypergraph partitioning result and satisfy the load balancing constraint. In the fine-grained process, the coarse-grained hypergraph can be mapped back to a layer of fine-grained hypergraphs, which is iteratively executed until the hypergraph is mapped to a layer.

[0055] S3 transfers a plurality of the tasks based on the partitioning result to obtain a current executable task;

[0056] The specific manner is:

[0057] S31 find the data center with the longest task completion time based on the partition result and traverse to obtain a target task (a task with the smallest data dependency);

[0058] Specifically, the total completion time of a task is calculated

[0059] CT l = ET l + WT l + CDT l

[0060] S32 assign the target task to the data center with the shortest current task completion time to obtain a current executable task.

[0061] Specifically, the total completion time of a task on each data center is compared, the data center with the longest task completion time is found, and the target task is re-assigned to the data center with the shortest current task completion time.

[0062] S4 select a target data file from multiple data files and add it to the transmission queue of the corresponding data center, and add the current executable task to the task queue;

[0063] The specific manner is:

[0064] S41 add multiple data files to a file set, find a target file from the file set, and add it to a frequency queue;

[0065] Specifically, the file to be transmitted is added to items, items is a set of files to be transmitted, a target file (a file with the most occurrences) is found from items, and is added to Maxoccuringchars.

[0066] S42 compare the data sizes of the files in the frequency queue, find the data file with the smallest data size, and obtain a target data file (a data file with the strongest dependency and the smallest data size);

[0067] S43 traverse multiple data centers to find a data center corresponding to the target data file;

[0068] S44 add the target data file to the transmission queue of the corresponding data center;

[0069] S45 add the current executable task to the task queue.

[0070] Specifically, the transmission queue and the task queue are task sets based on hypergraph partitioning.

[0071] S5 iteratively compares the execution time of the tasks in the task queue and the transmission time of the data files in the transmission queue to obtain an optimal task and an optimal transmission data file.

[0072] Specifically, at this time, the execution time of the dequeued task in ExecuteTask is calculated, and the current task queue and transmission time are updated. The completion time of C l is calculated, and the next cycle is entered; when the ExecuteTask queue and the Transfer queue are empty at the same time, the cycle ends.

[0073] The difference between the present application and the existing research work is that we fully consider the dependency relationship among the task, the data center, and the data file, and use the hypergraph technology to model it. We propose a task scheduling and data transmission method based on hypergraph, which fully captures the dependency relationship among various factors in the complex scenario of cloud computing. Through the three steps of multi-level coarsening, recursive bisection, and remapping, the modeled hypergraph is partitioned. For the results of hypergraph partitioning, the present application further optimizes the idea of Min-Min algorithm, reassigns the tasks, and optimizes the data transmission stage.

[0074] Advantages:

[0075] The task scheduling and data transmission method based on hypergraph proposed by the present application can effectively reduce the workload of the data center, reduce the consumption of network bandwidth, and significantly speed up the completion time of the task under the cloud computing environment. Compared with the traditional centralized single data center processing task, the method proposed by the present application can fully meet the task demand submitted by the user, reduce the waiting time when processing the task, and reduce the workload of the data center. Compared with the ordinary distributed data center, the data transmission between the data centers is significantly reduced, the consumption of network bandwidth is reduced, the cost of processing the task is reduced, and the completion time of the task is further accelerated.

[0076] The above only discloses a preferred embodiment of the task scheduling and data transmission method based on hypergraph, and of course cannot limit the scope of the present application. Those skilled in the art can understand that the above-mentioned embodiment can be implemented in whole or in part, and equivalent changes made according to the claims of the present application still belong to the scope covered by the present application.

Claims

1. A method for hypergraph-based task scheduling and data transmission, characterized in that, The method comprises the following steps: Modeling a plurality of tasks, a plurality of data files and a plurality of data centers as hypergraphs respectively to obtain a plurality of original hypergraphs; Modeling the three of tasks, data files and data centers in the system framework as hypergraphs, taking the tasks and data centers as vertices in the hypergraphs, taking the data files as hyperedges in the hypergraphs, and assigning the size of the tasks and the processing capacity of the data centers as weights of the vertices, and assigning the data size of the data files as weights of the hyperedges; An initialized weight is set C l ∈C is the weight of the data center, for calculating the average weight, the sum of the weights of each data center Dividing by the total number of data centers |M|, and α is a weight factor, Then, the weight on each data center is recalculated as: Partitioning a plurality of the original hypergraphs to obtain a partitioning result; The specific way of partitioning a plurality of the original hypergraphs to obtain a partitioning result is: Compressing each of the original hypergraphs into a plurality of compressed hypergraphs by using a matching clustering method; Dividing each of the compressed hypergraphs by using a recursive bisection method to obtain divided hypergraphs; Mapping each of the divided hypergraphs back to the original hypergraph to obtain a partitioning result; The specific way of mapping each of the divided hypergraphs back to the original hypergraph to obtain a partitioning result is: Mapping each of the divided hypergraphs back to the original hypergraph by using a BFM algorithm to adjust the nodes between a plurality of the divided hypergraphs and then through multi-level fine-grained segmentation to obtain a partitioning result; Transferring a plurality of the tasks based on the partitioning result to obtain a currently executable task; Selecting a target data file from a plurality of data files and adding the target data file to a transmission queue of a corresponding data center, and adding the currently executable task to a task queue; Iteratively comparing the execution time of the tasks in the task queue and the transmission time of the data files in the transmission queue to obtain an optimal task and an optimal transmission data file.

2. The hypergraph-based task scheduling and data transmission method according to claim 1, wherein The specific way of transferring a plurality of the tasks based on the partitioning result to obtain a currently executable task is: Comparing the total completion time of the tasks on a plurality of data centers based on the partitioning result, finding a data center with the longest task completion time for traversal to obtain a target task; Assigning the target task to a data center with the shortest current task completion time to obtain a currently executable task.

3. The hypergraph-based task scheduling and data transmission method according to claim 2, wherein The specific way of selecting a target data file from a plurality of data files and adding the target data file to a transmission queue of a corresponding data center, and adding the currently executable task to a task queue is: Adding a plurality of data files to a file set, finding a target file from the file set and adding the target file to a frequency queue; Comparing the data size of the files in the frequency queue to find a data file with the smallest data size to obtain a target data file; Traversing a plurality of the data centers to find a data center corresponding to the target data file; Adding the target data file to a transmission queue of the corresponding data center; Adding the currently executable task to a task queue.

Images