Task scheduling method and device, electronic equipment and readable storage medium

Abstract

The application provides a task scheduling method and device, electronic equipment and readable storage medium. The method comprises: obtaining a first relationship topology graph corresponding to a first data slice combination; determining P first data slice scheduling modes corresponding to P data slice scheduling sequences according to the first relationship topology graph, wherein the P data slice scheduling sequences each comprise m target data slices, and the scheduling sequences of the m target data slices are different in different data slice scheduling sequences; the first data slice scheduling mode is a data slice scheduling mode with the lowest scheduling cost in at least two scheduling modes corresponding to the data slice scheduling sequence, and the scheduling cost is determined based on at least one scheduling cost contribution value; and scheduling the m target data slices according to a first target data slice scheduling mode in the P first data slice scheduling modes. The application considers the global influence of task scheduling, thereby improving the global execution efficiency.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a task scheduling method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] MapReduce task scheduling directly affects the execution time of MapReduce computing jobs. Therefore, how to schedule tasks is crucial for improving the overall computing speed.

[0003] In existing technologies, the task node (Tasktracker or Task Tracker) that requests a task first executes the "data slice of this node". If there is no "data slice of this node", the "data slice of this rack" is executed first. If the "data slice of this rack" still does not exist, the "data slice of a different rack" is executed randomly. That is, existing technologies use a local optimum strategy to schedule tasks, without considering the impact of the current task execution on the global MapReduce system, which can easily lead to low global execution efficiency. Summary of the Invention

[0004] This application provides a task scheduling method, apparatus, electronic device, and readable storage medium to solve the problem of low overall execution efficiency caused by the prior art not considering the impact of task scheduling of each task node on the overall MapReduce system.

[0005] To solve the above problems, this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a task scheduling method applied to a MapReduce system, wherein the MapReduce system includes n task nodes, where n is an integer greater than 1; including:

[0007] Obtain a first relational topology graph corresponding to the first data slice combination, wherein the first relational topology graph is a relational topology graph between the m target data slices included in the first data slice combination and the n task nodes. In the first relational topology graph, the n task nodes are respectively connected to the m target data slices; the connection line between the i-th task node and the j-th target data slice is associated with: at least one scheduling cost contribution value of the i-th task node for the j-th target data slice, where m is an integer greater than 1, i is a positive integer less than or equal to n, and j is a positive integer less than or equal to m.

[0008] Based on the first relational topology graph, P first data slice scheduling methods are determined, each corresponding to one of the P data slice scheduling sequences. Each of the P data slice scheduling sequences includes the m target data slices. The scheduling order of the m target data slices differs in different data slice scheduling sequences, and P is a positive integer. The first data slice scheduling method is the data slice scheduling method with the lowest scheduling cost among at least two scheduling methods corresponding to the data slice scheduling sequence. The scheduling cost is determined based on the at least one scheduling cost contribution value.

[0009] The scheduling of the m target data slices is executed according to the first target data slice scheduling method among the P first data slice scheduling methods.

[0010] Secondly, embodiments of this application also provide a task scheduling device applied to a MapReduce system, the MapReduce system comprising n task nodes, where n is an integer greater than 1; including:

[0011] The first acquisition module is used to acquire a first relational topology graph corresponding to the first data slice combination. The first relational topology graph is a relational topology graph between the m target data slices included in the first data slice combination and the n task nodes. In the first relational topology graph, the n task nodes are respectively connected to the m target data slices. The connection line between the i-th task node and the j-th target data slice is associated with at least one scheduling cost contribution value of the i-th task node for the j-th target data slice, where m is an integer greater than 1, i is a positive integer less than or equal to n, and j is a positive integer less than or equal to m.

[0012] The first determining module is configured to determine, based on the first relational topology graph, P first data slice scheduling methods corresponding one-to-one with P data slice scheduling sequences, wherein each of the P data slice scheduling sequences includes the m target data slices, and the scheduling order of the m target data slices is different in different data slice scheduling sequences, and P is a positive integer; the first data slice scheduling method is the data slice scheduling method with the lowest scheduling cost among at least two scheduling methods corresponding to the data slice scheduling sequence, and the scheduling cost is determined based on the at least one scheduling cost contribution value;

[0013] The execution module is used to execute the scheduling of the m target data slices according to the first target data slice scheduling method among the P first data slice scheduling methods.

[0014] Thirdly, embodiments of this application also provide an electronic device, including: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; the processor is configured to read the program in the memory to implement the steps in the method described in the first aspect above; or, the steps in the method described in the second aspect above.

[0015] Fourthly, embodiments of this application also provide a readable storage medium for storing a program, which, when executed by a processor, implements the steps in the method described in the first aspect above, or implements the steps in the method described in the second aspect above.

[0016] In this embodiment, a relationship topology graph between data slices and task nodes is constructed. This graph includes the scheduling cost contribution value of each task node for each data slice. Based on this topology graph, the scheduling method with the lowest scheduling cost corresponding to each data slice scheduling sequence is determined, resulting in P first data slice scheduling methods. Then, one scheduling method, i.e., the first target optimal scheduling method, is selected from these P first data slice scheduling methods to execute the data slice scheduling. Therefore, this embodiment considers the global impact of task scheduling and selects the scheduling method with the lowest scheduling cost corresponding to a certain data slice scheduling sequence to execute the data slice scheduling, thereby improving overall execution efficiency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts of the task scheduling method provided in the embodiments of this application;

[0019] Figure 2 This is a schematic diagram of the relationship topology between data slices and task nodes provided in the embodiments of this application;

[0020] Figure 3 This is a schematic diagram illustrating the topological logical relationship between data blocks and data slices provided in the embodiments of this application;

[0021] Figure 4 This is a schematic diagram of scheduling operations provided in an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the distribution of task nodes and data slices provided in the embodiments of this application;

[0023] Figure 6 This is a schematic diagram illustrating the prediction of the estimated task execution time provided in an embodiment of this application;

[0024] Figure 7 This is the second flowchart of the task scheduling method provided in the embodiments of this application;

[0025] Figure 8 This is a structural diagram of the task scheduling device provided in this application.

[0026] Figure 9 This is a structural diagram of the electronic device provided in this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] The terms "first," "second," etc., used in the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. Additionally, the use of "and / or" in this application indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: including A alone, B alone, C alone, and the presence of both A and B, both B and C, both A and C, and the presence of A, B, and C.

[0029] For ease of understanding, the following describes some aspects of the embodiments of this application.

[0030] The MapReduce computation process is broadly divided into the Map phase and the Reduce phase. The Map phase should generally adhere to the locality principle (the Tasktracker processes data slices on its own node). When a MapReduce job has a non-local task, the compute node processing that task needs to wait for remote data transmission, resulting in wasted computing resources. Most non-local tasks appear after the job's Shuffle phase begins, and these tasks compete for network bandwidth resources with the Shuffle phase itself. Therefore, non-local tasks not only consume network bandwidth and waste computing resources but also delay the job's execution progress, ultimately affecting its efficiency. The higher the data locality of a job, the less network bandwidth resources it consumes. With limited network bandwidth, less bandwidth consumption means less waiting time for each compute node, ultimately leading to higher job execution efficiency.

[0031] The task scheduling method provided in the embodiments of this application will be described below.

[0032] The task scheduling method in this application embodiment can be applied to, but is not limited to, MapReduce systems. That is, the task scheduling method can be used, but is not limited to, implementing task scheduling within a MapReduce system. The MapReduce system may include n task nodes, where n is an integer greater than 1. The n task nodes can be all or some of the task nodes included in the MapReduce system, and can be determined according to actual needs. This application embodiment does not limit this.

[0033] In this embodiment, data slices can be scheduled using a topology graph showing the relationship between data slices and task nodes. Additionally, data slices can be generated by dividing data blocks using the same topology graph.

[0034] For ease of understanding, the data slice scheduling method that "schedules data slices based on the relationship topology between data slices and task nodes" will be referred to as the "first scheduling method". Other data slice scheduling methods can be referred to as the "second scheduling method", such as: first, executing the "local node data slice", if the "local node data slice" does not exist, executing the "local rack data slice" first, and if the "local rack data slice" still does not exist, then randomly executing the "non-local rack data slice".

[0035] The data block partitioning method that "partitions data blocks based on the relationship topology between data slices and task nodes" is called the "first partitioning method". Other data block partitioning methods can be called the "second partitioning method", such as partitioning data blocks by using a data slice that includes a preset number of data blocks (e.g., one data block), or partitioning data blocks by using the i-th slice that includes u+1 data blocks, where u is the number of data blocks included in the (i-1)-th slice.

[0036] It is understood that in practical applications, the task scheduling device can choose any of the above-mentioned partitioning methods to implement data block partitioning, and choose any of the above-mentioned scheduling methods to implement data slice scheduling, according to actual needs. This application embodiment does not limit this. For example:

[0037] In some embodiments, after determining the data blocks to be processed by the MapReduce system, the data blocks can be divided using at least one of a first partitioning method and a second partitioning method to obtain data slices to be scheduled by the MapReduce system. Then, the first scheduling method can be used to schedule the data slices. For example, in an optional embodiment, after determining all the data blocks to be processed by the MapReduce system, these data blocks can be divided into two sets. For the data blocks included in the first set, the second partitioning method described above can be used to partition these data blocks, and the first scheduling method described above can be used to schedule the data slices; in parallel, for the data blocks included in the second set, the first partitioning method described above can be used to partition these data blocks, and the first scheduling method described above can be used to schedule the data slices.

[0038] In other embodiments, after determining the data blocks to be processed by the MapReduce system, these data blocks can be divided using a first partitioning method to obtain data slices to be scheduled by the MapReduce system. Then, a second scheduling method can be used to schedule the data slices.

[0039] It should be noted that, in the embodiments of this application, a data slice may include one or more (i.e., at least two) data blocks. The number of data blocks included in different data slices may be the same or different, that is, the capacity of different data slices may be the same or different, which can be determined according to the actual situation, and the embodiments of this application do not limit this.

[0040] See Figure 1 , Figure 1 This is one of the flowcharts of the task scheduling method provided in the embodiments of this application. For example... Figure 1 As shown, the task scheduling method in this application embodiment may include the following steps:

[0041] Step 101: Obtain the first relational topology graph corresponding to the first data slice combination, wherein the first relational topology graph is the relational topology graph between the m target data slices included in the first data slice combination and the n task nodes. In the first relational topology graph, the n task nodes are respectively connected to the m target data slices; the connection line between the i-th task node and the j-th target data slice is associated with: at least one scheduling cost contribution value of the i-th task node for the j-th target data slice, where m is an integer greater than 1, i is a positive integer less than or equal to n, and j is a positive integer less than or equal to m.

[0042] In specific implementation, the first relation topology graph can be obtained in any of the following ways: construction acquisition, reception acquisition, configuration acquisition, or search acquisition. The specific method can be determined according to actual needs, and this application embodiment does not limit it.

[0043] The first data slice combination includes m target data slices, which can be understood as data slices to be scheduled, i.e., the first data slice combination is the actual data slice combination to be scheduled. In specific implementation, the m target data slices can be obtained by dividing them using the first slicing method or the second slicing method described above; the capacity of different data slices among the m target data slices can be the same or different, which can be determined according to actual needs, and this application embodiment does not limit this.

[0044] In this embodiment, the scheduling cost contribution value of a task node for a data slice may include, but is not limited to, at least one of the following: the estimated task execution time K for the task node for the data slice; and the input / output (IO) cost J for the task node for the data slice. The value of K for the i-th task node for the j-th data slice can be denoted as Kij. ij The J value of the i-th task node for the j-th data slice can be denoted as J. ij .

[0045] The K value of a task node for a data slice is related to the following information: the size of the data slice; and the execution capability of the task node. In other words, the size of the data slice is positively correlated with the K value; that is, the larger the data slice size, the larger the K value, and vice versa. The execution capability of a task node is negatively correlated with the K value; that is, the stronger the execution capability of the task node, the smaller the K value, and vice versa.

[0046] The J value of a task node relative to a data slice is related to the following information: the compute node to which the task node and the data slice belong, and their respective architectures. The compute node and architecture to which the task node and data slice belong determine the scheduling distance between them. It can be understood that the scheduling distance between a task node and a data slice is positively correlated with the J value; that is, the smaller the scheduling distance, the larger the J value, and vice versa.

[0047] The embodiments of this application do not limit the method of obtaining the at least one scheduling cost contribution value. Any method that can determine the at least one scheduling cost contribution value can fall within the protection scope of this application.

[0048] To facilitate understanding of the topology diagram relating data slices and task nodes in the embodiments of this application, please refer to... Figure 2 . Figure 2 The diagram shows the topology of the relationship between m data slices (m splits) and n task nodes (n Tasktrackers).

[0049] like Figure 2 As shown, the relational topology graph includes the m data slices and the n task nodes, and each of the n task nodes is connected to the m data slices. It can be understood that the relational topology graph includes n×m connecting lines.

[0050] Additionally, in this relational topology, each connection line is associated with a K value and a J value. The K value of the connection line is the K value of the connected task node relative to its connected data slice, and the J value is the J value of the connected task node relative to its connected data slice. Figure 2 The diagram only shows the K and J values ​​associated with the connection point between the first task node and the first data slice, and the K and J values ​​associated with the connection point between the nth task node and the mth target data slice.

[0051] Step 102: Based on the first relational topology graph, determine P first data slice scheduling methods that correspond one-to-one with the P data slice scheduling sequences, wherein each of the P data slice scheduling sequences includes the m target data slices, and the scheduling order of the m target data slices is different in different data slice scheduling sequences, and P is a positive integer; the first data slice scheduling method is the data slice scheduling method with the lowest scheduling cost among at least two scheduling methods corresponding to the data slice scheduling sequence, and the scheduling cost is determined based on the at least one scheduling cost contribution value.

[0052] In the embodiments of this application, it is considered that different data slice scheduling orders for the same data slice combination will result in different scheduling methods for the data slice combination.

[0053] For example, assume that the m target data slices include data slice 1, data slice 2, and data slice 3. The n task nodes include task node 1 and task node 2.

[0054] In the data slice scheduling sequence a, the scheduling order of the m target data slices from first to last is: data slice 1, data slice 2 and data slice 3.

[0055] In the data slice scheduling sequence b, the scheduling order of the m target data slices from first to last is: data slice 3, data slice 2 and data slice 1.

[0056] For data slice scheduling sequence a, the determined scheduling method may be: scheduling data slice 1 to task node 1, scheduling data slice 2 to task node 2, and scheduling data slice 3 to task node 1.

[0057] For data slice scheduling sequence b, the determined scheduling method may be: scheduling data slice 3 to task node 1, scheduling data slice 2 to task node 1, and scheduling data slice 1 to task node 2.

[0058] It is evident that when the same data slice is scheduled using data slice scheduling sequence a and data slice scheduling sequence b, the scheduling methods for the data slice may differ.

[0059] It is understandable that different data slice scheduling methods may have different scheduling costs. The embodiments of this application aim to reduce the overall scheduling cost of the data blocks to be scheduled. Therefore, the embodiments of this application can pre-determine P data slice scheduling sequences corresponding to the m target data slices. Then, a first data slice scheduling method corresponding to each data slice scheduling sequence is determined, and subsequently, a data slice scheduling method is selected from the first data slice scheduling methods corresponding to each data slice scheduling sequence to execute the scheduling of the m target data slices. In this way, the overall scheduling cost of the m target data slices can be reduced, and the global efficiency of the MapReduce system can be improved.

[0060] For each data slice scheduling sequence, all possible data slice scheduling methods can be determined based on the connecting lines of the first relational topology graph. The scheduling cost corresponding to each data slice scheduling method is then determined based on the at least one scheduling cost contribution value associated with each connecting line in the first relational topology graph. Next, the data slice scheduling method with the lowest scheduling cost is selected as the first data slice scheduling method corresponding to that data slice scheduling sequence. In practice, a greedy algorithm can be used to implement the above process, but it is not limited to this. The first data slice scheduling method corresponding to a certain data slice scheduling sequence can be understood as the optimal data slice scheduling method corresponding to that data slice scheduling sequence.

[0061] Step 103: Execute the scheduling of the m target data slices according to the first target data slice scheduling method among the P first data slice scheduling methods.

[0062] In a specific implementation, one approach is to randomly select one of the P first data slice scheduling methods as the first target data slice scheduling method.

[0063] In another embodiment, after determining the scheduling methods of the P first data slices that correspond one-to-one with the scheduling sequences of the P data slices, and before executing the scheduling of the m target data slices, the method may further include:

[0064] Obtain the P scheduling costs that correspond one-to-one with the P first data slice scheduling methods;

[0065] The first data slice scheduling method corresponding to the lowest scheduling cost among the P scheduling costs is determined as the first target optimal scheduling method.

[0066] That is, from the P first data slice scheduling methods, the first data slice scheduling method with the lowest scheduling cost can be selected as the first target data slice scheduling method. In this way, the overall scheduling cost of the m target data slices can be minimized, further improving the overall execution efficiency.

[0067] Steps 101 to 103 can be regarded as specific implementations of the first scheduling method in the embodiments of this application.

[0068] The task scheduling method in this embodiment constructs a topology graph showing the relationship between data slices and task nodes. This topology graph includes the scheduling cost contribution value of each task node for each data slice. Based on this topology graph, the scheduling method with the lowest scheduling cost corresponding to each data slice scheduling sequence is determined, resulting in P first data slice scheduling methods. Then, a scheduling method, i.e., the first target optimal scheduling method, is selected from these P first data slice scheduling methods to execute the data slice scheduling. Therefore, this embodiment considers the global impact of task scheduling and selects the scheduling method with the lowest scheduling cost corresponding to a certain data slice scheduling sequence to execute the data slice scheduling, thereby improving overall execution efficiency.

[0069] The following provides a detailed explanation of the establishment of the relationship topology diagram between data slices and task nodes in the embodiments of this application.

[0070] In some embodiments, the at least one scheduling cost contribution value includes: estimated task execution time and input / output I / O cost;

[0071] The step of obtaining the first relational topology graph corresponding to the first data slice combination may include:

[0072] Construct a second topological logical relationship between m target data slices and n task nodes, in which the n task nodes are respectively connected to the m target data slices;

[0073] Based on at least one model, predict the estimated execution time of each of the n task nodes for the m target data slices, and obtain the estimated execution times of n×m tasks.

[0074] Based on the storage relationship between the m target data slices and the computing nodes corresponding to the n task nodes, the IO cost of each task node for the m target data slices is determined, resulting in n×m IO costs.

[0075] Based on the second topological logical relationship, the estimated execution time of the n×m tasks, and the cost of the n×m IO operations, a first relational topology graph corresponding to the first data slice combination is constructed.

[0076] In this embodiment, a topological logical relationship can be constructed between m target data slices and the n task nodes, namely the second topological logical relationship.

[0077] In addition, the K and J values ​​of each task node for each target data slice can be obtained separately.

[0078] In practice, the K value can be predicted using at least one model. The model's input is the target data slice, and its output is the K value corresponding to the target data slice. During implementation, the model can be dynamically adjusted based on prediction bias, and this bias can be compensated for to improve the accuracy of the model's predictions.

[0079] For the J value, the scheduling distance between the task node and the data slice can be determined based on the storage relationship between the m target data slices and the computing nodes corresponding to the n task nodes. Then, based on the scheduling distance between the task node and the data slice, the IO cost of the task node to the data slice is determined.

[0080] Then, by associating the connection lines, J values, and K values ​​corresponding to the same task node and the same data slice, a first relational topology graph corresponding to the first data slice combination can be generated.

[0081] Through this embodiment, the K value can be predicted using at least one model, and the J value can be determined by the storage relationship between task nodes and data slices. This can improve the accuracy of determining the scheduling cost of data slices for task nodes, thereby improving the reliability of task scheduling.

[0082] The determination of the first data slice scheduling method is explained below.

[0083] In some embodiments, determining the P first data slice scheduling methods corresponding one-to-one with the P data slice scheduling sequences based on the first relationship topology graph may include:

[0084] According to the scheduling order of the m target data slices in the first data slice scheduling sequence, and based on the first relation topology graph, the second data slice scheduling methods corresponding to the m target data slices are determined in turn to obtain m second data slice scheduling methods. The second data slice scheduling method can be the data slice scheduling method with the lowest scheduling cost among the at least two data slice scheduling methods corresponding to the target data slice.

[0085] The set of m second data slice scheduling methods is determined as the first data slice scheduling method corresponding to the first data slice scheduling sequence;

[0086] Wherein, the first data slice scheduling sequence is any one of the P data slice scheduling sequences.

[0087] Since each of the m target data slices is connected to the n task nodes, it means that each of the m target data slices may be scheduled to any one of the n task nodes, that is, each target data slice has n possible scheduling methods.

[0088] However, the scheduling of the current task node may affect the scheduling cost of the next task node. For example, assuming the first data slice is scheduled to the first task node, when scheduling the second data slice, there are two possible implementations: First, if it is still scheduled to the first task node, the second data slice needs to wait for the first data slice to complete its scheduling before being scheduled to that task node. Second, if it is scheduled to the second task node, there is no need to wait for the first data slice to complete its scheduling, thus saving scheduling waiting time.

[0089] It is evident that different scheduling methods for the same data slice may correspond to different scheduling costs. Therefore, in this embodiment, according to the sequential scheduling order of the m target data slices in the first data slice scheduling sequence, all possible data slice scheduling methods corresponding to each of the m target data slices can be determined in turn. Then, for each target data slice, the data slice scheduling method with the lowest scheduling cost is selected as its corresponding second data slice scheduling method. In specific implementation, a greedy algorithm can be used to implement the above process, but it is not limited to this. The second data slice scheduling method corresponding to a certain data slice can be understood as: the optimal data slice scheduling method corresponding to that data slice.

[0090] In this embodiment, the data slice scheduling method corresponding to the data slice scheduling sequence can be used to determine: the task nodes of each data slice in all data slices included in the data slice scheduling sequence. The data slice scheduling method corresponding to a data slice can be used to determine: the task node of that data slice. Therefore, after obtaining the m second data slice scheduling methods, the set of the m second data slice scheduling methods can be determined as the first data slice scheduling method corresponding to the first data slice scheduling sequence.

[0091] The following section provides a detailed explanation of how to determine the scheduling method for the second data slice corresponding to the data slice.

[0092] In some embodiments, the step of determining the second data slice scheduling methods corresponding to the m target data slices in the first data slice scheduling sequence according to the first relational topology graph, based on the sequential scheduling order of the m target data slices in the first data slice scheduling sequence, to obtain m second data slice scheduling methods, may include:

[0093] From the first relational topology graph, obtain the at least one scheduling cost contribution value of each of the n task nodes for the first data slice, where the first data slice is the first scheduled data slice in the first data slice scheduling sequence;

[0094] Based on the at least one scheduling cost contribution value of each task node for the first data slice, determine the first scheduling cost of each task node for the first data slice, and obtain n first scheduling costs;

[0095] The method of scheduling the first data slice to the first task node is determined as the second data slice scheduling method corresponding to the first data slice, and the first task node is the task node corresponding to the lowest scheduling cost among the n first scheduling costs;

[0096] Update the task execution waiting time of the first task node based on the estimated task execution time of the first task node for the first data slice.

[0097] In specific implementation, when determining the scheduling method of the second data slice corresponding to the first scheduled data slice in the data slice scheduling sequence, the total scheduling cost contribution value of each of the n task nodes for that data slice can be obtained from the first relationship topology graph. Then, using the obtained values, the scheduling cost of each task node for that data slice is calculated.

[0098] In one optional implementation, the scheduling cost Z of the i-th task node for the first data slice (i.e., the 1st data slice) is... i1 It can be calculated using the following formula:

[0099] Z i1 =a×K i1 +b×J i1

[0100] Where a represents the weight of the K value and b represents the weight of the J value, which can be set according to actual needs.

[0101] In another alternative implementation, K can be directly used. i1 With J i1 The sum of the values ​​is determined as the scheduling cost Z of the i-th task node for the first data slice.

[0102] After obtaining the n first scheduling costs, the task node corresponding to the first scheduling cost with the lowest scheduling cost among the n first scheduling costs, i.e., the first task node, can be determined as the task node for scheduling the first data slice. That is, the scheduling method for the second data slice corresponding to the first data slice is: to schedule the first data slice to the first task node.

[0103] Furthermore, since the first data slice is scheduled to the first task node, the first task node is no longer an idle task node. If other data slices are scheduled to this task node, the scheduling of the first data slice must be completed. Therefore, to improve the accuracy of determining the scheduling cost of the first task node scheduling other data slices, the task execution waiting time of the first task node can be updated. Then, the scheduling cost of the first task node scheduling other data slices can be calculated using the task execution waiting time of the first task node.

[0104] This embodiment minimizes the scheduling cost of the first data slice, thereby reducing the overall scheduling cost of the data slice to be scheduled.

[0105] In some embodiments, the step of determining the second data slice scheduling methods corresponding to the m target data slices in the first data slice scheduling sequence according to the first relational topology graph, based on the sequential scheduling order of the m target data slices in the first data slice scheduling sequence, to obtain m second data slice scheduling methods, may further include:

[0106] From the first relational topology graph, obtain at least one scheduling cost contribution value of each of the n task nodes for the (k+1)th data slice, where the 0th data slice is the 0th scheduled data slice in the first data slice scheduling sequence, and 0 is a positive integer greater than 1 and less than n.

[0107] Obtain the task execution waiting time of each of the n task nodes;

[0108] Based on the at least one scheduling cost contribution value of each task node for the o-th data slice, and the task execution waiting time of each task node, the second scheduling cost of each task node for the o-th data slice is determined, resulting in n second scheduling costs.

[0109] The method of scheduling the o-th data slice to the second task node is determined as the second data slice scheduling method corresponding to the o-th data slice, and the second task node is the task node corresponding to the lowest scheduling cost among the n second scheduling costs;

[0110] Update the task execution wait time of the second task node based on the estimated task execution time of the second task node for the o-th data slice.

[0111] In specific implementation, when determining the second data slice scheduling method corresponding to the o-th scheduled data slice in the data slice scheduling sequence, the total scheduling cost contribution value of each of the n task nodes for that data slice can be obtained from the first relationship topology graph. Additionally, the task execution waiting time of each of the n task nodes is obtained. Then, using the obtained values, the scheduling cost of each task node for that data slice is calculated.

[0112] In one optional implementation, the scheduling cost Z of the i-th task node for the o-th scheduled data slice (i.e., the o-th data slice) is... io It can be calculated using the following formula:

[0113] Z io =a×K io +b×J io +c×T i

[0114] Among them, T iThis represents the task execution waiting time for the i-th task node; a represents the weight of the K value, b represents the weight of the J value, and c represents the weight of the T value, which can be set according to actual needs.

[0115] In another alternative implementation, K can be directly used. io J io With T i The sum of these values ​​is determined as the scheduling cost Z of the i-th task node for the first data slice. io .

[0116] After obtaining the n second scheduling costs, the task node corresponding to the second scheduling cost with the lowest scheduling cost among the n second scheduling costs, i.e., the second task node, can be determined as the task node for scheduling the o-th data slice. That is, the scheduling method for the second data slice corresponding to the o-th data slice is: to schedule the o-th data slice to the second task node.

[0117] Additionally, the task execution waiting time of the second task node can be updated based on the estimated task execution time for the o-th data slice. Specifically, the estimated task execution time for the o-th data slice can be added to the historical task execution waiting time of the second task node to obtain the updated task execution waiting time. This improves the accuracy of determining the scheduling cost of the second task node for scheduling other data slices.

[0118] This embodiment minimizes the scheduling cost of the o-th data slice, thereby reducing the overall scheduling cost of the data slice to be scheduled.

[0119] The following combination Figure 3 The specific implementation of the first segmentation method in the embodiments of this application will be described in detail.

[0120] In some embodiments, the method may further include:

[0121] Obtain q data blocks to be processed by the MapReduce system, where q is an integer greater than m;

[0122] Construct the topological logical relationship between the q data blocks and the m sample data slices;

[0123] Based on the topological logical relationship, the q data blocks are sliced ​​and grouped to obtain R data slice combinations. Each of the R data slice combinations includes m test data slices. In different data slice combinations, at least one test data slice includes different data blocks. R is a positive integer.

[0124] Based on the relationship topology diagrams corresponding to the R data slice combinations, determine the scheduling methods for the R first target data slices that correspond one-to-one with the R data slice combinations.

[0125] The scheduling method with the lowest scheduling cost among the R first target data slice scheduling methods is determined as the second target data slice scheduling method;

[0126] The m test data slices included in the target data slice combination are determined as the m target data slices, and the target data slice combination is the data slice combination corresponding to the second target data slice scheduling method.

[0127] The sample data slice can be understood as a data slice to be determined. That is, when constructing the topological logical relationship, it can be determined that the q data blocks should be divided into m data slices, but the division method of the q data blocks is uncertain, that is, the data blocks included in each of the m data slices are uncertain.

[0128] After constructing the topological logical relationship, at least two slicing grouping methods for the q data blocks can be obtained based on this relationship. Specifically, data blocks can be sliced ​​to meet the minimum distance scheduling cost within a slice. Each slicing grouping method yields a data slice combination, and each data slice combination includes m test data slices. Finally, one data slice combination can be selected as the actual scheduled data slice combination.

[0129] After obtaining R data slice combinations, R relational topology graphs corresponding one-to-one with the R data slice combinations can be constructed. Using these R relational topology graphs, and employing a similar implementation method to steps 101 to 103, the first target data slice scheduling method corresponding to each data slice combination is determined, resulting in R first target data slice scheduling methods. Then, the data slice combination corresponding to the first target data slice scheduling method with the lowest scheduling cost is selected as the actual scheduled data slice combination; that is, the m test data slices included in the target data slice combination are determined as the m target data slices.

[0130] This embodiment allows for flexible adjustment of the data slice size and minimizes the scheduling cost of the data slices actually used for scheduling, thereby further reducing the global scheduling cost of data blocks.

[0131] The various optional implementation methods described in the embodiments of this application can be combined with each other or implemented individually without conflict. The embodiments of this application do not limit this.

[0132] For ease of understanding, the following example is provided:

[0133] In the embodiments of this application, such as Figure 4 As shown, the Map-Reduce scheduling process can be divided into the Split stage, the Map stage, and the Reduce stage. (1) Split stage: The data blocks are split. (2) Map stage: Map tasks or mapper tasks are performed on the data slices. (3) Shuffle stage: The intermediate values ​​of the data are optimized, and the partitioned data is distributed to each Reduce. (4) Reduce stage: The output results of the Map operation are split and scheduled to finally form the result. (5) Result stage: The result is output.

[0134] This application embodiment constructs a three-level topology relationship between data blocks, data slices, and task nodes, establishing a topological logical relationship between Block, Split, and MapTask. Based on fully considering the impact of the storage structure of multiple data backup files, a scheduling cost evaluation parameter, namely Z below, is introduced. Through a greedy algorithm, the optimal slice size and optimal scheduling scheme are solved, effectively reducing the IO scheduling cost of the MapReduce cluster.

[0135] 1. Data Slicing: To facilitate computation, the platform can default to using data slices where the split size equals the block size.

[0136] 2. Sliced ​​Data Storage: MapReduce's input data files are stored in a distributed file system (HDFS). Figure 5 In HDFS, each data block has three redundant backups, stored on three different compute nodes in two different racks to meet data security requirements. Therefore, the data after slicing also meets HDFS's redundancy backup requirements. Due to this redundant storage structure, different backups of the same data slice may correspond to multiple nodes, meaning that executing any data slice will affect the other nodes.

[0137] 3. Data preprocessing: After the cluster completes the data slicing, it constructs the data storage topology (i.e., the topological logical relationship mentioned above), abstracts it into a binary structure, establishes an index, and records the data slice storage tree structure.

[0138] 4. MapTask task execution scheduling.

[0139] 4.1 Map Execution Time Estimation: Estimate the execution time of the Map phase, such as... Figure 6As shown, multiple models can be used for prediction, and compensation parameters can be introduced. The prediction algorithm engine contains multiple data models (including prediction algorithms such as linear regression), dynamically adjusts the prediction algorithm model through prediction deviation, and introduces the prediction deviation into compensation.

[0140] 4.2 Data Slicing and Task Node Relationship Topology Establishment: The overall IO overhead of the system and the data processing topology of each node are as follows, with a total of m Splits and n Tasktrackers.

[0141] Tasktracker n For Split m The estimated execution time of the task is K. nm Tasktracker n For Split m The cost of scheduling I / O is J nm .

[0142] Set K nm Let the weight coefficient be a, and set J nm The weighting coefficient is b.

[0143] Then Tasktracker n For Split m Scheduling cost Z nm Z nm =a×K nm +b×J nm .

[0144] 4.3 Task Scheduling Solution: Based on Figure 2 The topology shown is used for global greedy algorithm calculation. That is, starting from data slice 1, greedy scheduling is performed sequentially to obtain m locally optimal results, which are then compared to obtain the optimal solution. The process can be found in [link to relevant documentation]. Figure 7 .

[0145] exist Figure 7 In this process, for the selected task node, its task execution waiting time can be updated to improve the calculation of the scheduling cost of subsequent data slices and the selection of task nodes.

[0146] 5. Slice capacity optimization

[0147] Setting the split size to be equal to the block size in the initial configuration is a common practice in big data MapReduce. However, in actual execution, the split size will be adjusted to meet the overall scheduling requirements of the cluster, taking into account factors such as parallelism.

[0148] 5.1 In this case, such as Figure 3As shown, the traditional sequential slicing algorithm can be abandoned, and a topological logical relationship between Split and block can be established.

[0149] 5.2 Perform block slicing and grouping, i.e., split1 = (block1, block..., block...) y ), where y can be a positive integer less than q. To satisfy the minimum distance scheduling cost of data blocks within a slice. Then, perform block slice grouping.

[0150] 5.3 Based on Figure 2 The topology shown is used for global greedy algorithm calculation. That is, starting from data slice 1, greedy scheduling is performed sequentially to obtain m local optimal results, which are then compared to obtain the optimal solution.

[0151] 5.4 performs the optimal scheduling solution for splitting, similar to the implementation in 4.

[0152] 5.5 The optimal slip size can be determined using the above algorithm, and the slice size adjustment can be completed.

[0153] The embodiments of this application include the following:

[0154] Based on the general MapReduce task, instead of sequential slicing, a topological logical relationship between splits and blocks is established, the global split storage situation is modeled, a tree structure is built, and then the optimal strategy scheduling is provided, which effectively improves the scheduling accuracy.

[0155] By employing multiple data models to predict the execution time of MapTasks and introducing compensation and correction algorithms to correct biases, more accurate prediction results can be obtained.

[0156] Based on the newly constructed data slice topology and task node TaskTracker relationship, the size of the slice can be adjusted to meet the overall scheduling requirements of the cluster.

[0157] Compared with the prior art, this application has the following beneficial effects:

[0158] Existing technologies employ a limited scheduling order of nodes, racks, and data centers during the MapReduce creation and scheduling process, without considering the impact of storage backup structure on the overall system. This method, based on general MapReduce tasks, models the global split storage situation, establishes a tree structure, and constructs scheduling cost parameters between data slices and task nodes. During the solution process, the scheduling impact of a single MapTask task on other backup nodes is taken into account, which can effectively improve overall efficiency.

[0159] This application employs multiple data models to predict the execution time of MapTasks, and introduces compensation and correction algorithms to correct biases, thereby obtaining more accurate prediction results.

[0160] This application employs a globally optimal solution for MapTask task scheduling, fully considering factors such as IO scheduling and MapTask execution time estimation, which can effectively improve scheduling accuracy.

[0161] See Figure 8 , Figure 8 This is a structural diagram of the task scheduling device provided in the embodiments of this application. Figure 7 As shown, the task scheduling device 800 includes:

[0162] The first acquisition module 801 is used to acquire a first relational topology graph corresponding to the first data slice combination. The first relational topology graph is a relational topology graph between the m target data slices included in the first data slice combination and the n task nodes. In the first relational topology graph, the n task nodes are respectively connected to the m target data slices. The connection line between the i-th task node and the j-th target data slice is associated with at least one scheduling cost contribution value of the i-th task node for the j-th target data slice, where m is an integer greater than 1, i is a positive integer less than or equal to n, and j is a positive integer less than or equal to m.

[0163] The first determining module 802 is configured to determine, based on the first relational topology graph, P first data slice scheduling methods corresponding one-to-one with P data slice scheduling sequences, wherein each of the P data slice scheduling sequences includes the m target data slices, and the scheduling order of the m target data slices is different in different data slice scheduling sequences, and P is a positive integer; the first data slice scheduling method is the data slice scheduling method with the lowest scheduling cost among at least two scheduling methods corresponding to the data slice scheduling sequence, and the scheduling cost is determined based on the at least one scheduling cost contribution value;

[0164] The execution module 803 is used to execute the scheduling of the m target data slices according to the first target data slice scheduling method among the P first data slice scheduling methods.

[0165] In one example, the device further includes:

[0166] The second acquisition module is used to acquire q data blocks to be processed by the MapReduce system, where q is an integer greater than m;

[0167] The construction module is used to construct the topological logical relationship between the q data blocks and the m sample data slices;

[0168] The grouping module is used to slice and group the q data blocks according to the topological logical relationship to obtain R data slice combinations. Each of the R data slice combinations includes m test data slices. In different data slice combinations, at least one test data slice includes different data blocks. R is a positive integer.

[0169] The second determining module is used to determine the scheduling method of the R first target data slices that correspond one-to-one with the R data slice combinations based on the relationship topology diagrams corresponding to the R data slice combinations.

[0170] The third determining module is used to determine the first target data slice scheduling method with the lowest scheduling cost among the R first target data slice scheduling methods as the second target data slice scheduling method;

[0171] The fourth determining module is used to determine the m test data slices included in the target data slice combination as the m target data slices, and the target data slice combination is the data slice combination corresponding to the second target data slice scheduling method.

[0172] In one example, the first determining module includes:

[0173] The first determining submodule is used to determine the second data slice scheduling methods corresponding to the m target data slices according to the first relationship topology diagram, based on the scheduling order of the m target data slices in the first data slice scheduling sequence from first to last, and to obtain m second data slice scheduling methods. The second data slice scheduling method can be the data slice scheduling method with the lowest scheduling cost among the at least two data slice scheduling methods corresponding to the target data slice.

[0174] The second determining submodule is used to determine the set of m second data slice scheduling methods as the first data slice scheduling method corresponding to the first data slice scheduling sequence;

[0175] Wherein, the first data slice scheduling sequence is any one of the P data slice scheduling sequences.

[0176] In one example, the first determining submodule includes:

[0177] The first acquisition unit is used to acquire, from the first relational topology graph, the at least one scheduling cost contribution value of each of the n task nodes for the first data slice, wherein the first data slice is the first scheduled data slice in the first data slice scheduling sequence.

[0178] The first determining unit is configured to determine the first scheduling cost of each task node for the first data slice based on the at least one scheduling cost contribution value of each task node for the first data slice, thereby obtaining n first scheduling costs.

[0179] The second determining unit is used to determine the method of scheduling the first data slice to the first task node as the second data slice scheduling method corresponding to the first data slice, wherein the first task node is the task node corresponding to the lowest scheduling cost among the n first scheduling costs.

[0180] The first update unit is used to update the task execution waiting time of the first task node based on the estimated task execution time of the first task node for the first data slice.

[0181] In one example, the first determining submodule includes:

[0182] The second acquisition unit is used to acquire, from the first relational topology graph, at least one scheduling cost contribution value of each of the n task nodes for the (k+1)th data slice, where the 0th data is the 0th scheduled data slice in the first data slice scheduling sequence, and 0 is a positive integer greater than 1 and less than n.

[0183] The third acquisition unit is used to acquire the task execution waiting time of each of the n task nodes;

[0184] The third determining unit is used to determine the second scheduling cost of each task node for the o-th data slice based on the at least one scheduling cost contribution value of each task node for the o-th data slice and the task execution waiting time of each task node, thereby obtaining n second scheduling costs.

[0185] The fourth determining unit is used to determine the method of scheduling the o-th data slice to the second task node as the second data slice scheduling method corresponding to the o-th data slice, wherein the second task node is the task node corresponding to the lowest scheduling cost among the n second scheduling costs;

[0186] The second update unit is used to update the task execution waiting time of the second task node based on the estimated task execution time of the second task node for the o-th data slice.

[0187] In one example, the at least one scheduling cost contribution value includes: the estimated task execution time and the input / output I / O cost;

[0188] The first acquisition module includes:

[0189] A construction submodule is used to construct a second topological logical relationship between m target data slices and n task nodes, in which the n task nodes are respectively connected to the m target data slices;

[0190] The prediction submodule is used to predict the estimated execution time of each of the n task nodes for the m target data slices based on at least one model, so as to obtain n×m estimated execution times of the tasks.

[0191] The third determining submodule is used to determine the IO cost of each task node in the n task nodes for the m target data slices based on the storage relationship between the m target data slices and the computing nodes corresponding to the n task nodes, so as to obtain n×m IO costs.

[0192] Based on the second topological logical relationship, the estimated execution time of the n×m tasks, and the cost of the n×m IO operations, a first relational topology graph corresponding to the first data slice combination is constructed.

[0193] In one example, the device further includes:

[0194] The third acquisition module is used to acquire P scheduling costs that correspond one-to-one with the P first data slice scheduling methods;

[0195] The fifth determining module is used to determine the first data slice scheduling method corresponding to the lowest scheduling cost among the P scheduling costs as the first target optimal scheduling method.

[0196] The task scheduling device 800 can realize the embodiments of this application. Figure 1 The various processes in the method embodiments, and the ways to achieve the same beneficial effects, will not be repeated here to avoid repetition.

[0197] This application also provides an electronic device. Please refer to [link to relevant documentation]. Figure 9 The electronic device may include a processor 901, a memory 902, and a program 9021 stored in the memory 902 and executable on the processor 901. When the program 9021 is executed by the processor 901, it can achieve... Figure 1 Any steps in the corresponding method embodiments and the achievement of the same beneficial effects will not be repeated here.

[0198] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by hardware related to program instructions, and the program can be stored in a readable medium. This application also provides a readable storage medium storing a computer program, which, when executed by a processor, can implement the above-described methods. Figure 1Any step in the method embodiment that can achieve the same technical effect will not be described again here to avoid repetition.

[0199] The storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0200] The above description represents the preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A task scheduling method, characterized in that, Applied to a MapReduce system, the MapReduce system comprising n task nodes, where n is an integer greater than 1; the method includes: Obtain a first relational topology graph corresponding to the first data slice combination, wherein the first relational topology graph is a relational topology graph between the m target data slices included in the first data slice combination and the n task nodes. In the first relational topology graph, the n task nodes are respectively connected to the m target data slices; the connection line between the i-th task node and the j-th target data slice is associated with: at least one scheduling cost contribution value of the i-th task node for the j-th target data slice, where m is an integer greater than 1, i is a positive integer less than or equal to n, and j is a positive integer less than or equal to m. Based on the first relational topology graph, P first data slice scheduling methods are determined, each corresponding to one of the P data slice scheduling sequences. Each of the P data slice scheduling sequences includes the m target data slices. The scheduling order of the m target data slices differs in different data slice scheduling sequences, and P is a positive integer. The first data slice scheduling method is the data slice scheduling method with the lowest scheduling cost among at least two scheduling methods corresponding to the data slice scheduling sequence. The scheduling cost is determined based on the at least one scheduling cost contribution value. The scheduling of the m target data slices is executed according to the first target data slice scheduling method among the P first data slice scheduling methods.

2. The method according to claim 1, characterized in that, The method further includes: Obtain q data blocks to be processed by the MapReduce system, where q is an integer greater than m; Construct the first topological logical relationship between the q data blocks and the m sample data slices; Based on the first topological logical relationship, the q data blocks are sliced ​​and grouped to obtain R data slice combinations. Each of the R data slice combinations includes m test data slices. In different data slice combinations, at least one test data slice includes different data blocks. R is a positive integer. Based on the relationship topology diagrams corresponding to the R data slice combinations, determine the scheduling methods for the R first target data slices that correspond one-to-one with the R data slice combinations. The scheduling method with the lowest scheduling cost among the R first target data slice scheduling methods is determined as the second target data slice scheduling method; The m test data slices included in the target data slice combination are determined as the m target data slices, and the target data slice combination is the data slice combination corresponding to the second target data slice scheduling method.

3. The method according to claim 1, characterized in that, The step of determining the P first data slice scheduling methods corresponding one-to-one with the P data slice scheduling sequences based on the first relationship topology graph includes: According to the scheduling order of the m target data slices in the first data slice scheduling sequence, and based on the first relation topology graph, the second data slice scheduling methods corresponding to the m target data slices are determined in turn to obtain m second data slice scheduling methods. The second data slice scheduling method is the data slice scheduling method with the lowest scheduling cost among the at least two data slice scheduling methods corresponding to the target data slice. The set of m second data slice scheduling methods is determined as the first data slice scheduling method corresponding to the first data slice scheduling sequence; Wherein, the first data slice scheduling sequence is any one of the P data slice scheduling sequences.

4. The method according to claim 3, characterized in that, The step involves determining the second data slice scheduling methods corresponding to the m target data slices in the first data slice scheduling sequence according to their sequential scheduling order, based on the first relationship topology graph, thereby obtaining m second data slice scheduling methods, including: From the first relational topology graph, obtain the at least one scheduling cost contribution value of each of the n task nodes for the first data slice, where the first data slice is the first scheduled data slice in the first data slice scheduling sequence; Based on the at least one scheduling cost contribution value of each task node for the first data slice, determine the first scheduling cost of each task node for the first data slice, and obtain n first scheduling costs; The method of scheduling the first data slice to the first task node is determined as the second data slice scheduling method corresponding to the first data slice, and the first task node is the task node corresponding to the lowest scheduling cost among the n first scheduling costs; Update the task execution waiting time of the first task node based on the estimated task execution time of the first task node for the first data slice.

5. The method according to claim 3, characterized in that, The step of determining the second data slice scheduling methods corresponding to the m target data slices in the first data slice scheduling sequence according to the first relational topology graph, based on the sequential scheduling order of the m target data slices, to obtain m second data slice scheduling methods, further includes: From the first relational topology graph, obtain at least one scheduling cost contribution value of each of the n task nodes for the (k+1)th data slice, where the oth data is the oth scheduled data slice in the first data slice scheduling sequence, and o is a positive integer greater than 1 and less than n; Obtain the task execution waiting time of each of the n task nodes; Based on the at least one scheduling cost contribution value of each task node for the o-th data slice, and the task execution waiting time of each task node, the second scheduling cost of each task node for the o-th data slice is determined, resulting in n second scheduling costs. The method of scheduling the o-th data slice to the second task node is determined as the second data slice scheduling method corresponding to the o-th data slice, and the second task node is the task node corresponding to the lowest scheduling cost among the n second scheduling costs; Update the task execution wait time of the second task node based on the estimated task execution time of the second task node for the o-th data slice.

6. The method according to claim 1, characterized in that, The at least one scheduling cost contribution value includes: estimated task execution time and input / output I / O cost; The step of obtaining the first relational topology graph corresponding to the first data slice combination includes: Construct a second topological logical relationship between m target data slices and n task nodes, in which the n task nodes are respectively connected to the m target data slices; Based on at least one model, predict the estimated execution time of each of the n task nodes for the m target data slices, and obtain the estimated execution times of n×m tasks. Based on the storage relationship between the m target data slices and the computing nodes corresponding to the n task nodes, the IO cost of each task node for the m target data slices is determined, resulting in n×m IO costs. Based on the second topological logical relationship, the estimated execution time of the n×m tasks, and the cost of the n×m IO operations, a first relational topology graph corresponding to the first data slice combination is constructed.

7. The method according to claim 1, characterized in that, After determining the scheduling methods of the P first data slices that correspond one-to-one with the scheduling sequences of the P data slices, and before executing the scheduling of the m target data slices, the method further includes: Obtain the P scheduling costs that correspond one-to-one with the P first data slice scheduling methods; The first data slice scheduling method corresponding to the lowest scheduling cost among the P scheduling costs is determined as the first target optimal scheduling method.

8. A task scheduling device, characterized in that, Applied to a MapReduce system, the MapReduce system comprising n task nodes, where n is an integer greater than 1; the device comprises: The first acquisition module is used to acquire a first relational topology graph corresponding to the first data slice combination. The first relational topology graph is a relational topology graph between the m target data slices included in the first data slice combination and the n task nodes. In the first relational topology graph, the n task nodes are respectively connected to the m target data slices. The connection line between the i-th task node and the j-th target data slice is associated with at least one scheduling cost contribution value of the i-th task node for the j-th target data slice, where m is an integer greater than 1, i is a positive integer less than or equal to n, and j is a positive integer less than or equal to m. The first determining module is configured to determine, based on the first relational topology graph, P first data slice scheduling methods corresponding one-to-one with P data slice scheduling sequences, wherein each of the P data slice scheduling sequences includes the m target data slices, and the scheduling order of the m target data slices is different in different data slice scheduling sequences, and P is a positive integer; the first data slice scheduling method is the data slice scheduling method with the lowest scheduling cost among at least two scheduling methods corresponding to the data slice scheduling sequence, and the scheduling cost is determined based on the at least one scheduling cost contribution value; The execution module is used to execute the scheduling of the m target data slices according to the first target data slice scheduling method among the P first data slice scheduling methods.

9. An electronic device, comprising: A transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that the processor is configured to read the program from the memory to implement the steps of the task scheduling method as described in any one of claims 1 to 7.

10. A readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the steps of the task scheduling method as described in any one of claims 1 to 7.

Images