A Reduce task execution method, device, equipment and storage medium in Spark framework

Abstract

The present application discloses a method, apparatus, device and storage medium for executing Reduce tasks in a Spark framework, including: obtaining a first number of Executors required to execute the Reduce tasks of an application, and determining a second number of available Executors in a Spark framework node; determining the first number of available Executors from the second number of available Executors based on the communication distance between the available Executors, so as to obtain a target Executor set with low communication delay corresponding to the application; wherein the communication distance characterizes the communication delay between the nodes where the available Executors are located; starting the available Executors in the target Executor set in the corresponding Spark framework node, so as to use the started available Executors to execute the Reduce tasks of the application. It can be seen that the present application uses the communication distance as the main consideration to start the Executors on nodes close to each other in the Spark framework, so as to reduce the communication delay between Reduce tasks when the job is running, and improve the task execution performance.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a Reduce task execution method, device, equipment and storage medium in a Spark framework. Background Art

[0002] In Spark, an application usually consists of one or more jobs, and each job consists of many stages. In Spark's DAG (Directed Acyclic Graph) execution model, stages are executed in sequence. This means that the tasks of the next stage can only start after the tasks of the previous stage are completed. And the output intermediate results of the previous stage serve as the input of the next stage. Since multiple tasks in a stage are executed in parallel, a large amount of data communication is required during the job process. In particular, if Figure 1 As shown in the figure, in the Map phase, the task reads the data block, processes it, and then outputs the intermediate data to the local disk. In the Reduce phase, the task obtains its intermediate data from all Map tasks for processing. This is an all-to-all communication mode. The data transmission delay largely determines the running time of the job and greatly limits the performance of the Spark system. Among them, the startup position of the Executor has a great impact on the data transmission delay. The existing methods cannot achieve the optimal communication efficiency, resulting in low task execution efficiency.

[0003] Therefore, how to improve the execution efficiency of many-to-many Reduce tasks in Spark is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a Reduce task execution method, device, equipment and storage medium in the Spark framework, which can reduce the communication delay between Reduce tasks during job execution and improve task execution performance. The specific scheme is as follows:

[0005] A first aspect of the present application provides a Reduce task execution method in a Spark framework, comprising:

[0006] Obtain a first number of executors required to execute Reduce tasks of the application program, and determine a second number of available executors in the Spark framework node;

[0007] Determine the first number of available executors from the second number of available executors based on the communication distance between the available executors, so as to obtain a target executor set with low communication delay corresponding to the application; wherein the communication distance represents the communication delay between the nodes where the available executors are located;

[0008] The available Executor in the target Executor set is started in the corresponding Spark framework node, so as to execute the Reduce task of the application program by using the started available Executor.

[0009] Optionally, determining the second number of available Executors in the Spark framework node includes:

[0010] The number of available Executors in each node is determined according to the free resources in each node in the Spark framework and the resource specifications of each required Executor, so as to obtain the second number of available Executors in the Spark framework node.

[0011] Optionally, before determining the first number of available executors from the second number of available executors based on the communication distances between the available executors, the method further includes:

[0012] Determine the network delay and the network bandwidth between each two of the available Executors to obtain the communication distance between each two of the available Executors;

[0013] The communication distance between the available Executors is determined based on the communication distance between every two of the available Executors.

[0014] Optionally, determining the network delay and the network bandwidth between each two of the available Executors to obtain the communication distance between each two of the available Executors includes:

[0015] Determine the proximity level between each two available executors according to the locations of the nodes and racks where the available executors are located; wherein the proximity level between two available executors located in the same node is the first level, the proximity level between two available executors located in different nodes of the same rack is the second level, and the proximity level between two available executors located in different racks is the third level;

[0016] Determine the network delay and the network bandwidth between each two of the available Executors, and obtain the communication distance between each two of the available Executors based on the network delay, the network bandwidth, the proximity level and the first relationship between each two of the available Executors;

[0017] The first relation is:

[0018]

[0019] Among them, d ij represents the communication distance between the i-th available executor and the j-th available executor, PL=1, PL=2 and PL=3 represent the proximity level as the first level, the second level and the third level respectively, NS Indicates the network bandwidth from the node to the switch, band SS Indicates the network bandwidth from switch to switch, latency NS Indicates the network delay from the node to the switch, latency SS Indicates the switch-to-switch network latency.

[0020] Optionally, the determining the first number of available executors from the second number of available executors based on the communication distance between the available executors to obtain a target executor set with low communication latency corresponding to the application program includes:

[0021] Taking each of the available executors as a center point and determining the first number of available executors including itself from the second number of available executors in order of the communication distance from the center point from small to large, so as to obtain a first executor set corresponding to each of the available executors;

[0022] The communication distance between all the available executors in each of the first executor sets is calculated, and the first executor set with the smallest communication distance between all the available executors is determined as the target executor set with low communication delay corresponding to the application.

[0023] Optionally, the determining the first number of available executors from the second number of available executors based on the communication distance between the available executors to obtain a target executor set with low communication latency corresponding to the application program includes:

[0024] A distance threshold is set according to the target requirement, and the available executor corresponding to the minimum value of the communication distance between two available executors that is not greater than the distance threshold is determined as the target executor to obtain a corresponding second executor set;

[0025] Determine whether the number of the available executors in the second Executor set is equal to the first number; if so, determine the second Executor set as the target Executor set with low communication latency corresponding to the application; if not, take each of the current available executors in the second Executor set as the center point and expand the available executors whose communication distance with the center point is not greater than the distance threshold to the second Executor set; stop expanding the second Executor set when the number of the available executors in the most recently expanded second Executor set is equal to the first number.

[0026] Optionally, the method of taking each of the currently available executors in the second executor set as a center point and expanding the available executors whose communication distance with the center point is not greater than the distance threshold to the second executor set further includes:

[0027] When the communication distance corresponding to each currently available executor in the second executor set after the latest expansion is not less than the distance threshold and the number of available executors in the second executor set after the latest expansion is still less than the first number, a new distance threshold is set, and the expansion step is re-executed for the second executor set after the latest expansion according to the set new distance threshold.

[0028] A second aspect of the present application provides a Reduce task execution device in a Spark framework, comprising:

[0029] An acquisition module, used to acquire a first number of executors required to execute a Reduce task of an application program, and determine a second number of available executors in a Spark framework node;

[0030] A determination module, configured to determine the first number of available executors from the second number of available executors based on the communication distance between the available executors, so as to obtain a target executor set with low communication delay corresponding to the application; wherein the communication distance represents the communication delay between the nodes where the available executors are located;

[0031] The execution module is used to start the available Executor in the target Executor set in the corresponding Spark framework node, so as to use the started available Executor to execute the Reduce task of the application.

[0032] A third aspect of the present application provides an electronic device, comprising a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the Reduce task execution method in the aforementioned Spark framework.

[0033] A fourth aspect of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the Reduce task execution method in the aforementioned Spark framework is implemented.

[0034] In this application, the first number of Executors required to execute the Reduce task of the application is first obtained, and the second number of available Executors in the Spark framework node is determined; then, based on the communication distance between the available Executors, the first number of available Executors are determined from the second number of available Executors to obtain a target Executor set with low communication delay corresponding to the application; wherein the communication distance characterizes the communication delay between the nodes where the available Executors are located; finally, the available Executors in the target Executor set are started in the corresponding Spark framework node to use the started available Executors to execute the Reduce task of the application. It can be seen that after determining the number of Executors required to execute the application and the number of available Executors on the node, this application uses the communication distance as the main consideration to start the Executors on nodes close to each other in the Spark framework, so as to reduce the communication delay between Reduce tasks when the job is running and improve the task execution performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0036] Figure 1 A schematic diagram of data transmission between stage tasks in a Spark framework provided by this application;

[0037] Figure 2 A flow chart of a Reduce task execution method in a Spark framework provided in this application;

[0038] Figure 3 A schematic diagram of the distribution of available Executors on a Spark framework node provided for this application;

[0039] Figure 4 A schematic diagram of a SpreadOut distribution strategy provided for this application;

[0040] Figure 5 A schematic diagram of a NoSpreadOut distribution strategy provided for this application;

[0041] Figure 6 A schematic diagram of Executor distribution based on communication distance provided for this application;

[0042] Figure 7 A schematic diagram of the structure of a Reduce task execution device in a Spark framework provided in this application;

[0043] Figure 8 This is a structural diagram of an electronic device for executing Reduce tasks in a Spark framework provided in this application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] In the prior art, Spark provides two different Executor allocation methods: SpreadOut and NoSpreadOut. The former tries to disperse the Executor on each node, and the latter tries to allocate the Executor on fewer nodes. However, neither of them considers the impact of the Executor location on data communication between tasks, which may cause a large amount of cross-node / rack network traffic, or consume too much time due to the long transmission distance, thereby reducing job performance. In response to the above-mentioned technical defects, the present application provides a Reduce task execution solution in the Spark framework, which uses communication distance as the main consideration to start the Executor on nodes close to each other in the Spark framework, so as to reduce the communication delay between Reduce tasks when the job is running and improve the task execution performance.

[0046] Figure 2 A flow chart of a Reduce task execution method in a Spark framework provided in an embodiment of the present application. Figure 2 As shown, the Reduce task execution method in the Spark framework includes:

[0047] S11: Obtain a first number of Executors required to execute Reduce tasks of the application program, and determine a second number of available Executors in the Spark framework node.

[0048] In this embodiment, the first number of Executors required to execute the Reduce task of the application is first obtained, which is determined by the application itself. Then the second number of available Executors in the Spark framework node is determined. Specifically, the number of available Executors in each node can be determined based on the free resources in each node in the Spark framework and the resource specifications of each required Executor to obtain the second number of available Executors in the Spark framework node. When the application is submitted to the cluster, Spark registers with the resource manager (such as YARN) and applies for running Executor resources. Then, the maximum number of Executors (the second number) allowed to be started on each node is determined based on the free resources (mainly CPU and memory) on the node.

[0049] In this embodiment, the scheme needs to be explained in conjunction with the modeling process. For ease of understanding, some important variable declarations are listed in Table 1.

[0050] Table 1 Variable declaration

[0051]

[0052] Let α and β be the number of nodes and racks respectively. First, initialize the network topology of the cluster as the node set {N 0 ,N 1 ,…,N α-1} and rack {R 0 ,R 1 ,…,R β-1}, 1≤β≤α. For any node N l , N l Located in rack R r , r∈{0,1,…,β-1}. In the initial state of Executor allocation, some specific data structures are formalized. Among them, AE represents a set of Executors that are allowed to start, with a number of m. Elements Indicates that the ith executor allowed to start is located on the lth node (if specified). The number of executors allowed to start on each node is calculated based on the free resources on the node. Assuming that a node has 2 free CPUs and 3GB of RAM, and the resource requirements for each executor are 1 CPU and 2GB of RAM, the number of executors allowed to start on the node is 1.

[0053] When an application is submitted to the cluster, Spark starts a certain number of Executors on the node. As the execution container of the task, if these Executors are distributed close to each other on the node, the communication distance between the tasks executed in the Executor will be shorter, and the data communication delay will be lower, and vice versa. In this embodiment, the communication delay depends on the network delay and the network bandwidth. Therefore, it is necessary to determine the network delay and the network bandwidth between each two available Executors to obtain the communication distance between each two available Executors. For the convenience of calculation, the proximity level (PL) of the two Executors is first divided into 3 levels. That is, the proximity level between each two available Executors is determined according to the location of the node and rack where the available Executor is located; wherein, the proximity level between the two available Executors located in the same node is the first level, the proximity level between the two available Executors located in different nodes of the same rack is the second level, and the proximity level between the two available Executors located in different racks is the third level. If two executors are in the same node, PL is equal to 1; if two executors are in different nodes of the same rack, PL is equal to 2; if two executors are in nodes of different racks, PL is equal to 3. According to the above definition, the network delay and the network bandwidth between each two available executors are determined, and the communication distance between each two available executors is obtained based on the network delay, the network bandwidth, the proximity level and the first relational expression between each two available executors. The first relational expression is:

[0054]

[0055] Among them, d ij represents the communication distance between the i-th available executor and the j-th available executor, PL=1, PL=2 and PL=3 represent the proximity level as the first level, the second level and the third level respectively, NS Indicates the network bandwidth from the node to the switch, band SS Indicates the network bandwidth from switch to switch, latency NS Indicates the network delay from the node to the switch, latency SS Indicates the switch-to-switch network latency.

[0056] On this basis, the communication distance between the available executors is determined based on the communication distance between every two of the available executors. In this embodiment, an n×n matrix ED is defined to represent the communication distance between the available executors:

[0057]

[0058] Among them, the element d ij Represents Executor i and Executore i e j The communication distance between them.

[0059] In this model, our goal is to make the required executors close to each other in node distribution. Assuming that the number of executors required by the application is k, the optimal executor allocation problem can be described as how to select a subset U∈AE and |U|=k so that the total communication distance of the started executors is minimized. This problem can be formalized as follows using integer programming:

[0060]

[0061]

[0062] Among them, x i It is a binary variable, 1 means Executore is selected i , 0 means that the Executor is not selected. It is not difficult to prove that the above problem belongs to NP-Hard problem.

[0063] The following is a comparative explanation. Assume that a cluster contains 4 nodes: Node0, Node1, Node2, and Node3. The number of executors allowed to be started on each node is 3, 3, 2, and 1 respectively. The number of executors required by the application is 5. First, sort the nodes by the number of executors that can be allocated, such as Figure 3 Then, SpreadOut starts Executors on each node in turn until the required number is reached, as shown in Figure 4 In contrast, NoSpreadOut starts the maximum number of Executors on the nodes in sequence until the number requirement is met, as shown in Figure 5As shown in the figure, we can see that the executors in SpreadOut are far away from each other, which will cause a large communication delay for the task. Although NoSpreadOut makes the executors close to each other, it does not consider the network topology of the cluster, so it may actually cause a longer transmission distance. Figure 6 The total communication distance between executors is used to measure their proximity to each other. For convenience, only the number of switch hops is used to calculate the communication distance. It can be concluded that the total communication distances of executors in the three solutions are 21, 20, and 18 respectively. Figure 6 The solution is optimal.

[0064] S12: Determine the first number of available executors from the second number of available executors based on the communication distance between the available executors to obtain a target executor set with low communication delay corresponding to the application; wherein the communication distance represents the communication delay between the nodes where the available executors are located.

[0065] In this embodiment, after the communication distance is obtained, the first number of available executors are determined from the second number of available executors based on the communication distance between the available executors to obtain a target executor set with low communication latency corresponding to the application. In a Spark cluster, the communication distance matrix may or may not satisfy the triangle inequality. Different target executor set determination algorithms are used for these two situations.

[0066] In one embodiment, the communication distance matrix satisfies the triangle inequality, that is, the distance between executors satisfies the following relationship: ij ≤d iu +d uj , an approximate algorithm can be used for processing. First, take each of the available executors as the center point and determine the first number of available executors including itself from the second number of available executors in order of the communication distance from the center point from small to large, so as to obtain the first executor set corresponding to each available executor. Then calculate the communication distance between all the available executors in each of the first executor sets, and determine the first executor set with the smallest communication distance between all the available executors as the target executor set with low communication delay corresponding to the application.

[0067] Specifically, the algorithm is for each Executore i Select the k closest Executors (including e i ). For Executor e_i, the set of its k nearest Executors is denoted as S(e i ), the sum of its distances to the other (k-1) Executors is represented by C(e i Then, find the smallest C(e) among all Executors. v ), and the Executor set S(e v ) is assigned to MinSet. In addition, the sum of the distances between all Executors in MinSet, MinCost, is calculated, and MinSet is returned. In this approximate algorithm, using the best algorithm to find the closest k Executors will take O(m) time. For m Executors, the time required is m×O(m). Therefore, the time complexity of the algorithm is O(m 2 ), where m is the number of executors allowed to be started. It is not difficult to prove that the approximation factor of the solution of the model problem defined in the above embodiment by the approximation algorithm is 2.

[0068] In another embodiment, the communication distance matrix does not satisfy the triangle inequality, that is, the distance between Executors satisfies the following relationship: ij >d iu +d uj, a set expansion algorithm can be used for processing. First, a distance threshold is set according to the target demand, and the available executor corresponding to the minimum value in the communication distance between the two available executors that is not greater than the distance threshold is determined as the target executor to obtain the corresponding second executor set. It is not difficult to understand that there can be multiple minimum values, which means that there are multiple elements (minimum communication distances) in the matrix ED that are not greater than the distance threshold and are equal. Of course, the above steps include determining the two available executors corresponding to any minimum value in the communication distance between the two available executors that is not greater than the distance threshold as the target executor, and then taking each available executor as the center point to determine the available executor whose communication distance with the center point is not greater than the distance threshold and put it into the second executor set as the target executor. However, it should be noted that in this case, there is only the available executor whose communication distance with the center point is equal to the distance threshold, and there is no available executor whose communication distance is less than the distance threshold. The second executor set obtained by both is consistent, but the processing efficiency of the former is higher than that of the latter.

[0069] Then determine whether the number of available executors in the second Executor set is equal to the first number. If yes, determine the second Executor set as the target Executor set with low communication delay corresponding to the application. If no, take each of the current available executors in the second Executor set as the center point and expand the available executors whose communication distance with the center point is not greater than the distance threshold to the second Executor set. When the number of available executors in the second Executor set after the latest expansion is equal to the first number, stop the step of expanding the second Executor set. Since the second Executor set is dynamically expanded, the center point should be selected from a dynamic perspective, that is, the available executor that has been expanded to the second Executor set should also be used as the center point for dynamic expansion. Finally, each of the available executors in the target Executor set has been used as the center point for expansion. In addition, if the distance threshold is set too small and the required number of executors cannot be obtained even after expansion, the expansion operation can be performed again by raising the distance threshold. That is, when the communication distance corresponding to each of the currently available executors in the second executor set after the latest expansion is not less than the distance threshold and the number of available executors in the second executor set after the latest expansion is still less than the first number, a new distance threshold is set, and the expansion step is re-executed for the second executor set after the latest expansion according to the set new distance threshold. Specifically, according to the communication distance matrix of the executor, the set expansion algorithm uses a distance threshold (Threshold) to select qualified executors. In order to minimize the total communication distance of the started executors, starting from a minimum Threshold, select executor pairs with a distance less than or equal to the Threshold. Then, according to the selected executors, the executor set is expanded using the sub-algorithm. Finally, it is determined whether the final executor meets the quantity requirement. If so, the selected executor set S is returned; otherwise, larger thresholds are selected in turn, and the above steps are repeated. The time complexity of the set expansion algorithm is 0(m 2), m is the number of executors. The sub-algorithm uses a given distance threshold to add executors. If the set can be expanded to a size of k, it is returned; otherwise, the executors that can be added are empty. Because k≤m, as long as the threshold is set reasonably, it can always return an executor set of size k. The time complexity of the sub-algorithm is O(n×m 2 ), n is the number of selected Executors.

[0070] S13: starting the available Executor in the target Executor set in the corresponding Spark framework node, so as to use the started available Executor to execute the Reduce task of the application.

[0071] In this embodiment, the available executors in the target executor set are started in the corresponding Spark framework node, so as to use the started available executors to execute the Reduce task of the application. In Spark, the executors are started on nodes close to each other, such as the same node or rack, to reduce the communication delay between tasks during job runtime. For data-intensive Spark applications, the data transmission time of the task largely determines the job time. In this embodiment, reasonable executor placement can improve performance.

[0072] It can be seen that the embodiment of the present application first obtains the first number of Executors required to execute the Reduce task of the application, and determines the second number of available Executors in the Spark framework node; then, based on the communication distance between the available Executors, the first number of available Executors are determined from the second number of available Executors to obtain a target Executor set with low communication delay corresponding to the application; wherein the communication distance characterizes the communication delay between the nodes where the available Executors are located; finally, the available Executors in the target Executor set are started in the corresponding Spark framework node to use the started available Executors to execute the Reduce task of the application. After determining the number of Executors required to execute the application and the number of available Executors on the node, this embodiment uses the communication distance as the main consideration to start the Executors on nodes close to each other in the Spark framework, so as to reduce the communication delay between Reduce tasks when the job is running and improve the task execution performance.

[0073] See also Figure 7As shown, the embodiment of the present application also discloses a Reduce task execution device in a Spark framework, including:

[0074] An acquisition module 11 is used to acquire a first number of executors required to execute a Reduce task of an application program, and determine a second number of available executors in a Spark framework node;

[0075] A determination module 12, configured to determine the first number of available executors from the second number of available executors based on the communication distance between the available executors, so as to obtain a target executor set with low communication delay corresponding to the application; wherein the communication distance represents the communication delay between the nodes where the available executors are located;

[0076] The execution module 13 is used to start the available Executor in the target Executor set in the corresponding Spark framework node, so as to use the started available Executor to execute the Reduce task of the application.

[0077] It can be seen that the embodiment of the present application first obtains the first number of Executors required to execute the Reduce task of the application, and determines the second number of available Executors in the Spark framework node; then, based on the communication distance between the available Executors, the first number of available Executors are determined from the second number of available Executors to obtain a target Executor set with low communication delay corresponding to the application; wherein the communication distance characterizes the communication delay between the nodes where the available Executors are located; finally, the available Executors in the target Executor set are started in the corresponding Spark framework node to use the started available Executors to execute the Reduce task of the application. After determining the number of Executors required to execute the application and the number of available Executors on the node, this embodiment uses the communication distance as the main consideration to start the Executors on nodes close to each other in the Spark framework, so as to reduce the communication delay between Reduce tasks when the job is running and improve the task execution performance.

[0078] In some specific embodiments, the acquisition module 11 is specifically used to determine the number of available executors in each node according to the free resources in each node in the Spark framework and the resource specifications of each required executor, so as to obtain the second number of available executors in the Spark framework node.

[0079] In some specific embodiments, the Reduce task execution device in the Spark framework further includes:

[0080] A first determination submodule is used to determine the network delay and the network bandwidth between every two of the available Executors to obtain the communication distance between every two of the available Executors;

[0081] A second determining submodule, configured to determine the communication distance between the available Executors based on the communication distance between every two of the available Executors;

[0082] Accordingly, the first determining submodule specifically includes:

[0083] A level division unit, used for determining the level of proximity between every two available executors according to the positions of the nodes and racks where the available executors are located; wherein the level of proximity between two available executors located in the same node is the first level, the level of proximity between two available executors located in different nodes of the same rack is the second level, and the level of proximity between two available executors located in different racks is the third level;

[0084] a computing unit, configured to determine the network delay and the network bandwidth between each two of the available Executors, and obtain the communication distance between each two of the available Executors based on the network delay, the network bandwidth, the proximity level and a first relationship between each two of the available Executors;

[0085] The first relation is:

[0086]

[0087] Among them, d ij represents the communication distance between the i-th available executor and the j-th available executor, PL=1, PL=2 and PL=3 represent the proximity level as the first level, the second level and the third level respectively, NS Indicates the network bandwidth from the node to the switch, bandSS Indicates the network bandwidth from switch to switch, latency NS Indicates the network delay from the node to the switch, latency SS Indicates the switch-to-switch network latency.

[0088] In some specific embodiments, the determining module 12 specifically includes:

[0089] A sorting unit, configured to respectively take each of the available executors as a center point and determine the first number of the available executors including itself from the second number of the available executors in order of the communication distance from the center point from small to large, so as to obtain a first executor set corresponding to each of the available executors;

[0090] A screening unit is used to calculate the communication distance between all the available executors in each of the first executor sets, and determine the first executor set with the smallest communication distance between all the available executors as the target executor set with low communication delay corresponding to the application.

[0091] In some specific embodiments, the determining module 12 specifically includes:

[0092] A setting unit, configured to set a distance threshold according to a target requirement, and determine the available executor corresponding to the minimum value of the communication distance between two available executors that is not greater than the distance threshold as a target executor, so as to obtain a corresponding second executor set;

[0093] A judging unit is used to judge whether the number of the available executors in the second Executor set is equal to the first number; if so, the second Executor set is determined as the target Executor set with low communication delay corresponding to the application; if not, each of the current available executors in the second Executor set is used as a center point and the available executors whose communication distance with the center point is not greater than the distance threshold are expanded to the second Executor set; when the number of the available executors in the second Executor set after the latest expansion is equal to the first number, the step of expanding the second Executor set is stopped.

[0094] Accordingly, in some specific embodiments, the Reduce task execution device in the Spark framework further includes:

[0095] A loop module is used to set a new distance threshold when the communication distance corresponding to each of the currently available executors in the second executor set after the latest expansion is not less than the distance threshold and the number of the available executors in the second executor set after the latest expansion is still less than the first number, and re-execute the expansion step for the second executor set after the latest expansion according to the set new distance threshold.

[0096] Furthermore, an embodiment of the present application also provides an electronic device. Figure 8 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0097] Figure 8 The present invention provides a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the Reduce task execution method in the Spark framework disclosed in any of the aforementioned embodiments.

[0098] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0099] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0100] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the Reduce task execution method in the Spark framework executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. The data 223 can include the task data collected by the electronic device 20.

[0101] Furthermore, an embodiment of the present application also discloses a storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the Reduce task execution method steps in the Spark framework disclosed in any of the aforementioned embodiments are implemented.

[0102] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0103] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0104] The Reduce task execution method, device, equipment and storage medium in the Spark framework provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A Reduce task execution method in a Spark framework, characterized in that: include: Obtain a first number of executors required to execute Reduce tasks of the application program, and determine a second number of available executors in the Spark framework node; Determine the first number of available executors from the second number of available executors based on the communication distance between the available executors, so as to obtain a target executor set with low communication delay corresponding to the application; wherein the communication distance represents the communication delay between the nodes where the available executors are located; Starting the available Executor in the target Executor set in the corresponding Spark framework node, so as to use the started available Executor to execute the Reduce task of the application; The determining the first number of available executors from the second number of available executors based on the communication distance between the available executors to obtain a target executor set with low communication delay corresponding to the application program includes: A distance threshold is set according to the target requirement, and the available executor corresponding to the minimum value of the communication distance between two available executors that is not greater than the distance threshold is determined as the target executor to obtain a corresponding second executor set; Determine whether the number of the available executors in the second Executor set is equal to the first number; if so, determine the second Executor set as the target Executor set with low communication latency corresponding to the application; if not, take each of the current available executors in the second Executor set as the center point and expand the available executors whose communication distance with the center point is not greater than the distance threshold to the second Executor set; stop expanding the second Executor set when the number of the available executors in the most recently expanded second Executor set is equal to the first number.

2. The Reduce task execution method in the Spark framework according to claim 1, characterized in that: The determining the second number of available Executors in the Spark framework node includes: The number of available Executors in each node is determined according to the free resources in each node in the Spark framework and the resource specifications of each required Executor, so as to obtain the second number of available Executors in the Spark framework node.

3. The Reduce task execution method in the Spark framework according to claim 1, characterized in that: Before determining the first number of the available executors from the second number of the available executors based on the communication distances between the available executors, the method further includes: Determine the network delay and the network bandwidth between each two of the available Executors to obtain the communication distance between each two of the available Executors; The communication distance between the available Executors is determined based on the communication distance between every two of the available Executors.

4. The Reduce task execution method in the Spark framework according to claim 3, characterized in that: The determining of the network delay and the network bandwidth between each two of the available Executors to obtain the communication distance between each two of the available Executors includes: Determine the proximity level between each two available executors according to the locations of the nodes and racks where the available executors are located; wherein the proximity level between two available executors located in the same node is the first level, the proximity level between two available executors located in different nodes of the same rack is the second level, and the proximity level between two available executors located in different racks is the third level; Determine the network delay and the network bandwidth between each two of the available Executors, and obtain the communication distance between each two of the available Executors based on the network delay, the network bandwidth, the proximity level and the first relationship between each two of the available Executors; The first relation is: Among them, d ij represents the communication distance between the i-th available executor and the j-th available executor, PL=1, PL=2 and PL=3 represent the proximity level as the first level, the second level and the third level respectively, and band NS Indicates the network bandwidth from the node to the switch, band SS Indicates the network bandwidth from switch to switch, latency NS Indicates the network delay from the node to the switch, latency SS Indicates the switch-to-switch network latency.

5. The Reduce task execution method in the Spark framework according to any one of claims 1 to 4, characterized in that: The determining the first number of available executors from the second number of available executors based on the communication distance between the available executors to obtain a target executor set with low communication delay corresponding to the application program includes: Taking each of the available executors as a center point and determining the first number of available executors including itself from the second number of available executors in order of the communication distance from the center point from small to large, so as to obtain a first executor set corresponding to each of the available executors; The communication distance between all the available executors in each of the first executor sets is calculated, and the first executor set with the smallest communication distance between all the available executors is determined as the target executor set with low communication delay corresponding to the application.

6. The Reduce task execution method in the Spark framework according to claim 1, characterized in that: After taking each of the currently available executors in the second executor set as a center point and expanding the available executors whose communication distance with the center point is not greater than the distance threshold to the second executor set, the method further includes: When the communication distance corresponding to each currently available executor in the second executor set after the latest expansion is not less than the distance threshold and the number of available executors in the second executor set after the latest expansion is still less than the first number, a new distance threshold is set, and the expansion step is re-executed for the second executor set after the latest expansion according to the set new distance threshold.

7. A Reduce task execution device in a Spark framework, characterized in that: include: An acquisition module, used to acquire a first number of executors required to execute a Reduce task of an application program, and determine a second number of available executors in a Spark framework node; A determination module, configured to determine the first number of available executors from the second number of available executors based on the communication distance between the available executors, so as to obtain a target executor set with low communication delay corresponding to the application; wherein the communication distance represents the communication delay between the nodes where the available executors are located; An execution module, used to start the available Executor in the target Executor set in the corresponding Spark framework node, so as to use the started available Executor to execute the Reduce task of the application; A setting unit, configured to set a distance threshold according to a target requirement, and determine the available executor corresponding to the minimum value of the communication distance between two available executors that is not greater than the distance threshold as a target executor, so as to obtain a corresponding second executor set; A judging unit is used to judge whether the number of the available executors in the second Executor set is equal to the first number; if so, the second Executor set is determined as the target Executor set with low communication delay corresponding to the application; if not, each of the current available executors in the second Executor set is used as a center point and the available executors whose communication distance with the center point is not greater than the distance threshold are expanded to the second Executor set; when the number of the available executors in the second Executor set after the latest expansion is equal to the first number, the step of expanding the second Executor set is stopped.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the Reduce task execution method in the Spark framework according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Used to store computer executable instructions, which, when loaded and executed by a processor, implement the Reduce task execution method in the Spark framework according to any one of claims 1 to 6.

Images