Executor allocation method, device, equipment and storage medium based on Spark framework

Abstract

The present application discloses an Executor allocation method, apparatus, device and storage medium based on the Spark framework, including: determining the communication cost for all Map tasks in the Map stage to obtain the corresponding data block on each first idle node in the Spark framework, so as to obtain the first communication cost corresponding to each first idle node; sorting all the first idle nodes in order of the size of the first communication cost, and sequentially allocating the maximum number of available Executors for the first idle nodes, a number of first Executors; when the total number of the first Executors allocated is the first required number of Executors, stopping allocating the first Executors to obtain the first Executor set that executes the Map task in the Map stage, including the currently allocated first Executors. It can be seen that the present application allocates Executors on nodes close to the input data blocks, which can improve data locality in Spark task scheduling and effectively reduce the network traffic and data access delay of tasks.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an Executor allocation method, apparatus, device and storage medium based on a Spark framework. Background Art

[0002] As applications in the big data era demand increasingly more responsive data, the emerging Spark distributed computing framework, with its superior features, has attracted significant attention and gained widespread adoption, particularly in Google, Yahoo!, Baidu, and Tencent. Compared to Hadoop and other distributed computing frameworks, Spark introduces the concept of Resilient Distributed Datasets (RDDs), which leverage in-memory computing to efficiently execute tasks, particularly for iterative computations. The large amount of data transferred during the execution of Spark applications can extend task runtimes and cause cluster network congestion, thus impacting system performance.

[0003] Therefore, how to solve the network communication bottleneck of Spark applications is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide an Executor allocation method, apparatus, device, and storage medium based on the Spark framework, which can improve data locality in Spark task scheduling and effectively reduce task network traffic and data access latency. The specific solution is as follows:

[0005] The first aspect of the present application provides an Executor allocation method based on the Spark framework, comprising:

[0006] Determine the communication cost of all Map tasks in the Map phase acquiring the corresponding data block from each first idle node in the Spark framework, so as to obtain a first communication cost corresponding to each first idle node;

[0007] Sort all the first idle nodes in order of the first communication costs, and sequentially allocate first executors corresponding to the maximum number of available executors of the first idle nodes to the sorted first idle nodes;

[0008] When the total number of the allocated first Executors reaches the first required number of Executors, the allocation of the first Executors is stopped to obtain a first Executor set including the currently allocated first Executors that executes the Map task in the Map phase.

[0009] Optionally, determining the communication cost of all Map tasks in the Map phase to obtain the corresponding data block from each first idle node in the Spark framework to obtain a first communication cost corresponding to each first idle node includes:

[0010] Determine a second communication cost for each of the Map tasks in the Map phase to obtain a corresponding data block from each of the first idle nodes in the Spark framework;

[0011] The second communication cost on each of the first idle nodes is summed to obtain the first communication cost corresponding to each of the first idle nodes.

[0012] Optionally, the second communication cost is positively correlated with the size of the data block and the network distance between the first idle node where the Map task is located and the node where the data block is located.

[0013] Optionally, determining the second communication cost for each Map task in the Map stage to obtain a corresponding data block from each first idle node in the Spark framework includes:

[0014] Determining the network distance between the first idle node where each Map task is located in the Map phase and the nodes where multiple copies of the data block are located; wherein the data block is stored in the form of multiple copies;

[0015] The communication cost of the Map task in acquiring the copy on the node with the smallest network distance at the first idle node is determined as the second communication cost.

[0016] Optionally, when the total number of the first executors allocated reaches the required number of executors, after stopping allocating executors to obtain a first executor set including the currently allocated first executors that executes the Map task in the Map phase, the method further includes:

[0017] Determine the communication cost for all Reduce tasks in the Reduce phase to obtain corresponding partitioned data from each second idle node in the Spark framework, so as to obtain a third communication cost corresponding to each second idle node; the partitioned data is the data in the partition obtained by dividing the intermediate data output by each Map task in the Map phase into regions consistent with the number of Reduce tasks according to bucket type;

[0018] Sort all the second idle nodes in order of the third communication cost, and sequentially allocate second executors corresponding to the maximum number of available executors of the second idle nodes to the sorted second idle nodes;

[0019] When the total number of allocated second Executors reaches the second required number of Executors, allocation of the second Executors is stopped to obtain a second Executor set including the currently allocated second Executors that executes the Reduce task in the Reduce phase.

[0020] Optionally, determining the communication cost of all Reduce tasks in the Reduce phase to obtain corresponding partition data from each second idle node in the Spark framework to obtain a third communication cost corresponding to each second idle node includes:

[0021] Determining a fourth communication cost for each of the Reduce tasks in the Reduce phase to obtain the corresponding partition data from each of the second idle nodes in the Spark framework;

[0022] The fourth communication cost on each of the second idle nodes is summed to obtain the third communication cost corresponding to each of the second idle nodes.

[0023] Optionally, the fourth communication cost is positively correlated with the data size of the intermediate data in each partition and the network distance between the second idle node where the Reduce task is located and the node where the intermediate data in each partition is located.

[0024] A second aspect of the present application provides an Executor allocation device based on the Spark framework, comprising:

[0025] A first determining module is used to determine the communication cost of all Map tasks in the Map phase to obtain corresponding data blocks from each first idle node in the Spark framework, so as to obtain a first communication cost corresponding to each first idle node;

[0026] A first allocation module is configured to sort all the first idle nodes in order of the first communication costs, and sequentially allocate first executors having a maximum number of available executors of the first idle nodes to the sorted first idle nodes;

[0027] The first generation module is used to stop allocating the first Executor when the total number of the allocated first Executors reaches the first required number of Executors, and obtain a first Executor set including the currently allocated first Executor that executes the Map task in the Map phase.

[0028] A third aspect of the present application provides an electronic device, comprising a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the aforementioned Executor allocation method based on the Spark framework.

[0029] A fourth aspect of the present application provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are loaded and executed by a processor, the aforementioned Executor allocation method based on the Spark framework is implemented.

[0030] In this application, the communication cost of all Map tasks in the Map stage to obtain the corresponding data block on each first idle node in the Spark framework is first determined to obtain the first communication cost corresponding to each first idle node; then all the first idle nodes are sorted in order of the size of the first communication cost, and the maximum number of available Executors of the first idle nodes are allocated to the sorted first idle nodes in turn; finally, when the total number of the first Executors allocated is the first required number of Executors, the allocation of the first Executors is stopped to obtain the first Executor set that executes the Map task in the Map stage, including the currently allocated first Executors. It can be seen that this application allocates Executors on nodes close to the input data blocks, which can improve data locality in Spark task scheduling and effectively reduce the network traffic and data access delay of tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0032] Figure 1 A flow chart of the Executor allocation method based on the Spark framework provided in this application;

[0033] Figure 2 A directed acyclic graph of a Spark program provided in this application;

[0034] Figure 3 This is a schematic diagram of the structure of an Executor allocation device based on the Spark framework provided in this application;

[0035] Figure 4 This is a structural diagram of an Executor allocation electronic device based on the Spark framework provided in this application. DETAILED DESCRIPTION

[0036] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0037] In the prior art, Spark provides two Executor allocation algorithms, spreadOut and noSpreadOut, to determine on which nodes the Executor is started. However, unlike the Hadoop framework, the tasks in Spark run in parallel in the Executor in a multi-threaded manner. As the execution container of the task, the location of the Executor will directly affect the locality acquisition of the task. Neither spreadOut nor noSpreadOut fully considers the data locality factor, resulting in low network communication efficiency. In response to the above technical defects, the present application provides an Executor allocation scheme based on the Spark framework, which allocates the Executor on the node close to the input data block, which can improve the data locality in Spark task scheduling and effectively reduce the network traffic and data access delay of the task.

[0038] Figure 1 This is a flow chart of an Executor allocation method based on the Spark framework provided in this application embodiment. Figure 1 As shown, the Executor allocation method based on the Spark framework includes:

[0039] S11: Determine the communication cost of all Map tasks in the Map phase to obtain corresponding data blocks from each first idle node in the Spark framework, so as to obtain a first communication cost corresponding to each first idle node.

[0040] Spark applications form a directed acyclic graph (DAG) in terms of computational logic, which consists of many stages based on the lineage relationship of RDDs. From the perspective of the MapReduce programming model, these stages can be divided into two types, namely the Map stage and the Reduce stage. Figure 2 As shown in the figure, in the Map phase, a partition of an RDD corresponds to a partition of its parent RDD, namely Stage 1 and Stage 3. In the Reduce phase, a partition of an RDD corresponds to multiple partitions of its parent RDD, namely Stage 2 and Stage 4. A Map task and a Reduce task are responsible for computing a partition of the RDD, and both can result in large amounts of data transfer when obtaining input data. In the Map phase, if the task and the data block are on different nodes, the task will read data across nodes / racks. In the Reduce phase, a task pulls its portion of intermediate data from all output tasks in the previous phase, a process called shuffle, which is a many-to-many communication model. The large amount of data transfer generated in these two phases can extend task runtimes, cause cluster network congestion, and thus affect system performance. To address these network communication bottlenecks, Spark considers data locality as a primary factor in task scheduling in both the Map and Reduce phases. Data locality means placing computations / tasks close to the data, thereby reducing transmission latency and network I / O overhead.

[0041] It is not difficult to understand that if the Executor is started on a node far away from the input data block, it will be difficult for the Map task to access the data locally. In this embodiment, the task scheduler uses a classic delayed scheduling algorithm in the Map stage, which tries to assign the Map task to the node where the data block is located to avoid remote data copying. Therefore, it is determined that all Map tasks in the Map stage obtain the communication cost of the corresponding data block from each first idle node in the Spark framework respectively, so as to obtain the first communication cost corresponding to each of the first idle nodes. Specifically, it is first determined that each of the Map tasks in the Map stage obtains the second communication cost of the corresponding data block from each first idle node in the Spark framework respectively; then the second communication cost on each of the first idle nodes is summed to obtain the first communication cost corresponding to each of the first idle nodes.

[0042] The second communication cost is positively correlated with the data block size and the network distance between the first idle node where the Map task resides and the node where the data block resides. Data blocks typically exist in multiple copies. In the Spark framework, a task retrieves the data block closest to it on the network. This means that the network distance between the first idle node where each Map task resides and the nodes where multiple copies of the data block reside are determined separately during the Map phase. Based on this, the communication cost for the Map task to obtain a copy from the node with the smallest network distance from the first idle node is determined as the second communication cost.

[0043] For ease of understanding, this embodiment is described in conjunction with a data model. Table 1 shows some important variable declarations.

[0044] Table 1 Variable declaration

[0045]

[0046] In the Map phase, each Map task is responsible for processing a data block and outputting the intermediate data to the local disk. First, the network topology of the Spark distributed computing cluster can be formalized as a node set NS = {N0, N1, ..., N α-1} and rack set RS={R0,R1,…,R β-1}, 1≤β≤α. For any node N l , N l Located on rack R r , r∈{0,1,…,β-1}. When allocating the initial state of the Executor in the Map phase, some specific data structures are formalized as follows:

[0047] NS map It contains n_1 idle nodes that are allowed to start Executor and is a subset of NS. Its element N i Indicates the i-th idle node, its remaining CPU and memory capacity are represented as free_cpu i and free_memory i Therefore, the number of Executors allowed to start on each node can be calculated as:

[0048]

[0049]

[0050] in, is node N iThe number of executors allowed to start on the node, cpu_conf and memory_conf are the number of CPUs and memory capacity configured for each executor. For example, node N i The remaining resources are 5 CPUs and 16 GB of memory. Each Executor requires 2 CPUs and 4 GB of memory. The number of Executors allowed to start on the node is calculated as:

[0051]

[0052] BK is a vector of length m1, whose elements b j Indicates the jth data block. In HDFS, the data set is divided into many data blocks, and each block is replicated multiple times, which can be represented as b j ={b j0 ,b j1 ,…,b j(λ-1)}, usually the replication factor λ = 3. MT is a set of Map tasks, whose elements t j Indicates processing of the corresponding data block b j The jth task of . Because each Map task is responsible for one data block, the number of tasks is equal to m1. During task scheduling, each Map task may be scheduled to an Executor on any node. To perceive data locality, we define the communication cost when each Map task obtains a data block from a different node, expressed as an n1×m1 matrix CM:

[0053]

[0054] Among them, c i,j Indicates that when task t is assigned j To node N i Get data block b j communication cost.

[0055] For network data transmission, the communication cost depends on two factors: the amount of data transmitted and the network distance. That is, the less the amount of data, the shorter the network distance, and the smaller the communication overhead, and vice versa. Therefore, the communication cost c i,j It can be calculated as:

[0056] c i,j =d(i,j)×|b j |

[0057] Among them, d(i,j) is the data block b j and node N i The network distance between |b j| is the data block size. Based on the HDFS splitting strategy, each data block defaults to 128MB, except for the last block left after splitting. However, in some special cases, inconsistent data block sizes may be necessary due to the presence of large, indivisible records.

[0058] Each data block b j There are λ copies stored in different nodes, Map task t j Assigned to N i The closest copy will be obtained. Therefore, the distance d(i,j) can be calculated as N i to b i The distance to the nearest replica:

[0059] d(i,j)=min{d(i,j0),d(i,j1),…,d(i,j(λ-1))}

[0060] Where d(i,jk) is the node N i and copy b jk The network distance between the nodes.

[0061] In addition, the distribution of data blocks among cluster nodes can be known in advance, and the network distance between nodes can be predefined based on the network topology of the cluster, which can be expressed as an α×α matrix D:

[0062] D=[DV0,DV1,…,DV l ,…,DV α-1 ] T

[0063] and

[0064] DV l =[dis l,0 ,dis l,1 ,…,dis l,p ,…,dis l,(α-1) ]

[0065] where the vector DV l is node N l The network distance between other nodes, dis l,p is node N l With N p The network distance between nodes is usually 0.

[0066] S12: Sort all the first idle nodes in order of the first communication costs, and sequentially allocate first executors having the maximum number of available executors of the first idle nodes to the sorted first idle nodes.

[0067] S13: When the total number of the allocated first Executors reaches the first required number of Executors, stop allocating the first Executors and obtain a first Executor set that includes the currently allocated first Executors and executes the Map task in the Map phase.

[0068] In this embodiment, all the first idle nodes are first sorted in order of the first communication cost, and then the first executors corresponding to the maximum number of available executors of the first idle nodes are sequentially allocated to the sorted first idle nodes. When the total number of allocated first executors reaches the first required number of executors, allocation of the first executors is stopped to obtain a first executor set including the currently allocated first executors that executes the Map task in the Map phase.

[0069] For a single Map task t j The best execution location is when the task runs on node N i When it is in an Executor, it can get the minimum communication cost c ij However, for node N i , because any task can run in its Executor, we should consider the data locality of all tasks, rather than a specific task. Based on the above theoretical analysis, we define a node N i The overall data locality level that can be provided is expressed as data_locality(i), and NS map The idle nodes in are sorted in ascending order according to the value of data_localiy, which can be calculated as the sum of the communication costs when each task runs in the Executor of the node:

[0070]

[0071] Among them, m1 is the number of Map tasks, n1 is the number of idle nodes. Obviously, the larger the value of data_locality(i), the greater the communication cost, so the node N i The better the overall data locality that can be provided, the worse the performance is, and vice versa. Specifically, in the Map phase, data_locality(i) can be further calculated as:

[0072]

[0073] Based on the above model, in order to provide better data locality for tasks, an executor needs to be started on a node close to the input data. In other words, all tasks running on this executor can obtain lower communication costs. Assume that the number of executors required is u, and the set of executors selected for startup is SE = {e0,e1,…,e u-1}, then the optimal Executor allocation scheme can be formalized as:

[0074]

[0075] Among them, cost(e k ) means that all Map tasks run on Executore k The sum of the communication costs in , which can be further calculated as:

[0076]

[0077] in, When the task t j Running on Executor k When the data block b is obtained j Assume that e k Start on node N i Above, recorded as Then we have:

[0078] In this embodiment, during the Reduce phase, tasks are dispatched to nodes storing the majority of intermediate data, reducing the amount of remote data transmission. The intermediate data generated by the Map phase is partitioned using a partitioner (e.g., Hash or Range). Then, during the Reduce phase, each Reduce task pulls its portion of intermediate data from the output of all Map tasks in the previous phase. This creates a many-to-many communication model across the network.

[0079] Determine the communication cost for all Reduce tasks in the Reduce phase to obtain corresponding partitioned data from each second idle node in the Spark framework, thereby obtaining a third communication cost corresponding to each second idle node. The partitioned data is the data in the partitions obtained by partitioning the intermediate data output by each Map task in the Map phase into regions corresponding to the number of Reduce tasks according to bucket type. Accordingly, first determine the communication cost for all Reduce tasks in the Reduce phase to obtain corresponding partitioned data from each second idle node in the Spark framework, thereby obtaining a third communication cost corresponding to each second idle node. The partitioned data is the data in the partitions obtained by partitioning the intermediate data output by each Map task in the Map phase into regions corresponding to the number of Reduce tasks according to bucket type. Then, sort all second idle nodes in order of the third communication cost, and sequentially allocate second executors corresponding to the maximum number of available executors for the second idle nodes to the sorted second idle nodes. When the total number of allocated second executors reaches the second required number of executors, stop allocating second executors, and obtain a second executor set that executes the Reduce tasks in the Reduce phase, including the currently allocated second executors. The process for determining the third communication cost is as follows: first, determining the fourth communication cost for each Reduce task in the Reduce phase to obtain the corresponding partition data from each second idle node in the Spark framework, and then summing the fourth communication costs on each second idle node to obtain the third communication cost corresponding to each second idle node. The fourth communication cost is positively correlated with the size of the intermediate data in each partition and the network distance between the second idle node where the Reduce task is located and the node where the intermediate data in each partition is located.

[0080] The solution of this embodiment is also described in conjunction with the data model. Table 2 shows some important variable declarations.

[0081] Table 2 Variable declaration

[0082]

[0083] The job phases in Spark are executed sequentially, which means that the Reduce task must wait until all tasks in the previous phase are completed before it can start. When allocating the initial state of the Executor in the Redcue phase, some specific data structures are defined as follows:

[0084] NSreduce It contains n2 idle nodes that are allowed to start Executor, which is a subset of NS. Its element N i Represents the i-th idle node, which is required to have the minimum remaining computing resources. Similarly, the number of Executors allowed to start on each node can be calculated in advance, represented by EN reduce :

[0085]

[0086] in, is node N i The number of Executors that are allowed to start.

[0087] BT is a matrix of length m1×m2, which represents the distribution of the intermediate data output by the Map task in the previous stage after partitioning. Its element b kj Represents the jth bucket, whose data comes from Map task t k m1 is the number of Map tasks, m2 is the number of partitions. PN is a vector of length m2, whose element p j Indicates the jth partition. During the Shuffle process, the intermediate data of the jth bucket of each Map task constitutes partition p j , which can be formalized as:

[0088]

[0089] RT is a set of Reduce tasks. Its elements t j Indicates processing partition p j Since each Reduce task computes one partition, the number of Reduce tasks is the same as the number of partitions.

[0090] For Reduce task scheduling, each task can run in the Executor of any node. We define the communication cost of each Redcue task when obtaining partitions on different nodes, expressed as an n2×m2 matrix CR:

[0091]

[0092] Among them, c i,j Indicates that when task t is assigned j To node N i Get partition p j communication cost.

[0093] As analyzed above, partition p j It is composed of the j-th bucket output by all Map tasks, so the Redcue task tj It is necessary to pull data from the node where each Map task is located. i,j It is task t j The sum of the communication costs for each bucket can be calculated as:

[0094]

[0095] Among them, d(i,kj) is bucket b kj and node N i The network distance between |b kj | is bucketb kj The data size of d(i,kj) is calculated as the node N i and Map task t k The network distance between the nodes.

[0096] For a single Reduce task t j , we define a node N i The overall data locality level that can be provided is expressed as data_locality(i), and nS map The idle nodes in are sorted in ascending order according to the value of data_localiy, which can be calculated as the sum of the communication costs when each task runs in the Executor of the node:

[0097]

[0098] Among them, m2 is the number of Reduce tasks, and n2 is the number of idle nodes. Obviously, the larger the value of tata_locality(i), the greater the communication cost. Therefore, the node N i The better the overall data locality that can be provided, the worse the performance is, and vice versa. Specifically, in the Reduce phase, data_locality(i) can be further calculated as:

[0099]

[0100] It can be seen that the embodiment of the present application first determines the communication cost of all Map tasks in the Map stage to obtain the corresponding data block on each first idle node in the Spark framework, so as to obtain the first communication cost corresponding to each first idle node; then sorts all the first idle nodes in order of the size of the first communication cost, and sequentially allocates the first Executors with the maximum number of available Executors of the first idle nodes on the sorted first idle nodes; finally, when the total number of the first Executors allocated is the first required number of Executors, stops allocating the first Executor and obtains the first Executor set that executes the Map task in the Map stage, including the currently allocated first Executors. The embodiment of the present application allocates the Executor on the node close to the input data block, which can improve the data locality in Spark task scheduling and effectively reduce the network traffic and data access delay of the task.

[0101] See also Figure 3 As shown, the embodiment of the present application also discloses an Executor allocation device based on the Spark framework, including:

[0102] A first determining module 11 is configured to determine the communication cost for all Map tasks in the Map phase to obtain corresponding data blocks from each first idle node in the Spark framework, so as to obtain a first communication cost corresponding to each first idle node;

[0103] A first allocation module 12 is configured to sort all the first idle nodes in order of the first communication costs, and sequentially allocate first executors corresponding to the maximum number of available executors of the first idle nodes to the sorted first idle nodes;

[0104] The first generating module 13 is configured to stop allocating the first Executor when the total number of the allocated first Executors reaches the first required number of Executors, and obtain a first Executor set including the currently allocated first Executors for executing the Map task in the Map phase.

[0105] It can be seen that the embodiment of the present application first determines the communication cost of all Map tasks in the Map stage to obtain the corresponding data block on each first idle node in the Spark framework, so as to obtain the first communication cost corresponding to each first idle node; then sorts all the first idle nodes in order of the size of the first communication cost, and sequentially allocates the first Executors with the maximum number of available Executors of the first idle nodes on the sorted first idle nodes; finally, when the total number of the first Executors allocated is the first required number of Executors, stops allocating the first Executor and obtains the first Executor set that executes the Map task in the Map stage, including the currently allocated first Executors. The embodiment of the present application allocates the Executor on the node close to the input data block, which can improve the data locality in Spark task scheduling and effectively reduce the network traffic and data access delay of the task.

[0106] In some specific embodiments, the first determining module 11 specifically includes:

[0107] A first determining submodule is configured to determine a second communication cost for each of the Map tasks in the Map phase to obtain a corresponding data block from each of the first idle nodes in the Spark framework;

[0108] The second determining submodule is configured to sum the second communication costs on each of the first idle nodes to obtain the first communication cost corresponding to each of the first idle nodes.

[0109] In some specific embodiments, the first determining submodule specifically includes:

[0110] A first determining unit is configured to respectively determine the network distance between the first idle node where each Map task is located in the Map phase and the nodes where the multiple copies of the data block are located; wherein the data block is stored in the form of multiple copies;

[0111] The second determining unit is configured to determine the communication cost of the Map task in acquiring the copy on the node with the smallest network distance at the first idle node as the second communication cost.

[0112] In some specific embodiments, the Spark framework-based Executor allocation device further includes:

[0113] The second determination module is configured to determine the communication cost for all Reduce tasks in the Reduce phase to obtain corresponding partitioned data from each second idle node in the Spark framework, so as to obtain a third communication cost corresponding to each second idle node; the partitioned data is the data in the partitions obtained by dividing the intermediate data output by each Map task in the Map phase into regions consistent with the number of Reduce tasks according to bucket type;

[0114] A second allocation module is configured to sort all the second idle nodes in order of the size of the third communication cost, and sequentially allocate second executors having a maximum number of available executors of the second idle nodes to the sorted second idle nodes;

[0115] The second generation module is used to stop allocating the second Executor when the total number of allocated second Executors reaches the second required number of Executors, and obtain a second Executor set including the currently allocated second Executor that executes the Reduce task in the Reduce phase.

[0116] In some specific embodiments, the second determining module specifically includes:

[0117] A third determining submodule is configured to determine a fourth communication cost for each of the Reduce tasks in the Reduce phase to obtain the corresponding partition data from each of the second idle nodes in the Spark framework;

[0118] The fourth determining submodule is configured to sum the fourth communication costs on each of the second idle nodes to obtain the third communication cost corresponding to each of the second idle nodes.

[0119] Furthermore, an embodiment of the present application also provides an electronic device. Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0120] Figure 4 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the Executor allocation method based on the Spark framework disclosed in any of the aforementioned embodiments.

[0121] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0122] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon may include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0123] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, enabling the processor 21 to operate and process the massive amount of data 223 in the memory 22. The operating system 221 can be Windows Server, NetWare, Unix, Linux, etc. In addition to including computer programs capable of implementing the Spark framework-based Executor allocation method disclosed in any of the aforementioned embodiments and executed by the electronic device 20, the computer program 222 can further include computer programs capable of performing other specific tasks. The data 223 can include task data collected by the electronic device 20.

[0124] Furthermore, an embodiment of the present application also discloses a storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the Executor allocation method based on the Spark framework disclosed in any of the aforementioned embodiments are implemented.

[0125] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0126] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0127] The above is a detailed introduction to the Executor allocation method, device, equipment and storage medium based on the Spark framework provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. An Executor allocation method based on the Spark framework, characterized in that: include: Determine the communication cost of all Map tasks in the Map phase acquiring the corresponding data block from each first idle node in the Spark framework, so as to obtain a first communication cost corresponding to each first idle node; Sort all the first idle nodes in order of the first communication costs, and sequentially allocate first executors corresponding to the maximum number of available executors of the first idle nodes to the sorted first idle nodes; When the total number of the allocated first executors reaches the first required number of executors, stop allocating the first executors and obtain a first executor set including the currently allocated first executors that executes the map task in the map phase; Determining the communication cost for all Map tasks in the Map phase to obtain the corresponding data block from each first idle node in the Spark framework to obtain the first communication cost corresponding to each first idle node includes: determining the second communication cost for each Map task in the Map phase to obtain the corresponding data block from each first idle node in the Spark framework; summing the second communication cost on each first idle node to obtain the first communication cost corresponding to each first idle node, including: Among them, m1 is the number of Map tasks, n1 is the number of idle nodes, d(i,j) is the data block b j and node N i The network distance between i,j Indicates that when task t is assigned j To node N i Get data block b j The communication cost, |b j | is the size of the data block; The calculation process of the maximum number of available executors of the first idle node and the first executor includes: In the Map phase, each Map task is responsible for processing a data block and outputting intermediate data to the local disk; the network topology of the Spark distributed computing cluster is formalized as a node set NS = {N0, N1, ..., N α-1 } and rack set RS={R0,R1,…,R β-1 }, 1≤β≤α, for any node N l , N l Located on rack R r , r∈{0,1,…,β-1}; when allocating the initial state of the Executor in the Map phase, some specific data structures are formalized as follows: NS map It contains n_1 idle nodes that are allowed to start Executor, which is a subset of NS; its element N i Indicates the i-th idle node, its remaining CPU and memory capacity are represented as free_cpu i and free_memory i ; The number of Executors allowed to start on each node is calculated as: in, is node N i The number of Executors allowed to be started, cpu_conf and memory_conf are the number of CPUs and memory capacity configured for each Executor.

2. The Executor allocation method based on the Spark framework according to claim 1, characterized in that: The second communication cost is positively correlated with the size of the data block and the network distance between the first idle node where the Map task is located and the node where the data block is located.

3. The Executor allocation method based on the Spark framework according to claim 2, characterized in that: The determining of the second communication cost for each Map task in the Map stage to obtain the corresponding data block from each first idle node in the Spark framework includes: Determining the network distance between the first idle node where each Map task is located in the Map phase and the nodes where multiple copies of the data block are located; wherein the data block is stored in the form of multiple copies; The communication cost of the Map task in acquiring the copy on the node with the smallest network distance at the first idle node is determined as the second communication cost.

4. The Executor allocation method based on the Spark framework according to any one of claims 1 to 3, characterized in that: When the total number of the first executors allocated reaches the required number of executors, after stopping allocating executors to obtain a first executor set including the currently allocated first executors that executes the map task in the map phase, the method further includes: Determine the communication cost for all Reduce tasks in the Reduce phase to obtain corresponding partitioned data from each second idle node in the Spark framework, so as to obtain a third communication cost corresponding to each second idle node; the partitioned data is the data in the partition obtained by dividing the intermediate data output by each Map task in the Map phase into regions consistent with the number of Reduce tasks according to bucket type; Sort all the second idle nodes in order of the third communication cost, and sequentially allocate second executors corresponding to the maximum number of available executors of the second idle nodes to the sorted second idle nodes; When the total number of allocated second Executors reaches the second required number of Executors, allocation of the second Executors is stopped to obtain a second Executor set including the currently allocated second Executors that executes the Reduce task in the Reduce phase.

5. The Executor allocation method based on the Spark framework according to claim 4, characterized in that: The determining of the communication cost for all Reduce tasks in the Reduce phase to obtain corresponding partition data from each second idle node in the Spark framework to obtain a third communication cost corresponding to each second idle node includes: Determining a fourth communication cost for each of the Reduce tasks in the Reduce phase to obtain the corresponding partition data from each of the second idle nodes in the Spark framework; The fourth communication cost on each of the second idle nodes is summed to obtain the third communication cost corresponding to each of the second idle nodes.

6. The Executor allocation method based on the Spark framework according to claim 5, characterized in that: The fourth communication cost is positively correlated with the data size of the intermediate data in each partition and the network distance between the second idle node where the Reduce task is located and the node where the intermediate data in each partition is located.

7. An Executor allocation device based on the Spark framework, characterized in that: include: A first determining module is used to determine the communication cost of all Map tasks in the Map phase to obtain corresponding data blocks from each first idle node in the Spark framework, so as to obtain a first communication cost corresponding to each first idle node; A first allocation module is configured to sort all the first idle nodes in order of the first communication costs, and sequentially allocate first executors having a maximum number of available executors of the first idle nodes to the sorted first idle nodes; A first generating module is configured to stop allocating the first executor when the total number of allocated first executors reaches the first required number of executors, and obtain a first executor set including the currently allocated first executor for executing the map task in the map phase; The first determining module is specifically configured to determine a second communication cost for each Map task in the Map phase to obtain a corresponding data block from each first idle node in the Spark framework; Summing up the second communication cost on each of the first idle nodes to obtain the first communication cost corresponding to each of the first idle nodes includes: Among them, m1 is the number of Map tasks, n1 is the number of idle nodes, d(i,j) is the data block b j and node N i The network distance between i,j Indicates that when task t is assigned j To node N i Get data block b j The communication cost, |b j | is the size of the data block; The calculation process of the maximum number of available executors of the first idle node and the first executor includes: In the Map phase, each Map task is responsible for processing a data block and outputting intermediate data to the local disk; the network topology of the Spark distributed computing cluster is formalized as a node set NS = {N0, N1, ..., N α-1 } and rack set RS={R0,R1,…,R β-1 }, 1≤β≤α, for any node N l , N l Located on rack R r , r∈{0,1,…,β-1}; when allocating the initial state of the Executor in the Map phase, some specific data structures are formalized as follows: NS map It contains n_1 idle nodes that are allowed to start Executor, which is a subset of NS; its element N i Indicates the i-th idle node, its remaining CPU and memory capacity are represented as free_cpu i and free_memory i ; The number of Executors allowed to start on each node is calculated as: in, is node N i The number of Executors allowed to be started, cpu_conf and memory_conf are the number of CPUs and memory capacity configured for each Executor.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the Executor allocation method based on the Spark framework according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that Used to store computer-executable instructions, which, when loaded and executed by a processor, implement the Executor allocation method based on the Spark framework according to any one of claims 1 to 6.

Images