Reduce task execution method and device in Spark framework, equipment and storage medium

Abstract

The invention discloses a Reduce task execution method and device in a Spark framework, equipment and a storage medium, and the method comprises the steps: obtaining a first number of Executors required for executing a Reduce task of an application program, and determining a second number of available Executors in a Spark framework node; determining the first number of the available Executors from the second number of the available Executors on the basis of the communication distance between the available Executors, so as to obtain a target Executor set which is corresponding to the application program and has low communication delay; wherein the communication distance represents the communication delay between the nodes where the available Executors are located; and the available Executor in the target Executor set is started in a corresponding Spark framework node, so that the Reduce task of the application program is executed by utilizing the started available Executor. It can be seen that the Executors are started on the nodes close to each other in the Spark framework by taking the communication distance as a main consideration factor, so that communication delay between Reduce tasks during operation running is reduced, and task execution performance is improved.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a method, device, device and storage medium for executing a Reduce task in a Spark framework. Background technique [0002] In Spark, an application usually contains one or more jobs, and each job consists of many stages. In Spark's DAG (Directed Acyclic Graph) execution model, stages are executed sequentially. This means that the tasks in the later stage can only start after the tasks in the previous stage are completed. And the output intermediate result of the previous stage is used as the input of the next stage. Due to the parallel execution of multiple tasks in the stage, a large amount of data communication is required during the job process. In particular, as figure 1 As shown, in the Map phase, the task reads the data block and processes it and then outputs the intermediate data to the local disk. In the Reduce phase, the task obtains its part of the inter...

AI Technical Summary

Problems solved by technology

In the Reduce phase, the task obtains its part of the intermediate data from all Map tasks for processing. This is an All-to-All communication mode. The data transmission delay largely determines the running time of the job, and Greatly limit the performance of the Spark system

Among them, the starting position of the Executor has a great influence on the data transmission delay, and the existing methods cannot achieve the optimal communication efficiency, making the task execution efficiency low

Images