Zeppelin-based task scheduling module, device and method

Abstract

The present invention discloses a Zeppelin-based task scheduling module, device, and method. The method comprises: an Azkaban scheduling platform schedules Spark tasks defined on a Zeppelin platform page; a task scheduling module submits tasks to the Zeppelin platform based on event triggering; after the Zeppelin platform receives the task information, it establishes a connection with a Spark interpreter process and submits the task to the Spark interpreter process; after receiving the task submitted by the Zeppelin platform, the Spark interpreter submits the task to a YARN-specified queue based on the YARN queue information contained in the task, thereby executing the Spark task. The present invention enables a task scheduling module to simultaneously schedule multiple Spark tasks to be submitted to a YARN resource pool, thereby completing the parallel execution of multiple tasks and significantly improving overall execution efficiency.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a Zeppelin-based task scheduling module, device, and method. Background Art

[0002] Currently, Spark tasks are still scheduled using the Zeppelin platform's cron scheduling module. This involves setting a batch of Spark tasks' execution cycles and times, and then submitting them in batches to the backend service when the execution time arrives. However, this approach has the following drawbacks: 1. Scheduling based on the cron scheduling module is inflexible and cannot define dependencies between task executions; 2. Tasks are submitted to the Spark interpreter, which supports a FIFO scheduling method, requiring tasks to be submitted in a queue; 3. After tasks are submitted to YARN, YARN's default scheduling pool is default, which uses FIFO scheduling, requiring tasks to be queued for execution; 4. This approach fails to effectively leverage the advantages of parallelism, with overall task execution time increasing linearly with the number of tasks.

[0003] The above problems need to be solved urgently. Summary of the Invention

[0004] To solve related technical problems, the present invention provides a task scheduling module, device and method based on Zeppelin to solve the problems mentioned in the above background technology section.

[0005] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides a task scheduling module based on Zeppelin, which includes:

[0007] The task ID query unit is used to query the paragraph ID (i.e., task ID) contained in the note ID on the Zeppelin platform according to the execution note ID when the task starts to be scheduled;

[0008] An event encapsulation unit, configured to encapsulate the task ID and configuration information into an event after obtaining the task ID;

[0009] The event sending unit is used to send the event encapsulated by the event encapsulation unit to the event bus;

[0010] An event monitoring unit is used to monitor events in the event bus through a listener, and when a task arrives at the event bus, trigger several listeners to obtain task information;

[0011] A user authentication unit, configured to obtain information about a user who is performing a task from a user list based on the task information obtained by the event monitoring unit;

[0012] The task submission unit is used to merge the user information and the task information and submit them to the Zeppelin platform.

[0013] Furthermore, the event sending unit is specifically configured to asynchronously send the event encapsulated by the event encapsulation unit to the event bus.

[0014] In a second aspect, an embodiment of the present invention further provides a task scheduling device based on Zeppelin, wherein the device adopts the task scheduling module based on Zeppelin provided in the above embodiment (hereinafter referred to as the task scheduling module), specifically comprising:

[0015] Azkaban scheduling platform, used to schedule spark tasks defined on the Zeppelin platform page;

[0016] The task scheduling module is used to submit tasks based on event triggers. It predefines n users who submit tasks, completes parallel submission of tasks between multiple users, and serial submission of tasks within a single user;

[0017] Zeppelin platform, used to edit task content;

[0018] Spark interpreter, used to initialize the task execution environment and submit tasks submitted by the client to the YARN platform;

[0019] Yarn resource pool, used for the actual running of tasks.

[0020] Furthermore, the azkaban scheduling platform is also used for task scheduling, and the execution order of tasks can be flexibly set through configuration.

[0021] Furthermore, the editing tasks in the Zeppelin platform include but are not limited to writing Spark codes in Scala and Python formats using web pages.

[0022] Furthermore, the Zeppelin-based task scheduling device divides the Yarn resource pool to separate queues specifically for executing scheduled tasks, with parallel execution between resource pools and serial execution within a resource pool.

[0023] Furthermore, the Zeppelin platform is specifically used to first check whether the user contained in the task information has the authority to execute the task after receiving the task information submitted by the task scheduling module. Only after the authority verification is passed will it establish a connection with the spark interpreter process and submit the task to the spark interpreter process.

[0024] Furthermore, the spark interpreter is specifically used to submit the task to the yarn specified queue of the Yarn resource pool according to the yarn queue information contained in the task after receiving the task submitted by the zeppelin platform, and execute the spark task.

[0025] Furthermore, the Zeppelin-based task scheduling device binds the user's name to the name of the YARN queue.

[0026] In a third aspect, an embodiment of the present invention further provides a task scheduling method based on Zeppelin, the method comprising the following steps:

[0027] The azkaban scheduling platform schedules the spark tasks defined on the zeppelin platform page;

[0028] The task scheduling module submits tasks to the Zeppelin platform based on event triggering;

[0029] After receiving the task information submitted by the task scheduling module, the Zeppelin platform establishes a connection with the Spark interpreter process and submits the task to the Spark interpreter process;

[0030] After receiving the task submitted by the Zeppelin platform, the Spark interpreter submits the task to the YARN specified queue of the YARN resource pool according to the YARN queue information contained in the task, and executes the Spark task.

[0031] The Zeppelin-based task scheduling module proposed in the embodiment of the present invention submits tasks based on event triggering, realizing parallel submission of tasks between multiple users and serial submission of tasks within a single user. The Zeppelin-based task scheduling device and method proposed in the embodiment of the present invention adopts a Zeppelin-based task scheduling module, realizing that one task scheduling module simultaneously schedules multiple Spark tasks to be submitted to the YARN resource pool, completing the parallel execution of multiple tasks and greatly improving the overall execution efficiency. Compared with the prior art, the technical solution of the embodiment of the present invention utilizes asynchrony and parallelism to accelerate the submission and execution of tasks, reduces the execution time of the overall task, and improves timeliness; the same task can be executed multiple times according to the change of external parameters, and is passed to the task by passing parameters; by separating the overall task scheduling from the task execution, the execution time of the overall task can be intuitively viewed through the Azkaban scheduling platform, which is convenient for monitoring and optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate and understand the technical solutions in the embodiments of the present invention, a brief introduction is given below to the background technology of the present invention and the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the contents of the embodiments of the present invention and these drawings without any creative work.

[0033] Figure 1 This is a block diagram of the Zeppelin-based task scheduling module provided in Example 1 of the present invention;

[0034] Figure 2 This is a block diagram of the principle of a task scheduling device based on Zeppelin provided in the third embodiment of the present invention;

[0035] Figure 3 This is a flow chart of the Zeppelin-based task scheduling method provided in Example 4 of the present invention. DETAILED DESCRIPTION

[0036] To make the technical problems solved, the technical solutions adopted, and the technical effects achieved by the present invention more clearly understood, the technical solutions of the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the described embodiments are only some of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0037] Example 1

[0038] like Figure 1 As shown, Figure 1This is a principle block diagram of a Zeppelin-based task scheduling module provided in the first embodiment of the present invention.

[0039] The Zeppelin-based task scheduling module in this embodiment includes a task ID query unit 101 , an event encapsulation unit 102 , an event sending unit 103 , an event monitoring unit 104 , a user authentication unit 105 and a task submission unit 106 .

[0040] The task ID query unit 101 is used to query the paragraphId contained in the noteId, i.e., the task ID, on the zeppelin platform according to the execution noteId when the task starts to be scheduled;

[0041] The event encapsulation unit 102 is used to encapsulate the task ID and configuration information into an event after obtaining the task ID;

[0042] The event sending unit 103 is used to send the event encapsulated by the event encapsulation unit 102 to the event bus;

[0043] The event monitoring unit 104 is used to monitor events in the event bus through a listener, and when a task arrives at the event bus, it triggers several listeners to obtain task information;

[0044] The user authentication unit 105 is used to obtain the user information of the task from the user list according to the task information obtained by the event monitoring unit 104;

[0045] The task submission unit 106 is used to merge the user information and the task information and submit them to the Zeppelin platform.

[0046] Exemplarily, in this embodiment, the event sending unit 103 is specifically configured to asynchronously send the event encapsulated by the event encapsulation unit 102 to the event bus.

[0047] The zeppelin-based task scheduling module proposed in the embodiment of the present invention is implemented in an event-driven manner. By encapsulating events, the programming model is simplified, and the functions can be flexibly expanded by extending event types and registering listeners. The technical solution of the embodiment of the present invention utilizes the interpreter user isolation mechanism in zeppelin to predefine n users who submit tasks, thereby realizing parallel submission of tasks between multiple users and serial submission of tasks within a single user; wherein, the interpreter user isolation mechanism in zeppelin means that when the isolation level set by the interpreter is the user level, each user and the interpreter will have a separate session, and data cannot be shared between the sessions. It is worth mentioning that although the task scheduling module in this embodiment completes the parallel submission of tasks through an event-triggered listening mechanism, it is not limited to this. The event-triggered listening mechanism in this embodiment can be replaced by parallel mechanisms such as multi-process, multi-threading, and coroutines.

[0048] Example 2

[0049] The Zeppelin-based task scheduling device in this embodiment adopts the Zeppelin-based task scheduling module (hereinafter referred to as the task scheduling module) provided in the above-mentioned embodiment 1, which specifically includes the Azkaban scheduling platform, the task scheduling module, the Zeppelin platform, the Spark interpreter, and the Yarn resource pool:

[0050] The azkaban scheduling platform is used to schedule the spark tasks defined on the zeppelin platform page;

[0051] The task scheduling module is used to submit tasks based on event triggering, pre-define n users who submit tasks, complete parallel submission of tasks between multiple users, and serial submission of tasks within a single user;

[0052] The zeppelin platform is used to edit task content;

[0053] The Spark interpreter is used to initialize the task execution environment and submit the tasks submitted by the client to the YARN platform;

[0054] The YARN resource pool is used for the actual execution of tasks.

[0055] Exemplarily, in this embodiment, the azkaban scheduling platform is also used for task timing scheduling, and the execution order of tasks can be flexibly set through configuration.

[0056] For example, the editing task content in the Zeppelin platform described in this embodiment includes but is not limited to writing Spark code in Scala and Python formats using web pages.

[0057] Illustratively, in the spark interpreter described in this embodiment, initializing the task execution environment includes but is not limited to sparkSession initialization.

[0058] For example, in this embodiment, the Zeppelin-based task scheduling device divides the Yarn resource pool to separate queues specifically for executing scheduled tasks, with parallel execution between resource pools and serial execution within a resource pool.

[0059] The Zeppelin-based task scheduling device proposed in the embodiment of the present invention adopts the Zeppelin-based task scheduling module proposed in Example 1, realizing that one task scheduling module simultaneously schedules multiple Spark tasks to be submitted to the YARN resource pool, completing the parallel execution of multiple tasks and greatly improving the overall execution efficiency.

[0060] Example 3

[0061] like Figure 2 As shown, Figure 2 This is a principle block diagram of a Zeppelin-based task scheduling device provided in the third embodiment of the present invention.

[0062] The Zeppelin-based task scheduling device in this embodiment adopts the Zeppelin-based task scheduling module provided in the above-mentioned embodiment 1, specifically including the Azkaban scheduling platform, the task scheduling module, the Zeppelin platform, the Spark interpreter, and the Yarn resource pool:

[0063] The azkaban scheduling platform is used to schedule the spark tasks defined on the zeppelin platform page;

[0064] The task scheduling module is used to submit tasks based on event triggering, pre-define n users who submit tasks, complete parallel submission of tasks between multiple users, and serial submission of tasks within a single user;

[0065] The zeppelin platform is used to edit task content;

[0066] The Spark interpreter is used to initialize the task execution environment and submit the tasks submitted by the client to the YARN platform;

[0067] The YARN resource pool is used for the actual execution of tasks.

[0068] Exemplarily, in this embodiment, the azkaban scheduling platform is also used for task timing scheduling, and the execution order of tasks can be flexibly set through configuration.

[0069] For example, the editing task content in the Zeppelin platform described in this embodiment includes but is not limited to writing Spark code in Scala and Python formats using web pages.

[0070] Illustratively, in the spark interpreter described in this embodiment, initializing the task execution environment includes but is not limited to sparkSession initialization.

[0071] For example, in this embodiment, the Zeppelin-based task scheduling device divides the Yarn resource pool to separate queues specifically for executing scheduled tasks, with parallel execution between resource pools and serial execution within a resource pool.

[0072] For example, in this embodiment, the Zeppelin platform is specifically used to, after receiving the task information submitted by the task scheduling module, first check whether the user contained in the task information has the authority to execute the task. Only after the authority verification is passed will it establish a connection with the spark interpreter process and submit the task to the spark interpreter process.

[0073] Exemplarily, in this embodiment, the spark interpreter is specifically used to submit the task to the yarn specified queue of the Yarn resource pool according to the yarn queue information contained in the task after receiving the task submitted by the zeppelin platform, and execute the spark task.

[0074] For example, in the Zeppelin-based task scheduling device of this embodiment, the user name is bound to the name of the yarn queue. For example, the yarn queue submitted by the task00 user is named task00 pool by default, which simplifies the logic.

[0075] The Zeppelin-based task scheduling device proposed in the embodiment of the present invention adopts the Zeppelin-based task scheduling module proposed in Example 1, realizing that one task scheduling module simultaneously schedules multiple Spark tasks to be submitted to the YARN resource pool, completing the parallel execution of multiple tasks and greatly improving the overall execution efficiency.

[0076] Compared with the existing technology, the technical solution of the embodiment of the present invention utilizes asynchrony and parallelism to accelerate the submission and execution of tasks, reduce the execution time of the overall task, and improve timeliness; the same task can be executed multiple times according to the changes in external parameters. For example, if we need to calculate the sum of certain data for 10 minutes, 20 minutes, 30 minutes, and 60 minutes, we can use time as a parameter, configure it in the scheduling file, and pass it to the task by passing parameters; by separating the overall task scheduling from the task execution, the execution time of the overall task can be intuitively viewed through the azkaban scheduling platform, which is convenient for monitoring and optimization.

[0077] Example 4

[0078] like Figure 3 As shown, Figure 3 This is a flow chart of the Zeppelin-based task scheduling method provided in Example 4 of the present invention.

[0079] This embodiment discloses a task scheduling method based on Zeppelin, which includes the following steps:

[0080] S301.azkaban scheduling platform schedules the spark tasks defined on the zeppelin platform page;

[0081] S302. The task scheduling module submits tasks to the zeppelin platform based on event triggering;

[0082] S303. After receiving the task information submitted by the task scheduling module, the Zeppelin platform establishes a connection with the Spark interpreter process and submits the task to the Spark interpreter process;

[0083] S304. After receiving the task submitted by the Zeppelin platform, the spark interpreter submits the task to the yarn specified queue of the Yarn resource pool according to the yarn queue information contained in the task, and executes the spark task.

[0084] For example, in this embodiment, the task scheduling module triggers task submission based on events, pre-defines n users who submit tasks, and completes parallel task submission between multiple users and serial task submission within a single user. For example, in this embodiment, the azkaban scheduling platform supports scheduled task scheduling, and flexibly sets the order of task execution through configuration.

[0085] Exemplarily, in this embodiment, editing task content in the Zeppelin platform includes, but is not limited to, writing Spark code in Scala or Python using a web page. Exemplarily, initializing the task execution environment in the Spark interpreter includes, but is not limited to, initializing a SparkSession. Exemplarily, in this embodiment, the Zeppelin-based task scheduling device partitions the Yarn resource pool, creating queues specifically for executing scheduled tasks, with parallel execution across resource pools and serial execution within a resource pool.

[0086] Exemplarily, in this embodiment, after receiving the task information submitted by the task scheduling module, the Zeppelin platform first checks whether the user contained in the task information has the authority to execute the task. Only after the authority verification is passed will it establish a connection with the Spark interpreter process and submit the task to the Spark interpreter process. Exemplarily, in this embodiment, after receiving the task submitted by the Zeppelin platform, the Spark interpreter submits the task to the YARN designated queue of the YARN resource pool according to the YARN queue information contained in the task, and executes the Spark task. Exemplarily, in this embodiment, the user's name is bound to the name of the YARN queue. For example, the default YARN queue submitted by the task00 user is named task00pool, which simplifies the logic.

[0087] The Zeppelin-based task scheduling method proposed in the embodiment of the present invention adopts the Zeppelin-based task scheduling module proposed in Example 1, realizing that a single task scheduling module simultaneously schedules multiple Spark tasks to be submitted to the YARN resource pool, completing the parallel execution of multiple tasks and significantly improving overall execution efficiency. Compared with the prior art, the technical solution of the embodiment of the present invention utilizes asynchrony and parallelism to accelerate the submission and execution of tasks, reducing the execution time of the overall task and improving timeliness. The same task can be executed multiple times based on changes in external parameters, which are passed to tasks by passing parameters. By separating overall task scheduling from task execution, the execution time of the overall task can be intuitively viewed through the Azkaban scheduling platform, facilitating monitoring and optimization.

[0088] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A task scheduling device based on Zeppelin, characterized in that: The device specifically includes: Azkaban scheduling platform, used to schedule spark tasks defined on the Zeppelin platform page; The task scheduling module is used to submit tasks based on event triggers. It predefines n users who submit tasks, completes parallel submission of tasks between multiple users, and serial submission of tasks within a single user; Zeppelin platform, used to edit task content; Spark interpreter, used to initialize the task execution environment and submit tasks submitted by the client to the YARN platform; Yarn resource pool, used for the actual running of tasks; The task scheduling module is a Zeppelin-based task scheduling module, including: The task ID query unit is used to query the paragraph ID (i.e., task ID) contained in the note ID on the Zeppelin platform according to the execution note ID when the task starts to be scheduled; An event encapsulation unit, configured to encapsulate the task ID and configuration information into an event after obtaining the task ID; The event sending unit is used to send the event encapsulated by the event encapsulation unit to the event bus; An event monitoring unit is used to monitor events in the event bus through a listener, and when a task arrives at the event bus, trigger several listeners to obtain task information; A user authentication unit, configured to obtain information about a user who is performing a task from a user list based on the task information obtained by the event monitoring unit; The task submission unit is used to merge the user information and the task information and submit them to the Zeppelin platform.

2. The Zeppelin-based task scheduling device according to claim 1, wherein the event sending unit is specifically configured to asynchronously send the event encapsulated by the event encapsulation unit to the event bus.

3. The task scheduling device based on Zeppelin according to claim 1, characterized in that: The azkaban scheduling platform is also used for task timing scheduling, and the execution order of tasks can be flexibly set through configuration.

4. The task scheduling device based on Zeppelin according to claim 1, characterized in that: The editing tasks in the Zeppelin platform include but are not limited to writing Spark code in Scala and Python formats using web pages.

5. The task scheduling device based on Zeppelin according to claim 1, characterized in that: By dividing the YARN resource pool, queues are separated to execute scheduled tasks, with parallel execution between resource pools and serial execution within a resource pool.

6. The zeppelin-based task scheduling device according to claim 3, characterized in that: The Zeppelin platform is specifically used to first check whether the user contained in the task information has the authority to execute the task after receiving the task information submitted by the task scheduling module. Only after the authority verification is passed will it establish a connection with the spark interpreter process and submit the task to the spark interpreter process.

7. The zeppelin-based task scheduling device according to claim 6, characterized in that: The spark interpreter is specifically used to submit the task to the yarn specified queue of the Yarn resource pool according to the yarn queue information contained in the task after receiving the task submitted by the zeppelin platform, and execute the spark task.

8. The zeppelin-based task scheduling device according to claim 7, characterized in that: The Zeppelin-based task scheduling device binds the user's name to the name of the YARN queue.

9. A Zeppelin-based task scheduling method using the Zeppelin-based task scheduling device according to claim 8, characterized in that: The method comprises the following steps: The azkaban scheduling platform schedules the spark tasks defined on the zeppelin platform page; The task scheduling module submits tasks to the Zeppelin platform based on event triggering; After receiving the task information submitted by the task scheduling module, the Zeppelin platform establishes a connection with the Spark interpreter process and submits the task to the Spark interpreter process; After receiving the task submitted by the Zeppelin platform, the Spark interpreter submits the task to the YARN specified queue of the YARN resource pool according to the YARN queue information contained in the task, and executes the Spark task.

Images