Notebook task scheduling method, device, and computer-readable storage medium

Abstract

This invention provides a Notebook task scheduling method, comprising: synchronizing a Notebook script to the Zeppelin management platform; configuring Notebook scheduling on the Zeppelin management platform and creating a scheduled task on Kubernetes; initiating execution of a Kubernetes Notebook task driver program using the scheduled task; and updating the Kubernetes Notebook task driver program Pod status. By integrating Kubernetes containerization technology with Zeppelin Notebook, this invention enables users to configure periodic task scheduling for Notebooks, achieving automated task scheduling. After completing data analysis or data modeling, users only need to configure the page to achieve automated task scheduling, improving work efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a Notebook task scheduling method, device, and computer-readable storage medium. Background Art

[0002] Most internet companies offer products similar to Notebooks, which use interactive methods for data analysis, data modeling, and data visualization. Apache Zeppelin is an open-source product for big data analysis, visualization, and modeling. Some companies use Apache Zeppelin for customized enterprise development. This tool allows data analysts and algorithm modelers to interactively analyze data from various data sources using different languages ​​and computing engines within a web-based notebook, and then visualize the analysis results. It supports a variety of common computing engines and language tools, including Flink, Spark, Python, JDBC, Markdown, Shell, and ES.

[0003] However, in practice, Zeppelin Notebook only supports manual interactive script execution. After conducting data analysis, especially data modeling exploration, users often need to schedule tasks, particularly periodic data analysis script execution and model training. Zeppelin cannot meet these scheduling requirements for Notebook scripts. Therefore, an automated Notebook task scheduling method is needed. Summary of the Invention

[0004] The present invention aims to provide a method, device, and computer-readable storage medium for scheduling Notebook tasks, aiming to solve the problem that Zeppelin Notebook cannot meet the requirements for automated scheduling of Notebook scripts.

[0005] In a first aspect, the present invention provides a Notebook task scheduling method, the method comprising:

[0006] S101. Synchronize the Notebook script to the Zeppelin management platform;

[0007] S102. Configure Notebook scheduling on the Zeppelin management platform and create a scheduled task on Kubernetes.

[0008] S103, the scheduled task starts the Kubernetes Notebook task driver program for execution;

[0009] S104: Update the Kubernetes Notebook task driver Pod status.

[0010] In a second aspect, the present invention provides a Notebook task scheduling device, comprising:

[0011] one or more processors;

[0012] Memory; and

[0013] One or more computer programs, the processor and the memory are connected via a bus, wherein the one or more computer programs are stored in the memory and are configured to be executed by the one or more processors, wherein the processor implements the steps of the Notebook task scheduling method described above when executing the computer program.

[0014] In a third aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program implements the steps of the Notebook task scheduling method described above when executed by a processor.

[0015] By integrating Kubernetes containerization technology and Zeppelin Notebook, this invention allows users to configure periodic task scheduling in Notebook and realize automated task scheduling. After completing data analysis or data modeling, users only need to configure the page to realize automated task scheduling, which improves work efficiency. At the same time, by persisting each instance task, users can easily view historical records and task status. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flowchart of the notebook task scheduling method provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and beneficial effects of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0018] In order to illustrate the technical solution of the present invention, specific embodiments are provided below.

[0019] Example 1:

[0020] like Figure 1 As shown, the first embodiment of the present invention provides a method for scheduling a Notebook task, including the following steps:

[0021] S101. Synchronize the Notebook script to the Zeppelin management platform. This includes the following steps:

[0022] Modify the Zeppelin software and add the function of calling the Zeppelin management platform interface in the creation logic of the Notebook script;

[0023] After the Zeppelin management platform script creation interface obtains script related information, it saves the newly created Note to a note database, such as a MySQL database. The script related information includes the script ID, and may also include one or more of the script name, script creator, and script creation time.

[0024] The Zeppelin management platform queries the Note library table in the note data and displays the Notebooks created in Zeppelin in a list.

[0025] S102. Configure Notebook scheduling on the Zeppelin management platform and create a scheduled task on Kubernetes. The specific steps are as follows:

[0026] Configure scheduled scripts on the Zeppelin management platform and save the scheduling information to the database;

[0027] Generate a Kubernetes cronjob on the Zeppelin management platform backend. This involves obtaining the note ID and note request link metadata to be scheduled, as well as the note creator information. It also builds the image required for the cronjob and the namespace information used to submit the cronjob, thereby generating a cronjob object on Kubernetes.

[0028] Call the Kubernetes API server to create a Kubernetes scheduled task. Kubernetes persists the scheduled task information. When the scheduled scheduling conditions are met, the scheduled task is triggered to start the Notebook driver to execute the schedule.

[0029] S103. Scheduled tasks start the Kubernetes Notebook task driver program. The Kubernetes Notebook task driver program package is provided as an image. You can write Python, Shell, Java, Go, or other scripts to implement the following steps:

[0030] When the scheduled time for the Kubernetes scheduled task is reached, the instance information of the corresponding note is inserted into the database and the instance status is set to ready, so that users can view the history and task status.

[0031] The Kubernetes Notebook task driver obtains the configuration information from the database and starts running the corresponding notebook by calling Zeppelin's restful interface;

[0032] After the Kubernetes scheduled task is driven to run, the Kubernetes Notebook task driver program synchronously queries the task execution status to update the execution result of the current instance.

[0033] S104. Update the Kubernetes Notebook task driver Pod status. During the execution of the Pod, abnormal exit at the Pod level may occur, resulting in inconsistent status. Therefore, it is necessary to capture the Pod status to update the Pod running status. The specific steps include:

[0034] Monitor the Pod status. The driver Pod will go through several life cycles, including creation, suspension, running, failure, end, and unknown. The monitoring logic is implemented to capture the Pod life cycle state migration events. Here, the main focus is to obtain the corresponding note original data information, including the corresponding note ID and instance ID information from the pod to update the instance information of the corresponding database.

[0035] Poll Pod status for updates. Set a scheduled Pod status query task to query the Kubernetes API server at regular intervals (e.g., 60 seconds) to obtain the status of all task driver Pods. Based on the Pod status, refresh the corresponding notebook instance's running status, start time, end time, container ID information, etc. Because system or network anomalies may cause status to not be captured properly, a polling strategy must be designed to avoid missing relevant time.

[0036] Example 2:

[0037] A second embodiment of the present invention provides a device for scheduling Notebook tasks, including:

[0038] one or more processors;

[0039] Memory; and

[0040] One or more computer programs, the processor and the memory are connected via a bus, wherein the one or more computer programs are stored in the memory and are configured to be executed by the one or more processors, wherein when the processor executes the computer program, the steps of the Notebook task scheduling method provided in Example 1 are implemented.

[0041] Example 3:

[0042] A third embodiment of the present invention provides a computer-readable storage medium storing a computer program. The computer-readable storage medium is characterized in that when the computer program is executed by a processor, the steps of the notebook task scheduling method provided in the first embodiment are implemented.

[0043] In this invention, by integrating Kubernetes containerization technology and Zeppelin Notebook, users can configure periodic task scheduling for Notebook and realize automated task scheduling. After completing data analysis or data modeling exploration, users only need to configure the page to realize automated task scheduling, which improves work efficiency. At the same time, by persisting each instance task, users can easily view historical records and task status.

[0044] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A Notebook task scheduling method, characterized in that: The method comprises: S101. Synchronize the Notebook script to the Zeppelin management platform; S102. Configure Notebook scheduling on the Zeppelin management platform and create a scheduled task on Kubernetes. S103, the scheduled task starts the Kubernetes Notebook task driver program for execution; S104. Update the Kubernetes Notebook task driver Pod status; Configuring Notebook scheduling on the management platform and creating scheduled tasks on Kubernetes include: Configure a scheduled schedule for the script on the Zeppelin management platform and save the scheduling information in a database; Generate a Kubernetes scheduled task object on the backend of the Zeppelin management platform; Call the Kubernetes API server to create a Kubernetes scheduled task, and Kubernetes persists the scheduled task information; The Kubernetes Notebook task driver software package is provided in a mirrored manner. When the Kubernetes Notebook task driver is executed, the following steps are implemented: When the scheduled time for the Kubernetes scheduled task is reached, the instance information of the corresponding note is inserted into the database and the instance status is set to ready; After obtaining the configuration information, the Kubernetes Notebook task driver calls the Zeppelin restful interface to start running the corresponding notebook; After the Kubernetes scheduled task is driven to run, the Kubernetes Notebook driver program synchronously queries the task execution status to update the execution result of the current instance.

2. The method according to claim 1, wherein Synchronizing the Notebook script to the Zeppelin management platform includes: Modify the Zeppelin software and add a function to call the Zeppelin management platform interface in the creation logic of the Notebook script; After the Zeppelin management platform script creation interface obtains the script related information, it saves the newly created note into the note database. The script related information includes the script ID. The Zeppelin management platform queries the note database table and displays the Notebooks created in Zeppelin in a list.

3. The method according to claim 2, wherein The script-related information further includes one or more of a script name, a script creator, or a script creation time.

4. The method according to claim 1, wherein Generating a Kubernetes scheduled task object on the backend of the Zeppelin management platform specifically includes: Get the note ID and note request link metadata to be scheduled, build the image required for the scheduled task and the namespace information used to submit the scheduled task, and jointly generate the scheduled task object on Kubernetes.

5. The method according to claim 4, wherein When obtaining the note ID that needs to be scheduled and the original information of the note request link, obtain the note creator information.

6. The method according to claim 1, wherein Updating the Kubernetes Notebook task driver Pod status includes listening to and polling the Pod status to update the Pod status.

7. A Notebook task scheduling device, comprising: one or more processors; Memory; as well as One or more computer programs, the processor and the memory are connected via a bus, wherein the one or more computer programs are stored in the memory and are configured to be executed by the one or more processors, wherein when the processor executes the computer program, the steps of the Notebook task scheduling method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the Notebook task scheduling method according to any one of claims 1 to 6 are implemented.

Images