Method and platform for real-time monitoring of spark engine output progress based on distributed scheduling

Abstract

The application discloses a kind of based on distributed scheduling's real-time monitoring Spark engine output progress method and platform, belong to information technology field, including: task creation, task execution and progress monitoring and task state and progress monitoring display.The application integrates distributed scheduling and Spark computing engine, adopts MasterServer, WorkerServer and SeaTunnel engine to carry out task execution and progress monitoring, obtains relevant information of output progress from Spark computing engine end in real time and sends back to scheduling system, so that user can see the output progress condition of task in real time from the page of scheduling monitoring conveniently while using reliable and stable performance scheduling ability and high-performance scalable computing ability, improve data reconciliation efficiency, reduce data operation cost.

Description

Technical Field

[0001] This invention relates to the fields of information technology and big data technology, and in particular to a method for real-time monitoring of the output progress of the Spark engine based on distributed scheduling. Background Technology

[0002] The computing engine and the task scheduling system are the two most core components of building a data platform.

[0003] Spark is an open-source, memory-based distributed computing engine that supports ad-hoc SQL queries, real-time stream computing, machine learning, graph computing, and more. Spark can be deployed on top of the YARN resource manager, providing a one-stop big data solution and playing an important role in all aspects of data collection, cleaning, processing, and data mining.

[0004] The scheduling system provides functions such as scheduled scheduling, task-dependent scheduling, manual scheduling, scheduling monitoring, and fault tolerance recovery for the execution of various data tasks in the data platform.

[0005] Currently, the main scheduling systems used for scheduling Spark tasks include Oozie, Airflow, DolphinScheduler, and Azkaban. These systems can monitor the start time, end time, success, and failure of Spark tasks. However, for tasks that output data to external storage, the output progress cannot be monitored in real time. This is especially true for long-running tasks processing massive amounts of data, where users struggle to take timely and effective measures to prevent anomalies or restore business operations. Furthermore, if a task fails, the system cannot display the amount of successful or failed output, making it difficult for users to understand the task's status and perform data reconciliation. Summary of the Invention

[0006] The purpose of this invention is to address the above-mentioned problems by providing a method for real-time monitoring of Spark engine output progress based on distributed scheduling. This method obtains relevant information on output progress from the computing engine in real time and sends it back to the scheduling system, making it easy to see the task output progress in real time from the scheduling monitoring page, thereby improving data reconciliation efficiency and reducing data operation and maintenance costs.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A method for real-time monitoring of Spark engine output progress based on distributed scheduling includes the following:

[0009] Step S10, Task Creation: Create a Spark task, set the input data source, output data source, data processing, computation rules, and resources used; set scheduling rules, select the task to be scheduled, and set the rules for timed triggering or the rules that the task depends on.

[0010] Step S20, task execution and progress monitoring, includes the following specific processing flow:

[0011] Step S21: Distribute tasks: The MasterServer periodically polls the task scheduling information to obtain tasks that meet the trigger execution conditions, and then distributes the tasks to the WorkerServer.

[0012] Step S22, Execute the task: The WorkerServer receives the task distributed by the MasterServer, then executes the shell to start the SeaTunnel engine, and listens for the exit code of the shell to check whether the task has started successfully. Then it sends the task start response information to the MasterServer to update the task running status.

[0013] Step S23: Submit Spark task: After SeaTunnel parses the configuration file specified by WorkerServer, it submits the application to YARN and then generates the data processing chain of the Spark application.

[0014] Step S24: Monitor task progress: YARN counts and caches the total number of outputs to be completed on the driver side, then obtains and accumulates the total number of outputs of each computing node in real time, and then sends task progress information to the MasterServer to update the task progress through the progress reporting thread.

[0015] Step S30, Task Status and Progress Monitoring Display: Obtain and display the task status and progress monitoring information.

[0016] The specific content of step S21 is as follows: After the MasterServer service starts, it periodically polls the task scheduling information through JDBC to obtain tasks that meet the trigger execution conditions. Then, it obtains the available Worker address and port information through Zookeeper, and then uses Netty to distribute the tasks to the WorkerServer.

[0017] Step S22 is as follows: The WorkerServer receives the task distributed by the MasterServer, generates a temporary execution directory, a SeaTunnel configuration file, and a shell script to start SeaTunnel based on the received task information, and then executes the shell script to start the SeaTunnel engine. After SeaTunnel starts, the WorkerServer uses Netty to send a response message to the MasterServer. After receiving the response message, the MasterServer connects to the database via JDBC and updates the task status to running. When the task is completed or fails, the WorkerServer uses Netty to send the task result information to the MasterServer. After receiving the result information, the MasterServer connects to the database via JDBC and updates the task status to completed or failed.

[0018] Step S23 is as follows: After SeaTunnel parses the configuration file specified by the WorkerServer, it submits the application to YARN. After YARN allocates resources, it initializes the Driver program, initializes SparkSession according to the resource configuration information submitted by SeaTunnel, obtains the input data source configuration information, creates a DataFrameReader using SparkSession, obtains the Dataset from the DataFrameReader, and configures and defines the steps for Dataset transformation and calculation according to the data processing and calculation rules. Then, it generates a DataFrameWriter from the Dataset after the steps are defined according to the output data source configuration, forming one or more Spark data processing links of "read -> transformation and calculation -> output". Each link is ultimately triggered to execute through the save interface of DataFrameWriter. The computation tasks are distributed to each node of YARN to initialize executors for distributed execution.

[0019] Step S24 is as follows: After YARN initializes SparkSession, it starts a progress reporting thread. Every heartbeat interval, this thread uses Netty to send the task progress information to the MasterServer, which persists it to the database. The total number of outputs to be completed is obtained by calling the count interface from the "Dataset after completing the defined steps". After obtaining the total number of outputs, the Dataset and the statistical value are cached, and the progress reporting thread retrieves the value from the cache. The total number of outputs to be completed needs to be collected from the output values ​​of each executor and accumulated, and then sent back to the progress reporting thread. After initializing SparkSession, a LongAccumulator is created and set in the write operator access field of DataFrameWriter. After each output submission by the write operator, the LongAccumulator accumulates the output value. Then the progress reporting thread can use the value interface of LongAccumulator to get the current total number of outputs in real time.

[0020] By adopting the above technical solution, the present invention has the following beneficial effects:

[0021] This invention integrates distributed scheduling with the Spark computing engine, employing MasterServer, WorkerServer, and SeaTunnel engines for task execution and progress monitoring. It retrieves output progress information from the Spark computing engine in real time and sends it back to the scheduling system, allowing users to conveniently view the task output progress in real time from the scheduling monitoring page while utilizing reliable and stable scheduling capabilities and high-performance scalable computing power. This improves data reconciliation efficiency and reduces data operation and maintenance costs. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method for monitoring the output progress of the Spark engine according to the present invention.

[0023] Figure 2 This is a schematic diagram of the platform architecture and services for monitoring the output progress of the Spark engine according to the present invention.

[0024] Figure 3 This is a schematic diagram of the Spark data processing chain of the present invention. Detailed Implementation

[0025] The specific implementation of the invention will be further described below with reference to the accompanying drawings.

[0026] See Figure 1As described above, the method for monitoring the output progress of the Spark engine includes the processing program in steps S10-S24, which is stored in memory and configured to be executed by a processor. The relevant memory and processor utilize existing technologies and will not be elaborated further here. Based on this method for monitoring the output progress of the Spark engine, a platform for real-time monitoring of the Spark engine's output progress based on distributed scheduling can be constructed. The following will provide a detailed explanation of the method and platform for monitoring the output progress of the Spark engine.

[0027] like Figure 1 and Figure 2 As shown, the platform for monitoring the output progress of the Spark engine includes a UI, APIServer, MasterServer, WorkerServer, and the SeaTunnel engine. The interaction and business logic design follows the MVC layered theory and adopts a front-end / back-end separation model. The task scheduling and execution end uses a decentralized Master / Worker cluster model, with the Worker side generating and submitting Spark tasks through SeaTunnel. The entire platform mainly includes a UI service, APIServer, MasterServer, WorkerServer, and the SeaTunnel engine. The UI and APIServer are used for task creation and monitoring / displaying task status and progress. The MasterServer, WorkerServer, and SeaTunnel engine are used for task execution and progress monitoring. The main functions of each service are described below.

[0028] UI service: The system's front-end page, providing various visual operation interfaces for the system, including task creation, task monitoring, pausing, resuming, and rerunning.

[0029] ApiServer: The API interface layer, primarily responsible for handling requests from the front-end UI layer. This service provides a unified RESTful API for external request services. The interface includes workflow creation, definition, querying, modification, publishing, decommissioning, manual start, stopping, pausing, resuming, and starting execution from a given node, etc.

[0030] MasterServer: The MasterServer adopts a distributed, decentralized design. It is primarily responsible for DAG task splitting, task submission monitoring, and simultaneously monitoring the health status of other MasterServers and WorkerServers. Upon startup, the MasterServer registers an ephemeral node with ZooKeeper, using changes to these ephemeral nodes for fault tolerance. The MasterServer provides its monitoring service based on Netty.

[0031] WorkerServer: WorkerServer also adopts a distributed, decentralized design, supports custom task plugins, and is mainly responsible for task execution and providing logging services. When the WorkerServer service starts, it registers a temporary node with ZooKeeper and maintains a heartbeat.

[0032] SeaTunnel Engine: SeaTunnel can translate configuration files into Spark tasks and submit the generated tasks and dependencies to the Yarn cluster for distributed computing.

[0033] As described above, the processing flow between the various services is as follows:

[0034] Step S10, Task Creation: Spark task creation is accomplished by two services: UI Service and ApiServer. These services communicate via HTTP. UI acts as the request client, and ApiServer provides a RESTful API as the server. The process is as follows:

[0035] a) After configuring external data source information on the UI, the user calls the ApiServer interface to save the data source configuration. ApiServer saves the configuration information to the DB via JDBC.

[0036] b) The user creates a task on the UI. The UI calls the ApiServer interface to obtain available data sources. The ApiServer reads the configuration information through JDBC and returns it to the UI. The user selects an available data source on the UI, configures the input, output, processing logic, and resources used for the data processing task, and then submits it. The UI calls the ApiServer interface. After receiving the request, the ApiServer converts the configuration into a SeaTunnel task configuration file and saves it.

[0037] c) Users configure scheduling settings on the UI. After selecting the task to be configured, they set the rules for timed triggering or the rules that the task depends on, and then submit. The UI calls the ApiServer interface, and after receiving the request, the ApiServer saves the scheduling information to the database via JDBC.

[0038] Step S2, Task Execution and Progress Monitoring: The execution and output progress monitoring of Spark tasks are completed collaboratively by three parts: MasterServer, WorkerServer, and SeaTunnel. The process is as follows:

[0039] a) MasterServer obtains and distributes tasks: After the MasterServer service starts, the background thread polls the task scheduling information through JDBC to obtain tasks that meet the trigger execution conditions, then obtains the available Worker addresses and port information through Zookeeper, and then uses Netty to distribute the tasks to the WorkerServer.

[0040] b) WorkerServer receives and executes tasks: After receiving a task distributed by the MasterServer, the WorkerServer generates a temporary execution directory, a SeaTunnel configuration file, and a shell script to start SeaTunnel based on the received task information. It then executes the shell script to start the SeaTunnel engine. After startup, the WorkerServer uses Netty to send a startup response message to the MasterServer that distributed the task. Upon receiving the response message, the MasterServer updates the task status (startup status) to "running" via JDBC connection to the database. When a task completes or fails, the WorkerServer again uses Netty to send task status information to the MasterServer that distributed the task. Upon receiving the status information, the MasterServer updates the task status via JDBC connection to the database.

[0041] c) SeaTunnel generates and submits Spark tasks: After the WorkerServer starts SeaTunnel via shell, SeaTunnel parses the configuration file specified by the WorkerServer and submits the application to YARN. YARN allocates resources and initializes the Driver program. Based on the resource configuration information submitted by SeaTunnel, it initializes the SparkSession, then obtains the input data source configuration information and uses the SparkSession to create a DataFrameReader. Next, it retrieves the Dataset from the DataFrameReader and configures the transformation and computation steps of the Dataset according to the data processing and computation rules. Finally, it generates a DataFrameWriter from the Dataset after the definition steps (the Dataset after the definition steps are completed) according to the output data source configuration, forming one or more Spark data processing chains of "read -> transformation and computation -> output" (see...). Figure 3 Each link is ultimately triggered for execution through the save interface of DataFrameWriter, and the computation tasks are distributed to each node of YARN to initialize the executor for distributed execution.

[0042] d) Task Progress and Status Information Acquisition and Storage: Acquiring task progress information requires two metrics: the total number of tasks yet to be output and the total number of tasks already output. In process c), after initializing SparkSession, a background thread is started to send the metric data back to the scheduling service in real time. Every heartbeat interval, this thread uses Netty to send the metric data to the MasterServer, which then persists it to the database. The total number of tasks yet to be output is obtained from the "Dataset after the defined transformation and calculation steps" in process c) by calling the count interface. After obtaining the value, the Dataset and the statistical value are cached to avoid duplicate calculations. The progress reporting thread retrieves the value from the cache. However, the total number of tasks already output cannot use pre-statistics and caching methods because the latest value needs to be obtained in real time every heartbeat interval. In YARN, Spark tasks distribute computational tasks to executors on the same or different nodes. To obtain the current total output, the output values ​​from each executor need to be collected, accumulated, and then sent back to the driver's reporting thread. This is achieved using Spark's LongAccumulator: In process c), after initializing the SparkSession, a LongAccumulator is created and set in the write operator access field of the DataFrameWriter. Each time the write operator completes an output commit, the LongAccumulator increments the corresponding output value. The progress reporting thread can then use the LongAccumulator's value interface to obtain the current total output in real time. Obtaining the task status is relatively simple. Since the task is started by the WorkerServer using a shell, after completion, the WorkerServer uses the shell's exit status code to determine success or failure, then uses Netty to send the status to the MasterServer. Finally, the MasterServer persists the task status to the database via JDBC.

[0043] Step S30, Task Status and Progress Monitoring and Display: The display of Spark task status and progress information is completed by two services: UI service and ApiServer. Users select the relevant task to view through the UI service. The UI calls the ApiServer interface to obtain task status and progress information. The ApiServer obtains the DB-related status and progress information updated in real time by the MasterServer through JDBC and returns it to the UI, which is finally displayed to the user.

[0044] As described above, the interaction steps for users accessing the platform's web page through a browser in this Spark engine output monitoring platform are as follows:

[0045] 1) Create a data source: Users enter the data source management page, click the Create Data Source button, and then set relevant parameters such as IP address and port in the form. After testing the connection and confirming its availability, the user saves the settings.

[0046] 2) Create a task: Users enter the task management page, click the create task button, set the input data source, output data source, data processing and calculation rules, resources used, etc. on the page, and then save.

[0047] 3) Set up scheduling: Users enter the task management page, select the task to be scheduled, click the scheduling settings button, set the trigger rules in the form, and then save.

[0048] 4) View scheduling status: Users can access the scheduling management page to view the scheduling status, execution history and other information of tasks.

[0049] 5) View task execution status: Users can access the task instance management page to view the running status and output progress related metrics of Spark tasks.

[0050] 6) Perform data reconciliation: Users should check the total number of data to be written (total number of data to be output) against the total number of data already written (total number of data already output) in step 5) according to their specific business situation. If there is a discrepancy, appropriate action should be taken.

[0051] As mentioned above, this invention uses two key indicators, the number of completed tasks and the total number of tasks, as basic indicators for the progress monitoring method. The following will further analyze and explain the advantages of this invention's method and platform for monitoring the output progress of the Spark engine from the perspectives of technical details and application scenarios.

[0052] The method and platform for monitoring the output progress of the Spark engine in this invention obtains metrics accurate to the task of the external data source for data output. The obtained metrics are "total number of data rows output" and "total number of data rows to be output". The key steps for obtaining the metrics are: after initializing SparkSession, create LongAccumulator and set it in the output operator access domain of DataFrameWriter. First, use the count interface to count the "total number of data rows to be output", and then execute the output task. After each output operator completes an output submission, LongAccumulator accumulates the corresponding output value. Then, the driver obtains the "total number of data rows output" through the value interface of LongAccumulator.

[0053] As a comparative example, monitoring using the Spark REST API to obtain the number of completed tasks and the total number of tasks is only suitable for Spark applications with a single job. This is because the Spark engine schedules jobs sequentially; it only splits and schedules the next job after all tasks in the current job have been completed. In other words, it cannot obtain the total number of tasks at the start of the job process, leading to inaccurate progress displays when there are multiple jobs. Furthermore, this "task-specific monitoring: a list of all tasks in a stage" is specific to the line of code and requires a deep understanding of Spark to comprehend; it necessitates professional developers to interpret it. If applied to a data reconciliation scenario for business personnel (non-professional developers), they would only know how many tasks were completed, but not how many data entries each task output, or the total number of data entries required, making data reconciliation impossible.

[0054] In contrast, the method for monitoring the output progress of the Spark engine according to the present invention does not have this limitation. This method for monitoring the output progress of the Spark engine can be applied to both single jobs and multiple jobs, and the metrics can be accurate to the number of data entries, providing fine-grained monitoring.

[0055] This invention is applicable to tasks that monitor the output of data to external storage in real time. Users can precisely understand how many data entries are pending output and how many have already been output. If a task encounters an anomaly, users can identify the number of successful and failed outputs during data reconciliation, allowing for accurate data replenishment. Users only need to focus on the data processing and do not require any Spark-related knowledge. The monitoring granularity of this invention is precise down to the number of data output entries, making it suitable for data professionals performing data reconciliation.

[0056] The above provides a detailed explanation of the methods and platforms for monitoring the output progress of the Spark engine, combined with... Figure 2 and Figure 3 The following will take the data entry and reconciliation process of a certain unit's data platform as an example to further illustrate the specific working process of the present invention.

[0057] The data entry and reconciliation process of a certain organization's data platform can be divided into the following steps:

[0058] Step 1. On the platform's data source management page, the user adds the business data source information that needs to be included in the lake. After submission, the UI calls the ApiServer through the RestApi interface, and the ApiServer persists the data source information to the DB.

[0059] Step 2. Users create tasks on the platform's task management page, setting information such as the data source, entry rules, and resources to be used for the business data to be entered into the lake. After submission, the UI calls the ApiServer through the RestApi interface, and the ApiServer persists the task information to the database via JDBC.

[0060] Step 3. On the platform's task management page, the user selects the task to be scheduled, sets the scheduling rules, and submits the settings. After submission, the UI calls the ApiServer via the RestApi interface, and the ApiServer persists the task's scheduling rule information to the database via JDBC.

[0061] Step 4. The MasterServer periodically polls the database via JDBC. When a user-created task meets the execution conditions, the MasterServer distributes the task to the WorkerService.

[0062] Step 5. After receiving the user-created task, the WorkerService starts SeaTunnel via shell and listens for the exit code of the shell to see if the task has started successfully. Then, it uses Netty to send the task status information (running, completed, or failed) to the MasterServer for persistence.

[0063] Step 6. After SeaTunnel starts, it submits the user's data ingestion task to YARN. Then, it generates the data processing chain of the Spark application according to the configuration, counts and caches the total number of data to be written on the driver side, and then obtains and accumulates the total number of data written on each computing node in real time. Finally, it uses Netty to send the progress information to the MasterServer for persistence.

[0064] Step 7. The user enters the platform's scheduling management page. The UI calls the ApiServer to obtain the latest task status data from the database. The user can then view the task scheduling status, running history, and other information, and audit or confirm whether the scheduling is normal. If any abnormalities are found, they can be dealt with promptly.

[0065] Step 8. The user enters the task instance management page of the platform. The UI calls the ApiServer to obtain the latest task progress indicator data from the DB. The user can then view the running status of the Spark task, output progress related indicators, confirm whether there are any abnormalities in the output progress, and verify the data according to the specific situation of the data entering the lake. If there is a deviation in the value, appropriate action will be taken.

[0066] As described above, the method and platform for monitoring the output progress of the Spark engine of this invention provide a method and platform that integrates distributed scheduling and the Spark computing engine. It generates SeaTunnel task configuration through page configuration, and adds an indicator acquisition operator to the Spark data processing link generated by SeaTunnel. This enables real-time acquisition of relevant information on the output progress from the computing engine and sending it back to the scheduling system. This allows users to conveniently see the output progress of tasks in real time from the scheduling monitoring page while utilizing reliable and stable scheduling capabilities and high-performance scalable computing capabilities, thereby improving data reconciliation efficiency and reducing data operation and maintenance costs.

[0067] It should be noted that the examples of the above embodiments can preferably be combined with one or more of each other according to actual needs, and the accompanying drawings of multiple examples adopt a set of combined technical features, which will not be described in detail here.

[0068] The above description is a detailed explanation and illustration of the preferred embodiments of the present invention. However, these descriptions are not intended to limit the scope of protection claimed by the present invention. All equivalent changes or modifications made under the technical teachings of the present invention should fall within the patent protection scope covered by the present invention.

Claims

1. A method for real-time monitoring of Spark engine output progress based on distributed scheduling, characterized in that, Includes the following: Step S10, Task Creation: Create a Spark task, set the input data source, output data source, data processing, calculation rules, and resources used; Set scheduling rules: Select the tasks to be scheduled, and set rules for timed triggering or task dependencies; Step S20, task execution and progress monitoring, includes the following specific processing flow: Step S21: Distribute tasks: The MasterServer periodically polls the task scheduling information to obtain tasks that meet the trigger execution conditions, and then distributes the tasks to the WorkerServer. Step S22, Execute the task: The WorkerServer receives the task distributed by the MasterServer, then executes the shell to start the SeaTunnel engine, and listens for the exit code of the shell to check whether the task has started successfully. Then it sends the task start response information to the MasterServer to update the task running status. Step S23: Submit Spark task: After SeaTunnel parses the configuration file specified by WorkerServer, it submits the application to YARN and then generates the data processing chain of the Spark application. Step S24: Monitor task progress: YARN counts and caches the total number of outputs to be completed on the driver side, then obtains and accumulates the total number of outputs of each computing node in real time, and then the progress reporting thread sends task progress information to the MasterServer to update the task progress. Step S30, Task Status and Progress Monitoring Display: Obtain and display the task status and progress monitoring information.

2. The method for real-time monitoring of Spark engine output progress based on distributed scheduling according to claim 1, characterized in that: Step S21 is as follows: After the MasterServer service starts, it periodically polls the task scheduling information through JDBC to obtain tasks that meet the trigger execution conditions. Then, it obtains the available Worker addresses and port information through Zookeeper, and then uses Netty to distribute the tasks to the WorkerServer.

3. The method for real-time monitoring of Spark engine output progress based on distributed scheduling according to claim 1, characterized in that: Step S22 is as follows: The WorkerServer receives the task distributed by the MasterServer, generates a temporary execution directory, a SeaTunnel configuration file, and a shell script to start SeaTunnel based on the received task information, and then executes the shell script to start the SeaTunnel engine. After SeaTunnel starts, the WorkerServer uses Netty to send a response message to the MasterServer. After receiving the response message, the MasterServer connects to the database via JDBC and updates the task status to running. When the task is completed or fails, the WorkerServer uses Netty to send the task result information to the MasterServer. After receiving the result information, the MasterServer connects to the database via JDBC and updates the task status to completed or failed.

4. The method for real-time monitoring of Spark engine output progress based on distributed scheduling according to claim 1, characterized in that: Step S23 is as follows: After SeaTunnel parses the configuration file specified by the WorkerServer, it submits the application to YARN. After YARN allocates resources, it initializes the Driver program, initializes SparkSession according to the resource configuration information submitted by SeaTunnel, obtains the input data source configuration information, uses SparkSession to create a DataFrameReader, obtains the Dataset from the DataFrameReader, and configures and defines the steps for Dataset transformation and calculation according to the data processing and calculation rules. Then, it generates a DataFrameWriter from the Dataset after the steps are defined according to the output data source configuration, forming one or more Spark data processing links of "read -> transformation and calculation -> output". Each link is ultimately triggered to execute through the save interface of DataFrameWriter. The computation tasks are distributed to each node of YARN to initialize executors for distributed execution.

5. The method for real-time monitoring of Spark engine output progress based on distributed scheduling according to claim 1, characterized in that: Step S24 is as follows: After YARN initializes SparkSession, it starts a progress reporting thread. Every heartbeat interval, this thread uses Netty to send the task progress information to the MasterServer, which persists it to the database. The total number of outputs to be completed is obtained by calling the count interface from the "Dataset after completing the defined steps". After obtaining the total number of outputs, the Dataset and the statistical value are cached, and the progress reporting thread retrieves the value from the cache. The total number of outputs to be completed needs to be collected from the output values ​​of each executor and accumulated, and then sent back to the progress reporting thread. After initializing SparkSession, a LongAccumulator is created and set in the write operator access field of DataFrameWriter. After each output submission by the write operator, the LongAccumulator accumulates the output value. Then the progress reporting thread can use the value interface of LongAccumulator to get the current total number of outputs in real time.

6. A platform for real-time monitoring of Spark engine output progress based on distributed scheduling, characterized by: This includes the UI, APIServer, MasterServer, WorkerServer, and SeaTunnel engine, used for the following processing flow: Step S10, the UI and API Server are used to create tasks: create Spark tasks, set input data sources, output data sources, data processing, calculation rules and resources used; Set scheduling rules: Select the tasks to be scheduled, and set rules for timed triggering or task dependencies; Step S20, MasterServer, WorkerServer, and the SeaTunnel engine are used for task execution and progress monitoring. The specific processing flow includes: Step S21: Distribute tasks: The MasterServer periodically polls the task scheduling information to obtain tasks that meet the trigger execution conditions, and then distributes the tasks to the WorkerServer. Step S22, Execute the task: The WorkerServer receives the task distributed by the MasterServer, then executes the shell to start the SeaTunnel engine, and listens for the exit code of the shell to check whether the task has started successfully. Then it sends the task start response information to the MasterServer to update the task running status. Step S23: Submit Spark task: After SeaTunnel parses the configuration file specified by WorkerServer, it submits the application to YARN and then generates the data processing chain of the Spark application. Step S24: Monitor task progress: YARN counts and caches the total number of outputs to be completed on the driver side, then obtains and accumulates the total number of outputs of each computing node in real time, and then sends task progress information to the MasterServer to update the task progress through the progress reporting thread. Step S30: The UI and ApiServer are used to monitor and display task status and progress: The UI calls the ApiServer interface to obtain task status and progress monitoring information for display.

7. The platform for real-time monitoring of Spark engine output progress based on distributed scheduling according to claim 6, characterized in that: Step S21 is as follows: After the MasterServer service starts, it periodically polls the task scheduling information through JDBC to obtain tasks that meet the trigger execution conditions. Then, it obtains the available Worker addresses and port information through Zookeeper, and then uses Netty to distribute the tasks to the WorkerServer.

8. The platform for real-time monitoring of Spark engine output progress based on distributed scheduling according to claim 6, characterized in that: Step S22 is as follows: The WorkerServer receives the task distributed by the MasterServer, generates a temporary execution directory, a SeaTunnel configuration file, and a shell script to start SeaTunnel based on the received task information, and then executes the shell script to start the SeaTunnel engine. After SeaTunnel starts, the WorkerServer uses Netty to send a response message to the MasterServer. After receiving the response message, the MasterServer connects to the database via JDBC and updates the task status to running. When the task is completed or fails, the WorkerServer uses Netty to send the task result information to the MasterServer. After receiving the result information, the MasterServer connects to the database via JDBC and updates the task status to completed or failed.

9. The platform for real-time monitoring of Spark engine output progress based on distributed scheduling according to claim 6, characterized in that: Step S23 is as follows: After SeaTunnel parses the configuration file specified by the WorkerServer, it submits the application to YARN. After YARN allocates resources, it initializes the Driver program, initializes SparkSession according to the resource configuration information submitted by SeaTunnel, obtains the input data source configuration information, uses SparkSession to create a DataFrameReader, obtains the Dataset from the DataFrameReader, and configures and defines the steps for Dataset transformation and calculation according to the data processing and calculation rules. Then, it generates a DataFrameWriter from the Dataset after the steps are defined according to the output data source configuration, forming one or more Spark data processing links of "read -> transformation and calculation -> output". Each link is ultimately triggered to execute through the save interface of DataFrameWriter. The computation tasks are distributed to each node of YARN to initialize executors for distributed execution.

10. The platform for real-time monitoring of Spark engine output progress based on distributed scheduling according to claim 6, characterized in that: Step S24 is as follows: After YARN initializes SparkSession, it starts a progress reporting thread. Every heartbeat interval, this thread uses Netty to send the task progress information to the MasterServer, which persists it to the database. The total number of outputs to be completed is obtained by calling the count interface from the "Dataset after completing the defined steps". After obtaining the total number of outputs, the Dataset and the statistical value are cached, and the progress reporting thread retrieves the value from the cache. The total number of outputs to be completed needs to be collected from the output values ​​of each executor and accumulated, and then sent back to the progress reporting thread. After initializing SparkSession, a LongAccumulator is created and set in the write operator access field of DataFrameWriter. After each output submission by the write operator, the LongAccumulator accumulates the output value. Then the progress reporting thread can use the value interface of LongAccumulator to get the current total number of outputs in real time.

Images