Real-time data integration method based on DolphinScheduler

Abstract

The invention discloses a real-time data integration method based on DolphinScheduler, relates to the field of data processing, and solves the technical problems that Flink CDC is high in starting difficulty, and an Apache DolphinScheduler data processing mode is high in processing delay and insufficient in real-time performance. The method comprises the following steps of: 1, setting a real-time integration task interface, checking parameters in the real-time integration task interface, and creating a real-time data integration task according to the parameters of the real-time integration task interface; 2, starting the real-time data integration task; and 3, stopping the real-time data integration task through the real-time integration task interface, and when the real-time data integration task is started again, updating data in the real-time data integration task into a database. According to the invention, the data change is updated to the database immediately, the data delay is reduced, and the data processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and more specifically, to a real-time data integration method based on DolphinScheduler. Background Art

[0002] In the field of real-time data integration and streaming data processing, Apache Flink, as a high-performance, scalable stream processing framework, has been widely used to process large-scale data streams. As a component of Flink, Flink CDC (Change Data Capture) can synchronize and process database change events in real time, so that data changes can be immediately reflected in the data warehouse, indicator calculation or other systems. However, although Flink CDC performs well in real-time data synchronization, it requires professional knowledge and skills for users to use, and lacks flexibility and ease of use. As a distributed workflow task scheduling system, Apache DolphinScheduler provides a wealth of scheduling strategies, but its support for real-time tasks needs to be improved, and its user experience and compatibility with non-technical users need to be further explored and optimized. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a real-time data integration method based on DolphinScheduler in view of the shortcomings of the prior art, so as to solve the technical problems that Flink CDC is difficult to use and the data processing method of Apache DolphinScheduler has high processing delay and insufficient real-time performance.

[0004] The present invention discloses a real-time data integration method based on DolphinScheduler. The method comprises the following steps: step 1: adding a real-time integration task interface, setting page parameters in the real-time integration task interface, verifying the page parameters and creating a real-time data integration task;

[0005] Step 2: Start the real-time data integration task;

[0006] Step 3: If the real-time data integration task is stopped through the real-time integration task interface, when the real-time data integration task is started again, the data in the real-time data integration task is updated into the database.

[0007] As a further improvement, in step 1, the page parameters include a source data table, a target data table, a field mapping relationship between the source data table and the target data table, a yarn queue, and flink task-related startup parameters.

[0008] Furthermore, in step 1, the method for verifying the page parameters is that, when the target data table is a hive-type data table, querying whether the user has the operation authority for the target data table through the Ranger restFul interface;

[0009] When the target data table does not exist, the target data table is automatically created according to the field mapping relationship between the source data table and the target data table;

[0010] When the target data table exists, checking whether the field mapping relationship between the source data table and the target data table conforms to the fields of the target data table;

[0011] The page parameters are converted into backend entity classes and stored in relevant database tables.

[0012] Furthermore, in step 2, the step of starting the real-time data integration task is:

[0013] Step 1: setting execution parameters including savePoint location, detection point interval number, failure retry number and delayed execution time parameter on the real-time integration task interface;

[0014] Step 2: Encapsulate the execution parameters into a task start command message, and send the task start command message to the dolphinscheduler master component through netty communication;

[0015] The TaskExecuteStartProcessor real-time task processor in the dolphinscheduler master component creates a StreamTaskExecuteRunnable streaming task execution runnable according to the task start command message, and the StreamTaskExecuteRunnable streaming task execution runnable runs in the StreamTaskExecuteThreadPool streaming task execution thread pool;

[0016] Encapsulate the task parameters in the StreamTaskExecuteRunnable stream task execution runnable into a task execution command message, and send the task execution command message to the dolphinscheduler worker component through netty communication;

[0017] Step 3: The TaskExecuteProcessor task execution processor of the dolphinscheduler worker component creates a TaskExecuteThread task execution thread according to the task execution command message, and the TaskExecuteThread task execution thread runs in the workerExecService task execution thread pool. The execution path of the real-time data integration task and the storage path of the log are set in the TaskExecuteThread task execution thread. The execution path and the storage path of the log are called to build a start command according to the task type of the real-time data integration task, and the start command is executed. The real-time data integration task is executed in yarn;

[0018] Step 4: Obtain the task status and task log of the real-time data integration task. The dolphinscheduler worker component obtains the jobmanager address of each real-time task by parsing the task log, fills in the jobmanager address and generates a response, sends the response to the dolphinschedulermaster component and stores it in the database.

[0019] Furthermore, in the third step,

[0020] When the startup mode of the startup command is initial startup, select the run-application yarn-application command;

[0021] When the savePoint site is set in the first step, the run-applicationfromSavepoint command is selected.

[0022] Furthermore, after the run-application fromSavepoint command is selected, the parameters of the task execution context in the dolphinscheduler master component and the dolphinscheduler worker component are converted into a string in json format according to the parameters defined in the jar package according to the pre-uploaded jar package, and then the string is spliced ​​to the back of the start command to update the start command.

[0023] Furthermore, the jar package is developed based on flinkCDC, and the jar package performs data synchronization by receiving predefined parameters to create a flinkCDC connector of a source data table and a target data table.

[0024] Furthermore, the predefined parameters include a source data table connection method, a target data table connection method, a field mapping relationship between the source data table and the target data table, and flink task related startup parameters.

[0025] Furthermore, in the step three, after the real-time data integration task is stopped, when the real-time data integration task is started again, the database updates the jobmanager address.

[0026] Beneficial Effects

[0027] The advantages of the present invention are:

[0028] 1. The present invention realizes instant updating of data changes to the database, reduces data delays, and improves data processing efficiency by updating the jobmanager address in the database when the real-time data integration task is restarted after the real-time data integration task is stopped.

[0029] 2. The present invention creates and operates real-time data integration tasks by adding a real-time integration task interface, which simplifies user operations, improves user experience, increases operational friendliness for non-technical users, and reduces user difficulty.

[0030] 3. The present invention combines DolphinScheduler and FlinkCDC technology. The pre-developed jar package contains flink task-related startup parameters, and uses dolphinScheduler master component and dolphinSchedulerworker component to schedule real-time data integration tasks, which solves the shortcomings of DolphinScheduler in monitoring, status tracking and log viewing after real-time task submission, and enhances the visual monitoring and management capabilities in real-time task processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flow chart of the method for creating a real-time data task of the present invention;

[0032] Figure 2 The first part of the flow chart of the method for starting a real-time data task of the present invention;

[0033] Figure 3 This is a flow chart of the second part of the method for starting a real-time data task of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with the embodiments, but it does not constitute any limitation to the present invention. Any limited number of modifications made by anyone within the scope of the claims of the present invention are still within the scope of the claims of the present invention.

[0035] See also Figure 1-Figure 3 , a real-time data integration method based on DolphinScheduler of the present invention is as follows: Figure 1 As shown in the figure, step 1: add a real-time integration task interface, set page parameters in the real-time integration task interface, verify the page parameters and create a real-time data integration task. By adding a real-time integration task interface to create and operate real-time data integration tasks, user operations are simplified, user experience is improved, and operation friendliness is improved for non-technical users.

[0036] In step 1, the page parameters include the source data table, the target data table, the field mapping relationship between the source data table and the target data table, the yarn queue, and the flink task-related startup parameters. The real-time integration task interface parameters are uploaded to the server.

[0037] After receiving the form of the real-time integration task interface parameters, the server verifies the real-time page parameters in step 1. When the target data table is a hive data table, the Ranger restFul interface is used to query whether the user has the operation authority of the target data table.

[0038] When the target data table does not exist, it is automatically created based on the field mapping relationship between the source data table and the target data table.

[0039] When the target data table exists, check whether the field mapping relationship between the source data table and the target data table is consistent with the fields of the target data table.

[0040] The page parameters submitted to the server are converted into backend entity classes and stored in the relevant database tables. At this point, the real-time data integration task is created.

[0041] like Figure 2-3 As shown in the figure, step 2: start the real-time data integration task.

[0042] In step 2, the steps to start the real-time data integration task are:

[0043] Step 1: Set the execution parameters including the savePoint location, detection point interval, failure retry times and delayed execution time parameters on the real-time integration task interface, and then click Start Task on the real-time integration task interface to output the start task instruction.

[0044] Step 2: After the server (dolphinscheduler API service) receives the start task instruction, it encapsulates the start parameters into a task start command message and sends the task start command message to the dolphinscheduler master component through netty communication;

[0045] The TaskExecuteStartProcessor real-time task processor in the dolphinscheduler master component creates a StreamTaskExecuteRunnable streaming task execution runnable according to the task start command message. The StreamTaskExecuteRunnable streaming task execution runnable runs in the StreamTaskExecuteThreadPool streaming task execution thread pool.

[0046] Create a TaskExecutionContext task execution context in the StreamTaskExecuteRunnable streaming task execution runner, encapsulate all previous task parameters in the StreamTaskExecuteRunnable streaming task execution runner into a task execution command message, and send the task execution command message to the dolphinscheduler worker component through netty communication.

[0047] Step 3: The TaskExecuteProcessor task execution processor of the dolphinscheduler worker component creates a TaskExecuteThread task execution thread according to the task execution command message. The TaskExecuteThread task execution thread runs in the workerExecService task execution thread pool.

[0048] The execution path of the real-time data integration task and the log storage path are set in the TaskExecuteThread task execution thread. According to the task type, the corresponding plug-in is called to build a start command (here, the FlinkStreamTask plug-in is called) to convert the relevant parameters in the task execution context into a start command.

[0049] If the startup mode of the startup command is initial startup, select the run-application yarn-application command.

[0050] When the savePoint site is set in the first step, select run-application-

[0051] fromSavepoint command.

[0052] like Figure 1 As shown in the figure, developers upload the developed and public real-time integration jar package to HDFS in the background for use in subsequent real-time integration tasks. This jar package is developed based on flinkCDC, and its function is to accept defined parameters (the jar package includes the source table connection method, the target data table connection method, the field mapping relationship between the source data table and the target data table, the flink task related startup parameters, etc.) to create the source data table flinkCDC connector execution data and the target data table flinkCDC connector execution data synchronization. Users need to synchronize the source data table and the target data table in real time on the real-time integration task interface.

[0053] According to the pre-uploaded jar package, the parameters of the task execution context in the dolphinscheduler master component and the dolphinschedulerworker component are converted into a string in json format according to the parameters defined in the jar package, and then the string is concatenated to the end of the start command to generate a new start command.

[0054] Execute the start command, and the real-time data integration task is executed in yarn. Yarn is the abbreviation of Yet Another Resource Negotiator, which is an open source, distributed resource management system for large distributed computing clusters.

[0055] Step 4: After executing step 3, the entire task process ends when the Dolphinscheduler scheduling service successfully submits the real-time task to yarn. To obtain the task status and task log of the real-time data integration task, the Dolphinscheduler worker component obtains the jobmanager address of each real-time task by parsing the task log (the log storage path in step 3), fills in the jobmanager address and generates a response, sends the response to the Dolphinscheduler master component and stores it in the database.

[0056] Step 3: Stop the real-time data integration task through the real-time integration task interface.

[0057] In step 3, the server (dolphinscheduler API service) can call the restful interface to query the task status, task log and other information of the flink task according to the jobmanager address in the database. It can also perform savepoint or stop operations on the real-time task being executed. If the stop operation is performed, the system will store the latest jobmanager address in the database when the real-time integration task is started again, and the database will update the jobmanager address. A functional closed loop is achieved. Data changes are updated to the database in real time, reducing data delays and improving data processing efficiency.

[0058] The present invention uses dolphin Scheduler master component and dolphin Scheduler worker component to schedule real-time tasks, solves the shortcomings of Dolphin Scheduler in monitoring, status tracking and log viewing after real-time task submission, and enhances the visual monitoring and management capabilities in real-time task processing.

[0059] The above is only a preferred embodiment of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the structure of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A real-time data integration method based on DolphinScheduler, characterized in that: The method comprises the following steps: step 1: adding a real-time integration task interface, setting page parameters in the real-time integration task interface, verifying the page parameters and creating a real-time data integration task; Step 2: Start the real-time data integration task; Step 3: If the real-time data integration task is stopped through the real-time integration task interface, when the real-time data integration task is started again, the data in the real-time data integration task is updated into the database.

2. A real-time data integration method based on DolphinScheduler according to claim 1, characterized in that: In step 1, the page parameters include the source data table, the target data table, the field mapping relationship between the source data table and the target data table, the yarn queue, and the flink task related startup parameters.

3. A real-time data integration method based on DolphinScheduler according to claim 2, characterized in that: The method for verifying the page parameters in step 1 is: when the target data table is a hive data table, querying whether the user has the operation authority for the target data table through the Ranger restFul interface; When the target data table does not exist, the target data table is automatically created according to the field mapping relationship between the source data table and the target data table; When the target data table exists, checking whether the field mapping relationship between the source data table and the target data table conforms to the fields of the target data table; The page parameters are converted into backend entity classes and stored in relevant database tables.

4. The real-time data integration method based on DolphinScheduler according to claim 1, characterized in that: In step 2, the steps of starting the real-time data integration task are: Step 1: setting execution parameters including savePoint location, detection point interval number, failure retry number and delayed execution time parameter on the real-time integration task interface; Step 2: Encapsulate the execution parameters into a task start command message, and send the task start command message to the dolphinscheduler master component through netty communication; The TaskExecuteStartProcessor real-time task processor in the dolphinscheduler master component creates a StreamTaskExecuteRunnable streaming task execution runnable according to the task start command message, and the StreamTaskExecuteRunnable streaming task execution runnable runs in the StreamTaskExecuteThreadPool streaming task execution thread pool; Encapsulate the task parameters in the StreamTaskExecuteRunnable stream task execution runnable into a task execution command message, and send the task execution command message to the dolphinschedulerworker component through netty communication; Step 3: The TaskExecuteProcessor task execution processor of the dolphinscheduler worker component creates a TaskExecuteThread task execution thread according to the task execution command message, and the TaskExecuteThread task execution thread runs in the workerExecService task execution thread pool. The execution path of the real-time data integration task and the storage path of the log are set in the TaskExecuteThread task execution thread. The execution path and the storage path of the log are called to build a start command according to the task type of the real-time data integration task, and the start command is executed. The real-time data integration task is executed in yarn; Step 4: Obtain the task status and task log of the real-time data integration task. The dolphinschedulerworker component obtains the jobmanager address of each real-time task by parsing the task log, fills in the jobmanager address and generates a response, sends the response to the dolphinscheduler master component and stores it in the database.

5. A real-time data integration method based on DolphinScheduler according to claim 4, characterized in that: In the third step, When the startup mode of the startup command is initial startup, select the run-application yarn-application command; When the savePoint site is set in the first step, the run-application fromSavepoint command is selected.

6. A real-time data integration method based on DolphinScheduler according to claim 5, characterized in that: After the run-application fromSavepoint command is selected, the parameters of the task execution context in the dolphinscheduler master component and the dolphinscheduler worker component are converted into a string in json format according to the parameters defined in the jar package according to the pre-uploaded jar package, and then the string is spliced ​​to the end of the start command to update the start command.

7. A real-time data integration method based on DolphinScheduler according to claim 6, characterized in that: The jar package is developed based on flinkCDC, and the jar package performs data synchronization by receiving predefined parameters to create a flinkCDC connector for a source data table and a target data table.

8. The real-time data integration method based on DolphinScheduler according to claim 7, characterized in that: The predefined parameters include the source data table connection mode, the target data table connection mode, the field mapping relationship between the source data table and the target data table, and the flink task related startup parameters.

9. The real-time data integration method based on DolphinScheduler according to claim 4, characterized in that: In the step three, after the real-time data integration task is stopped, when the real-time data integration task is started again, the database updates the jobmanager address.

Images