ETL task processing method, system, medium and equipment for data integration platform

Abstract

The present invention provides an ETL task processing method, system, medium, and device for a data integration platform, comprising: submitting a DAG (Directed Acyclic Graph) generated by dragging and dropping on a user-side interface to an ETL parser; the ETL parser extracting a streaming operator and regenerating a new DAG execution graph; the streaming operator starting with the start of the previous task operator; adding a send end message after the batch operator task ends; the ETL parser submitting the new DAG execution graph to a scheduler to schedule the operator execution; and after the data output operator task ends, scheduling a task end interface to notify the task end. The present invention defines the ETL execution process as different types of operator logic, defines the conversion and output operators as streaming processes, and starts with the start of the previous task operator. This enables synchronous execution of streaming and batch tasks, thereby accelerating the processing speed of the entire ETL task.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an ETL task processing method, system, medium and device for a data integration platform. Background Art

[0002] With the advent of the big data era, data analysis scenarios have become more complex and diverse. Therefore, in the data processing process, a platform that can reuse processing modules and perform unified and standardized management of different processing modules is needed.

[0003] Currently, the development process for such platforms defines various ETL transformation tasks as various execution operator modules for management. However, these operator task types are not carefully categorized. During job execution, the next operator task must be started only after the previous one has completed, significantly reducing overall task processing efficiency.

[0004] Patent document CN110232085A (application number: CN201910359658.4) discloses a method and system for orchestrating big data ETL tasks. The patent defines task types as stream processing and batch processing. During the execution of a single task operator, the operator must wait until the previous task operator completes before executing the next operator task. However, in practice, tasks such as filtering and field mapping only process one piece of data at a time and do not perform calculations on other data. Therefore, these tasks can be started synchronously with the start of the previous task. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the present invention aims to provide an ETL task processing method, system, medium and device for a data integration platform.

[0006] The ETL task processing method for a data integration platform provided by the present invention includes:

[0007] Step 1: Use Spark coding to generate execution operators based on different computing logics. Execution operators are classified into transformation operators and calculation operators.

[0008] Step 2: Based on different data sources, use different data collection frameworks to generate different data input operators and data output operators. The data input operator uses batch processing or streaming processing according to different requirements, and the data output operator uses streaming processing;

[0009] Step 3: Based on the execution logic type of the operator, define the calculation operator and data input operator as batch operators, and define the conversion operator and data output operator as streaming operators. The data input operator issues a task completion message to notify subsequent operator tasks.

[0010] Step 4: The user drags and drops the generated DAG on the client interface and submits it to the ETL parser;

[0011] Step 5: The ETL parser extracts the streaming operators and regenerates a new DAG execution graph;

[0012] Step 6: When the next operator connected is a streaming operator, when the previous streaming operator task starts, the next streaming operator is notified to start the task and read the calculation results of the previous streaming operator; when the next operator is a batch operator, the batch operator starts after the previous task ends;

[0013] Step 7: After the batch operator task is completed, add a send end message; after the streaming operator receives the end message, the streaming operator task is completed;

[0014] Step 8: The ETL parser submits the new DAG execution graph to the scheduler, which schedules the operator execution.

[0015] Step 9: After the data output operator task is completed, the task completion interface is dispatched to notify the task completion.

[0016] Preferably, when the calculation operator performs cumulative calculation tasks including summation and average, it uses sparkcore batch processing logic to execute;

[0017] When the transformation operator performs data transformation logic tasks including filtering and mapping, Spark Streaming is used for streaming processing.

[0018] Preferably, based on the original DAG graph generated by the user by dragging and dropping in the interface, the ETL parser first identifies the streaming operator and the batch operator, and extracts the execution path J of the streaming operator and the previous operator, generates a DAG execution graph composed entirely of batch operators, and reassembles the DAG execution graph based on the execution path J.

[0019] Preferably, the dolphinScheduler is used as the scheduler, and the FIFO strategy is used to schedule operators to execute batch operator and streaming operator tasks on different machines or clusters.

[0020] The ETL task processing system for a data integration platform provided by the present invention includes:

[0021] Module M1: Generates execution operators using Spark coding based on different computing logics. Execution operators are categorized into conversion operators and calculation operators.

[0022] Module M2: Based on different data sources, different data collection frameworks are used to generate different data input operators and data output operators. The data input operator uses batch processing or streaming processing according to different requirements, and the data output operator uses streaming processing;

[0023] Module M3: Based on the execution logic of the operator, the calculation operator and data input operator are defined as batch operators, and the conversion operator and data output operator are defined as streaming operators. The data input operator issues a task completion message to notify subsequent operator tasks.

[0024] Module M4: The user drags and drops the generated DAG on the user interface and submits it to the ETL parser;

[0025] Module M5: The ETL parser extracts the streaming operators and regenerates a new DAG execution graph;

[0026] Module M6: When the next operator connected is a streaming operator, when the previous streaming operator task starts, the next streaming operator is notified to start the task and read the calculation results of the previous streaming operator; when the next operator is a batch operator, the batch operator starts after the previous task ends;

[0027] Module M7: After the batch operator task is completed, add a send end message; after the streaming operator receives the end message, the streaming operator task is completed;

[0028] Module M8: The ETL parser submits the new DAG execution graph to the scheduler to schedule operator execution;

[0029] Module M9: After the data output operator task is completed, the task completion interface is scheduled to notify the task completion.

[0030] Preferably, when the calculation operator performs cumulative calculation tasks including summation and average, it uses sparkcore batch processing logic to execute;

[0031] When the transformation operator performs data transformation logic tasks including filtering and mapping, Spark Streaming is used for streaming processing.

[0032] Preferably, based on the original DAG graph generated by the user by dragging and dropping in the interface, the ETL parser first identifies the streaming operator and the batch operator, and extracts the execution path J of the streaming operator and the previous operator, generates a DAG execution graph composed entirely of batch operators, and reassembles the DAG execution graph based on the execution path J.

[0033] Preferably, the dolphinScheduler is used as the scheduler, and the FIFO strategy is used to schedule operators to execute batch operator and streaming operator tasks on different machines or clusters.

[0034] According to the computer-readable storage medium storing a computer program provided by the present invention, when the computer program is executed by a processor, the steps of the ETL task processing method for a data integration platform are implemented.

[0035] The electronic device provided according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the ETL task processing method for a data integration platform are implemented.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention defines the ETL execution process as different types of operator logic, defines the conversion and output operators as streaming processing processes, and starts them when the previous task operator starts, thereby realizing the synchronous execution of streaming tasks and batch tasks, thereby speeding up the processing speed of the entire ETL task. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0039] Figure 1 This is a flow chart of the ETL task processing method of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0041] Example 1

[0042] like Figure 1 The present invention provides an ETL task processing method for a data integration platform, comprising the following steps:

[0043] Step 1: Use Spark coding to generate execution operators based on different computing logics. Execution operators are categorized into conversion operators and calculation operators. Calculation operators such as summation and average are executed using Spark Core batch processing logic. Transformation operators such as filtering and mapping perform data conversion logic tasks using Spark Streaming for streaming processing.

[0044] Step 2: Based on different data sources, use different data collection frameworks to generate different data input operators and data output operators. The data input operator can be either batch processing or stream processing according to different requirements; the data output operator is stream processing;

[0045] Step 3: Based on whether the operator's execution logic is batch processing or streaming processing, define the computation operator as a batch operator, and define the transformation operator and data output operator as streaming operators. Due to the special nature of the data input operator, the data input operator is classified as a batch operator. The data input operator will also issue a task completion message to notify subsequent operator tasks.

[0046] The difference between stream processing logic and batch processing logic lies in whether processing can be performed without waiting for all data to arrive. For example, summing and filtering require waiting for all data from 1 to 10 to arrive before cumulative calculation, while filtering only requires checking whether each data item is greater than or less than 1.

[0047] Stream processing logic: Each piece of data can be calculated independently after it arrives;

[0048] Batch processing logic: Calculation can only be performed after all data arrives.

[0049] Step 4: The user drags and drops the generated DAG on the client interface and submits it to the ETL parser;

[0050] Step 5: The ETL parser extracts the streaming operators and regenerates a new DAG execution graph;

[0051] Based on the original DAG graph (A-A1-A2-A3-A4-A5) generated by the user by dragging and dropping in the interface, the ETL parser will first identify the streaming operators and batch operators (such as streaming operators A2 and A4, and batch operators A, A1, A3, and A5). It will then extract the execution path J of the streaming operator and the previous operator (A1-A2, A3-A4), generate a DAG execution graph consisting entirely of batch operators (A-A1-A3-A5), and reassemble the DAG execution graph (A-A1-A3-A5, A-A2-A3-A5, A-A1-A4-A5, and A-A2-A4-A5) based on execution path J.

[0052] Step 6: When the next operator connected is a streaming operator, when the previous operator task starts, the streaming operator will be notified to start the task at the same time, and the calculation results of the previous operator will be read in a streaming manner. If the next operator is a batch operator, the batch operator will start after the previous task ends.

[0053] Step 7: After the batch operator task is completed, a completion message will be sent; after the streaming operator receives the completion message, the streaming operator task is completed;

[0054] Step 8: The ETL parser submits the new DAG execution graph to the scheduler, which schedules the operator execution.

[0055] The dolphinScheduler is used as the scheduler, and the FIFO strategy is used to schedule operators to execute batch and streaming operator tasks on different machines or clusters.

[0056] Step 9: After the data output operator task is completed, the task completion interface is dispatched to notify the task completion.

[0057] Example 2

[0058] The present invention also provides an ETL task processing system for a data integration platform. The ETL task processing system for a data integration platform can be implemented by executing the process steps of the ETL task processing method for a data integration platform. That is, those skilled in the art can understand the ETL task processing method for a data integration platform as a preferred implementation of the ETL task processing system for a data integration platform.

[0059] The ETL task processing system for a data integration platform provided by the present invention includes: module M1: using Spark coding to generate execution operators according to different calculation logics, and the execution operators are classified into conversion operators and calculation operators; module M2: using different data acquisition frameworks to generate different data input operators and data output operators according to different data sources, the data input operator uses batch processing or stream processing according to different requirements, and the data output operator uses stream processing; module M3: defining the calculation operator and the data input operator as batch operators, and the conversion operator and the data output operator as stream operators according to the type of the operator's execution logic, and the data input operator issues a task end message to notify subsequent operator tasks; module M4: the user drags and drops the generated DAT on the user terminal interface G directed acyclic graph is submitted to the ETL parser; Module M5: The ETL parser extracts the streaming operator and regenerates a new DAG execution graph; Module M6: When the next operator connected is a streaming operator, when the previous streaming operator task starts, the next streaming operator is notified to start the task and read the calculation result of the previous streaming operator; when the next operator is a batch operator, the batch operator starts after the previous task ends; Module M7: After the batch operator task ends, an end message is added to send; after the streaming operator receives the end message, the streaming operator task ends; Module M8: The ETL parser submits the new DAG execution graph to the scheduler and schedules the operator execution; Module M9: After the data output operator task ends, the scheduling task end interface is used to notify the task end.

[0060] When the calculation operator performs cumulative calculation tasks including summation and average, it uses SparkCore batch processing logic execution; when the transformation operator performs data transformation logic tasks including filtering and mapping, it uses SparkStreaming for streaming processing.

[0061] Based on the original DAG graph generated by the user by dragging and dropping in the interface, the ETL parser first identifies the streaming operator and the batch operator, extracts the execution path J of the streaming operator and the previous operator, generates a DAG execution graph consisting entirely of batch operators, and reassembles the DAG execution graph based on the execution path J.

[0062] The dolphinScheduler is used as the scheduler, and the FIFO strategy is used to schedule operators to execute batch operator and streaming operator tasks on different machines or clusters.

[0063] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0064] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. An ETL task processing method for a data integration platform, characterized in that: include: Step 1: Use Spark coding to generate execution operators based on different computing logics. Execution operators are classified into transformation operators and calculation operators. Step 2: Based on different data sources, use different data collection frameworks to generate different data input operators and data output operators. The data input operator uses batch processing or streaming processing according to different requirements, and the data output operator uses streaming processing; Step 3: Based on the execution logic type of the operator, define the calculation operator and data input operator as batch operators, and define the conversion operator and data output operator as streaming operators. The data input operator issues a task completion message to notify subsequent operator tasks. Step 4: The user drags and drops the generated DAG on the client interface and submits it to the ETL parser; Step 5: The ETL parser extracts the streaming operators and regenerates a new DAG execution graph; Step 6: When the next operator connected is a streaming operator, when the previous streaming operator task starts, the next streaming operator is notified to start the task and read the calculation results of the previous streaming operator; when the next operator is a batch operator, the batch operator starts after the previous task ends; Step 7: After the batch operator task is completed, add a send end message; after the streaming operator receives the end message, the streaming operator task is completed; Step 8: The ETL parser submits the new DAG execution graph to the scheduler, which schedules the operator execution. Step 9: After the data output operator task is completed, the task completion interface is dispatched to notify the task completion.

2. The ETL task processing method for a data integration platform according to claim 1, characterized in that: When the calculation operator performs cumulative calculation tasks including summation and average, it uses SparkCore batch processing logic. When the transformation operator performs data transformation logic tasks including filtering and mapping, Spark Streaming is used for streaming processing.

3. The ETL task processing method for a data integration platform according to claim 1, characterized in that: Based on the original DAG graph generated by the user by dragging and dropping in the interface, the ETL parser first identifies the streaming operator and the batch operator, extracts the execution path J of the streaming operator and the previous operator, generates a DAG execution graph consisting entirely of batch operators, and reassembles the DAG execution graph based on the execution path J.

4. The ETL task processing method for a data integration platform according to claim 1, characterized in that: The dolphinScheduler is used as the scheduler, and the FIFO strategy is used to schedule operators to execute batch operator and streaming operator tasks on different machines or clusters.

5. An ETL task processing system for a data integration platform, characterized in that: include: Module M1: Generates execution operators using Spark coding based on different computing logics. Execution operators are categorized into conversion operators and calculation operators. Module M2: Based on different data sources, different data collection frameworks are used to generate different data input operators and data output operators. The data input operator uses batch processing or streaming processing according to different requirements, and the data output operator uses streaming processing; Module M3: Based on the execution logic of the operator, the calculation operator and data input operator are defined as batch operators, and the conversion operator and data output operator are defined as streaming operators. The data input operator issues a task completion message to notify subsequent operator tasks. Module M4: The user drags and drops the generated DAG on the user interface and submits it to the ETL parser; Module M5: The ETL parser extracts the streaming operators and regenerates a new DAG execution graph; Module M6: When the next operator connected is a streaming operator, when the previous streaming operator task starts, the next streaming operator is notified to start the task and read the calculation results of the previous streaming operator; when the next operator is a batch operator, the batch operator starts after the previous task ends; Module M7: After the batch operator task is completed, add a send end message; after the streaming operator receives the end message, the streaming operator task is completed; Module M8: The ETL parser submits the new DAG execution graph to the scheduler to schedule operator execution; Module M9: After the data output operator task is completed, the task completion interface is scheduled to notify the task completion.

6. The ETL task processing system for a data integration platform according to claim 5, characterized in that: When the calculation operator performs cumulative calculation tasks including summation and average, it uses SparkCore batch processing logic. When the transformation operator performs data transformation logic tasks including filtering and mapping, Spark Streaming is used for streaming processing.

7. The ETL task processing system for a data integration platform according to claim 5, characterized in that: Based on the original DAG graph generated by the user by dragging and dropping in the interface, the ETL parser first identifies the streaming operator and the batch operator, extracts the execution path J of the streaming operator and the previous operator, generates a DAG execution graph consisting entirely of batch operators, and reassembles the DAG execution graph based on the execution path J.

8. The ETL task processing system for a data integration platform according to claim 5, characterized in that: The dolphinScheduler is used as the scheduler, and the FIFO strategy is used to schedule operators to execute batch operator and streaming operator tasks on different machines or clusters.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the ETL task processing method for a data integration platform according to any one of claims 1 to 4 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the ETL task processing method for a data integration platform according to any one of claims 1 to 4 are implemented.

Images