Data offline synchronization method and device based on SeaTunnel

Abstract

The invention belongs to the technical field of data processing, and relates to a SeaTunnel-based data offline synchronization method and device, computer equipment and a storage medium, and the method comprises the following steps: formulating a visual parameter configuration component page according to a connection component and a conversion component in SeaTunnel; binding a component generation strategy according to page component configuration; according to the component generation strategy, a SeaTunnel script is generated; and the generated SeaTunnel script is synchronized to a DolphinScheduler task, and the generated SeaTunnel script is synchronized to the DolphinScheduler task. SeaTunnel script configuration is carried out in a visual mode, the development difficulty is reduced, the specification of the SeaTunnel script is defined, and the script abnormal rate is reduced, so that the development efficiency is improved, and the problems that development is difficult, errors are prone to occurring, and the efficiency is low can be solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a SeaTunnel-based data offline synchronization method, device, computer equipment and storage medium. Background Art

[0002] Data synchronization means that a handheld computer can quickly synchronize data and share information with a desktop computer or laptop computer, so that your data can maintain integrity and consistency. Data synchronization is achieved through various data transmission interfaces, such as USB synchronization docks. I believe that many people use multiple computers at the same time: at home, in the office, or in other secret places. Multiple computers correspond to multiple hard drives, and also to multiple folders. How to synchronize data is a headache for many people. If you rely entirely on USB flash drives for synchronization, you will not be able to remember which version is the latest and which version needs to be modified. In the end, it is common to make mistakes in files. At this time, data synchronization is needed.

[0003] Data synchronization is a simple synchronization, which means executing the same operation at the same time. Data synchronization locates the object as data. Most of the time, it is understood as a backup operation between different storage devices or terminals, or between terminals and servers. However, complete data synchronization should be real-time, that is, the two parties of the current operation should be mirror images of each other. For example, the backup process until the end is data synchronization, but this operation example should belong to backup.

[0004] SeaTunnel is a very simple and easy-to-use product with high performance for real-time streaming and offline batch processing of massive data. Apache DolphinScheduler is a distributed and scalable open source visual DAG workflow task scheduling system. Suitable for enterprise-level scenarios, it provides a solution for visual operation tasks, workflows, and full life cycle data processing. Integrate SeaTunnel and DolphinScheduler technology to open up data collection and task scheduling channels. Users configure Seatunnel custom tasks on the front-end interface, save and submit them to the distributed task scheduler DolphinScheduler for scheduling in a workflow manner. In the actual SeaTunnel script development process, developers and business personnel need to develop scripts in accordance with the SeaTunnel official documentation, which will lead to low development efficiency, script exceptions, and high learning costs. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a method, device, computer equipment and storage medium for offline data synchronization based on SeaTunnel, so as to solve the problem that script development needs to refer to the official documentation of SeaTunnel in the prior art, resulting in low development efficiency, script abnormality and high learning cost.

[0006] In order to solve the above technical problems, the present invention provides a data offline synchronization method based on SeaTunnel, which adopts the following technical solution, including:

[0007] According to the Connector component and Transform component in SeaTunnel, formulate the component page for visual parameter configuration;

[0008] Bind component generation strategy based on page component configuration;

[0009] Generate a SeaTunnel script according to the component generation strategy;

[0010] Synchronize the generated SeaTunnel script to the DolphinScheduler task.

[0011] Preferably, the step of formulating a visualization parameter configuration component page according to the connection component Connector and the transformation component Transform in SeaTunnel specifically includes:

[0012] Get the parameters of the SeaTunnel official website Connector component;

[0013] Get the Transform component parameters of the SeaTunnel official website;

[0014] The obtained Connector component parameters and Transform component parameters are configured according to different component types.

[0015] Preferably, the step of binding components to generate strategies according to page component configuration specifically includes:

[0016] Get page component configuration;

[0017] The backend performs configuration analysis of different components based on the component generation strategy.

[0018] Preferably, the step of generating a SeaTunnel script according to the component generation strategy specifically includes:

[0019] Obtain the grammatical rules of each component according to the official website documentation;

[0020] The backend will parse the configuration and generate SeaTunnel script syntax according to the syntax rules.

[0021] Preferably, the step of synchronizing the generated SeaTunnel script to the DolphinScheduler task specifically includes:

[0022] Synchronizing DolphinScheduler requires operating five tables of the DolphinScheduler system, namely, task definition record table t_ds_task_definition_log, task definition table t_ds_task_definition, process definition record table t_ds_process_definition_log, process definition table t_ds_process_definition, process task relationship table t_ds_process_task_relation;

[0023] The newly added SeaTunnel script tasks are stored in table records according to each table field.

[0024] Preferably, after the step of synchronizing the generated SeaTunnel script to the DolphinScheduler task, the method further includes:

[0025] Submit the defined task and check whether the defined task exists in DolphinScheduler.

[0026] In order to solve the above technical problems, the present invention further provides a data offline synchronization device based on SeaTunnel, which adopts the following technical solution, including:

[0027] The configuration module is used to formulate the visualization parameter configuration component page according to the connection component Connector and the transformation component Transform in SeaTunnel;

[0028] The binding module is used to bind components to generate strategies according to page component configuration;

[0029] A script module, used to generate a SeaTunnel script according to the component generation strategy;

[0030] The synchronization module is used to synchronize the generated SeaTunnel script to the DolphinScheduler task.

[0031] Preferably, the configuration module includes:

[0032] The first parameter acquisition module is used to obtain the parameters of the SeaTunnel official website Connector component;

[0033] The second parameter acquisition module is used to obtain the Transform component parameters of the SeaTunnel official website;

[0034] The type configuration module is used to configure the obtained Connector component parameters and Transform component parameters according to different component types.

[0035] In order to solve the above technical problems, the present invention also provides a computer device, which adopts the following technical solution, including a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned SeaTunnel-based data offline synchronization method when executing the computer-readable instructions.

[0036] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, which adopts the following technical solution, and stores computer-readable instructions on the computer-readable storage medium. When the computer-readable instructions are executed by the processor, the steps of the above-mentioned SeaTunnel-based data offline synchronization method are implemented.

[0037] Compared with the prior art, the present invention has the following beneficial effects: by integrating SeaTunnel and DolphinScheduler, SeaTunnel script configuration is performed in a visual manner, data collection and task scheduling channels are opened up, Seatunnel custom tasks can be configured on the front-end interface, saved and submitted to the distributed task scheduler DolphinScheduler, and data offline task scheduling can be performed in a workflow manner. Compared with the prior art SeaTunnel script development process, in which developers and business personnel need to refer to SeaTunnel official documents for script development, which results in low development efficiency, script exceptions, and high learning costs, a data offline synchronization mode based on SeaTunnel is constructed, which reduces the development difficulty, defines the specifications of SeaTunnel scripts, reduces the script exception rate, and thus improves development efficiency, and can solve the problems of difficult development, easy errors, and low efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the scheme in the present invention, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1is a flow chart of an embodiment of a SeaTunnel-based data offline synchronization method of the present invention;

[0040] Figure 2 is a schematic diagram of data source configuration used in the SeaTunnel-based data offline synchronization method of the present invention;

[0041] Figure 3 is a schematic diagram of data input used in the SeaTunnel-based data offline synchronization method of the present invention;

[0042] Figure 4 is a schematic diagram of data output used in the SeaTunnel-based data offline synchronization method of the present invention;

[0043] Figure 5 is a schematic diagram of data processing used in the SeaTunnel-based data offline synchronization method of the present invention;

[0044] Figure 6 is a schematic diagram of basic information of configuration tasks used in the SeaTunnel-based data offline synchronization method of the present invention;

[0045] Figure 7 is a schematic diagram of a backend menu structure used in the SeaTunnel-based data offline synchronization method of the present invention;

[0046] Figure 8 1. It is a schematic diagram of data input for offline synchronization of student tables used in the offline data synchronization method based on SeaTunnel of the present invention;

[0047] Fig. 9 1. It is a data output diagram of offline synchronization of a student table used in the offline data synchronization method based on SeaTunnel of the present invention;

[0048] Fig.10 1. It is a schematic diagram of task information configuration of offline synchronization of student table synchronization used in the offline data synchronization method based on SeaTunnel of the present invention;

[0049] Fig.11 It is a schematic diagram showing whether a task of checking whether a student table synchronization offline synchronization exists used in the SeaTunnel-based data offline synchronization method of the present invention;

[0050] Fig.12 is a structural diagram of an embodiment of a SeaTunnel-based data offline synchronization device of the present invention;

[0051] Fig.13 It is a schematic structural diagram of an embodiment of a computer device of the present invention. DETAILED DESCRIPTION

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present invention; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present invention or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0053] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0054] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0055] It should be noted that the SeaTunnel-based data offline synchronization method provided in the embodiment of the present invention is generally executed by a server / terminal device, and accordingly, the SeaTunnel-based data offline synchronization device is generally disposed in the server / terminal device.

[0056] It should be understood that the numbers of the above terminal devices, networks and servers are only illustrative. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0057] In the following embodiments, the basic environment package uses SeaTunnel V2.3.2 and DolphinScheduler V3.4.1 for illustration.

[0058] Embodiment 1

[0059] Continue to refer Figure 1 , shows a flow chart of an embodiment of a SeaTunnel-based data offline synchronization method of the present invention. The SeaTunnel-based data offline synchronization method comprises the following steps:

[0060] Step S1, according to the connection component Connector and the transformation component Transform in SeaTunnel, formulate a visualization parameter configuration component page.

[0061] In this embodiment, the electronic device (e.g., server / terminal device) on which the SeaTunnel-based data offline synchronization method runs can receive the SeaTunnel-based data offline synchronization request through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAXX connection, Zigbee connection, UWB (ultrawideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0062] SeaTunnel script development includes four component configurations, namely env, Source, Transform, and Sink. The functions of each component are shown below:

[0063] env: engine configuration component, used to add some optional engine parameters and select the engine according to the actual deployment environment. SeaTunnel supports three engine options: SeaTunnel (own), FLink, and Spark;

[0064] Source: data source component, supporting multiple data source types, such as JDBC, Http, Elasticsearch, FtpFile, kafka, etc.

[0065] Transform: Data transformation component, which provides a variety of transformation components for data processing, such as Copy, Filter, Replace, Split, SQL, etc.

[0066] Sink: data receiving component, supporting multiple data receiving types, such as Doris, Elasticsearch, Hbase, kafka, JDBC, S3File, etc.

[0067] In this embodiment, step S1 may further include:

[0068] S11. Get the parameters of the Connector component on the SeaTunnel official website.

[0069] S12. Get the Transform component parameters of the SeaTunnel official website.

[0070] S13. Configure the obtained Connector component parameters and Transform component parameters according to different component types.

[0071] When implementing the component page for visual parameter configuration, a stream task configuration method can be adopted. Stream task configuration refers to the configuration operation of the previous step and the next step, which refers to the step-by-step task configuration.

[0072] First, configure the data source and target for offline data synchronization. Figure 2 Schematic diagram of data source configuration used in the SeaTunnel-based data offline synchronization method of the present invention. Figure 2 As shown in the figure, data source configuration includes configuring data input, data output, and data processing. It should be noted that data input refers to defining the data source (the data source to be extracted), which corresponds to the Source component of SeaTunnel. Data output refers to defining the data target (the target source where the data needs to be saved), which corresponds to the Sink component of SeaTunnel. Data processing refers to defining the logic that needs to be processed during the data extraction process (multi-layer logic, executed from top to bottom), which corresponds to the Transform component of SeaTunnel.

[0073] Figure 3 Schematic diagram of data input used in the SeaTunnel-based data offline synchronization method of the present invention. Figure 3 As shown, data input supports the selection of Mysql, Oracle, PostgreSQL, HANA, Trino, Hive, Impala, Doris, Phoenix, ClickHouse, and Kafka data source types, and different configuration parameters are implemented for each data source type. Figure 3 It is displayed as a Mysql data source type.

[0074] Figure 4 Schematic diagram of data output used in the SeaTunnel-based data offline synchronization method of the present invention. Figure 4 As shown, data output supports the selection of Mysql, Oracle, PostgreSQL, HANA, Trino, Hive, Impala, Doris, Phoenix, ClickHouse, and Kafka data source types, and implements different configuration parameters for each data source type. Figure 4 It is displayed as a Mysql data source type.

[0075] Figure 5 Schematic diagram of data processing used in the SeaTunnel-based data offline synchronization method of the present invention. Figure 5 As shown, data processing: supports SQL, Copy, Split, Filter, Replace, FieldMapper, and FilterRowKind processing functions, implements different configurations for each processing function, and also supports multiple data processing functions (multiple processing functions are executed in sequence).

[0076] Step S2, binding components to generate strategies according to page component configuration.

[0077] The component generation strategy, or component generation method, means that each component type will have a different implementation in the backend code, and the backend code has a strategy that generates syntax based on different component types.

[0078] In this embodiment, step S2 may further include:

[0079] S21. Get page component configuration.

[0080] S22. The backend performs configuration analysis of different components based on the component generation strategy.

[0081] Figure 6 Schematic diagram of basic information of configuration tasks used in the SeaTunnel-based offline data synchronization method of the present invention. Figure 6 As shown in the figure, the SeaTunnel configures the basic information of the task, including the task name, task description information definition, workgroup selection, timeout alarm strategy and notification strategy definition, notification group selection, etc. On the task information configuration page, you can define the task name, task description information, select the workgroup, define the timeout alarm strategy and notification strategy, notification group selection, etc. The task name refers to the defined task name. The task description information refers to the description information added to the task, such as the usage scenario of the task. The workgroup refers to the workgroup that executes the task, which can be understood as the server that provides the execution environment for the task. The timeout alarm strategy can include two strategies: timeout failure and timeout alarm. Timeout failure means that if the task execution is greater than the defined timeout duration threshold, the task execution result will be defined as a failure. Timeout alarm means that the task execution result is defined as a timeout. The notification strategy includes four notification strategies: 1-None (no matter success or failure, no notification); 2-Failure send (only failure send); 3-Success send (only success send); 4-All send (no matter success or failure, all send). The notification group refers to the members that need to be notified, and these members are all in this notification group.

[0082] Step S3: Generate a SeaTunnel script according to the component generation strategy.

[0083] In this embodiment, step S3 may further include:

[0084] S31. Obtain the grammatical rules of each component according to the official website documentation.

[0085] S32. The backend will parse the configuration and generate SeaTunnel script syntax according to the syntax rules.

[0086] The SeaTunnel script is automatically generated by the Java program. The backend adopts the registrar mode, assembling the data types and function configurations of the env, sink, Source, and Transform modules into Bean objects, which are managed by the registrar and registered to obtain the class when used. Finally, all components are merged into a complete syntax. The backend refers to the code written in Java language, because it is run by the backend and cannot be seen by users.

[0087] Figure 7 Schematic diagram of the backend menu structure used in the SeaTunnel-based data offline synchronization method of the present invention. Figure 7 As shown, in the backend menu structure, each directory has an interface class (defining behavior), a registrar (registering and obtaining), a data source type, and a function implementation class.

[0088] The interface class (taking sink as an example) can be:

[0089] public interface Sink {

[0090] 1usage 5implementations

[0091] String createSink(String sinkType, int index, String sinkConfig, TBdpDatasourceDetailDT0 data);

[0092] }

[0093] public class SinkRegister{

[0094] 2 usages

[0095] private static final Map<String,Sink> sinkMap

[0096] =new LinkedCaseInsensitiveMap<>();

[0097] 11usages

[0098] public static void sinkFactory(String sinkType,

[0099] Sink sink){sinkMap.put(sinkType,sink);}

[0100] 1usage

[0101] public static Sink findOrCreate(String sinkType){

[0102] Sink sink=sinkMap.get(String.valueOf(sinkType));

[0103] if(sink!=null){

[0104] return sink;

[0105] }

[0106] return new JdbcSink();

[0107] }

[0108] static{

[0109] sinkFactory(SeaTunnelSinkTypeEnum.CLICKH0USE.getType(),

[0110] new ClickhouseSink());

[0111] sinkFactory(SeaTunnelSinkTypeEnum.DORIS.getType(),

[0112] new DorisSink());

[0113] sinkFactory(SeaTunnelSinkTypeEnum.HIVE.getType(),

[0114] nem HiveSink());

[0115] sinkFactory(SeaTunnelSinkTypeEnum.MYSQL.getTypec),new JdbcSink());

[0116] sinkFactory(SeaTunnelSinkTypeEnum.0RACLE.getType(),

[0117] nem JdbcSink());

[0118] sinkFactory(SeaTunnelSinkTypeEnum.TRIN0.getType(),

[0119] new JdbcSink());

[0120] sinkFactory(SeaTunnelSinkTypeEnum.POSTGRESQL.getType(),

[0121] new JdbeSink());

[0122] sinkFactory(SeaTunnelSinkTypeEnum.PHOENIX.getType(),

[0123] new JdbcSink());

[0124] sinkFactorg(SeaTunnelSinkTypeEnum.HANA.getType(),

[0125] new JdbcSink[));

[0126] sinkFactory(SeaTunnelSinkTypeEnum.IMPALAngetType(,

[0127] nem JdbcSink()3;

[0128] sinkFactorg(SeaTunnelSinkTypeEnum.KAFKA,getType(),

[0129] new KafkaSink();

[0130] }

[0131] Registrar (taking sink as an example): Each data source type corresponds to a Sink implementation. When using it, you only need to call the findOrCreate method of the class and pass in the data source type to get the corresponding type implementation. The registrar here refers to the registrar in the backend code, which is used to manage the SeaTunnel syntax generation of various component types. For example, HiveSink in the example is a SeaTunnel syntax generation code written for the Hive component. The registrar can actively find the SeaTunnel syntax generation code of HiveSink through the hive component to generate the SeaTunnel syntax of the Hive component.

[0132] HiveSink implements the Sink interface and has all the hiveSink attribute parameters in the official document. In createSink, the SeaTunnel syntax of the hive data source type is spliced ​​to obtain a Sink script.

[0133] Note: For env, source, transform and other data source types of sink, the function configuration backend is implemented as shown in the hive example operation implementation.

[0134] When all configuration syntaxes are obtained, they are assembled into a complete SeaTunnel script in the order of env>source>transform>sink (Transform can be empty or multiple). The SeaTunnel script exists in Dolphinscheduler as a task.

[0135] Step S4: Synchronize the generated SeaTunnel script to the DolphinScheduler task.

[0136] In this embodiment, step S4 may further include:

[0137] S41. Synchronizing DolphinScheduler requires operating five tables of the DolphinScheduler system, namely, the task definition record table t_ds_task_definition_log, the task definition table t_ds_task_definition, the process definition record table t_ds_process_definition_log, the process definition table t_ds_process_definition, and the process task relationship table t_ds_process_task_relation.

[0138] The task definition record table t_ds_task_definition_log is used for version records, and the process definition record table t_ds_process_definition_log can also be used for version records.

[0139] S42. The newly added SeaTunnel script task is stored in table records according to each table field.

[0140] After the storage is successfully synchronized, you can see the created SeaTunnel script task on the DolphinScheduler task flow page, and you can run the task, view the log, set the schedule, and other operations. The task here refers to the SeaTunnel script obtained in step S3 in the form of a task in DolphinScheduler. The SeaTunnel scripts in DolphinScheduler are tasks, and they need to be run, so they are called running tasks. Running these tasks will generate logs, and you need to view the logs to obtain the running status of these tasks. You can set the scheduled execution of these tasks.

[0141] In some optional implementations of this embodiment, after step S4, the electronic device may further perform the following steps:

[0142] S5. Submit the defined task and check whether the defined task exists in DolphinScheduler.

[0143] For example, the task list can be used to check whether the defined task exists. Through step S5, it can be determined whether there is a task for offline data synchronization. If not, it indicates that the offline data synchronization has failed; if it exists, it indicates that the offline data synchronization has succeeded.

[0144] The implementation of this embodiment has the following beneficial effects: by integrating SeaTunnel and DolphinScheduler, SeaTunnel script configuration is performed in a visual manner, data collection and task scheduling channels are opened up, Seatunnel custom tasks can be configured in the front-end interface, saved and submitted to the distributed task scheduler DolphinScheduler, and data offline task scheduling can be performed in a workflow manner. Compared with the prior art SeaTunnel script development process, in which developers and business personnel need to refer to SeaTunnel official documents for script development, which results in low development efficiency, script exceptions, and high learning costs, a SeaTunnel-based data offline synchronization mode is constructed, which reduces the development difficulty, defines the specifications of SeaTunnel scripts, reduces the script exception rate, and thus improves development efficiency, and can solve the problems of difficult development, easy errors, and low efficiency.

[0145] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0147] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0148] Embodiment 2

[0149] Take student table synchronization (synchronize the student table of MySQL data source type to t_ods_student of Trino data source type offline) as an example to explain:

[0150] Step 1: Define data input, select the Mysql data source type, select the data source where student is located, and write the SQL query statement: select * from student, such as Figure 8 As shown, Figure 8 It is a data input schematic diagram of offline synchronization of student tables used in the offline data synchronization method based on SeaTunnel of the present invention.

[0151] Step 2: Define data output, select Trino data source type, select Trino data source, database, data table, and write insert SQL statement:

[0152] Insert into t_ods_student(s_no,s_name,s_sex,borndate,class_name,telephone,enroll_date,address,comment)values(?,?,?,?,?,?,?,?,?), such as Fig. 9 As shown, Fig. 9 It is a data output schematic diagram of offline synchronization of student tables used in the offline data synchronization method based on SeaTunnel of the present invention.

[0153] Step 3: Data processing definition (omitted), because this example does not require data processing. You can set the links that require data processing according to the needs of the actual usage scenario.

[0154] Step 4: Click Next to configure the task information, fill in the workflow name and description, select the Worker group (task execution engine server), timeout alarm strategy, define the notification strategy, and select the notification group (task exception notification person). Fig.10 As shown, Fig.10 It is a schematic diagram of task information configuration of offline synchronization of student tables used in the offline data synchronization method based on SeaTunnel of the present invention.

[0155] Step 5: Submit the defined task and check whether the task exists in DolphinScheduler. Check the task list to see if the "Student Table Synchronization" task exists. Fig.11 As shown, Fig.11 It is a schematic diagram of whether a task of checking whether a student table synchronization offline synchronization used in the SeaTunnel-based data offline synchronization method of the present invention exists.

[0156] View the generated SeaTunnel syntax:

[0157]

[0158]

[0159]

[0160] Step 6: The student table synchronization task already exists. Execute the operation, check the operation log, and wait for the task to complete.

[0161] The implementation of this embodiment has the following beneficial effects: by integrating SeaTunnel and DolphinScheduler, SeaTunnel script configuration is performed in a visual manner, data collection and task scheduling channels are opened up, Seatunnel custom tasks can be configured in the front-end interface, saved and submitted to the distributed task scheduler DolphinScheduler, and data offline task scheduling can be performed in a workflow manner. Compared with the prior art SeaTunnel script development process, in which developers and business personnel need to refer to SeaTunnel official documents for script development, which results in low development efficiency, script exceptions, and high learning costs, a SeaTunnel-based data offline synchronization mode is constructed, which reduces the development difficulty, defines the specifications of SeaTunnel scripts, reduces the script exception rate, and thus improves development efficiency, and can solve the problems of difficult development, easy errors, and low efficiency.

[0162] Embodiment 3

[0163] Further references Fig.12 , as a response to the above Figure 1 The present invention provides an embodiment of a data offline synchronization device based on SeaTunnel. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0164] like Fig.12 As shown, the SeaTunnel-based data offline synchronization device 50 of this embodiment includes: a configuration module 51, a binding module 52, a script module 53 and a synchronization module 54. Among them:

[0165] The configuration module 51 is used to formulate a visualization parameter configuration component page according to the connection component Connector and the transformation component Transform in SeaTunnel;

[0166] A binding module 52, used to bind components to generate strategies according to page component configuration;

[0167] The script module 53 is used to generate a SeaTunnel script according to the component generation strategy;

[0168] The synchronization module 54 is used to synchronize the generated SeaTunnel script to the DolphinScheduler task.

[0169] In some optional implementations of this embodiment, the configuration module 51 further includes: a first parameter acquisition module 511, used to obtain the SeaTunnel official website Connector component parameters; a second parameter acquisition module 512, used to obtain the SeaTunnel official website Transform component parameters; a type configuration module 513, used to configure the obtained Connector component parameters and Transform component parameters according to different component types.

[0170] The binding module 52 further includes: a page component configuration acquisition module 521, which is used to acquire the page component configuration; and a parsing module 521, which is used for the back end to perform configuration parsing of different components on the component generation strategy.

[0171] The script module 53 further includes: a grammar rule acquisition module 531, which is used to obtain the grammar rules of each component according to the official website document; and a script grammar generation module 532, which is used for the backend to generate the SeaTunnel script grammar according to the grammar rules after parsing the configuration.

[0172] The synchronization module 54 further includes: a table synchronization module 541, which is used to synchronize DolphinScheduler. Five tables of the DolphinScheduler system need to be operated, namely, the task definition record table t_ds_task_definition_log, the task definition table t_ds_task_definition, the process definition record table t_ds_process_definition_log, the process definition table t_ds_process_definition, and the process task relationship table t_ds_process_task_relation; a table record storage module 542, which is used to store the newly added SeaTunnel script tasks in table records according to each table field.

[0173] The implementation of this embodiment has the following beneficial effects: by integrating SeaTunnel and DolphinScheduler, SeaTunnel script configuration is performed in a visual manner, data collection and task scheduling channels are opened up, Seatunnel custom tasks can be configured in the front-end interface, saved and submitted to the distributed task scheduler DolphinScheduler, and data offline task scheduling can be performed in a workflow manner. Compared with the prior art SeaTunnel script development process, in which developers and business personnel need to refer to SeaTunnel official documents for script development, which results in low development efficiency, script exceptions, and high learning costs, a SeaTunnel-based data offline synchronization mode is constructed, which reduces the development difficulty, defines the specifications of SeaTunnel scripts, reduces the script exception rate, and thus improves development efficiency, and can solve the problems of difficult development, easy errors, and low efficiency.

[0174] Embodiment 4

[0175] To solve the above technical problems, the embodiment of the present invention also provides a computer device. Fig.13 , Fig.13 This is a basic structural block diagram of the computer device in this embodiment.

[0176] The above-mentioned computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 6 having components of a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0177] The computer device may be a desktop computer, a notebook computer, a PDA, a cloud server, etc. The computer device may interact with the user through a keyboard, a mouse, a remote control, a touch pad, or a voice control device.

[0178] The memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 6. Of course, the memory 61 may also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions of the data offline synchronization method based on SeaTunnel. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0179] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run the computer-readable instructions stored in the memory 61 or process data, such as running the computer-readable instructions of the SeaTunnel-based data offline synchronization method.

[0180] The network interface 63 may include a wireless network interface or a wired network interface. The network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0181] The implementation of this embodiment has the following beneficial effects: by integrating SeaTunnel and DolphinScheduler, SeaTunnel script configuration is performed in a visual manner, data collection and task scheduling channels are opened up, Seatunnel custom tasks can be configured in the front-end interface, saved and submitted to the distributed task scheduler DolphinScheduler, and data offline task scheduling can be performed in a workflow manner. Compared with the prior art SeaTunnel script development process, in which developers and business personnel need to refer to SeaTunnel official documents for script development, which results in low development efficiency, script exceptions, and high learning costs, a SeaTunnel-based data offline synchronization mode is constructed, which reduces the development difficulty, defines the specifications of SeaTunnel scripts, reduces the script exception rate, and thus improves development efficiency, and can solve the problems of difficult development, easy errors, and low efficiency.

[0182] Embodiment 5

[0183] The present invention also provides another embodiment, that is, providing a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the SeaTunnel-based data offline synchronization method as described above.

[0184] The implementation of this embodiment has the following beneficial effects: by integrating SeaTunnel and DolphinScheduler, SeaTunnel script configuration is performed in a visual manner, data collection and task scheduling channels are opened up, Seatunnel custom tasks can be configured in the front-end interface, saved and submitted to the distributed task scheduler DolphinScheduler, and data offline task scheduling can be performed in a workflow manner. Compared with the prior art SeaTunnel script development process, in which developers and business personnel need to refer to SeaTunnel official documents for script development, which results in low development efficiency, script exceptions, and high learning costs, a SeaTunnel-based data offline synchronization mode is constructed, which reduces the development difficulty, defines the specifications of SeaTunnel scripts, reduces the script exception rate, and thus improves development efficiency, and can solve the problems of difficult development, easy errors, and low efficiency.

[0185] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the various embodiment methods of the present invention.

[0186] Obviously, the embodiments described above are only some embodiments of the present invention, rather than all embodiments. The preferred embodiments of the present invention are given in the accompanying drawings, but they do not limit the patent scope of the present invention. The present invention can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive. Although the present invention has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific embodiments, or to perform equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of the present invention, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present invention.

Claims

1. A SeaTunnel-based data offline synchronization method, characterized in that: The steps include: According to the Connector component and Transform component in SeaTunnel, formulate the component page for visual parameter configuration; Bind component generation strategy based on page component configuration; Generate a SeaTunnel script according to the component generation strategy; Synchronize the generated SeaTunnel script to the DolphinScheduler task.

2. The SeaTunnel-based data offline synchronization method according to claim 1, characterized in that: The steps of formulating a visualization parameter configuration component page according to the connection component Connector and the transformation component Transform in SeaTunnel specifically include: Get the parameters of the SeaTunnel official website Connector component; Get the Transform component parameters of the SeaTunnel official website; The obtained Connector component parameters and Transform component parameters are configured according to different component types.

3. The SeaTunnel-based data offline synchronization method according to claim 1, characterized in that: The step of binding components to generate strategies according to page component configuration specifically includes: Get page component configuration; The backend performs configuration analysis of different components based on the component generation strategy.

4. The SeaTunnel-based data offline synchronization method according to claim 3, characterized in that: The step of generating a SeaTunnel script according to the component generation strategy specifically includes: Obtain the grammatical rules of each component according to the official website documentation; The backend will parse the configuration and generate SeaTunnel script syntax according to the syntax rules.

5. The SeaTunnel-based data offline synchronization method according to claim 1, characterized in that: The step of synchronizing the generated SeaTunnel script to the DolphinScheduler task specifically includes: Synchronizing DolphinScheduler requires operating five tables of the DolphinScheduler system, namely, task definition record table t_ds_task_definition_log, task definition table t_ds_task_definition, process definition record table t_ds_process_definition_log, process definition table t_ds_process_definition, process task relationship table t_ds_process_task_relation; The newly added SeaTunnel script tasks are stored in table records according to each table field.

6. The SeaTunnel-based data offline synchronization method according to claim 5, characterized in that: After the step of synchronizing the generated SeaTunnel script to the DolphinScheduler task, the method further includes: Submit the defined task and check whether the defined task exists in DolphinScheduler.

7. A data offline synchronization device based on SeaTunnel, characterized in that: include: The configuration module is used to formulate the visualization parameter configuration component page according to the connection component Connector and the transformation component Transform in SeaTunnel; The binding module is used to bind components to generate strategies according to page component configuration; A script module, used to generate a SeaTunnel script according to the component generation strategy; The synchronization module is used to synchronize the generated SeaTunnel script to the DolphinScheduler task.

8. The SeaTunnel-based data offline synchronization device according to claim 7, characterized in that: The configuration module includes: The first parameter acquisition module is used to obtain the parameters of the SeaTunnel official website Connector component; The second parameter acquisition module is used to obtain the Transform component parameters of the SeaTunnel official website; The type configuration module is used to configure the obtained Connector component parameters and Transform component parameters according to different component types.

9. A computer device, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the SeaTunnel-based data offline synchronization method are implemented as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the SeaTunnel-based data offline synchronization method are implemented as described in any one of claims 1 to 6.

Images