Hudi data synchronization method based on DataX

Abstract

The invention particularly relates to a method for synchronizing hudi data based on Data X. The method for synchronizing the hudi data based on the Data X comprises the following steps of: compiling a package.xml file and a plugin.json file, and explaining basic information of a plug-in to a framework; writing a hudiwriter.java script based on a java language to prepare an object or a data stream for writing data, starting a write operation, and refreshing a cache; an auxiliary support type HudiWriterErrorCode.java file is compiled and used for capturing abnormities generated in the data transmission process, and error information is output to a log after being subjected to unified processing; and a framework of the Data X tool reads the file through a corresponding reserved interface, so that data extraction, conversion, loading and abnormal error reporting among the data sources are realized. According to the data X-based hudi data synchronization method, the data synchronization between the hudi and the relational library is realized by performing plug-in secondary development on the Data X tool, and the application range of the Data X tool in the field of big data integration is longitudinally extended.

Description

Technical Field

[0001] The present invention relates to the field of big data acquisition technology, and in particular to a method for hudi data synchronization based on Data X. Background Art

[0002] In the context of big data, there are many types of data, complex and huge content, and the processing speed is relatively fast, which are characteristics that traditional information data do not have. Especially with the development of information technology such as the Internet, big data has been widely used in all walks of life and has given full play to its advantages. However, with the in-depth mining of data value, privacy-sensitive information in the data is also facing the risk of being leaked and abused, so strengthening the identification and protection of sensitive data has become an urgent problem to be solved in the field of data security.

[0003] According to monitoring, the amount of global data doubles approximately every two years. The advent of big data has led to a technological revolution, prompting the rapid development of big data and cloud service technologies. Faced with the application of massive amounts of data in storage, mining, and analysis, users have an increasingly strong demand for big data security, and data security plays an increasingly important role. How to more safely and timely protect data privacy and sensitive data has become a challenge that the industry is increasingly concerned about.

[0004] Data X is an open-source data synchronization tool of Alibaba Cloud. It supports data synchronization between multiple data sources by default, but currently does not support data synchronization between real-time data warehouses (such as Hudi) and other data sources.

[0005] In order to realize data synchronization between the real-time data warehouse hudi and the relational database in Data X, the present invention proposes a hudi data synchronization method based on Data X. Summary of the invention

[0006] In order to make up for the defects of the prior art, the present invention provides a simple and efficient method for hudi data synchronization based on Data X.

[0007] The present invention is achieved through the following technical solutions:

[0008] A method for hudi data synchronization based on Data X, characterized in that it includes the following steps:

[0009] Step S1, write a package.xml file; the package.xml file is the configuration file of the framework scanning and identification plug-in plugin of the Data X tool;

[0010] The package.xml file is configured according to the path of the writer-plugin package and the path of the plugin.json file;

[0011] Step S2, write the plugin.json file, fill in the plugin name, class path, description and developer information in the plugin.json file to explain the basic information of the plugin to the framework;

[0012] Step S3: Write a hudiwriter.java script based on the Java language to prepare an object or data stream for writing data, start a write operation, and refresh the cache;

[0013] Step S4, write the auxiliary support class HudiWriterErrorCode.java file to capture the exceptions generated during the data transmission process, and output the error information to the log after unified processing;

[0014] Step S5: The framework of the Data X tool reads the package.xml file, plugin.json file, hudiwriter.java script, and HudiWriterErrorCode.java file through the corresponding reserved interfaces, thereby realizing data extraction, conversion, loading, and exception reporting between data sources.

[0015] In step S3, the implementation logic of the hudiwriter.java script is as follows:

[0016] 1) Writer initialization, prepare the object or data stream (writer) for writing data. All secondary development plug-ins inherit the object or data stream (writer) for writing data of the DataX tool;

[0017] 2) Start the write operation startwriter, start processing data, convert the data format, assemble the basic information of the data source, and write the processed data into the hudi data source;

[0018] The basic information of the data source includes data source type, data table and fields.

[0019] 3) Refresh the cache freshcache. Data is transmitted in the framework in the form of data stream. According to the needs of data transmission (new data, updated data), the data stream is written to the hudi data source in a new or updated manner.

[0020] A hudi data synchronization system based on Data X, comprising:

[0021] The package.xml file writing module is used to write the package.xml file and configure the framework scanning and identification plug-in of the Data X tool;

[0022] The package.xml file is configured according to the path of the writer-plugin package and the path of the plugin.json file;

[0023] The plugin.json file writing module is used to write the plugin.json file. Fill in the plugin name, class path, description and developer information in the plugin.json file to explain the basic information of the plugin to the framework.

[0024] The Java script writing module is used to write the hudiwriter.java script based on the Java language, prepare objects or data streams for writing data, start write operations, and refresh the cache;

[0025] Auxiliary support class file writing module, used to write the auxiliary support class HudiWriterErrorCode.java file, capture the exceptions generated during data transmission, and process the error information uniformly and output it to the log;

[0026] The framework interface of the Data X tool is used to read the package.xml file, plugin.json file, hudiwriter.java script, and HudiWriterErrorCode.java file, and then implement data extraction, conversion, loading, and exception reporting between data sources.

[0027] The implementation logic of the hudiwriter.java script is as follows:

[0028] 1) Writer initialization, prepare the object or data stream (writer) for writing data. All secondary development plug-ins inherit the object or data stream (writer) for writing data of the DataX tool;

[0029] 2) Start the write operation startwriter, start processing data, convert the data format, assemble the basic information of the data source, and write the processed data into the hudi data source;

[0030] The basic information of the data source includes data source type, data table and fields.

[0031] 3) Refresh the cache freshcache. Data is transmitted in the framework in the form of data stream. According to the needs of data transmission (new data, updated data), the data stream is written to the hudi data source in a new or updated manner.

[0032] A device for hudi data synchronization based on Data X, characterized in that it includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above method steps when executing the computer program.

[0033] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program implements the above method steps when executed by a processor.

[0034] The beneficial effect of the present invention is that the method for hudi data synchronization based on Data X realizes data synchronization between hudi and the relational database by secondary development of plug-ins for the Data X tool, and vertically extends the application scope of the Data X tool in the field of big data integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Attached Figure 1 It is a schematic diagram of the method for hudi data synchronization based on Data X of the present invention. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0038] The method for hudi data synchronization based on Data X includes the following steps:

[0039] Step S1, write a package.xml file; the package.xml file is the configuration file of the framework scanning and identification plug-in plugin of the Data X tool;

[0040] The package.xml file is configured according to the path of the writer-plugin package and the path of the plugin.json file; the plugin.json file describes the description of this plugin. The specific configuration information is as follows:

[0041] <assembly

[0042] xmlns="http: / / maven.apache.org / plugins / maven-assembly-plugin / assembly / 1.1.0"

[0043] xmlns:xsi="http: / / www.w3.org / 2001 / XMLSchema-instance"

[0044] xsi:schemaLocation="http: / / maven.apache.org / plugins / maven-assembly-plugin / assembly / 1.1.0http: / / maven.apache.org / xsd / assembly-1.1.0.xsd">

[0045] <id>< / id>

[0046] <formats>

[0047] <format>dir< / format>

[0048] < / formats>

[0049] <includebasedirectory>false< / includebasedirectory>

[0050] <filesets>

[0051] <fileset>

[0052] <directory>src / main / resources< / directory>

[0053] <includes>

[0054] <include>plugin.json< / include>

[0055] <include>plugin_job_template.json< / include>

[0056] < / includes>

[0057] <outputdirectory>plugin / writer / hudiwriter< / outputdirectory>

[0058] < / fileset>

[0059] <fileset>

[0060] <directory>target / < / directory>

[0061] <includes>

[0062] <include>hudiwriter-0.0.1-SNAPSHOT.jar< / include>

[0063] < / includes>

[0064] <outputdirectory>plugin / writer / hudiwriter< / outputdirectory>

[0065] < / fileset>

[0066] < / filesets>

[0067] <dependencysets>

[0068] <dependencyset>

[0069] <useprojectartifact>false< / useprojectartifact>

[0070] <outputdirectory>plugin / writer / hudiwriter / libs< / outputdirectory>

[0071] <scope>runtime< / scope>

[0072] < / dependencyset>

[0073] < / dependencysets>

[0074]

[0075] Step S2, write the plugin.json file, fill in the plugin name, class path, description and developer information in the plugin.json file to explain the basic information of the plugin to the framework; the specific content is as follows:

[0076] {

[0077] "name":"hudiwriter",

[0078] "class":"com.alibaba.datax.plugin.writer.hudiwriter.HudiWriter",

[0079] "description":"useScene:test.mechanism:use datax framework totransport data to hudi.warn:The more you know about the data,the lessproblems you encounter.",

[0080] "developer":"xxxxx"

[0081] }

[0082] Step S3: Write a hudiwriter.java script based on the Java language to prepare an object or data stream for writing data, start a write operation, and refresh the cache;

[0083] Step S4, write the auxiliary support class HudiWriterErrorCode.java file to capture the exceptions generated during the data transmission process, and output the error information to the log after unified processing;

[0084] Step S5: The framework of the Data X tool reads the package.xml file, plugin.json file, hudiwriter.java script, and HudiWriterErrorCode.java file through the corresponding reserved interfaces, thereby realizing data extraction, conversion, loading, and exception reporting between data sources.

[0085] In step S3, the implementation logic of the hudiwriter.java script is as follows:

[0086] 1) Writer initialization, prepare the object or data stream (writer) for writing data. All secondary development plug-ins inherit the object or data stream (writer) for writing data of the DataX tool;

[0087] 2) Start the write operation startwriter, start processing data, convert the data format, assemble the basic information of the data source, and write the processed data into the hudi data source;

[0088] The basic information of the data source includes data source type, data table and fields.

[0089] 3) Refresh the cache freshcache. Data is transmitted in the framework in the form of data stream. According to the needs of data transmission (new data, updated data), the data stream is written to the hudi data source in a new or updated manner.

[0090] This method of hudi data synchronization based on Data X redevelops the hudi plugin based on the Data X tool to support data synchronization between hudi and the relational database. Its underlying technology is implemented based on the framework of the Data X tool.

[0091] The framework is the core framework of the Data X tool. Data extraction, conversion, and loading between all data sources are implemented in this framework. The Framework reserves an interface protocol to support developers to secondary develop plugins for other data sources based on custom requirements and interface protocols. Developers can horizontally expand the data synchronization function between other data sources and Hudi and other data sources based on this.

[0092] The hudi data synchronization system based on Data X includes:

[0093] The package.xml file writing module is used to write the package.xml file and configure the framework scanning and identification plug-in of the Data X tool;

[0094] The package.xml file is configured according to the path of the writer-plugin package and the path of the plugin.json file; the plugin.json file describes the description information of this plugin.

[0095] The plugin.json file writing module is used to write the plugin.json file. Fill in the plugin name, class path, description and developer information in the plugin.json file to explain the basic information of the plugin to the framework.

[0096] The Java script writing module is used to write the hudiwriter.java script based on the Java language, prepare objects or data streams for writing data, start write operations, and refresh the cache;

[0097] Auxiliary support class file writing module, used to write the auxiliary support class HudiWriterErrorCode.java file, capture the exceptions generated during data transmission, and process the error information uniformly and output it to the log;

[0098] The framework interface of the Data X tool is used to read the package.xml file, plugin.json file, hudiwriter.java script, and HudiWriterErrorCode.java file, and then implement data extraction, conversion, loading, and exception reporting between data sources.

[0099] The implementation logic of the hudiwriter.java script is as follows:

[0100] 1) Writer initialization, prepare the object or data stream (writer) for writing data. All secondary development plug-ins inherit the object or data stream (writer) for writing data of the DataX tool;

[0101] 2) Start the write operation startwriter, start processing data, convert the data format, assemble the basic information of the data source, and write the processed data into the hudi data source;

[0102] The basic information of the data source includes data source type, data table and fields.

[0103] 3) Refresh the cache freshcache. Data is transmitted in the framework in the form of data stream. According to the needs of data transmission (new data, updated data), the data stream is written to the hudi data source in a new or updated manner.

[0104] The device for hudi data synchronization based on Data X includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above method steps when executing the computer program.

[0105] The readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method steps are implemented.

[0106] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for hudi data synchronization based on Data X, characterized in that: The following steps are involved: Step S1, write package.xml file; The package.xml file is the configuration file of the Data X tool framework scanning and identification plug-in plugin; The package.xml file is configured according to the path of the writer-plugin package and the path of the plugin.json file; Step S2, write the plugin.json file, fill in the plugin name, class path, description and developer information in the plugin.json file to explain the basic information of the plugin to the framework; Step S3: Write a hudiwriter.java script based on the Java language to prepare an object or data stream for writing data, start a write operation, and refresh the cache; Step S4, write the auxiliary support class HudiWriterErrorCode.java file to capture the exceptions generated during the data transmission process, and output the error information to the log after unified processing; Step S5: The framework of the Data X tool reads the package.xml file, plugin.json file, hudiwriter.java script, and HudiWriterErrorCode.java file through the corresponding reserved interfaces, thereby realizing data extraction, conversion, loading, and exception reporting between data sources.

2. The method for hudi data synchronization based on Data X according to claim 1 is characterized in that: In step S3, the implementation logic of the hudiwriter.java script is as follows: 1) Writer initialization, prepare the object or data stream for writing data. All secondary development plug-ins inherit the object or data stream for writing data of the DataX tool; 2) Start the write operation startwriter, start processing data, convert the data format, assemble the basic information of the data source, and write the processed data into the hudi data source; 3) Refresh the cache freshcache. Data is transmitted in the framework in the form of data stream. According to the needs of data transmission, the data stream is written to the hudi data source in a new or updated manner.

3. The method for hudi data synchronization based on Data X according to claim 2 is characterized in that: The basic information of the data source includes data source type, data table and fields.

4. A hudi data synchronization system based on Data X, characterized by: include: The package.xml file writing module is used to write the package.xml file and configure the framework scanning and identification plug-in of the Data X tool; The package.xml file is configured according to the path of the writer-plugin package and the path of the plugin.json file; The plugin.json file writing module is used to write the plugin.json file. Fill in the plugin name, class path, description and developer information in the plugin.json file to explain the basic information of the plugin to the framework. The Java script writing module is used to write the hudiwriter.java script based on the Java language, prepare objects or data streams for writing data, start write operations, and refresh the cache; Auxiliary support class file writing module, used to write the auxiliary support class HudiWriterErrorCode.java file, capture the exceptions generated during data transmission, and process the error information uniformly and output it to the log; The framework interface of the Data X tool is used to read the package.xml file, plugin.json file, hudiwriter.java script, and HudiWriterErrorCode.java file, and then implement data extraction, conversion, loading, and exception reporting between data sources.

5. The system for hudi data synchronization based on Data X according to claim 4 is characterized in that: The implementation logic of the hudiwriter.java script is as follows: 1) Writer initialization, prepare the object or data stream for writing data. All secondary development plug-ins inherit the object or data stream for writing data of the DataX tool; 2) Start the write operation startwriter, start processing data, convert the data format, assemble the basic information of the data source, and write the processed data into the hudi data source; 3) Refresh the cache freshcache. Data is transmitted in the framework in the form of data stream. According to the needs of data transmission, the data stream is written to the hudi data source in a new or updated manner.

6. The system for hudi data synchronization based on Data X according to claim 5 is characterized in that: The basic information of the data source includes data source type, data table and fields.

7. A device for hudi data synchronization based on Data X, characterized by: The method comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method according to any one of claims 1 to 3 when executing the computer program.

8. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images