User-defined plug-in based on DataX framework and development method thereof
By developing custom plug-ins in the DataX framework, the read and write of Geometry-type data of PostgreSQL database is solved, and the problem that DataX does not support Geometry-type data is simplified, and the data transmission process is reduced and the cost is reduced.
Patent Information
- Application Number
- CN202411778387.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-05
- Publication Date
- 2025-03-28
AI Technical Summary
The current version of DataX does not support the read and write of Geometry-type data of PostgreSQL database, which makes it impossible to synchronize and transmit Geometry-type data, which increases the complexity and implementation cost of the project.
Develop a custom plug-in based on the DataX framework, including a common read plug-in for relational databases, a universal write plug-in for relational databases and a PostgreSQL database write plug-in. These plug-ins realize the reading and writing of PostgreSQL data source Geometry format data.
It effectively expands the functions of the DataX framework, allowing it to support PostgreSQL's Geometry-type data reading and writing, simplifies the data transmission process, reduces implementation costs, and improves user experience.
Smart Images
Figure CN119718463A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data technology, and in particular to a custom plug-in based on a DataX framework and a development method thereof. Background Art
[0002] PostgreSQL is a widely used open source object-relational database management system that supports multiple data types and allows users to define data types and functions. It is widely used in small applications to large enterprise solutions. Geometry type data is used to represent spatial data, such as points, lines, surfaces, and other geometric figures in two-dimensional or three-dimensional space. By installing the PostGIS component on the PostgreSQL database, the PostgreSQL database can store, query, and analyze geospatial data.
[0003] DataX is an open source data transfer tool that is mainly used for efficient data synchronization and transmission between different data sources. It supports a variety of data sources, including relational databases such as PostgreSQL, NoSQL databases, and big data storage, and is suitable for large-scale data migration and integration projects.
[0004] However, the current version of Datax does not support the reading and writing of Geometry type data in the PostgreSQL database. Data tools based on the DataX framework cannot synchronize and transmit Geometry type data, causing users to choose other data tools to implement functions, increasing the complexity and implementation cost of the project.
[0005] In order to increase the data types that the DataX framework can adapt to the PostgreSQL database, the present invention proposes a custom plug-in based on the DataX framework and a development method thereof. Summary of the invention
[0006] In order to remedy the deficiencies of the prior art, the present invention provides a simple and efficient custom plug-in based on the DataX framework and a development method thereof.
[0007] The present invention is achieved through the following technical solutions:
[0008] A custom plug-in based on the DataX framework and a development method thereof, characterized in that: it is used to implement the reading and writing of Geometry format data of a PostgreSQL data source, including:
[0009] A general read plug-in for relational databases, used to read Geometry format data from PostgreSQL data sources;
[0010] A general write plug-in for relational databases, used to write Geometry format data into a PostgreSQL database;
[0011] PostgreSQL database writing plug-in, used to handle special writing of Geometry type data.
[0012] The relational database universal reading plug-in is responsible for reading data from the data source and obtaining query results through the ResultSet interface; when processing the data of the ResultSet interface, if it is detected that the data type of the current column is Geometry, it is read as a string in WKT (Well-KnownText) format;
[0013] Use the getObject() method of the ResultSet interface to obtain the column object. If it is not empty, convert it into a WKT string. Then, add the obtained WKT string to the Record object, encapsulate it with the string column StringColumn, write it to the RecordSender component, and write the data into the cache queue through the RecordSender component.
[0014] The relational database universal write plug-in is responsible for reading the data sent by the relational database universal read plug-in through the RecordReceiver component (record receiver), and writing it to the target data source, using the PreparedStatement method to fill in the data.
[0015] The PostgreSQL database writing plug-in is responsible for decoding the WKB format geometry data into binary format through the built-in decode function of the PostgreSQL database when the column data type is detected to be in Geometry format during the data preparation process, and then using the function of the PostGIS component to construct a geometry object to convert the input binary geometry data into the Geometry data type of the PostGIS component.
[0016] A method for developing a custom plug-in based on the DataX framework includes the following steps:
[0017] Step S1, selecting a PostGIS extension component of a corresponding version according to the server PostgreSQL database version, and enabling the PostGIS extension function for the PostgreSQL database to implement the database geospatial function, allowing users to store, query and process spatial data including geographic coordinates and map data;
[0018] Step S2, writing Task class plug-in codes, including writing relational database general read plug-in, relational database general write plug-in and PostgreSQL database write plug-in;
[0019] The Task class is responsible for task initialization, data reading / writing, local post-operation, and destruction. The entry class of the plugin must extend the Reader or Writer abstract class and implement the Task internal abstract class.
[0020] The relational database universal reading plug-in obtains the query result through the ResultSet interface, and when the ResultSet interface detects that the data type of the current column is in Geometry format, reads it as a string in WKT format and writes it into a Record object;
[0021] The relational database general write plug-in reads data through the RecordReceiver component, writes the RecordReceiver component to the target data source, and fills the data using the PreparedStatement method;
[0022] The PostgreSQL database writing plug-in processes various data types through a switch statement during data preparation; when it is detected that the column data type is in Geometry format, the geometry data in WKB format is decoded into binary format through the decode function built into the PostgreSQL database, and then the function of constructing geometry objects of the PostGIS component is used to convert the input binary geometry data into the Geometry data type of the PostGIS component;
[0023] Step S3: Package the above plug-in written according to the DataX plug-in specification and put it into the server plug-in directory;
[0024] Step S4: Write the task configuration JSON file and execute the read and write tasks.
[0025] In the step S2, a general read plug-in for a relational database is written to add a Geometry format data reading function;
[0026] The relational database universal reading plug-in reads data from the data source and obtains the query result through the ResultSet interface; when processing the data of the ResultSet interface, if it is detected that the data type of the current column is Geometry, it is read as a string in WKT (Well-KnownText) format;
[0027] Use the getObject() method of the ResultSet interface to obtain the column object. If it is not empty, convert it into a WKT string. Then, add the obtained WKT string to the Record object, encapsulate it with the string column StringColumn, write it to the RecordSender component, and write the data into the cache queue through the RecordSender component.
[0028] In the step S2, a general writing plug-in for a relational database is written to add a Geometry format data writing function;
[0029] The data is read from the source data by the relational database universal read plug-in and sent to the RecordReceiver component through the RecordSender component.
[0030] It writes to the target data source;
[0031] The relational database general write plug-in traverses each Record, fills in the PreparedStatement, and prepares the data.
[0032] In step S3, the Maven build tool is used to package all dependencies and codes of the project into an independent executable JAR file through the plug-in maven-assembly-plugin.
[0033] In step S4, the task configuration JSON file is used to define the global configuration of the synchronization task, including the transmission speed limit and the number of tolerant error records, and specifies the relational database general read plug-in configuration to read the database connection information, and uses the querySql mode to define the SQL query field; specifies the relational database general write plug-in and the PostgreSQL database write plug-in, configures the write database connection information, executes the task to write data, and completes the data synchronization task.
[0034] The beneficial effects of the present invention are: the custom plug-in based on the DataX framework and its development method effectively expand the functionality of the DataX framework, enabling it to support the reading and writing of Geometry type data of PostgreSQL, simplifying the data transmission process, reducing implementation costs, and improving user experience. Through unified data processing tools, users can more efficiently perform large-scale data migration and integration, providing a more flexible and feasible solution for the digital transformation of the industry. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0036] Attached Figure 1 Schematic diagram of the development method of a custom plug-in based on the DataX framework of the present invention. DETAILED DESCRIPTION
[0037] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0038] This custom plug-in based on the DataX framework is used to read and write data in Geometry format of PostgreSQL data source, including:
[0039] A general read plug-in for relational databases, used to read Geometry format data from PostgreSQL data sources;
[0040] A general write plug-in for relational databases, used to write Geometry format data into a PostgreSQL database;
[0041] PostgreSQL database writing plug-in, used to handle special writing of Geometry type data.
[0042] The relational database universal reading plug-in is responsible for reading data from the data source and obtaining query results through the ResultSet interface; when processing the data of the ResultSet interface, if it is detected that the data type of the current column is Geometry, it is read as a string in WKT (Well-KnownText) format;
[0043] Use the getObject() method of the ResultSet interface to obtain the column object. If it is not empty, convert it into a WKT string. Then, add the obtained WKT string to the Record object, encapsulate it with the string column StringColumn, write it to the RecordSender component, and write the data into the cache queue through the RecordSender component.
[0044] The relational database universal write plug-in is responsible for reading the data sent by the relational database universal read plug-in through the RecordReceiver component (record receiver), and writing it to the target data source, using the PreparedStatement method to fill in the data.
[0045] The PostgreSQL database writing plug-in is responsible for decoding the WKB format geometry data into binary format through the built-in decode function of the PostgreSQL database when the column data type is detected to be in Geometry format during the data preparation process, and then using the function of the PostGIS component to construct a geometry object to convert the input binary geometry data into the Geometry data type of the PostGIS component.
[0046] The development method of the custom plug-in based on the DataX framework includes the following steps:
[0047] Step S1, selecting a PostGIS extension component of a corresponding version according to the server PostgreSQL database version, and enabling the PostGIS extension function for the PostgreSQL database to implement the database geospatial function, allowing users to store, query and process spatial data including geographic coordinates and map data;
[0048] Step S2, writing Task class plug-in codes, including writing relational database general read plug-in, relational database general write plug-in and PostgreSQL database write plug-in;
[0049] The Task class is responsible for task initialization, data reading / writing, local post-operation, and destruction. The entry class of the plugin must extend the Reader or Writer abstract class and implement the Task internal abstract class.
[0050] The relational database universal reading plug-in obtains the query result through the ResultSet interface, and when the ResultSet interface detects that the data type of the current column is in Geometry format, reads it as a string in WKT format and writes it into a Record object;
[0051] The relational database general write plug-in reads data through the RecordReceiver component, writes the RecordReceiver component to the target data source, and fills the data using the PreparedStatement method;
[0052] The PostgreSQL database writing plug-in processes various data types through a switch statement during data preparation; when it is detected that the column data type is in Geometry format, the geometry data in WKB format is decoded into binary format through the decode function built into the PostgreSQL database, and then the function of constructing geometry objects of the PostGIS component is used to convert the input binary geometry data into the Geometry data type of the PostGIS component;
[0053] Step S3: Package the above plug-in written according to the DataX plug-in specification and put it into the server plug-in directory;
[0054] Step S4: Write the task configuration JSON file and execute the read and write tasks.
[0055] In the step S2, a general read plug-in for a relational database is written to add a Geometry format data reading function;
[0056] The relational database universal reading plug-in reads data from the data source and obtains the query result through the ResultSet interface; when processing the data of the ResultSet interface, if it is detected that the data type of the current column is Geometry, it is read as a string in WKT (Well-KnownText) format;
[0057] Use the getObject() method of the ResultSet interface to obtain the column object. If it is not empty, convert it into a WKT string. Then, add the obtained WKT string to the Record object, encapsulate it with the string column StringColumn, write it to the RecordSender component, and write the data into the cache queue through the RecordSender component.
[0058] In the step S2, a general writing plug-in for a relational database is written to add a Geometry format data writing function;
[0059] The data is read from the source data by the relational database universal read plug-in and sent to the RecordReceiver component through the RecordSender component.
[0060] It writes to the target data source;
[0061] The relational database general write plug-in traverses each Record, fills in the PreparedStatement, and prepares the data.
[0062] In step S3, the Maven build tool is used to package all dependencies and codes of the project into an independent executable JAR file through the plug-in maven-assembly-plugin.
[0063] In step S4, the task configuration JSON file is used to define the global configuration of the synchronization task, including the transmission speed limit and the number of tolerant error records, and specifies the relational database general read plug-in configuration to read the database connection information, and uses the querySql mode to define the SQL query field; specifies the relational database general write plug-in and the PostgreSQL database write plug-in, configures the write database connection information, executes the task to write data, and completes the data synchronization task.
[0064] This custom plug-in based on the DataX framework and its development method effectively solves the problem of insufficient support for Geometry type data in the PostgreSQL database by DataX. By writing a custom plug-in, the DataX framework is successfully extended to support the reading and writing of Geometry type data in the PostgreSQL database. Users can use a unified data transfer tool (DataX) to read and write various data types, reducing the need to switch between multiple tools, thereby simplifying the data transfer process, reducing development and maintenance costs, and improving user experience and work efficiency.
[0065] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.
Claims
1. A custom plug-in based on the DataX framework, characterized by: Used to read and write data in Geometry format of PostgreSQL data source, including: A general read plug-in for relational databases, used to read Geometry format data from PostgreSQL data sources; A general write plug-in for relational databases, used to write Geometry format data into a PostgreSQL database; PostgreSQL database writing plug-in, used to handle special writing of Geometry type data.
2. The custom plug-in based on the DataX framework according to claim 1, characterized in that: The relational database universal reading plug-in is responsible for reading data from the data source and obtaining query results through the ResultSet interface; when processing the data of the ResultSet interface, if it is detected that the data type of the current column is Geometry, it is read as a string in WKT format; Use the getObject() method of the ResultSet interface to obtain the column object. If it is not empty, convert it into a WKT string. Then, add the obtained WKT string to the Record object, encapsulate it with the string column StringColumn, write it to the RecordSender component, and write the data into the cache queue through the RecordSender component.
3. The custom plug-in based on the DataX framework according to claim 1, characterized in that: The relational database universal write plug-in is responsible for reading the data sent by the relational database universal read plug-in through the component in RecordReceiver, and writing it to the target data source, using the PreparedStatement method to fill in the data.
4. The custom plug-in based on the DataX framework according to claim 1, characterized in that: The PostgreSQL database writing plug-in is responsible for decoding the WKB format geometry data into binary format through the built-in decode function of the PostgreSQL database when the column data type is detected to be in Geometry format during the data preparation process, and then using the function of the PostGIS component to construct a geometry object to convert the input binary geometry data into the Geometry data type of the PostGIS component.
5. A method for developing a custom plug-in based on the DataX framework, characterized in that: The following steps are involved: Step S1, selecting a PostGIS extension component of a corresponding version according to the server PostgreSQL database version, and enabling the PostGIS extension function for the PostgreSQL database to implement the database geospatial function, allowing users to store, query and process spatial data including geographic coordinates and map data; Step S2, writing Task class plug-in codes, including writing relational database general read plug-in, relational database general write plug-in and PostgreSQL database write plug-in; The relational database universal reading plug-in obtains the query result through the ResultSet interface, and when the ResultSet interface detects that the data type of the current column is in Geometry format, reads it as a string in WKT format and writes it into the Record object; The relational database general writing plug-in reads data through the RecordReceiver component, writes the RecordReceiver component to the target data source, and fills the data using the PreparedStatement method; The PostgreSQL database writing plug-in processes various data types through a switch statement during data preparation; when it is detected that the column data type is in Geometry format, the geometry data in WKB format is decoded into binary format through the decode function built into the PostgreSQL database, and then the function of constructing geometry objects of the PostGIS component is used to convert the input binary geometry data into the Geometry data type of the PostGIS component; Step S3: Package the above plug-in written according to the DataX plug-in specification and put it into the server plug-in directory; Step S4: Write the task configuration JSON file and execute the read and write tasks.
6. The method for developing a custom plug-in based on the DataX framework according to claim 5, characterized in that: In the step S2, a general read plug-in for a relational database is written to add a Geometry format data reading function; The relational database universal reading plug-in reads data from the data source and obtains the query result through the ResultSet interface; when processing the data of the ResultSet interface, if it is detected that the data type of the current column is Geometry, it is read as a string in WKT format; Use the getObject() method of the ResultSet interface to obtain the column object. If it is not empty, convert it into a WKT string. Then, add the obtained WKT string to the Record object, encapsulate it with the string column StringColumn, write it to the RecordSender component, and write the data into the cache queue through the RecordSender component.
7. The method for developing a custom plug-in based on the DataX framework according to claim 5, characterized in that: In the step S2, a general writing plug-in for a relational database is written to add a Geometry format data writing function; The data is read from the source data by the relational database universal read plug-in and sent to the RecordReceiver component through the RecordSender component. The relational database universal write plug-in reads the data and writes it to the target data source. The relational database general write plug-in traverses each Record and fills the PreparedStatement to prepare data.
8. The method for developing a custom plug-in based on the DataX framework according to claim 5, characterized in that: In step S3, the Maven build tool is used to package all dependencies and codes of the project into an independent executable JAR file through the plug-in maven-assembly-plugin.
9. The method for developing a custom plug-in based on the DataX framework according to claim 5, characterized in that: In step S4, the task configuration JSON file is used to define the global configuration of the synchronization task, including the transmission speed limit and the number of tolerant error records, and specifies the relational database general read plug-in configuration to read the database connection information, and uses the querySql mode to define the SQL query field; specifies the relational database general write plug-in and the PostgreSQL database write plug-in, configures the write database connection information, executes the task to write data, and completes the data synchronization task.