An operatorized multi-source big data processing method for cross-type migration

Abstract

The application provides an operatorized multi-source big data processing method for cross-type migration, which can solve the problem of complicated data integration operation of traditional data migration and realize programming-free data operation across data source types. The method comprises the following steps: step one, creating a data operation operator according to a data integration method of different types of data sources, configuring a data source in the data operation operator, and configuring metadata structure information of input data and output data according to a structure of the data source to obtain a data operation flow file; step two, using spark to read data in the data source as source data according to the data operation flow file and data operation operator configuration information, and performing type conversion on the read data set according to the source data; and step three, performing data processing by using each data operation operator according to a data processing operation flow drawn in the data operation flow file, and writing the data set into a target data source.

Description

Technical Field

[0001] This invention belongs to the field of big data processing technology and relates to an operator-based multi-source big data processing method for cross-type migration. Background Technology

[0002] As business demands continue to expand, data scale becomes increasingly complex, and data sources become more diversified. In daily data development and practical applications, data may originate from multiple data sources, and depending on the usage scenario, data is often stored in different types of databases. In this situation, not only are there potential inconsistencies in field types between different databases, but the structural differences brought about by heterogeneous databases also need to be addressed. This makes using heterogeneous data sources and data migration a massive undertaking, forcing developers to spend a significant amount of time dealing with data inconsistencies and performance issues, which greatly impacts data analysis and processing.

[0003] Traditional data migration or data integration operations often require writing specialized programs for data reading, writing, and processing. When multiple data sources are involved, it is necessary to consider not only the matching of field types in the data sources themselves, but also the differences in data storage structures between relational and non-relational databases. This makes the code for data operations very cumbersome and prevents rapid development to achieve production purposes. Summary of the Invention

[0004] To address the aforementioned shortcomings, this invention proposes an operator-based multi-source big data processing method for cross-type migration, which can solve the problem of cumbersome data integration operations such as traditional data migration and realize cross-data source type data operations without programming.

[0005] The present invention is achieved through the following technical solution.

[0006] An operator-based multi-source big data processing method for cross-type transfer includes:

[0007] Step 1: Create data operation operators according to the data integration methods of different types of data sources, configure the data source in the data operation operator, and configure the metadata structure information of the input data and output data according to the structure of the data source to obtain the data operation process file;

[0008] Step 2: Based on the data operation process file and its data operation operator configuration information, use Spark to read data from the data source as source data; perform type conversion on the read dataset based on the source data to finally obtain the data stream to be operated on, and transmit it through the output port defined by the operator;

[0009] Step 3: According to the data processing operation flow drawn in the data operation flow file, use each data operation operator to process the data according to the flow. The data transfer between the operators is adapted through the ports defined by each operator. Finally, the processed operator data is converted into a dataset that meets the field structure of the target data source, and the dataset is written into the target data source.

[0010] The beneficial effects of this invention are:

[0011] Compared to existing technologies, this invention encapsulates the actual data processing process and code logic using an operator-based approach, allowing business personnel to obtain data integration and processing results in a simpler way. The data processing process provides timely feedback on data operation status and allows real-time viewing of data operation results, facilitating user control over the data processing process. Simultaneously, this invention supports data type conversion and data migration between heterogeneous data sources, avoiding the cumbersome type adaptation operations in traditional heterogeneous data migration processes. Furthermore, the provided big data processing strategies ensure the smooth completion of large-volume data operations, ultimately achieving non-programming cross-heterogeneous data source type data operations. Detailed Implementation

[0012] The present invention will now be described in further detail.

[0013] This specific embodiment of an operator-based multi-source big data processing method for cross-type migration includes:

[0014] Step 1: Create data operation operators according to the data integration methods of different types of data sources, configure the data source in the data operation operator, and configure the metadata structure information of the input data and output data according to the structure of the data source to obtain the data operation process file;

[0015] The input data refers to the data read from the selected data source, and the output data refers to the data to be stored in the destination data source or the data to be processed. The data operation operators are adapted and connected through ports, which are used to identify the application type of the data flowing in the operator, including datasets, graphs, texts, JSONs, etc.

[0016] Step 2: Based on the data operation process file and its data operation operator configuration information, use Spark to read data from the data source as source data; perform type conversion on the read dataset based on the source data to finally obtain the data stream to be operated on, and transmit it through the output port defined by the operator;

[0017] In this embodiment, the data read from the data source using Spark includes JDBC driver, database authorization information, database name, table name, data range, metadata information, etc.

[0018] Step 3: According to the data processing operation flow drawn in the data operation flow file, use each data operation operator to process the data according to the flow. The data transfer between the operators is adapted through the ports defined by each operator. Finally, the processed operator data is converted into a dataset that meets the field structure of the target data source, and the dataset is written into the target data source.

[0019] In this embodiment, the data operation operators include, in addition to input and output related operators, preprocessing operators such as data filtering, deduplication, sorting, format conversion, and missing value handling; statistical analysis operators such as data retrieval and conditional filtering; machine learning operators such as classification, clustering, and regression; and other data business operators.

[0020] In this embodiment, the data processing using various data operation operators specifically includes the following steps:

[0021] Step 1: Configure the data operation operators according to the data source type and data field type in the data operation process file, and determine the metadata structure of the selected data;

[0022] In this embodiment, the data operation operator includes an operator configuration HTML file and a JAR file exported from the execution code entity. Specifically, database read / write operations of the data source type can be encapsulated using operators, including relational databases like MySQL, non-relational databases like MongoDB, Elasticsearch, and HBase, as well as HDFS and various local document read / write operations.

[0023] Step 2: Connect the data operation operators to process and improve the data content, determine the execution order and data flow of the data operation operators, generate the DAG graph corresponding to the data import and export process and its corresponding JSON data, and submit the execution process request.

[0024] In this embodiment, the data content is improved by connecting data retrieval statistics operators with graphical operators such as bar charts and line charts.

[0025] Step 3: The process execution service module receives the execution process request, obtains the DAG graph data, and parses out the selected operators, operator configuration parameters, and operator execution order in the process. Based on the parsing results, it executes each data operator in sequence, performs type processing according to the data source structure type and record metadata in the operator, reads or writes the dataset, and finally completes the data processing process.

[0026] In this embodiment, when reading or writing the dataset, in order to avoid service and process interruption due to excessive server memory usage during data processing, a batch import strategy is adopted, that is, when reading data from the source data source, the dataset is pulled in batches and then imported into the target data source.

[0027] In this embodiment, the type processing in the operator according to the structure type of the data source and the metadata of the record specifically includes:

[0028] When processing MySQL structured databases, the SQL statement is concatenated by setting the data offset and the number of records to be read in a single operation. This SQL statement is then executed using JDBC to retrieve the dataset. The same method is then used to concatenate an INSERT statement to insert the data into the dataset. However, setting the data offset can lead to a full table scan, which is inefficient. Therefore, when importing or exporting data from tables with a unique primary key, a pointer-based approach is used for data location. The WHERE clause determines the starting position of the data to be read, ensuring that records outside the target range are not considered during data retrieval.

[0029] When processing unstructured databases, using RestHighLevelClient to read Elasticsearch data typically employs a `from + size` pagination method. However, this method is inefficient for deep pagination. Furthermore, due to the default Elasticsearch limit of 10,000 records per read, a `scroll` approach is needed, using a cursor to read data in batches until the `hits` field in the result set is empty. Similarly, when reading HBase data, since the total number of records cannot be obtained, and using `from + size` is too inefficient, a cursor is also required. `scanner.next()` is used to read data in batches before importing or exporting.

[0030] When collaboratively processing import / export operations across different storage types, a dataset is used to transform the source dataset into a structured dataset before importing it into the target data source. During operator configuration, metadata information is read based on the selected data source structure, and the metadata type is configured in the interface, converting it to common data types. The metadata is then finally configured according to the field requirements of the target data and the field types of the destination data source. During operator execution, the dataset's data type is converted based on changes in the configured metadata. The `withColumn()` and `withColumnRenamed()` methods provided by the dataset are used for data type conversion or data field renaming, ultimately achieving the goal of collaborative processing of heterogeneous data sources.

[0031] During collaborative processing, when reading from Elasticsearch, there may be a situation where null values ​​cannot be read. You need to configure the parameter "es.field.read.empty.as.null" to "no" in the API call. When reading from HBase, you need to first read the data as a JavaPairRDD and then further convert it into a dataset, etc.

[0032] Although embodiments of the present invention have been described, those skilled in the art can make several improvements without departing from the principles of the present invention, and these improvements should also be considered to fall within the scope of protection of the present invention.

Claims

1. An operator-based multi-source big data processing method for cross-type transfer, characterized in that, include: Step 1: Create data operation operators according to the data integration methods of different types of data sources, configure the data source in the data operation operator, and configure the metadata structure information of the input data and output data according to the structure of the data source to obtain the data operation process file; Step 2: Based on the data operation process file and its data operation operator configuration information, use Spark to read data from the data source as the source data; The type of the read dataset is converted based on the source data to obtain the data stream to be operated on, and then transmitted through the output port defined by the operator. Step 3: According to the data processing operation flow drawn in the data operation flow file, use each data operation operator to process the data according to the flow. The data transfer between the operators is adapted through the ports defined by each operator. Finally, the processed operator data is converted into a dataset that meets the field structure of the target data source, and the dataset is written into the target data source. The data processing using various data manipulation operators specifically includes the following steps: Step 1: Configure the data operation operators according to the data source type and data field type in the data operation process file, and determine the metadata structure of the selected data; Step 2: Connect the data operation operators to process and improve the data content, determine the execution order and data flow of the data operation operators, generate the DAG graph corresponding to the data import and export process and its corresponding JSON data, and submit the execution process request. Step 3: The process execution service module receives the execution process request, obtains the DAG graph data, and parses out the selected operators, operator configuration parameters, and operator execution order in the process. Based on the parsing results, it executes each data operator in sequence, performs type processing according to the data source structure type and record metadata in the operator, reads or writes the dataset, and finally completes the data processing process.

2. The operator-based multi-source big data processing method for cross-type migration as described in claim 1, characterized in that, The input data refers to the data read from the selected data source, and the output data refers to the data to be stored in the destination data source or the data to be processed; the data operation operators are adapted and connected through ports.

3. The operator-based multi-source big data processing method for cross-type migration as described in claim 2, characterized in that, The port is used to identify the application type of the data flowing in the operator, including dataset, graph, text, and json.

4. The operator-based multi-source big data processing method for cross-type migration as described in claim 1, 2, or 3, characterized in that, The data read from the data source using Spark includes the JDBC driver, database authorization information, database name, table name, data range, and metadata information.

5. The operator-based multi-source big data processing method for cross-type migration as described in claim 1, characterized in that, The data manipulation operators include an operator configuration HTML file and a JAR file exported from the execution code entity.

6. The operator-based multi-source big data processing method for cross-type migration as described in claim 5, characterized in that, Operator encapsulation is performed on database read and write operations of the data source types, including relational database MySQL, non-relational databases MongoDB, Elasticsearch, HBase, HDFS, and various types of local documents.

7. The operator-based multi-source big data processing method for cross-type migration as described in claim 1, 5, or 6, characterized in that, The data content is improved by using data retrieval statistical operators and graphical operators such as bar charts and line charts.

8. The operator-based multi-source big data processing method for cross-type migration as described in claim 1, 5, or 6, characterized in that, When reading or writing the dataset, a batch import strategy is adopted, that is, when reading data from the source data source, the dataset is pulled in batches and then imported into the target data source.

9. The operator-based multi-source big data processing method for cross-type migration as described in claim 1, 5, or 6, characterized in that, The type processing in the operator according to the structure type of the data source and the metadata of the record is as follows: When dealing with MySQL structured databases, the SQL statement is concatenated by setting the data offset and the number of data to be read at one time. The SQL statement is then executed using JDBC to obtain the dataset. Then, the same method is used to concatenate the INSERT statement to insert the dataset.

10. The operator-based multi-source big data processing method for cross-type migration as described in claim 9, characterized in that, The data offset setting uses a pointer-based approach for data positioning. The starting position of the data to be read is determined by the WHERE clause, so that data records outside the target range do not need to be considered when reading data.

11. The operator-based multi-source big data processing method for cross-type migration as described in claim 1, 5, or 6, characterized in that, The type processing in the operator according to the structure type of the data source and the metadata of the record is as follows: When performing collaborative import / export operations across storage types, the source dataset is transformed into a structured dataset using a dataset before being imported into the target data source. When configuring the operator, the metadata information of the data is read according to the selected data source structure, and the type of the metadata is configured in the interface to convert it into common data types; and the metadata is finally configured according to the field requirements of the target data and the field types of the destination data source. During operator execution, the data type of the dataset is converted according to the changes in the configured metadata. The data type conversion or data field renaming is performed through the withColumn() and withColumnRenamed() methods provided by the dataset, ultimately achieving the goal of collaborative processing of heterogeneous data sources.