Data integration method, device, equipment and product

Abstract

The invention provides a data integration method and device, equipment and a product, and the method comprises the steps: obtaining data integration configuration information which comprises task information and engine information for executing tasks; according to the engine information, determining a corresponding engine type, and according to the task information, obtaining a data integration rule of the corresponding engine type and generating a data integration statement; and executing the data integration statement to read the target data from the first data source, performing data conversion on the target data, and writing the target data into the second data source. The engine type is determined according to the obtained data integration configuration information, it is ensured that the subsequently obtained data integration rule is closely combined with the characteristics and functions of the selected engine, the data integration statement capable of being actually executed is generated according to the data integration rule, and the data integration statement is executed based on the selected engine; unified management and dynamic switching of the data integration engine are achieved, the accuracy and logicality of data processing are guaranteed, and the quality and efficiency of data integration are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular, to a data integration method, device, equipment and product. Background Art

[0002] With the development of database technology and the popularization of the network, on the one hand, a large amount of data is stored in databases, forming "information islands" that are not conducive to data sharing. On the other hand, with the intensification of global market competition, more and more information systems need to share data in databases, which requires the integration of data information.

[0003] Currently, in the data integration scenario, there are multiple sets of data integration engines. Due to the different design architectures of different data integration engines, it is impossible to achieve unified management and engine switching of multiple sets of engines. Users need additional learning costs to use different engines. Summary of the Invention

[0004] The present invention provides a data integration method, device, equipment and product, which is used to solve the defect that multiple sets of data integration engines in the prior art cannot be unifiedly managed, and realizes the unified management and dynamic switching of data integration engines.

[0005] The present invention provides a data integration method, including: obtaining data integration configuration information, where the data integration configuration information includes task information and engine information for executing the task. The task information is a data integration task configured based on transferring target data from a first data source to a second data source. The first data source is used to represent the database providing the target data, and the second data source is used to represent the database to receive the target data; determining the corresponding engine type according to the engine information, and obtaining the data integration rule corresponding to the engine type and generating a data integration statement according to the task information; where the data integration rule is a logical rule configured based on the corresponding task; executing the data integration statement to read the target data from the first data source, perform data conversion on the target data, and write it into the second data source.

[0006] A data integration method provided by the present invention, the data integration statement includes a data reading statement, a data conversion statement, and a data writing statement; according to the engine information, determine the corresponding engine type, and according to the task information, obtain the data integration rules of the corresponding engine type, including: according to the engine information, determine the corresponding engine type, and according to the task information, obtain the first data source configuration information and the data source configuration information of the corresponding engine type, and generate a data reading statement according to the first data source configuration information and the data source configuration information; according to the engine information, determine the corresponding engine type, and according to the task information, obtain the data conversion configuration information and the data conversion association configuration information of the corresponding engine type, and generate a data conversion statement according to the data conversion configuration information and the data conversion association configuration information; according to the engine information, determine the corresponding engine type, and according to the task information, obtain the second data source configuration information and the data destination configuration information of the corresponding engine type, and generate a data writing statement according to the second data source configuration information and the data destination configuration information.

[0007] A data integration method provided by the present invention, the task information includes a first data source, target data, and a second data source; according to the task information, obtain the first data source configuration information and the data source configuration information of the corresponding engine type, including: according to the first data source, obtain the first data source configuration information of the corresponding engine type, and the first data source configuration information is previously configured based on the connection information with the corresponding first data source and the data reading strategy; according to the target data read from the first data source, obtain the data source configuration information of the corresponding engine type, and the data source configuration information is previously configured based on the data source information and the data reading condition information of the target data to be read from the corresponding first data source; According to the task information, obtain the data conversion configuration information and the data conversion association configuration information of the corresponding engine type, including: according to the target data, obtain the data conversion configuration information of the corresponding engine type, and the data conversion configuration information is previously configured according to the data conversion method corresponding to the target data; according to the data conversion configuration information, obtain the data conversion association configuration information, and the data conversion association configuration information is previously configured according to other data conversion methods corresponding to the corresponding data conversion configuration information; According to the task information, obtain the second data source configuration information and the data destination configuration information of the corresponding engine type, including: according to the second data source, obtain the second data source configuration information of the corresponding engine type, and the second data source configuration information is previously configured based on the connection information with the second data source and the data writing strategy; according to the target data to be written into the second data source, obtain the data destination configuration information of the corresponding engine type, and the data destination configuration information is configured based on the storage location information and the data storage condition information of the target data to be written into the corresponding second data source.

[0008] A data integration method provided by the present invention, which executes a data integration statement, includes: after generating a data reading statement, executing the data reading statement to establish a connection with a first data source and read target data in the first data source; after generating a corresponding data conversion statement, executing the data conversion statement to perform data conversion on the target data; and after generating a data writing statement, executing the data writing statement to establish a connection with a second data source and write the target data after data conversion into the second data source.

[0009] A data integration method provided by the present invention, which obtains a data integration rule corresponding to an engine type according to task information, further includes: after generating a data reading statement, a data writing statement, and a data writing statement, splicing the data reading statement, the data conversion statement, and the data writing statement in a preset splicing order to obtain a data integration statement; wherein the preset splicing order is used to define the splicing order of the data reading statement, the data conversion statement, and the data writing statement; or, after generating any two of the data writing statement, the data conversion statement, and the data writing statement, splicing the two generated statements in a preset splicing order to obtain a spliced statement, and after generating the remaining one of the data writing statement, the data conversion statement, and the data writing statement, synthesizing the spliced statement and the data writing statement in a preset splicing order to obtain a data integration statement.

[0010] A data integration method provided by the present invention, which executes a data integration statement, includes: executing the data integration statement to establish a connection with a first data source and read target data in the first data source; performing data conversion on the target data; establishing a connection with a second data source and writing the target data after data conversion into the second data source.

[0011] A data integration method provided by the present invention, which executes a data integration statement, includes: based on the engine type, loading a preset loading rule of a corresponding data integration engine, and using the data integration engine after loading the corresponding preset loading rule to execute the data integration statement, and the preset loading rule is selected in advance based on the engine type.

[0012] The present invention also provides a data integration device, including: a data acquisition module, which acquires data integration configuration information. The data integration configuration information includes task information and engine information for executing the task. The task information is a data integration task configured based on transferring target data from a first data source to a second data source. The first data source is used to represent the database providing the target data, and the second data source is used to represent the database to receive the target data; a statement generation module, which determines the corresponding engine type according to the engine information, and acquires the data integration rule of the corresponding engine type and generates a data integration statement according to the task information. The data integration rule is a logical rule configured based on the corresponding task; a data integration module, which executes the data integration statement to read the target data from the first data source, perform data conversion on the target data, and write it into the second data source.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the data integration method as described in any one of the above.

[0014] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the data integration method as described in any one of the above.

[0015] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the data integration method as described in any one of the above.

[0016] The data integration method, device, equipment, and product provided by the present invention clarify the details of the data integration task to be performed through the acquired data integration configuration information, thus laying a foundation for subsequent data integration work. Further, the engine type is determined according to the data integration configuration information to ensure that the subsequent acquired data integration rules are closely combined with the characteristics and functions of the selected engine, thereby giving play to the advantages of the corresponding data integration engine in data processing. And the data integration statement that can be actually executed is generated according to the acquired data integration rule, so as to facilitate the subsequent execution of the data integration statement based on the selected engine, realize the unified management and dynamic switching of the data integration engine, ensure that the data is accurately processed according to the pre-set business requirements, make the data integration process more standardized and standardized, ensure the accuracy and logic of data processing, improve the quality and efficiency of data integration, and provide strong data support for various subsequent business activities based on these data. Description of the Drawings

[0017] To more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0018] Figure 1 is one of the schematic flowcharts of the data integration method provided by the present invention; Figure 2 is the second of the schematic flowcharts of the data integration method provided by the present invention; Figure 3 is the schematic structural diagram of the data integration device provided by the present invention; Figure 4 is the schematic structural diagram of the electronic device provided by the present invention. Detailed implementation manners

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention with reference to the drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0020] Figure 1 is the schematic flowchart of the data integration method provided by the present invention. As Figure 1 shown, the method includes: S11. Obtain data integration configuration information, where the data integration configuration information includes task information and engine information for executing the task. The task information is a data integration task configured based on transferring target data from a first data source to a second data source. The first data source is used to represent the database providing the target data, and the second data source is used to represent the database to receive the target data; S12. Determine the corresponding engine type according to the engine information, and obtain the data integration rules of the corresponding engine type and generate a data integration statement according to the task information; wherein, the data integration rules are logical rules configured based on the corresponding task; S13. Execute the data integration statement to read the target data from the first data source, perform data conversion on the target data, and write it into the second data source.

[0021] It should be noted that the execution subject of this specification is an integration platform. The step numbers "S1N" in this specification do not represent the sequence of the data integration method. The following specifically combines Figure 2 to describe the data integration method of the present invention.

[0022] Step S11: Obtain data integration configuration information, which includes task information and engine information for executing the task. The task information is a data integration task configured based on transferring target data from a first data source to a second data source. The first data source is used to represent the database providing the target data, and the second data source is used to represent the database to receive the target data.

[0023] It should be added that the task information is used to indicate reading target data from the first data source and performing data transformation on the read target data to write the target data after data transformation into the second data source. Accordingly, the task information includes the first data source, target data, data source, data reading method, data transformation method, second data source, data storage location, data writing method, etc., which are specifically determined according to the user's data integration requirements. The user configures and submits the data integration configuration information based on the data integration requirements, and the platform receives the data integration configuration information.

[0024] In addition, the first data source and the second data source can be of the same data source type or different data source types. The data source type can be a relational database management system (MySQL), a data warehouse (Hive), Oracle's relational database management system (Oracle), a message queue (kafka), etc., which can be specifically selected according to actual requirements and are not further limited here.

[0025] For example, the task information can be to transfer data from MySQL to Hive (mysql2hive); or, the task information can be to transfer, transform, and import data from a MySQL database to an Oracle database (Oracle); or, the task information can be to send the data in the MySQL database to the Kafka message queue (mysql2kafka); or, the task information can be to synchronize MySQL data to the Hive data warehouse while capturing data changes in MySQL (such as new records, modified records, deleted records, etc.) and synchronizing these changes to Hive in real-time or near real-time (mysql2hive_cdc); or, the task information can be to import the data in the Kafka message queue into the Hive data warehouse (kafka2hive).

[0026] In an optional embodiment, the engine information can be determined according to the data integration engine actually selected by the user. The data integration engine includes a big data distributed streaming computing engine (Flink), a big data computing engine (Spark), a heterogeneous data source synchronization engine (DataX), a streaming data integration engine (FLinkCDC), and a data integration engine (Bitsail), etc., which are not further limited here.

[0027] Step S12: Determine the corresponding engine type according to the engine information, and obtain the data integration rules of the corresponding engine type according to the task information, and generate data integration statements; wherein, the data integration rules are logical rules configured based on the corresponding task.

[0028] In this embodiment, the data integration statements include data reading statements, data conversion statements, and data writing statements. Correspondingly, referring to Figure 2 , determine the corresponding engine type according to the engine information, and obtain the data integration rules of the corresponding engine type according to the task information, including: determine the corresponding engine type according to the engine information, and obtain the first data source configuration information and data source configuration information of the corresponding engine type according to the task information, and generate a data reading statement according to the first data source configuration information and the data source configuration information; determine the corresponding engine type according to the engine information, and obtain the data conversion configuration information and data conversion association configuration information of the corresponding engine type according to the task information, and generate the corresponding data conversion statement according to the data conversion configuration information and the data conversion association configuration information; determine the corresponding engine type according to the engine information, and obtain the second data source configuration information and data destination configuration information of the corresponding engine type according to the task information, and generate a data writing statement according to the second data source configuration information and the data destination configuration information.

[0029] It should be noted that, according to the engine information, determine the corresponding engine type, so as to obtain the corresponding data integration rules according to the corresponding engine type, thereby realizing the compatibility with multiple data integration engines and realizing the unified management and operation of multiple data integration engines.

[0030] Specifically, the task information includes a first data source, target data, and a second data source.

[0031] Correspondingly, obtain the first data source configuration information and data source configuration information of the corresponding engine type according to the task information, including: obtain the first data source configuration information of the corresponding engine type according to the first data source, and the first data source configuration information is previously configured based on the connection information with the corresponding first data source and the data reading strategy; obtain the data source configuration information of the corresponding engine type according to the target data read from the first data source, and the data source configuration information is previously configured based on the data source information and data reading condition information of the target data to be read from the corresponding first data source.

[0032] It should be added that when the task information is to migrate MySQL data to Hive, if the engine type is Flink, the corresponding first data source configuration information can be obtained through mysqlConfigForFlink; if the engine type is Spark, the corresponding first data source configuration information can be obtained through mysqlConfigForSpark; and the corresponding data source configuration information can be obtained through sourceConfig. In addition, the first data source configuration information includes the connection information to the first data source and the data reading strategy. The connection information of the first data source includes the connection address, port number, username, password, etc. of the first data source, and the data reading strategy includes data reading methods such as full table reading or reading according to conditions.

[0033] It should be noted that the first data source configuration information obtained through mysqlConfigForFlink or mysqlConfigForSpark enables the corresponding Flink or Spark engine to successfully connect to the MySQL database. Based on mysqlConfigForFlink or mysqlConfigForSpark, the specific data source can be further clarified through sourceConfig. For example, sourceConfig determines from which specific databases and tables in the MySQL database to read data according to the task information, and defines whether to perform filtered reading according to specific conditions, such as only reading specific data within a specific time period or data that meets certain conditions, so as to obtain more accurate data that meets the requirements.

[0034] In addition, according to the task information, obtain the data conversion configuration information and data conversion association configuration information corresponding to the engine type, including: obtain the data conversion configuration information corresponding to the engine type according to the target data, and the data conversion configuration information is configured in advance according to the data conversion method corresponding to the target data; obtain the data conversion association configuration information according to the data conversion configuration information, and the data conversion association configuration information is configured in advance according to other data conversion methods associated with the corresponding data conversion configuration information.

[0035] It should be added that, in the case where the task information is to migrate MySQL data to Hive, if the data conversion method is encryption, the corresponding data conversion configuration information can be obtained through EncTransformConfig, and the corresponding data conversion related configuration information can be obtained through transformConfig. In addition, the data conversion configuration information includes configuration information such as encryption, routing and diversion. For example, if the data conversion method is encryption, the corresponding data conversion configuration information needs to include detailed information on how to encrypt the data, such as the selection of algorithms, the management of encryption keys, and the specific location of encryption operations in the entire data processing link. The data conversion related configuration information includes data format conversion, data cleaning, data standardization, data enrichment and derivation, etc. The data conversion related configuration information needs to be configured according to the data conversion configuration information. For example, after EncTransformConfig configures the corresponding data conversion configuration information according to the encryption configuration, transformConfig needs to consider how to convert the format of the encrypted data or how to associate and process it with other unencrypted data, etc., so as to determine the corresponding data conversion related configuration information, so as to transition from a single security protection perspective to a more comprehensive data processing and utilization perspective. Under the premise of ensuring data security, TransformConfig is used to further optimize and utilize data to meet a wider range of business needs.

[0036] In addition, according to the task information, the second data source configuration information and data destination configuration information of the corresponding engine type are obtained, including: according to the second data source, the second data source configuration information of the corresponding engine type is obtained, and the second data source configuration information is previously configured based on the connection information with the second data source and the data writing strategy; according to the target data to be written into the second data source, the data destination configuration information of the corresponding engine type is obtained, and the data destination configuration information is configured based on the storage location information and data storage condition information of the target data to be written into the corresponding second data source.

[0037] It should be added that, when the task information is to migrate MySQL data to Hive, if the engine type is Flink, the corresponding second data source configuration information can be obtained through HiveConfigForFlink, and if the engine type is Spark, the corresponding second data source configuration information can be obtained through HiveConfigForSpark; and the corresponding data destination configuration information can be obtained through sinkConfig. In addition, the second data source configuration information includes the connection information and data writing strategy with the second data source. The connection information of the second data source includes the connection address, port number, user name and password of the second data source, and the data writing strategy includes batch writing, line by line writing and other data writing methods.

[0038] It should be noted that through HiveConfigForFlink and HiveConfigForSpark, the corresponding configuration information is set for the Flink and Spark engines to interact with the Hive data warehouse, as well as the special configurations based on the interaction between a specific engine and Hive, such as the data writing method and how to handle data format differences, so as to ensure that the engine can effectively exchange and operate data with Hive; then through sinkConfig, on the basis of HiveConfigForFlink or HiveConfigForSpark, the destination of the target data after data conversion is refined and supplemented, such as the data storage location, data storage format, and the association with other storage systems or data processing processes, etc., so as to transition from the interaction configuration between a specific engine and Hive to a more general data storage destination configuration, making the planning of data storage more complete and specific, and better meeting complex data processing and storage requirements.

[0039] Step S13, execute the data integration statement to read the target data from the first data source, and write the target data after data conversion into the second data source.

[0040] In a possible implementation, executing the data integration statement includes: after generating the data reading statement, execute the data reading statement to establish a connection with the first data source and read the target data in the first data source; after generating the corresponding data conversion statement, execute the data conversion statement to perform data conversion on the target data; after generating the data writing statement, execute the data writing statement to establish a connection with the second data source and write the target data after data conversion into the second data source.

[0041] It should be added that a connection with the first data source can be established through the first connection service (connector-service), and the corresponding first data source service can be connected through the interaction interface (databus-connector-service[interface]) to read the corresponding target data from the first data source through the first data source service; then the target data is subjected to data conversion through the data conversion service (transform-service) through the interaction interface; finally, a connection with the second data source is established through the second connection service (connector-service), and the target data after data conversion is written into the corresponding second data source through the second data source service. For example, if the first data source is MySQL, the corresponding first data source service is mysql-service, the second data source is Hive, and the second data source service is hive-service.

[0042] In an alternative embodiment, obtaining the data integration rules corresponding to the engine type according to the task information further includes: after generating the data read statement, the data conversion statement, and the data write statement, splicing the data read statement, the data conversion statement, and the data write statement in a preset splicing order to obtain a data set integration statement; wherein the preset splicing order is used to define the splicing order of the data read statement, the data conversion statement, and the data write statement. For example, if the preset splicing order is the splicing order of the data read statement, the data conversion statement, and the data write statement from first to last, correspondingly, splicing the data read statement, the data conversion statement, and the data write statement in sequence according to the preset splicing order to obtain a data set integration statement.

[0043] In an alternative embodiment, obtaining the data integration rules corresponding to the engine type according to the task information further includes: after generating any two of the data write statement, the data conversion statement, and the data write statement, splicing the two generated statements in a preset splicing order to obtain a spliced statement, and after generating the remaining one of the data write statement, the data conversion statement, and the data write statement, synthesizing the spliced statement and the data write statement in a preset splicing order to obtain a data set integration statement.

[0044] Similarly, if the preset splicing order is the splicing order of the data read statement, the data conversion statement, and the data write statement from first to last, assuming the two generated statements are the data write statement and the data conversion statement, or the data conversion statement and the data write statement, then synthesizing the spliced statement and the data write statement in a preset splicing order can be splicing the data write statement after the end position of the spliced statement obtained by splicing the data write statement and the data conversion statement, or splicing the data read statement before the start position of the spliced statement obtained by splicing the data conversion statement and the data write statement.

[0045] In addition, assuming the two generated statements are the data write statement and the data write statement, and the spliced statement is splicing the data write statement after the end position of the data write statement, then synthesizing the spliced statement and the data write statement in a preset splicing order can be inserting the data conversion statement between the data write statements of the spliced statement, that is, inserting the data conversion statement after the end position of the data write statement of the spliced statement and before the start position of the data write statement.

[0046] Correspondingly, executing the data set integration statement includes: executing the data set integration statement to establish a connection with the first data source and read the target data in the first data source; performing data conversion on the target data; establishing a connection with the second data source and writing the target data after data conversion into the second data source.

[0047] In an optional embodiment, executing the data integration statement includes: based on the engine type, loading a preset loading rule corresponding to the data integration engine, and using the data integration engine after loading the corresponding preset loading rule to execute the data integration statement, where the preset loading rule is selected in advance based on the engine type.

[0048] Specifically, Spark, as a distributed computing framework based on in-memory computing, is good at processing batch processing and interactive queries of large-scale data. Therefore, the corresponding preset loading rules need to focus on the allocation of computing resources and data partitioning; Flink is an open-source framework that unifies stream processing and batch processing, focusing on real-time data stream processing and emphasizing low latency and high throughput. Therefore, the corresponding preset loading rules focus on aspects such as the initialization task parallelism, memory information, and checkpoint information.

[0049] Accordingly, loading the preset loading rule corresponding to the data integration engine based on the engine type includes: if the engine type is Spark, loading the preset loading rule corresponding to the data integration engine, where the preset loading rule includes resources and the number of data partitions; if the engine type is Flink, loading the preset loading rule corresponding to the data integration engine, where the preset loading rule includes the initialization task parallelism, memory information, and checkpoint information.

[0050] It should be added that by defining a standardized interaction interface (databus-connector-service[interface]) and protocol, it is convenient for data integration engines of different engine types to access the platform through the adapter pattern, reducing the dependence on the underlying implementation of each engine, simplifying the access and management of the engine, reducing the operation and maintenance costs and complexity of the system, and meeting the needs of future business expansion.

[0051] In summary, the embodiment of the present invention clarifies the details of the data integration task to be performed through the obtained data integration configuration information, thereby laying a foundation for the subsequent data integration work. Further, the engine type is determined according to the data integration configuration information to ensure that the subsequent obtained data integration rules are closely combined with the characteristics and functions of the selected engine, so as to give play to the advantages of the corresponding data integration engine in data processing, and generate executable data integration statements according to the obtained data integration rules, so as to facilitate the subsequent execution of the data integration statements based on the selected engine, realize the unified management and dynamic switching of the data integration engine, ensure that the data is processed accurately according to the pre-set business requirements, make the data integration process more standardized and standardized, ensure the accuracy and logic of data processing, improve the quality and efficiency of data integration, and provide strong data support for various business activities carried out based on these data subsequently.

[0052] The data integration device provided by the present invention will be described below. The data integration device described below can be correspondingly referred to the data integration method described above.

[0053] Figure 3 A schematic structural diagram of a data integration device is shown. The device includes: A data acquisition module 31 that acquires data integration configuration information. The data integration configuration information includes task information and engine information for executing the task. The task information is a data integration task configured based on transferring target data from a first data source to a second data source. The first data source is used to represent the database providing the target data, and the second data source is used to represent the database to receive the target data. A statement generation module 32 that determines the corresponding engine type according to the engine information, and acquires the data integration rule of the corresponding engine type and generates a data integration statement according to the task information. The data integration rule is a logical rule configured based on the corresponding task. A data integration module 33 that executes the data integration statement to read the target data from the first data source, perform data conversion on the target data, and write it into the second data source.

[0054] In this embodiment, the data integration statement includes a data reading statement, a data conversion statement, and a data writing statement. Correspondingly, the statement generation module 32 includes: a first statement generation unit that determines the corresponding engine type according to the engine information, acquires the first data source configuration information and data source configuration information of the corresponding engine type according to the task information, and generates a data reading statement according to the first data source configuration information and data source configuration information; a second statement generation unit that determines the corresponding engine type according to the engine information, acquires the data conversion configuration information and data conversion association configuration information of the corresponding engine type according to the task information, and generates a corresponding data conversion statement according to the data conversion configuration information and data conversion association configuration information; a third statement generation unit that determines the corresponding engine type according to the engine information, acquires the second data source configuration information and data destination configuration information of the corresponding engine type according to the task information, and generates a data writing statement according to the second data source configuration information and data destination configuration information.

[0055] Specifically, the task information includes a first data source, target data, and a second data source. Correspondingly, the first statement generation unit includes: a first information configuration subunit, which obtains first data source configuration information corresponding to the engine type according to the first data source, and the first data source configuration information was previously configured based on the connection information and data reading strategy corresponding to the first data source; a second information configuration subunit, which obtains data source configuration information corresponding to the engine type according to the target data read from the first data source, and the data source configuration information was previously configured based on the data source information and data reading condition information of the target data to be read from the corresponding first data source.

[0056] In addition, the second statement generation unit includes: a third information configuration subunit, which obtains data conversion configuration information corresponding to the engine type according to the target data, and the data conversion configuration information was previously configured according to the data conversion method corresponding to the target data; a fourth information configuration subunit, which obtains data conversion association configuration information according to the data conversion configuration information, and the data conversion association configuration information was previously configured according to other data conversion methods associated with the corresponding data conversion configuration information.

[0057] Furthermore, the third statement generation unit includes: a fifth information configuration subunit, which obtains second data source configuration information corresponding to the engine type according to the second data source, and the second data source configuration information was previously configured based on the connection information and data writing strategy with the second data source; a sixth information configuration subunit, which obtains data destination configuration information corresponding to the engine type according to the target data to be written into the second data source, and the data destination configuration information is configured based on the storage location information and data storage condition information of the target data to be written into the corresponding second data source.

[0058] In a possible implementation manner, the data integration module 33 includes: a first statement execution unit, which, after generating a data reading statement, executes the data reading statement to establish a connection with the first data source and read the target data in the first data source; a second statement execution unit, which, after generating a corresponding data conversion statement, executes the data conversion statement to perform data conversion on the target data; a third statement execution unit, which, after generating a data writing statement, executes the data writing statement to establish a connection with the second data source and write the target data after data conversion into the second data source.

[0059] In an alternative embodiment, the statement generation module 32 further includes: a statement splicing unit, which, after generating a data reading statement, a data writing statement, and a data writing statement, splices the data reading statement, the data conversion statement, and the data writing statement in a preset splicing order to obtain a data integration statement; wherein the preset splicing order is used to define the splicing order of the data reading statement, the data conversion statement, and the data writing statement.

[0060] In an alternative embodiment, the statement generation module 32 further includes: a statement splicing unit that, after generating any two of the data writing statement, the data conversion statement, and the data writing statement, splices the two generated statements in a preset splicing order to obtain a spliced statement; and a statement synthesis unit that, after generating the remaining one of the data writing statement, the data conversion statement, and the data writing statement, synthesizes the spliced statement with the data writing statement in a preset splicing order to obtain a dataset synthesis statement.

[0061] Correspondingly, the data integration module 33 includes: executing the dataset synthesis statement to establish a connection with the first data source and read target data in the first data source; performing data conversion on the target data; establishing a connection with the second data source, and writing the target data after data conversion into the second data source.

[0062] In an alternative embodiment, the data integration module 33 further includes: a rule loading unit that, based on the engine type, loads the preset loading rules of the corresponding data integration engine and uses the data integration engine after loading the corresponding preset loading rules to execute the dataset synthesis statement, and the preset loading rules are selected in advance based on the engine type.

[0063] Specifically, the rule loading unit is configured to: if the engine type is Spark, load the preset loading rules of the corresponding data integration engine, and the preset loading rules include resources and the number of data partitions; if the engine type is Flink, load the preset loading rules of the corresponding data integration engine, and the preset loading rules include the initialization task parallelism, memory information, and checkpoint information.

[0064] In summary, the data integration configuration information obtained by the data acquisition module in the embodiments of the present invention clarifies the details of the data integration task to be performed, thereby laying a foundation for subsequent data integration work. Further, the statement generation module determines the engine type according to the data integration configuration information to ensure that the subsequent obtained data integration rules are closely combined with the characteristics and functions of the selected engine, thereby exerting the advantages of the corresponding data integration engine in data processing, and generating an actually executable dataset synthesis statement according to the obtained data integration rules, so as to facilitate the subsequent execution of the dataset synthesis statement by the data integration module based on the selected engine, realize the unified management and dynamic switching of the data integration engine, ensure that the data is accurately processed according to the pre-set business requirements, make the data integration process more standardized and standardized, ensure the accuracy and logic of data processing, improve the quality and efficiency of data integration, and provide strong data support for subsequent various business activities based on these data.

[0065] Figure 4 Illustrates a schematic physical structure diagram of an electronic device, such as Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440. Among them, the processor 410, the communications interface 420, and the memory 430 complete communication with each other through the communication bus 440. The processor 410 may call the logical instructions in the memory 430 to execute a data integration method, which includes: obtaining data integration configuration information, where the data integration configuration information includes task information and engine information for executing the task. The task information is a data integration task configured based on transferring target data from a first data source to a second data source. The first data source is used to represent the database providing the target data, and the second data source is used to represent the database to receive the target data; determining the corresponding engine type according to the engine information, and obtaining the data integration rule of the corresponding engine type and generating a data integration statement according to the task information; where the data integration rule is a logical rule configured based on the corresponding task; executing the data integration statement to read the target data from the first data source, perform data conversion on the target data, and write it into the second data source.

[0066] In addition, when the logical instructions in the above-mentioned memory 430 can be implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0067] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data integration method provided by each of the above methods. The method includes: obtaining data integration configuration information, where the data integration configuration information includes task information and engine information for executing the task. The task information is a data integration task configured based on transferring target data from a first data source to a second data source. The first data source is used to represent the database providing the target data, and the second data source is used to represent the database to receive the target data; determining the corresponding engine type according to the engine information, and obtaining the data integration rule of the corresponding engine type and generating a data integration statement according to the task information; where the data integration rule is a logical rule configured based on the corresponding task; executing the data integration statement to read the target data from the first data source, perform data conversion on the target data, and then write it into the second data source.

[0068] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it realizes the data integration method provided by each of the above methods. The method includes: obtaining data integration configuration information, where the data integration configuration information includes task information and engine information for executing the task. The task information is a data integration task configured based on transferring target data from a first data source to a second data source. The first data source is used to represent the database providing the target data, and the second data source is used to represent the database to receive the target data; determining the corresponding engine type according to the engine information, and obtaining the data integration rule of the corresponding engine type and generating a data integration statement according to the task information; where the data integration rule is a logical rule configured based on the corresponding task; executing the data integration statement to read the target data from the first data source, perform data conversion on the target data, and then write it into the second data source.

[0069] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0070] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data integration method, characterized in that: include: Acquire data integration configuration information, the data integration configuration information including task information and engine information for executing the task, the task information being a data integration task configured based on transferring target data from a first data source to a second data source, the first data source being used to represent a database providing the target data, and the second data source being used to represent a database to receive the target data; According to the engine information, determine the corresponding engine type, and according to the task information, obtain the data integration rule of the corresponding engine type and generate a data integration statement; wherein the data integration rule is a logical rule based on the corresponding task configuration; The data integration statement is executed to read target data from the first data source, and the target data is converted and then written into the second data source.

2. The data integration method according to claim 1, characterized in that: The data integration statements include data reading statements, data conversion statements and data writing statements; Determining a corresponding engine type according to the engine information, and acquiring a data integration rule corresponding to the engine type according to the task information, including: Determine the corresponding engine type according to the engine information, obtain first data source configuration information and data source configuration information of the corresponding engine type according to the task information, and generate a data reading statement according to the first data source configuration information and the data source configuration information; Determine a corresponding engine type according to the engine information, obtain data conversion configuration information and data conversion associated configuration information of the corresponding engine type according to the task information, and generate a data conversion statement according to the data conversion configuration information and the data conversion associated configuration information; According to the engine information, the corresponding engine type is determined, and according to the task information, the second data source configuration information and data destination configuration information of the corresponding engine type are obtained, and according to the second data source configuration information and the data destination configuration information, a data write statement is generated.

3. The data integration method according to claim 2, characterized in that: The task information includes a first data source, target data, and a second data source; According to the task information, first data source configuration information and data source configuration information of the corresponding engine type are obtained, including: According to the first data source, obtaining first data source configuration information corresponding to the engine type, wherein the first data source configuration information is previously configured based on connection information corresponding to the first data source and a data reading strategy; According to the target data read from the first data source, obtain a data source of a corresponding engine type to configure information, wherein the data source configuration information is configured based on the data source information and data reading condition information of the target data to be read from the corresponding first data source; According to the task information, data conversion configuration information and data conversion associated configuration information of the corresponding engine type are obtained, including: According to the target data, acquiring data conversion configuration information corresponding to the engine type, wherein the data conversion configuration information is configured in advance according to the data conversion mode corresponding to the target data; According to the data conversion configuration information, acquiring data conversion associated configuration information, wherein the data conversion associated configuration information is configured in advance according to other data conversion modes associated with the corresponding data conversion configuration information; According to the task information, second data source configuration information and data destination configuration information of the corresponding engine type are obtained, including: According to the second data source, obtaining second data source configuration information corresponding to the engine type, where the second data source configuration information is previously configured based on connection information with the second data source and a data writing strategy; According to the target data to be written into the second data source, data destination configuration information of the corresponding engine type is obtained, wherein the data destination configuration information is configured based on the storage location information and data storage condition information of the target data to be written into the corresponding second data source.

4. The data integration method according to claim 2, characterized in that: Executing the data integration statement includes: After generating the data reading statement, executing the data reading statement to establish a connection with the first data source and read target data in the first data source; After generating the corresponding data conversion statement, executing the data conversion statement to perform data conversion on the target data; After the data write statement is generated, the data write statement is executed to establish a connection with the second data source, and the target data after data conversion is written into the second data source.

5. The data integration method according to claim 2, characterized in that: According to the task information, obtaining a data integration rule corresponding to the engine type also includes: After generating the data reading statement, the data writing statement and the data writing statement, the data reading statement, the data conversion statement and the data writing statement are spliced ​​in a preset splicing order to obtain a data integration statement; wherein the preset splicing order is used to limit the splicing order of the data reading statement, the data conversion statement and the data writing statement; or, After generating any two of the data writing statement, the data conversion statement and the data writing statement, the two generated statements are spliced ​​in a preset splicing order to obtain a spliced ​​statement, and after generating the remaining one of the data writing statement, the data conversion statement and the data writing statement, the spliced ​​statement is synthesized with the data writing statement in the preset splicing order to obtain a data integration statement.

6. The data integration method according to claim 5, characterized in that: Executing the data integration statement includes: Executing the data integration statement to establish a connection with the first data source and read target data in the first data source; Performing data conversion on the target data; A connection is established with the second data source, and the target data after data conversion is written into the second data source.

7. The data integration method according to claim 1, characterized in that: Executing the data integration statement includes: Based on the engine type, a preset loading rule of a corresponding data integration engine is loaded, and the data integration statement is executed using the data integration engine after loading the corresponding preset loading rule, wherein the preset loading rule is selected based on the engine type in advance.

8. A data integration device, characterized in that: include: A data acquisition module acquires data integration configuration information, wherein the data integration configuration information includes task information and information about an engine for executing the task, wherein the task information is a data integration task configured based on transferring target data from a first data source to a second data source, wherein the first data source is used to represent a database that provides the target data, and the second data source is used to represent a database to receive the target data; A statement generation module determines the corresponding engine type according to the engine information, obtains the data integration rule of the corresponding engine type according to the task information, and generates a data integration statement; wherein the data integration rule is a logical rule based on the corresponding task configuration; The data integration module executes the data integration statement to read target data from the first data source, performs data conversion on the target data, and writes the target data into the second data source.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the data integration method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data integration method according to any one of claims 1 to 6 is implemented.

Images