Multi-data-source data synchronization method, system, equipment and medium

Abstract

The invention discloses a multi-data-source data synchronization method, system and device and a medium, belongs to the technical field of data synchronization, and aims to solve the technical problems that the existing data synchronization system supports fewer data sources, and errors are easily caused in the data passing process. Generating a task configuration file; task scheduling: starting an executor according to the concurrent quantity, issuing a task configuration file, and recording log information and error information in a task execution process; task execution: each task execution is an independently executed Java thread, a core-plug architecture is adopted, data source expansion is supported, and support for realizing multiple database types is built in.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization, and in particular to a method, system, device and medium for synchronizing data from multiple data sources. Background Art

[0002] Data synchronization technology has evolved from simple database direct connection synchronization, file synchronization, to synchronization based on database log parsing. Although database direct connection synchronization is simple to configure, it has a greater impact on the business system; file synchronization generates data text files for synchronization, which improves transmission efficiency; database log parsing synchronization achieves quasi-real-time data synchronization by parsing logs such as MySQL binlog, reducing the impact on the business system.

[0003] In order to achieve synchronization between multiple data sources, middleware or services are needed to coordinate the data flow between different data sources. For example, Tencent Cloud TDSQL's multi-source synchronization module supports TDSQL as the source or target end, realizes real-time synchronization and distribution of data to multiple platforms, and uses log-based CDC replication technology and Kafka message queues to ensure data order and consistency.

[0004] The core technologies of data synchronization include data mapping and conversion, conflict resolution, data version management, etc. Data mapping and conversion ensure the consistency of data structure and format; conflict resolution strategy ensures data consistency; data version management tracks data change history and status. With the development of new technologies such as cloud computing and the Internet of Things, data synchronization technology will be applied in more fields, such as finance, logistics, medical care, education, etc., to support the digital transformation and business innovation of enterprises.

[0005] In today's information age, data has become the core asset of enterprises, and data synchronization technology is a key means to ensure data consistency and availability. With the increase in business systems, the need for data synchronization between systems is becoming increasingly frequent, and existing data platforms have limitations in supporting data source types and data synchronization stability. Summary of the invention

[0006] The technical task of the present invention is to provide a multi-data source data synchronization method, system, device and medium to solve the problem that the existing data synchronization system supports fewer data sources and the data transmission process is prone to errors.

[0007] The technical task of the present invention is achieved in the following manner: a method for synchronizing data from multiple data sources, the method being specifically as follows:

[0008] Task construction: create a task form, perform task validity check, and generate a task configuration file;

[0009] Task scheduling: Start the executor according to the number of concurrent tasks, issue the task configuration file, and record the log information and error information of the task execution process;

[0010] Task execution: Each task is executed as an independently executed Java thread, adopts the core-plugin architecture, supports data source expansion, and has built-in support for multiple database types.

[0011] As a preferred option, the task construction is as follows:

[0012] Task form creation: Users enter task configuration through the system front-end page. Task configuration content includes inputting data source connection information, fields, query conditions, target data source connection information, write fields, pre-SQL and post-SQL information;

[0013] Task validity check: After the form task is created, a validity check is performed to check the validity of the input data source and the target data source and whether the fields match each other.

[0014] Generate task configuration: After the task is verified, generate a task configuration file, and parse the task form content stored in the task configuration file in JSON format.

[0015] As a preferred embodiment, the task scheduling is as follows:

[0016] Task division: Count the total number of entries in the database and divide the data area; specifically: count the total number of entries in the database, divide the total number by the number of concurrent connections to get the amount of data to be processed by each task, and then use the sliding window to divide the data area to be processed by each task; after the task division is completed, send the data area information to be processed by each task to the executor;

[0017] Executor management: send task configuration files and data area information; specifically: after the task is created, start the corresponding number of executors according to the set concurrency number, and then send the task configuration file to each executor. In order to facilitate the recording of the task execution process, the task log information and the error information of each node are recorded.

[0018] As a preference, the tasks are performed as follows:

[0019] Receive and send data through the core-plugin architecture;

[0020] Supports data synchronization of multiple database types.

[0021] Preferably, the core in the core-plugin architecture is the central conversion module, which is used to receive data from the plugin and send the data to the target plugin after conversion. In the core-plugin architecture mode, both the input data source and the output data source are regarded as a plugin. When expanding a new database type, you only need to implement the plugin. Each plugin also implements read() and write() to send data to the core and write data sent by the core.

[0022] A multi-data source data synchronization system, the system includes a task construction module, a task scheduling module and a task execution module;

[0023] Among them, the task construction module is used to create task forms, perform task validity checks, and generate task configuration files;

[0024] The task scheduling module is used to start the executor according to the number of concurrent tasks, issue task configuration files, and record log information and error information during task execution;

[0025] The task execution module is used to perform data synchronization; data synchronization includes data source expansion and data conversion; each task execution module is an independently executed Java thread, and the task execution module adopts a core-plugin architecture, supports data source expansion, and has built-in support for multiple database types.

[0026] Preferably, the task building module includes:

[0027] The user interface submodule is used to allow the user to input data source connection information, fields, query conditions, target data source connection information, write fields, pre-SQL and post-SQL information;

[0028] The validity verification mechanism submodule is used to verify the validity of the input data source and the target data source and the field matching consistency;

[0029] The task scheduling module includes:

[0030] The task division mechanism submodule is used to count the total number of entries in the database and divide the data area to be processed by each task through a sliding window;

[0031] The executor management submodule is used to send task configuration files and data area information to each executor.

[0032] Preferably, the task execution module includes:

[0033] The core module is used to receive data from the plugin and send the data to the target plugin after conversion;

[0034] Plugins for various database types, including MySQL, Oracle, MariaDB, MongoDB, King Base, and Hangao database.

[0035] An electronic device comprising: a memory and at least one processor;

[0036] Wherein, the memory stores a computer program;

[0037] The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the multi-data source data synchronization method as described above.

[0038] A computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the multi-data source data synchronization method as described above.

[0039] The multi-data source data synchronization method, system, device and medium of the present invention have the following advantages:

[0040] (1) The present invention solves the problem that the existing data platform supports few data source types and the data synchronization process is unstable. By setting a task construction module, a task scheduling module and a task execution module, the present invention realizes efficient and stable synchronization between multiple data sources.

[0041] (ii) In order to meet the needs of diversified data sources during data synchronization, the present invention has built-in implementations of MySQL, Oracle, MongoDB, Maria DB, and domestic King Base and Hangao databases. Full and incremental data synchronization can be achieved between any two types of databases. At the same time, in order to facilitate developers to expand data sources, developers only need to implement the plugin of the corresponding data source, package it into a jar package and put it in the lib directory to load it into the system, meeting the needs of customized development;

[0042] (III) The present invention aims to build an efficient, reliable and flexible multi-data source data synchronization system to meet the needs of data synchronization in different business scenarios;

[0043] (iv) The present invention can quickly process a large amount of data and improve the efficiency of data synchronization through the concurrent processing capability of the task scheduling module;

[0044] (V) Users can customize task configurations through the front-end page, including data source connection information, field mapping, query conditions, etc., so that the system can adapt to various complex data synchronization requirements;

[0045] (vi) The present invention performs validity checks during the task construction phase and records detailed log information and error information during the task scheduling phase, so as to facilitate timely discovery and resolution of problems;

[0046] (VII) The present invention adopts a core-plugin architecture, so that the system can be easily expanded to support new database types, and only the corresponding plugin needs to be implemented;

[0047] (VIII) The present invention has realized the support for MySQL, Oracle, MariaDB, Mongo DB, King Base, Hangao and other databases, and can meet the needs of different users for different database types;

[0048] (IX) The present invention ensures the stability and reliability of data synchronization through the independent Java thread execution of the task execution module and the design of the core conversion module;

[0049] (10) The present invention can reasonably allocate resources through task division and sliding window technology to ensure that each executor can efficiently process the allocated data area;

[0050] (XI) The present invention supports setting the number of concurrent connections according to actual business needs, so that the data synchronization process can be flexibly adjusted according to system resources and business pressure;

[0051] (XII) The present invention can execute specific SQL statements before and after data synchronization through the support of pre-SQL and post-SQL to ensure data consistency and integrity;

[0052] (XIII) Users can configure and manage tasks through an intuitive front-end page, which reduces operational difficulty and improves user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The present invention is further described below in conjunction with the accompanying drawings.

[0054] Attached Figure 1 A flowchart of a method for synchronizing data from multiple data sources;

[0055] Attached Figure 2 A flowchart constructed for a task. DETAILED DESCRIPTION

[0056] The multi-data source data synchronization method, system, device and medium of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments of the specification.

[0057] Embodiment 1:

[0058] As attached Figure 1 As shown, this embodiment provides a method for synchronizing data from multiple data sources, and the method is specifically as follows:

[0059] S1. Task construction: create a task form, perform task validity check, and generate a task configuration file;

[0060] S2, Task Scheduling: Start the executor according to the number of concurrent tasks, issue the task configuration file, and record the log information and error information of the task execution process;

[0061] S3, Task execution: Each task is executed as an independently executed Java thread, adopts the core-plugin architecture, supports data source expansion, and has built-in support for multiple database types.

[0062] As attached Figure 2 As shown, the task construction in S1 of this embodiment is as follows:

[0063] S101, task form creation: the user enters the task configuration through the system front-end page, and the task configuration content includes inputting data source connection information, fields, query conditions, target data source connection information, write fields, pre-SQL and post-SQL information;

[0064] S102, task validity check: after the form task is created, a validity check is performed to check the validity of the input data source and the target data source and whether the fields match each other;

[0065] S103, generating task configuration: after the task is verified, generating a task configuration file, and parsing the task form content stored in the task configuration file in JSON format.

[0066] The task scheduling in step S2 of this embodiment is specifically as follows:

[0067] S201, task division: count the total number of entries in the database and divide the data area; specifically: count the total number of entries in the database, divide the total number by the number of concurrent entries to obtain the amount of data to be processed by each task, and then divide the data area to be processed by each task through a sliding window; after the task division is completed, send the data area information to be processed by each task to the executor;

[0068] S202, executor management: send task configuration files and data area information; specifically: after the task is created, start the corresponding number of executors according to the set concurrency number, and then send the task configuration file to each executor. In order to facilitate the recording of the task execution process, the task log information and the error information of each node are recorded.

[0069] The task execution in step S3 of this embodiment is specifically as follows:

[0070] S301, receiving and sending data through the core-plugin architecture;

[0071] S302. Support data synchronization of multiple database types.

[0072] The core in the core-plugin architecture in step S301 of this embodiment is a central conversion module, which is used to receive data sent by the plugin and send the data to the target plugin after conversion. In the core-plugin architecture mode, the input data source and the output data source are both regarded as a plugin; when expanding a new database type, only the plugin needs to be implemented. Each plugin implements read() and write() at the same time to send data to the core and write data sent by the core.

[0073] Embodiment 2:

[0074] This embodiment provides a multi-data source data synchronization system, which includes a task construction module, a task scheduling module and a task execution module;

[0075] Among them, the task construction module is used to create task forms, perform task validity checks, and generate task configuration files;

[0076] The task scheduling module is used to start the executor according to the number of concurrent tasks, issue task configuration files, and record log information and error information during task execution;

[0077] The task execution module is used to perform data synchronization; data synchronization includes data source expansion and data conversion; each task execution module is an independently executed Java thread, and the task execution module adopts a core-plugin architecture, supports data source expansion, and has built-in support for multiple database types.

[0078] The task building modules in this embodiment include:

[0079] The user interface submodule is used to allow the user to input data source connection information, fields, query conditions, target data source connection information, write fields, pre-SQL and post-SQL information;

[0080] The validity verification mechanism submodule is used to verify the validity of the input data source and the target data source and the field matching consistency.

[0081] The task scheduling module in this embodiment includes:

[0082] The task division mechanism submodule is used to count the total number of entries in the database and divide the data area to be processed by each task through a sliding window;

[0083] The executor management submodule is used to send task configuration files and data area information to each executor.

[0084] The task execution module in this embodiment includes:

[0085] The core module is used to receive data from the plugin and send the data to the target plugin after conversion;

[0086] Plugins for various database types, including MySQL, Oracle, MariaDB, MongoDB, King Base, and Hangao database.

[0087] Embodiment 3:

[0088] This embodiment also provides an electronic device, including: a memory and a processor;

[0089] Wherein, the memory stores computer-executable instructions;

[0090] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the multi-data source data synchronization method in any embodiment of the present invention.

[0091] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0092] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0093] Embodiment 4:

[0094] This embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the multi-data source data synchronization method in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0095] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0096] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0097] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0098] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for synchronizing data from multiple data sources, characterized in that: The method is as follows: Task construction: create a task form, perform task validity check, and generate a task configuration file; Task scheduling: Start the executor according to the number of concurrent tasks, issue the task configuration file, and record the log information and error information of the task execution process; Task execution: Each task is executed as an independently executed Java thread, adopts the core-plugin architecture, supports data source expansion, and has built-in support for multiple database types.

2. The method for synchronizing data from multiple data sources according to claim 1, characterized in that: The task construction is as follows: Task form creation: Users enter task configuration through the system front-end page. Task configuration content includes inputting data source connection information, fields, query conditions, target data source connection information, write fields, pre-SQL and post-SQL information; Task validity check: After the form task is created, a validity check is performed to check the validity of the input data source and the target data source and whether the fields match each other. Generate task configuration: After the task is verified, generate a task configuration file, and parse the task form content stored in the task configuration file in JSON format.

3. The method for synchronizing data from multiple data sources according to claim 1, characterized in that: The task scheduling is as follows: Task division: Count the total number of entries in the database and divide the data area. Specifically, count the total number of entries in the database, divide the total number by the number of concurrent connections to get the amount of data to be processed by each task, and then use the sliding window to divide the data area to be processed by each task. After the task division is completed, the data area information to be processed by each task is sent to the executor; Executor management: Send task configuration files and data area information; specifically: after the task is created, start the corresponding number of executors according to the set concurrency number, and then send the task configuration file to each executor, record the task log information and the error information of each node.

4. The method for synchronizing data from multiple data sources according to claim 1, characterized in that: The tasks are executed as follows: Receive and send data through the core-plugin architecture; Supports data synchronization of multiple database types.

5. The method for synchronizing data from multiple data sources according to claim 4, characterized in that: The core in the core-plugin architecture is the central conversion module, which is used to receive data from the plugin and send the data to the target plugin after conversion. In the core-plugin architecture mode, both the input data source and the output data source are regarded as a plugin. When expanding a new database type, you only need to implement the plugin. Each plugin also implements read() and write() to send data to the core and write data sent by the core.

6. A multi-data source data synchronization system, characterized in that: The system includes a task construction module, a task scheduling module and a task execution module; Among them, the task construction module is used to create task forms, perform task validity checks, and generate task configuration files; The task scheduling module is used to start the executor according to the number of concurrent tasks, issue task configuration files, and record log information and error information during task execution; The task execution module is used to perform data synchronization; data synchronization includes data source expansion and data conversion; each task execution module is an independently executed Java thread, and the task execution module adopts a core-plugin architecture, supports data source expansion, and has built-in support for multiple database types.

7. The multi-data source data synchronization system according to claim 6, characterized in that: Task building blocks include: The user interface submodule is used to allow the user to input data source connection information, fields, query conditions, target data source connection information, write fields, pre-SQL and post-SQL information; The validity verification mechanism submodule is used to verify the validity of the input data source and the target data source and the field matching consistency; The task scheduling module includes: The task division mechanism submodule is used to count the total number of entries in the database and divide the data area to be processed by each task through a sliding window; The executor management submodule is used to send task configuration files and data area information to each executor.

8. The multi-data source data synchronization system according to claim 6 or 7, characterized in that: The task execution module includes: The core module is used to receive data from the plugin and send the data to the target plugin after conversion; Plugins for various database types, including MySQL, Oracle, MariaDB, MongoDB, King Base, and Hangao database.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the multi-data source data synchronization method according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the multi-data source data synchronization method according to any one of claims 1 to 5.

Images